Thanks a lot for the response Samba Surabhi. Can someone please explain, what happens, If I doesn’t set the rotation windows for HDFS write operator ? Its side effects when single file size grows large ??
All the data that is coming will be written to a single file if we don’t set the rotation ? We get around 12 GB of data approximately written to a single file per day, Will such an amount of data into a single file without setting rotation window will cause any issue ? I currently see our application with HDFS write operator continuously failing with some issue… and keep trying again… and creating lots of files in the destination hdfs directory!! Today I observed around 1 TB of data written into a directory where we normally not even get a 20gb of data….!! Can some throw some light into what could be the wrong ? Thanks a lot again !! Regards, Raja. From: Samba Surabhi <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Thursday, August 4, 2016 at 1:25 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: RE: HDFS Write File Operator struggling to start If it is the issue with file size, you can rotate the output file. writer.setAlwaysWriteToTmp(true); writer.setRotationWindows(240); Thanks, Samba Surabhi. ________________________________ From: [email protected]<mailto:[email protected]> To: [email protected]<mailto:[email protected]> Subject: HDFS Write File Operator struggling to start Date: Thu, 4 Aug 2016 14:49:16 +0000 Hi I have a HDFS file write operator in my DAG, which is failing to start a new operator and keep on trying to start one!! It created approx. 800 temporary files in the destination hdfs directory!! How can I fix this issue…. And debug the root cause…. ? All I can see in container log is, File corrupted message…!! Can someone please help me fix this ? Regards, Raja.
