Re: data missing in AbstractFileOutPutOperator

2017-08-10 Thread Vivek Bhide
Hi Chiru, Have you tried waiting till the time your output file in HDFS rolls over (from .tmp to .0)? I have observed this in our case that if you query .tmp file, it may not show all the records written to it. The reason could be that the file is still eligible to be written to and output

Re: data missing in AbstractFileOutPutOperator

2017-08-10 Thread chiranjeevi vasupilli
I hope you got the issue. its a random issue, the same record is not missing in all the runs. In DT console we can see at the writer operator processed required records but the count is not matching with the data written in HDFS. the sequence of missing tuples , we are not sure because its

Re: data missing in AbstractFileOutPutOperator

2017-08-09 Thread Sanjay Pujare
When the few tuples are missing are they always the trailing ones or random in between ones? Also are you shutting down the app or also killing it (which is when you see missing tuples?) On Wed, Aug 9, 2017 at 11:03 PM, chiranjeevi vasupilli wrote: > Hi Venky, > > The

Re: data missing in AbstractFileOutPutOperator

2017-08-09 Thread chiranjeevi vasupilli
Hi Venky, The records are not skipped intentionally, after shutdown the application we are getting the tuples. But some times few tuples are missing. we have identified those missing tuples and tested it. we dont have any conditions to drop those records. The other issue is its random issue,

Re: data missing in AbstractFileOutPutOperator

2017-08-09 Thread Venkatesh Kottapalli
You can try adding logs in the operator to see where your records are getting skipped. -Venky. > On Aug 9, 2017, at 7:08 AM, chiranjeevi vasupilli wrote: > > Hi Team, > > In my use case im seeing some random issue in writing data to HDFS. > > Im using