Have you tried waiting till the time your output file in HDFS rolls over
(from .tmp to .0)? I have observed this in our case that if you query .tmp
file, it may not show all the records written to it. The reason could be
that the file is still eligible to be written to and output
I hope you got the issue.
its a random issue, the same record is not missing in all the runs. In DT
console we can see at the writer operator processed required records but
the count is not matching with the data written in HDFS.
the sequence of missing tuples , we are not sure because its
When the few tuples are missing are they always the trailing ones or random
in between ones?
Also are you shutting down the app or also killing it (which is when you
see missing tuples?)
On Wed, Aug 9, 2017 at 11:03 PM, chiranjeevi vasupilli
> Hi Venky,
The records are not skipped intentionally, after shutdown the application
we are getting the tuples.
But some times few tuples are missing. we have identified those missing
tuples and tested it. we dont have any conditions to drop those records.
The other issue is its random issue,
You can try adding logs in the operator to see where your records are getting
> On Aug 9, 2017, at 7:08 AM, chiranjeevi vasupilli wrote:
> Hi Team,
> In my use case im seeing some random issue in writing data to HDFS.
> Im using