In my use case im seeing some random issue in writing data to HDFS.
Im using AbstractFileOutPutOperator to write the data to HDFS, and upstream
operator generate the data. In DT console we can see the writer operator
processed 100 tuples but in hdfs we can 80-90 records. When we
kill/shutdown the application we are getting 95-100 records.
Its a random behavior, not same tuple is missing in every run. Please
suggest further without killing/shutdown the app we need to write all
incoming tuples to HFDS.
let me know if you need more information.