Hi Raja, How many partitions do you have for the file output operator and what would you save your data write rate is in bytes/second.
Thanks On Thu, Jul 13, 2017 at 8:13 AM, Raja.Aravapalli <raja.aravapa...@target.com > wrote: > Team, > > > > We have an apex application that is reading from Kafka and wring to HDFS. > > > > The data flow for kafka topic is very huge… say 2500 messages per sec!! > > > > The issue we are facing is: > > > > The operator (which *extends *AbstractFileOutputOperator) is writing to > hdfs is building latency over time and failing eventually. Can someone pls > share your thoughts on how I can handle this ? > > > > > > Thanks a lot. > > > > > > Regards, > > Raja. >
