Hi Vivek, Can you send across few more details, like the configuration of the operator and some sample data? Also do confirm if the key could have been null?
~ Bhupesh _______________________________________________________ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org On Fri, Mar 23, 2018 at 1:44 AM, Vivek Bhide <vivek.bh...@target.com> wrote: > Hi All, > > While using the TimeBasedDedupOperator for deduping, I see that operator > keeps failing with below NullPointer exception. I don't know if it has > anything to do with configuration. > > I also see that operator is always high on CPU usages. Almost reaching > 100%. > I tried increasing vcores and also memory for operator but its of no use > > 2018-03-22 15:10:10,037 INFO stram.FSRecoveryHandler > (FSRecoveryHandler.java:rotateLog(103)) - Creating > hdfs://littleredns/user/SVDATHDP/datatorrent/apps/ > application_1519410901484_187748/recovery/log > 2018-03-22 15:10:10,056 INFO stram.StreamingContainerParent > (StreamingContainerParent.java:log(170)) - child msg: Stopped running due > to > an exception. java.lang.NullPointerException > at org.apache.hadoop.io.file.tfile.TFile$Writer.append( > TFile.java:387) > at com.datatorrent.lib.fileaccess.TFileWriter.append( > TFileWriter.java:66) > at > org.apache.apex.malhar.lib.state.managed.BucketsFileSystem. > writeBucketData(BucketsFileSystem.java:179) > at > org.apache.apex.malhar.lib.state.managed.IncrementalCheckpointManager. > transferWindowFiles(IncrementalCheckpointManager.java:139) > at > org.apache.apex.malhar.lib.state.managed.IncrementalCheckpointManager$ > 1.run(IncrementalCheckpointManager.java:110) > at java.util.concurrent.Executors$RunnableAdapter. > call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > context: > PTContainer[id=3(container_e3125_1519410901484_187748_01_ > 000012),state=ACTIVE,operators=[PTOperator[id=3,name=dedupeOperator,state= > ACTIVE]]] > 2018-03-22 15:10:10,915 WARN stram.StreamingContainerManager > (StreamingContainerManager.java:processOperatorFailure(1439)) - Operator > failure: PTOperator[id=3,name=dedupeOperator,state=INACTIVE] count: 1 > > Regards > Vivek > > > > > > -- > Sent from: http://apache-apex-users-list.78494.x6.nabble.com/ >
