- This does not happen when I don't use the reshuffle hack - HTTP QPS seems to be improved with reshuffling, but also seems to burst-and-pause - The "processing lull" log entry occurs 4 times every 5 minutes - Right now, I'm guessing that "processing lull" means that a map operation is taking too long
Jacob On Sun, Oct 29, 2017 at 9:15 PM, Jacob Marble <jmar...@kochava.com> wrote: > Good evening- > > What should I make of the log warning "processing lull for [instant] in > state windmill-read" ? > > - This happens in a streaming pipeline, in Dataflow. > - The DoFn that emits the log entry makes HTTP requests to a third-party. > - This only happens when I added a side input to the PTransform, to > prevent fusing. > - The side input is a SingletonView, just an empty string value. > > Thanks as usual, > > Jacob > > Processing lull for PT300.124S in state windmill-read of [step name] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at com.google.cloud.dataflow.worker.repackaged.com.google. > common.util.concurrent.AbstractFuture.get(AbstractFuture.java:469) > at com.google.cloud.dataflow.worker.repackaged.com.google. > common.util.concurrent.AbstractFuture$TrustedFuture. > get(AbstractFuture.java:76) > at com.google.cloud.dataflow.worker.MetricTrackingWindmillServerSt > ub.getStateData(MetricTrackingWindmillServerStub.java:188) > at com.google.cloud.dataflow.worker.WindmillStateReader. > startBatchAndBlock(WindmillStateReader.java:405) > at com.google.cloud.dataflow.worker.WindmillStateReader$ > WrappedFuture.get(WindmillStateReader.java:306) > at com.google.cloud.dataflow.worker.WindmillStateInternals$ > WindmillValue.read(WindmillStateInternals.java:384) > at com.google.cloud.dataflow.worker.StreamingSideInputFetcher. > blockedMap(StreamingSideInputFetcher.java:249) > at com.google.cloud.dataflow.worker.StreamingSideInputFetcher. > storeIfBlocked(StreamingSideInputFetcher.java:186) > at com.google.cloud.dataflow.worker.StreamingSideInputDoFnRunner. > processElement(StreamingSideInputDoFnRunner.java:70) > at com.google.cloud.dataflow.worker.SimpleParDoFn. > processElement(SimpleParDoFn.java:233) > at com.google.cloud.dataflow.worker.util.common.worker. > ParDoOperation.process(ParDoOperation.java:48) > at com.google.cloud.dataflow.worker.util.common.worker. > OutputReceiver.process(OutputReceiver.java:52) > at com.google.cloud.dataflow.worker.SimpleParDoFn$1.output( > SimpleParDoFn.java:183) >
