Re: "processing lull"
- Upstream steps seem to be slowed down by this PTransform (system lag up and elements/sec down) - Unbounded source is PubSubIO Jacob On Sun, Oct 29, 2017 at 9:22 PM, Jacob Marble <jmar...@kochava.com> wrote: > - This does not happen when I don't use the reshuffle hack > - HTTP QPS seems to be improved with reshuffling, but also seems to > burst-and-pause > - The "processing lull" log entry occurs 4 times every 5 minutes > - Right now, I'm guessing that "processing lull" means that a map > operation is taking too long > > Jacob > > On Sun, Oct 29, 2017 at 9:15 PM, Jacob Marble <jmar...@kochava.com> wrote: > >> Good evening- >> >> What should I make of the log warning "processing lull for [instant] in >> state windmill-read" ? >> >> - This happens in a streaming pipeline, in Dataflow. >> - The DoFn that emits the log entry makes HTTP requests to a third-party. >> - This only happens when I added a side input to the PTransform, to >> prevent fusing. >> - The side input is a SingletonView, just an empty string value. >> >> Thanks as usual, >> >> Jacob >> >> Processing lull for PT300.124S in state windmill-read of [step name] >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) >> at com.google.cloud.dataflow.worker.repackaged.com.google.commo >> n.util.concurrent.AbstractFuture.get(AbstractFuture.java:469) >> at com.google.cloud.dataflow.worker.repackaged.com.google.commo >> n.util.concurrent.AbstractFuture$TrustedFuture.get( >> AbstractFuture.java:76) >> at com.google.cloud.dataflow.worker.MetricTrackingWindmillServe >> rStub.getStateData(MetricTrackingWindmillServerStub.java:188) >> at com.google.cloud.dataflow.worker.WindmillStateReader.startBa >> tchAndBlock(WindmillStateReader.java:405) >> at com.google.cloud.dataflow.worker.WindmillStateReader$Wrapped >> Future.get(WindmillStateReader.java:306) >> at com.google.cloud.dataflow.worker.WindmillStateInternals$Wind >> millValue.read(WindmillStateInternals.java:384) >> at com.google.cloud.dataflow.worker.StreamingSideInputFetcher.b >> lockedMap(StreamingSideInputFetcher.java:249) >> at com.google.cloud.dataflow.worker.StreamingSideInputFetcher.s >> toreIfBlocked(StreamingSideInputFetcher.java:186) >> at com.google.cloud.dataflow.worker.StreamingSideInputDoFnRunne >> r.processElement(StreamingSideInputDoFnRunner.java:70) >> at com.google.cloud.dataflow.worker.SimpleParDoFn.processElemen >> t(SimpleParDoFn.java:233) >> at com.google.cloud.dataflow.worker.util.common.worker.ParDoOpe >> ration.process(ParDoOperation.java:48) >> at com.google.cloud.dataflow.worker.util.common.worker.OutputRe >> ceiver.process(OutputReceiver.java:52) >> at com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(Simp >> leParDoFn.java:183) >> > >
Re: "processing lull"
- This does not happen when I don't use the reshuffle hack - HTTP QPS seems to be improved with reshuffling, but also seems to burst-and-pause - The "processing lull" log entry occurs 4 times every 5 minutes - Right now, I'm guessing that "processing lull" means that a map operation is taking too long Jacob On Sun, Oct 29, 2017 at 9:15 PM, Jacob Marble <jmar...@kochava.com> wrote: > Good evening- > > What should I make of the log warning "processing lull for [instant] in > state windmill-read" ? > > - This happens in a streaming pipeline, in Dataflow. > - The DoFn that emits the log entry makes HTTP requests to a third-party. > - This only happens when I added a side input to the PTransform, to > prevent fusing. > - The side input is a SingletonView, just an empty string value. > > Thanks as usual, > > Jacob > > Processing lull for PT300.124S in state windmill-read of [step name] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at com.google.cloud.dataflow.worker.repackaged.com.google. > common.util.concurrent.AbstractFuture.get(AbstractFuture.java:469) > at com.google.cloud.dataflow.worker.repackaged.com.google. > common.util.concurrent.AbstractFuture$TrustedFuture. > get(AbstractFuture.java:76) > at com.google.cloud.dataflow.worker.MetricTrackingWindmillServerSt > ub.getStateData(MetricTrackingWindmillServerStub.java:188) > at com.google.cloud.dataflow.worker.WindmillStateReader. > startBatchAndBlock(WindmillStateReader.java:405) > at com.google.cloud.dataflow.worker.WindmillStateReader$ > WrappedFuture.get(WindmillStateReader.java:306) > at com.google.cloud.dataflow.worker.WindmillStateInternals$ > WindmillValue.read(WindmillStateInternals.java:384) > at com.google.cloud.dataflow.worker.StreamingSideInputFetcher. > blockedMap(StreamingSideInputFetcher.java:249) > at com.google.cloud.dataflow.worker.StreamingSideInputFetcher. > storeIfBlocked(StreamingSideInputFetcher.java:186) > at com.google.cloud.dataflow.worker.StreamingSideInputDoFnRunner. > processElement(StreamingSideInputDoFnRunner.java:70) > at com.google.cloud.dataflow.worker.SimpleParDoFn. > processElement(SimpleParDoFn.java:233) > at com.google.cloud.dataflow.worker.util.common.worker. > ParDoOperation.process(ParDoOperation.java:48) > at com.google.cloud.dataflow.worker.util.common.worker. > OutputReceiver.process(OutputReceiver.java:52) > at com.google.cloud.dataflow.worker.SimpleParDoFn$1.output( > SimpleParDoFn.java:183) >
"processing lull"
Good evening- What should I make of the log warning "processing lull for [instant] in state windmill-read" ? - This happens in a streaming pipeline, in Dataflow. - The DoFn that emits the log entry makes HTTP requests to a third-party. - This only happens when I added a side input to the PTransform, to prevent fusing. - The side input is a SingletonView, just an empty string value. Thanks as usual, Jacob Processing lull for PT300.124S in state windmill-read of [step name] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at com.google.cloud.dataflow.worker.repackaged.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:469) at com.google.cloud.dataflow.worker.repackaged.com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:76) at com.google.cloud.dataflow.worker.MetricTrackingWindmillServerStub.getStateData(MetricTrackingWindmillServerStub.java:188) at com.google.cloud.dataflow.worker.WindmillStateReader.startBatchAndBlock(WindmillStateReader.java:405) at com.google.cloud.dataflow.worker.WindmillStateReader$WrappedFuture.get(WindmillStateReader.java:306) at com.google.cloud.dataflow.worker.WindmillStateInternals$WindmillValue.read(WindmillStateInternals.java:384) at com.google.cloud.dataflow.worker.StreamingSideInputFetcher.blockedMap(StreamingSideInputFetcher.java:249) at com.google.cloud.dataflow.worker.StreamingSideInputFetcher.storeIfBlocked(StreamingSideInputFetcher.java:186) at com.google.cloud.dataflow.worker.StreamingSideInputDoFnRunner.processElement(StreamingSideInputDoFnRunner.java:70) at com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:233) at com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:48) at com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:52) at com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:183)
