Confirmed that KafkaIO is consuming and producing as of f4809446b931c02e1dc5da0d86f01faf00b53581.
On Fri, Jun 10, 2016 at 6:40 PM Raghu Angadi <[email protected]> wrote: > KafkaIO reader reports Long.MIN_VALUE > <https://github.com/apache/incubator-beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L1074> > for watermark in this case (it hasn't read any records yet). That is fine, > right? > > On Fri, Jun 10, 2016 at 5:46 PM, Thomas Groh <[email protected]> wrote: > >> If we're reading from an unbounded read and it reports the watermark as >> BoundedWindow#TIMESTAMP_MAX_VALUE, the InProcessRunner won't reinvoke the >> source; the call to start() returning false by itself just means that we >> should call into it later, but the output watermark should still be held by >> the source. >> >> On Fri, Jun 10, 2016 at 4:44 PM, Raghu Angadi <[email protected]> wrote: >> >>> It looks like InProcessPipelineRunner instantiates the source, calls >>> start() on it, and immediately closes it. In this case start() returns >>> false and the runner seems to think the source is done (which is incorrect?) >>> >>> On Fri, Jun 10, 2016 at 4:24 PM, Jesse Anderson <[email protected]> >>> wrote: >>> >>>> Raghu and I spent some time on a hangout looking at this issue. Looks >>>> like there is an issue with unbounded collections with KafkaIO >>>> on InProcessPipelineRunner. >>>> >>>> We changed the code to be a bounded collection with >>>> withMaxNumRecords and used DirectPipelineRunner. That worked and processed >>>> the messages. >>>> >>>> Next, we used InProcessPipelineRunner with a bounded collection. That >>>> worked and processed the messages. >>>> >>>> We changed it back to an unbounded collection >>>> using InProcessPipelineRunner. That didn't work and continued to output the >>>> error messages similar to the ones I've shown on the thread. >>>> >>>> Thanks, >>>> >>>> Jesse >>>> >>>> >>>> On Wed, Jun 8, 2016 at 7:12 PM Jesse Anderson <[email protected]> >>>> wrote: >>>> >>>>> I tried an 0.9.0 broker and I got the same error. Not sure if it makes >>>>> a difference, but I'm using Confluent platform 2.0 and 3.0 for this >>>>> testing. >>>>> >>>>> On Wed, Jun 8, 2016 at 5:20 PM Jesse Anderson <[email protected]> >>>>> wrote: >>>>> >>>>>> Still open to screensharing and resolving over a hangout. >>>>>> >>>>>> On Wed, Jun 8, 2016 at 5:19 PM Raghu Angadi <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> On Wed, Jun 8, 2016 at 1:56 PM, Jesse Anderson < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> [pool-2-thread-1] INFO org.apache.beam.sdk.io.kafka.KafkaIO - >>>>>>>> Reader-0: resuming eventsim-0 at default offset >>>>>>>> >>>>>>> [...] >>>>>>>> >>>>>>> [pool-2-thread-1] INFO org.apache.kafka.common.utils.AppInfoParser - >>>>>>>> Kafka commitId : 23c69d62a0cabf06 >>>>>>>> [pool-2-thread-1] WARN org.apache.beam.sdk.io.kafka.KafkaIO - >>>>>>>> Reader-0: getWatermark() : no records have been read yet. >>>>>>>> [pool-77-thread-1] INFO org.apache.beam.sdk.io.kafka.KafkaIO - >>>>>>>> Reader-0: Returning from consumer pool loop >>>>>>>> [pool-78-thread-1] WARN org.apache.beam.sdk.io.kafka.KafkaIO - >>>>>>>> Reader-0: exception while fetching latest offsets. ignored. >>>>>>>> >>>>>>> >>>>>>> this reader is closed before the exception. The exception is due to >>>>>>> an action during close and can be ignored. The main question is why this >>>>>>> was closed... >>>>>>> >>>>>> >>> >> >
