Re: Using startingOffsets latest - no data from structured streaming kafka query
For some reason, after restarting the app and trying again, latest now works as expected. Not sure why it didn’t work before. On Tue, Apr 21, 2020 at 1:46 PM Ruijing Li wrote: > Yes, we did. But for some reason latest does not show them. The count is > always 0. > > On Sun, Apr 19, 2020 at 3:42 PM Jungtaek Lim > wrote: > >> Did you provide more records to topic "after" you started the query? >> That's the only one I can imagine based on such information. >> >> On Fri, Apr 17, 2020 at 9:13 AM Ruijing Li wrote: >> >>> Hi all, >>> >>> Apologies if this has been asked before, but I could not find the answer >>> to this question. We have a structured streaming job, but for some reason, >>> if we use startingOffsets = latest with foreachbatch mode, it doesn’t >>> produce any data. >>> >>> Rather, in logs I see it repeats the message “ Fetcher [Consumer] >>> Resetting offset for partition to offset” over and over again.. >>> >>> However with startingOffsets=earliest, we don’t get this issue. I’m >>> wondering then how we can use startingOffsets=latest as I wish to start >>> from the latest offset available. >>> -- >>> Cheers, >>> Ruijing Li >>> >> -- > Cheers, > Ruijing Li > -- Cheers, Ruijing Li
Re: Using startingOffsets latest - no data from structured streaming kafka query
Yes, we did. But for some reason latest does not show them. The count is always 0. On Sun, Apr 19, 2020 at 3:42 PM Jungtaek Lim wrote: > Did you provide more records to topic "after" you started the query? > That's the only one I can imagine based on such information. > > On Fri, Apr 17, 2020 at 9:13 AM Ruijing Li wrote: > >> Hi all, >> >> Apologies if this has been asked before, but I could not find the answer >> to this question. We have a structured streaming job, but for some reason, >> if we use startingOffsets = latest with foreachbatch mode, it doesn’t >> produce any data. >> >> Rather, in logs I see it repeats the message “ Fetcher [Consumer] >> Resetting offset for partition to offset” over and over again.. >> >> However with startingOffsets=earliest, we don’t get this issue. I’m >> wondering then how we can use startingOffsets=latest as I wish to start >> from the latest offset available. >> -- >> Cheers, >> Ruijing Li >> > -- Cheers, Ruijing Li
Re: Using startingOffsets latest - no data from structured streaming kafka query
Did you provide more records to topic "after" you started the query? That's the only one I can imagine based on such information. On Fri, Apr 17, 2020 at 9:13 AM Ruijing Li wrote: > Hi all, > > Apologies if this has been asked before, but I could not find the answer > to this question. We have a structured streaming job, but for some reason, > if we use startingOffsets = latest with foreachbatch mode, it doesn’t > produce any data. > > Rather, in logs I see it repeats the message “ Fetcher [Consumer] > Resetting offset for partition to offset” over and over again.. > > However with startingOffsets=earliest, we don’t get this issue. I’m > wondering then how we can use startingOffsets=latest as I wish to start > from the latest offset available. > -- > Cheers, > Ruijing Li >
