Yes, this helped. I have some additional questions.
Does StreamThread have one consumer? (Looks like it, but just want to
confirm)
Is there a separate StreamThread for each topic including the KTable?
If a KTable is a StreamThread and there is a StreamTask for that KTable,
could my buffer be getting filled up, and the mainConsumer for the KTable
be getting paused? I see this code in StreamTask#addRecords.
// if after adding these records, its partition queue's buffered size has
been
// increased beyond the threshold, we can then pause the
consumption for this partition
if (newQueueSize > maxBufferedSize) {
mainConsumer.pause(singleton(partition));
}
Is there any specific logging that I can set to debug or trace that would
help me troubleshoot? I'd prefer not to turn debug and/or trace on for
every single class.
Thanks,
Chad
On Sat, Oct 30, 2021 at 5:20 AM Luke Chen <show...@gmail.com> wrote:
> Hi Chad,
> > I'm wondering if someone can point me to the Kafka streams internal code
> that reads records for the join?
> --> You can check StreamThread#pollPhase, where stream thread (main
> consumer) periodically poll records. And then, it'll process each topology
> node with these polled records in stream tasks (StreamTask#process).
>
> Hope that helps.
>
> Thanks.
> Luke
>
>
> On Sat, Oct 30, 2021 at 5:42 PM Gilles Philippart
> <gilles.philipp...@fundingcircle.com.invalid> wrote:
>
> > Hi Chad, this talk around 24:00 clearly explains what you’re seeing
> >
> https://www.confluent.io/events/kafka-summit-europe-2021/failing-to-cross-the-streams-lessons-learned-the-hard-way/
> > <
> >
> https://www.confluent.io/events/kafka-summit-europe-2021/failing-to-cross-the-streams-lessons-learned-the-hard-way/
> > >
> >
> > Gilles
> >
> > > On 30 Oct 2021, at 04:02, Chad Preisler <chad.preis...@gmail.com>
> wrote:
> > >
> > > Hello,
> > >
> > > I have a stream application that does a KStream to KTable left join. We
> > > seem to be occasionally missing joins (KTable side is null).
> > >
> > > I'm wondering if someone can point me to the Kafka streams internal
> code
> > > that reads records for the join? I've poked around the Kafka code base,
> > but
> > > there is a lot there. I imagine there is some consumer poll for each
> side
> > > of the join, and possibly a background thread for reading the KTable
> > topic.
> > >
> > > I figure there are several possible causes of this issue, and since
> > nothing
> > > is really jumping out in my code, I was going to start looking at the
> > Kafka
> > > code to see if there is something I can do to fix this.
> > >
> > > Thanks,
> > > Chad
> >
> >
> > --
> >
> >
> >
> >
> > Funding Circle Limited is authorised and regulated by the Financial
> > Conduct Authority under firm registration number 722513. Funding Circle
> is
> > not covered by the Financial Services Compensation Scheme. Registered in
> > England (Co. No. 06968588) with registered office at 71 Queen Victoria
> > Street, London EC4V 4AY.
> >
>
