So I think, You should write to your db the partition and the offset, while initing the real time consumer you'd read from database where to set the consumer starting point, kind-of the "exactly once" programming approach,
בתאריך יום ו׳, 28 במאי 2021, 21:38, מאת Ronald Fenner < rfen...@gamecircus.com>: > That might work if my consumers were in the same process but the db > consumer is a python job running under Airflow and the realtime consumer > wold be running as a backend service on another server. > > Also how would I seed the realtime consumer at startup if the consumer > isn't running which would could be possible if it hit the end stream, > > The db consumer is designed to read until no new message is delivered then > exit till it's next spawned. > > Ronald Fenner > Network Architect > Game Circus LLC. > > rfen...@gamecircus.com > > > On May 28, 2021, at 12:05 AM, Ran Lupovich <ranlupov...@gmail.com> > wrote: > > > > > https://kafka.apache.org/0110/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#seek(org.apache.kafka.common.TopicPartition,%20long) > > > > בתאריך יום ו׳, 28 במאי 2021, 08:04, מאת Ran Lupovich < > ranlupov...@gmail.com > >> : > > > >> While your DB consumer is running you get the access to the partition > >> ${partition} @ offset ${offset} > >> > >> > https://github.com/confluentinc/examples/blob/6.1.1-post/clients/cloud/nodejs/consumer.jswhen > >> setting your second consumers for real time just set them tostart from > that > >> point > >> > >> > >> בתאריך יום ו׳, 28 במאי 2021, 01:51, מאת Ronald Fenner < > >> rfen...@gamecircus.com>: > >> > >>> I'm trying to figure out how to pragmatically read a consumer groups > >>> offset for a topic. > >>> What I'm trying to do is read the offsets of our DB consumers that run > >>> once an hour and batch lad all new messages. I then would have another > >>> consumer that monitors the offsets that have been consumed and consume > the > >>> message not yet loaded storing them in memory to be able to send them > to a > >>> viewer. As messages get consumed they then get pruned from the in > memory > >>> cache. > >>> > >>> Basically I'm wanting to create window on the messages that haven't > been > >>> loaded into the db. > >>> > >>> I've seen ways of getting it from the command line but I'd like to from > >>> with in code. > >>> > >>> Currently I'm using node-rdkafka. > >>> > >>> I guess as a last resort I could shell the command line for the offsets > >>> then parse it and get it that way. > >>> > >>> > >>> Ronald Fenner > >>> Network Architect > >>> Game Circus LLC. > >>> > >>> rfen...@gamecircus.com > >>> > >>> > >
