RE: consumer loses offset
Hi, thanks for your response. Is there any chance the offset is never committed on the "__consumer_offsets” topic, although auto commit is enabled every 5 seconds? We are checking daily and the offset is always set to NULL. Lorenzo Rovere Technology Reply Via Avogadri, 2 31057 - Silea (TV) - ITALY phone: +39 0422 1836521 l.rov...@reply.it www.reply.it -Original Message- From: Luke Chen Sent: 7 October, 2022 2:15 PM To: users@kafka.apache.org Subject: Re: consumer loses offset Hi Lorenzo, Sounds like it is caused by this bug: https://issues.apache.org/jira/browse/KAFKA-13636 If you're not in the versions of fixed version list or newer, please try to upgrade it. Thanks. Luke On Fri, Oct 7, 2022 at 5:36 PM Lorenzo Rovere wrote: > Hi everyone, I have a simple question about Kafka offsets. > > We have 1 producer and 1 consumer. > > Imagine the consumer reading till offset 5 (for example) and then > suddenly stops for some hours. In the meantime the producer keeps > writing messages and the offset of the last message is 10 (always for > example). When we restart the microservice that contains the consumer, > where does it start to read from, if we have auto.offset.reset=latest? > > > > I ask this because one of our costumers complains about starting > reading from 10, thus losing messages between 5 and 10. Is that correct? > > > > Other configs: > enable.auto.commit = true > > auto.commit.interval.ms = 5000 > > log.retention.hours=168 > > offsets.retention.minutes (7d) > > > > We also noticed that on the "__consumer_offsets” topic offset is > always set to NULL > > [consumer_group, topic_name,partition]::NULL > > > > Can you help me understanding what’s happening? Thanks a lot > > > Lorenzo Rovere > > Technology Reply > Via Avogadri, 2 > 31057 - Silea (TV) - ITALY > phone: +39 0422 1836521 > l.rov...@reply.it > www.reply.it > > [image: Technology Reply] >
JDBC Sink connector
Hi
whenever I try to create a new connector it gets timed outcurl -s -X POST -H
"Accept:application/json" -H "Content-Type:application/json" -d
@connector-config.json http://localhost:8083/connectors
{"error_code":500,"message":"Request timed out"}
4:13
where can we see logs related to it ..
4:13
in connect.log, all i can see is
[2022-10-10 10:07:44,524] TRACE [Worker clientId=connect-1,
groupId=connect-demo-group] Submitting connector config write request
oss-atp-connector
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:864)
this is connector-config.json
{
"name": "oss-atp-connector",
"config": {
"connector.class": "JdbcSinkConnector",
"tasks.max": "2",
"connection.url": "jdbc:oracle:thin:@prime?TNS_ADMIN=/opt/jdbc/wallet",
"connection.user": "csdnd",
"connection.password": "Rsdbjdf",
"mode": "incrementing",
"incrementing.column.name": "ID",
"topic":"CTRANSACTION_SINK"
}
}
thanks
Charandeep
This message contains information that may be privileged or confidential and is
the property of the Capgemini Group. It is intended only for the person to whom
it is addressed. If you are not the intended recipient, you are not authorized
to read, print, retain, copy, disseminate, distribute, or use this message or
any part thereof. If you receive this message in error, please notify the
sender immediately and delete all copies of this message.
Re: Apply to be a contributor for Kafka
Hi Divya, Done. Thanks for the interest in Apache Kafka. Luke On Mon, Oct 10, 2022 at 6:20 AM Divya A L wrote: > I’m a developer of kafka, and I want to contribute for the project. Can I > be added as a contributor, as I would like to pick up an issue from the > jira board. > > jira ID : divyaal22 > > Thanks! > Divya >
Re: JDBC Sink connector
Hi Charandeep,
What is in the connect logs? That generally has a full call stack.
On 10/10/2022 7:11 AM, Singh, Charandeep wrote:
Hi
whenever I try to create a new connector it gets timed outcurl -s -X POST -H
"Accept:application/json" -H "Content-Type:application/json" -d
@connector-config.json http://localhost:8083/connectors
{"error_code":500,"message":"Request timed out"}
4:13
where can we see logs related to it ..
4:13
in connect.log, all i can see is
[2022-10-10 10:07:44,524] TRACE [Worker clientId=connect-1,
groupId=connect-demo-group] Submitting connector config write request
oss-atp-connector
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:864)
this is connector-config.json
{
"name": "oss-atp-connector",
"config": {
"connector.class": "JdbcSinkConnector",
"tasks.max": "2",
"connection.url": "jdbc:oracle:thin:@prime?TNS_ADMIN=/opt/jdbc/wallet",
"connection.user": "csdnd",
"connection.password": "Rsdbjdf",
"mode": "incrementing",
"incrementing.column.name": "ID",
"topic":"CTRANSACTION_SINK"
}
}
thanks
Charandeep
This message contains information that may be privileged or confidential and is
the property of the Capgemini Group. It is intended only for the person to whom
it is addressed. If you are not the intended recipient, you are not authorized
to read, print, retain, copy, disseminate, distribute, or use this message or
any part thereof. If you receive this message in error, please notify the
sender immediately and delete all copies of this message.
[ANNOUNCE] New committer: Deng Ziming
Hi All The PMC for Apache Kafka has invited Deng Ziming to become a committer, and we are excited to announce that he has accepted! Ziming has been contributing to Kafka for about three years. He has authored more than 100 patches and helped to review nearly as many. In particular, he made significant contributions to the KRaft project which had a big part in reaching our production readiness goal in the 3.3 release: https://blogs.apache.org/kafka/entry/what-rsquo-s-new-in. Please join me in congratulating Ziming! Thanks for all of your contributions! -- Jason, on behalf of the Apache Kafka PMC
Re: [ANNOUNCE] New committer: Deng Ziming
Congratulations! On Mon, 10 Oct 2022, 11:30 Jason Gustafson, wrote: > Hi All > > The PMC for Apache Kafka has invited Deng Ziming to become a committer, > and we are excited to announce that he has accepted! > > Ziming has been contributing to Kafka for about three years. He has > authored > more than 100 patches and helped to review nearly as many. In particular, > he made significant contributions to the KRaft project which had a big part > in reaching our production readiness goal in the 3.3 release: > https://blogs.apache.org/kafka/entry/what-rsquo-s-new-in. > > Please join me in congratulating Ziming! Thanks for all of your > contributions! > > -- Jason, on behalf of the Apache Kafka PMC >
Re: [ANNOUNCE] New committer: Deng Ziming
Congratulations Ziming! Ismael On Mon, Oct 10, 2022 at 9:30 AM Jason Gustafson wrote: > Hi All > > The PMC for Apache Kafka has invited Deng Ziming to become a committer, > and we are excited to announce that he has accepted! > > Ziming has been contributing to Kafka for about three years. He has > authored > more than 100 patches and helped to review nearly as many. In particular, > he made significant contributions to the KRaft project which had a big part > in reaching our production readiness goal in the 3.3 release: > https://blogs.apache.org/kafka/entry/what-rsquo-s-new-in. > > Please join me in congratulating Ziming! Thanks for all of your > contributions! > > -- Jason, on behalf of the Apache Kafka PMC >
Re: [ANNOUNCE] New committer: Deng Ziming
Congrats Ziming! Regards, Bill On Mon, Oct 10, 2022 at 5:32 PM Ismael Juma wrote: > Congratulations Ziming! > > Ismael > > On Mon, Oct 10, 2022 at 9:30 AM Jason Gustafson > > wrote: > > > Hi All > > > > The PMC for Apache Kafka has invited Deng Ziming to become a committer, > > and we are excited to announce that he has accepted! > > > > Ziming has been contributing to Kafka for about three years. He has > > authored > > more than 100 patches and helped to review nearly as many. In particular, > > he made significant contributions to the KRaft project which had a big > part > > in reaching our production readiness goal in the 3.3 release: > > https://blogs.apache.org/kafka/entry/what-rsquo-s-new-in. > > > > Please join me in congratulating Ziming! Thanks for all of your > > contributions! > > > > -- Jason, on behalf of the Apache Kafka PMC > > >
Re: Metadata Refresh and TimeoutException when MAX_BLOCK_MS_CONFIG set 0
Hi Luke, Thanks for the pointers. Sorry for being late I was out. I would like to propose the following which might be a little different from the Old one: Kafka Producer must distinguish between *broker down state* vs *metadata NOT available* for a given topic. Like the boot-strap server option, many applications (like ours) do not dynamically create topics and publish/subscribe to predefine topics. So, the Kafka producer can have a configuration option to “*predefine-topics*”. When a predefine-topic is configured, Metadata is fetched for those pre-defined topics before the producer is initialized. Also, these pre-defined topics will always guarantee that Metadata will be refreshed before it expires meaning (the metadata cache will expire at X time, then the producer should automatically fetch metadata refresh request X-(3000) ms so the cache will always have the latest mapping of topic partition states continue to fetch everyone seconds till it expires cache X-2000 and X-1000). This will guarantee the non-blocking behavior for pre-defined topics. Blocking behavior is acceptable for topics that are NOT defined ahead of time or dynamic topics. Another configuration we should have is to *drop-message-on-broker-down* (true or false), even if the metadata has expired just DROP the message till the broker is online. Do NOT block the application thread which puts stuff on the Kafka in-memory queue. Of course, the Kafka producer will have to keep track of all brokers and it states the in-memory data structure and update it periodically (when send is a success or ping to port (IP: port) is a success). Luke and others let me know what you think about it. I can write documents if there is interest in the topic. Thanks, Bhavesh On Sun, Sep 25, 2022 at 8:44 PM Luke Chen wrote: > Hi Bhavesh, > > I understand your point. > There was an old KIP with the similar idea which was not accepted by the > community in the end. > Maybe you can try to bring it back to the community again, or try to > propose your own KIP for this idea? > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-286%3A+producer.send%28%29+should+not+block+on+metadata+update > > Thank you. > Luke > > On Sat, Sep 24, 2022 at 6:36 AM Bhavesh Mistry > > wrote: > > > Hello Kafka Team, > > > > I would appreciate any insight into how to distinguish between Brocker > Down > > vs Metadata Refresh not available due to timing issues. > > > > Thanks, > > > > Bhavesh > > > > On Mon, Sep 19, 2022 at 12:50 PM Bhavesh Mistry < > > mistry.p.bhav...@gmail.com> > > wrote: > > > > > Hello Kafka Team, > > > > > > > > > > > > We have an environment where Kafka Broker can go down for whatever > > reason. > > > > > > > > > > > > Hence, we had configured MAX_BLOCK_MS_CONFIG=0 because we wanted to > drop > > > messages when brokers were NOT available. > > > > > > > > > > > > Now the issue is we get data loss due to METADATA not being available > and > > > get this exception “*Topic not present in metadata after 0 > ms.”. > > > *This is due to the fast metadata has expired and the next request to > > > send an event does not have metadata. > > > > > > > > > > > > Why does Kafka have his design? Why can’t Kafka distinguish between > > > Broker down vs metadata refresh not available? Is it reasonable to > > expect > > > metadata would refresh BEFORE it expires so metadata refresh doesn’t > need > > > before it expires? Have Metadata ready before expires? Any particular > > > reason send() has wait for metadata refresh vs background thread that > > > automatically refreshes metadata before it expires, hence send() method > > > never incur wait(). > > > > > > > > > Let me know what suggestion you have to prevent the application thread > > > from blocking (MAX_BLOCK_MS_CONFIG) when the Kafka brokers are DOWN vs > > > metadata is NOT available due to expiration. > > > > > > > > > > > > Let me know your suggestions and what you think about metadata refresh. > > > Should Kafka Producer be proactively refreshing metadata intelligently > > > rather than what the producer does today? > > > > > > > > > > > > > > > > > > Thanks, > > > Bhavesh > > > > > >
Re: consumer loses offset
Hi Lorenzo, In theory, it should commit every 5 secs IF you keep polling the server. But I saw you "stopped" the consumer for some hours, which means the commit won't happen during this period. So, if it exceeds the retention period, it'll get deleted. That's my assumption. You need to check the logs (both client and server) to find the clues. Luke On Mon, Oct 10, 2022 at 5:53 PM Lorenzo Rovere wrote: > Hi, thanks for your response. > > Is there any chance the offset is never committed on the > "__consumer_offsets” topic, although auto commit is enabled every 5 seconds? > We are checking daily and the offset is always set to NULL. > > > > Lorenzo Rovere > > Technology Reply > Via Avogadri, 2 > 31057 - Silea (TV) - ITALY > phone: +39 0422 1836521 > l.rov...@reply.it > www.reply.it > -Original Message- > From: Luke Chen > Sent: 7 October, 2022 2:15 PM > To: users@kafka.apache.org > Subject: Re: consumer loses offset > > Hi Lorenzo, > > Sounds like it is caused by this bug: > https://issues.apache.org/jira/browse/KAFKA-13636 > If you're not in the versions of fixed version list or newer, please try > to upgrade it. > > Thanks. > Luke > > On Fri, Oct 7, 2022 at 5:36 PM Lorenzo Rovere wrote: > > > Hi everyone, I have a simple question about Kafka offsets. > > > > We have 1 producer and 1 consumer. > > > > Imagine the consumer reading till offset 5 (for example) and then > > suddenly stops for some hours. In the meantime the producer keeps > > writing messages and the offset of the last message is 10 (always for > > example). When we restart the microservice that contains the consumer, > > where does it start to read from, if we have auto.offset.reset=latest? > > > > > > > > I ask this because one of our costumers complains about starting > > reading from 10, thus losing messages between 5 and 10. Is that correct? > > > > > > > > Other configs: > > enable.auto.commit = true > > > > auto.commit.interval.ms = 5000 > > > > log.retention.hours=168 > > > > offsets.retention.minutes (7d) > > > > > > > > We also noticed that on the "__consumer_offsets” topic offset is > > always set to NULL > > > > [consumer_group, topic_name,partition]::NULL > > > > > > > > Can you help me understanding what’s happening? Thanks a lot > > > > > > Lorenzo Rovere > > > > Technology Reply > > Via Avogadri, 2 > > 31057 - Silea (TV) - ITALY > > phone: +39 0422 1836521 > > l.rov...@reply.it > > www.reply.it > > > > [image: Technology Reply] > > >
