Topic stopped processing messages

[Sebastian Schmitz]

Hello,

I just had a weird problem in one of my topics.

It accepted messages from producers but consumers couldn't see those 
messages. The producers didn't get any errors when sending messages to 
the topic, but even my test-console-consumer didn't see anything in the 
topic.
Also checking the topic with the kafka-consumer-groups.sh using the 
option of --describe returned a TimeoutException on failing to get 
offsets in 30001ms.


My quick and dirty solution was to delete the topic and with 
auto-creation enabled is was recreated and messages started showing up 
to consumersand --describe working again.


As I have no idea where to start to find the issue of this, and then of 
course open an issue on the website, I'd need some hints on what to look 
for


Thank you

Sebastian


--
DISCLAIMER
This email contains information that is confidential and which 
may be 
legally privileged. If you have received this email in error please 


notify the sender immediately and delete the email.
This email is intended 
solely for the use of the intended recipient and you may not use or 
disclose this email in any way. 

Help in Configuring Kafka on my local Mac

[Santohsh B]

Hi Community,
I am accessing events from the enterorise Kafka topics via Windows command line 
via keytab successfully. However, I have transitioned to macbook this week and 
the same set of commands are not working in Mac. Any documentation on how to 
connect to kafka topic via macbook and using keytab file?

In windows, I first use the below environment variables for setup and then 
proceed with Kafka consuming

0) Specify where jaas.conf file is located.  Need to set an environment 
variable.

KAFKA_OPTS= 
-Djava.security.auth.login.config=C:\myfolder\Kafka_Configs\jaas.conf

KAFKA_REALM= -Djava.security.krb5.conf=krb5.conf

1) Authenticate app id to Kerberos with ticket cached on server

kinit -kt C:\myfolder\Kafka_Configs\keytab_file.keytab  
your_kerberos_principal_n...@domain.com

2) Listen to a message on a topic

/kafka-console-consumer. --bootstrap-server   --topic test --from-beginning --consumer.config 
C:\myfolder\Kafka_Configs\consumer.config



Thanks,

Santohsh

Re: [DISCUSS] Apache Kafka 2.4.0 release

[Manikumar]

Hi all,

I put together a draft release plan with Oct 2019 as the release month and
a list of KIPs that have already been voted:

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=125307901

Here are the dates:

1) KIP Freeze:  Sep 25, 2019 (A KIP must be accepted by this date in order
to be considered for this release)

2) Feature Freeze:  Oct 2, 2019 (Major features merged & working on
stabilization, minor features have PR,
 release branch cut; anything not in this state will be automatically moved
to the next release in JIRA.

3) Code Freeze:  Oct 16, 2019

4) Release Date: Oct 30, 2019 (tentative)

Please plan accordingly for the features you want push into Apache Kafka
2.4.0 release.

Regards,
Manikumar

On Mon, Aug 12, 2019 at 9:08 PM Ismael Juma  wrote:

> Thanks for volunteering Manikumar. +1
>
> Ismael
>
> On Mon, Aug 12, 2019 at 7:54 AM Manikumar 
> wrote:
>
> > Hi all,
> >
> > I would like to volunteer to be the release manager for our next
> time-based
> > feature release (v2.4.0).
> >
> > If that sounds good, I'll post the release plan over the next few days.
> >
> > Thanks,
> > Manikumar
> >
>

Re: kafka client 2.0.1 compatibility with kafka broker 0.10.0.1

[Upendra Yadav]

FYI:

I changed the dependencies of brooklin to use kafka 0.10.0.1.
But compilation is getting failed.
1st failure related test case where it trying to create KafkaServer, where
constructor args mismatch.
I have changed KafkaServer creation API to old one, but another set of
issues:
brooklin having LIKafkaVersion - 1.0.4, and due to this it is downloading
kafka-client 2.0.1.
so I have changed LIKafkaVersion to 0.0.6(where dependency kafka-client
0.10.1.1).
once again, while preparing build test cases are failing: some where
related to kafka topic creation test cases.

So now I have 2 options:
1. move my kafka brokers with 2.0.1. not easy because have multiple
dependencies.
2. or develop my own kafka-mirror or try some other open-source projects.

If you know any good kafka mirror opensource projects then please let me
know.

Thank you.


On Tue, Aug 27, 2019 at 10:11 AM Upendra Yadav 
wrote:

> Hi,
>
> We have kafka brokers with kafka version 0.10.0.1.
> And brooklin(https://github.com/linkedin/brooklin) is using kafka client
> - 2.0.1.
> I'm testing brooklin for kafka mirroring,
>
> Due to kafka version mismatch, Do you see any issue while using brooklin
> kafka-mirror with our kafka brokers?
>
> Or in general: Do you see any issue while using kafka client 2.0.1
> with kafka broker 0.10.0.1?
>
> I went through below link:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
>
> but I'm not sure : is brooklin using such any new APIs that is not
> supported in kafka broker 0.10.0.1.
>
> If someone have time then please look into this and let me know.
> Thank you so much...
>

Byzantine Fault Tolerance Implementation

[Nayak, Soumya R.]

Hi Eliza,

I used the wrong word stability. Rather it should be a byzantine fault tolerant 
network.

Regards,
Soumya

-Original Message-
From: Eliza  
Sent: Tuesday, August 27, 2019 12:55 PM
To: users@kafka.apache.org
Subject: Re: Byzantine Fault Tolerance Implementation

Hi

on 2019/8/27 12:36, Nayak, Soumya R. wrote:
> Zookeeper uses a version of Paxos - Zookeeper atomic broadcast. Is there any 
> plan in future or current in progress where zookeeper will be implemented 
> with a BFT algorithm. This might help to have a more stable distributed 
> environment when we have the cluster across different machines.

Why do you think BFT algorithm will improve the stability of the cluster? :)

regards.
**
This message may contain confidential or proprietary information intended only 
for the use of the 
addressee(s) named above or may contain information that is legally privileged. 
If you are
not the intended addressee, or the person responsible for delivering it to the 
intended addressee,
you are hereby notified that reading, disseminating, distributing or copying 
this message is strictly 
prohibited. If you have received this message by mistake, please immediately 
notify us by replying 
to the message and delete the original message and any copies immediately 
thereafter.

If you received this email as a commercial message and would like to opt out of 
future commercial 
messages, please let us know and we will remove you from our distribution list.

Thank you.~
**
FAFLD

Re: Metrics of BytesRejectedPerSec keeping rising

[李海军]

Hi Liam,

Thank you for you kind reply. I will check it out to see if this helps.

There are amost 5K topics in our cluster with 6 nodes. We collect the
metrics by collectd (https://collectd.org/) and visualize with Grafana. If
all these per topic metrics are collected, will it affect the performance
of the cluster?

Haijun

Liam Clarke  于2019年8月27日周二 下午4:15写道：

> Hi,
>
> The broker exposes per topic error rates, that might help?
>
> https://docs.confluent.io/current/kafka/monitoring.html#per-topic-metrics
>
> Kind regards,
>
> Liam Clarke
>
>
> On Tue, 27 Aug. 2019, 8:07 pm 李海军,  wrote:
>
> > Hi all,
> >
> > We have a kafka cluster consisted of 6 brokers with the version of
> > kafka_2.11-0.10.2.1. These days we saw a continuous increasing on the
> > metrics of "BytesRejectedPerSec" on broker 1. I already knew that this
> > could happen when the message records were too large and raise a
> > "RecordTooLargeException" on the client side, which could be seen in the
> > client's log by using "Callback()" in the send method or just adapting
> the
> > log level to debug.
> >
> > However, from the perspective of the server side, there are no logs about
> > the exceptions at all. Since there are too many producers involved in the
> > clusters and checking all the client logs is out of the question, how
> could
> > I find these certain topics causing "BytesRejectedPerSec" in the server
> > logs?
> >
> > Thanks in advance.
> >
> > Sinsecrely,
> > Haijun
> >
>

Re: Metrics of BytesRejectedPerSec keeping rising

[Liam Clarke]

Hi,

The broker exposes per topic error rates, that might help?

https://docs.confluent.io/current/kafka/monitoring.html#per-topic-metrics

Kind regards,

Liam Clarke


On Tue, 27 Aug. 2019, 8:07 pm 李海军,  wrote:

> Hi all,
>
> We have a kafka cluster consisted of 6 brokers with the version of
> kafka_2.11-0.10.2.1. These days we saw a continuous increasing on the
> metrics of "BytesRejectedPerSec" on broker 1. I already knew that this
> could happen when the message records were too large and raise a
> "RecordTooLargeException" on the client side, which could be seen in the
> client's log by using "Callback()" in the send method or just adapting the
> log level to debug.
>
> However, from the perspective of the server side, there are no logs about
> the exceptions at all. Since there are too many producers involved in the
> clusters and checking all the client logs is out of the question, how could
> I find these certain topics causing "BytesRejectedPerSec" in the server
> logs?
>
> Thanks in advance.
>
> Sinsecrely,
> Haijun
>

Metrics of BytesRejectedPerSec keeping rising

[李海军]

Hi all,

We have a kafka cluster consisted of 6 brokers with the version of
kafka_2.11-0.10.2.1. These days we saw a continuous increasing on the
metrics of "BytesRejectedPerSec" on broker 1. I already knew that this
could happen when the message records were too large and raise a
"RecordTooLargeException" on the client side, which could be seen in the
client's log by using "Callback()" in the send method or just adapting the
log level to debug.

However, from the perspective of the server side, there are no logs about
the exceptions at all. Since there are too many producers involved in the
clusters and checking all the client logs is out of the question, how could
I find these certain topics causing "BytesRejectedPerSec" in the server
logs?

Thanks in advance.

Sinsecrely,
Haijun

Re: Byzantine Fault Tolerance Implementation

[Eliza]

Hi

on 2019/8/27 12:36, Nayak, Soumya R. wrote:

Zookeeper uses a version of Paxos - Zookeeper atomic broadcast. Is there any 
plan in future or current in progress where zookeeper will be implemented with 
a BFT algorithm. This might help to have a more stable distributed environment 
when we have the cluster across different machines.


Why do you think BFT algorithm will improve the stability of the cluster? :)

regards.