Re: Request to be added to kafka contributors list

:14 AM, Fan Yang wrote: > Hi Matthia, > > I tried sign out and sign in, still can't find the "Assign" button, my JIRA > ID is fanyan, could you help me set it again? > > Best, > Fan > > > From: Matthias J. Sax > Se

Re: Request to be added to kafka contributors list

复: Request to be added to kafka contributors list Did you sign out and sign in again? On 5/17/24 9:49 AM, Yang Fan wrote: > Thanks Matthias, > > I still can't find "Assign to me" button beside Assignee and Reporter. Could > you help me set it again? > > Best regards, > Fan

回复: Request to be added to kafka contributors list

ntributors list Thanks for reaching out Yang. You should be all set. -Matthias On 5/16/24 7:40 AM, Yang Fan wrote: > Dear Apache Kafka Team, > > I hope this email finds you well. My name is Fan Yang, JIRA ID is fanyan, I > kindly request to be added to the contributors list for Apach

Request to be added to kafka contributors list

Dear Apache Kafka Team, I hope this email finds you well. My name is Fan Yang, JIRA ID is fanyan, I kindly request to be added to the contributors list for Apache Kafka. Being part of this list would allow me to be assigned to JIRA tickets and work on them. Thank you for considering my request

Re: Enable TLS Authentication + Enable External Access using LoadBalancer

: Tuesday, December 21, 2021 at 8:55 AM To: Kafka Users Subject: Re: Enable TLS Authentication + Enable External Access using LoadBalancer Hi Wei Yang, > On Kafka cluster side, how to configure *advertised.listeners* for external access? All 3 LoadBalancer IPs + port, or any 1 LoadBalancer IP + p

Enable TLS Authentication + Enable External Access using LoadBalancer

lancer IPs + port, or any 1 LoadBalancer IP + port? * On external client side, does it need all 3 broker’s certificates? * How does the client know using which certificate while creating request to Kafka cluster? Thanks and regards, Wei Yang Cloud Infrastructure Engineer [/var/

RE: [VOTE] 2.7.0 RC4

unsubsribe

graceful shutdown may be cause loss message

hello, currently CONTROLLED_SHUTDOWN API does not ensure the current leader sends all data to followers which in isr. it may cause data loss if the client does not use ack=all. I think semantic is a little weak. I can accept data loss if the broker crash, but we can try best to avoid data loss

metrics for checking whether a broker throttles requests based on its quota limits?

Hi, Recently we enabled Kafka quota management for our Kafka clusters. We are looking for Kafka metrics that can be used for alerting on whether a Kafka broker throttles requests based on quota. There are a few throttle related metrics on Kafka. But none of them can tell accurately whether the

slow log segment loading during kafka 0.10.2 -> 1.0.1 upgrade

Hi, We are experimenting upgrading our kafka service from 0.10.2 to 1.0.1, and noticed that it was slow in restarting the broker after updating the kafka binary. On a test cluster that have ~120G data on each broker, it took about 20 minutes to load log segments. I am wondering if there is any

Re: 答复: kafka controller setting for detecting broker failure and re-electing a new leader for partitions?

ler/ControllerChannelManager.scala#L222 <https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/controller/ControllerChannelManager.scala#L245> On Thu, Jan 25, 2018 at 12:02 AM, Hu Xi <huxi...@hotmail.com> wrote: > Yu Yang, > > > There does exist

kafka controller setting for detecting broker failure and re-electing a new leader for partitions?

Hi everyone, Recently we had a cluster in which the controller failed to connect to a broker A for an extended period of time. I had expected that the controller would identify the broker as a failed broker, and re-elect another broker as the leader for partitions that were hosted on broker A.

Why doesn't MirrorMaker apply the Exactly once and transactional semantics at 0.11.x?

semantics on production work. Thanks & Have a good day! -- Yang Cui FreeWheel | Beijing +86 1381-1441-685

Re: Topic Creation fails - Need help

please use zk client to check the path：/brokers/ids in ZK 发自我的 iPhone > 在 2017年8月18日，下午3:14，Raghav 写道： > > Hi > > I have a 1 broker and 1 zookeeper on the same VM. I am using Kafka 10.2.1. > I am trying to create a topic using below command: > > "bin/kafka-topics.sh

Re: Topic Creation fails - Need help

your broker is not running 发自我的 iPhone > 在 2017年8月18日，下午3:14，Raghav 写道： > > Hi > > I have a 1 broker and 1 zookeeper on the same VM. I am using Kafka 10.2.1. > I am trying to create a topic using below command: > > "bin/kafka-topics.sh --create --zookeeper

Should set type of log.segment.bytes from INT to LONG

We wish to enlarge the segment file size from 1GB to 2GB, but we found that it will throw the exception as: “Invalid value 2147483648 for configuration log.segment.bytes: Not a number of type INT”. It is because of the overflow of INT, if we set l”og.segment.bytes” to 2147483648. Now we are

Re: Brokers is down by “java.io.IOException: Too many open files”

this should work: # /etc/security/limits.conf * - nofile 65536 On Fri, May 12, 2017 at 6:34 PM, Yang Cui <y...@freewheel.tv> wrote: > Our Kafka cluster is broken down by the problem “java.io.IOException: Too > many open files” three time

Brokers is down by “java.io.IOException: Too many open files”

Our Kafka cluster is broken down by the problem “java.io.IOException: Too many open files” three times in 3 weeks. We encounter these problem on both 0.9.0.1 and 0.10.2.1 version. The error is like: java.io.IOException: Too many open files at

How to rolling restart Kafka cluster?

Hi guys, I have a confusion about whether it will lost data when I rolling restart Kafka cluster. Suppose that Kafka cluster has 4 brokers: A, B, C, D, and 4 partitions of a topic which name is “test-topic” are allocated on these brokers with balance: p1 for A, p2 for B, p3 for C, p4 for

Re: About "org.apache.kafka.common.protocol.types.SchemaException" Problem

Hi All, Have anyone can help answer this question? Thanks a lot! On 26/04/2017, 8:00 PM, "Yang Cui" <y...@freewheel.tv> wrote: Dear All, I am using Kafka cluster 2.11_0.9.0.1, and the new consumer of 2.11_0.9.0.1. When I set the quo

About "org.apache.kafka.common.protocol.types.SchemaException" Problem

Dear All, I am using Kafka cluster 2.11_0.9.0.1, and the new consumer of 2.11_0.9.0.1. When I set the quota configuration is: quota.producer.default=100 quota.consumer.default=100 And I used the new consumer to consume data, then the error happened sometimes:

recommendation on kafka go client : sarama or confluent-kafka-go

Hi all, We need to write a service in Go to consume from kafka. There are two popular kafka clients: sarama and confluent-kafka-go . Sarama client is written purely in Go. This allow it to show the full Go

How many levels can Message Set nest recursively?

I am thinking about that: 1 If a Producer compresses a Message set which is nested more than 2 levels records recursively and sends it to broker, how does broker know which offset should be allocated to this message set without uncompressed all levels and getting the all records? 2

Kafka mirror maker issue

Hi I am doing some experiments on kafka mirror maker. I have two Kafka cluster, one is kafka_2.11-0.9.0.1 with 3 nodes as source cluster, another is kafka_2.10-0.9.0.1 with one node as target cluster. First I mirror the message for topic TEST_TOPIC from source cluster to target cluster with

Re: how to ingest a database with a Kafka Connect cluster in parallel?

Hi Ewen, OK. Thanks a lot for your feedback! Best regards, Yang 2017-01-03 22:42 GMT+01:00 Ewen Cheslack-Postava <e...@confluent.io>: > It's an implementation detail of the JDBC connector. You could potentially > write a connector that parallelizes at that level, but yo

Re: how to ingest a database with a Kafka Connect cluster in parallel?

Hi Ewen, Thanks a lot for your reply. So it means we cannot parallelize ingestion of one table with multiple processes. Is it because of Kafka Connect or the JDBC connector? Have a nice day. Best regards, Yang 2017-01-03 20:55 GMT+01:00 Ewen Cheslack-Postava <e...@confluent.io>: >

how to ingest a database with a Kafka Connect cluster in parallel?

ct2:8083" }, "tasks":[ { "state":"RUNNING", "id":0, "worker_id":"connect3:8083" } ] } This result is the same for all instances. Does it mean the ingestion tasks are not parallelized? Is there anything important I am missing or this type of clustering is simply not supported? Any comments and suggestions are highly appreciated. Have a nice day! Best regards, Yang

RE: consumer with version 0.10.0

No worries. I figure that out already. Thanks all. Best regards, Jack -Original Message- From: Jack Yang [mailto:j...@uow.edu.au] Sent: Monday, 29 August 2016 10:13 AM To: users@kafka.apache.org Subject: RE: consumer with version 0.10.0 Hi there, My fault. When I produce messages

RE: consumer with version 0.10.0

with version 0.10.0 Is anyone producing any (new) messages to the topics you are subscribing to in that consumer? -Jaikiran On Friday 26 August 2016 10:14 AM, Jack Yang wrote: > Hi all, > I am using kafka 0.10.0.1, and I set up my listeners like: > > listeners=PLAINTEXT://myh

consumer with version 0.10.0

Hi all, I am using kafka 0.10.0.1, and I set up my listeners like: listeners=PLAINTEXT://myhostName:9092 then I have one consumer going using the new api. However, I did not see anything return for the api. The log from kafka is: [2016-08-26 14:39:28,548] INFO [GroupCoordinator 0]: Preparing

Kafka running on Ceph

Hi All, Does anyone have any performance metrics running Kafka on Ceph? I briefly gathered at the 2016 Kafka Summit that there's an ongoing work between the Kafka community and RedHat in getting Kafka running successfully on Ceph. Is this correct? If so, what's timeline for that? Thanks

Kafka 0.8.2.1 fatal error under high cpu load

Earlier this year, our cluster crashed multiple times due to the same Fatal error regarding replica offset. In our production cluster, we set replication factor to 3 and min isr to 2. On the day this error occurred we temporarily doubled the number of producers and all kafka brokers were under

Re: Not able to read committed offset on nodes failures with replication setup.

Hi Here is the doc in the ReplicaFetcherThread.handleOffsetOutOfRange which maybe the answer: /** * Handle a partition whose offset is out of range and return a new fetch offset. */ def handleOffsetOutOfRange(topicAndPartition: TopicAndPartition): Long = { val replica =

Re: Support customized security protocol

@Ismael, what's the status of the SASL/PLAIN PR, https://github.com/apache/kafka/pull/341? On Tue, Jan 19, 2016 at 6:25 PM, tao xiao wrote: > The PR provides a new SASL mech but it doesn't provide a pluggable way to > implement user's own logic to do authentication. So I

Re: cannot make another partition reassignment due to the previous partition reassignment failure

, 2015 at 8:15 AM, Yu Yang yuyan...@gmail.com wrote: HI, We have a kafka 0.8.1.1 cluster. Recently I did a partition assignment for some topic partitions in the cluster. Due to broker failure, the partition reassignment failed. I cannot do another partition assignment now, and always get

cannot make another partition reassignment due to the previous partition reassignment failure

HI, We have a kafka 0.8.1.1 cluster. Recently I did a partition assignment for some topic partitions in the cluster. Due to broker failure, the partition reassignment failed. I cannot do another partition assignment now, and always get errors as follows. How can we work around this? I have tried

Re: Does Kafka 0.8.2 producer has a lower throughput in sync-mode, comparing with 0.8.1.x?

...@nmsworks.co.in wrote: 1 . On Mon, Mar 9, 2015 at 1:03 PM, Yu Yang yuyan...@gmail.com wrote: The confluent blog http://blog.confluent.io/2014/12/02/whats-coming-in-apache-kafka-0-8-2/ mentions that the the batching is done whenever possible now. The sync producer, under load

Does Kafka 0.8.2 producer has a lower throughput in sync-mode, comparing with 0.8.1.x?

Hi, Kafka 0.8.1.1 allows us to send a list of messages in sync mode: public void send(ListKeyedMessageK,V messages); I did not find a counter-part of this api in the new producer that is introduced in kafka 0.8.2. It seems that we can use the following method to do sync send in kafka

Re: Camus reads from multiple offsets in parallel?

Thanks for that info Jun. On Tue, Mar 3, 2015 at 3:56 PM, Jun Rao j...@confluent.io wrote: Camus only fetches from different partitions in parallel. Thanks, Jun On Fri, Feb 27, 2015 at 4:24 PM, Yang tedd...@gmail.com wrote: we have a single partition, and the topic contains 300k

Re: cross-colo writing/reading?

place. This way clients can pick a local or a global view of the data, and we can have everyone still stay in their local Kafka clusters. -Todd On Fri, Feb 27, 2015 at 4:21 PM, Yang tedd...@gmail.com wrote: we tested our new application that reads and writes to kafka. at first we

Re: kafka producer does not distribute messages to partitions evenly?

like this : hash(key) % numPartitions. Thanks, Mayuresh On Mon, Mar 2, 2015 at 3:52 PM, Yang tedd...@gmail.com wrote: we have 10 partitions for a topic, and omit the explicit partition param in the message creation: KeyedMessageString, String data = new KeyedMessageString

kafka producer does not distribute messages to partitions evenly?

are empty. what went wrong? thanks Yang

Re: what groupID does camus use?

and is hadoop-etl in some places (when it initializes the offsets) and camus in other places. The name camus is defined in camus.properties in kafka.client.name, so you can override it. Gwen On Tue, Feb 24, 2015 at 12:26 PM, Yang tedd...@gmail.com wrote: I looked at the example camus.properties

cross-colo writing/reading?

techniques/properties to set , or maybe enhancements to be made to the kafka code, that could cater to such situations ? Thanks Yang

Camus reads from multiple offsets in parallel?

group, which a previous thead has clarified to be not the case, i.e. camus does not use group api), it would be very slow. Thanks Yang

what groupID does camus use?

I looked at the example camus.properties, it does not specify this. how could we set the groupId ? thanks Yang

latest messages do not show up on consumers

the consumer on the 3rd line to read out the messages? in fact if I give it --from-beginning, it works fine this is kafka_2.9.2-0.8.1.1 Thanks Yang

Re: latest messages do not show up on consumers

24, 2015 at 1:51 PM, Yang tedd...@gmail.com wrote: in the following code, I just create a new topic, send some messages onto the new topic with console producer, then read it with console consumer. but after the 3rd line, nothing is printed out. export topic=ttnew2 sudo -u kafka bin

Re: [VOTE] 0.8.2.1 Candidate 1

+1 On Feb 18, 2015 7:23 PM, Matt Narrell matt.narr...@gmail.com wrote: +1 On Feb 18, 2015, at 7:56 PM, Jun Rao j...@confluent.io wrote: This is the first candidate for release of Apache Kafka 0.8.2.1. This only fixes one critical issue (KAFKA-1952) in 0.8.2.0. Release Notes for the

Camus add third party decoder jar, doesn't work

. thanks Yang

Immediate Kafka security solution before 0.9 release

Hi All, Before Kafka 0.9 release is available, is there an immediate security solution that we can leverage? I've come across https://github.com/relango/kafka/tree/kafka_security and the IP address filter patch from Kafka 0.8.3, which has not have a set release date. Thanks, Connie

one message consumed by both consumers in the same group?

--. all the parameters are the same then I use console producer to dump some messages into the queue. the message does show up on both ends. where am I doing wrong? thanks Yang

java API equivalent of --from-beginning?

we are trying to achive the equivalent of bin/kafka-console-consumer.sh we tried props.put(auto.offset.reset, smallest); but it still doesn't work. how can I achieve --from-beginning in java code? thanks (we are using 8.1.1)

Re: java API equivalent of --from-beginning?

://www.twitter.com/allthingshadoop / On Fri, Feb 6, 2015 at 5:48 AM, Yang tedd...@gmail.com wrote: we are trying to achive the equivalent of bin/kafka-console-consumer.sh we tried props.put(auto.offset.reset, smallest); but it still doesn't

generics type for Producer and Consumer do not need to match?

? in other words, how do we determine the K,V types on consumer side? thanks Yang

Re: generics type for Producer and Consumer do not need to match?

as it knows how to convert those raw bytes to CK, CV. In the example if CK == byte[] and CV == byte[] it is effectively a no-op conversion. On Thu, Feb 05, 2015 at 12:11:03PM -0800, Yang wrote: in the example https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example we use

does kafka support COMMIT of a batch ?

. does kafka support a BATCH mode with commit ? so basically at the start of my session, I declare start transaction, and after all records have been pulled and sent to kafka, I declare commit . Thanks Yang

Re: does kafka support COMMIT of a batch ?

thanks for the info yang On Mon, Jan 26, 2015 at 2:09 PM, Jay Kreps jay.kr...@gmail.com wrote: We did a relatively complete prototype but it isn't integrated into the mainline code yet and there isn't a target release date. There is rather a lot of testing and compatability work that would

Re: does kafka support COMMIT of a batch ?

well I guess this is what I need https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka anyone knows how far this effort has gone? thanks Yang On Mon, Jan 26, 2015 at 12:06 PM, Yang tedd...@gmail.com wrote: we are pulling from a DB, maybe 100k rows at a time

Re: How to handle broker disk failure

I think the best way is raid not jbod. If one disk of jbod goes wrong , broker shutdown, then it takes long time to recovery . Brokes which run for long time will be more and more leaders of partitions. I/O pressure will be unbalanced. btw, I use kafka 0.8.0-beta1

Re: Best practice for upgrading Kafka cluster from 0.8.1 to 0.8.1.1

: Yu, Are you enabling message compression in 0.8.1 now? If you have already then upgrading to 0.8.2 will not change its behavior. Guozhang On Tue, Dec 2, 2014 at 4:21 PM, Yu Yang yuyan...@gmail.com wrote: Hi Neha, Thanks for the reply! We know that Kafka 0.8.2 will be released soon

Re: Best practice for upgrading Kafka cluster from 0.8.1 to 0.8.1.1

based on the message type when consuming the data. Guozhang On Wed, Dec 3, 2014 at 11:33 AM, Yu Yang yuyan...@gmail.com wrote: Guozhang, We haven't enable message compression yet. In this case, what shall we do when we upgrade to 0.8.2? Must we launch a new cluster, redirect

Best practice for upgrading Kafka cluster from 0.8.1 to 0.8.1.1

Hi, We have a kafka cluster that runs Kafka 0.8.1 that we are considering upgrade to 0.8.1.1. The Kafka documentation http://kafka.apache.org/documentation.html#upgrade mentions upgrading from 0.8 to 0.8.1, but not from 0.8.1 to 0.8.1.1. Will doing one broker at a time by brining the broker

Re: kafka web console running error

do you see error msg Too many open files? it tips you should modify nofile On Tue, Nov 25, 2014 at 1:26 PM, Jun Rao jun...@gmail.com wrote: Which web console are you using? Thanks, Jun On Fri, Nov 21, 2014 at 8:34 AM, Sa Li sal...@gmail.com wrote: Hi, all I am trying to get kafka

Re: ConsumerOffsetChecker returns negative value for log lag

, 2014 at 8:06 PM, Jun Rao jun...@gmail.com wrote: Haven't seen this one before. Are you manually committing offsets yourself? Thanks, Jun On Fri, Oct 17, 2014 at 11:10 PM, Yu Yang yuyan...@gmail.com wrote: Hi, I have a Kafka 0.8.1 cluster. I used the ConsumerOffsetChecker tool

Re: how to do disaster recovery for kafka 0.8 cluster with consumers that uses high-level consumer api?

Thanks, Jun! Yes, I set the topic replication factor to 3. On Sun, Oct 19, 2014 at 8:09 PM, Jun Rao jun...@gmail.com wrote: Did you set the replication factor to be more than 1? Thanks, Jun On Sat, Oct 18, 2014 at 2:32 AM, Yu Yang yuyan...@gmail.com wrote: Hi all, We have a kafka

ConsumerOffsetChecker returns negative value for log lag

Hi, I have a Kafka 0.8.1 cluster. I used the ConsumerOffsetChecker tool to check the lag of consumer groups. I found that for some partition, the tool returns negative value for the lag column. Is this a known issue that has been seen before? I find that the negative value prevents the consumer

Re: How to recover from a disk full situation in Kafka cluster?

through some level of testing now and we will appreciate any feedback/bugs that you can report back. Thanks, Neha On Fri, Jul 18, 2014 at 4:23 PM, Connie Yang cybercon...@gmail.com wrote: Sure, I will try to take a snapshot of the data distribution when it happens next time. Assuming

Re: How to recover from a disk full situation in Kafka cluster?

, is 10 mins. So it effectively does not distribute data evenly across all partitions. If you see the same behavior next time, try to take a snapshot of data distribution across all partitions to verify this theory. Thanks, Neha On Thu, Jul 17, 2014 at 5:43 PM, Connie Yang cybercon...@gmail.com

producer consumer fail if the leader failed-over

Hi List, I'm new to kafka, feel sorry if this has been asked, I didn't figure out my answer by googling, so asked here. Thanks in advance! I'm following the kafka quick start http://kafka.apache.org/documentation.html#quickstart and managed setup a kafka cluster with two brokers, which

Re: producer consumer fail if the leader failed-over

? -- Thanks, Chengwei Thanks, Jun On Thu, May 29, 2014 at 2:38 AM, Chengwei Yang chengwei.yang...@gmail.comwrote: Hi List, I'm new to kafka, feel sorry if this has been asked, I didn't figure out my answer by googling, so asked here. Thanks in advance! I'm following the kafka

Message lost during leader fail-over

need to pass in multiple brokers in --broker-list in the producer. To obtain the new leaders, the producer needs to talk to a live broker to get the new metadata. Thanks, Jun On Thu, May 29, 2014 at 2:38 AM, Chengwei Yang chengwei.yang...@gmail.comwrote: Hi List, I'm new

Re: Message lost during leader fail-over

On Thu, May 29, 2014 at 08:39:47PM -0700, Jun Rao wrote: Have you looked at request.required.acks in http://kafka.apache.org/documentation.html#producerconfigs ? Thanks Jun, that's make sense to me. -- Thanks, Chengwei Thanks, Jun

What happens to Kafka when ZK lost its quorum?

Hi all, Can Kafka producers, brokers and consumers still be processing messages and functioning in their normal states if Zookeeper lost its quorum? Thanks, Connie

What happens to Kafka when ZK lost its quorum or becomes unstable?

Hi, Can the producers, brokers and consumers still be processing messages when their ZK cluster lost its quorum or becomes unstable? I know this is rather general question as it may depends on what configuration these use. So, please enumerate all of those combinations. Thanks, Connie

Re: at-least-once guarantee?

, the consumer has to take on the responsibility to make sure of at least once consumption (if this is required by application), by means such as atomic commit of the offset and data, as you mentioned. Regards Yang On Wed, Aug 7, 2013 at 4:26 PM, Jay Kreps jay.kr...@gmail.com wrote: Yeah I'm not sure how

how can I let camus etl point to hdfs instead of local?

a conf that points to a hdfs setup? I already set my env HADOOP_CONF_DIR to my running hadoop thanks Yang

Re: how can I let camus etl point to hdfs instead of local?

...@googlegroups.com seems less active :) thanks Yang On Tue, Sep 3, 2013 at 6:31 PM, Neha Narkhede neha.narkh...@gmail.comwrote: I'm sorry I'm not too familiar with the camus code base. Have you tried pinging camus_...@googlegroups.com ? Thanks, Neha On Tue, Sep 3, 2013 at 1:40 PM, Yang

at-least-once guarantee?

are able to assert between 2 cases nothing vs =1 delivered, which can be seen as 2 different answers 0 and 1. isn't this as hard as the common Byzantine general problem? Thanks Yang

Re: Kafka throw InvalidMessageException and lost data

-- Yang Zhou（周阳） Department of Computer Science and Engineering Shanghai Jiao Tong University