Announcing the Community Over Code 2024 Streaming Track

Hi all, Community Over Code , the ASF conference, will be held in Denver, Colorado, October 7-10, 2024. The call for presentations is open

Announcing the Community Over Code 2023 Streaming Track

Hi all, Community Over Code , the ASF conference, will be held in Halifax, Nova Scotia October 7-10, 2023. The call for presentations is open now through July 13, 2023. I am one of the co-chairs for the

JOB | Platform Lead (Singapore or London)

to discuss I can be reached using "JamesBTobin (at) Gmail (dot) Com". Kind regards, James

KIP-794 inhibits Partitioner delegation with no fallback option

Prior to KIP-794 it was possible to create a custom Partitioner that could delegate to the DefaultPartitioner. This has been deprecated so we can now only delegate to BuiltInPartitioner.partitionForKey which does not handle a non-keyed message. Hence there is now no mechanism for a custom

Re: Apache Kafka :-Producer-Consumer at different network

Happy to help, Namita. Regards, Jim Ziesig On Fri, Sep 30, 2022 at 7:51 AM Namita Jaokar wrote: > Hi James, > > Thanks alot James for the solution provided. The below configurations did > work when producer and consumer applications were checked with the > scenarios mentione

Re: Apache Kafka :-Producer-Consumer at different network

and hostname to /etc/hosts. I believe your producer is on Windows so it might be best to stick with the IP address. Good luck, Jim Ziesig On Wed, Sep 28, 2022 at 8:21 AM Namita Jaokar wrote: > Hi James, > > As suggested, I did try below steps but still my producer is unable to

Re: Apache Kafka :-Producer-Consumer at different network

Hi Namita, This is likely a listener configuration issue. The bootstrap server setting only gets you so far, your client must be able to contact the server via a listener address. I have seen this occur on a few systems where a hostname could not be identified by Kafka, so the server advertises

Re: mTLS client hostname validation is not working

better. > > Kind regards, > > > Richard Bosch > > Developer Advocate > > Axual BV > > https://axual.com/ > > > On Thu, Sep 15, 2022 at 9:55 PM James Ziesig > wrote: > > > Hi, > > > > I have configured mTLS on a three server Kafka c

mTLS client hostname validation is not working

Hi, I have configured mTLS on a three server Kafka cluster. The servers and clients are all communicating properly, except I am having trouble with client hostname validation when the client is using a cert from a different host. I would expect this to fail on handshake like it does when the

Re: [ANNOUNCE] Apache Kafka 3.2.0

https://cwiki.apache.org/confluence/display/KAFKA/KIP-791%3A+Add+Record+Metadata+to+State+Store+Context> > adds method recordMetadata() to the StateStoreContext, I know that the page has already been published, but should we fix that typo? Thanks! -James > On May 17, 2022, at 9:01 AM, Brun

Re: Unexpected loss of Offsets

, or the 2.8.2 if that's not possible. What can we do to make this happen ASAP? Regards, James. On 29/04/2022, at 14:50, Luke Chen mailto:show...@gmail.com>> wrote: Hi James, So far, v2.8.2 is not planned, yet. And usually, the patch release only has one, that is, v2.8.0, and

Re: Unexpected loss of Offsets

Luke, Do you know if 2.8.2 will be released anytime soon? It appears to be waiting on https://issues.apache.org/jira/browse/KAFKA-13805 for which fixes are available. Regards, James. On 11/04/2022, at 14:22, Luke Chen mailto:show...@gmail.com>> wrote: Hi James, This looks like this

Unexpected loss of Offsets

gic is actually checking for no consumer now and no offsets for 7 days instead? Server and Client are 2.7.2. Sorry I don't have any more detailed server-side logs. Regards, James.

Re: Consumer failure after rolling Broker upgrade

not occur with 2.5.1 and 2.7.0 Clients. This make me suspect that https://issues.apache.org/jira/browse/KAFKA-10793 introduced this issue. Regards, James. On 24/11/2021, at 14:35, James Olsen mailto:ja...@inaseq.com>> wrote: Luke, We did not upgrade to resolve the issue. We simply rest

Re: Consumer failure after rolling Broker upgrade

Luke, We did not upgrade to resolve the issue. We simply restarted the failing clients. Regards, James. On 23/11/2021, at 16:10, Luke Chen mailto:show...@gmail.com>> wrote: Hi James, > Bouncing the clients resolved the issue Could you please describe which version you upgrade to, t

Consumer failure after rolling Broker upgrade

d if any experts out there can identify if there is any weakness in the 2.7.1 version. Regards, James.

Re: How is __consumer_offsets partitioned?

If it's of any value to you, we use the following test to check that we have a well balanced set of consumer group ids. Note that in the code, ConsumerGroups.ALL_GROUPS is simply a list of all our consumer group ids. Spreading the offset commit load across these partitions evenly helps in

Re: [ANNOUNCE] Apache Kafka 2.6.1

Thank you Mickael for running the release. Good job everyone! -James Sent from my iPhone > On Jan 11, 2021, at 2:17 PM, Mickael Maison wrote: > > The Apache Kafka community is pleased to announce the release for > Apache Kafka 2.6.1. > > This is a bug fix release and

Re: [ANNOUNCE] New committer: David Jacot

Congratulations, David! -James > On Oct 16, 2020, at 9:01 AM, Gwen Shapira wrote: > > The PMC for Apache Kafka has invited David Jacot as a committer, and > we are excited to say that he accepted! > > David Jacot has been contributing to Apache Kafka since July 2015 (!) &

Re: [ANNOUNCE] New committer: Chia-Ping Tsai

Congratulations Chia-Ping! -James > On Oct 19, 2020, at 10:24 AM, Guozhang Wang wrote: > > Hello all, > > I'm happy to announce that Chia-Ping Tsai has accepted his invitation to > become an Apache Kafka committer. > > Chia-Ping has been contributing to Kafka since Ma

Re: [ANNOUNCE] New committer: A. Sophie Blee-Goldman

Congratulations, Sophie! -James > On Oct 19, 2020, at 9:40 AM, Matthias J. Sax wrote: > > Hi all, > > I am excited to announce that A. Sophie Blee-Goldman has accepted her > invitation to become an Apache Kafka committer. > > Sophie is actively contributing to Kafk

Re: MM2 max.request.size setting

Even with the errors complaining, it does appear to have worked. Thanks much! James Lavoy Senior Infrastructure Engineer Verizon - Protectwise 1601 Wewatta Street #700 Denver, Co > On Aug 31, 2020, at 2:16 PM, nitin agarwal wrote: > > I am not sure why it is picked in ConsumerCo

Re: MM2 max.request.size setting

. James Lavoy Senior Infrastructure Engineer Verizon - Protectwise 1601 Wewatta Street #700 Denver, Co > On Aug 31, 2020, at 1:08 PM, nitin agarwal wrote: > > If you have clusters like DC1 and DC2 then you can add the following > configuration: > DC1->DC2.producer.overrid

Re: MM2 max.request.size setting

Through the connect-mirror-maker.sh helper script. James Lavoy Senior Infrastructure Engineer Verizon - Protectwise 1601 Wewatta Street #700 Denver, Co > On Aug 31, 2020, at 1:04 PM, nitin agarwal wrote: > > How are you running MM2 ? > > Thanks, > Nitin > > On Mo

MM2 max.request.size setting

. This is the specific value I'm talking about: [2020-08-31 15:42:17,066] INFO ProducerConfig values: max.request.size = 1048576 Does anybody know off-hand the right config line to override this limit? Thank you for your time and attention. James Lavoy Senior Infrastructure Engineer Verizon

Re: Frequent consumer offset commit failures

and also incorporate an equivalent to auto.commit.interval.ms. Regards, James. On 20/06/2020, at 02:15, Ricardo Ferreira mailto:rifer...@riferrei.com>> wrote: James, If I were you I would start investigating what is causing this network drops between your cluster and your consumers. The following messages

Frequent consumer offset commit failures

.. plus many more with increasing 50=OffsetAndMetadata{offset=xxx ...} (the busy Partition). Thanks, James.

Consumer Group got stuck in CompletingRebalance

:20,308] INFO [GroupCoordinator 6]: Stabilized group trackers-etl generation 4987 (__consumer_offsets-29) (kafka.coordinator.group.GroupCoordinator) Has anyone seen this before? Should I file a JIRA ticket? Was there a better process than restarting the broker? -- James Brown Systems Engineer

Re: acks this category properties in Which *.java Defined???

org.apache.kafka.clients.producer.ProducerConfig org.apache.kafka.clients.consumer.ConsumerConfig > On 3/04/2020, at 04:30, 一直以来 <279377...@qq.com> wrote: > > Properties props = new Properties(); props.put("bootstrap.servers", > "localhost:9092"); props.put("acks", "all");

Re: Reg : Slowness in Kafka

ven’t raised an issue for it as I would have no expectation of it being fixed. I just made my client and server versions match and implemented a workaround for the issue I wanted the 2.4.0 client to resolve. Regards, James. On 31/03/2020, at 12:16, JALADY Christophe mailto:christophe.jal...@lapos

Re: Problems when Consuming from multiple Partitions

Resolved by downgrading Client to 2.2.2 and implementing an application level heartbeat on every Producer to avoid he UNKNOWN_PRODUCER_ID issue. > On 9/03/2020, at 16:08, James Olsen wrote: > > P.S. I guess the big question is what is the best way to handle or avoid > UNKNOWN_PROD

Re: Reg : Slowness in Kafka

Also check your Kafka Client and Server versions. There are serious latency issues when mixing different client and server versions IF your consumers handle multiple partitions. > On 27/03/2020, at 12:59, Chris Larsen wrote: > > Hi Vidhya, > > How many tasks are you running against the

Re: Problems when Consuming from multiple Partitions

P.S. I guess the big question is what is the best way to handle or avoid UNKNOWN_PRODUCER_ID when running versions that don’t include KAFKA-7190 / KAFKA-8710 ? We are using non-transactional idempotent Producers. > On 9/03/2020, at 12:59 PM, James Olsen wrote: > > For completene

Re: Problems when Consuming from multiple Partitions

? We can choose 2.2.1 or 2.3.1 for the Broker (AWS recommend 2.2.1 although don’t state why). Based on the experiences below, I would then go with the corresponding 2.2.2 or 2.3.1 Client version. Which combo would people recommend? > On 9/03/2020, at 12:03 PM, James Olsen wrote: > &

Re: Problems when Consuming from multiple Partitions

wanted the fix for https://issues.apache.org/jira/browse/KAFKA-7190 as we may have some very quiet Topics.2.3.x does have some handling for this as implied by https://issues.apache.org/jira/browse/KAFKA-8483 but I’m not sure it is as complete. Regards, James. On 9/03/2020, at 11:54 AM, Jamie

Re: Problems when Consuming from multiple Partitions

=consumer-LedgerService-group-1, groupId=LedgerService-group] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(Ledger-1), toForget=(), implied=(Ledger-0)) to broker localhost:9093 (id: 1001 rack: null) > On 9/03/2020, at 10:48 AM, James Olsen wrote: > > Thanks for your respo

Re: Problems when Consuming from multiple Partitions

rned fetch data (error=NONE, highWaterMark=41, lastStableOffset = 41, logStartOffset = 0, preferredReadReplica = absent, abortedTransactions = null, recordsSizeInBytes=573) On 5/03/2020, at 11:45 PM, M. Manna mailto:manme...@gmail.com>> wrote: Hi James, 3 Consumers in a group means y

Problems when Consuming from multiple Partitions

I’m seeing behaviour that I don’t understand when I have Consumers fetching from multiple Partitions from the same Topic. There are two different conditions arising: 1. A subset of the Partitions allocated to a given Consumer not being consumed at all. The Consumer appears healthy, the

Re: Significant CPU utilization increase in 2.4.0

I filed https://issues.apache.org/jira/browse/KAFKA-9339 On Fri, Dec 27, 2019 at 12:24 PM Ismael Juma wrote: > Hi James, > > Thanks for the report. This is not expected. How many partitions per broker > and what Java version? I suggest filing a JIRA and, if possible, attach a >

Significant CPU utilization increase in 2.4.0

upgrade something with more load on it? -- James Brown Engineer

New topic gets stuck in bad state after Zookeeper error

came up, it assumed leadership of the partition and everything started working fine. Has anyone else seen this behavior before? The fact that a partition was unavailable but the mbean showed 0 under-replicated and 0 un-available topics is extremely concerning to me. -- James Brown Systems Engineer

Re: please care and vote for Chinese people under cruel autocracy of CCP, great thanks!

Why is this group being targeted by the false political propaganda? The sender did not even use a valid email account. On Thu, Aug 29, 2019 at 7:05 AM ant_fighter wrote: > Hi all, > Sorry for disturbing you guys. Though I don't think here as a proper place > to do this, I need your help, your

Re: [ANNOUNCE] New Kafka PMC member: Matthias J. Sax

Congrats!! -James Sent from my iPhone > On Apr 18, 2019, at 2:35 PM, Guozhang Wang wrote: > > Hello Everyone, > > I'm glad to announce that Matthias J. Sax is now a member of Kafka PMC. > > Matthias has been a committer since Jan. 2018, and since then he con

Re: Proxying the Kafka protocol

any fundamental issues with this approach? James Grant Developer - Expedia Group On Tue, 19 Mar 2019 at 16:13, Hans Jespersen wrote: > > > You might want to take a look at kafka-proxy ( see > https://github.com/grepplabs/kafka-proxy < > https://github.com/grepplabs/kafka-prox

Proxying the Kafka protocol

anybody know of any reason it would be impossible (or a bad idea) to do? James Grant Developer - Expedia Group

Re: [ANNOUNCE] New Committer: Randall Hauch

Congrats, Randall! Well deserved! -James Sent from my iPhone > On Feb 14, 2019, at 6:16 PM, Guozhang Wang wrote: > > Hello all, > > The PMC of Apache Kafka is happy to announce another new committer joining > the project today: we have invited Randall Hauch as a project

Re: [ANNOUNCE] New Committer: Vahid Hashemian

Congrats, Vahid!! -James > On Jan 15, 2019, at 2:44 PM, Jason Gustafson wrote: > > Hi All, > > The PMC for Apache Kafka has invited Vahid Hashemian as a project committer > and > we are > pleased to announce that he has accepted! > > Vahid has made numer

Re: [ANNOUNCE] Apache Kafka 2.1.0

Thanks Dong for running the release, and congrats to everyone in the community! -James Sent from my iPhone > On Nov 21, 2018, at 10:09 AM, Dong Lin wrote: > > The Apache Kafka community is pleased to announce the release for Apache > Kafka 2.1.0 > > > This is a major

Re: Kafka on Eclipse Setup

Matthias, Thanks for your suggestion but it did not work. I finally have to remove all reference of test libraries and just use main. I was able to build all libraries without any errors. James > On Oct 15, 2018, at 12:45 PM, Matthias J. Sax wrote: > > You can try to

Re: Kafka on Eclipse Setup

and stream codes. I can manually copy all Kafka related jar files but is this the right way? I thought after I download the codes from Github, everything should work. James > On Oct 13, 2018, at 10:08 AM, Vahid Hashemian > wrote: > > This article was written a while back but stil

Kafka on Eclipse Setup

or path not defined messages. I don’t know if I need to fix them one by one or if there is any jar files that I need to include. James

Re: [ANNOUNCE] New committer: Colin McCabe

Congrats, Colin! -James > On Sep 25, 2018, at 1:39 AM, Ismael Juma wrote: > > Hi all, > > The PMC for Apache Kafka has invited Colin McCabe as a committer and we are > pleased to announce that he has accepted! > > Colin has contributed 101 commits and 8 KI

Re: unable to start kafka when zookeeper cluster is in working but unhealthy state

to be resolved. Or does it also work for this case? Thanks for your responses. This is a UTF-8 formatted mail --- James C.-C.Yu +886988713275 +8615618429976 2018-09-26 19:36 GMT+08:00 Manikumar : > You can try using Kafka 2.0 release. Original is

unable to start kafka when zookeeper cluster is in working but unhealthy state

thought we can start a kafka as long as the zookeeper cluster is working.regardless there is one zookeeper node (3 in total) fails to run? This is a UTF-8 formatted mail --- James C.-C.Yu +886988713275 +8615618429976

Re: Need info

MySQL and PostgrepSQL. > On Sep 12, 2018, at 12:34 PM, Mich Talebzadeh > wrote: > > Hi James, > > As a matter of interest this streaming data is fed into some Operational > Data Store ODS) like MongoDB? > > In general using this method will create a near real time sn

Re: Need info

We have banking customers sending data from DB2 z to Kafka linux (not cloud) with transaction rate 30K per seconds. Kafka can handle more than this rate. > On Sep 12, 2018, at 2:31 AM, Chanchal Chatterji > wrote: > > Hi, > > In the process of mainframe modernization, we are attempting to

Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

Congrats Dong! -James > On Aug 20, 2018, at 3:54 AM, Ismael Juma wrote: > > Hi everyone, > > Dong Lin became a committer in March 2018. Since then, he has remained > active in the community and contributed a number of patches, reviewed > several pull requests and partici

Re: [ANNOUNCE] Apache Kafka 2.0.0 Released

Congrats and great job, everyone! Thanks Rajini for driving the release! -James Sent from my iPhone > On Jul 30, 2018, at 3:25 AM, Rajini Sivaram wrote: > > The Apache Kafka community is pleased to announce the release for > > Apache Kafka 2.0.0. > > > > &

Does anyone publish or subscribe kafka topic in TestNG?

er<String, String> producer = new KafkaProducer<String, String>(props); for(int i = 0; i < 10; i++) { producer.send(new ProducerRecord<String, String>(topicName, Integer.toString(i), Integer.toString(i))); } System.out.println("Message sent successfully"); producer.close(); } } This is a UTF-8 formatted mail --- James C.-C.Yu +886988713275 +8615618429976

org.apache.kafka.common.errors.UnsupportedVersionException in Java

Factory() { ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>(); factory.setConsumerFactory(consumerFactory()); factory.getContainerProperties().setAckMode(AckMode.MANUAL_IMMEDIATE); return factory; } Thank yo

Re: [ANNOUNCE] Apache Kafka 1.1.0 Released

Thanks Damian and Rajini for running the release! Congrats and good job everyone! -James Sent from my iPhone > On Mar 29, 2018, at 2:27 AM, Rajini Sivaram <rsiva...@apache.org> wrote: > > The Apache Kafka community is pleased to announce the release for > > Apache Kafka

Re: [ANNOUNCE] New Committer: Dong Lin

Congrats, Dong! -James > On Mar 28, 2018, at 10:58 AM, Becket Qin <becket@gmail.com> wrote: > > Hello everyone, > > The PMC of Apache Kafka is pleased to announce that Dong Lin has accepted > our invitation to be a new Kafka committer. > > Dong started wor

Re: [ANNOUNCE] Apache Kafka 1.0.1 Released

Congrats, everyone! Thanks for driving the release, Ewen! -James > On Mar 6, 2018, at 1:22 PM, Guozhang Wang <wangg...@gmail.com> wrote: > > Ewen, thanks for driving the release!! > > > Guozhang > > On Tue, Mar 6, 2018 at 1:14 PM, Ewen Cheslack-P

Consume latest offset after rebalance

y to catch the "NOT_COORDINATOR_FOR_GROUP" error I could manually alter the offset? I'm currently using 0.9.1. Thanks -James

Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram

Congrats Rajini! -James Sent from my iPhone > On Jan 17, 2018, at 10:48 AM, Gwen Shapira <g...@confluent.io> wrote: > > Dear Kafka Developers, Users and Fans, > > Rajini Sivaram became a committer in April 2017. Since then, she remained > active in the community and c

Re: [VOTE] KIP-247: Add public test utils for Kafka Streams

+1 (non-binding) -James Sent from my iPhone > On Jan 17, 2018, at 6:09 PM, Matthias J. Sax <matth...@confluent.io> wrote: > > Hi, > > I would like to start the vote for KIP-247: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-247%3A+Add+public+test

what are common ways to convert info on a web site into a log entry?

the time when it is likely to only do a few seconds of work each month? Many thanks, James.

Re: what are common ways to convert info on a web site into a log entry?

thanks! I will write my reply into the mail group once I have fiddled with your suggestions. James. > On Jan 15, 2018, at 2:20 PM, Jacob Sheck <shec0...@gmail.com> wrote: > > You will need to find or create something to accomplish this. Topics in > Kafka primarily act

what are common ways to convert info on a web site into a log entry?

the time when it is likely to only do a few seconds of work each month? Many thanks, James.

Re: [ANNOUNCE] New committer: Matthias J. Sax

Congrats, Matthias!! Well deserved! -James > On Jan 12, 2018, at 2:59 PM, Guozhang Wang <wangg...@gmail.com> wrote: > > Hello everyone, > > The PMC of Apache Kafka is pleased to announce Matthias J. Sax as our > newest Kafka committer. > > Matthias has made tre

Re: Insanely long recovery time with Kafka 0.11.0.2

to see what I/O the broker was doing, it was reading ALL the segments for the partitions. -James > On Jan 11, 2018, at 10:56 AM, Vincent Rischmann <vinc...@rischmann.fr> wrote: > > If anyone else has any idea, I'd love to hear it. > > Meanwhile, I'll resume upgrading my broker

JOB | Permanent Sysadmin (Amsterdam, Netherlands OR remote)

"JamesBTobin (at) Gmail (dot) Com". Kind regards, James

Re: [ANNOUNCE] Apache Kafka 1.0.0 Released

Good job, everyone! And thanks Guozhang for running the release! -James > On Nov 1, 2017, at 7:27 AM, Guozhang Wang <wangg...@gmail.com> wrote: > > The Apache Kafka community is pleased to announce the release for Apache > Kafka 1.0.0. > > This is a major rele

How do I instantiate a metrics reporter in Kafka Streams, with custom config?

zed keys in the properties object? And actually, for the metrics.reporter class itself, am I supposed to specify it as: metrics.reporter or metric.reporters producer.metric.reporters consumer.metric.reporters Thanks, -James

JOB | Permanent Remote Cassandra Database Engineer

ot;. Kind regards, James

Re: Metrics: committed offset, client version

188+-+Add+new+metrics+to+support+health+checks> -James > On Sep 20, 2017, at 2:37 AM, Stas Chizhov <schiz...@gmail.com> wrote: > > Hi! > > I am wondering if there are broker/client metrics for: > - client version (to keep track of clients that needs an upgrade) &g

Re: Kafka Internals Video/Blog

e soon. -James > On Sep 20, 2017, at 9:49 AM, Raghav <raghavas...@gmail.com> wrote: > > Hi > > Just wondering if there is any video/blog that goes over Kafka Internal and > under the hood design and implementation details. I am a newbie and I would > like to dabble wi

Re: Improving Kafka State Store performance

.level="debug", you can see a bunch of metrics around the state stores. Stuff like put-latency-avg, for example. See http://kafka.apache.org/documentation/#kafka_streams_store_monitoring <http://kafka.apache.org/documentation/#kafka_streams_store_monitoring> -James > On Sep 16, 20

Re: Consumer group metadata retention

The offsets.retention.minutes value (1440 = 24 hours = 1 day) is a broker level configuration, and can't be changed dynamically during runtime. You would have to modify the broker configurations, and restart the brokers. -James > On Jul 25, 2017, at 9:43 PM, Raghu Angadi <ang...@gma

Re: Tuning up mirror maker for high thruput

tuning affect those consumers that are close to the source datacenter? Will they continue to function well? -James > On Jul 23, 2017, at 7:16 AM, Todd Palino <tpal...@gmail.com> wrote: > > One of the best pieces of advice I can offer is that you really need to run > the mirror

Re: Tuning up mirror maker for high thruput

and video at https://youtu.be/oQe7PpDDdzA -James Sent from my iPhone > On Jul 21, 2017, at 9:25 AM, Sunil Parmar <sunilosu...@gmail.com> wrote: > > We're trying to set up mirror maker to mirror data from EU dc to US dc. The > network delay is ~150 ms. In recent test; we realized

Re: Consumer offsets partitions size much bigger than others

in the event you don't have enough brokers. -James > On Jul 18, 2017, at 8:44 AM, Luciano Afranllie <listas.luaf...@gmail.com> > wrote: > > Hi > > One of our Kafka brokers was running out of disk space and when we checked > the file size in the kafka log dir we observed

Re: Mirroring multiple clusters into one

Answers inline below. -James Sent from my iPhone > On Jul 7, 2017, at 1:18 AM, Vahid S Hashemian <vahidhashem...@us.ibm.com> > wrote: > > James, > > Thanks for sharing your thoughts and experience. > Could you please also confirm whether > - you do any en

Re: Mirroring multiple clusters into one

at. https://www.slideshare.net/mobile/JiangjieQin/producer-performance-tuning-for-apache-kafka-63147600 -James Sent from my iPhone > On Jul 6, 2017, at 1:00 AM, Vahid S Hashemian <vahidhashem...@us.ibm.com> > wrote: > > The literature suggests running the MM on the target cluster when pos

Re: mirroring Kafka while preserving the order

. MirrorMaker supports custom message handlers. You can use those to map the source partition to the destination partition, which will allow you to avoid the above two problems. Here's an example of how to do it. https://github.com/gwenshap/kafka-examples/tree/master/MirrorMakerHandler -James Sent from

question about document

Hello, At https://kafka.apache.org/intro, I found a sentence: Each partition has one server which acts as the "leader" and zero or more servers which act as "followers". In my opinion, it is possible that more than one brokers exist on a machine with different ports and different

Re: Slow Consumer Group Startup

Bryan, This sounds related to https://cwiki.apache.org/confluence/display/KAFKA/KIP-134%3A+Delay+initial+consumer+group+rebalance and https://issues.apache.org/jira/browse/KAFKA-4925. -James > On Jun 13, 2017, at 7:02 AM, Bryan Baugher <bjb...@gmail.com> wrote: > > The topic

Re: [ANNOUNCE] New committer: Damian Guy

Congrats Damian! -James > On Jun 9, 2017, at 1:34 PM, Guozhang Wang <wangg...@gmail.com> wrote: > > Hello all, > > > The PMC of Apache Kafka is pleased to announce that we have invited Damian > Guy as a committer to the project. > > Damian has made tremendo

Re: topics stuck in "Leader: -1" after crash while migrating topics

For what it's worth, shutting down the entire cluster and then restarting it did address this issue. I'd love anyone's thoughts on what the "correct" fix would be here. On Fri, Apr 28, 2017 at 10:58 AM, James Brown <jbr...@easypost.com> wrote: > The following is also appearin

Re: topics stuck in "Leader: -1" after crash while migrating topics

The following is also appearing in the logs a lot, if anyone has any ideas: INFO Partition [easypost.syslog,7] on broker 1: Cached zkVersion [647] not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition) On Fri, Apr 28, 2017 at 10:43 AM, James Brown <jbr...@easypost.com>

topics stuck in "Leader: -1" after crash while migrating topics

e whatever the third broker was in their replica-set as their replica set? Do I need to temporarily enable unclean elections? I've never seen a cluster fail this way...​ -- James Brown Engineer

Re: stunning error - Request of length 1550939497 is not valid, it is larger than the maximum size of 104857600 bytes

e protocol that uses the word "\quit"? Like IRC or SMTP or IMAP something? I'm not sure. Anyway, thought you might find that interesting! -James > On Dec 12, 2016, at 9:39 AM, Todd Palino <tpal...@gmail.com> wrote: > > Are you actually getting requests that are

how can I contribute to this project?

Hi Because I love this project, so I want to take part of it. But I'm brand new to opensource project. How can I get started to make contribution? Can you give me some advise or something? By the way, I already have JIRA account which called "james.c" Sincerely, James.C

Re: ISR churn

aven't been able it isolate a cause. I think it's related to flaky connections to the zookeeper cluster. -James > On Mar 22, 2017, at 11:46 AM, Marcos Juarez <mjua...@gmail.com> wrote: > > We're seeing the same exact pattern of ISR shrinking/resizing, mostly on >

Re: Offset commit request failing

gets too big to go through the topic. There is a long thread here that discusses it. I don't remember what specific actions came out of that discussion. http://search-hadoop.com/m/Kafka/uyzND1yd26N1rFtRd1?subj=+DISCUSS+scalability+limits+in+the+coordinator -James Sent from my iPhone > On

Re: How does offsets.retention.minutes work

Yes, that is correct. I filed a JIRA about that issue here: https://issues.apache.org/jira/browse/KAFKA-4682 -James > On Mar 15, 2017, at 8:51 PM, tao xiao <xiaotao...@gmail.com> wrote: > > Hi team, > > I know that Kafka deletes offset for a consumer group after a period

Re: Clarification on min.insync.replicas​

> On Mar 7, 2017, at 12:18 PM, James Cheng <wushuja...@gmail.com> wrote: > > >> On Mar 7, 2017, at 7:44 AM, Shrikant Patel <spa...@pdxinc.com> wrote: >> >> Thanks for clarification. I am seeing strange behavior in that case, >> >> When

Re: Clarification on min.insync.replicas​

try min.insync.replicas=2 in server.properties and acks=all on producer? min.insync.replicas only applies when acks=all. -James > > -Original Message- > From: Todd Palino [mailto:tpal...@gmail.com] > Sent: Monday, March 06, 2017 6:48 PM > To: users@kafka.apache.org > Subject: R

Re: Question about messages in __consumer_offsets topic

Yup, this got fixed in 0.10.2 https://issues.apache.org/jira/browse/KAFKA-2000 <https://issues.apache.org/jira/browse/KAFKA-2000> -James > On Feb 23, 2017, at 11:10 AM, Jeff Widman <j...@netskope.com> wrote: > > The topic deletion only triggers tombstone on brokers

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

Woohoo! Thanks for running the release, Ewen! -James > On Feb 22, 2017, at 12:33 AM, Ewen Cheslack-Postava <ewe...@apache.org> wrote: > > The Apache Kafka community is pleased to announce the release for Apache > Kafka 0.10.2.0. This is a feature release which includes the

  1   2   3   >