Re: Is there any command can be used to get broker.id on one of broker with command?

JMX mbean. > > > > On Thu, Feb 16, 2017 at 12:32 PM, Stevo Slavić <ssla...@gmail.com> > wrote: > > > > > You can just grep it from broker's config/server.properties > > > > > > On Thu, Feb 16, 2017 at 7:42 AM, kay James <myvide...@gmail.com> &

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

would most likely only make sense to one partition at time. It might make sense to *require* topic and partition, when moving to a specific offset. -James > On Feb 8, 2017, at 3:36 PM, Gwen Shapira <g...@confluent.io> wrote: > > Just to clarify, we'll need to allow specifyi

Is there any command can be used to get broker.id on one of broker with command?

Is there any command can be used to get broker.id on one of broker with command? I can check zookeeper or check the broker starting log to find broker.id, but I wander whether there is any command to get the broker.id?

Is anyone running Kafka on CoreOS?

on the first node. Is anyone else doing this and if so, can you share some ideas/approaches? We have an ugly solution to #1, which is to hook into the systemd dependency chain to allow shutdown. But we don't have a great way to do 2). Thanks, -James

Fw: Exception when using simple consumer fetching offsets.

:?] at kafka.consumer.SimpleConsumer.fetchOffsets(SimpleConsumer.scala:169) [kafka_2.11-0.9.0.1.jar:?] any comment is appreciated. james.

users@kafka.apache.org

join in mailing list.

Re: Kafka docs for current trunk

+1 In particular, this would help when there is a doc change submitted to trunk which is applicable to the currently released version. It would help the change get out there faster. -James > On Feb 1, 2017, at 9:03 AM, Guozhang Wang <wangg...@gmail.com> wrote: > > +

Re: [ANNOUNCE] New committer: Grant Henke

Congrats, Grant!! -James > On Jan 11, 2017, at 11:51 AM, Gwen Shapira <g...@confluent.io> wrote: > > The PMC for Apache Kafka has invited Grant Henke to join as a > committer and we are pleased to announce that he has accepted! > > Grant contributed 88 patches, 90 code

Re: Under-replicated Partitions while rolling Kafka nodes in AWS

once 1 is back up. But that's a bunch of work you've have to do yourself. -James > This is causing us some pain because it means that we get pages whenever we > roll out changes to Zookeeper. > > Does anybody have any ideas why this would be happening, and how we can > avoid it? > > Thanks. > > -Jack Lund > Braintree Payments

Re: Lost message with Kafka configuration

> On Jan 5, 2017, at 8:23 AM, Hoang Bao Thien <hbthien0...@gmail.com> wrote: > > Yes, the problem is from producer configuration. And James Cheng has told > me how to fix it. > However I still get other poblem with a large file: > > org.apache.kafka.common.err

Re: Is this a bug or just unintuitive behavior?

> On Jan 5, 2017, at 12:57 PM, Jeff Widman <j...@netskope.com> wrote: > > Thanks James and Hans. > > Will this also happen when we expand the number of partitions in a topic? > > That also will trigger a rebalance, the consumer won't subscribe to the > partitio

Re: Lost message with Kafka configuration

acks in the producer configuration section of the kafka docs (http://kafka.apache.org/documentation/#producerconfigs <http://kafka.apache.org/documentation/#producerconfigs>) -James > On Jan 4, 2017, at 1:25 AM, Hoang Bao Thien <hbthien0...@gmail.com> wrote: > > Hi all, >

Re: Is this a bug or just unintuitive behavior?

the following JIRAs: https://issues.apache.org/jira/browse/KAFKA-3848 <https://issues.apache.org/jira/browse/KAFKA-3848> https://issues.apache.org/jira/browse/KAFKA-3370 <https://issues.apache.org/jira/browse/KAFKA-3370> -James > On Jan 4, 2017, at 4:37 PM, h...@confluent.io wrote: > >

Why does consumer.subscribe(Pattern) require a ConsumerRebalanceListener?

think through what happens as new topic/partitions appear and disappear? -James

When using mirrormaker, how are people creating topics?

creation in an interceptor Anyway, was hoping to get some thoughts from people who are already doing this. Thanks! -James

Re: consumer_offsets partition skew and possibly ignored retention

Jeff: This was with 0.9.0.1. It has not recurred since upgrading to 0.10.1.0. On Fri, Oct 28, 2016 at 9:28 PM, Jeff Widman <j...@netskope.com> wrote: > James, > What version did you experience the problem with? > > On Oct 28, 2016 6:26 PM, "James Brown" <jbr.

Re: Duplicate consumer group in ListGroups in Kafka 0.10.1?

', client_id='tracking.etl', client_host='/fd00:ea51:d057:0:1:0:0:2', member_metadata=b'', member_assignment=b'')])]) On Tue, Nov 1, 2016 at 1:50 PM, James Brown <jbr...@easypost.com> wrote: > Here's another strange bug that we're seeing after upgrading to Kafka > 0.10.1.0: one of our con

Duplicate consumer group in ListGroups in Kafka 0.10.1?

there was anything else folks wanted me to look at first. -- James Brown Engineer

Re: Segments being deleted too early after upgrading 0.9.0.1 to 0.10.1.0

, not into the past... On Mon, Oct 31, 2016 at 5:02 PM, Jun Rao <j...@confluent.io> wrote: > Hi, James, > > Thanks for testing and reporting this. What you observed is actually not > the expected behavior in 0.10.1 based on the design. The way that retention > works in 0.10.1 is that

Segments being deleted too early after upgrading 0.9.0.1 to 0.10.1.0

t it's too late to un-do anything related to this, and I don't actually think any of our consumers were relying on this data, but I figured I'd send along this report and see if anybody else has seen behavior like this. Thanks,​ -- James Brown Engineer

Re: [ANNOUNCE] New committer: Jiangjie (Becket) Qin

Congrats, Becket! -James > On Oct 31, 2016, at 10:35 AM, Joel Koshy <jjkosh...@gmail.com> wrote: > > The PMC for Apache Kafka has invited Jiangjie (Becket) Qin to join as a > committer and we are pleased to announce that he has accepted! > > Becket has made significan

Re: consumer_offsets partition skew and possibly ignored retention

ur config output includes the following entries: > > log.cleaner.enable = true > > offsets.retention.minutes = 1440 > > > I tried looking through the issues on JIRA but didn't see a reported > issue. Does anyone know what's going on, and how I can fix this? > > Thanks. > -- James Brown Engineer

Re: Read all record from a Topic.

Jean-Baptiste, I wrote a blog post recently on this exact subject. https://logallthethings.com/2016/06/28/how-to-read-to-the-end-of-a-kafka-topic/ Let me know if you find it useful. -James Sent from my iPhone > On Jul 13, 2016, at 7:16 AM, g...@netcourrier.com wrote: > > Hi, &g

Re: Halting because log truncation is not allowed for topic __consumer_offsets

the JIRAs as well. FYI, I was the original filer of one of KAFKA-3410. My repro only talks about the case where a broker was completely wiped, but we've seen it happen during normal operation as well. -James Sent from my iPhone > On Jun 26, 2016, at 1:41 PM, Peter Davis <davi...@gmail.com

Re: 10MB message

is having issues right now, so it may take a (long) while to load. I've notified the website owner to let her know. -James > On Jun 14, 2016, at 1:33 AM, R Krishna <krishna...@gmail.com> wrote: > > There are options to compress on the wire and in the topic. > > On Tue, May 31,

Re: Will segments on no-traffic topics get deleted/compacted?

that for in-active topics, segments will never roll. See https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/log/Log.scala#L383 and https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/log/Log.scala#L617-L634 Tom, does that sound right? -James > On May 20, 2016, at 6:05

Will segments on no-traffic topics get deleted/compacted?

on the topic? Thanks! -James

Re: unknown (kafka) offsets after restart

-cleaner.log debuglog file to see if there are any error messages there. -James > On May 6, 2016, at 6:28 AM, Jörg Wagner <joerg.wagn...@1und1.de> wrote: > > After a bit more looking into this we found out that the Offsetmanager is > single threaded and due to our setup (few

Re: leader election bug

hut down, but it's still leader for the partition (at >> least for a couple of hours as I monitor it). >> Zookeeper cluster is healthy. >> >> ls /brokers/ids >> [104224874, 104224875, 104224863, 104224864, 104224871, 104224867, >> 104224868, 104224865, 104224866, 104224876, 104224877, 104224869, >> 104224878, 104224879] >> >> That broker is not registered in ZK. >> -- James Brown Engineer

Re: Consumers disappearing form __consumer_offsets

This may be related to offsets.retention.minutes. offsets.retention.minutes Log retention window in minutes for offsets topic It defaults to 1440 minutes = 24 hours. -James > On Apr 11, 2016, at 1:36 PM, Morellato, Wanny <wanny.morell...@concur.com> > wrote: > > Hi, > &g

Re: Reg. Partition Rebalancing

move so much data around at unpredictable times. Is that the main performance issue you were talking about, or was there something else? -James > The other thing we mean by rebalance is partition rebalancing, or changing > which brokers are replicas for a given partition to spread out

Re: kafka 0.9.0.1: FATAL exception on startup

Hi, we ran into this problem too. The only way we were able to bypass this was by stopping Kafka and deleting the log directory of the affected partition. Which means, we lost data for that partition on this broker. -James > On Mar 8, 2016, at 1:07 AM, Anatoly Deyneka <anatoly.deyn...@

Re: Multi-threaded consumer?

a single topic. -James > On Mar 22, 2016, at 5:21 PM, BYEONG-GI KIM <bg...@bluedigm.com> wrote: > > Hello. > > I'd like to know how to implement a multi-threaded consumer, which retrieve > message(s) from a topic per thread. > > I read the Kafka Consumer 0.9

What happens if controlled shutdown can't complete within controlled.shutdown.max.retries attempts?

shutdown never happened? Or does it do an uncontrolled shutdown? Thanks, -James This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any

Re: Questions about unclean leader election and "Halting because log truncation is not allowed"

Anthony, I filed https://issues.apache.org/jira/browse/KAFKA-3410 to track this. -James > On Feb 25, 2016, at 2:16 PM, Anthony Sparks <anthony.spark...@gmail.com> > wrote: > > Hello James, > > We received this exact same error this past Tuesday (we are on 0.8.2).

Re: Uneven GC behavior between nodes

are seeing more load than the others? -James > On Mar 4, 2016, at 9:15 AM, Cees de Groot <c...@pagerduty.com> wrote: > > We're seeing something funny in one of our production clusters that I > cannot explain away. Everything works fine, but as we're ramping up on > Kafka, I really wa

Re: Writing a Producer from Scratch

do you have any way to do HTTP calls on your platform? There exist some REST servers that you speak HTTP to and then they will produce to Kafka on your behalf. Here is one: http://docs.confluent.io/2.0.1/kafka-rest/docs/index.html -James On Mar 3, 2016, at 2:47 AM, Hopson, Stephen <st

Re: About the number of partitions

Kim, Here's a good blog post from Confluent with advice on how to choose the number of partitions. http://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/ -James > On Mar 1, 2016, at 4:11 PM, BYEONG-GI KIM <bg...@bluedigm.com> wrote: > &g

Unavailable partitions (Leader: -1 and ISR is empty) and we can't figure out how to get them back online

came back online. Thanks, -James in zookeeper --- $ get /brokers/topics/the.topic.name {"version":1,"partitions":{"0":[5]}} server.log [2016-03-01 06:29:13,869] WARN Found an corrupted index file, /TivoData/kafka/the.topic.name-0/.i

Re: Kafka Rest Proxy

Jan, I don't use the rest proxy, but Confluent has a mailing list where you can probably get more info: Here's the direct link: https://groups.google.com/forum/#!forum/confluent-platform And it is linked off of here: http://www.confluent.io/developer#documentation -James > On Mar 1, 2

Questions about unclean leader election and "Halting because log truncation is not allowed"

ion. But I did venture into "unclean leader election" territory, so I'm not sure if the comment still applies. Thanks, -James This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient.

Re: Discrepancy between JMX OfflinePartitionCount and kafka-topics.sh?

on the leader broker and the broker that was out of the ISR, and it showed that the partition was fully replicated to the follower. So JMX+DumpLogSegments disagreed with kafka-topics.sh+zookeeper. So that is the opposite scenario of yours. Mine was on 0.9.0. -James > On Feb 9, 2016, at 12:02

Re: Detecting broker version programmatically

> On Feb 4, 2016, at 8:28 PM, Manikumar Reddy <ku...@nmsworks.co.in> wrote: > > Currently it is available through JMX Mbean. It is not available on wire > protocol/requests. > The name of the JMX Mbean is kafka.server:type=app-info,id=4 Not sure what the id=4 means. -J

Re: at-least-once delivery

y appear first." > Franco, Also, you can avoid the message reordering issue in that description by setting max.in.flight.requests.per.connector to 1. This slide deck has good guidelines on the types of things you are talking about: http://www.slideshare.net/JiangjieQin/no-data-lo

Re: MongoDB Kafka Connect driver

ou'll need "understand" mongodb's syntax and commands? Although maybe it is simply sets/unsets/deletes, in which case it's maybe pretty simple. -James > On Jan 29, 2016, at 9:39 AM, Jay Kreps <j...@confluent.io> wrote: > > Also, most database provide a "full logging&qu

Re: Accumulating data in Kafka Connect source tasks

o/mysql-binlog-connector-java is a fork. So it seems we're along the same path. Maxwell in particular is doing the things you are talking about in this thread. It captures MySQL schema changes and stores them in such a way that they are correlated with the binlog coordinates, and so they can be

Re: Accumulating data in Kafka Connect source tasks

> imagine is all you really need -- it'd just be convenient to have the > connection info for the Kafka brokers. > > Finally, I'd love to know which DB you're reading the transaction log from > and if you're planning on open sourcing the connector:) > +1! I'd like to know what DB you're w

Re: Offset storage issue with kafka(0.8.2.1)

ded format, so you'll need to specify a message formatter. >From slide 32 of this deck >http://www.slideshare.net/jjkoshy/offset-management-in-kafka, ./bin/kafka-console-consumer.sh --topic __consumer_offsets --zookeeper localhost:2181 --formatter "kafka.server.OffsetManager$OffsetMess

Re: Kafka + ZooKeeper on the same hardware?

t; (init scripts etc) which I'd have expected > in a full release. > > (sorry if this is OT, I don't know of a specific Confluent mailing list). > Confluent-specific mailing list is linked off of here: http://www.confluent.io/developer#documentation -James > > > On 14 Januar

Re: how to reset kafka offset in zookeeper

-Parcels/td-p/20392, Cloudera's default config is to put a zookeeper chroot of "/kafka" So you might have to look at /kafka/consumers/ -James > On Dec 19, 2015, at 3:14 PM, Todd Palino <tpal...@gmail.com> wrote: > > There’s no simple command. You’ll need to use

0.8.2 high level consumer with one-time-use group.id's?

deleted? Thanks, -James This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you

Re: Where is replication factor stored?

0xc017a mtime = Wed Aug 05 22:48:12 UTC 2015 pZxid = 0xc017a cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 79 numChildren = 0 I tried that for a number of different topics, and none of them have it. -James > Guozhang > > On Fri, Oct 16, 2015 at 12:

Where is replication factor stored?

Hi, Where is the replication factor for a topic stored? It isn't listed at https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper. But the kafka-topics --describe command returns something. Where is it finding that? Thanks, -James

Re: New consumer client compatible with old broker

tter.com/lorax_james/status/652607463057829888 -James This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. I

Re: Dealing with large messages

Here’s an article that Gwen wrote earlier this year on handling large messages in Kafka. http://ingest.tips/2015/01/21/handling-large-messages-kafka/ -James > On Oct 5, 2015, at 11:20 AM, Pradeep Gollakota <pradeep...@gmail.com> wrote: > > Fellow Kafkaers, > > We have

Re: custom message handlers?

nt i am building kafka from source to get this > feature. > Doug, are you asking about custom message handling in Mirror Maker? -James > many thanks, > doug > This email and any attachments may contain confidential and privileged material for t

Re: Log Cleaner Thread Stops

we want to make use of __consumer_offsets, then we cannot use compressed messages in any topic that has compaction turned on. Is that right? -James > -Todd > > > On Fri, Sep 18, 2015 at 9:54 AM, John Holland < > john.holl...@objectpartners.com> wrote: > >> Thank

Documentation typo for offsets.topic.replication.factor ?

will be created with fewer replicas. I'm guessing there's a typo there? I'm guessing it should be: If the offsets topic is created when fewer brokers than the replication factor [are active], then the offsets topic will be created with fewer replicas. Or something along those lines? Thanks, -James

Checkpointing with custom metadata

, and that in order to checkpoint with custom metadata, we have to issue the OffsetCommitRequest ourselves. Is that correct? Thanks, -James

Re: New consumer - poll/seek javadoc confusing, need clarification

if there is a broker failure and a new broker becomes the partition leader? Does the high level consumer start listening to the new partition leader at the in-memory position, or does it restart based on saved offsets? Thanks, -James -Ewen Could be it's just me not understanding this from

Consuming from Kafka but don't need to save offsets

? Thanks, -James

Re: New producer in production

it elsewhere, but this is where I found it. -James On Jul 17, 2015, at 9:49 AM, Jay Kreps j...@confluent.io wrote: Hey Sivananda, That's actually no longer true and likely a documentation bug. Where did you see that? -Jay On Fri, Jul 17, 2015 at 9:35 AM, Sivananda Reddy sivananda2

Re: New producer in production

producer to the new producer. Siva, this is less important for you, since you are starting to use kafka for the first time. And this is not important at all if you provide your own partitioning algorithm. -James On Jul 17, 2015, at 1:23 PM, Sivananda Reddy sivananda2...@gmail.com wrote: Hi

Re: Using Kafka as a persistent store

the Kinesis stream itself. And the data is structured such that when you are done reading from S3, you can connect to the Kinesis stream at the point where the S3 archive left off. This effectively allowed me to add a configurable retention period when consuming from Kinesis. -James On Jul 13

New producer and ordering of Callbacks when sending to multiple partitions

will also be called prior to the ones from A1 A2 A3? Thanks, -James

How to manage the consumer group id?

. Thanks, -James

Re: Changing replication factor for an existing topic

AirBNB's kafkat tool has a set-replication-factor option. I've never tried it myself. https://github.com/airbnb/kafkat -James On Jun 10, 2015, at 4:20 PM, Aditya Auradkar aaurad...@linkedin.com.INVALID wrote: The replica list that you specify can be used to increment/decrement

Re: Is fetching from in-sync replicas possible?

partition topic across 10 nodes where each leader handles 1/10th of the consumer traffic for that topic, I could change that and instead have 100 partition topic across 100 nodes, and then each leader would only have to handle 1/100th of the consumer traffic for that topic. -James Theo's use

Re: Is fetching from in-sync replicas possible?

that the network/disk bandwidth of a particular leader is the bottleneck. If consumers could read from in-sync replicas, then a single node no longer is the bottleneck for reads. You could scale out your read capacity as far as you want. -James The use case is a Kafka cluster running in EC2 across

Re: Is there a way to know when I've reached the end of a partition (consumed all messages) when using the high-level consumer?

of the solutions look good. Will, that patch does exactly what I want, but I'm not sure I want to patch Kafka right now. I'll keep it in mind. Thanks. -James On May 9, 2015, at 10:42 AM, Charlie Knudsen charlie.knud...@smartthings.com wrote: Hi James, What are you trying to do exactly? If all you

Re: Log end offset

Vamsi, There is another thread going on right now about this exact topic: Is there a way to know when I've reached the end of a partition (consumed all messages) when using the high-level consumer? http://search-hadoop.com/m/uyzND1Eb3e42NMCWl -James On May 10, 2015, at 11:48 PM, Achanta Vamsi

Is there a way to know when I've reached the end of a partition (consumed all messages) when using the high-level consumer?

/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-FetchResponse Does the Java high-level consumer expose the HighwaterMarkOffset in any way? I looked but I couldn't find such a thing. Thanks, -James

Re: New and old producers partition messages differently

question: How do you even identify who your consumers are? The only way I can think of is by looking at something like consumer groups and offsets, and find mappings between those and the people who wrote those.) -James If we advertise: upgrading to the new producer API may break consumers, we

New and old producers partition messages differently

of the main reasons to have a custom hashing algorithm is so that you are full control of the partitioning and can keep it stable (as opposed to being reliant on hashCode()). Thanks, -James

Re: serveral questions about auto.offset.reset

the no initial offset in Zookeeper case. But that doesn't describe the offset out of range case. -James On Apr 13, 2015, at 10:28 PM, Ewen Cheslack-Postava e...@confluent.io wrote: On Mon, Apr 13, 2015 at 10:10 PM, bit1...@163.com bit1...@163.com wrote: Hi, Kafka experts: I got serveral

How to consume from a specific topic, as well as a wildcard of topics?

? Can a ConsumerConnector be used like this, or should I have 2 ConsumerConnectors; one for the specific topic, and another for the wildcarded topics? It works when I use 2 ConsumerConnectors, but I just wanted to check if this is expected or not. Thanks, -James

Kafka Consumer

I created a topic using: bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test How do I find out what group it belongs to? Thank you.

Re: [ANN] sqlstream: Simple MySQL binlog to Kafka stream

I created a wiki page that lists all the MySQL replication options that people posted, plus a couple others. People may/may not find it useful. https://github.com/wushujames/mysql-cdc-projects/wiki I wasn't sure where to host it, so I put it up on a Github Wiki. -James On Mar 17, 2015, at 11

Re: Post on running Kafka at LinkedIn

) Bytes out: 650 TB (20% growth) Total brokers: 1100 (56% growth) That much growth in just 2 months? Wowzers. -James On Mar 20, 2015, at 11:30 AM, James Cheng jch...@tivo.com wrote: For those who missed it: The Kafka Audit tool was also presented at the 1/27 Kafka meetup: http

Re: Post on running Kafka at LinkedIn

/recorded/58109076 -James On Mar 20, 2015, at 9:47 AM, Todd Palino tpal...@gmail.com wrote: For those who are interested in detail on how we've got Kafka set up at LinkedIn, I have just published a new posted to our Engineering blog titled Running Kafka at Scale https

Re: Post on running Kafka at LinkedIn

Oops. Thank you Tao Xiao! -James On Mar 20, 2015, at 11:49 AM, tao xiao xiaotao...@gmail.com wrote: here is the slide http://www.slideshare.net/JonBringhurst/kafka-audit-kafka-meetup-january-27th-2015 On Sat, Mar 21, 2015 at 2:36 AM, Xiao lixiao1...@gmail.com wrote: Hi, James

Re: Kafka 0.9 consumer API

Those are pretty much the best javadocs I've ever seen. :) Nice job, Kafka team. -James On Mar 19, 2015, at 9:40 PM, Jay Kreps jay.kr...@gmail.com wrote: Err, here: http://kafka.apache.org/083/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html -Jay On Thu, Mar 19

Re: [ANN] sqlstream: Simple MySQL binlog to Kafka stream

to date easily. -James On Mar 17, 2015, at 8:21 AM, Hisham Mardam-Bey hisham.mardam...@gmail.com wrote: Pretty much a hijack / plug as well (= https://github.com/mardambey/mypipe MySQL binary log consumer with the ability to act on changed rows and publish changes to different systems

Re: createMessageStreams vs createMessageStreamsByFilter

Ah, I understand now. I didn't realize that there was one fetcher thread per broker. Thanks Tao Guozhang! -James On Mar 11, 2015, at 5:00 PM, tao xiao xiaotao...@gmail.com wrote: Fetcher thread is per broker basis, it ensures that at lease one fetcher thread per broker. Fetcher thread

Re: createMessageStreams vs createMessageStreamsByFilter

On Mar 11, 2015, at 9:12 AM, Guozhang Wang wangg...@gmail.com wrote: Hi James, What I meant before is that a single fetcher may be responsible for putting fetched data to multiple queues according to the construction of the streams setup, where each queue may be consumed by a different

Re: Database Replication Question

On Mar 5, 2015, at 12:59 AM, Xiao lixiao1...@gmail.com wrote: Hi, James, This design regarding the restart point has a few potential issues, I think. - The restart point is based on the messages that you last published. The message could be pruned. How large is your

Re: Database Replication Question

. I will probably have a consumer that will read that all tables topic, and split the data out into separate topics, for consumers who just want a subset of the data. -James On Mar 4, 2015, at 9:28 AM, Jonathan Hodges hodg...@gmail.com wrote: Yes you are right on the oplog per partition

Re: Database Replication Question

message? Send 1 Wait for ack Send 2 Wait for ack etc If I try to send 1,2,3,4,5 in a batch, is it possible that the broker could receive 1,2 and 4,5, and that only 3 would fail? Or is it always a contiguous chunk, and then the first failure would cause the rest of the batch to abort? -James

Re: Kafka 0.8.2 log cleaner

snapshot of the final stage (otherwise delete tombstones may be collected before they complete their scan). This setting can be overridden on a per-topic basis (see the per-topic configuration section). http://kafka.apache.org/documentation.html#brokerconfigs -James On Mar 2, 2015, at 8:57 AM, Ivan

If you run Kafka in AWS or Docker, how do you persist data?

replication? Or do you bring back up the broker with the persisted files? Trying to learn about what people are doing, beyond on premises and dedicated hardware. Thanks, -James

Re: KafkaProducer.send contract

is unfortunate since it's artificially slowing message rate down just in case the send() fails. I appreciate it'd be awesome if Kafka could track this; but if it's failing to write my message, it's likely having issues that would impede its ability to track the failure state too. JAmes On Mon

KafkaProducer.send contract

correct in believing that I'll have to manually manage having one message in flight per node at a time? Or is there a mechanism to say This message and all messages after it for this partition were rejected? (or something similar) Thanks! JAmes

Re: AWS EC2 deployment best practices

-4EO3PQ_YOLbvf6HKpqBTNQ8fyYeRuDMrlDYlK0yQ/pubchart?oid=1634430904format=interactive -James On Sep 30, 2014, at 7:47 AM, Philip O'Toole philip.oto...@yahoo.com.INVALID wrote: OK, yeah, speaking from experience I would be comfortable with using the ephemeral storage if it's replicated across

JOB | Permanent Java Developer (London, UK)

of this list may like to discuss with me further off-list using the email address JamesBTobin (at) Gmail (dot) Com. All the best, James

Re: questions about ISR

. Regards, Libo -- Friendly regards, *James Wu https://plus.google.com/u/0/100829801349304669533 *

Re: Questions about the leader election

, Aug 22, 2013 at 8:51 PM, James Wu jameswu...@gmail.com wrote: Hi, I am wondering what is the mechanism that Kafka elects the leader of partitions ? Does it handle by the controller process or ? If the leader crashed, who will decide the new leader ? and the process is running

Questions about the leader election

Hi, I am wondering what is the mechanism that Kafka elects the leader of partitions ? Does it handle by the controller process or ? If the leader crashed, who will decide the new leader ? and the process is running on Zookeeper or Kafka? Thanks. -- -- Friendly regards, *James Wu http

Re: How to get broker back to ISR

, James Wu jameswu...@gmail.com wrote: Hi, I am wondering if my leader broker crash, how to get it back to ISR after restart kafak ? In the initial status the kafka-list-topic.sh shows: topic: failover-test partition: 0 leader: 0 replicas: 0,1 isr: 0,1 If I terminate the leader

Re: How to get broker back to ISR

Sorry, the link is : http://mail-archives.apache.org/mod_mbox/kafka-users/201308.mbox/raw/%3CFF142F6B499AE34CAED4D263F6CA32901D34E289%40EXTXMB19.nam.nsroot.net%3E/1 On Thu, Aug 22, 2013 at 12:46 AM, James Wu jameswu...@gmail.com wrote: Thank you Neha. Actually my situation was more like

Re: use kafka as dependency in sbt from central maven repository

attributes for the zookeeper dependency in the custom inline xml included in kafkas Build.scala anybody else having these problems and maybe some advice for a solution/workaround? regards lorenz -- -- Friendly regards, *James Wu http://www.facebook.com/jameswu629

<    1   2   3   >