Re: Kafka integration with Hadoop

Also, Flume 1.6 has Kafka source, sink and channel. Using Kafka source or channel with Dataset sink should work. Gwen On Tue, May 12, 2015 at 6:16 AM, Warren Henning warren.henn...@gmail.com wrote: You could start by looking at Linkedin's Camus and go from there? On Mon, May 11, 2015 at 8:10

Re: Log end offset

Hi James, The thread was different from the one I was asking. When I issue a OffsetFetchRequest to the channel, I am getting a ClosedChannelException and all of them seem to come from a single broker. I don't see anything problematic with the broker (config, exceptions.etc.) but requests for

Kafka broker and producer max message default config

Hi, I see configuration for broker max.message.bytes 1,000,000 and configuration for producer max.request.size 1,048,576 Why is default config for broker is less than producer? If that is the case then there will be message sent by producer which is bigger than what broker could receive. Could

Reassigning partitions off of a permanently failed broker

Hi there, I'm working on a script that fails kafka v8.2 brokers from the cluster, mostly intended for dealing with long term downtimes such as hardware failures. The script generates a new partition assignment, moving any replica on the failed host to other available hosts. The problem I'm

Re: New Producer Async - Metadata Fetch Timeout

The way it works I suppose is that, the producer will do fetchMetadata, if the last fetched metadata is stale (the refresh interval has expired) or if it is not able to send data to a particular broker in its current metadata (This might happen in some cases like if the leader moves). It cannot

Re: New Producer Async - Metadata Fetch Timeout

I could not follow the reasoning behind blocking the send method if the metadata is not up-to-date. Though, I see that it as per design, it requires the metadata to batch the message into appropriate topicPartition queue. Also, if the metadata could not be updated in the specified interval, it

Re: Could this be happening?

Hi Scott, what producer client are you using? Reordering is possible in async producers in the case of temporary broker failures and the combination of request.required.acks != 0 and retries 0. Consider the case where a producer has 20 messages in-flight to the broker, out of those messages #

Re: Kafka broker and producer max message default config

The max.request.size effectively caps the largest size message the producer will send, but the actual purpose is, as the name implies, to limit the size of a request, which could potentially include many messages. This keeps the producer from sending very large requests to the broker. The

Re: Kafka log compression change in 0.8.2.1?

Hi Jun, I figured it out this morning and opened https://issues.apache.org/jira/browse/KAFKA-2189 -- it turned out to be a bug in versions 1.1.1.2 through 1.1.1.6 of snappy-java that has recently been fixed (I was very happy to see their new unit test named

Re: New Producer Async - Metadata Fetch Timeout

That¹s right. Send() will first try to get metadata of a topic, that is a blocking operation. On 5/12/15, 2:48 AM, Rendy Bambang Junior rendy.b.jun...@gmail.com wrote: Hi, sorry if my understanding is incorrect. I am integrating kafka producer with application, when i try to shutdown all kafka

Compression and batching

Hi, I'm wondering when you call kafka.javaapi.Producer.send() with a list of messages, and also have compression on (snappy in this case), how does it decide how many messages to put together as one? The reason I'm asking is that even though my messages are only 70kb uncompressed, the broker

Re: Kafka broker and producer max message default config

Thanks, I get the difference now. This is assuming request to be sent contains at least 1 messages. Isn't it? Rendy On May 13, 2015 3:55 AM, Ewen Cheslack-Postava e...@confluent.io wrote: The max.request.size effectively caps the largest size message the producer will send, but the actual

questions

hi, How can i send data from a log file to kafka server? Can i use kafka with flume. or is there any other way to do it Thanks

Kafka 0.9 release and future 0.8.2.1 client compatibility

Hi, I would like to start using Kafka, can I start from 0.9 or is it better to develop on 0.8.2.1 and than migrate to 0.9 ? My plans are to be in production by september Will a 0.8.2.1 client (producer/consumer) be able to talk to 0.9 brokers ? Is there any public maven artifact for Kafka 0.9 ?

Re: Kafka log compression change in 0.8.2.1?

Hi, Andrew, Thanks for finding this out. I marked KAFKA-2189 as a blocker for 0.8.3. Could you share your experience on snappy 1.1.1.7 in the jira once you have tried it out? If the result looks good, we can upgrade the snappy version in trunk. Jun On Tue, May 12, 2015 at 1:23 PM, Olson,Andrew

Re: New Producer Async - Metadata Fetch Timeout

Send() will only block if the metadata is *not available* for the topic. It won’t block if metadata there is stale. The metadata refresh is async to send(). However, if you send the message to a topic for the first time, send() will trigger a metadata refresh and block until it has metadata for

Using 0.8.1.1 kafka producer/consumer with 0.8.2.1 broker

I am in the process of testing and migrating our prod kafka from 0.8.1.1 to 0.8.2.1. Wanted to do a quick check with the community if anyone has observed any issue with writing/reading data to 0.8.2.1 kafka broker(s), using 0.8.1.1 producer and consumer. Any gotchas to watch for or any

RE: Kafka 0.8.2.1 - Listing partitions owned by consumers

Perhaps you could try the ConsumerOffsetChecker. The Owner field might be what you want.. Aditya From: Bharath Srinivasan [bharath...@gmail.com] Sent: Tuesday, May 12, 2015 7:29 PM To: users@kafka.apache.org Subject: Kafka 0.8.2.1 - Listing partitions

Re: Compression and batching

Well, the batch size is decided by the value set for the property : batch.size; The producer will attempt to batch records together into fewer requests whenever multiple records are being sent to the same partition. This helps performance on both the client and the server. This configuration

Re: Kafka 0.8.2.1 - Listing partitions owned by consumers

As we need to do this programmatically, i tried to strip out the relevant parts from ConsumerOffsetChecker. It did work. Thanks for the suggestions. On Tue, May 12, 2015 at 8:58 PM, Mayuresh Gharat gharatmayures...@gmail.com wrote: Well, there is no separate tool available for importing and

Re: Compression and batching

Mayuresh, this is about the old producer instead of the new Java producer. Jamie, In the old producer, if you use sync mode, the list of message will be sent as a batch. On the other hand, if you are using async mode, the messages are just put into the queue and batched with other messages.

Re: Kafka 0.8.2.1 - Listing partitions owned by consumers

Well, there is no separate tool available for importing and exporting offsets from kafka, which will also provide this functionality. We are working on it. You can try the consumerOffsetChecker as Aditya mentioned. Thanks, Mayuresh On Tue, May 12, 2015 at 8:11 PM, Aditya Auradkar

Re: Message corruption with new Java client + snappy + broker restart

Oops. I originally sent this to the dev list but meant to send it here. Hi, When using Samza 0.9.0 which uses the new Java producer client and snappy enabled, I see messages getting corrupted on the client side. It never happens with the old producer and it never happens with lz4, gzip, or

Re: questions

Kafka with Flume is one way (Just use Flume's SpoolingDirectory source with Kafka Channel or Kafka Sink). Also, Kafka itself has a Log4J appender as part of the project, this will work if the log is written with log4j. Gwen On Tue, May 12, 2015 at 12:52 PM, ram kumar ramkumarro...@gmail.com