Re: Kafka encryption

Thanks, How can we do file system encryption? we are using aws environment. Thanks, Snehalata - Original Message - From: "Gerard Klijs" To: "Users" Sent: Tuesday, May 24, 2016 7:26:27 PM Subject: Re: Kafka encryption For both old

Re: FW: [ANNOUCE] Apache Kafka 0.10.0.0 Released

To be fair to Spark, we are encouraging the client integrations to be very conservative with new releases. Spark with 0.8.x client will work on 0.10.0.0 broker, but not vice-versa. So it doesn't make tons of sense for Spark to upgrade until there is critical mass of users on the new release. We

Re: kafka-consumer-group.sh failed on 0.10.0 but works on 0.9.0.1

Thanks Jason for that insight. I will use the 0.9 tools until i upgrade all the brokers. I suppose it should be documented somewhere so others dont run into the same issue and think something is wrong? On Tue, May 24, 2016 at 7:25 PM, Jason Gustafson wrote: > I went ahead

Re: Will segments on no-traffic topics get deleted/compacted?

I ran some tests, and also looked through the code. log.roll.ms isn't sufficient. Segments appear to get rolled only if log.roll.ms or segment.ms has passed log.segment.bytes has passed But those rules are only evaluated when a new record is appended to the log. So it appears that for

Re: kafka-consumer-group.sh failed on 0.10.0 but works on 0.9.0.1

Maybe i am doing this wrong [ac...@ekk001.scl ~]$ cat /opt/kafka/kafka_2.11-0.10.0.0/config/log4j.properties .. log4j.rootLogger=DEBUG, kafkaAppender .. See no extra logs when running the consumer-group.sh tool. On Tue, May 24, 2016 at 5:46 PM, Jason Gustafson wrote: >

FW: [ANNOUCE] Apache Kafka 0.10.0.0 Released

Does anyone know if spark plans to upgrade? I think the current version is 0.8x? Kind regards Andy From: Gwen Shapira Reply-To: Date: Tuesday, May 24, 2016 at 9:24 AM To: , ,

Re: kafka-consumer-group.sh failed on 0.10.0 but works on 0.9.0.1

I went ahead and tried this locally and the new topic metadata request does appear to be the problem. Unfortunately, the tools are bound by the same compatibility model as the clients, which means there is no guarantee that they work with older versions. As a workaround, I guess you can use the

kafka-consumer-group.sh failed on 0.10.0 but works on 0.9.0.1

I upgraded one of my brokers to 0.10.0. I followed the upgrade guide and added these to my server.properties: inter.broker.protocol.version=0.9.0.1 log.message.format.version=0.9.0.1 When checking the lag i get this error. [ac...@ekk001.scl ~]$ sudo

Re: kafka-consumer-group.sh failed on 0.10.0 but works on 0.9.0.1

I am pretty sure consumer-group.sh uses tools-log4j.properties On Tue, 24 May 2016 at 17:59 allen chan wrote: > Maybe i am doing this wrong > > [ac...@ekk001.scl ~]$ cat > /opt/kafka/kafka_2.11-0.10.0.0/config/log4j.properties > .. > log4j.rootLogger=DEBUG,

RE: Large kafka deployment on virtual hardware

Thanks for the response Jens. That is one of the first things we looked at, was the ZK performance. ZK seems nominal, low load and no exhaustion of any resources. ZK offset commit is not set by us explicitly, so default. Kind regards, Jahn Roux -Original Message- From: Jens Rantil

Re: Large kafka deployment on virtual hardware

I'm sure you checked this but since these are virtual machines, is it possible there is just contention for resources? Network clogged or some other simpler explanation like that? On Mon, May 23, 2016 at 9:42 PM, Jahn Roux wrote: > I have a large Kafka deployment on virtual

RE: Large kafka deployment on virtual hardware

Thank you for the response. Yes, we have had a number of experts investigate the underlying resource provision and there are no clear issues that stand out - from a virtual and host hardware/resource perspective the system is busy but nothing indicates it is overburdened. Kind regards, Jahn

Re: Large kafka deployment on virtual hardware

Hi Jahn, I'm assuming your using Java, maybe try another Java client? I know you said that CPU was nominal but maybe this client from blackberry will help: https://github.com/blackberry/Krackle Thanks, Ben On Tue, May 24, 2016 at 8:34 AM, Jahn Roux wrote: > Thanks for the

Re: kafka 0.8.2 broker behaviour

yeh mostly they are up always, sometimes when we do new deployment they might be down for about 15 mins or so. seems to have become slightly better now, i changed the replica socket buffer size and also the number of replica fetcher threads. On Mon, May 23, 2016 at 6:24 PM, Gerard Klijs

Kafka encryption

Hi All, We have requirement of encryption in kafka. As per docs, we can configure kafka with ssl, for secured communication. But does kafka also stores data in encrypted format? Thanks, Snehalata

Re: Kafka encryption

Yes,it does that.What specifically you are looking for? On 5/24/16, 3:52 PM, "Snehalata Nagaje" wrote: >Hi All, > > >We have requirement of encryption in kafka. > >As per docs, we can configure kafka with ssl, for secured communication. > >But does

Reporting security issues

Hi all, Since Kafka implements a number of security features, we need a procedure for reporting potential security vulnerabilities privately (as per http://www.apache.org/security/). We have added a simple page to the website that describes the procedure (thanks Flavio):

Re: Kafka encryption

Thanks for quick reply. Do you mean If I see messages in kafka, those will not be readable? And also, we are using new producer but old consumer , does old consumer have ssl support? As mentioned in document, its not there. Thanks, Snehalata - Original Message - From: "Mudit

Re: Kafka encryption

Hi, There's no encryption at rest. It's recommended to use filesystem encryption, or encryption of each individual message before producing it for this. Only the new producer and consumers have SSL support. Thanks Tom Crayford Heroku Kafka On Tue, May 24, 2016 at 11:33 AM, Snehalata Nagaje <

Re: Large kafka deployment on virtual hardware

Jahn, Are all these brokers running on the same underlying machine? Doing so seems highly against the usual fault tolerance properties of Kafka, and I'd expect there to be some hidden performance issues in the hypervisor at that point. Are you running with the new producer or the old one? Are

RE: Large kafka deployment on virtual hardware

Hi Tom, I appreciate you taking the time to respond to my request. I believe at the moment we only have 2 to 4 virtuals running on a single host. This is probably not ideal, but this is what we are stuck with - essentially "cloud" VM hardware. I am not sure about the producer, I believe it to

Re: Large kafka deployment on virtual hardware

2-4 seems ok with me, as long as the network isn't bound or dropping packets/etc then you're probably ok. The new producer is this class: https://kafka.apache.org/082/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html - if you're using that then it's new producer, otherwise it's old

Re: Kafka encryption

For both old and new consumers/producers you can make your own (de)serializer to do some encryption, maybe that could be an option? On Tue, May 24, 2016 at 2:40 PM Tom Crayford wrote: > Hi, > > There's no encryption at rest. It's recommended to use filesystem > encryption,

Re: Kafka Scalability with the Number of Partitions

What's your server setup for the brokers and consumers? Generally I'd expect something to be exhausted here and that to end up being the bottleneck. Thanks Tom Crayford Heroku Kafka On Mon, May 23, 2016 at 7:32 PM, Yazeed Alabdulkarim < y.alabdulka...@gmail.com> wrote: > Hi, > I am running

Re: Which should scale: Producer or Topic

Is that "one instance of the producer class per topic"? I'd recommend just having a single producer shared per process. 1 million records in a week is not very many records, it works down to ~1.6 records a second on average, which is nothing (we typically see 1 million+ messages per second on our

Which should scale: Producer or Topic

Hello Folks, I am using Kafka (0.9) in my company and it is expected that we are going to receive 1 million records in next week. I have many topics for solely different purposes. Is it good that I define one producer per topic or create one producer for every topic? Right now, I have only 4

Re: Which should scale: Producer or Topic

Tom, Thank you for your answer. No, I am talking about one PRODUCER for each topic, not one instance of same producer class. I am asking for general concept only. Actually we are just growing and not so much far from the case of 1 million records per sec. Just considering our future case, I need

Re: Which should scale: Producer or Topic

Hi, I think I'm a bit confused. When you say "one producer per topic", do you mean one instance of the JVM application that's producing per topic? Thanks Tom On Tue, May 24, 2016 at 4:19 PM, Hafsa Asif wrote: > Tom, > > Thank you for your answer. No, I am talking

Re: KafkaSpout support for Kafka 0.9 version

We're using 0.9 with Storm 0.10 and haven't found any issues so far. The only real down side is that you're probably not going to be able to use the security features since it uses 0.9 in 0.8 compatibility mode on the client side. On Tue, May 24, 2016 at 11:24 AM, Kanagha

Re: Which should scale: Producer or Topic

Ok, let me rephrase (may be I am not using correct terms): Simply consider I have 2 topics, and I have both Java and NodeJS client for Kafka. *NodeJS:* Is it good that I write two producers per each topic like that : *var producer1 = new Producer(client);producer1.on('ready', function

Re: Which should scale: Producer or Topic

One more question: How many topics can be easily handled by one producer? Hafsa 2016-05-24 17:39 GMT+02:00 Hafsa Asif : > Ok, let me rephrase (may be I am not using correct terms): > Simply consider I have 2 topics, and I have both Java and NodeJS client > for Kafka.

KafkaSpout support for Kafka 0.9 version

Hi, Does Apache Storm KafkaSpout support 0.9 kafka version? Thanks Kanagha -- Kanagha

Re: Which should scale: Producer or Topic

Aha, yep that helped a lot. One producer per process. There's not really a per producer topic limit. There's buffering and batching space, but assuming you have sufficient memory (which is by the partition, not by topic), you'll be fine. Thanks Tom Crayford Heroku Kafka On Tue, May 24, 2016 at

Re: Precautions Before Restarting a Down Kafka Instance

No real precautions need to be taken on starting a down instance, assuming you have replication in play and the controller is up and active. We routinely restart downed broker processes, and have never had an issue with it (running thousands of clusters with an ops team of ~3). Thanks Tom

Re: Which should scale: Producer or Topic

Interesting discussion! What do you mean here by a process? Is that a thread or the JVM process? On Tue, May 24, 2016 at 5:49 PM, Tom Crayford wrote: > Aha, yep that helped a lot. > > One producer per process. There's not really a per producer topic limit. > There's

Re: [ANNOUCE] Apache Kafka 0.10.0.0 Released

I had an offline chat with Gwen and the 0.10.0.0-rc6 tag is the right one. I've updated the 0.10.0.0 tag to be the same: https://github.com/apache/kafka/tree/0.10.0.0 Ismael On Tue, May 24, 2016 at 6:04 PM, Ismael Juma wrote: > Hmm, sorry. The tag seems wrong. The commit

Re: [ANNOUCE] Apache Kafka 0.10.0.0 Released

Awesome! Thanks for managing the release Gwen! On Tue, May 24, 2016 at 11:24 AM, Gwen Shapira wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka 0.10.0.0. > This is a major release with exciting new features, including first > release

Re: [ANNOUCE] Apache Kafka 0.10.0.0 Released

Woohoo!!! :-) -Jay On Tue, May 24, 2016 at 9:24 AM, Gwen Shapira wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka 0.10.0.0. > This is a major release with exciting new features, including first > release of KafkaStreams and many

Re: 0.10 Kafka with 0.9 brokers

You may face issues if client version is higher than broker version. On 5/24/16, 10:13 PM, "Oleg Zhurakousky" wrote: >With 0.10 Kafka being announced, are there any known compatibility issues with >0.9 brokers? > >Thanks >Oleg

Re: [ANNOUCE] Apache Kafka 0.10.0.0 Released

Can I just confirm that https://github.com/apache/kafka/commit/b8642491e78c5a137f5012e31d347c01f3b02339 is the official commit for the release? The source download doesn't have the git repo and I can't see a sha anywhere in the downloaded source. On Tue, May 24, 2016 at 5:42 PM, Becket Qin

[ANNOUCE] Apache Kafka 0.10.0.0 Released

The Apache Kafka community is pleased to announce the release for Apache Kafka 0.10.0.0. This is a major release with exciting new features, including first release of KafkaStreams and many other improvements. All of the changes in this release can be found:

Re: [ANNOUCE] Apache Kafka 0.10.0.0 Released

Awesome! On Tue, May 24, 2016 at 9:41 AM, Jay Kreps wrote: > Woohoo!!! :-) > > -Jay > > On Tue, May 24, 2016 at 9:24 AM, Gwen Shapira wrote: > > > The Apache Kafka community is pleased to announce the release for Apache > > Kafka 0.10.0.0. > > This is a

0.10 Kafka with 0.9 brokers

With 0.10 Kafka being announced, are there any known compatibility issues with 0.9 brokers? Thanks Oleg

Re: [ANNOUCE] Apache Kafka 0.10.0.0 Released

Hi Tom, The official commit can always be found via the relevant Git tag: https://github.com/apache/kafka/tree/0.10.0.0 https://github.com/apache/kafka/commit/1b5879653e0d956c79556301d1d11987baf6f2d7 Ismael On Tue, May 24, 2016 at 5:57 PM, Tom Crayford wrote: > Can I

Re: [ANNOUCE] Apache Kafka 0.10.0.0 Released

Hmm, sorry. The tag seems wrong. The commit you linked Tom seems the correct one: https://github.com/apache/kafka/commit/b8642491e78c5a137f5012e31d347c01f3b02339 Gwen, is this right? Ismael On Tue, May 24, 2016 at 6:03 PM, Ismael Juma wrote: > Hi Tom, > > The official

Re: [ANNOUCE] Apache Kafka 0.10.0.0 Released

Awesome, thanks for running the release Gwen. :) Ismael On Tue, May 24, 2016 at 5:24 PM, Gwen Shapira wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka 0.10.0.0. > This is a major release with exciting new features, including first >

Re: [ANNOUCE] Apache Kafka 0.10.0.0 Released

Great!!! cheers :) Thanks, Mayuresh On Tue, May 24, 2016 at 10:07 AM, Ismael Juma wrote: > Awesome, thanks for running the release Gwen. :) > > Ismael > > On Tue, May 24, 2016 at 5:24 PM, Gwen Shapira wrote: > > > The Apache Kafka community is pleased

Re: 0.10 Kafka with 0.9 brokers

Oleg, Please read this section for details: http://kafka.apache.org/documentation.html#upgrade Guozhang On Tue, May 24, 2016 at 9:45 AM, Mudit Kumar wrote: > You may face issues if client version is higher than broker version. > > > > > On 5/24/16, 10:13 PM, "Oleg

Re: [ANNOUCE] Apache Kafka 0.10.0.0 Released

Thanks for running the release Gwen!! On Tue, May 24, 2016 at 11:47 AM, Grant Henke wrote: > Awesome! Thanks for managing the release Gwen! > > On Tue, May 24, 2016 at 11:24 AM, Gwen Shapira > wrote: > > > The Apache Kafka community is pleased to