Re: Kafka security

Don't hard code it. Martin's suggestion allows it to be read from a configuration file or injected from another source such as an environment variable at runtime. If you neither of these are acceptable for corporate policy I suggest asking how it has been handled before at your company.

Re: Encryption at Rest

"We need to be capable of changing encryption keys on regular intervals and in case of expected key compromise." is achievable with full disk encryption particularly if you are willing to add and remove Kafka servers so that you replicate the data to new machines/disks with new keys and take the

Re: Encryption at Rest

>From what I know of previous discussions encryption at rest can be handled with transparent disk encryption. When that's sufficient it's nice and easy. Christian On Thu, Apr 21, 2016 at 2:31 PM, Tauzell, Dave wrote: > Has there been any discussion or work on at

Re: Kafka over Satellite links

I would not do that. I admit I may be a bit biased due to working for Buddy Platform (IoT backend stuff including telemetry collection), but you want to send the data via some protocol (HTTP? MQTT? COAP?) to the central hub and then have those servers put the data into Kafka. Now if you want to

Re: Kafka behind AWS ELB

Dillian, On Mon, May 4, 2015 at 1:52 PM, Dillian Murphey crackshotm...@gmail.com wrote: I'm interested in this topic as well. If you put kafka brokers inside an autoscaling group, then AWS will automatically add brokers if demand increases, and the ELB will automatically round-robin across

Re: Kafka Poll: Version You Use?

Do you have a anything on the number of voters, or audience breakdown? Christian On Wed, Mar 4, 2015 at 8:08 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hello hello, Results of the poll are here! Any guesses before looking? What % of Kafka users are on 0.8.2.x already? What %

Re: kafka producer does not distribute messages to partitions evenly?

I believe you are seeing the behavior where the random partitioner is sticky. http://mail-archives.apache.org/mod_mbox/kafka-users/201309.mbox/%3ccahwhrrxax5ynimqnacsk7jcggnhjc340y4qbqoqcismm43u...@mail.gmail.com%3E has details. So with the default 10 minute refresh if your test is only an hour or

Re: Tips for working with Kafka and data streams

I wouldn't say no to some discussion of encryption. We're running on Azure EventHubs (with preparations for Kinesis for EC2, and Kafka for deployments in customer datacenters when needed) so can't just use disk level encryption (which would have its own overhead). We're putting all of our messages

Re: Tips for working with Kafka and data streams

and producers, still leaving it up to the clients, but facilitating the key handling. Julio On 2/25/15, 4:24 PM, Christian Csar christ...@csar.us wrote: The questions we get from customers typically end up being general so we break out our answer into network level and on disk scenarios

Re: Tips for working with Kafka and data streams

). Christian On Wed, Feb 25, 2015 at 3:57 PM, Jay Kreps jay.kr...@gmail.com wrote: Hey Christian, That makes sense. I agree that would be a good area to dive into. Are you primarily interested in network level security or encryption on disk? -Jay On Wed, Feb 25, 2015 at 1:38 PM, Christian Csar

Re: Proper Relationship Between Partition and Threads

Ricardo, The parallelism of each logical consumer (consumer group) is the number of partitions. So with four partitions it could make sense to have one logical consumer (application) have two processes on different machines each with two threads, or one process with four. While with two logical

Re: How to mark a message as needing to retry in Kafka?

noodles, Without an external mechanism you won't be able to mark individual messages/offsets as needing to be retried at a later time. Guozhang is describing a way to get the offset of a message that's been received so that you can find it later. You would need to save that into a 'failed

Re: Is kafka suitable for our architecture?

% of cases the size is less than 50Kb) Regards On 9 October 2014 20:02, Christian Csar cac...@gmail.com wrote: Apart from your data locality problem it sounds like what you want is a workqueue. Kafka's consumer structure doesn't lend itself too well to that use case as a single partition of a topic

Re: Is kafka suitable for our architecture?

Apart from your data locality problem it sounds like what you want is a workqueue. Kafka's consumer structure doesn't lend itself too well to that use case as a single partition of a topic should only have one consumer instance per logical subscriber of the topic, and that consumer would not be

Re: Use case

The thought experiment I did ended up having a set of front end servers corresponding to a given chunk of the user id space, each of which was a separate subscriber to the same set of partitions. The you have one or more partitions corresponding to that same chunk of users. You want the chunk/set

Re: Handling send failures with async producer

TLDR: I use one Callback per job I send to Kafka and include that sort of information by reference in the Callback instance. Our system is currently moving data from beanstalkd to Kafka due to historical reasons so we use the callback to either delete or release the message depending on success.

Re: EBCDIC support

Having been spared any EBCDIC experience whatsoever (ie from a positio of thorough ignorance), if you are transmitting text or things with a designated textual form (presumably) I would recommend that your conversion be to unicode rather than ascii if you don't already have consumers expecting a

Re: Which producer to use?

I ended up coding against the new one, org.apache.kafka.clients.producer.Producer, though it is not yet in production here. It might be slightly more painful to select a partition since there isn't a place to plug in a partitioner class, but overall it was quite easy and had the key feature of an

0.8.1 Java Producer API Callbacks

I'm looking at using the java producer api for 0.8.1 and I'm slightly confused by this passage from section 4.4 of https://kafka.apache.org/documentation.html#theproducer Note that as of Kafka 0.8.1 the async producer does not have a callback, which could be used to register handlers to catch send

Re: 0.8.1 Java Producer API Callbacks

On 05/01/2014 07:22 PM, Christian Csar wrote: I'm looking at using the java producer api for 0.8.1 and I'm slightly confused by this passage from section 4.4 of https://kafka.apache.org/documentation.html#theproducer Note that as of Kafka 0.8.1 the async producer does not have a callback