RE: Security for individual partitions

Hi, Yash I don't know the version and language of consumer client you use. Java client [kafka-clients-0.9.0.0] provides a method [public void assign(List partitions)] in KafkaConsumer.class to manually assign a list of partition to this consumer. This may be what you want. >-Original

Re: NPE in low level Kafka topology

Hi Frank, One issue that I can remember is this one: https://issues.apache.org/jira/browse/KAFKA-4324 Since I have not seen your full topology building code including com.dexels.kafka.streams.remotejoin.StoreProcessor, com.dexels.kafka.streams.remotejoin.PreJoinProcessor and

Re: ConsumerConnector and 0.10 message format

The performance impact happens if the consumer doesn't support the message format defined by log.message.format.version. The old consumer in Kafka 0.10 supports the message format introduced in 0.10 (including timestamps and the ability to avoid recompression in the produce path) so what you

Re: [VOTE] 2.0.0 RC0

+1 (non-binding) Heroku has run our usual set of upgrade and performance tests, and we haven't found any notable issues through that. On Sat, Jun 23, 2018 at 12:30 AM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > +1 (non-binding) > > Built from source and ran quickstart successfully

Re: [kafka-clients] Re: [VOTE] 1.1.1 RC1

+1 (non-binding) Ran tests, Verified quick start, producer/consumer perf tests On Sat, Jun 23, 2018 at 8:11 AM Dong Lin wrote: > Thank you for testing and voting the release! > > I noticed that the date for 1.1.1-rc1 is wrong. Please kindly test and > vote by Tuesday, June 26, 12 pm PT. > >

Re: Kafka Streams Session store fetch latency very high with caching turned on

Hello Sam, That is an interesting find. My reasoning is similar to yours: since you have 1K / sec input traffic, it means 3600K / hour. Since you mean there are about 500K / hour unique keys, it means each key will be updated roughly about 7 times per hour. Assuming the traffic is even not

Re: Retries

Well it does matter just because the variable wasn’t even in the 1.1 build, because as you probably know there was already a Jira opened that the variable was set as private and thus couldn’t be used. It was fixed but if you want to keep your build aligned with official releases then you can’t

Re: Retries

`default.production.exception.handler` is the correct parameter. It should not make any difference if you pass in a plain String or `StreamConfig#DEFAULT_PRODUCTION_EXCEPTION_HANDLER_CLASS_CONFIG` -- the variable is also just a String... When you start the application, it should the config in

[VOTE] KIP-312: Add Overloaded StreamsBuilder Build Method to Accept java.util.Properties

All, I'd like to start a vote for this KIP now. Thanks, Bill

Re: Security for individual partitions

Kafka ACLs are at the topic level, not partition level. Probably better to make 10 topics of 1 partition each and use topic ACLs to control access. -hans > On Jun 25, 2018, at 9:50 PM, Yash Ganthe wrote: > > Hi, > > If I have a topic with 10 partitions, I would like each partition to be >

Re: Security for individual partitions

Hi Yash, I'm afraid this is not easily possible with existing functionality. Even if you created your own authorizer, I'm fairly certain that the partition is not available as part of the resource that is being accessed. Is there any specific reason why you can't create more than one topic and

Is possible to have an infinite delete.retention.ms?

Hi all, I would like to setup a compaction policy on a topic where a message can be deleted (GDPR..) using a tombstone with the same key that the message to be removed. My problem is that I would like to use empty payload messages also for identifying that an entity has been deleted, but these

Security for individual partitions

Hi, If I have a topic with 10 partitions, I would like each partition to be accessible to only certain consumers. Consumer 1 should be able to read from partition 1 but no other partition and so on. Is this possible in Kafka? -Yash