Re: [DISCUSS] KIP-561: Regex Expressions Support for ConsumerGroupCommand

[Alexander Dunayevsky]

Any additional feedback on this?

Best Regards,
Alex Dunayevsky


On Thu, 23 Jan 2020, 11:39 Alexander Dunayevsky, 
wrote:

> Hello guys,
>
> Let's discuss KIP-561 Regex Support for ConsumerGroupCommand:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-561%3A+Regex+Support+for+ConsumerGroupCommand
>
> Functionality already implemented and waiting to be reviewed.
>
> Best Regards,
> Alex Dunayevsky
>
>
> On Thu, 16 Jan 2020, 14:25 Alex D,  wrote:
>
>> Hello, guys,
>>
>> Please review Regex Expressions Support for ConsumerGroupCommand
>> improvement proposal
>>
>>- *Previous Discussion 1*: Re: Multiple Consumer Group Management
>>
>>- *Previous Discussion 2*: Re: ConsumerGroupCommand tool improvement?
>>
>>
>> *JIRA*: KAFKA-7817 Multiple Consumer Group Management with Regex
>> 
>>
>> *PR*: #6700 
>>
>

Build failed in Jenkins: kafka-trunk-jdk8 #4185

[Apache Jenkins Server]

See 


Changes:

[matthias] MINOR: Fix topology builder debug log message (#8005)


--
[...truncated 2.84 MB...]

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutIfAbsentWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutIfAbsentWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardInit 
PASSED

> Task 

Build failed in Jenkins: kafka-trunk-jdk11 #1109

[Apache Jenkins Server]

See 


Changes:

[matthias] MINOR: Fix topology builder debug log message (#8005)


--
[...truncated 2.86 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

Re: [KAFKA-557] Add emit on change support for Kafka Streams

[Richard Yu]

Hello to all,

I've finished making some initial modifications to the KIP.
I have decided to keep the implementation section in the KIP for
record-keeping purposes.

For now, we should focus on only the proposed behavior changes instead.

See if you have any comments!

Cheers,
Richard

On Sat, Jan 25, 2020 at 11:12 AM Richard Yu 
wrote:

> Hi all,
>
> Thanks for all the discussion!
>
> @John and @Bruno I will survey other possible systems and see what I can
> do.
> Just a question, by systems, I suppose you would mean the pros and cons of
> different reporting strategies?
>
> I'm not completely certain on this point, so it would be great if you can
> clarify on that.
>
> So here's what I got from all the discussion so far:
>
>- Since both Matthias and John seems to have come to a consensus on
>this, then we will go for an all-round behavorial change for KTables. After
>some thought, I decided that for now, an opt-out config will not be added.
>As John have pointed out, no-op changes tend to explode further down the
>topology as they are forwarded to more and more processor nodes downstream.
>- About using hash codes, after some explanation from John, it looks
>like hash codes might not be as ideal (for implementation). For now, we
>will omit that detail, and save it for the PR.
>- @Bruno You do have valid concerns. Though, I am not completely
>certain if we want to do emit-on-change only for materialized KTables. I
>will put it down in the KIP regardless.
>
> I will do my best to address all points raised so far on the discussion.
> Hope we could keep this going!
>
> Best,
> Richard
>
> On Fri, Jan 24, 2020 at 6:07 PM Bruno Cadonna  wrote:
>
>> Thank you Matthias for the use cases!
>>
>> Looking at both use cases, I think you need to elaborate on them in
>> the KIP, Richard.
>>
>> Emit from plain KTable:
>> I agree with Matthias that the lower timestamp makes sense because it
>> marks the start of the validity of the record. Idempotent records with
>> a higher timestamp can be safely ignored. A corner case that I
>> discussed with Matthias offline is when we do not materialize a KTable
>> due to optimization. Then we cannot avoid the idempotent records
>> because we do not keep the first record with the lower timestamp to
>> compare to.
>>
>> Emit from KTable with aggregations:
>> If we specify that an aggregation result should have the highest
>> timestamp of the records that participated in the aggregation, we
>> cannot ignore any idempotent records. Admittedly, the result of an
>> aggregation usually changes, but there are aggregations where the
>> result may not change like min and max, or sum when the incoming
>> records have a value of zero. In those cases, we could benefit of the
>> emit on change, but only if we define the semantics of the
>> aggregations to not use the highest timestamp of the participating
>> records for the result. In Kafka Streams, we do not have min, max, and
>> sum as explicit aggregations, but we need to provide an API to define
>> what timestamp should be used for the result of an aggregation if we
>> want to go down this path.
>>
>> All of this does not block this KIP and I just wanted to put this
>> aspects up for discussion. The KIP can limit itself to emit from
>> materialized KTables. However, the limits should be explicitly stated
>> in the KIP.
>>
>> Best,
>> Bruno
>>
>>
>>
>> On Fri, Jan 24, 2020 at 10:58 AM Matthias J. Sax 
>> wrote:
>> >
>> > IMHO, the question about semantics depends on the use case, in
>> > particular on the origin of a KTable.
>> >
>> > If there is a changlog topic that one reads directly into a KTable,
>> > emit-on-change does actually make sense, because the timestamp indicates
>> > _when_ the update was _effective_. For this case, it is semantically
>> > sound to _not_ update the timestamp in the store, because the second
>> > update is actually idempotent and advancing the timestamp is not ideal
>> > (one could even consider it to be wrong to advance the timestamp)
>> > because the "valid time" of the record pair did not change.
>> >
>> > This reasoning also applies to KTable-KTable joins.
>> >
>> > However, if the KTable is the result of an aggregation, I think
>> > emit-on-update is more natural, because the timestamp reflects the
>> > _last_ time (ie, highest timestamp) of all input records the contributed
>> > to the result. Hence, updating the timestamp and emitting a new record
>> > actually sounds correct to me. This applies to windowed and non-windowed
>> > aggregations IMHO.
>> >
>> > However, considering the argument that the timestamp should not be
>> > update in the first case in the store to begin with, both cases are
>> > actually the same, and both can be modeled as emit-on-change: if a
>> > `table()` operator does not update the timestamp if the value does not
>> > change, there is _no_ change and thus nothing is emitted. At the same
>> > time, if an aggregation operator does 

[jira] [Created] (KAFKA-9477) Doc: Add RoundRobinAssignor as an option to consumerc onfigs

[Alexandra Rodoni (Jira)]

Alexandra Rodoni created KAFKA-9477:
---

 Summary: Doc: Add RoundRobinAssignor as an option to consumerc 
onfigs
 Key: KAFKA-9477
 URL: https://issues.apache.org/jira/browse/KAFKA-9477
 Project: Kafka
  Issue Type: Improvement
Reporter: Alexandra Rodoni


Add: org.apache.kafka.clients.consumer.RoundRobinAssignor
as an option to partition.assignment.strategy
under https://kafka.apache.org/documentation/#consumerconfigs

Source is 
https://github.com/apache/kafka/blob/99a4068c5ca61951d70b9e647ead3b08a2af4309/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java#L106



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-546: Add quota-specific APIs to the Admin Client, redux

[Colin McCabe]

Thanks, Brian.

+1 (binding)

best,
Colin

On Sun, Jan 26, 2020, at 07:56, Brian Byrne wrote:
> Hello all,
> 
> I'd like to start a vote on KIP-456: Add quota-specific APIs to the Admin
> Client, redux
> 
> The KIP is here:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-546%3A+Add+quota-specific+APIs+to+the+Admin+Client%2C+redux
> 
> The discussion thread is here:
> https://lists.apache.org/thread.html/802de7a2167677bf647a729df74e828b6acd4185f2eb0a25ddb939f6%40%3Cdev.kafka.apache.org%3E
> 
> Thanks,
> Brian
>

Streams Processing meetup on Wednesday, February 5, 2020 at LinkedIn, Sunnyvale

[Joel Koshy]

*[bcc: (users,dev)@kafka.apache.org ]*

Hello,

The Streams Infra team invites you to attend the Streams Processing meetup
to be held on Wednesday, February 5, 2020. This meetup will focus on Apache
Kafka, Apache Samza and related streaming technologies.

*Where*: Unify Conference room, 950 W Maude Ave, Sunnyvale

*When*: 5:00 - 8:00 PM

*RSVP*: Please RSVP here (only if attending in person)
https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/267283444/
A streaming link will be posted approximately 30 minutes prior to the event.

*Agenda:*

 - 5:00 PM: Doors open and catered food available

5:00 - 6:00 PM: Networking

6:00 - 6:30 PM:
*High-performance data replication at Salesforce with MirusPaul Davidson,
Salesforce*

At Salesforce we manage high-volume Apache Kafka clusters in a growing
number of data centers around the globe. In the past we relied on Kafka's
Mirror Maker tool for cross-data center replication but, as the volume and
variety of data increased, we needed a new solution to maintain a high
standard of service reliability. In this talk, we will describe Mirus, our
open-source data replication tool based on Kafka Connect. Mirus was
designed for reliable, high-performance data replication at scale. It
successfully replaced MirrorMaker at Salesforce and has now been running
reliably in production for more than a year. We will give an overview of
the Mirus design and discuss the lessons we learned deploying, tuning, and
operating Mirus in a high-volume production environment.

6:30 - 7:00 PM: *Defending users from Abuse using Stream Processing at
LinkedIn, Bhargav Golla, LinkedIn *

When there are more than half a billion users, how can one effectively,
reliably and scalably classify them as good and bad users? This talk will
highlight how the Anti-Abuse team at LinkedIn leverages Streams Processing
technologies like Samza and Brooklin to keep the good users in a trusted
environment devoid of bad actors.

7:00 - 7:30 PM: *Enabling Mission-critical Stateful Stream Processing with
Samza, Ray Manpreet Singh Matharu, LinkedIn*

Samza powers a variety of large-scale business-critical stateful stream
processing applications at LinkedIn. Their scale necessitates using
persistent and replicated local state. Unfortunately, hard failures can
cause a loss of this local state, and re-caching it can incur downtime
ranging from a few minutes to hours! In this talk, we describe the systems
and protocols that we've devised that bound the down time to a few seconds.
We detail the tradeoffs our approach brings and how we tackle them in
production at LinkedIn.

7:30 - 8:00 PM: Additional networking and Q

Hope to see you there!

*[Streams Infra team @ LinkedIn]*

Re: [DISCUSS] KIP-553: Enable TLSv1.3 by default and disable all protocols except [TLSV1.2, TLSV1.3]

[Nikolay Izhikov]

Hello, Rajini.

I’m tried to run all system tests but failed for now.
It happens, that system tests generates a lot of logs.
I had a 250GB of the free space but it all was occupied by the log from half of 
the system tests.

Do you have any idea what is summary disc space I need to run all system tests? 
 

> 7 янв. 2020 г., в 14:49, Rajini Sivaram  написал(а):
> 
> Hi Nikolay,
> 
> There a couple of things you could do:
> 
> 1) Run all system tests that use SSL with TLSv1.3. I had run a subset, but
> it will be good to run all of them. You can do this locally using docker
> with JDK 11 by updating the files in tests/docker. You will need to update
> tests/kafkatest/services/security/security_config.py to enable only
> TLSv1.3. Instructions for running system tests using docker are in
> https://github.com/apache/kafka/blob/trunk/tests/README.md.
> 2) For integration tests, we run a small number of tests using TLSv1.3 if
> the tests are run using JDK 11 and above. We need to do this for system
> tests as well. There is an open JIRA:
> https://issues.apache.org/jira/browse/KAFKA-9319. Feel free to assign this
> to yourself if you have time to do this.
> 
> Regards,
> 
> Rajini
> 
> 
> On Tue, Jan 7, 2020 at 5:15 AM Николай Ижиков  wrote:
> 
>> Hello, Rajini.
>> 
>> Can you, please, clarify, what should be done?
>> I can try to do tests by myself.
>> 
>>> 6 янв. 2020 г., в 21:29, Rajini Sivaram 
>> написал(а):
>>> 
>>> Hi Brajesh.
>>> 
>>> No one is working on this yet, but will follow up with the Confluent
>> tools
>>> team to see when this can be done.
>>> 
>>> On Mon, Jan 6, 2020 at 3:29 PM Brajesh Kumar 
>> wrote:
>>> 
 Hello Rajini,
 
 What is the plan to run system tests using JDK 11? Is someone working on
 this?
 
 On Mon, Jan 6, 2020 at 3:00 PM Rajini Sivaram 
 wrote:
 
> Hi Nikolay,
> 
> We can leave the KIP open and restart the discussion once system tests
 are
> running.
> 
> Thanks,
> 
> Rajini
> 
> On Mon, Jan 6, 2020 at 2:46 PM Николай Ижиков 
 wrote:
> 
>> Hello, Rajini.
>> 
>> Thanks, for the feedback.
>> 
>> Should I mark this KIP as declined?
>> Or just wait for the system tests results?
>> 
>>> 6 янв. 2020 г., в 17:26, Rajini Sivaram 
>> написал(а):
>>> 
>>> Hi Nikolay,
>>> 
>>> Thanks for the KIP. We currently run system tests using JDK 8 and
 hence
>> we
>>> don't yet have full system test results with TLS 1.3 which requires
 JDK
>> 11.
>>> We should wait until that is done before enabling TLS1.3 by default.
>>> 
>>> Regards,
>>> 
>>> Rajini
>>> 
>>> 
>>> On Mon, Dec 30, 2019 at 5:36 AM Николай Ижиков 
>> wrote:
>>> 
 Hello, Team.
 
 Any feedback on this KIP?
 Do we need this in Kafka?
 
> 24 дек. 2019 г., в 18:28, Nikolay Izhikov 
 написал(а):
> 
> Hello,
> 
> I'd like to start a discussion of KIP.
> Its goal is to enable TLSv1.3 and disable obsolete versions by
> default.
> 
> 
 
>> 
> 
 
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956
> 
> Your comments and suggestions are welcome.
> 
 
 
>> 
>> 
> 
 
 
 --
 Regards,
 Brajesh Kumar
 
>> 
>> 

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

[Mickael Maison]

Hi,

We are now at 4 non-binding votes but still no binding votes.
I have not seen any outstanding questions in the DISCUSS thread. If
you have any feedback, please let me know.

Thanks


On Thu, Jan 16, 2020 at 2:03 PM M. Manna  wrote:
>
> MIckael,
>
>
>
> On Thu, 16 Jan 2020 at 14:01, Mickael Maison 
> wrote:
>
> > Hi Manna,
> >
> > In your example, the topic 'dummy' is not under replicated. It just
> > has 1 replica. A topic under replicated is a topic with less ISRs than
> > replicas.
> >
> > Having under replicated topics is relatively common in a Kafka
> > cluster, it happens everytime is broker is down. However Kafka does
> > not permit it to happen at topic creation. Currently at creation,
> > Kafka requires to have at least as many brokers as the replication
> > factor. This KIP addresses this limitation.
> >
> > Regarding your 2nd point. When rack awareness is enabled, Kafka tries
> > to distribute partitions across racks. When all brokers in a rack are
> > down (ie: a zone is offline), you can end up with partitions not well
> > distributed even with rack awareness. There are currently no easy way
> > to track such partitions so I decided to not attempt addressing this
> > issue in this KIP.
> >
> > I hope that answers your questions.
> >
>
>  It does and I appreciate you taking time and explaining this.
>
>  +1 (binding) if I haven't already.
>
> >
> >
> >
> > On Wed, Jan 15, 2020 at 4:10 PM Kamal Chandraprakash
> >  wrote:
> > >
> > > +1 (non-binding). Thanks for the KIP!
> > >
> > > On Mon, Jan 13, 2020 at 1:58 PM M. Manna  wrote:
> > >
> > > > Hi Mikael,
> > > >
> > > > Apologies for last minute question, as I just caught up with it.
> > Thanks for
> > > > your work on the KIP.
> > > >
> > > > Just trying to get your thoughts on one thing (I might have
> > misunderstood
> > > > it) - currently it's possible (even though I am strongly against it) to
> > > > create Kafka topics which are under-replicated; despite all brokers
> > being
> > > > online. This the the output of an intentionally under-replicated topic
> > > > "dummy" with p=6 and RF=1 (with a 3 node cluster)
> > > >
> > > >
> > > > virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$ ./kafka-topics.sh
> > > > --create --topic dummy --partitions 6 --replication-factor 1
> > > > --bootstrap-server localhost:9092
> > > > virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$ ./kafka-topics.sh
> > > > --describe --topic dummy  --bootstrap-server localhost:9092
> > > > Topic:dummy PartitionCount:6ReplicationFactor:1
> > > >
> > > >
> > Configs:compression.type=gzip,min.insync.replicas=2,cleanup.policy=delete,segment.bytes=10485760,max.message.bytes=10642642,retention.bytes=20971520
> > > > Topic: dummyPartition: 0Leader: 3   Replicas: 3
> > > > Isr: 3
> > > > Topic: dummyPartition: 1Leader: 1   Replicas: 1
> > > > Isr: 1
> > > > Topic: dummyPartition: 2Leader: 2   Replicas: 2
> > > > Isr: 2
> > > > Topic: dummyPartition: 3Leader: 3   Replicas: 3
> > > > Isr: 3
> > > > Topic: dummyPartition: 4Leader: 1   Replicas: 1
> > > > Isr: 1
> > > > Topic: dummyPartition: 5Leader: 2   Replicas: 2
> > > > Isr: 2
> > > >
> > > >  This is with respect to the following statement on your KIP (i.e.
> > > > under-replicated topic creation is also permitted when none is
> > offline):
> > > >
> > > > *but note that this may already happen (without this KIP) when
> > > > > topics/partitions are created while all brokers in a rack are offline
> > > > (ie:
> > > > > an availability zone is offline). Tracking topics/partitions not
> > > > optimally
> > > > > spread across all racks can be tackled in a follow up KIP.  *
> > > >
> > > >
> > > >
> > > >
> > > > Did you mean to say that such under-replicated topics (including
> > > > human-created ones) will be handled in a separete KIP ?
> > > >
> > > > Regards,
> > > >
> > > >
> > > > On Mon, 13 Jan 2020 at 10:15, Mickael Maison  > >
> > > > wrote:
> > > >
> > > > > Hi all.
> > > > >
> > > > > With 2.5.0 approaching, bumping this thread once more as feedback or
> > > > > votes would be nice.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Wed, Dec 18, 2019 at 1:59 PM Tom Bentley 
> > wrote:
> > > > > >
> > > > > > +1 non-binding. Thanks!
> > > > > >
> > > > > > On Wed, Dec 18, 2019 at 1:05 PM Sönke Liebau
> > > > > >  wrote:
> > > > > >
> > > > > > > Hi Mickael,
> > > > > > >
> > > > > > > thanks for your response! That all makes perfect sense and I
> > cannot
> > > > > > > give any actual use cases for where what I asked about would be
> > > > useful
> > > > > > > :)
> > > > > > > It was more the idle thought if this might be low hanging fruit
> > while
> > > > > > > changing this anyway to avoid having to circle back later on and
> > > > > > > wanted to at least mention it.
> > > > > > >
> > > > > > > I am totally happy either way!
> > > > > > >
> > > > > > > Best regards,
> > > > > 

Re: [DISCUSS] KIP-518: Allow listing consumer groups per state

[Mickael Maison]

Hi David,

Did that answer your questions? or do you have any further feedback?

Thanks

On Thu, Jan 16, 2020 at 4:11 PM Mickael Maison  wrote:
>
> Hi David,
>
> Thanks for taking a look.
> 1) Yes, updated
>
> 2) I had not considered that but indeed this would be useful if the
> request contained multiple states and would avoid doing another call.
> The ListGroups response already includes the group ProtocolType, so I
> guess we could add the State as well. The response will still be
> significantly smaller than DescribeGroups. With this change, one thing
> to note is that having Describe on the Cluster resource will allow
> retrieving the state of all groups. Currently retrieving the state of
> a group requires Describe on the Group.
>
> 3) Yes if ListGroups response includes the state, it makes sense to
> expose it via the command line tool and the AdminClient. With
> ConsumerGroupCommand, to avoid compatibility issues we can only print
> states when the --states flag is specified.
>
> I've updated the KIP accordingly.
>
> On Mon, Jan 13, 2020 at 12:20 PM David Jacot  wrote:
> >
> > Hi Michael,
> >
> > Please, excuse me for my late feedback. I've got a few questions/comments
> > while reviewing the KIP.
> >
> > 1. I would suggest to clearly state in the documentation of the state field
> > that omitting it or providing an empty list means "all".
> >
> > 2. Have you considered including the state in the response? The API allows
> > to search for multiple states so it could be
> > convenient to have the state in the response to let the user differentiate
> > the groups.
> >
> > 3. If 2. makes sense, I would suggest to also include it in the information
> > printed out by the ConsumerGroupCommand tool. Putting
> > myself in the shoes of an operator, I would like to see the state of each
> > group if I select specific states. Perhaps we could
> > use a table instead of the simple list used today. What do you think?
> >
> > Thanks for the KIP!
> >
> > Best,
> > David
> >
> > On Mon, Nov 11, 2019 at 12:40 PM Mickael Maison 
> > wrote:
> >
> > > Hi all,
> > >
> > > If there's no more feedback, I'll open a vote in the next few days.
> > >
> > > Thanks
> > >
> > >
> > > On Fri, Nov 1, 2019 at 4:27 PM Mickael Maison 
> > > wrote:
> > > >
> > > > Hi Tom,
> > > >
> > > > Thanks for taking a look at the KIP!
> > > > You are right, even if we serialize the field as a String, we should
> > > > use ConsumerGroupState in the API.
> > > > As suggested, I've also updated the API so a list of states is 
> > > > specified.
> > > >
> > > > Regards,
> > > >
> > > >
> > > > On Tue, Oct 22, 2019 at 10:03 AM Tom Bentley 
> > > wrote:
> > > > >
> > > > > Hi Mickael,
> > > > >
> > > > > Thanks for the KIP.
> > > > >
> > > > > The use of String to represent the desired state in the API seems less
> > > > > typesafe than would be ideal. Is there a reason not to use the 
> > > > > existing
> > > > > ConsumerGroupState enum (even if the state is serialized as a String)?
> > > > >
> > > > > While you say that the list-of-names result from listConsumerGroups is
> > > a
> > > > > reason not to support supplying a set of desired states I don't find
> > > that
> > > > > argument entirely convincing. Sure, if the results are going to be
> > > shown to
> > > > > a user then it would be ambiguous and multiple queries would be
> > > needed. But
> > > > > it seems quite possible that the returned list of groups will
> > > immediately
> > > > > be used in a describeConsumerGroups query (for example, so show a user
> > > > > additional information about the groups of interest, for example). In
> > > that
> > > > > case the grouping by state could be done on the descriptions, and some
> > > RPCs
> > > > > could be avoided. It would also avoid the race inherent in making
> > > multiple
> > > > > listConsumerGroups requests. So supporting a set of states isn't
> > > entirely
> > > > > worthless and it wouldn't really add very much complexity.
> > > > >
> > > > > Kind regards,
> > > > >
> > > > > Tom
> > > > >
> > > > > On Mon, Oct 21, 2019 at 5:54 PM Mickael Maison <
> > > mickael.mai...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Bump
> > > > > > Now that the rush for 2.4.0 is ending I hope to get some feedback
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Mon, Sep 9, 2019 at 5:44 PM Mickael Maison <
> > > mickael.mai...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I have created a KIP to allow listing groups per state:
> > > > > > >
> > > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-518%3A+Allow+listing+consumer+groups+per+state
> > > > > > >
> > > > > > > Have a look and let me know what you think.
> > > > > > > Thank you
> > > > > >
> > >

Re: KIP-560 Discuss

[Sang wn Lee]

thank you John Roesle

It is a good idea
"—all-input-topics"

I agree with you

I'll update right away


On 2020/01/24 14:14:17, "John Roesler"  wrote: 
> Hi all, thanks for the explanation. I was also not sure how the kip would be 
> possible to implement. 
> 
> No that it does seem plausible, my only feedback is that the command line 
> option could align better with the existing one. That is, the existing option 
> is called “—input-topics”, so it seems like the new one should be called 
> “—all-input-topics”. 
> 
> Thanks,
> John
> 
> On Fri, Jan 24, 2020, at 01:42, Boyang Chen wrote:
> > Thanks Sophie for the explanation! I read Sang's PR and basically he did
> > exactly what you proposed (check it here
> >  in case I'm wrong).
> > 
> > I think Sophie's response answers Gwen's question already, while in the
> > meantime for a KIP itself we are not required to mention all the internal
> > details about how to make the changes happen (like how to actually get the
> > external topics), considering the change scope is pretty small as well. But
> > again, it would do no harm if we mention it inside Proposed Change session
> > specifically so that people won't get confused about how.
> > 
> > 
> > On Thu, Jan 23, 2020 at 8:26 PM Sophie Blee-Goldman 
> > wrote:
> > 
> > > Hi all,
> > >
> > > I think what Gwen is trying to ask (correct me if I'm wrong) is how we can
> > > infer which topics are associated with
> > > Streams from the admin client's topic list. I agree that this doesn't seem
> > > possible, since as she pointed out the
> > > topics list (or even description) lacks the specific information we need.
> > >
> > > What we could do instead is use the admin client's
> > > `describeConsumerGroups` API to get the information
> > > on the Streams app's consumer group specifically -- note that the Streams
> > > application.id config is also used
> > > as the consumer group id, so each app forms a group to read from the input
> > > topics. We could compile a list
> > > of these topics just by looking at each member's assignment (and even 
> > > check
> > > for a StreamsPartitionAssignor
> > > to verify that this is indeed a Streams app group, if we're being
> > > paranoid).
> > >
> > > The reset tool actually already gets the consumer group description, in
> > > order to validate there are no active
> > > consumers in the group. We may as well grab the list of topics from it
> > > while it's there. Or did you have something
> > > else in mind?
> > >
> > > On Sat, Jan 18, 2020 at 6:17 PM Sang wn Lee  wrote:
> > >
> > > > Thank you
> > > >
> > > > I understand you
> > > >
> > > > 1. admin client has topic list
> > > > 2. applicationId can only have one stream, so It won't be a problem!
> > > > 3. For example, --input-topic [reg]
> > > > Allowing reg solves some inconvenience
> > > >
> > > >
> > > > On 2020/01/18 18:15:23, Gwen Shapira  wrote:
> > > > > I am not sure I follow. Afaik:
> > > > >
> > > > > 1. Topics don't include client ID information
> > > > > 2. Even if you did, the same ID could be used for topics that are not
> > > > Kafka
> > > > > Streams input
> > > > >
> > > > > The regex idea sounds doable, but I'm not sure it solves much?
> > > > >
> > > > >
> > > > > On Sat, Jan 18, 2020, 7:12 AM Sang wn Lee 
> > > wrote:
> > > > >
> > > > > > Thank you
> > > > > > Gwen Shapira!
> > > > > > We'll add a flag to clear all topics by clientId
> > > > > > It is ‘reset-all-external-topics’
> > > > > >
> > > > > > I also want to use regex on the input topic flag to clear all
> > > matching
> > > > > > topics.
> > > > > >
> > > > > > On 2020/01/17 19:29:09, Gwen Shapira  wrote:
> > > > > > > Seem like a very nice improvement to me. But I have to admit that 
> > > > > > > I
> > > > > > > don't understand how this will how - how could you infer the input
> > > > > > > topics?
> > > > > > >
> > > > > > > On Thu, Jan 16, 2020 at 10:03 AM Sang wn Lee  > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > Starting this thread to discuss KIP-560:
> > > > > > > > wiki link :
> > > > > > > >
> > > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-560%3A+Auto+infer+external+topic+partitions+in+stream+reset+tool
> > > > > > > >
> > > > > > > > I'm newbie
> > > > > > > > I would like to receive feedback on the following features!
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> 

Re: [VOTE] KIP-553: Disable all SSL protocols except TLSV1.2 by default.

[Nikolay Izhikov]

Thanks everyone!

After 3+ business days since this thread started, I'm concluding the vote
on KIP-553.

The KIP has passed with:

4 binding votes from Mickael Maison, Manikumar, Rajini Sivaram, M. Manna.
2 non-binding vote from Ted Yu, Ron Dagostino.

Thank you all for voting!

> 22 янв. 2020 г., в 14:43, M. Manna  написал(а):
> 
> +1 (binding). A simple, and yet powerful enforcement of TLS version.
> 
> Thanks for this KIP :)
> 
> On Tue, 21 Jan 2020 at 20:39, Mickael Maison 
> wrote:
> 
>> +1 (binding)
>> Thanks
>> 
>> On Tue, Jan 21, 2020 at 7:58 PM Ron Dagostino  wrote:
>>> 
>>> +1 (non-binding)
>>> 
>>> Ron
>>> 
>>> On Tue, Jan 21, 2020 at 11:29 AM Manikumar 
>> wrote:
 
 +1 (binding).
 
 Thanks for the KIP.
 
 
 On Tue, Jan 21, 2020 at 9:56 PM Ted Yu  wrote:
 
> +1
> 
> On Tue, Jan 21, 2020 at 8:24 AM Rajini Sivaram <
>> rajinisiva...@gmail.com>
> wrote:
> 
>> +1 (binding)
>> 
>> Thanks for the KIP!
>> 
>> Regards,
>> 
>> Rajini
>> 
>> 
>> On Tue, Jan 21, 2020 at 3:43 PM Николай Ижиков <
>> nizhi...@apache.org>
>> wrote:
>> 
>>> Hello.
>>> 
>>> I would like to start vote for KIP-553: Disable all SSL protocols
> except
>>> TLSV1.2 by default.
>>> 
>>> KIP -
>>> 
>> 
> 
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956
>>> Discussion thread -
>>> 
>> 
> 
>> https://lists.apache.org/thread.html/9c6201fe403a24f84fc3aa27f47dd06b718c1d80de0ee3412b9b877c%40%3Cdev.kafka.apache.org%3E
>> 
> 
>>