Re: [ANNOUNCE] Apache Kafka 2.0.0 Released

Thanks for driving the release! On Mon, Jul 30, 2018 at 3:55 PM Rajini Sivaram wrote: > The Apache Kafka community is pleased to announce the release for > > Apache Kafka 2.0.0. > > > > > > This is a major release and includes significant new features from > > 40 KIPs. It contains fixes and

Re: [ANNOUNCE] Apache Kafka 2.0.0 Released

This is great news! Thanks for driving this Rajini!! On Mon, Jul 30, 2018 at 3:25 AM, Rajini Sivaram wrote: > The Apache Kafka community is pleased to announce the release for > > Apache Kafka 2.0.0. > > > > > > This is a major release and includes significant new features from > > 40 KIPs. It

[jira] [Created] (KAFKA-7216) Exception while running kafka-acls.sh from 1.0 env on target Kafka env with 1.1.1

Satish Duggana created KAFKA-7216: - Summary: Exception while running kafka-acls.sh from 1.0 env on target Kafka env with 1.1.1 Key: KAFKA-7216 URL: https://issues.apache.org/jira/browse/KAFKA-7216

Re: [ANNOUNCE] Apache Kafka 2.0.0 Released

Such a good news on a Monday morning ... Thank you Rajini for driving the release! --Vahid From: Mickael Maison To: Users Cc: dev , annou...@apache.org, kafka-clients Date: 07/30/2018 04:37 AM Subject:Re: [ANNOUNCE] Apache Kafka 2.0.0 Released Great news! Thanks

Re: [ANNOUNCE] Apache Kafka 2.0.0 Released

Thanks to everyone who contributed to the release (including testing and bug reports)! And thank you Rajini for managing the release. Ismael On Mon, 30 Jul 2018, 03:25 Rajini Sivaram, wrote: > The Apache Kafka community is pleased to announce the release for > > Apache Kafka 2.0.0. > > > > > >

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

Hey Dong, Thanks for the detailed review. Responses below: 1/2: Thanks for noticing the inconsistency. Would it be reasonable to simply call it LeaderEpoch for both APIs? 3: I agree it should be a map. I will update. 4: Fair point. I think we should always be able to identify an offset. Let's

Re: [ANNOUNCE] Apache Kafka 2.0.0 Released

Thank you for your great works! Thanks again for the commiters and all the contributors! On Tue, Jul 31, 2018 at 1:05 AM Ismael Juma wrote: > Thanks to everyone who contributed to the release (including testing and > bug reports)! And thank you Rajini for managing the release. > > Ismael > > On

Re: [ANNOUNCE] Apache Kafka 2.0.0 Released

Excellent! Thanks for running the release Rajini! On Mon, 30 Jul 2018 at 11:25 Rajini Sivaram wrote: > The Apache Kafka community is pleased to announce the release for > > Apache Kafka 2.0.0. > > > > > > This is a major release and includes significant new features from > > 40 KIPs. It

Dynamic matching of topics using regex for connector sink

Hi Dev, With the release of the Kafka 1.1.0 (KIP-215) the support for the regex was added for the sink connector, however it is appears that the topics cannot be added dynamically matching the regex without restarting the connector. Can we add this in issues list? I am fairly new to Kafka.

Re: KIP-213 - Scalable/Usable Foreign-Key KTable joins - Rebooted.

Hi Guozhang et al I was just reading the 2.0 release notes and noticed a section on Record Headers. https://cwiki.apache.org/confluence/display/KAFKA/KIP-244%3A+Add+Record+Header+support+to+Kafka+Streams+Processor+API I am not yet sure if the contents of a RecordHeader is propagated all the way

[ANNOUNCE] Apache Kafka 2.0.0 Released

The Apache Kafka community is pleased to announce the release for Apache Kafka 2.0.0. This is a major release and includes significant new features from 40 KIPs. It contains fixes and improvements from 246 JIRAs, including a few critical bugs. Here is a summary of some notable changes: **

Re: [ANNOUNCE] Apache Kafka 2.0.0 Released

Great news! Thanks for running the release On Mon, Jul 30, 2018 at 12:20 PM, Manikumar wrote: > Thanks for driving the release! > > > > On Mon, Jul 30, 2018 at 3:55 PM Rajini Sivaram wrote: > >> The Apache Kafka community is pleased to announce the release for >> >> Apache Kafka 2.0.0. >> >> >>

Re: [DISCUSS] KIP-332: Update AclCommand to use AdminClient API

Bumping this up! On Mon, Jul 23, 2018 at 8:00 PM Manikumar wrote: > Hi all, > > I have created a KIP to use AdminClient API in AclCommand (kafka-acls.sh) > > > *https://cwiki.apache.org/confluence/display/KAFKA/KIP-332%3A+Update+AclCommand+to+use+AdminClient+API* > >

Re: [VOTE] KIP-328: Ability to suppress updates for KTables

+1 On Mon, Jul 30, 2018 at 11:46 AM John Roesler wrote: > Hello devs, > > The discussion of KIP-328 has gone some time with no new comments, so I am > calling for a vote! > > Here's the KIP: https://cwiki.apache.org/confluence/x/sQU0BQ > > The basic idea is to provide: > * more usable control

Re: [ANNOUNCE] Apache Kafka 2.0.0 Released

Congrats! Is there any reason I don't see 2.0.0 listed on the docs page? https://kafka.apache.org/documentation/ On Mon, Jul 30, 2018 at 12:54 PM, Stephane Maarek < steph...@simplemachines.com.au> wrote: > Congratulations ! > > On Mon., 30 Jul. 2018, 11:51 am Sean Glover, > wrote: > > >

[jira] [Created] (KAFKA-7217) Loading dynamic topic data into kafka connector sink using regex

Pratik Gaglani created KAFKA-7217: - Summary: Loading dynamic topic data into kafka connector sink using regex Key: KAFKA-7217 URL: https://issues.apache.org/jira/browse/KAFKA-7217 Project: Kafka

[DISCUSS] Applying scalafmt to core code

I had started on KAFKA-2423 (was Scalastyle, now Expand scalafmt to core).  As part of the cleanup, applying the "gradlew spotlessApply" command ended up affecting too many (435 out of 439) files.  Since this will affect every file, this sort of change does risk polluting the git logs. So,

Re: [ANNOUNCE] Apache Kafka 2.0.0 Released

Congratulations ! On Mon., 30 Jul. 2018, 11:51 am Sean Glover, wrote: > Congrats to everyone involved. Releasing "2.0.0" is a big achievement :) > > On Mon, Jul 30, 2018 at 12:35 PM Dongjin Lee wrote: > > > Thank you for your great works! Thanks again for the commiters and all > the > >

Re: [VOTE] KIP-328: Ability to suppress updates for KTables

Hi John, Thanks for the updated KIP, +1 from me, and one minor suggestion: Following your suggestion of the differentiation of `skipped-records` v.s. `late-event-drop`, we should probably consider moving the scenarios where records got ignored due the window not being available any more in

[VOTE] KIP-328: Ability to suppress updates for KTables

Hello devs, The discussion of KIP-328 has gone some time with no new comments, so I am calling for a vote! Here's the KIP: https://cwiki.apache.org/confluence/x/sQU0BQ The basic idea is to provide: * more usable control over update rate (vs the current state store caches) * the

Re: [ANNOUNCE] Apache Kafka 2.0.0 Released

Congrats to everyone involved. Releasing "2.0.0" is a big achievement :) On Mon, Jul 30, 2018 at 12:35 PM Dongjin Lee wrote: > Thank you for your great works! Thanks again for the commiters and all the > contributors! > > On Tue, Jul 31, 2018 at 1:05 AM Ismael Juma wrote: > > > Thanks to

Re: [DISCUSS] KIP-334 Include partitions in exceptions raised during consumer record deserialization/validation

Hey Stanislav, Thanks for the KIP. I think the goal is to allow users to seek past a records which cannot be parsed for whatever reason. However, it's a little annoying that you need to catch two separate types to handle this. I'm wondering if it makes sense to expose an interface like

Re: [DISCUSS] KIP-332: Update AclCommand to use AdminClient API

Look good to me. On Mon, Jul 23, 2018 at 7:30 AM Manikumar wrote: > Hi all, > > I have created a KIP to use AdminClient API in AclCommand (kafka-acls.sh) > > * > https://cwiki.apache.org/confluence/display/KAFKA/KIP-332%3A+Update+AclCommand+to+use+AdminClient+API* > < >

Re: [DISCUSS] KIP-332: Update AclCommand to use AdminClient API

Hi Manikumar, It's great that you are taking a look at this! Much needed. Just one note: I assume that --authorizer-properties is no longer required if the --bootstrap-server option is specified. We should probably spell this out somewhere in the KIP. thanks, Colin On Mon, Jul 30, 2018,

Re: [VOTE] KIP-328: Ability to suppress updates for KTables

Thanks for the KIP! +1 -Bill On Mon, Jul 30, 2018 at 3:42 PM Ted Yu wrote: > +1 > > On Mon, Jul 30, 2018 at 11:46 AM John Roesler wrote: > > > Hello devs, > > > > The discussion of KIP-328 has gone some time with no new comments, so I > am > > calling for a vote! > > > > Here's the KIP:

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

Thanks for your review, Dong. Ack that these configs will have a bigger impact for users. On the other hand, I would argue that the request queue becoming full may or may not be a rare scenario. How often the request queue gets full depends on the request incoming rate, the request processing

Re: [Vote] KIP-321: Update TopologyDescription to better represent Source and Sink Nodes

We need one more binding vote. Binding Votes: - Matthias J. Sax - Guozhang Wong Community Votes: - Bill Bejeck - Ted Yu Best, Nishanth Pradeep On Fri, Jul 27, 2018 at 10:02 AM Bill Bejeck wrote: > Thanks for the KIP! > > +1 > > -Bill > > On Thu, Jul 26, 2018 at 2:39 AM Guozhang

Re: [VOTE] KIP-328: Ability to suppress updates for KTables

Thanks Guozhang, Thanks for that catch. to clarify, currently, events are "late" only when they are older than the retention period. Currently, we detect this in the processor and record it as a "skipped-record". We then do not attempt to store the event in the window store. If a user provided a

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Guozhang, Thanks for giving us a great starting point. A few questions that come to mind right away: 1) What do you think a reasonable group-change-timeout would be? I am thinking on the order of minutes (5 minutes?) 2) Will the nodes that are still alive continue to make progress during a

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Hey Guozhang, Thanks for the detailed response. Really quick about the fencing issue, I think host/port will not be sufficient because it cannot handle disconnects. For example, if the coordinator moves to another broker, then there is no way we'd be able to guarantee the same host/port.

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Jason, I really appreciate the broader conversation that you are bringing up here. I've been working on an application that does streaming joins for a while now, and we face a similar issue with group membership being dynamic. We are currently using our own StickyAssignor and take special care

Re: [VOTE] KIP-328: Ability to suppress updates for KTables

Another thing that came up after I started working on an implementation is that in addition to deprecating "retention" from the Windows interface, we also need to deprecate "segmentInterval", for the same reasons. I simply overlooked it previously. I've updated the KIP accordingly. Hopefully,

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Hello Boyang / Jason / Mike, Thanks for your thoughtful inputs! Regarding the fencing issue, I've thought about leveraging the epoch notion from PID of transactional messaging before, but since in this proposal we do not always require member ids from clients, and hence could have a mixed of

Re: [VOTE] KIP-328: Ability to suppress updates for KTables

Yes, the addendum lgtm as well. Thanks! On Mon, Jul 30, 2018 at 3:34 PM, John Roesler wrote: > Another thing that came up after I started working on an implementation is > that in addition to deprecating "retention" from the Windows interface, we > also need to deprecate "segmentInterval", for

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Hi Boyang, Thanks for the response. I think the main point I was trying to make is the need for fencing. I am not too concerned about how to generate a unique id on the client side. The approach you suggested for streams seems reasonable. However, any time you reuse an id, you need to be careful

Permission to create KIP

Hi, I am trying to create a KIP, but I don't access to do so. Can anyone provide me with the access ? Thanks, Ratish

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

@Mike: 1) This should be configurable for each consumer group id, and the actual value will depend on how fast users expect their instances to start / shutdown concurrently, and fail-over. I think a default value of 30 seconds should be good, note that it means on starting / shutting down the

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

@Jason, Good point about disconnects. And with that I think I agree that a registry id maybe a better idea to enable fencing than validating on host / port. Guozhang On Mon, Jul 30, 2018 at 5:40 PM, Jason Gustafson wrote: > Hey Guozhang, > > Thanks for the detailed response. Really quick

Re: [DISCUSS] KIP-332: Update AclCommand to use AdminClient API

Hi Colin, Yes, "--authorizer-properties" option is not required with "--bootstrap-server" option. Updated the KIP. Thanks, On Tue, Jul 31, 2018 at 1:30 AM Ted Yu wrote: > Look good to me. > > On Mon, Jul 23, 2018 at 7:30 AM Manikumar > wrote: > > > Hi all, > > > > I have created a KIP to

Re: [ANNOUNCE] Apache Kafka 2.0.0 Released

Congrats and great job, everyone! Thanks Rajini for driving the release! -James Sent from my iPhone > On Jul 30, 2018, at 3:25 AM, Rajini Sivaram wrote: > > The Apache Kafka community is pleased to announce the release for > > Apache Kafka 2.0.0. > > > > > > This is a major release and

Build failed in Jenkins: kafka-trunk-jdk8 #2847

See Changes: [lindong28] KAFKA-7180; Fixing the flaky test -- [...truncated 889.64 KB...] kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask PASSED

[jira] [Created] (KAFKA-7215) Improve LogCleaner behavior on error

Stanislav Kozlovski created KAFKA-7215: -- Summary: Improve LogCleaner behavior on error Key: KAFKA-7215 URL: https://issues.apache.org/jira/browse/KAFKA-7215 Project: Kafka Issue Type: