Re: [VOTE] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils

+1 Original message From: Bill Bejeck Date: 3/9/18 12:29 PM (GMT-08:00) To: dev@kafka.apache.org Subject: Re: [VOTE] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils Thanks for the KIP. +1 -Bill On Fri, Mar 9, 2018 at 3:16 PM, John

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

Thanks for your comment Clemens. It make sense what you are saying. However, your described pattern is to split partitions and use linear hashing to avoid random key distribution. But this is what Jan thinks we should not do... Also, I just picked an example with 2 -> 3 partitions, but if you

Re: [VOTE] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils

Thanks for the KIP. +1 -Bill On Fri, Mar 9, 2018 at 3:16 PM, John Roesler wrote: > Hello all, > > I'd like to start voting on KIP-267, to introduce a MockProcessorContent > enabling Processor, Transformer, and ValueTransformer authors to unit-test > their components. > >

Build failed in Jenkins: kafka-trunk-jdk9 #459

See Changes: [wangguoz] KAFKA-4831: Extract WindowedSerde to public APIs (#3307) -- [...truncated 1.48 MB...] kafka.coordinator.transaction.TransactionCoordinatorTest >

Re: Want to contribute

Matthias, Thank you! Thanks, Eduardo Mello On Fri, Mar 9, 2018 at 6:14 PM, Matthias J. Sax wrote: > Eduardo, > > I added you to the contributor list. > > -Matthias > > On 3/9/18 7:07 AM, Eduardo Mello wrote: > > Thank you! > > > > Ill look for this label! > > > > I

Build failed in Jenkins: kafka-trunk-jdk8 #2461

See Changes: [github] MINOR: Fix record conversion time in metrics (#4671) [wangguoz] KAFKA-4831: Extract WindowedSerde to public APIs (#3307) -- [...truncated 419.40

Jenkins build is back to normal : kafka-trunk-jdk7 #3238

See

Re: [DISCUSS] KIP-258: Allow to Store Record Timestamps in RocksDB

Matthias, Thanks for the KIP, it's a +1 from me. I do have one question regarding the retrieval methods on the new interfaces. Would want to consider adding one method with a Predicate that would allow for filtering records by the timestamp stored with the record? Or is this better left for

Re: [DISCUSS] KIP-258: Allow to Store Record Timestamps in RocksDB

@Bill: I think a filter predicate should be part of user code. And even if we want to add something like this, I would prefer to do it in a separate KIP. @James: I would love to avoid a second rolling bounce. But from my understanding it would not be possible. The purpose of the second rolling

Re: [VOTE] KIP-186: Increase offsets retention default to 7 days

Thanks for voting everyone, KIP is accepted with 5 binding +1 and 5 non-binding +1s. PR is already merged and there are upgrade notes about the potential memory impact. -Ewen On Wed, Mar 7, 2018 at 9:25 PM, Ted Yu wrote: > +1 > Original message From:

[jira] [Resolved] (KAFKA-6622) GroupMetadataManager.loadGroupsAndOffsets decompresses record batch needlessly

[ https://issues.apache.org/jira/browse/KAFKA-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-6622. Resolution: Fixed Fix Version/s: 1.1.0 > GroupMetadataManager.loadGroupsAndOffsets

Re: [DISCUSS] KIP-258: Allow to Store Record Timestamps in RocksDB

Matthias, For all the upgrade paths, is it possible to get rid of the 2nd rolling bounce? For the in-place upgrade, it seems like primary difference between the 1st rolling bounce and the 2nd rolling bounce is to decide whether to send Subscription Version 2 or Subscription Version 3.

Build failed in Jenkins: kafka-1.1-jdk7 #71

See Changes: [rajinisivaram] MINOR: Fix record conversion time in metrics (#4671) -- [...truncated 416.07 KB...] kafka.tools.ConsoleConsumerTest >

[VOTE] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils

Hello all, I'd like to start voting on KIP-267, to introduce a MockProcessorContent enabling Processor, Transformer, and ValueTransformer authors to unit-test their components.

Re: [VOTE] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils

Guozhang, renaming the class is an internal change and I think it's not required to cover it in the KIP. +1 (binding) -Matthias On 3/9/18 1:26 PM, Guozhang Wang wrote: > Hi John, > > You mentioned you are going to do the renaming for the existing class > to InternalMockProcessorContext > but

Re: [VOTE] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils

Hi John, You mentioned you are going to do the renaming for the existing class to InternalMockProcessorContext but that seems not reflected on the KIP wiki, could you add that in the proposed changes section? Other than that, I'm +1 on the KIP. Thanks! Guozhang On Fri, Mar 9, 2018 at 12:39

Re: Want to contribute

Eduardo, I added you to the contributor list. -Matthias On 3/9/18 7:07 AM, Eduardo Mello wrote: > Thank you! > > Ill look for this label! > > I created a Jira user too: emello > > Thanks, > Eduardo Mello > > On Fri, Mar 9, 2018 at 12:00 PM, Sanket Band > wrote: >

Re: [DISCUSS] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils

Hey John, Re: Mock Processor Context: That's a good point, I'm convinced that we should keep them as two classes. Re: test-utils module: I think I agree with your proposed changes, in fact in order to not scatter the test classes in two places maybe it's better to move all of them to the new

Build failed in Jenkins: kafka-trunk-jdk7 #3237

See Changes: [github] MINOR: Fix record conversion time in metrics (#4671) -- [...truncated 414.91 KB...] kafka.coordinator.transaction.TransactionCoordinatorTest >

Re: [DISCUSS] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils

Sweet! I think this pretty much wraps up all the discussion points. I'll update the KIP with all the relevant aspects we discussed and call for a vote. I'll also comment on the TopologyTestDriver ticket noting this modular test strategy. Thanks, everyone. -John On Fri, Mar 9, 2018 at 10:57 AM,

Build failed in Jenkins: kafka-trunk-jdk8 #2460

See Changes: [ismael] MINOR: Remove unused local variable in SocketServer (#4669) -- [...truncated 3.93 MB...]

Re: [VOTE] KIP-251: Allow timestamp manipulation in Processor API

+1 I am also closing this vote. The KIP is accepted with 3 binding votes (Damian, Guozhang, Matthias) and 2 non-binding votes (Ted, Bill) Thanks for your comments and votes! -Matthias On 3/8/18 10:36 PM, Guozhang Wang wrote: > Thanks Matthias, that sounds good to me. I'm +1 on the KIP

RE: [DISCUSS]KIP-235 DNS alias and secured connections

Hi, There has been further discussion on the ticket and it seems having an additional option to trigger the DNS lookup behaviour would be the best approach. https://issues.apache.org/jira/browse/KAFKA-6195 Updated the KIP

Build failed in Jenkins: kafka-0.10.2-jdk7 #204

See Changes: [ismael] MINOR: Tag AWS instances with Jenkins build url (#4657) -- Started by an SCM change Started by an SCM change [EnvInject] - Loading node environment

Re: [VOTE] 1.1.0 RC1

Great! Have a good weekend. On Fri, Mar 9, 2018 at 4:41 PM, Ismael Juma wrote: > Thanks Jeff: > > https://github.com/apache/kafka/pull/4678 > > Ismael > > On Fri, Mar 9, 2018 at 1:56 AM, Damian Guy wrote: > > > Hi Jeff, > > > > Thanks, we will look into

Build failed in Jenkins: kafka-1.1-jdk7 #72

See Changes: [jason] KAFKA-6622; Fix performance issue loading consumer offsets (#4661) [junrao] MINOR: Fix deadlock in ZooKeeperClient.close() on session expiry (#4672) --

[jira] [Created] (KAFKA-6634) Delay initiating the txn on producers until initializeTopology with EOS turned on

Guozhang Wang created KAFKA-6634: Summary: Delay initiating the txn on producers until initializeTopology with EOS turned on Key: KAFKA-6634 URL: https://issues.apache.org/jira/browse/KAFKA-6634

Re: [DISCUSS] KIP-258: Allow to Store Record Timestamps in RocksDB

Thanks for the details, Matthias. bq. change the metadata protocol only if a future release, encoding both used and supported version might be an advantage Looks like encoding both versions wouldn't be implemented in this KIP. Please consider logging a JIRA with the encoding proposal. Cheers

Re: [DISCUSS] KIP-255: OAuth Authentication via SASL/OAUTHBEARER

Hi Rajini. Thanks for the great feedback. See below for my thoughts/conclusions. I haven't implemented any of it yet or changed the KIP, but I will start to work on the areas where we are in agreement immediately, and I will await your feedback on the areas where an additional iteration is

Re: [DISCUSS] KIP-258: Allow to Store Record Timestamps in RocksDB

Ted, I still consider changing the KIP to include it right away -- if not, I'll create a JIRA. Need to think it through in more detail first. (Same for other open questions like interface names -- I collect feedback and update the KIP after we reach consensus :)) -Matthias On 3/9/18 3:35 PM,

Re: [VOTE] 1.1.0 RC1

Thanks Jeff: https://github.com/apache/kafka/pull/4678 Ismael On Fri, Mar 9, 2018 at 1:56 AM, Damian Guy wrote: > Hi Jeff, > > Thanks, we will look into this. > > Regards, > Damian > > On Thu, 8 Mar 2018 at 18:27 Jeff Chao wrote: > > > Hello, > > >

Build failed in Jenkins: kafka-trunk-jdk9 #460

See Changes: [ismael] MINOR: Add PayloadGenerator to Trogdor (#4640) [jason] KAFKA-6622; Fix performance issue loading consumer offsets (#4661) [ismael] MINOR: Tag AWS instances with Jenkins build url (#4657)

Build failed in Jenkins: kafka-trunk-jdk7 #3239

See Changes: [ismael] MINOR: Add PayloadGenerator to Trogdor (#4640) [jason] KAFKA-6622; Fix performance issue loading consumer offsets (#4661) [ismael] MINOR: Tag AWS instances with Jenkins build url (#4657)

Build failed in Jenkins: kafka-1.0-jdk7 #165

See Changes: [jason] KAFKA-6622; Fix performance issue loading consumer offsets (#4661) -- [...truncated 373.68 KB...] kafka.controller.ZookeeperClientTest >

Re: [DISCUSS] KIP-258: Allow to Store Record Timestamps in RocksDB

Hey James and Matthias, It seems clear (to me) that there's no way to avoid a double bounce for this release. But I do think we should figure out whether there's a feature we can put in right now to allow future releases to be single-bounce. I'm just thinking that this double bounce thing is the

Build failed in Jenkins: kafka-trunk-jdk8 #2462

See Changes: [ismael] MINOR: Add PayloadGenerator to Trogdor (#4640) -- [...truncated 414.80 KB...] kafka.coordinator.transaction.TransactionCoordinatorTest >

Re: [DISCUSS] KIP-258: Allow to Store Record Timestamps in RocksDB

@John: For the protocol version upgrade, it is only for the encoded metadata bytes protocol, which are just bytes-in bytes-out from Consumer's pov, so I think this change should be in the Streams layer as well. @Matthias: for 2), I agree that adding a "newest supported version" besides the

Re: Request to be added to the contributor list so that i can assign JIRAs to myself

Its the same - sband. I did not have an account on apache jira so I created one with the same ID. Thanks Sanket On Fri, 9 Mar 2018, 12:40 Matthias J. Sax, wrote: > I meant your JIRA ID -- sorry for the confusion. > > > -Matthias > > On 3/8/18 10:48 PM, Sanket Band wrote:

[jira] [Created] (KAFKA-6633) Is KafkaProducer still thread safe in version 1.0.1

Herbert Koelman created KAFKA-6633: -- Summary: Is KafkaProducer still thread safe in version 1.0.1 Key: KAFKA-6633 URL: https://issues.apache.org/jira/browse/KAFKA-6633 Project: Kafka Issue

[jira] [Created] (KAFKA-6632) Very slow hashCode methods in Kafka Connect types

Maciej Bryński created KAFKA-6632: - Summary: Very slow hashCode methods in Kafka Connect types Key: KAFKA-6632 URL: https://issues.apache.org/jira/browse/KAFKA-6632 Project: Kafka Issue

[jira] [Resolved] (KAFKA-6633) Is KafkaProducer still thread safe in version 1.0.1

[ https://issues.apache.org/jira/browse/KAFKA-6633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mickael Maison resolved KAFKA-6633. --- Resolution: Not A Problem > Is KafkaProducer still thread safe in version 1.0.1 >

Re: [VOTE] 1.1.0 RC1

Hi Jeff, Thanks, we will look into this. Regards, Damian On Thu, 8 Mar 2018 at 18:27 Jeff Chao wrote: > Hello, > > We at Heroku have run 1.1.0 RC1 through our normal performance and > regression test suite and have found performance to be comparable to 1.0.0. > > That

Re: [DISCUSS] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils

Hmm.. it seems to be a general issue then, since we were planning to also replace the KStreamTestDriver and ProcessorTopologyTestDriver with the new TopologyTestDriver soon, so if the argument that testing dependency could still cause circular dependencies holds it means we cannot do that as well.

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

I think it's fair to assume that topics will always be increased by an integer factor - e.g. from 2 partitions to 4 partitions. Then the mapping is much easier. Why anyone would increase partitions by lass than x2 is a mystery to me. If your two partitions cannot handle the load, then with

Want to contribute

Hello, I want to begin to code for open sources projects in my free time and I want to help the Apache Kafka! I already have read the How to Contribute and Code Guidelines. Do u guys have a label for "easy" Jira's issue or something like this? Att, Eduardo Mello

Re: Want to contribute

Hi Damian, Matthias Could you add me to the KAFKA project on jira, my jira id is sband Thanks Sanket On Fri, 9 Mar 2018, 20:23 Damian Guy, wrote: > Hi Eduardo, > > There are usually JIRAs that have the newbie label. This would be a good > place to start. > > Cheers, >

Re: [DISCUSS] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils

John, Sorry for the delayed response. Thanks for the KIP, I'm +1 on it, and I don't have any further comments on the KIP itself aside from the comments that others have raised. Regarding the existing MockProcessorContext and its removal in favor of the one added from this KIP, I'm actually in

Re: Want to contribute

Hi Eduardo, There are usually JIRAs that have the newbie label. This would be a good place to start. Cheers, Damian On Fri, 9 Mar 2018 at 13:57 Eduardo Mello wrote: > Hello, > I want to begin to code for open sources projects in my free time and I > want to help the

Re: Want to contribute

Hi Sanket, Done. Thanks, Damian On Fri, 9 Mar 2018 at 14:56 Sanket Band wrote: > Hi Damian, Matthias > > Could you add me to the KAFKA project on jira, my jira id is sband > > Thanks > Sanket > > On Fri, 9 Mar 2018, 20:23 Damian Guy, wrote: > > >

Re: Want to contribute

That was quick ! Thanks Thanks Sanket On Fri, Mar 9, 2018 at 8:29 PM, Damian Guy wrote: > Hi Sanket, > > Done. > Thanks, > Damian > > On Fri, 9 Mar 2018 at 14:56 Sanket Band wrote: > > > Hi Damian, Matthias > > > > Could you add me to the KAFKA

Re: Want to contribute

Thank you! Ill look for this label! I created a Jira user too: emello Thanks, Eduardo Mello On Fri, Mar 9, 2018 at 12:00 PM, Sanket Band wrote: > That was quick ! Thanks > > Thanks > Sanket > > On Fri, Mar 9, 2018 at 8:29 PM, Damian Guy wrote:

Re: [DISCUSS] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils

Hi Guozhang and Bill, I'll summarize what I'm currently thinking in light of all the discussion: Mock Processor Context: === Here's how I see the use cases for the two mocks differing: 1. o.a.k.test.MPC: Crafted for testing Streams use cases. Implements