Re: [DISCUSS] KIP-382: MirrorMaker 2.0

> Oh - got it, it checks the entire prefix, which seems obvious to me in retrospect :) Rhys, I've changed the wording to make this more clear, thanks for calling it out. Ryanne On Tue, Oct 16, 2018 at 4:16 PM McCaig, Rhys wrote: > > > In your example, us-west.us-east.us-central.us-west.topic

Re: [EXTERNAL] Incremental Cooperative Rebalancing

Konstantine, thanks for the explanation, makes sense. Ryanne On Tue, Oct 16, 2018, 1:51 PM Konstantine Karantasis < konstant...@confluent.io> wrote: > Matthias, Ryanne, Rhys, Guozhang, thank you all for your comments! > > Ryanne, to try to address your specific comments, let me start by saying

Re: [ANNOUNCE] New Committer: Manikumar Reddy

Congratulations Manikumar! Keep up the good work. On Tue, Oct 16, 2018 at 12:30 AM Jungtaek Lim wrote: > Congrats Mani! > On Tue, 16 Oct 2018 at 1:45 PM Abhimanyu Nagrath < > abhimanyunagr...@gmail.com> > wrote: > > > Congratulations Manikumar > > > > On Tue, Oct 16, 2018 at 10:09 AM Satish

[jira] [Resolved] (KAFKA-7512) java.lang.ClassCastException: java.util.Date cannot be cast to java.lang.Number

[ https://issues.apache.org/jira/browse/KAFKA-7512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Yokota resolved KAFKA-7512. -- Resolution: Duplicate > java.lang.ClassCastException: java.util.Date cannot be cast to >

Re: [DISCUSS] 2.0.1 bug fix release

Thanks for managing the release Manikumar! Ismael On Tue, 16 Oct 2018, 12:13 Manikumar, wrote: > Hi all, > > I would like to volunteer to be the release manager for 2.0.1 bug fix > release. > 2.0 was released July 30, 2018 and 44 issues are fixed so far. > > Please find all the resolved

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

> In your example, us-west.us-east.us-central.us-west.topic is an invalid > "remote topic" name because us-west appears twice. MM2 will not replicate > us-east.us-central.us-west.topic into us-west a second time, because the > source topic already has us-west in the prefix. This is what I mean by

Build failed in Jenkins: kafka-trunk-jdk8 #3145

See Changes: [wangguoz] MINOR: Update Streams Scala API for addition of Grouped (#5793) [github] KAFKA-7513: Fix timing issue in SaslAuthenticatorFailureDelayTest --

Build failed in Jenkins: kafka-2.1-jdk8 #29

See Changes: [rajinisivaram] KAFKA-7513: Fix timing issue in SaslAuthenticatorFailureDelayTest -- [...truncated 437.95 KB...]

[DISCUSS] 2.0.1 bug fix release

Hi all, I would like to volunteer to be the release manager for 2.0.1 bug fix release. 2.0 was released July 30, 2018 and 44 issues are fixed so far. Please find all the resolved tickets here:

Build failed in Jenkins: kafka-trunk-jdk11 #39

See Changes: [github] KAFKA-7513: Fix timing issue in SaslAuthenticatorFailureDelayTest -- [...truncated 1.84 MB...] org.apache.kafka.connect.transforms.CastTest >

Re: [EXTERNAL] Incremental Cooperative Rebalancing

Matthias, Ryanne, Rhys, Guozhang, thank you all for your comments! Ryanne, to try to address your specific comments, let me start by saying that a key concept behind this proposal is the concept of overlapping 'communication' with 'computation', which is known for often reducing the overall cost

Re: [VOTE] KIP-376: Implement AutoClosable on appropriate classes that want to be used in a try-with-resource statement

Bumping this thread up again, thanks! On Fri, Oct 12, 2018, 4:53 PM Colin McCabe wrote: > On Fri, Oct 12, 2018, at 15:45, Yishun Guan wrote: > > Hi Colin, > > > > Thanks for your suggestions. I have modified the current KIP with your > > comments. However, I still think I should keep the entire

[jira] [Resolved] (KAFKA-7196) Remove heartbeat delayed operation for those removed consumers at the end of each rebalance

[ https://issues.apache.org/jira/browse/KAFKA-7196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-7196. Resolution: Fixed > Remove heartbeat delayed operation for those removed consumers at the end of

Help Needed

Hello Kafka Dev, We need help on lagging issue we are seeing on one of the environment which doesn’t have much load. We are running kafka on multiple environement, and on one of our environemnt we do see events are taking huge time (some time more then a day) to get process from kafka. The

Re: [DISCUSS] KIP-354 Time-based log compaction policy

Mayuresh, Thanks for the comments. The requirement is that we need to pick up segments that are older than maxCompactionLagMs for compaction. maxCompactionLagMs is an upper-bound, which implies that picking up segments for compaction earlier doesn't violated the policy. We use the creation time

Re: [DISCUSS] Apache Kafka 2.1.0 Release Plan

Hey everyone, Thanks for all the contribution! Just a kind reminder that the code is now frozen for 2.1.0 release. Thanks, Dong On Mon, Oct 1, 2018 at 4:31 PM Dong Lin wrote: > Hey everyone, > > Hope things are going well! > > Just a kind reminder that the feature freeze time is end of day

Re: [DISCUSS] KIP-381 Connect: Tell about records that had their offsets flushed in callback

Steff, > Guess people have used it, assuming that all records that have been polled > at the time of callback to "commit", have also had their offsets committed. > But that is not true. (excerpt from KIP) The documentation for SourceTask.commit() reads: > Commit the offsets, up to the offsets

Build failed in Jenkins: kafka-2.1-jdk8 #28

See Changes: [wangguoz] MINOR: Update Streams Scala API for addition of Grouped (#5793) -- [...truncated 437.92 KB...]

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

> Could you comment on the approach of > your method vs. using other open source tools like Uber's uReplicator or > the recently open-sourced Mirus from Salesforce? Eno, a primary differentiator is that KIP-382 is "opinionated" about how replication should be done, e.g. by applying topic renaming

Re: [DISCUSS] KIP-377: TopicCommand to use AdminClient

Hi Colin, Thanks, it makes sense and simplifies this KIP tremendously. I'll move this section to the rejected alternatives with a note that KIP-142 will have this feature. On a similar note: is there a KIP for describe topics protocol or have you been thinking about it? I guess there it's the

Build failed in Jenkins: kafka-trunk-jdk11 #38

See Changes: [wangguoz] MINOR: Update Streams Scala API for addition of Grouped (#5793) -- [...truncated 1.84 MB...] org.apache.kafka.connect.transforms.CastTest >

Build failed in Jenkins: kafka-trunk-jdk8 #3144

See Changes: [rajinisivaram] KAFKA-7496: Handle invalid filters gracefully in -- [...truncated 2.35 MB...] org.apache.kafka.connect.transforms.CastTest >

[jira] [Resolved] (KAFKA-7513) Flaky test SaslAuthenticatorFailureDelayTest.testInvalidPasswordSaslPlain

[ https://issues.apache.org/jira/browse/KAFKA-7513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram resolved KAFKA-7513. --- Resolution: Fixed Reviewer: Ismael Juma > Flaky test

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

> But one big obstacle in this was always that group coordination happened on the source cluster. Jan, thank you for bringing up this issue with legacy MirrorMaker. I totally agree with you. This is one of several problems with MirrorMaker I intend to solve in MM2, and I already have a design

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

no worries, glad i could clarify On 16.10.2018 15:14, Andrew Otto wrote: > O ok apologies. Interesting! > > On Tue, Oct 16, 2018 at 9:06 AM Jan Filipiak > wrote: > >> Hi Andrew, >> >> thanks for your message, you missed my point. >> >> Mirrormaker collocation with target is for sure

[jira] [Resolved] (KAFKA-7496) KafkaAdminClient#describeAcls should handle invalid filters gracefully

[ https://issues.apache.org/jira/browse/KAFKA-7496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram resolved KAFKA-7496. --- Resolution: Fixed Reviewer: Rajini Sivaram Fix Version/s: 2.1.0 >

Build failed in Jenkins: kafka-trunk-jdk11 #37

See Changes: [rajinisivaram] KAFKA-7496: Handle invalid filters gracefully in -- [...truncated 1.84 MB...] org.apache.kafka.connect.transforms.CastTest >

Build failed in Jenkins: kafka-2.1-jdk8 #27

See Changes: [rajinisivaram] KAFKA-7496: Handle invalid filters gracefully in -- [...truncated 438.41 KB...] ^

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

O ok apologies. Interesting! On Tue, Oct 16, 2018 at 9:06 AM Jan Filipiak wrote: > Hi Andrew, > > thanks for your message, you missed my point. > > Mirrormaker collocation with target is for sure correct. > But then group coordination happens across WAN which is unnecessary. > And I request

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

Hi Andrew, thanks for your message, you missed my point. Mirrormaker collocation with target is for sure correct. But then group coordination happens across WAN which is unnecessary. And I request to be thought about again. I made a PR back then for zk Consumer to allow having 2 zookeeper

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

> I would generally say a LAN is better than a WAN for doing group > coordinaton For sure, but a LAN is better than a WAN for producing messages too. If there is network congestion during network production, messages will be dropped. With MirrorMaker currently, you can either skip these dropped

[jira] [Created] (KAFKA-7515) Trogdor - Add Consumer Group Benchmark Specification

Stanislav Kozlovski created KAFKA-7515: -- Summary: Trogdor - Add Consumer Group Benchmark Specification Key: KAFKA-7515 URL: https://issues.apache.org/jira/browse/KAFKA-7515 Project: Kafka

[jira] [Created] (KAFKA-7514) Trogdor - Support Multiple Threads in ConsumeBenchWorker

Stanislav Kozlovski created KAFKA-7514: -- Summary: Trogdor - Support Multiple Threads in ConsumeBenchWorker Key: KAFKA-7514 URL: https://issues.apache.org/jira/browse/KAFKA-7514 Project: Kafka

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

Hi, Currently MirrorMaker is usually run collocated with the target cluster. This is all nice and good. But one big obstacle in this was always that group coordination happened on the source cluster. So when then network was congested, you sometimes loose group membership and have to

Build failed in Jenkins: kafka-trunk-jdk8 #3143

See -- [...truncated 505 B...] > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10 Fetching upstream changes from https://github.com/apache/kafka.git > git

[jira] [Created] (KAFKA-7513) Flaky test SaslAuthenticatorFailureDelayTest.testInvalidPasswordSaslPlain

Rajini Sivaram created KAFKA-7513: - Summary: Flaky test SaslAuthenticatorFailureDelayTest.testInvalidPasswordSaslPlain Key: KAFKA-7513 URL: https://issues.apache.org/jira/browse/KAFKA-7513 Project:

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

This update is much needed, thank you! Could you comment on the approach of your method vs. using other open source tools like Uber's uReplicator or the recently open-sourced Mirus from Salesforce? ( https://engineering.salesforce.com/open-sourcing-mirus-3ec2c8a38537). I strongly believe

Build failed in Jenkins: kafka-trunk-jdk8 #3142

See Changes: [colin] KAFKA-6764: Improve the whitelist command-line option for [github] KAFKA-7080 and KAFKA-7222: Cleanup overlapping KIP changes (#5804) [mjsax] KAFKA-7223: Suppression documentation (#5787)

Re: KAFKA-6654 custom SSLContext

Hi Clement, I think many of the previous discussions around custom SslFactory was to do with config options (e.g. avoid cleartext password, configure alias to choose certificate, OCSP support etc.) For these, alternative config options make the most sense because these are generic config options

[jira] [Created] (KAFKA-7512) java.lang.ClassCastException: java.util.Date cannot be cast to java.lang.Number

Rohit Kumar Gupta created KAFKA-7512: Summary: java.lang.ClassCastException: java.util.Date cannot be cast to java.lang.Number Key: KAFKA-7512 URL: https://issues.apache.org/jira/browse/KAFKA-7512

Build failed in Jenkins: kafka-2.1-jdk8 #26

See Changes: [matthias] KAFKA-7080 and KAFKA-7222: Cleanup overlapping KIP changes (#5804) [matthias] KAFKA-7223: Suppression documentation (#5787) [wangguoz] MINOR: Doc changes for KIP-312 (#5789)

Re: [ANNOUNCE] New Committer: Manikumar Reddy

Congrats Mani! On Tue, 16 Oct 2018 at 1:45 PM Abhimanyu Nagrath wrote: > Congratulations Manikumar > > On Tue, Oct 16, 2018 at 10:09 AM Satish Duggana > wrote: > > > Congratulations Mani! > > > > > > On Fri, Oct 12, 2018 at 9:41 PM Colin McCabe wrote: > > > > > > Congratulations, Manikumar!

Throwing away prefetched records optimisation.

Hi there Kafka developers, I am currently trying to find a solution to an issue that has been manifesting itself in the Akka streams implementation of the Kafka connector. When it comes to consuming messages, the implementation relies heavily on the fact that we can pause and resume partitions.

[jira] [Created] (KAFKA-7511) Fetch offset 2287878 is out of range for partition CommitTrn-0, resetting offset

Hitesh Kumar created KAFKA-7511: --- Summary: Fetch offset 2287878 is out of range for partition CommitTrn-0, resetting offset Key: KAFKA-7511 URL: https://issues.apache.org/jira/browse/KAFKA-7511

Jenkins build is back to normal : kafka-trunk-jdk8 #3141

See

Build failed in Jenkins: kafka-trunk-jdk11 #36

See Changes: [mjsax] MINOR: Doc changes for KIP-324 (#5788) [mjsax] MINOR: doc changes for KIP-372 (#5790) [colin] KAFKA-6764: Improve the whitelist command-line option for [github] KAFKA-7080 and KAFKA-7222: