Re: [VOTE] KIP-545 support automated consumer offset sync across clusters in MM 2.0

My current plan is to implement this in "MirrorCheckpointTask" On 2019/11/02 03:30:11, Xu Jianhai wrote: > I think this kip will implement a task in sinkTask ? right? > > On Sat, Nov 2, 2019 at 1:06 AM Ryanne Dolan wrote: > > > Hey y'all, Ning Zhang and I

request permission to create KIP

Hello, I am working on this jira https://issues.apache.org/jira/browse/KAFKA-9076 which requires a KIP. Since I am new to Confluence, I would like to request permission to create KIP. My confluence User ID: yangguo1220 My registered email: ning2008w...@gmail.com Thanks, Ning

[DISCUSS] KIP-545: support automated consumer offset sync across clusters in MM 2.0

Hello All, MirrorMaker 2.0 (MM 2.0) is a new Kafka community initiative that is designed to address the limitations of current MirrorMaker. The offset translation is one of MM2.0's features to serve the foundation of

Re: [VOTE] KIP-545 support automated consumer offset sync across clusters in MM 2.0

; > > > > > > >+1 (binding) > > > >Thanks for the KIP! > > > > > > > >On Thu, Dec 5, 2019 at 12:56 AM Ryanne Dolan > > > > > > wrote: > > > >> > > > >> Bump. We've got 2

Re: [DISCUSS] KIP-656: MirrorMaker2 Exactly-once Semantics

ource > (MirrorCheckpoint) connector for MM2 you will need 2 Connect runtimes. > So this does not work well for users of Connect. I've not really > looked into it yet but I wonder if we should include a Sink connector > for checkpoints too > > On Thu, Sep 3, 2020 at 6:51 AM

Re: [DISCUSS] KIP-656: MirrorMaker2 Exactly-once Semantics

we should include a Sink connector > for checkpoints too > > On Thu, Sep 3, 2020 at 6:51 AM Ning Zhang wrote: > > > > bump for another potential more discussion > > > > On 2020/08/27 23:31:38, Ning Zhang wrote: > > > Hello Mickael, > > > > > &

Re: [DISCUSS] KIP-661: Expose task configurations in Connect REST API

That seems a useful interface for display and troubleshooting. On 2020/09/14 16:39:30, Mickael Maison wrote: > Hi all, > > As it's a relatively simple KIP, if I don't see any feedback in the > next few days, I'll start a vote early next week > > Thanks > > On Thu, Aug 20, 2020 at 5:47 PM

Re: [DISCUSS] KIP-656: MirrorMaker2 Exactly-once Semantics

bump for another potential more discussion On 2020/08/27 23:31:38, Ning Zhang wrote: > Hello Mickael, > > > 1. How does offset translation work with this new sink connector? > > Should we also include a CheckpointSinkConnector? > > CheckpointSourceConnector wil

Re: [VOTE] KIP-665 Kafka Connect Hash SMT

Hello, I think this SMT feature is parallel to https://docs.confluent.io/current/connect/transforms/index.html On 2020/10/15 15:24:51, Brandon Brown wrote: > Bumping this thread. > Please take a look at the KIP and vote or let me know if you have any > feedback. > > KIP: >

Re: [VOTE] KIP-661: Expose task configurations in Connect REST API

Looks a nice proposal. +1 On 2020/09/24 14:53:47, Mickael Maison wrote: > Hi, > > I'd like to start a vote on KIP-661: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-661%3A+Expose+task+configurations+in+Connect+REST+API > > Thanks >

Re: [VOTE] KIP-656: MirrorMaker2 Exactly-once Semantics

ore starting a vote. > > > On Mon, Aug 17, 2020 at 6:25 PM Ning Zhang wrote: > > > > Hello everyone, > > > > I'd like to start a vote on KIP-656. This KIP aims to make MirrorMaker 2 > > replicates messages exactly-once across clusters > > https://cwiki.ap

[DISCUSS] KIP-656: MirrorMaker2 Exactly-once Semantics

Hello, I wrote a KIP about MirrorMaker2 Exactly-once Semantics (EOS) https://cwiki.apache.org/confluence/display/KAFKA/KIP-656%3A+MirrorMaker2+Exactly-once+Semantics At the high-level, it resembles the idea of how HDFS Sink Connector achieves EOS across clusters by managing and storing the

Re: [VOTE] KIP-656: MirrorMaker2 Exactly-once Semantics

ring it's not a trivial feature, we should have a [DISCUSS] > thread before starting a vote. > > > On Mon, Aug 17, 2020 at 6:25 PM Ning Zhang wrote: > > > > Hello everyone, > > > > I'd like to start a vote on KIP-656. This KIP aims to make MirrorMaker 2 > &g

Re: [DISCUSS] KIP-656: MirrorMaker2 Exactly-once Semantics

21, 2020 at 4:58 PM Ryanne Dolan wrote: > > > > Awesome, this will be a huge advancement. I also want to point out that > > this KIP implements MirrorSinkConnector as well, finally, which is a very > > often requested missing feature in my experience. > > > > Ryanne >

[VOTE] KIP-656: MirrorMaker2 Exactly-once Semantics

Hello everyone, I'd like to start a vote on KIP-656. This KIP aims to make MirrorMaker 2 replicates messages exactly-once across clusters https://cwiki.apache.org/confluence/display/KAFKA/KIP-656%3A+MirrorMaker2+Exactly-once+Semantics

Re: [ANNOUNCE] New Kafka PMC Member: Mickael Maison

Congrats Mickael ! On 2020/07/31 15:14:30, Jun Rao wrote: > Hi, Everyone, > > Mickael Maison has been a Kafka committer since Nov. 5, 2019. He has > remained active in the community since becoming a committer. It's my > pleasure to announce that Mickael is now a member of Kafka PMC. > >

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

Hello Chris, That is an interesting KIP. I have a couple of questions: (1) in section of pseudo-code, what if the failure happens between 4(b) and 5(a), meaning after the producer commit the transaction, and before task.commitRecord(). (2) in section "source task life time", what is the

Re: [VOTE] KIP-720 Deprecate MirrorMaker v1

IMHO - I think there is no too much doubt on the effectiveness of KIP-712 and KIP-720, the tricky part may be the timing and the ordering of implementing KIP-712 and KIP-720 (if we do not want to execute both KIP in parallel). If it makes more sense to execute them in sequence, here may be a

Re: [VOTE] KIP-712: Shallow Mirroring

+1 from me. Will be very beneficial and efficient to the users who mirroring at large scale. On 2021/03/25 17:33:53, Henry Cai wrote: > Hi, > > > I'd like to start a vote on KIP-712: Shallow Mirroring. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-712%3A+Shallow+Mirroring >

Re: Proposing KIP-712: Shallow Mirroring

Hello Henry, This is a very interesting proposal. https://issues.apache.org/jira/browse/KAFKA-10728 reflects the similar concern of re-compressing data in mirror maker. Probably one thing may need to clarify is: how "shallow" mirroring is only applied to mirrormaker use case, if the changes

[jira] [Created] (KAFKA-3841) MirrorMaker topic renaming

Ning Zhang created KAFKA-3841: - Summary: MirrorMaker topic renaming Key: KAFKA-3841 URL: https://issues.apache.org/jira/browse/KAFKA-3841 Project: Kafka Issue Type: New Feature

[jira] [Updated] (KAFKA-3841) MirrorMaker topic renaming

[ https://issues.apache.org/jira/browse/KAFKA-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ning Zhang updated KAFKA-3841: -- Description: Our organization (walmart.com) has been a Kafka user since some years back

[jira] [Commented] (KAFKA-3841) MirrorMaker topic renaming

[ https://issues.apache.org/jira/browse/KAFKA-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15330877#comment-15330877 ] Ning Zhang commented on KAFKA-3841: --- Thanks. This is good to know. As MessageHandler was recently added

[jira] [Resolved] (KAFKA-3841) MirrorMaker topic renaming

[ https://issues.apache.org/jira/browse/KAFKA-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ning Zhang resolved KAFKA-3841. --- Resolution: Won't Fix > MirrorMaker topic renam

[jira] [Created] (KAFKA-9076) MirrorMaker 2.0 automated consumer offset sync

Ning Zhang created KAFKA-9076: - Summary: MirrorMaker 2.0 automated consumer offset sync Key: KAFKA-9076 URL: https://issues.apache.org/jira/browse/KAFKA-9076 Project: Kafka Issue Type

[jira] [Created] (KAFKA-9360) emitting checkpoint and heartbeat set to false will not disable the activity in their SourceTask

Ning Zhang created KAFKA-9360: - Summary: emitting checkpoint and heartbeat set to false will not disable the activity in their SourceTask Key: KAFKA-9360 URL: https://issues.apache.org/jira/browse/KAFKA-9360

[jira] [Created] (KAFKA-9352) unbalanced assignment of topic-partition to tasks

Ning Zhang created KAFKA-9352: - Summary: unbalanced assignment of topic-partition to tasks Key: KAFKA-9352 URL: https://issues.apache.org/jira/browse/KAFKA-9352 Project: Kafka Issue Type

[jira] [Created] (KAFKA-10483) extract common functions from SourceConnector and SourceTask

Ning Zhang created KAFKA-10483: -- Summary: extract common functions from SourceConnector and SourceTask Key: KAFKA-10483 URL: https://issues.apache.org/jira/browse/KAFKA-10483 Project: Kafka

[jira] [Reopened] (KAFKA-10133) Cannot compress messages in destination cluster with MM2

[ https://issues.apache.org/jira/browse/KAFKA-10133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ning Zhang reopened KAFKA-10133: There is no bug in the code, but need some efforts on doc to clarify on where some frequently used

[jira] [Created] (KAFKA-10339) MirrorMaker2 Exactly-once Semantics

Ning Zhang created KAFKA-10339: -- Summary: MirrorMaker2 Exactly-once Semantics Key: KAFKA-10339 URL: https://issues.apache.org/jira/browse/KAFKA-10339 Project: Kafka Issue Type: New Feature

[jira] [Created] (KAFKA-10370) WorkerSinkTask: IllegalStateException cased by consumer.seek(tp, offsets) when (tp, offsets) are supplied by WorkerSinkTaskContext

Ning Zhang created KAFKA-10370: -- Summary: WorkerSinkTask: IllegalStateException cased by consumer.seek(tp, offsets) when (tp, offsets) are supplied by WorkerSinkTaskContext Key: KAFKA-10370 URL: https

[jira] [Created] (KAFKA-10304) Revisit and improve the tests of MirrorMaker 2

Ning Zhang created KAFKA-10304: -- Summary: Revisit and improve the tests of MirrorMaker 2 Key: KAFKA-10304 URL: https://issues.apache.org/jira/browse/KAFKA-10304 Project: Kafka Issue Type: Test

[jira] [Resolved] (KAFKA-10424) MirrorMaker 2.0 does not replicates topic's "clean.policy"

[ https://issues.apache.org/jira/browse/KAFKA-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ning Zhang resolved KAFKA-10424. Resolution: Not A Bug > MirrorMaker 2.0 does not replicates topic's "clea

[jira] [Created] (KAFKA-10750) test failure scenarios of MirrorMaker 2

Ning Zhang created KAFKA-10750: -- Summary: test failure scenarios of MirrorMaker 2 Key: KAFKA-10750 URL: https://issues.apache.org/jira/browse/KAFKA-10750 Project: Kafka Issue Type: Improvement

[jira] [Reopened] (KAFKA-10704) Mirror maker with TLS at target

[ https://issues.apache.org/jira/browse/KAFKA-10704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ning Zhang reopened KAFKA-10704: > Mirror maker with TLS at target > --- > >

[jira] [Resolved] (KAFKA-10704) Mirror maker with TLS at target

[ https://issues.apache.org/jira/browse/KAFKA-10704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ning Zhang resolved KAFKA-10704. Resolution: Resolved > Mirror maker with TLS at tar

[jira] [Created] (KAFKA-10737) MirrorMaker 2: clarify how to produce to or consume from SSL-enabled cluster in README.md

Ning Zhang created KAFKA-10737: -- Summary: MirrorMaker 2: clarify how to produce to or consume from SSL-enabled cluster in README.md Key: KAFKA-10737 URL: https://issues.apache.org/jira/browse/KAFKA-10737