Re: [VOTE] KIP-720 Deprecate MirrorMaker v1

IMHO - I think there is no too much doubt on the effectiveness of KIP-712 and KIP-720, the tricky part may be the timing and the ordering of implementing KIP-712 and KIP-720 (if we do not want to execute both KIP in parallel). If it makes more sense to execute them in sequence, here may be a

Re: [VOTE] KIP-712: Shallow Mirroring

+1 from me. Will be very beneficial and efficient to the users who mirroring at large scale. On 2021/03/25 17:33:53, Henry Cai wrote: > Hi, > > > I'd like to start a vote on KIP-712: Shallow Mirroring. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-712%3A+Shallow+Mirroring >

Re: Proposing KIP-712: Shallow Mirroring

Hello Henry, This is a very interesting proposal. https://issues.apache.org/jira/browse/KAFKA-10728 reflects the similar concern of re-compressing data in mirror maker. Probably one thing may need to clarify is: how "shallow" mirroring is only applied to mirrormaker use case, if the changes

[jira] [Created] (KAFKA-10750) test failure scenarios of MirrorMaker 2

Ning Zhang created KAFKA-10750: -- Summary: test failure scenarios of MirrorMaker 2 Key: KAFKA-10750 URL: https://issues.apache.org/jira/browse/KAFKA-10750 Project: Kafka Issue Type: Improvement

[jira] [Created] (KAFKA-10737) MirrorMaker 2: clarify how to produce to or consume from SSL-enabled cluster in README.md

Ning Zhang created KAFKA-10737: -- Summary: MirrorMaker 2: clarify how to produce to or consume from SSL-enabled cluster in README.md Key: KAFKA-10737 URL: https://issues.apache.org/jira/browse/KAFKA-10737

[jira] [Resolved] (KAFKA-10704) Mirror maker with TLS at target

[ https://issues.apache.org/jira/browse/KAFKA-10704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ning Zhang resolved KAFKA-10704. Resolution: Resolved > Mirror maker with TLS at tar

[jira] [Reopened] (KAFKA-10704) Mirror maker with TLS at target

[ https://issues.apache.org/jira/browse/KAFKA-10704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ning Zhang reopened KAFKA-10704: > Mirror maker with TLS at target > --- > >

[jira] [Resolved] (KAFKA-10424) MirrorMaker 2.0 does not replicates topic's "clean.policy"

[ https://issues.apache.org/jira/browse/KAFKA-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ning Zhang resolved KAFKA-10424. Resolution: Not A Bug > MirrorMaker 2.0 does not replicates topic's "clea

Re: [VOTE] KIP-665 Kafka Connect Hash SMT

Hello, I think this SMT feature is parallel to https://docs.confluent.io/current/connect/transforms/index.html On 2020/10/15 15:24:51, Brandon Brown wrote: > Bumping this thread. > Please take a look at the KIP and vote or let me know if you have any > feedback. > > KIP: >

Re: [VOTE] KIP-656: MirrorMaker2 Exactly-once Semantics

ore starting a vote. > > > On Mon, Aug 17, 2020 at 6:25 PM Ning Zhang wrote: > > > > Hello everyone, > > > > I'd like to start a vote on KIP-656. This KIP aims to make MirrorMaker 2 > > replicates messages exactly-once across clusters > > https://cwiki.ap

Re: [VOTE] KIP-661: Expose task configurations in Connect REST API

Looks a nice proposal. +1 On 2020/09/24 14:53:47, Mickael Maison wrote: > Hi, > > I'd like to start a vote on KIP-661: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-661%3A+Expose+task+configurations+in+Connect+REST+API > > Thanks >

[jira] [Created] (KAFKA-10483) extract common functions from SourceConnector and SourceTask

Ning Zhang created KAFKA-10483: -- Summary: extract common functions from SourceConnector and SourceTask Key: KAFKA-10483 URL: https://issues.apache.org/jira/browse/KAFKA-10483 Project: Kafka

Re: [DISCUSS] KIP-661: Expose task configurations in Connect REST API

That seems a useful interface for display and troubleshooting. On 2020/09/14 16:39:30, Mickael Maison wrote: > Hi all, > > As it's a relatively simple KIP, if I don't see any feedback in the > next few days, I'll start a vote early next week > > Thanks > > On Thu, Aug 20, 2020 at 5:47 PM

Re: [DISCUSS] KIP-656: MirrorMaker2 Exactly-once Semantics

we should include a Sink connector > for checkpoints too > > On Thu, Sep 3, 2020 at 6:51 AM Ning Zhang wrote: > > > > bump for another potential more discussion > > > > On 2020/08/27 23:31:38, Ning Zhang wrote: > > > Hello Mickael, > > > > > &

Re: [DISCUSS] KIP-656: MirrorMaker2 Exactly-once Semantics

ource > (MirrorCheckpoint) connector for MM2 you will need 2 Connect runtimes. > So this does not work well for users of Connect. I've not really > looked into it yet but I wonder if we should include a Sink connector > for checkpoints too > > On Thu, Sep 3, 2020 at 6:51 AM

Re: [DISCUSS] KIP-656: MirrorMaker2 Exactly-once Semantics

bump for another potential more discussion On 2020/08/27 23:31:38, Ning Zhang wrote: > Hello Mickael, > > > 1. How does offset translation work with this new sink connector? > > Should we also include a CheckpointSinkConnector? > > CheckpointSourceConnector wil

Re: [DISCUSS] KIP-656: MirrorMaker2 Exactly-once Semantics

21, 2020 at 4:58 PM Ryanne Dolan wrote: > > > > Awesome, this will be a huge advancement. I also want to point out that > > this KIP implements MirrorSinkConnector as well, finally, which is a very > > often requested missing feature in my experience. > > > > Ryanne >

[jira] [Reopened] (KAFKA-10133) Cannot compress messages in destination cluster with MM2

[ https://issues.apache.org/jira/browse/KAFKA-10133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ning Zhang reopened KAFKA-10133: There is no bug in the code, but need some efforts on doc to clarify on where some frequently used

Re: [VOTE] KIP-656: MirrorMaker2 Exactly-once Semantics

ring it's not a trivial feature, we should have a [DISCUSS] > thread before starting a vote. > > > On Mon, Aug 17, 2020 at 6:25 PM Ning Zhang wrote: > > > > Hello everyone, > > > > I'd like to start a vote on KIP-656. This KIP aims to make MirrorMaker 2 > &g

[DISCUSS] KIP-656: MirrorMaker2 Exactly-once Semantics

Hello, I wrote a KIP about MirrorMaker2 Exactly-once Semantics (EOS) https://cwiki.apache.org/confluence/display/KAFKA/KIP-656%3A+MirrorMaker2+Exactly-once+Semantics At the high-level, it resembles the idea of how HDFS Sink Connector achieves EOS across clusters by managing and storing the

[VOTE] KIP-656: MirrorMaker2 Exactly-once Semantics

Hello everyone, I'd like to start a vote on KIP-656. This KIP aims to make MirrorMaker 2 replicates messages exactly-once across clusters https://cwiki.apache.org/confluence/display/KAFKA/KIP-656%3A+MirrorMaker2+Exactly-once+Semantics

[jira] [Created] (KAFKA-10370) WorkerSinkTask: IllegalStateException cased by consumer.seek(tp, offsets) when (tp, offsets) are supplied by WorkerSinkTaskContext

Ning Zhang created KAFKA-10370: -- Summary: WorkerSinkTask: IllegalStateException cased by consumer.seek(tp, offsets) when (tp, offsets) are supplied by WorkerSinkTaskContext Key: KAFKA-10370 URL: https

[jira] [Created] (KAFKA-10339) MirrorMaker2 Exactly-once Semantics

Ning Zhang created KAFKA-10339: -- Summary: MirrorMaker2 Exactly-once Semantics Key: KAFKA-10339 URL: https://issues.apache.org/jira/browse/KAFKA-10339 Project: Kafka Issue Type: New Feature

Re: [ANNOUNCE] New Kafka PMC Member: Mickael Maison

Congrats Mickael ! On 2020/07/31 15:14:30, Jun Rao wrote: > Hi, Everyone, > > Mickael Maison has been a Kafka committer since Nov. 5, 2019. He has > remained active in the community since becoming a committer. It's my > pleasure to announce that Mickael is now a member of Kafka PMC. > >

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

Hello Chris, That is an interesting KIP. I have a couple of questions: (1) in section of pseudo-code, what if the failure happens between 4(b) and 5(a), meaning after the producer commit the transaction, and before task.commitRecord(). (2) in section "source task life time", what is the

[jira] [Created] (KAFKA-10304) Revisit and improve the tests of MirrorMaker 2

Ning Zhang created KAFKA-10304: -- Summary: Revisit and improve the tests of MirrorMaker 2 Key: KAFKA-10304 URL: https://issues.apache.org/jira/browse/KAFKA-10304 Project: Kafka Issue Type: Test

Re: [VOTE] KIP-545 support automated consumer offset sync across clusters in MM 2.0

; > > > > > > >+1 (binding) > > > >Thanks for the KIP! > > > > > > > >On Thu, Dec 5, 2019 at 12:56 AM Ryanne Dolan > > > > > > wrote: > > > >> > > > >> Bump. We've got 2

[jira] [Created] (KAFKA-9360) emitting checkpoint and heartbeat set to false will not disable the activity in their SourceTask

Ning Zhang created KAFKA-9360: - Summary: emitting checkpoint and heartbeat set to false will not disable the activity in their SourceTask Key: KAFKA-9360 URL: https://issues.apache.org/jira/browse/KAFKA-9360

[jira] [Created] (KAFKA-9352) unbalanced assignment of topic-partition to tasks

Ning Zhang created KAFKA-9352: - Summary: unbalanced assignment of topic-partition to tasks Key: KAFKA-9352 URL: https://issues.apache.org/jira/browse/KAFKA-9352 Project: Kafka Issue Type

Re: [VOTE] KIP-545 support automated consumer offset sync across clusters in MM 2.0

My current plan is to implement this in "MirrorCheckpointTask" On 2019/11/02 03:30:11, Xu Jianhai wrote: > I think this kip will implement a task in sinkTask ? right? > > On Sat, Nov 2, 2019 at 1:06 AM Ryanne Dolan wrote: > > > Hey y'all, Ning Zhang and I

[DISCUSS] KIP-545: support automated consumer offset sync across clusters in MM 2.0

Hello All, MirrorMaker 2.0 (MM 2.0) is a new Kafka community initiative that is designed to address the limitations of current MirrorMaker. The offset translation is one of MM2.0's features to serve the foundation of

request permission to create KIP

Hello, I am working on this jira https://issues.apache.org/jira/browse/KAFKA-9076 which requires a KIP. Since I am new to Confluence, I would like to request permission to create KIP. My confluence User ID: yangguo1220 My registered email: ning2008w...@gmail.com Thanks, Ning

[jira] [Created] (KAFKA-9076) MirrorMaker 2.0 automated consumer offset sync

Ning Zhang created KAFKA-9076: - Summary: MirrorMaker 2.0 automated consumer offset sync Key: KAFKA-9076 URL: https://issues.apache.org/jira/browse/KAFKA-9076 Project: Kafka Issue Type

[jira] [Resolved] (KAFKA-3841) MirrorMaker topic renaming

[ https://issues.apache.org/jira/browse/KAFKA-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ning Zhang resolved KAFKA-3841. --- Resolution: Won't Fix > MirrorMaker topic renam

[jira] [Commented] (KAFKA-3841) MirrorMaker topic renaming

[ https://issues.apache.org/jira/browse/KAFKA-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15330877#comment-15330877 ] Ning Zhang commented on KAFKA-3841: --- Thanks. This is good to know. As MessageHandler was recently added

[jira] [Updated] (KAFKA-3841) MirrorMaker topic renaming

[ https://issues.apache.org/jira/browse/KAFKA-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ning Zhang updated KAFKA-3841: -- Description: Our organization (walmart.com) has been a Kafka user since some years back

[jira] [Created] (KAFKA-3841) MirrorMaker topic renaming

Ning Zhang created KAFKA-3841: - Summary: MirrorMaker topic renaming Key: KAFKA-3841 URL: https://issues.apache.org/jira/browse/KAFKA-3841 Project: Kafka Issue Type: New Feature