KIP 678: New Kafka Connect SMT for plainText => Struct(or Map) with Regex

Hi, I've opened KIP-678 which is intended to provide a new SMT in Kafka Connect. I'd be grateful for any feedback:https://cwiki.apache.org/confluence/display/KAFKA/KIP+678%3A+New+Kafka+Connect+SMT+for+plainText+%3D%3E+Struct%28or+Map%29+with+Regex== thanks, whsoul

Re: [DISCUSS] Checkstyle Import Order

As of Present: Committers: - Bruno: 2 and 3. - Gwen: (No Specific Preference) Non-Committers: - Brandon: 2. - Dongjin: 2 and 3. Let's hold on for 2 or 3 committers. Best, Dongjin On Fri, Oct 23, 2020 at 10:09 AM Gwen Shapira wrote: > I don't have any specific preference on the style. But

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #200

See Changes: [github] MINOR: Update docs to point to next release add notable features for 2.7 (#9483) [github] MINOR: Update raft/README.md and minor RaftConfig tweaks (#9484) [github] MINOR:

Build failed in Jenkins: Kafka » kafka-2.7-jdk8 #35

See Changes: [Bill Bejeck] MINOR: Add notable changes for 2.7 release -- [...truncated 3.43 MB...] org.apache.kafka.streams.test.OutputVerifierTest >

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #199

See Changes: [github] Handle ProducerFencedException on offset commit (#9479) -- [...truncated 6.90 MB...] org.apache.kafka.streams.test.OutputVerifierTest

Build failed in Jenkins: Kafka » kafka-2.7-jdk8 #34

See Changes: [John Roesler] Handle ProducerFencedException on offset commit (#9479) -- [...truncated 6.87 MB...]

Re: [DISCUSS] Checkstyle Import Order

I don't have any specific preference on the style. But I am glad you are bringing it up. Every other project I worked on had a specific import style, and the random import changes in PRs are pretty annoying. On Wed, Oct 14, 2020 at 10:36 PM Dongjin Lee wrote: > > Hello. I hope to open a

Build failed in Jenkins: Kafka » kafka-trunk-jdk8 #165

See Changes: [github] Handle ProducerFencedException on offset commit (#9479) -- [...truncated 6.84 MB...] org.apache.kafka.streams.TopologyTestDriverTest >

Jenkins build is back to normal : Kafka » kafka-trunk-jdk11 #174

See

[jira] [Created] (KAFKA-10636) Bypass log validation for writes to raft log

Jason Gustafson created KAFKA-10636: --- Summary: Bypass log validation for writes to raft log Key: KAFKA-10636 URL: https://issues.apache.org/jira/browse/KAFKA-10636 Project: Kafka Issue

Jenkins build is back to normal : Kafka » kafka-2.3-jdk8 #11

See

[jira] [Created] (KAFKA-10635) Streams application fails with OutOfOrderSequenceException after rolling restarts of brokers

Peeraya Maetasatidsuk created KAFKA-10635: - Summary: Streams application fails with OutOfOrderSequenceException after rolling restarts of brokers Key: KAFKA-10635 URL:

Re: Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #198

Build failed in Jenkins: Kafka » kafka-2.2-jdk8 #6

See Changes: [github] MINOR: Add Jenkinsfile to 2.2 (#9474) -- [...truncated 3.26 MB...] kafka.tools.ConsoleConsumerTest >

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #198

See Changes: [github] KAFKA-10284: Group membership update due to static member rejoin should be persisted (#9270) -- [...truncated 6.90 MB...]

Build failed in Jenkins: Kafka » kafka-trunk-jdk11 #173

See Changes: [github] KAFKA-10284: Group membership update due to static member rejoin should be persisted (#9270) -- [...truncated 3.44 MB...]

Build failed in Jenkins: Kafka » kafka-2.7-jdk8 #33

See Changes: [John Roesler] KAFKA-10284: Group membership update due to static member rejoin should be persisted (#9270) [Bill Bejeck] Update docs/documentation.html to point to 2.7 release and add

Jenkins build is back to normal : Kafka » kafka-trunk-jdk8 #164

See

[jira] [Created] (KAFKA-10634) LeaderChangeMessage should include the leader as one of the voters

Jose Armando Garcia Sancio created KAFKA-10634: -- Summary: LeaderChangeMessage should include the leader as one of the voters Key: KAFKA-10634 URL: https://issues.apache.org/jira/browse/KAFKA-10634

Re: Requesting Permission to submit Kafka Improvement Proposals

What is your username? On 10/22/20 2:23 AM, Aron Tigor Möllers wrote: > Hello Kafka Devs, > > I hereby kindly request permission to submit KIPs in the Wiki as described > in > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > Thank you and kind regards > > Aron

Re: Access request

Done. On 10/22/20 12:12 PM, Brad Davis wrote: > Ugh sorry, my Wiki username is "jherico" > > On Thu, Oct 22, 2020 at 12:11 PM Brad Davis wrote: > >> Requesting confluence wiki permission so I can create a KIP for a small >> feature add for Kafka Connect. >> >> Thanks, >> Brad >> >

Build failed in Jenkins: Kafka » kafka-2.6-jdk8 #41

See Changes: [John Roesler] KAFKA-10284: Group membership update due to static member rejoin should be persisted (#9270) -- [...truncated 6.32 MB...]

Access request

Requesting confluence wiki permission so I can create a KIP for a small feature add for Kafka Connect. Thanks, Brad

Re: Access request

Ugh sorry, my Wiki username is "jherico" On Thu, Oct 22, 2020 at 12:11 PM Brad Davis wrote: > Requesting confluence wiki permission so I can create a KIP for a small > feature add for Kafka Connect. > > Thanks, > Brad >

[jira] [Created] (KAFKA-10633) Constant probing rebalances in Streams 2.6

Bradley Peterson created KAFKA-10633: Summary: Constant probing rebalances in Streams 2.6 Key: KAFKA-10633 URL: https://issues.apache.org/jira/browse/KAFKA-10633 Project: Kafka Issue

Re: [DISCUSS] Apache Kafka 2.7.0 release

Hi Bill, I took a second look at the git history and now it actually seems to be a regression. Probably, a change in August that extended the error codes introduced this bug. Best, Bruno On 22.10.20 19:50, Bill Bejeck wrote: Hi Bruno, While technically it's not a regression, I think this

[jira] [Resolved] (KAFKA-9999) Internal topic creation failure should be non-fatal and trigger explicit rebalance

[ https://issues.apache.org/jira/browse/KAFKA-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Chen resolved KAFKA-. Resolution: Won't Fix > Internal topic creation failure should be non-fatal and trigger explicit

[jira] [Created] (KAFKA-10632) Raft client should push all committed data to listeners

Jason Gustafson created KAFKA-10632: --- Summary: Raft client should push all committed data to listeners Key: KAFKA-10632 URL: https://issues.apache.org/jira/browse/KAFKA-10632 Project: Kafka

Build failed in Jenkins: Kafka » kafka-trunk-jdk11 #172

See Changes: [github] MINOR: Add some class javadoc to Admin client (#9459) [github] MINOR: TopologyTestDriver should not require dummy parameters (#9477)

Re: [DISCUSS] Apache Kafka 2.7.0 release

Hi Bruno, While technically it's not a regression, I think this is an important fix with a low-risk to include, so we can leave it as a blocker. Thanks, Bill On Thu, Oct 22, 2020 at 1:25 PM Bruno Cadonna wrote: > Hi Bill, > > we encountered the following bug in our soak testing cluster. > >

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

Hi again, Colin. Related to the issue of how to communicate fenced vs. not fenced (and how to communicate or imply Offline vs not Offline), do we need to communicate in the log that a broker has gracefully shut down? We do distinguish between a broker being unavailable due to a controlled

Jenkins build is back to normal : Kafka » kafka-2.7-jdk8 #32

See

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

HI Colin. A FencedBrokerRecord appears in the metadata log when a broker is fenced. What appears in the metadata log to indicate that a broker is no longer fenced? Does a BrokerRecord appear? That seems to communicate a bunch of unnecessary data in this context (endpoints, features, rack). If

Re: [DISCUSS] Apache Kafka 2.7.0 release

Hi Bill, we encountered the following bug in our soak testing cluster. https://issues.apache.org/jira/browse/KAFKA-10631 I classified the bug as a blocker because it caused the death of a stream thread. It does not seem to be a regression, though. I opened a PR to fix the bug here:

[jira] [Created] (KAFKA-10631) ProducerFencedException is not Handled on Offest Commit

Bruno Cadonna created KAFKA-10631: - Summary: ProducerFencedException is not Handled on Offest Commit Key: KAFKA-10631 URL: https://issues.apache.org/jira/browse/KAFKA-10631 Project: Kafka

Re: [VOTE] KIP-665 Kafka Connect Hash SMT

Hey Gunnar, Those are great questions! 1) I went with it only selecting top level fields since it seems like that’s the way most of the out of the box SMTS work, however I could see a lot of value in it supporting nested fields. 2) I had not thought about adding salt but I think that would be

Re: [VOTE] KIP-665 Kafka Connect Hash SMT

Hey Brandon, I think that's an interesting idea, we got something as a built-in connector feature in Debezium, too [1]. Two questions: * Can "field" select nested fields, e.g. "after.email"? * Did you consider an option for specifying salt for the hash functions? --Gunnar [1]

Requesting Permission to submit Kafka Improvement Proposals

Hello Kafka Devs, I hereby kindly request permission to submit KIPs in the Wiki as described in https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals Thank you and kind regards Aron Tigor Moellers

Re: [DISCUSS] Apache Kafka 2.7.0 release

Hi All, We've hit code freeze. The current status for cutting an RC is there is one blocker issue. It looks like there is a fix in the works, so hopefully, it will get merged early next week. At that point, if there are no other blockers, I proceed with the RC process. Thanks, Bill On Wed,

Build failed in Jenkins: Kafka » kafka-trunk-jdk8 #163

See Changes: [github] MINOR: Add some class javadoc to Admin client (#9459) [github] MINOR: TopologyTestDriver should not require dummy parameters (#9477)

[jira] [Resolved] (KAFKA-10284) Group membership update due to static member rejoin should be persisted

[ https://issues.apache.org/jira/browse/KAFKA-10284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler resolved KAFKA-10284. -- Resolution: Fixed > Group membership update due to static member rejoin should be persisted >

[jira] [Created] (KAFKA-10630) State Directory config could be improved

John Roesler created KAFKA-10630: Summary: State Directory config could be improved Key: KAFKA-10630 URL: https://issues.apache.org/jira/browse/KAFKA-10630 Project: Kafka Issue Type: Task

[jira] [Created] (KAFKA-10629) TopologyTestDriver should not require a Properties arg

John Roesler created KAFKA-10629: Summary: TopologyTestDriver should not require a Properties arg Key: KAFKA-10629 URL: https://issues.apache.org/jira/browse/KAFKA-10629 Project: Kafka Issue

[jira] [Created] (KAFKA-10628) Follow-up: Remove all unnecessary dummy TopologyTestDriver configs

John Roesler created KAFKA-10628: Summary: Follow-up: Remove all unnecessary dummy TopologyTestDriver configs Key: KAFKA-10628 URL: https://issues.apache.org/jira/browse/KAFKA-10628 Project: Kafka

Jenkins build is back to normal : Kafka » kafka-trunk-jdk11 #171

See

[jira] [Created] (KAFKA-10627) Connect TimestampConverter transform does not support multiple formats for the same field and only allows one field to be transformed at a time

Joshua Grisham created KAFKA-10627: -- Summary: Connect TimestampConverter transform does not support multiple formats for the same field and only allows one field to be transformed at a time Key: KAFKA-10627

Re: [VOTE] KIP-676: Respect the logging hierarchy

Binding: 3 (John, Gwen, Mickael) Non-binding: 2 (Dongjin, Brandon) This KIP is now accepted. Thanks for participating! Best, Dongjin On Thu, Oct 22, 2020 at 7:52 PM Brandon Brown wrote: > +1 (non binding) > > Brandon Brown > > On Oct 22, 2020, at 5:03 AM, Mickael Maison > wrote: > > > > +1

Re: [VOTE] KIP-665 Kafka Connect Hash SMT

Gonna give this another little bump. :) Brandon Brown > On Oct 15, 2020, at 12:51 PM, Brandon Brown wrote: > >  > As I mentioned in the KIP, this transformer is slightly different from the > current MaskField SMT. > >> Currently there exists a MaskField SMT but that would completely remove

Re: [VOTE] KIP-676: Respect the logging hierarchy

+1 (non binding) Brandon Brown > On Oct 22, 2020, at 5:03 AM, Mickael Maison wrote: > > +1 (binding) > Thanks > >> On Sun, Oct 11, 2020 at 8:09 AM Dongjin Lee wrote: >> >> Binding: 2 (John, Gwen) >> Non-binding: 1 (Dongjin) >> >> We need one more binding now. >> >> Regards, >> Dongjin >>

Re: [DISCUSS] Checkstyle Import Order

I like option 2. Brandon Brown > On Oct 15, 2020, at 1:36 AM, Dongjin Lee wrote: > > Hello. I hope to open a discussion about the import order in Java code. > > As Nikolay stated recently[^1], Kafka uses a relatively strict code style > for Java code. However, it misses any rule on import

Re: [DISCUSS] Checkstyle Import Order

Hi Dongjin, Thank you for bringing this up. I like options 2 and 3. Best, Bruno On 15.10.20 07:36, Dongjin Lee wrote: Hello. I hope to open a discussion about the import order in Java code. As Nikolay stated recently[^1], Kafka uses a relatively strict code style for Java code. However, it

[jira] [Created] (KAFKA-10626) Add support for max.in.flight.requests.per.connection behaviour in MockClient

Rajini Sivaram created KAFKA-10626: -- Summary: Add support for max.in.flight.requests.per.connection behaviour in MockClient Key: KAFKA-10626 URL: https://issues.apache.org/jira/browse/KAFKA-10626

Re: [VOTE] KIP-676: Respect the logging hierarchy

+1 (binding) Thanks On Sun, Oct 11, 2020 at 8:09 AM Dongjin Lee wrote: > > Binding: 2 (John, Gwen) > Non-binding: 1 (Dongjin) > > We need one more binding now. > > Regards, > Dongjin > > On Sun, Oct 11, 2020 at 3:05 PM Gwen Shapira wrote: > > > +1 (binding) > > > > Makes sense, thank you. > > >

Build failed in Jenkins: Kafka » kafka-2.7-jdk8 #31

See Changes: [Bill Bejeck] KAFKA-10520; Ensure transactional producers poll if leastLoadedNode not available with max.in.flight=1 (#9406) -- [...truncated 6.86

2.7 recent commits

Hey all, Just a heads up that two recent commits on the 2.7 branch have had their commit ids changed (1 & 2 ). The commits themselves