Re: Do we want to add more SMTs to Apache Kafka?
I agree, if the desire is to keep the internal SMTs collection small then providing an ease of discovery like Gunnar suggestions would be extremely helpful. Brandon Brown > On Nov 19, 2021, at 6:13 PM, Gunnar Morling > wrote: > > Hi all, > > Just came across this thread, I hope the late reply is ok. > > FWIW, we're in a similar situation in Debezium, where users often request > new (Debezium-specific) SMTs, and we generally tend to recommend them to be > maintained by users themselves, unless they are truly generic. This > excludes a share of users though who aren't Java developers. > > What might help is having means of simple discoverability of externally > hosted SMTs, e.g. via some kind of catalog hosted on kafka.apache.org. That > way, people would have it easier to find and obtain SMTs from other places, > reducing the pressure to get them added to Apache Kafka proper. > > Best, > > --Gunnar > > > > >> Am So., 7. Nov. 2021 um 21:49 Uhr schrieb Brandon Brown < >> bran...@bbrownsound.com>: >> >> I like the idea of a select number of SMTs being offered and supported out >> of the box. The addition of SMTs via this process is nice because it allows >> for a rich set to be supported out of the box and without the need for >> extra work to deploy. >> >> Perhaps this is a spot where the community could express the interest of >> additional SMTs which maybe are available via an open source library and if >> enough usage occurs there could be a path to fold into the Kafka project at >> large? >> >> Brandon Brown >> >> >>>> On Nov 7, 2021, at 1:19 PM, Randall Hauch wrote: >>> >>> We have had several requests to add more Connect Single Message >>> Transforms (SMTs) to the project. When SMTs were first introduced with >>> KIP-66 (ref 1) in Jun 2017, the KIP mentioned the following: >>> >>>> Criteria: SMTs that are shipped with Kafka Connect should be general >> enough to apply to many data sources & serialization formats. They should >> also be simple enough to not cause any additional library dependency to be >> introduced. >>>> Beyond those being initially included with this KIP, transformations >> can be adopted for inclusion in future with JIRA/ML discussion to weigh the >> tradeoffs. >>> >>> In the 4+ years that we've had SMTs in the project, we've only >>> enhanced the framework with KIP-585 (ref 2), and fixed the initial >>> SMTs (including KIP-437, ref 3). We recently have had quite a few >>> requests to add new SMTs; a few samples of these include: >>> * https://issues.apache.org/jira/browse/KAFKA-10299 >>> * https://issues.apache.org/jira/browse/KAFKA-9436 >>> * https://issues.apache.org/jira/browse/KAFKA-9318 >>> * https://issues.apache.org/jira/browse/KAFKA-12443 >>> >>> Adding new or changing existing SMTs to the Apache Kafka project come >>> with requirements. First, AK releases are infrequent and necessarily >>> involve the entire project. Second, adding an SMT is an API change and >>> therefore requires a KIP. Third, all changes in behavior to SMTs >>> included in an prior AK release must be backward compatible, and >>> adding or changing an SMT's configuration requires a KIP. This last >>> one is also challenging if we're limiting ourselves to truly general >>> SMTs, since these are notoriously difficult to get right the first >>> time. All of these aspects mean that it's difficult to add, maintain, >>> and evolve/improve SMTs in AK. And unless a bug fix is critical, we're >>> likely not to create a patch release for AK just to fix a bug in an >>> SMT, simply because of the effort involved. >>> >>> On the other hand, anyone can easily implement their own SMT and >>> deploy them as a Connect plugin, whether that's part of a connector >>> plugin or a separate plugin dedicated for one or SMTs. Interestingly, >>> it's far simpler to implement and maintain custom SMTs outside of AK, >>> especially since those plugins can be released and deployed in any >>> Connect runtime version since at least 0.11.0. And if custom SMTs are >>> maintained in a relatively small project, they can be released often. >>> >>> Finally, KIP-26 (ref 4) specifically rejected maintaining connector >>> implementations in the AK project. So we have precedence for choosing >>> not to accept implementations. >>> >>> Given the above, I wonder if the time has come for us to prefer only >>> maintaining the SMT framework and existing SMTs, and to decline adding >>> new SMTs. >>> >>> Thoughts? >>> >>> Best regards, >>> >>> Randall Hauch >>> >>> (1) >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-66%3A+Single+Message+Transforms+for+Kafka+Connect >>> (2) >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-585%3A+Filter+and+Conditional+SMTs >>> (3) >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-437%3A+Custom+replacement+for+MaskField+SMT >>> (4) >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=58851767 >>
Re: Do we want to add more SMTs to Apache Kafka?
I like the idea of a select number of SMTs being offered and supported out of the box. The addition of SMTs via this process is nice because it allows for a rich set to be supported out of the box and without the need for extra work to deploy. Perhaps this is a spot where the community could express the interest of additional SMTs which maybe are available via an open source library and if enough usage occurs there could be a path to fold into the Kafka project at large? Brandon Brown > On Nov 7, 2021, at 1:19 PM, Randall Hauch wrote: > > We have had several requests to add more Connect Single Message > Transforms (SMTs) to the project. When SMTs were first introduced with > KIP-66 (ref 1) in Jun 2017, the KIP mentioned the following: > >> Criteria: SMTs that are shipped with Kafka Connect should be general enough >> to apply to many data sources & serialization formats. They should also be >> simple enough to not cause any additional library dependency to be >> introduced. >> Beyond those being initially included with this KIP, transformations can be >> adopted for inclusion in future with JIRA/ML discussion to weigh the >> tradeoffs. > > In the 4+ years that we've had SMTs in the project, we've only > enhanced the framework with KIP-585 (ref 2), and fixed the initial > SMTs (including KIP-437, ref 3). We recently have had quite a few > requests to add new SMTs; a few samples of these include: > * https://issues.apache.org/jira/browse/KAFKA-10299 > * https://issues.apache.org/jira/browse/KAFKA-9436 > * https://issues.apache.org/jira/browse/KAFKA-9318 > * https://issues.apache.org/jira/browse/KAFKA-12443 > > Adding new or changing existing SMTs to the Apache Kafka project come > with requirements. First, AK releases are infrequent and necessarily > involve the entire project. Second, adding an SMT is an API change and > therefore requires a KIP. Third, all changes in behavior to SMTs > included in an prior AK release must be backward compatible, and > adding or changing an SMT's configuration requires a KIP. This last > one is also challenging if we're limiting ourselves to truly general > SMTs, since these are notoriously difficult to get right the first > time. All of these aspects mean that it's difficult to add, maintain, > and evolve/improve SMTs in AK. And unless a bug fix is critical, we're > likely not to create a patch release for AK just to fix a bug in an > SMT, simply because of the effort involved. > > On the other hand, anyone can easily implement their own SMT and > deploy them as a Connect plugin, whether that's part of a connector > plugin or a separate plugin dedicated for one or SMTs. Interestingly, > it's far simpler to implement and maintain custom SMTs outside of AK, > especially since those plugins can be released and deployed in any > Connect runtime version since at least 0.11.0. And if custom SMTs are > maintained in a relatively small project, they can be released often. > > Finally, KIP-26 (ref 4) specifically rejected maintaining connector > implementations in the AK project. So we have precedence for choosing > not to accept implementations. > > Given the above, I wonder if the time has come for us to prefer only > maintaining the SMT framework and existing SMTs, and to decline adding > new SMTs. > > Thoughts? > > Best regards, > > Randall Hauch > > (1) > https://cwiki.apache.org/confluence/display/KAFKA/KIP-66%3A+Single+Message+Transforms+for+Kafka+Connect > (2) > https://cwiki.apache.org/confluence/display/KAFKA/KIP-585%3A+Filter+and+Conditional+SMTs > (3) > https://cwiki.apache.org/confluence/display/KAFKA/KIP-437%3A+Custom+replacement+for+MaskField+SMT > (4) https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=58851767
Re: Spam
Thanks! Brandon Brown > On Jan 5, 2021, at 6:07 PM, Justine Olshan wrote: > > The user has been blocked. https://issues.apache.org/jira/browse/INFRA-21268 > >> On Tue, Jan 5, 2021 at 2:52 PM Brandon Brown >> wrote: >> >> Is there any way to block Tim van der Kooi from making issues? I’m getting >> about 10 new email issues created a minute. >> >> Brandon Brown >> >>
Re: Spam
Is there any way to block Tim van der Kooi from making issues? I’m getting about 10 new email issues created a minute. Brandon Brown
Re: [VOTE] KIP-665 Kafka Connect Hash SMT
I’d like to give this one another friendly bump. If there are no disagreements I can update my existing Pr with the latest KIP changes. Thanks, -Brandon Brandon Brown > On Oct 26, 2020, at 8:29 PM, Brandon Brown wrote: > > I’ve update the KIP with suggestions from Gunnar. I’d like to bring this up > for a vote. > > Brandon Brown >> On Oct 22, 2020, at 12:53 PM, Brandon Brown wrote: >> >> Hey Gunnar, >> >> Those are great questions! >> >> 1) I went with it only selecting top level fields since it seems like that’s >> the way most of the out of the box SMTS work, however I could see a lot of >> value in it supporting nested fields. >> 2) I had not thought about adding salt but I think that would be a valid >> option as well. >> >> I think I’ll update the KIP to reflect those suggestions. One more, do you >> think this should allow a regex for fields or stick with the explicit naming >> of the fields? >> >> Thanks for the great feedback >> >> Brandon Brown >> >>>> On Oct 22, 2020, at 12:40 PM, Gunnar Morling >>>> wrote: >>> >>> Hey Brandon, >>> >>> I think that's an interesting idea, we got something as a built-in >>> connector feature in Debezium, too [1]. Two questions: >>> >>> * Can "field" select nested fields, e.g. "after.email"? >>> * Did you consider an option for specifying salt for the hash functions? >>> >>> --Gunnar >>> >>> [1] >>> https://debezium.io/documentation/reference/connectors/mysql.html#mysql-property-column-mask-hash >>> >>> >>> >>>> Am Do., 22. Okt. 2020 um 12:53 Uhr schrieb Brandon Brown < >>>> bran...@bbrownsound.com>: >>>> >>>> Gonna give this another little bump. :) >>>> >>>> Brandon Brown >>>> >>>>> On Oct 15, 2020, at 12:51 PM, Brandon Brown >>>> wrote: >>>>> >>>>> >>>>> As I mentioned in the KIP, this transformer is slightly different from >>>> the current MaskField SMT. >>>>> >>>>>> Currently there exists a MaskField SMT but that would completely remove >>>> the value by setting it to an equivalent null value. One problem with this >>>> would be that you’d not be able to know in the case of say a password going >>>> through the mask transform it would become "" which could mean that no >>>> password was present in the message, or it was removed. However this hash >>>> transformer would remove this ambiguity if that makes sense. The proposed >>>> hash functions would be MD5, SHA1, SHA256. which are all supported via >>>> MessageDigest. >>>>> >>>>> Given this take on things do you still think there would be value in >>>> this smt? >>>>> >>>>> >>>>> Brandon Brown >>>>>> On Oct 15, 2020, at 12:36 PM, Ning Zhang >>>> wrote: >>>>>> >>>>>> Hello, I think this SMT feature is parallel to >>>> https://docs.confluent.io/current/connect/transforms/index.html >>>>>> >>>>>>>> On 2020/10/15 15:24:51, Brandon Brown >>>> wrote: >>>>>>> Bumping this thread. >>>>>>> Please take a look at the KIP and vote or let me know if you have any >>>> feedback. >>>>>>> >>>>>>> KIP: >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-665%3A+Kafka+Connect+Hash+SMT >>>>>>> >>>>>>> Proposed: https://github.com/apache/kafka/pull/9057 >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Brandon Brown >>>>>>> >>>>>>>>> On Oct 8, 2020, at 10:30 PM, Brandon Brown >>>> wrote: >>>>>>>> >>>>>>>> Just wanted to give another bump on this and see if anyone had any >>>> comments. >>>>>>>> >>>>>>>> Thanks! >>>>>>>> >>>>>>>> Brandon Brown >>>>>>>> >>>>>>>>> On Oct 1, 2020, at 9:11 AM, "bran...@bbrownsound.com" < >>>> bran...@bbrownsound.com> wrote: >>>>>>>>> >>>>>>>>> Hey Kafka Developers, >>>>>>>>> >>>>>>>>> I’ve created the following KIP and updated it based on feedback from >>>> Mickael. I was wondering if we could get a vote on my proposal and move >>>> forward with the proposed pr. >>>>>>>>> >>>>>>>>> Thanks so much! >>>>>>>>> -Brandon >>>>>>> >>>>
Re: Contributing Avro Kafka Connect converter
I could be wrong but according to https://kafka.apache.org/documentation/#connect_running Avro should be supported out of the box. key.converter - Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the keys in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. value.converter - Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. Brandon Brown > On Dec 7, 2020, at 4:36 PM, Ravindra Nath Kakarla > wrote: > > Hi, > > I would like to contribute an Avro converter for Kafka Connect. I described > the approach on the Issue, https://issues.apache.org/jira/browse/KAFKA-10715 > > I am looking for a reviewer who can validate my approach and help commit > the change. Is anyone interested in helping me? > > Thank you!
Re: [DISCUSS] KIP-682: Connect TimestampConverter support for multiple fields and multiple input formats
I love this idea! I have a current KIP for a hash transform and am working adding multi field/nested support to that one. I can think of a few times we’ve needed functionality like that. I’d add that adding support for transforming nested fields would be a great feature for this. -Brandon Brandon Brown > On Nov 7, 2020, at 11:11 AM, Joshua Grisham wrote: > > Hello everyone! > (Thanks to Tom Bentley for directing me in this direction!) > > I have made a series of changes to some of the standard Connect transforms > to meet some of the challenges at my company to consume messages using > Connect, and have been running them for a few weeks now as custom SMTs. > > I realized that several of these changes might actually be really good > features to be included in the standard transforms so I will open some KIPs > to go along with PRs which I already submitted (apologize for going a bit > backwards in the process as this was my first time working with the Kafka > project). > > The first one I have created a KIP for is KIP-682 on the TimestampConverter > transform. > > Basically the 2 limitations that I am proposing to remove are the ability > to only convert one field at a time, but instead convert multiple fields > using a common config, and that if any producer sends a slightly different > date/timestamp string format on any message then the transformation > fails... so to instead add the ability to support variations in the string > input format. > > More info here: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-682%3A+Connect+TimestampConverter+support+for+multiple+fields+and+multiple+input+formats > > I would appreciate any kind of discussion to make improvements and it would > be great to see changes like this being pushed up. > > Thanks for now! > > Best regards, > Joshua Grisham
Re: [VOTE] KIP-665 Kafka Connect Hash SMT
I’ve update the KIP with suggestions from Gunnar. I’d like to bring this up for a vote. Brandon Brown > On Oct 22, 2020, at 12:53 PM, Brandon Brown wrote: > > Hey Gunnar, > > Those are great questions! > > 1) I went with it only selecting top level fields since it seems like that’s > the way most of the out of the box SMTS work, however I could see a lot of > value in it supporting nested fields. > 2) I had not thought about adding salt but I think that would be a valid > option as well. > > I think I’ll update the KIP to reflect those suggestions. One more, do you > think this should allow a regex for fields or stick with the explicit naming > of the fields? > > Thanks for the great feedback > > Brandon Brown > >> On Oct 22, 2020, at 12:40 PM, Gunnar Morling >> wrote: >> >> Hey Brandon, >> >> I think that's an interesting idea, we got something as a built-in >> connector feature in Debezium, too [1]. Two questions: >> >> * Can "field" select nested fields, e.g. "after.email"? >> * Did you consider an option for specifying salt for the hash functions? >> >> --Gunnar >> >> [1] >> https://debezium.io/documentation/reference/connectors/mysql.html#mysql-property-column-mask-hash >> >> >> >>> Am Do., 22. Okt. 2020 um 12:53 Uhr schrieb Brandon Brown < >>> bran...@bbrownsound.com>: >>> >>> Gonna give this another little bump. :) >>> >>> Brandon Brown >>> >>>> On Oct 15, 2020, at 12:51 PM, Brandon Brown >>> wrote: >>>> >>>> >>>> As I mentioned in the KIP, this transformer is slightly different from >>> the current MaskField SMT. >>>> >>>>> Currently there exists a MaskField SMT but that would completely remove >>> the value by setting it to an equivalent null value. One problem with this >>> would be that you’d not be able to know in the case of say a password going >>> through the mask transform it would become "" which could mean that no >>> password was present in the message, or it was removed. However this hash >>> transformer would remove this ambiguity if that makes sense. The proposed >>> hash functions would be MD5, SHA1, SHA256. which are all supported via >>> MessageDigest. >>>> >>>> Given this take on things do you still think there would be value in >>> this smt? >>>> >>>> >>>> Brandon Brown >>>>> On Oct 15, 2020, at 12:36 PM, Ning Zhang >>> wrote: >>>>> >>>>> Hello, I think this SMT feature is parallel to >>> https://docs.confluent.io/current/connect/transforms/index.html >>>>> >>>>>>> On 2020/10/15 15:24:51, Brandon Brown >>> wrote: >>>>>> Bumping this thread. >>>>>> Please take a look at the KIP and vote or let me know if you have any >>> feedback. >>>>>> >>>>>> KIP: >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-665%3A+Kafka+Connect+Hash+SMT >>>>>> >>>>>> Proposed: https://github.com/apache/kafka/pull/9057 >>>>>> >>>>>> Thanks >>>>>> >>>>>> Brandon Brown >>>>>> >>>>>>>> On Oct 8, 2020, at 10:30 PM, Brandon Brown >>> wrote: >>>>>>> >>>>>>> Just wanted to give another bump on this and see if anyone had any >>> comments. >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> Brandon Brown >>>>>>> >>>>>>>> On Oct 1, 2020, at 9:11 AM, "bran...@bbrownsound.com" < >>> bran...@bbrownsound.com> wrote: >>>>>>>> >>>>>>>> Hey Kafka Developers, >>>>>>>> >>>>>>>> I’ve created the following KIP and updated it based on feedback from >>> Mickael. I was wondering if we could get a vote on my proposal and move >>> forward with the proposed pr. >>>>>>>> >>>>>>>> Thanks so much! >>>>>>>> -Brandon >>>>>> >>>
Re: [VOTE] KIP-665 Kafka Connect Hash SMT
Hey Gunnar, Those are great questions! 1) I went with it only selecting top level fields since it seems like that’s the way most of the out of the box SMTS work, however I could see a lot of value in it supporting nested fields. 2) I had not thought about adding salt but I think that would be a valid option as well. I think I’ll update the KIP to reflect those suggestions. One more, do you think this should allow a regex for fields or stick with the explicit naming of the fields? Thanks for the great feedback Brandon Brown > On Oct 22, 2020, at 12:40 PM, Gunnar Morling > wrote: > > Hey Brandon, > > I think that's an interesting idea, we got something as a built-in > connector feature in Debezium, too [1]. Two questions: > > * Can "field" select nested fields, e.g. "after.email"? > * Did you consider an option for specifying salt for the hash functions? > > --Gunnar > > [1] > https://debezium.io/documentation/reference/connectors/mysql.html#mysql-property-column-mask-hash > > > >> Am Do., 22. Okt. 2020 um 12:53 Uhr schrieb Brandon Brown < >> bran...@bbrownsound.com>: >> >> Gonna give this another little bump. :) >> >> Brandon Brown >> >>> On Oct 15, 2020, at 12:51 PM, Brandon Brown >> wrote: >>> >>> >>> As I mentioned in the KIP, this transformer is slightly different from >> the current MaskField SMT. >>> >>>> Currently there exists a MaskField SMT but that would completely remove >> the value by setting it to an equivalent null value. One problem with this >> would be that you’d not be able to know in the case of say a password going >> through the mask transform it would become "" which could mean that no >> password was present in the message, or it was removed. However this hash >> transformer would remove this ambiguity if that makes sense. The proposed >> hash functions would be MD5, SHA1, SHA256. which are all supported via >> MessageDigest. >>> >>> Given this take on things do you still think there would be value in >> this smt? >>> >>> >>> Brandon Brown >>>> On Oct 15, 2020, at 12:36 PM, Ning Zhang >> wrote: >>>> >>>> Hello, I think this SMT feature is parallel to >> https://docs.confluent.io/current/connect/transforms/index.html >>>> >>>>>> On 2020/10/15 15:24:51, Brandon Brown >> wrote: >>>>> Bumping this thread. >>>>> Please take a look at the KIP and vote or let me know if you have any >> feedback. >>>>> >>>>> KIP: >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-665%3A+Kafka+Connect+Hash+SMT >>>>> >>>>> Proposed: https://github.com/apache/kafka/pull/9057 >>>>> >>>>> Thanks >>>>> >>>>> Brandon Brown >>>>> >>>>>>> On Oct 8, 2020, at 10:30 PM, Brandon Brown >> wrote: >>>>>> >>>>>> Just wanted to give another bump on this and see if anyone had any >> comments. >>>>>> >>>>>> Thanks! >>>>>> >>>>>> Brandon Brown >>>>>> >>>>>>> On Oct 1, 2020, at 9:11 AM, "bran...@bbrownsound.com" < >> bran...@bbrownsound.com> wrote: >>>>>>> >>>>>>> Hey Kafka Developers, >>>>>>> >>>>>>> I’ve created the following KIP and updated it based on feedback from >> Mickael. I was wondering if we could get a vote on my proposal and move >> forward with the proposed pr. >>>>>>> >>>>>>> Thanks so much! >>>>>>> -Brandon >>>>> >>
Re: [VOTE] KIP-665 Kafka Connect Hash SMT
Gonna give this another little bump. :) Brandon Brown > On Oct 15, 2020, at 12:51 PM, Brandon Brown wrote: > > > As I mentioned in the KIP, this transformer is slightly different from the > current MaskField SMT. > >> Currently there exists a MaskField SMT but that would completely remove the >> value by setting it to an equivalent null value. One problem with this would >> be that you’d not be able to know in the case of say a password going >> through the mask transform it would become "" which could mean that no >> password was present in the message, or it was removed. However this hash >> transformer would remove this ambiguity if that makes sense. The proposed >> hash functions would be MD5, SHA1, SHA256. which are all supported via >> MessageDigest. > > Given this take on things do you still think there would be value in this smt? > > > Brandon Brown >> On Oct 15, 2020, at 12:36 PM, Ning Zhang wrote: >> >> Hello, I think this SMT feature is parallel to >> https://docs.confluent.io/current/connect/transforms/index.html >> >>>> On 2020/10/15 15:24:51, Brandon Brown wrote: >>> Bumping this thread. >>> Please take a look at the KIP and vote or let me know if you have any >>> feedback. >>> >>> KIP: >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-665%3A+Kafka+Connect+Hash+SMT >>> >>> Proposed: https://github.com/apache/kafka/pull/9057 >>> >>> Thanks >>> >>> Brandon Brown >>> >>>>> On Oct 8, 2020, at 10:30 PM, Brandon Brown >>>>> wrote: >>>> >>>> Just wanted to give another bump on this and see if anyone had any >>>> comments. >>>> >>>> Thanks! >>>> >>>> Brandon Brown >>>> >>>>> On Oct 1, 2020, at 9:11 AM, "bran...@bbrownsound.com" >>>>> wrote: >>>>> >>>>> Hey Kafka Developers, >>>>> >>>>> I’ve created the following KIP and updated it based on feedback from >>>>> Mickael. I was wondering if we could get a vote on my proposal and move >>>>> forward with the proposed pr. >>>>> >>>>> Thanks so much! >>>>> -Brandon >>>
Re: [VOTE] KIP-676: Respect the logging hierarchy
+1 (non binding) Brandon Brown > On Oct 22, 2020, at 5:03 AM, Mickael Maison wrote: > > +1 (binding) > Thanks > >> On Sun, Oct 11, 2020 at 8:09 AM Dongjin Lee wrote: >> >> Binding: 2 (John, Gwen) >> Non-binding: 1 (Dongjin) >> >> We need one more binding now. >> >> Regards, >> Dongjin >> >>> On Sun, Oct 11, 2020 at 3:05 PM Gwen Shapira wrote: >>> >>> +1 (binding) >>> >>> Makes sense, thank you. >>> >>>> On Fri, Oct 9, 2020, 2:50 AM Tom Bentley wrote: >>> >>>> Hi all, >>>> >>>> KIP-676 is pretty trivial and the comments on the discussion thread seem >>> to >>>> be favourable, so I'd like to start a vote on it. >>>> >>>> >>>> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-676%3A+Respect+logging+hierarchy >>>> < >>>> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-676%3A+Respect+logging+hierarchy?moved=true >>>>> >>>> >>>> Please take a look if you have time. >>>> >>>> Many thanks, >>>> >>>> Tom >>>> >>> >> >> >> -- >> *Dongjin Lee* >> >> *A hitchhiker in the mathematical world.* >> >> >> >> >> *github: <http://goog_969573159/>github.com/dongjinleekr >> <https://github.com/dongjinleekr>keybase: https://keybase.io/dongjinleekr >> <https://keybase.io/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr >> <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: speakerdeck.com/dongjin >> <https://speakerdeck.com/dongjin>*
Re: [DISCUSS] Checkstyle Import Order
I like option 2. Brandon Brown > On Oct 15, 2020, at 1:36 AM, Dongjin Lee wrote: > > Hello. I hope to open a discussion about the import order in Java code. > > As Nikolay stated recently[^1], Kafka uses a relatively strict code style > for Java code. However, it misses any rule on import order. For this > reason, the code formatting settings of every local dev environment are > different from person to person, resulting in the countless meaningless > import order changes in the PR. > > For example, in `NamedCache.java` in the streams module, the `java.*` > imports are split into two chunks, embracing the other imports between > them. So, I propose to define an import order to prevent these kinds of > cases in the future. > > To define the import order, we have to regard the following three > orthogonal issues beforehand: > > a. How to group the type imports? > b. Whether to sort the imports alphabetically? > c. Where to place static imports: above the type imports, or below them. > > Since b and c seem relatively straightforward (sort the imports > alphabetically and place the static imports below the type imports), I hope > to focus the available alternatives on the problem a. > > I evaluated the following alternatives and checked how many files are get > effected for each case. (based on commit 1457cc652) And here are the > results: > > *1. kafka, org.apache.kafka, *, javax, java (5 groups): 1222 files.* > > ``` > > value="kafka,/^org\.apache\.kafka.*$/,*,javax,java"/> > > > > > > ``` > > *2. (kafka|org.apache.kafka), *, javax? (3 groups): 968 files.* > > ``` > > > > > > > > ``` > > *3. (kafka|org.apache.kafka), *, javax, java (4 groups): 533 files.* > > ``` > > value="(kafka|org\.apache\.kafka),*,javax,java"/> > > > > > > ``` > > *4. *, javax? (2 groups): 707 files.* > > ``` > > > > > > > > ``` > > *5. javax?, * (2 groups): 1822 files.* > > ``` > > > > > > > > ``` > > *6. java, javax, * (3 groups): 1809 files.* > > ``` > > > > > > > > ``` > > I hope to get some feedback on this issue here. > > For the WIP PR, please refer here: https://github.com/apache/kafka/pull/8404 > > Best, > Dongjin > > [^1]: > https://lists.apache.org/thread.html/r2bbee24b8a459842a0fc840c6e40958e7158d29f3f2d6c0d223be80b%40%3Cdev.kafka.apache.org%3E > [^2]: > https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/state/internals/NamedCache.java > > -- > *Dongjin Lee* > > *A hitchhiker in the mathematical world.* > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > <https://github.com/dongjinleekr>keybase: https://keybase.io/dongjinleekr > <https://keybase.io/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: speakerdeck.com/dongjin > <https://speakerdeck.com/dongjin>*
Re: [VOTE] KIP-665 Kafka Connect Hash SMT
As I mentioned in the KIP, this transformer is slightly different from the current MaskField SMT. > Currently there exists a MaskField SMT but that would completely remove the > value by setting it to an equivalent null value. One problem with this would > be that you’d not be able to know in the case of say a password going through > the mask transform it would become "" which could mean that no password was > present in the message, or it was removed. However this hash transformer > would remove this ambiguity if that makes sense. The proposed hash functions > would be MD5, SHA1, SHA256. which are all supported via MessageDigest. Given this take on things do you still think there would be value in this smt? Brandon Brown > On Oct 15, 2020, at 12:36 PM, Ning Zhang wrote: > > Hello, I think this SMT feature is parallel to > https://docs.confluent.io/current/connect/transforms/index.html > >> On 2020/10/15 15:24:51, Brandon Brown wrote: >> Bumping this thread. >> Please take a look at the KIP and vote or let me know if you have any >> feedback. >> >> KIP: >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-665%3A+Kafka+Connect+Hash+SMT >> >> Proposed: https://github.com/apache/kafka/pull/9057 >> >> Thanks >> >> Brandon Brown >> >>>> On Oct 8, 2020, at 10:30 PM, Brandon Brown wrote: >>> >>> Just wanted to give another bump on this and see if anyone had any >>> comments. >>> >>> Thanks! >>> >>> Brandon Brown >>> >>>> On Oct 1, 2020, at 9:11 AM, "bran...@bbrownsound.com" >>>> wrote: >>>> >>>> Hey Kafka Developers, >>>> >>>> I’ve created the following KIP and updated it based on feedback from >>>> Mickael. I was wondering if we could get a vote on my proposal and move >>>> forward with the proposed pr. >>>> >>>> Thanks so much! >>>> -Brandon >>
Re: [VOTE] KIP-661: Expose task configurations in Connect REST API
I like this proposal a lot! +1 by me. Brandon Brown > On Oct 15, 2020, at 11:12 AM, Mickael Maison wrote: > > Bumping this thread. > Please take a look at the KIP and vote or let me know if you have any > feedback. > Thanks > >> On Thu, Sep 24, 2020 at 4:53 PM Mickael Maison >> wrote: >> >> Hi, >> >> I'd like to start a vote on KIP-661: >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-661%3A+Expose+task+configurations+in+Connect+REST+API >> >> Thanks
Re: [VOTE] KIP-665 Kafka Connect Hash SMT
Bumping this thread. Please take a look at the KIP and vote or let me know if you have any feedback. KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-665%3A+Kafka+Connect+Hash+SMT Proposed: https://github.com/apache/kafka/pull/9057 Thanks Brandon Brown > On Oct 8, 2020, at 10:30 PM, Brandon Brown wrote: > > Just wanted to give another bump on this and see if anyone had any comments. > > Thanks! > > Brandon Brown > >> On Oct 1, 2020, at 9:11 AM, "bran...@bbrownsound.com" >> wrote: >> >> Hey Kafka Developers, >> >> I’ve created the following KIP and updated it based on feedback from >> Mickael. I was wondering if we could get a vote on my proposal and move >> forward with the proposed pr. >> >> Thanks so much! >> -Brandon
Re: [DISCUSSION] python code style checks
I love that idea, black is a really great formatter (opinionated with minimal config). Brandon Brown > On Oct 9, 2020, at 5:55 AM, Nikolay Izhikov wrote: > > Hello! > > Kafka uses relatively strict code style for Java code. > Code style enforces during project build. > > But, for now, we doesn’t check python test code style. > I’ve checked system tests code with the default pylint settings and got the > following results - "Your code has been rated at 5.98/10» > > I propose to add python code style checks to the codebase and process and fix > code style issues. > > What do you think?
Re: [VOTE] KIP-665 Kafka Connect Hash SMT
Just wanted to give another bump on this and see if anyone had any comments. Thanks! Brandon Brown > On Oct 1, 2020, at 9:11 AM, "bran...@bbrownsound.com" > wrote: > > Hey Kafka Developers, > > I’ve created the following KIP and updated it based on feedback from Mickael. > I was wondering if we could get a vote on my proposal and move forward with > the proposed pr. > > Thanks so much! > -Brandon
Re: [DISCUSS] KIP-665 Kafka Connect Hash SMT
Hi Tom, The reason I went fix was so that we could simplify the configuration for example you can say sha256 instead of having to remember that it’s SHA-256. Admittedly if other formats become implemented then it would require updating this as well. I’m flexible on changing it to a string and letting it be configured with the exact name. What do you think Mickael? Brandon Brown > On Sep 21, 2020, at 3:42 AM, Tom Bentley wrote: > > Hi Brandon and Mickael, > > Is it necessary to fix the supported digest? We could just support whatever > the JVM's MessageDigest supports? > > Kind regards, > > Tom > >> On Fri, Sep 18, 2020 at 6:00 PM Brandon Brown >> wrote: >> >> Thanks Michael! So proposed hash functions would be MD5, SHA1, SHA256. >> >> I can expand the motivation on the KIP but here’s where my head is at. >> MaskField would completely remove the value by setting it to an equivalent >> null value. One problem with this would be that you’d not be able to know >> in the case of say a password going through the mask transform it would >> become “” which could mean that no password was present in the message, or >> it was removed. However this hash transformer would remove this ambiguity >> if that makes sense. >> >> Do you think there are other hash functions that should be supported as >> well? >> >> Thanks, >> Brandon Brown >> >>> On Sep 18, 2020, at 12:00 PM, Mickael Maison >> wrote: >>> >>> Thanks Brandon for the KIP. >>> >>> There's already a built-in transformation (MaskField) that can >>> obfuscate fields. In the motivation section, it would be nice to >>> explain the use cases when MaskField is not suitable and when users >>> would need the proposed transformation. >>> >>> The KIP exposes a "function" configuration to select the hash function >>> to use. Which hash functions do you propose supporting? >>> >>>> On Thu, Aug 27, 2020 at 10:43 PM wrote: >>>> >>>> >>>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-665%3A+Kafka+Connect+Hash+SMT >>>> >>>> The current pr with the proposed changes >>>> https://github.com/apache/kafka/pull/9057 and the original 3rd party >>>> contribution which initiated this change >>>> >> https://github.com/aiven/aiven-kafka-connect-transforms/issues/9#issuecomment-662378057 >> . >>>> >>>> I'm interested in any suggestions for ways to improve this as I think >>>> it would make a nice addition to the existing SMTs provided by Kafka >>>> Connect out of the box. >>>> >>>> Thanks, >>>> Brandon >>>> >>>> >>>> >>
Re: [DISCUSS] KIP-665 Kafka Connect Hash SMT
Thanks Michael! So proposed hash functions would be MD5, SHA1, SHA256. I can expand the motivation on the KIP but here’s where my head is at. MaskField would completely remove the value by setting it to an equivalent null value. One problem with this would be that you’d not be able to know in the case of say a password going through the mask transform it would become “” which could mean that no password was present in the message, or it was removed. However this hash transformer would remove this ambiguity if that makes sense. Do you think there are other hash functions that should be supported as well? Thanks, Brandon Brown > On Sep 18, 2020, at 12:00 PM, Mickael Maison wrote: > > Thanks Brandon for the KIP. > > There's already a built-in transformation (MaskField) that can > obfuscate fields. In the motivation section, it would be nice to > explain the use cases when MaskField is not suitable and when users > would need the proposed transformation. > > The KIP exposes a "function" configuration to select the hash function > to use. Which hash functions do you propose supporting? > >> On Thu, Aug 27, 2020 at 10:43 PM wrote: >> >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-665%3A+Kafka+Connect+Hash+SMT >> >> The current pr with the proposed changes >> https://github.com/apache/kafka/pull/9057 and the original 3rd party >> contribution which initiated this change >> https://github.com/aiven/aiven-kafka-connect-transforms/issues/9#issuecomment-662378057. >> >> I'm interested in any suggestions for ways to improve this as I think >> it would make a nice addition to the existing SMTs provided by Kafka >> Connect out of the box. >> >> Thanks, >> Brandon >> >> >>
[DISCUSS] KIP-665 Kafka Connect Hash SMT
Hey Everybody I wondered if I could get some feedback on this. In a recent discussion it was brought up that there might be value in supporting the ability to hash multiple fields on a value message. Was wondering what y’all think and also if there’s any feedback on the current proposed KIP and the sample PR I’ve submitted. Thanks! -Brandon Brown
[DISCUSS] KIP-665 Kafka Connect Hash SMT
Hey everybody, I’ve created the following and would love some feedback. One place where this could be of use would be to say hashing the key used as an identifier for inserting into elasticsearch (which has a size limit) or obfuscating sensitive values like say passwords or ssn. https://cwiki.apache.org/confluence/display/KAFKA/KIP-665%3A+Kafka+Connect+Hash+SMT The current pr with the proposed changes https://github.com/apache/kafka/pull/9057 and the original 3rd party contribution which initiated this change https://github.com/aiven/aiven-kafka-connect-transforms/issues/9#issuecomment-662378057. I'm interested in any suggestions for ways to improve this as I think it would make a nice addition to the existing SMTs provided by Kafka Connect out of the box. Thanks, Brandon
Re: Requesting permission
Thanks! I was able to make my first KIP. -Brandon Brandon Brown (240) 506-8335 (m) > On Aug 27, 2020, at 4:19 PM, Jun Rao wrote: > > Hi, Brandon, > > Thanks for your interest. Just gave you the wiki permission. > > Jun > >> On Thu, Aug 27, 2020 at 1:17 PM Brandon Brown >> wrote: >> >> I’d like to request permission to create a KIP. My username is brbrown35. >> >> Thanks, >> Brandon >> >> Brandon Brown >> (240) 506-8335 (m)
Requesting permission
I’d like to request permission to create a KIP. My username is brbrown35. Thanks, Brandon Brandon Brown (240) 506-8335 (m)
[jira] [Created] (KAFKA-10299) Add a Hash SMT transformer
Brandon Brown created KAFKA-10299: - Summary: Add a Hash SMT transformer Key: KAFKA-10299 URL: https://issues.apache.org/jira/browse/KAFKA-10299 Project: Kafka Issue Type: New Feature Components: KafkaConnect Reporter: Brandon Brown A previous contribution to [https://github.com/aiven/aiven-kafka-connect-transforms] was suggested as by a member of confluent as being a nice addition to the out of the box Kafka Connect SMTs. The discussion is here [https://github.com/aiven/aiven-kafka-connect-transforms/issues/9#issuecomment-662378057] This change would add a new SMT which allows for either a hashing a key or a value using the configured hashing function. -- This message was sent by Atlassian Jira (v8.3.4#803005)
