[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15842359#comment-15842359
 ] 

ASF GitHub Bot commented on KAFKA-3209:
---

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/2458

KAFKA-3209: KIP-66: Flatten and Cast single message transforms



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka kafka-3209-even-more-transforms

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2458.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2458


commit 14692133dc80552410b2b0eac76b9fca8a73afe1
Author: Ewen Cheslack-Postava 
Date:   2017-01-27T07:26:02Z

KAFKA-3209: KIP-66: Flatten and Cast single message transforms




> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>Assignee: Ewen Cheslack-Postava
>Priority: Blocker
>  Labels: needs-kip
> Fix For: 0.10.2.0
>
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15832655#comment-15832655
 ] 

ASF GitHub Bot commented on KAFKA-3209:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2374


> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>Assignee: Shikhar Bhushan
>  Labels: needs-kip
> Fix For: 0.10.2.0
>
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822420#comment-15822420
 ] 

ASF GitHub Bot commented on KAFKA-3209:
---

GitHub user shikhar opened a pull request:

https://github.com/apache/kafka/pull/2374

KAFKA-3209: KIP-66: more single message transforms

WIP, in this PR I'd also like to add doc generation for transformations.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shikhar/kafka more-smt

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2374.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2374


commit f34cc71c9931ea7ec5dd045512c623196928a2a3
Author: Shikhar Bhushan 
Date:   2017-01-13T20:00:31Z

SetSchemaMetadata SMT

commit 09af66b3f1861bf2a0718aaf79ca1a3bd22adcec
Author: Shikhar Bhushan 
Date:   2017-01-13T21:44:57Z

Support schemaless and rename HoistToStruct->Hoist; add the inverse Extract 
transform

commit 022f4920c5f09d068bbf49e47091a1333dc48be2
Author: Shikhar Bhushan 
Date:   2017-01-13T21:51:43Z

InsertField transform -- fix bad name for interface containing config name 
constants

commit c5260a718e2f0ade66c4607a4b9c21abda61b90c
Author: Shikhar Bhushan 
Date:   2017-01-13T22:01:25Z

ValueToKey SMT




> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>Assignee: Shikhar Bhushan
>  Labels: needs-kip
> Fix For: 0.10.2.0
>
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795937#comment-15795937
 ] 

ASF GitHub Bot commented on KAFKA-3209:
---

GitHub user shikhar opened a pull request:

https://github.com/apache/kafka/pull/2299

KAFKA-3209: KIP-66: single message transforms

*WIP* particularly around testing

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shikhar/kafka smt-2017

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2299.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2299


commit 1178670f36d8fbdf5cbdbb2728ace7bf4f0e7300
Author: Shikhar Bhushan 
Date:   2017-01-03T19:21:17Z

KAFKA-3209: KIP-66: single message transforms




> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>Assignee: Shikhar Bhushan
>  Labels: needs-kip
> Fix For: 0.10.2.0
>
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[Nisarg Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743641#comment-15743641
 ] 

Nisarg Shah commented on KAFKA-3209:


Yeah sorry about that, I got caught up with school work. You would have a 
better depth of understanding of this problem as I haven't used this part in a 
professional environment. I would gladly collaborate with you. :) 

> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>Assignee: Shikhar Bhushan
>  Labels: needs-kip
> Fix For: 0.10.2.0
>
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[Shikhar Bhushan (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743300#comment-15743300
 ] 

Shikhar Bhushan commented on KAFKA-3209:


Thanks [~snisarg]. I self-assigned it as I don't believe you were actively 
working on it and the ticket was unassigned, but I'd be happy to collaborate if 
you have time.

Yes, I think we should continue the work by updating KIP-66. I'm working on 
drafting the proposal into KIP form and I'll send a ML update when it's ready.

> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>Assignee: Shikhar Bhushan
>  Labels: needs-kip
> Fix For: 0.10.2.0
>
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[Nisarg Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743278#comment-15743278
 ] 

Nisarg Shah commented on KAFKA-3209:


[~shikhar] the proposal looks details and flexible in terms of what it can do. 
I see you've assigned this to yourself, would you also like to update KIP-66?

> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>Assignee: Shikhar Bhushan
>  Labels: needs-kip
> Fix For: 0.10.2.0
>
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[Shikhar Bhushan (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729737#comment-15729737
 ] 

Shikhar Bhushan commented on KAFKA-3209:


[~snisarg] and [~jjchorrobe], I revived the discussion thread and I'd welcome 
your thoughts on there about this proposal: 
https://cwiki.apache.org/confluence/display/KAFKA/Connect+Transforms+-+Proposed+Design

> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>  Labels: needs-kip
> Fix For: 0.10.2.0
>
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[Nisarg Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674120#comment-15674120
 ] 

Nisarg Shah commented on KAFKA-3209:


Hey Juan. I have started working on this, have only been busy lately and will 
start again within a few weeks. You could help though. Have a look at the KIP 
(link above) and let's discuss suggestions and implementations. 

> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>  Labels: needs-kip
> Fix For: 0.10.2.0
>
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[Juan Jesus Chorro (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673513#comment-15673513
 ] 

Juan Jesus Chorro commented on KAFKA-3209:
--

Hi! I'm interested in this feature and I would like to know if anybody is 
working in this issue. 

> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>  Labels: needs-kip
> Fix For: 0.10.2.0
>
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[Nisarg Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371610#comment-15371610
 ] 

Nisarg Shah commented on KAFKA-3209:


Please give your comments and suggestions here
https://cwiki.apache.org/confluence/display/KAFKA/KIP-66%3A+Add+Kafka+Connect+Transformers+to+allow+transformations+to+messages


> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>Priority: Critical
>  Labels: needs-kip
> Fix For: 0.10.1.0
>
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[Nisarg Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15329669#comment-15329669
 ] 

Nisarg Shah commented on KAFKA-3209:


I'm still working on it. Got a little busy. I thought I'll get a working 
prototype ready so that it can be demonstrated and it's easier to convey how 
it'll work. 
I have it almost ready, working out small kinks of sending properties down the 
transformers behind the scenes. I will be sending out mails to the dev channel 
soon. 

> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328647#comment-15328647
 ] 

Ewen Cheslack-Postava commented on KAFKA-3209:
--

[~pablocasares] I don't think anybody is actively working on this -- Nisarg had 
started but I'm not sure if they have dropped work on it for now since we 
haven't seen anything for awhile. First step would be to sketch out a design 
and put together a KIP. We'll want to make sure the use cases and how the 
proposed interfaces work for them is clearly documented. Especially important 
will be discussion around configuration -- how will transforms be configured in 
the Connector configs, possibly allowing for multiple transformations while 
keeping configuration easy? Additionally, if we are going to ship anything with 
Kafka to provide very commonly needed transformations (dropping & renaming 
fields, flattening hierarchical structures, filters), we'll probably want to 
include a full description of that set in the KIP as well. (Alternatively, we 
might not ship any, but then they obviously aren't as useful out of the box -- 
like connectors and converters, you'll need to pull in separate jars for 
transforms). I imagine the implementation will be pretty straightforward once 
we've got it specced out.

> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[Pablo Casares (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327225#comment-15327225
 ] 

Pablo Casares commented on KAFKA-3209:
--

Hey, I'm interested in produce keys based on json fields. I was reading 
https://groups.google.com/forum/#!topic/confluent-platform/aVaqBtMiKkY and they 
suggest this improvement. I think that this can help me. I can contribute to 
this improvement if you want. What is the current status of this task?.

Regards! 

Pablo

> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269148#comment-15269148
 ] 

Ewen Cheslack-Postava commented on KAFKA-3209:
--

To help clarify [~Skandragon]'s comment a bit, the idea is that the records are 
going to be small compared to the headers. This means that the approach we 
might normally suggest -- doing the flatMap transformation with an application 
or stream processor, storing that data back to Kafka, then using Connect to 
store the data to another system -- will have very high overhead.

Whereas most of the message transforms we've discussed so far are either simple 
map() or filter() transformations, this is a case where we might want to 
generate multiple output messages from a single input message. The API for 
supporting this is obviously straightforward -- just support returning a list 
of messages from the transformation instead of a single message. However, I 
think the main challenge is that message offsets either aren't unique anymore 
or we'd need to extend the concept of offset to account for "sub-messages".

> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[Michael Graff (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268959#comment-15268959
 ] 

Michael Graff commented on KAFKA-3209:
--

We have a use case which is in effect "take a single Kafka input message and 
produce 1 or more output items" -- I understand people refer to this as a 
"flatMap" or something like that.

Specifically, our use case is to get a chunk formatted like this:

Header-1: value1
Header-2: value2

record1
record2
record3
...
records

Our transformation is to take some header fields and merge them into a record, 
then emit that record for all (or some) records in the message.


> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[Nisarg Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264644#comment-15264644
 ] 

Nisarg Shah commented on KAFKA-3209:


That does sound good. I realise it is not insanely complicated, but since I 
haven't done much in terms of open source, I thought I'll start with something 
like this. I also think this is useful nonetheless. 

> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264633#comment-15264633
 ] 

Ewen Cheslack-Postava commented on KAFKA-3209:
--

[~snisarg] I haven't started work in earnest. This JIRA may not end up being 
particularly complicated code-wise, but given that it will be new public API, 
the impact it can have, and that we want to make sure all transformations we 
want to support will work with the implementation, it'll need a KIP proposal 
and discussion. (Though an initial prototype patch might also help drive that 
discussion.)

If you're interested in picking this up, I'd be happy to guide you through the 
process of writing up the KIP.

> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[Gwen Shapira (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264627#comment-15264627
 ] 

Gwen Shapira commented on KAFKA-3209:
-

Thank you Nisarg. I believe [~ewencp] already started working on this one...

> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[Nisarg Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264548#comment-15264548
 ] 

Nisarg Shah commented on KAFKA-3209:


Hey, I'm looking to contribute to open source and want to take this up. :) 

> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)