[jira] [Commented] (IGNITE-4140) KafkaStreamer should use tuple extractor instead of decoders
[
https://issues.apache.org/jira/browse/IGNITE-4140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15768128#comment-15768128
]
Denis Magda commented on IGNITE-4140:
-
[~roman_s], presently the community has no plans for 1.9 release. So these
changes will be available in the master and 2.0.
> KafkaStreamer should use tuple extractor instead of decoders
>
>
> Key: IGNITE-4140
> URL: https://issues.apache.org/jira/browse/IGNITE-4140
> Project: Ignite
> Issue Type: Improvement
> Components: streaming
>Affects Versions: 1.7
>Reporter: Valentin Kulichenko
>Assignee: Anil
> Labels: patch-available
> Fix For: 1.9
>
>
> Current design of {{KafkaStreamer}} looks incorrect to me. In particular, it
> extends {{StreamAdapter}}, but ignores tuple extractors provided there and
> uses native Kafka decoders instead. This for example makes impossible to
> produce several entries from one message, like it can be done via
> {{StreamMultipleTupleExtractor}} in other streamers.
> To fix this, we should:
> # Declare the {{KafkaStreamer}} like this:
> {code}
> KafkaStreamer extends StreamAdapter,
> K, V>
> {code}
> # Remove {{keyDecoder}} and {{valDecoder}} in favor of tuple extractors.
> # Instead of doing {{getStreamer().addData(...)}} directly, call
> {{addMessage(...)}} method providing the raw message consumed from Kafka
> ({{MessageAndMetadata}}). This method will make sure that
> configured extractor is invoked and that all entries are added to
> {{IgniteDataStreamer}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (IGNITE-4140) KafkaStreamer should use tuple extractor instead of decoders
[
https://issues.apache.org/jira/browse/IGNITE-4140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730440#comment-15730440
]
Roman Shtykh commented on IGNITE-4140:
--
@avinogradov My concern is -- we create the next 1.9 release from the master
branch and this modification will go into 1.9 release instead of 2.0. Is it a
valid concern?
> KafkaStreamer should use tuple extractor instead of decoders
>
>
> Key: IGNITE-4140
> URL: https://issues.apache.org/jira/browse/IGNITE-4140
> Project: Ignite
> Issue Type: Improvement
> Components: streaming
>Affects Versions: 1.7
>Reporter: Valentin Kulichenko
>Assignee: Anil
> Labels: patch-available
> Fix For: 1.9
>
>
> Current design of {{KafkaStreamer}} looks incorrect to me. In particular, it
> extends {{StreamAdapter}}, but ignores tuple extractors provided there and
> uses native Kafka decoders instead. This for example makes impossible to
> produce several entries from one message, like it can be done via
> {{StreamMultipleTupleExtractor}} in other streamers.
> To fix this, we should:
> # Declare the {{KafkaStreamer}} like this:
> {code}
> KafkaStreamer extends StreamAdapter,
> K, V>
> {code}
> # Remove {{keyDecoder}} and {{valDecoder}} in favor of tuple extractors.
> # Instead of doing {{getStreamer().addData(...)}} directly, call
> {{addMessage(...)}} method providing the raw message consumed from Kafka
> ({{MessageAndMetadata}}). This method will make sure that
> configured extractor is invoked and that all entries are added to
> {{IgniteDataStreamer}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (IGNITE-4140) KafkaStreamer should use tuple extractor instead of decoders
[
https://issues.apache.org/jira/browse/IGNITE-4140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729068#comment-15729068
]
Anil commented on IGNITE-4140:
--
Thank you [~avinogradov] and [~vkulichenko]
> KafkaStreamer should use tuple extractor instead of decoders
>
>
> Key: IGNITE-4140
> URL: https://issues.apache.org/jira/browse/IGNITE-4140
> Project: Ignite
> Issue Type: Improvement
> Components: streaming
>Affects Versions: 1.7
>Reporter: Valentin Kulichenko
>Assignee: Anil
> Labels: patch-available
> Fix For: 1.9
>
>
> Current design of {{KafkaStreamer}} looks incorrect to me. In particular, it
> extends {{StreamAdapter}}, but ignores tuple extractors provided there and
> uses native Kafka decoders instead. This for example makes impossible to
> produce several entries from one message, like it can be done via
> {{StreamMultipleTupleExtractor}} in other streamers.
> To fix this, we should:
> # Declare the {{KafkaStreamer}} like this:
> {code}
> KafkaStreamer extends StreamAdapter,
> K, V>
> {code}
> # Remove {{keyDecoder}} and {{valDecoder}} in favor of tuple extractors.
> # Instead of doing {{getStreamer().addData(...)}} directly, call
> {{addMessage(...)}} method providing the raw message consumed from Kafka
> ({{MessageAndMetadata}}). This method will make sure that
> configured extractor is invoked and that all entries are added to
> {{IgniteDataStreamer}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (IGNITE-4140) KafkaStreamer should use tuple extractor instead of decoders
[
https://issues.apache.org/jira/browse/IGNITE-4140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15728199#comment-15728199
]
Anton Vinogradov commented on IGNITE-4140:
--
Anil,
Thaks for contribution!
Changes merged to master branch.
> KafkaStreamer should use tuple extractor instead of decoders
>
>
> Key: IGNITE-4140
> URL: https://issues.apache.org/jira/browse/IGNITE-4140
> Project: Ignite
> Issue Type: Improvement
> Components: streaming
>Affects Versions: 1.7
>Reporter: Valentin Kulichenko
>Assignee: Anil
> Labels: patch-available
> Fix For: 1.9
>
>
> Current design of {{KafkaStreamer}} looks incorrect to me. In particular, it
> extends {{StreamAdapter}}, but ignores tuple extractors provided there and
> uses native Kafka decoders instead. This for example makes impossible to
> produce several entries from one message, like it can be done via
> {{StreamMultipleTupleExtractor}} in other streamers.
> To fix this, we should:
> # Declare the {{KafkaStreamer}} like this:
> {code}
> KafkaStreamer extends StreamAdapter,
> K, V>
> {code}
> # Remove {{keyDecoder}} and {{valDecoder}} in favor of tuple extractors.
> # Instead of doing {{getStreamer().addData(...)}} directly, call
> {{addMessage(...)}} method providing the raw message consumed from Kafka
> ({{MessageAndMetadata}}). This method will make sure that
> configured extractor is invoked and that all entries are added to
> {{IgniteDataStreamer}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (IGNITE-4140) KafkaStreamer should use tuple extractor instead of decoders
[
https://issues.apache.org/jira/browse/IGNITE-4140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15716945#comment-15716945
]
Valentin Kulichenko commented on IGNITE-4140:
-
The ticket is scheduled for 2.0, so we can just change the API.
Regarding the code I have only these minor comments:
* You removed the license header from {{KafkaStreamer}}. Please bring it back.
* Why did you use explicit check instead of {{A.ensure}} and {{log.error}}
instead of {{U.error}}? This is against guidelines, so please revert.
* We are using spaces instead of tabs for indentation.
Please fix these and we will merge your contribution.
> KafkaStreamer should use tuple extractor instead of decoders
>
>
> Key: IGNITE-4140
> URL: https://issues.apache.org/jira/browse/IGNITE-4140
> Project: Ignite
> Issue Type: Improvement
> Components: streaming
>Affects Versions: 1.7
>Reporter: Valentin Kulichenko
>Assignee: Anil
> Labels: patch-available
> Fix For: 2.0
>
>
> Current design of {{KafkaStreamer}} looks incorrect to me. In particular, it
> extends {{StreamAdapter}}, but ignores tuple extractors provided there and
> uses native Kafka decoders instead. This for example makes impossible to
> produce several entries from one message, like it can be done via
> {{StreamMultipleTupleExtractor}} in other streamers.
> To fix this, we should:
> # Declare the {{KafkaStreamer}} like this:
> {code}
> KafkaStreamer extends StreamAdapter,
> K, V>
> {code}
> # Remove {{keyDecoder}} and {{valDecoder}} in favor of tuple extractors.
> # Instead of doing {{getStreamer().addData(...)}} directly, call
> {{addMessage(...)}} method providing the raw message consumed from Kafka
> ({{MessageAndMetadata}}). This method will make sure that
> configured extractor is invoked and that all entries are added to
> {{IgniteDataStreamer}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (IGNITE-4140) KafkaStreamer should use tuple extractor instead of decoders
[
https://issues.apache.org/jira/browse/IGNITE-4140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15715036#comment-15715036
]
ASF GitHub Bot commented on IGNITE-4140:
GitHub user adasari reopened a pull request:
https://github.com/apache/ignite/pull/1296
IGNITE-4140
KafkaDataStreamer - added new kafka streamer which uses the extractor.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/adasari/ignite IGNITE-4140
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/ignite/pull/1296.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1296
commit 15168221f38327aca89dfd7160fe473434526463
Author: unknown
Date: 2016-11-28T12:29:10Z
StreamAdapter - added empty check to avoid exception
KafkaDataStreamer - added new kafka streamer which uses the extractor.
commit b8bcc773604d0115764f86303a8f3ae7b14b666d
Author: unknown
Date: 2016-12-02T12:37:45Z
Removed KafkaDataStreamer and refactored KafkaStreamer
commit 27a30b5f9b0a2bcd088b508929e37cec422eedf8
Author: unknown
Date: 2016-12-02T12:39:04Z
Refactoring kafka streamer
> KafkaStreamer should use tuple extractor instead of decoders
>
>
> Key: IGNITE-4140
> URL: https://issues.apache.org/jira/browse/IGNITE-4140
> Project: Ignite
> Issue Type: Improvement
> Components: streaming
>Affects Versions: 1.7
>Reporter: Valentin Kulichenko
>Assignee: Anil
> Labels: patch-available
> Fix For: 2.0
>
>
> Current design of {{KafkaStreamer}} looks incorrect to me. In particular, it
> extends {{StreamAdapter}}, but ignores tuple extractors provided there and
> uses native Kafka decoders instead. This for example makes impossible to
> produce several entries from one message, like it can be done via
> {{StreamMultipleTupleExtractor}} in other streamers.
> To fix this, we should:
> # Declare the {{KafkaStreamer}} like this:
> {code}
> KafkaStreamer extends StreamAdapter,
> K, V>
> {code}
> # Remove {{keyDecoder}} and {{valDecoder}} in favor of tuple extractors.
> # Instead of doing {{getStreamer().addData(...)}} directly, call
> {{addMessage(...)}} method providing the raw message consumed from Kafka
> ({{MessageAndMetadata}}). This method will make sure that
> configured extractor is invoked and that all entries are added to
> {{IgniteDataStreamer}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (IGNITE-4140) KafkaStreamer should use tuple extractor instead of decoders
[
https://issues.apache.org/jira/browse/IGNITE-4140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15715035#comment-15715035
]
ASF GitHub Bot commented on IGNITE-4140:
Github user adasari closed the pull request at:
https://github.com/apache/ignite/pull/1296
> KafkaStreamer should use tuple extractor instead of decoders
>
>
> Key: IGNITE-4140
> URL: https://issues.apache.org/jira/browse/IGNITE-4140
> Project: Ignite
> Issue Type: Improvement
> Components: streaming
>Affects Versions: 1.7
>Reporter: Valentin Kulichenko
>Assignee: Anil
> Labels: patch-available
> Fix For: 2.0
>
>
> Current design of {{KafkaStreamer}} looks incorrect to me. In particular, it
> extends {{StreamAdapter}}, but ignores tuple extractors provided there and
> uses native Kafka decoders instead. This for example makes impossible to
> produce several entries from one message, like it can be done via
> {{StreamMultipleTupleExtractor}} in other streamers.
> To fix this, we should:
> # Declare the {{KafkaStreamer}} like this:
> {code}
> KafkaStreamer extends StreamAdapter,
> K, V>
> {code}
> # Remove {{keyDecoder}} and {{valDecoder}} in favor of tuple extractors.
> # Instead of doing {{getStreamer().addData(...)}} directly, call
> {{addMessage(...)}} method providing the raw message consumed from Kafka
> ({{MessageAndMetadata}}). This method will make sure that
> configured extractor is invoked and that all entries are added to
> {{IgniteDataStreamer}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (IGNITE-4140) KafkaStreamer should use tuple extractor instead of decoders
[
https://issues.apache.org/jira/browse/IGNITE-4140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15711788#comment-15711788
]
Anton Vinogradov commented on IGNITE-4140:
--
1) [~vkulichenko], what do you think?
Seems, we just need to replace old streamed with new one?
2) http://reviews.ignite.apache.org/ignite/view
Brief instructions:
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-ReviewWithUpsource
> KafkaStreamer should use tuple extractor instead of decoders
>
>
> Key: IGNITE-4140
> URL: https://issues.apache.org/jira/browse/IGNITE-4140
> Project: Ignite
> Issue Type: Improvement
> Components: streaming
>Affects Versions: 1.7
>Reporter: Valentin Kulichenko
>Assignee: Anil
> Labels: patch-available
> Fix For: 2.0
>
>
> Current design of {{KafkaStreamer}} looks incorrect to me. In particular, it
> extends {{StreamAdapter}}, but ignores tuple extractors provided there and
> uses native Kafka decoders instead. This for example makes impossible to
> produce several entries from one message, like it can be done via
> {{StreamMultipleTupleExtractor}} in other streamers.
> To fix this, we should:
> # Declare the {{KafkaStreamer}} like this:
> {code}
> KafkaStreamer extends StreamAdapter,
> K, V>
> {code}
> # Remove {{keyDecoder}} and {{valDecoder}} in favor of tuple extractors.
> # Instead of doing {{getStreamer().addData(...)}} directly, call
> {{addMessage(...)}} method providing the raw message consumed from Kafka
> ({{MessageAndMetadata}}). This method will make sure that
> configured extractor is invoked and that all entries are added to
> {{IgniteDataStreamer}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (IGNITE-4140) KafkaStreamer should use tuple extractor instead of decoders
[
https://issues.apache.org/jira/browse/IGNITE-4140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709025#comment-15709025
]
Anil commented on IGNITE-4140:
--
Hi [~avinogradov]
Thanks for comments.
#1 - Creating new steamer will provide backward compatibility and KafkaStreamer
can be Deprecated when KafkaDataStreamer is ready.
What do you say ? if team prefers refactoring of KafkaStreamer, i will do.
#2 - i am not sure how to use review tool. could you please share any
references to use the tool ?
Thanks
> KafkaStreamer should use tuple extractor instead of decoders
>
>
> Key: IGNITE-4140
> URL: https://issues.apache.org/jira/browse/IGNITE-4140
> Project: Ignite
> Issue Type: Improvement
> Components: streaming
>Affects Versions: 1.7
>Reporter: Valentin Kulichenko
>Assignee: Anil
> Labels: patch-available
> Fix For: 2.0
>
>
> Current design of {{KafkaStreamer}} looks incorrect to me. In particular, it
> extends {{StreamAdapter}}, but ignores tuple extractors provided there and
> uses native Kafka decoders instead. This for example makes impossible to
> produce several entries from one message, like it can be done via
> {{StreamMultipleTupleExtractor}} in other streamers.
> To fix this, we should:
> # Declare the {{KafkaStreamer}} like this:
> {code}
> KafkaStreamer extends StreamAdapter,
> K, V>
> {code}
> # Remove {{keyDecoder}} and {{valDecoder}} in favor of tuple extractors.
> # Instead of doing {{getStreamer().addData(...)}} directly, call
> {{addMessage(...)}} method providing the raw message consumed from Kafka
> ({{MessageAndMetadata}}). This method will make sure that
> configured extractor is invoked and that all entries are added to
> {{IgniteDataStreamer}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (IGNITE-4140) KafkaStreamer should use tuple extractor instead of decoders
[
https://issues.apache.org/jira/browse/IGNITE-4140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708955#comment-15708955
]
Anton Vinogradov commented on IGNITE-4140:
--
Anil,
Initial comments:
1) We have to refactor KafkaStreamer instead of creating KafkaDataStreamer.
2) It will be easy to review changes using Apache Ignite review tool:
http://reviews.ignite.apache.org/ignite/view
In case you're using IDEA just setup Upsource and start review, it will allow
to discuss changes in simple way.
Add me and Valentin Kulichenko as reviewers.
> KafkaStreamer should use tuple extractor instead of decoders
>
>
> Key: IGNITE-4140
> URL: https://issues.apache.org/jira/browse/IGNITE-4140
> Project: Ignite
> Issue Type: Improvement
> Components: streaming
>Affects Versions: 1.7
>Reporter: Valentin Kulichenko
>Assignee: Anil
> Labels: patch-available
> Fix For: 2.0
>
>
> Current design of {{KafkaStreamer}} looks incorrect to me. In particular, it
> extends {{StreamAdapter}}, but ignores tuple extractors provided there and
> uses native Kafka decoders instead. This for example makes impossible to
> produce several entries from one message, like it can be done via
> {{StreamMultipleTupleExtractor}} in other streamers.
> To fix this, we should:
> # Declare the {{KafkaStreamer}} like this:
> {code}
> KafkaStreamer extends StreamAdapter,
> K, V>
> {code}
> # Remove {{keyDecoder}} and {{valDecoder}} in favor of tuple extractors.
> # Instead of doing {{getStreamer().addData(...)}} directly, call
> {{addMessage(...)}} method providing the raw message consumed from Kafka
> ({{MessageAndMetadata}}). This method will make sure that
> configured extractor is invoked and that all entries are added to
> {{IgniteDataStreamer}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (IGNITE-4140) KafkaStreamer should use tuple extractor instead of decoders
[
https://issues.apache.org/jira/browse/IGNITE-4140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15702469#comment-15702469
]
Anil commented on IGNITE-4140:
--
Hi [~avinogradov]
Could you please review the changes ? Thanks
KafkaDataStreamer can be used when
1. Key of kafka message is key of the cache entry and message of the kafka
message as value of cache entry
2. Kafka message is actually number of cache entries
Tuple extractor is mandatory for both #1 and #2
I see two possible issues with KafkaStreamer and KafkaDataStreamer
a. Order of messages is not guaranteed as data streamer not guarantee the order
by design
b. possibility of streamer closed exception when node rejoin.
#a - entries must be added to cache directly instead of data streamer if the
order of messages is must
#b - could you please share your inputs ?
Thanks
> KafkaStreamer should use tuple extractor instead of decoders
>
>
> Key: IGNITE-4140
> URL: https://issues.apache.org/jira/browse/IGNITE-4140
> Project: Ignite
> Issue Type: Improvement
> Components: streaming
>Affects Versions: 1.7
>Reporter: Valentin Kulichenko
>Assignee: Anil
> Fix For: 2.0
>
>
> Current design of {{KafkaStreamer}} looks incorrect to me. In particular, it
> extends {{StreamAdapter}}, but ignores tuple extractors provided there and
> uses native Kafka decoders instead. This for example makes impossible to
> produce several entries from one message, like it can be done via
> {{StreamMultipleTupleExtractor}} in other streamers.
> To fix this, we should:
> # Declare the {{KafkaStreamer}} like this:
> {code}
> KafkaStreamer extends StreamAdapter,
> K, V>
> {code}
> # Remove {{keyDecoder}} and {{valDecoder}} in favor of tuple extractors.
> # Instead of doing {{getStreamer().addData(...)}} directly, call
> {{addMessage(...)}} method providing the raw message consumed from Kafka
> ({{MessageAndMetadata}}). This method will make sure that
> configured extractor is invoked and that all entries are added to
> {{IgniteDataStreamer}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (IGNITE-4140) KafkaStreamer should use tuple extractor instead of decoders
[
https://issues.apache.org/jira/browse/IGNITE-4140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15701869#comment-15701869
]
ASF GitHub Bot commented on IGNITE-4140:
GitHub user adasari opened a pull request:
https://github.com/apache/ignite/pull/1296
IGNITE-4140
KafkaDataStreamer - added new kafka streamer which uses the extractor.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/adasari/ignite IGNITE-4140
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/ignite/pull/1296.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1296
commit 15168221f38327aca89dfd7160fe473434526463
Author: unknown
Date: 2016-11-28T12:29:10Z
StreamAdapter - added empty check to avoid exception
KafkaDataStreamer - added new kafka streamer which uses the extractor.
> KafkaStreamer should use tuple extractor instead of decoders
>
>
> Key: IGNITE-4140
> URL: https://issues.apache.org/jira/browse/IGNITE-4140
> Project: Ignite
> Issue Type: Improvement
> Components: streaming
>Affects Versions: 1.7
>Reporter: Valentin Kulichenko
> Fix For: 2.0
>
>
> Current design of {{KafkaStreamer}} looks incorrect to me. In particular, it
> extends {{StreamAdapter}}, but ignores tuple extractors provided there and
> uses native Kafka decoders instead. This for example makes impossible to
> produce several entries from one message, like it can be done via
> {{StreamMultipleTupleExtractor}} in other streamers.
> To fix this, we should:
> # Declare the {{KafkaStreamer}} like this:
> {code}
> KafkaStreamer extends StreamAdapter,
> K, V>
> {code}
> # Remove {{keyDecoder}} and {{valDecoder}} in favor of tuple extractors.
> # Instead of doing {{getStreamer().addData(...)}} directly, call
> {{addMessage(...)}} method providing the raw message consumed from Kafka
> ({{MessageAndMetadata}}). This method will make sure that
> configured extractor is invoked and that all entries are added to
> {{IgniteDataStreamer}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (IGNITE-4140) KafkaStreamer should use tuple extractor instead of decoders
[
https://issues.apache.org/jira/browse/IGNITE-4140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15609531#comment-15609531
]
Valentin Kulichenko commented on IGNITE-4140:
-
NOTE: this change breaks API compatibility, so should be done in 2.0.
> KafkaStreamer should use tuple extractor instead of decoders
>
>
> Key: IGNITE-4140
> URL: https://issues.apache.org/jira/browse/IGNITE-4140
> Project: Ignite
> Issue Type: Improvement
> Components: streaming
>Affects Versions: 1.7
>Reporter: Valentin Kulichenko
> Fix For: 2.0
>
>
> Current design of {{KafkaStreamer}} looks incorrect to me. In particular, it
> extends {{StreamAdapter}}, but ignores tuple extractors provided there and
> uses native Kafka decoders instead. This for example makes impossible to
> produce several entries from one message, like it can be done via
> {{StreamMultipleTupleExtractor}} in other streamers.
> To fix this, we should:
> # Declare the {{KafkaStreamer}} like this:
> {code}
> KafkaStreamer extends StreamAdapter,
> K, V>
> {code}
> # Remove {{keyDecoder}} and {{valDecoder}} in favor of tuple extractors.
> # Instead of doing {{getStreamer().addData(...)}} directly, call
> {{addMessage(...)}} method providing the raw message consumed from Kafka
> ({{MessageAndMetadata}}). This method will make sure that
> configured extractor is invoked and that all entries are added to
> {{IgniteDataStreamer}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
