[
https://issues.apache.org/jira/browse/IGNITE-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15067927#comment-15067927
]
Roman Shtykh commented on IGNITE-2016:
--------------------------------------
Hi Denis,
I think Anton had similar questions when he reviewed Flume integration ;)
Please see my replies inline.
D> In general as I see you've implemented a new Kafka streamer using Kafka
Connect API, correct?
D> If so is there any reason why we can't extend existed StreamAdapter like
current Ignite's KafkaStreamer, MqttStreamer, etc. does? It will be more
consistent if you create a kind of KafkaConnectStreamer and put all the
implementation details in there.
R> First of all, from what I see, Kafka Connect is going to be a standard way
of integration for data import/export and streaming. This is coming with many
'perks' included like reliability, scalability, etc.
R> I understand you would like to have Kafka client being invoked from Ignite
and keep it consistent with other implementations, but here Kafka Connect is
central and you have to invoke Ignite from Connect API (similarly to how it is
done in Flume).
D> I wouldn't mode create one more module cause we already have "ignite-kafka".
So my preference if you reuse the existed one.
D> New streamer's configuration parameters should be passed through setters of
the implementation. Please refer to KafkaStreamer, MqttStreamer to as an
example.
R> Looking at how Kafka Connect and Flume is designed for integration with
other systems, I think it is valid to have an alternative to using
IgniteDataStreamer for these sort of integrations.
R> Do you prefer to handle all Kafka reliability and scalability issues, etc.
every time Kafka has modifications (or make users handle it, which is even
worse), or using these out of the box? ;) I prefer the latter.
R> That said, the current KafkaStreamer may be used too for some special cases.
We can keep two packages for a while.
But again I want to emphasize, it is not the duplicate solution.
> Update KafkaStreamer to fit new features introduced in Kafka 0.9
> ----------------------------------------------------------------
>
> Key: IGNITE-2016
> URL: https://issues.apache.org/jira/browse/IGNITE-2016
> Project: Ignite
> Issue Type: New Feature
> Components: streaming
> Reporter: Roman Shtykh
> Assignee: Roman Shtykh
>
> Particularly,
> - new consumer
> - Kafka Connect (Copycat)
> http://www.confluent.io/blog/apache-kafka-0.9-is-released
> This can be a a different integration task or a complete re-write of the
> current implementation, considering the fact that Kafka Connect is a new
> standard way for "large-scale, real-time data import and export for Kafka."
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
