[jira] [Commented] (IGNITE-7752) Update Ignite KafkaStreamer to use new KafkaConsmer configuration.
[ https://issues.apache.org/jira/browse/IGNITE-7752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564916#comment-16564916 ] ASF GitHub Bot commented on IGNITE-7752: Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4363 > Update Ignite KafkaStreamer to use new KafkaConsmer configuration. > -- > > Key: IGNITE-7752 > URL: https://issues.apache.org/jira/browse/IGNITE-7752 > Project: Ignite > Issue Type: Task > Components: streaming >Reporter: Andrew Mashenkov >Assignee: Roman Shtykh >Priority: Major > Labels: newbie, streaming > Fix For: 2.7 > > > Seems, for now it is impossible to use new style KafkaConsumer configuration > in KafkaStreamer. > The issue here is Ignite use > kafka.consumer.Consumer.createJavaConsumerConnector() method which creates > old consumer (ZookeeperConsumerConnector). > We should create a new KafkaConsumer instead which looks like support both, > old and new style configs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7752) Update Ignite KafkaStreamer to use new KafkaConsmer configuration.
[ https://issues.apache.org/jira/browse/IGNITE-7752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564835#comment-16564835 ] Roman Shtykh commented on IGNITE-7752: -- [~amashenkov] I will be using it in one of my projects, and if I notice the problems with reconnection again I'll share it. Thanks for reviewing. I will merge it then. > Update Ignite KafkaStreamer to use new KafkaConsmer configuration. > -- > > Key: IGNITE-7752 > URL: https://issues.apache.org/jira/browse/IGNITE-7752 > Project: Ignite > Issue Type: Task > Components: streaming >Reporter: Andrew Mashenkov >Assignee: Roman Shtykh >Priority: Major > Labels: newbie, streaming > Fix For: 2.7 > > > Seems, for now it is impossible to use new style KafkaConsumer configuration > in KafkaStreamer. > The issue here is Ignite use > kafka.consumer.Consumer.createJavaConsumerConnector() method which creates > old consumer (ZookeeperConsumerConnector). > We should create a new KafkaConsumer instead which looks like support both, > old and new style configs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7752) Update Ignite KafkaStreamer to use new KafkaConsmer configuration.
[ https://issues.apache.org/jira/browse/IGNITE-7752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564759#comment-16564759 ] Andrew Mashenkov commented on IGNITE-7752: -- [~roman_s], * *Now PR looks good for me. * Looks like there was a bug with consumer reconnection in earlier versions. Ok. Let's left this as is. * Sorry, I missed smth, there is no loop. All look good. > Update Ignite KafkaStreamer to use new KafkaConsmer configuration. > -- > > Key: IGNITE-7752 > URL: https://issues.apache.org/jira/browse/IGNITE-7752 > Project: Ignite > Issue Type: Task > Components: streaming >Reporter: Andrew Mashenkov >Assignee: Roman Shtykh >Priority: Major > Labels: newbie, streaming > Fix For: 2.7 > > > Seems, for now it is impossible to use new style KafkaConsumer configuration > in KafkaStreamer. > The issue here is Ignite use > kafka.consumer.Consumer.createJavaConsumerConnector() method which creates > old consumer (ZookeeperConsumerConnector). > We should create a new KafkaConsumer instead which looks like support both, > old and new style configs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7752) Update Ignite KafkaStreamer to use new KafkaConsmer configuration.
[
https://issues.apache.org/jira/browse/IGNITE-7752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564713#comment-16564713
]
Roman Shtykh commented on IGNITE-7752:
--
[~amashenkov] Thank you for having a look.
* Added javadoc comments.
* Honestly, I am not aware of this (resubscribing).
The _poll_ method says {color:#629755}@throws
{color}{color:#629755}org.apache.kafka.common.KafkaException for any other
unrecoverable errors{color}. I don't think we can do something here, just fail
(I will add an error message to log). Normally you have several brokers with
_n_ replication, so having your whole Kafka cluster is a rare case.
Do you recommend to resubscribe? (Looks like it's a consumer's job)
* I can't find any infinite loop. Can you explain?
> Update Ignite KafkaStreamer to use new KafkaConsmer configuration.
> --
>
> Key: IGNITE-7752
> URL: https://issues.apache.org/jira/browse/IGNITE-7752
> Project: Ignite
> Issue Type: Task
> Components: streaming
>Reporter: Andrew Mashenkov
>Assignee: Roman Shtykh
>Priority: Major
> Labels: newbie, streaming
> Fix For: 2.7
>
>
> Seems, for now it is impossible to use new style KafkaConsumer configuration
> in KafkaStreamer.
> The issue here is Ignite use
> kafka.consumer.Consumer.createJavaConsumerConnector() method which creates
> old consumer (ZookeeperConsumerConnector).
> We should create a new KafkaConsumer instead which looks like support both,
> old and new style configs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (IGNITE-7752) Update Ignite KafkaStreamer to use new KafkaConsmer configuration.
[ https://issues.apache.org/jira/browse/IGNITE-7752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563318#comment-16563318 ] Andrew Mashenkov commented on IGNITE-7752: -- [~roman_s], I've taken a look at the PR. Would you please fix javadocs to all classes and methods? Seems, they were missed in some places. Also you catch and ignore an Exception in ConsumerTask. AFAIK, consumer.poll can throw KafkaException when broker become temporary unavailable and consumer should be re-subscribed after such failure, isn't it? Also, stopped flag should be checked in this catch block to avoid infinite loop. Please, let me know if I'm wrong. > Update Ignite KafkaStreamer to use new KafkaConsmer configuration. > -- > > Key: IGNITE-7752 > URL: https://issues.apache.org/jira/browse/IGNITE-7752 > Project: Ignite > Issue Type: Task > Components: streaming >Reporter: Andrew Mashenkov >Assignee: Roman Shtykh >Priority: Major > Labels: newbie, streaming > Fix For: 2.7 > > > Seems, for now it is impossible to use new style KafkaConsumer configuration > in KafkaStreamer. > The issue here is Ignite use > kafka.consumer.Consumer.createJavaConsumerConnector() method which creates > old consumer (ZookeeperConsumerConnector). > We should create a new KafkaConsumer instead which looks like support both, > old and new style configs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7752) Update Ignite KafkaStreamer to use new KafkaConsmer configuration.
[ https://issues.apache.org/jira/browse/IGNITE-7752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544066#comment-16544066 ] ASF GitHub Bot commented on IGNITE-7752: GitHub user shroman opened a pull request: https://github.com/apache/ignite/pull/4363 IGNITE-7752: Update Ignite KafkaStreamer to use new consumer. You can merge this pull request into a Git repository by running: $ git pull https://github.com/shroman/ignite IGNITE-7752 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4363.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4363 commit b099602287ac1282b3b3fcb094e601c1f9e33e52 Author: shroman Date: 2018-07-14T05:56:00Z IGNITE-7752: Update Ignite KafkaStreamer to use new consumer. commit 8eafc4c305b6c069b7b63b7d4d567a7160b17043 Author: shroman Date: 2018-07-14T05:56:33Z IGNITE-7752: Update Ignite KafkaStreamer to use new consumer. > Update Ignite KafkaStreamer to use new KafkaConsmer configuration. > -- > > Key: IGNITE-7752 > URL: https://issues.apache.org/jira/browse/IGNITE-7752 > Project: Ignite > Issue Type: Task > Components: streaming >Reporter: Andrew Mashenkov >Assignee: Roman Shtykh >Priority: Major > Labels: newbie > Fix For: 2.7 > > > Seems, for now it is impossible to use new style KafkaConsumer configuration > in KafkaStreamer. > The issue here is Ignite use > kafka.consumer.Consumer.createJavaConsumerConnector() method which creates > old consumer (ZookeeperConsumerConnector). > We should create a new KafkaConsumer instead which looks like support both, > old and new style configs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7752) Update Ignite KafkaStreamer to use new KafkaConsmer configuration.
[ https://issues.apache.org/jira/browse/IGNITE-7752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543998#comment-16543998 ] Roman Shtykh commented on IGNITE-7752: -- Need this feature now, reassigned to myself. > Update Ignite KafkaStreamer to use new KafkaConsmer configuration. > -- > > Key: IGNITE-7752 > URL: https://issues.apache.org/jira/browse/IGNITE-7752 > Project: Ignite > Issue Type: Task > Components: streaming >Reporter: Andrew Mashenkov >Assignee: Roman Shtykh >Priority: Major > Labels: newbie > Fix For: 2.7 > > > Seems, for now it is impossible to use new style KafkaConsumer configuration > in KafkaStreamer. > The issue here is Ignite use > kafka.consumer.Consumer.createJavaConsumerConnector() method which creates > old consumer (ZookeeperConsumerConnector). > We should create a new KafkaConsumer instead which looks like support both, > old and new style configs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
