[jira] [Commented] (FLINK-10900) Mark Kafka 2.0 connector as beta feature
[ https://issues.apache.org/jira/browse/FLINK-10900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830135#comment-16830135 ] Till Rohrmann commented on FLINK-10900: --- FLINK-12030 seems to be a Kafka test instability. A part from that, I'm not aware of other instabilities. > Mark Kafka 2.0 connector as beta feature > > > Key: FLINK-10900 > URL: https://issues.apache.org/jira/browse/FLINK-10900 > Project: Flink > Issue Type: Task > Components: Connectors / Kafka >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Assignee: Piotr Nowojski >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > > Given the test problems with the Kafka 2.0 connector we should mark this > connector as a beta feature until we have fully understood why so many tests > deadlock. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10900) Mark Kafka 2.0 connector as beta feature
[ https://issues.apache.org/jira/browse/FLINK-10900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829381#comment-16829381 ] Seth Wiesman commented on FLINK-10900: -- [~till.rohrmann] Is the universal Kafka connector still showing instabilities? Are there any tickets I can follow? > Mark Kafka 2.0 connector as beta feature > > > Key: FLINK-10900 > URL: https://issues.apache.org/jira/browse/FLINK-10900 > Project: Flink > Issue Type: Task > Components: Connectors / Kafka >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Assignee: Piotr Nowojski >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > > Given the test problems with the Kafka 2.0 connector we should mark this > connector as a beta feature until we have fully understood why so many tests > deadlock. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10900) Mark Kafka 2.0 connector as beta feature
[
https://issues.apache.org/jira/browse/FLINK-10900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16689580#comment-16689580
]
ASF GitHub Bot commented on FLINK-10900:
pnowojski closed pull request #7109: [FLINK-10900][kafka][docs] Mark universal
Kafka connector as beta
URL: https://github.com/apache/flink/pull/7109
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/docs/dev/connectors/kafka.md b/docs/dev/connectors/kafka.md
index f81080a4b53..0630c6ec7d6 100644
--- a/docs/dev/connectors/kafka.md
+++ b/docs/dev/connectors/kafka.md
@@ -73,7 +73,7 @@ For most users, the `FlinkKafkaConsumer08` (part of
`flink-connector-kafka`) is
This connector supports https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+timestamps+to+Kafka+message;>Kafka
messages with timestamps both for producing and consuming.
-flink-connector-kafka-0.11_2.11
+flink-connector-kafka-0.11{{ site.scala_version_suffix }}
1.4.0
FlinkKafkaConsumer011
FlinkKafkaProducer011
@@ -81,13 +81,25 @@ For most users, the `FlinkKafkaConsumer08` (part of
`flink-connector-kafka`) is
Since 0.11.x Kafka does not support scala 2.10. This connector
supports https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging;>Kafka
transactional messaging to provide exactly once semantic for the
producer.
-flink-connector-kafka_2.11
+flink-connector-kafka{{ site.scala_version_suffix }}
1.7.0
FlinkKafkaConsumer
FlinkKafkaProducer
>= 1.0.0
-This Kafka connector attempts to track the latest version of the
Kafka client. The version of the client it uses may change between Flink
releases. Modern Kafka clients are backwards compatible with broker versions
0.10.0 or later. However for Kafka 0.11.x and 0.10.x versions, we recommend
using dedicated flink-connector-kafka-0.11 and link-connector-kafka-0.10
respectively.
-
+
+This universal Kafka connector attempts to track the latest version of
the Kafka client.
+The version of the client it uses may change between Flink releases.
+Modern Kafka clients are backwards compatible with broker versions
0.10.0 or later.
+However for Kafka 0.11.x and 0.10.x versions, we recommend using
dedicated
+flink-connector-kafka-0.11{{ site.scala_version_suffix }} and
link-connector-kafka-0.10{{ site.scala_version_suffix }} respectively.
+
+ Attention: as of Flink 1.7 the universal Kafka
connector is considered to be
+ in a BETA status and might not be as stable as the
0.11 connector.
+ In case of problems with the universal connector, you can try to use
flink-connector-kafka-0.11{{ site.scala_version_suffix }}
+ which should be compatible with all of the Kafka versions starting
from 0.11.
+
+
+
@@ -101,7 +113,8 @@ Then, import the connector in your maven project:
{% endhighlight %}
-Note that the streaming connectors are currently not part of the binary
distribution. See how to link with them for cluster execution [here]({{
site.baseurl}}/dev/linking.html).
+Note that the streaming connectors are currently not part of the binary
distribution.
+See how to link with them for cluster execution [here]({{
site.baseurl}}/dev/linking.html).
## Installing Apache Kafka
@@ -110,17 +123,21 @@ Note that the streaming connectors are currently not part
of the binary distribu
## Kafka 1.0.0+ Connector
-Starting with Flink 1.7, there is a new Kafka connector that does not track a
specific Kafka major version. Rather, it tracks the latest version of Kafka at
the time of the Flink release.
+Starting with Flink 1.7, there is a new universal Kafka connector that does
not track a specific Kafka major version.
+Rather, it tracks the latest version of Kafka at the time of the Flink release.
-If your Kafka broker version is 1.0.0 or newer, you should use this Kafka
connector. If you use an older version of Kafka (0.11, 0.10, 0.9, or 0.8), you
should use the connector corresponding to the broker version.
+If your Kafka broker version is 1.0.0 or newer, you should use this Kafka
connector.
+If you use an older version of Kafka (0.11, 0.10, 0.9, or 0.8), you should use
the connector corresponding to the broker version.
### Compatibility
-The modern Kafka connector is compatible with older and newer Kafka brokers
through the compatibility guarantees of the Kafka client API and broker. The
modern Kafka connector is compatible with broker versions 0.11.0 or later,
depending on the
[jira] [Commented] (FLINK-10900) Mark Kafka 2.0 connector as beta feature
[ https://issues.apache.org/jira/browse/FLINK-10900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16689579#comment-16689579 ] ASF GitHub Bot commented on FLINK-10900: pnowojski commented on a change in pull request #7109: [FLINK-10900][kafka][docs] Mark universal Kafka connector as beta URL: https://github.com/apache/flink/pull/7109#discussion_r234248328 ## File path: docs/dev/connectors/kafka.md ## @@ -110,13 +123,17 @@ Note that the streaming connectors are currently not part of the binary distribu ## Kafka 1.0.0+ Connector -Starting with Flink 1.7, there is a new Kafka connector that does not track a specific Kafka major version. Rather, it tracks the latest version of Kafka at the time of the Flink release. +Starting with Flink 1.7, there is a new Kafka connector that does not track a specific Kafka major version. +Rather, it tracks the latest version of Kafka at the time of the Flink release. -If your Kafka broker version is 1.0.0 or newer, you should use this Kafka connector. If you use an older version of Kafka (0.11, 0.10, 0.9, or 0.8), you should use the connector corresponding to the broker version. +If your Kafka broker version is 1.0.0 or newer, you should use this Kafka connector. +If you use an older version of Kafka (0.11, 0.10, 0.9, or 0.8), you should use the connector corresponding to the broker version. ### Compatibility -The modern Kafka connector is compatible with older and newer Kafka brokers through the compatibility guarantees of the Kafka client API and broker. The modern Kafka connector is compatible with broker versions 0.11.0 or later, depending on the features used. For details on Kafka compatibility, please refer to the [Kafka documentation](https://kafka.apache.org/protocol.html#protocol_compatibility). +The modern Kafka connector is compatible with older and newer Kafka brokers through the compatibility guarantees of the Kafka client API and broker. Review comment: I was just braking lines here, but I can fix this while I'm at it. Thanks for pointing out - I have fixed all `modern` references. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Mark Kafka 2.0 connector as beta feature > > > Key: FLINK-10900 > URL: https://issues.apache.org/jira/browse/FLINK-10900 > Project: Flink > Issue Type: Task > Components: Kafka Connector >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Assignee: Piotr Nowojski >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > > Given the test problems with the Kafka 2.0 connector we should mark this > connector as a beta feature until we have fully understood why so many tests > deadlock. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10900) Mark Kafka 2.0 connector as beta feature
[ https://issues.apache.org/jira/browse/FLINK-10900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16689568#comment-16689568 ] ASF GitHub Bot commented on FLINK-10900: pnowojski commented on a change in pull request #7109: [FLINK-10900][kafka][docs] Mark universal Kafka connector as beta URL: https://github.com/apache/flink/pull/7109#discussion_r234248328 ## File path: docs/dev/connectors/kafka.md ## @@ -110,13 +123,17 @@ Note that the streaming connectors are currently not part of the binary distribu ## Kafka 1.0.0+ Connector -Starting with Flink 1.7, there is a new Kafka connector that does not track a specific Kafka major version. Rather, it tracks the latest version of Kafka at the time of the Flink release. +Starting with Flink 1.7, there is a new Kafka connector that does not track a specific Kafka major version. +Rather, it tracks the latest version of Kafka at the time of the Flink release. -If your Kafka broker version is 1.0.0 or newer, you should use this Kafka connector. If you use an older version of Kafka (0.11, 0.10, 0.9, or 0.8), you should use the connector corresponding to the broker version. +If your Kafka broker version is 1.0.0 or newer, you should use this Kafka connector. +If you use an older version of Kafka (0.11, 0.10, 0.9, or 0.8), you should use the connector corresponding to the broker version. ### Compatibility -The modern Kafka connector is compatible with older and newer Kafka brokers through the compatibility guarantees of the Kafka client API and broker. The modern Kafka connector is compatible with broker versions 0.11.0 or later, depending on the features used. For details on Kafka compatibility, please refer to the [Kafka documentation](https://kafka.apache.org/protocol.html#protocol_compatibility). +The modern Kafka connector is compatible with older and newer Kafka brokers through the compatibility guarantees of the Kafka client API and broker. Review comment: I was just braking new lines here, but I can fix this while I'm at it. Thanks for pointing out. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Mark Kafka 2.0 connector as beta feature > > > Key: FLINK-10900 > URL: https://issues.apache.org/jira/browse/FLINK-10900 > Project: Flink > Issue Type: Task > Components: Kafka Connector >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Assignee: Piotr Nowojski >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > > Given the test problems with the Kafka 2.0 connector we should mark this > connector as a beta feature until we have fully understood why so many tests > deadlock. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10900) Mark Kafka 2.0 connector as beta feature
[ https://issues.apache.org/jira/browse/FLINK-10900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16689569#comment-16689569 ] ASF GitHub Bot commented on FLINK-10900: pnowojski commented on a change in pull request #7109: [FLINK-10900][kafka][docs] Mark universal Kafka connector as beta URL: https://github.com/apache/flink/pull/7109#discussion_r234248328 ## File path: docs/dev/connectors/kafka.md ## @@ -110,13 +123,17 @@ Note that the streaming connectors are currently not part of the binary distribu ## Kafka 1.0.0+ Connector -Starting with Flink 1.7, there is a new Kafka connector that does not track a specific Kafka major version. Rather, it tracks the latest version of Kafka at the time of the Flink release. +Starting with Flink 1.7, there is a new Kafka connector that does not track a specific Kafka major version. +Rather, it tracks the latest version of Kafka at the time of the Flink release. -If your Kafka broker version is 1.0.0 or newer, you should use this Kafka connector. If you use an older version of Kafka (0.11, 0.10, 0.9, or 0.8), you should use the connector corresponding to the broker version. +If your Kafka broker version is 1.0.0 or newer, you should use this Kafka connector. +If you use an older version of Kafka (0.11, 0.10, 0.9, or 0.8), you should use the connector corresponding to the broker version. ### Compatibility -The modern Kafka connector is compatible with older and newer Kafka brokers through the compatibility guarantees of the Kafka client API and broker. The modern Kafka connector is compatible with broker versions 0.11.0 or later, depending on the features used. For details on Kafka compatibility, please refer to the [Kafka documentation](https://kafka.apache.org/protocol.html#protocol_compatibility). +The modern Kafka connector is compatible with older and newer Kafka brokers through the compatibility guarantees of the Kafka client API and broker. Review comment: I was just brakinglines here, but I can fix this while I'm at it. Thanks for pointing out. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Mark Kafka 2.0 connector as beta feature > > > Key: FLINK-10900 > URL: https://issues.apache.org/jira/browse/FLINK-10900 > Project: Flink > Issue Type: Task > Components: Kafka Connector >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Assignee: Piotr Nowojski >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > > Given the test problems with the Kafka 2.0 connector we should mark this > connector as a beta feature until we have fully understood why so many tests > deadlock. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10900) Mark Kafka 2.0 connector as beta feature
[
https://issues.apache.org/jira/browse/FLINK-10900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16689566#comment-16689566
]
ASF GitHub Bot commented on FLINK-10900:
pnowojski commented on a change in pull request #7109:
[FLINK-10900][kafka][docs] Mark universal Kafka connector as beta
URL: https://github.com/apache/flink/pull/7109#discussion_r234247629
##
File path: docs/dev/connectors/kafka.md
##
@@ -73,21 +73,33 @@ For most users, the `FlinkKafkaConsumer08` (part of
`flink-connector-kafka`) is
This connector supports https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+timestamps+to+Kafka+message;>Kafka
messages with timestamps both for producing and consuming.
-flink-connector-kafka-0.11_2.11
+flink-connector-kafka-0.11{{ site.scala_version_suffix }}
1.4.0
FlinkKafkaConsumer011
FlinkKafkaProducer011
0.11.x
Since 0.11.x Kafka does not support scala 2.10. This connector
supports https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging;>Kafka
transactional messaging to provide exactly once semantic for the
producer.
-flink-connector-kafka_2.11
+flink-connector-kafka{{ site.scala_version_suffix }}
1.7.0
FlinkKafkaConsumer
FlinkKafkaProducer
>= 1.0.0
-This Kafka connector attempts to track the latest version of the
Kafka client. The version of the client it uses may change between Flink
releases. Modern Kafka clients are backwards compatible with broker versions
0.10.0 or later. However for Kafka 0.11.x and 0.10.x versions, we recommend
using dedicated flink-connector-kafka-0.11 and link-connector-kafka-0.10
respectively.
-
+
+This universal Kafka connector attempts to track the latest version of
the Kafka client.
+The version of the client it uses may change between Flink releases.
+Modern Kafka clients are backwards compatible with broker versions
0.10.0 or later.
+However for Kafka 0.11.x and 0.10.x versions, we recommend using
dedicated
+flink-connector-kafka-0.11 and link-connector-kafka-0.10 respectively.
+
+ Attention: as of Flink 1.7 the universal Kafka
connector is considered to be
+ in a BETA status and might not be as stable as the
0.11 connector.
+ In case of problems with the universal connector, you can try to use
flink-connector-kafka-0.11{{ site.scala_version_suffix }}
+ which should be compatible with all of the Kafka versions starting
from 0.11.
Review comment:
This is a good point, but as far as I know this is related only if someone
would want to upgrade `kafka.version` of the `flink-connector-kafka`. There
were couple of users that were successfully using `flink-connector-kafka-0.11`
against Kafka 1.0 and it should work for the same reasons why `universal`
connector should work with any Kafka version `0.11+`.
I would like to merge this before the next release candidate, but if I'm
wrong with this regard, please let me know here - we can always fix the
documentation after the release :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Mark Kafka 2.0 connector as beta feature
>
>
> Key: FLINK-10900
> URL: https://issues.apache.org/jira/browse/FLINK-10900
> Project: Flink
> Issue Type: Task
> Components: Kafka Connector
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Piotr Nowojski
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> Given the test problems with the Kafka 2.0 connector we should mark this
> connector as a beta feature until we have fully understood why so many tests
> deadlock.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10900) Mark Kafka 2.0 connector as beta feature
[
https://issues.apache.org/jira/browse/FLINK-10900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16689564#comment-16689564
]
ASF GitHub Bot commented on FLINK-10900:
pnowojski commented on a change in pull request #7109:
[FLINK-10900][kafka][docs] Mark universal Kafka connector as beta
URL: https://github.com/apache/flink/pull/7109#discussion_r234247629
##
File path: docs/dev/connectors/kafka.md
##
@@ -73,21 +73,33 @@ For most users, the `FlinkKafkaConsumer08` (part of
`flink-connector-kafka`) is
This connector supports https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+timestamps+to+Kafka+message;>Kafka
messages with timestamps both for producing and consuming.
-flink-connector-kafka-0.11_2.11
+flink-connector-kafka-0.11{{ site.scala_version_suffix }}
1.4.0
FlinkKafkaConsumer011
FlinkKafkaProducer011
0.11.x
Since 0.11.x Kafka does not support scala 2.10. This connector
supports https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging;>Kafka
transactional messaging to provide exactly once semantic for the
producer.
-flink-connector-kafka_2.11
+flink-connector-kafka{{ site.scala_version_suffix }}
1.7.0
FlinkKafkaConsumer
FlinkKafkaProducer
>= 1.0.0
-This Kafka connector attempts to track the latest version of the
Kafka client. The version of the client it uses may change between Flink
releases. Modern Kafka clients are backwards compatible with broker versions
0.10.0 or later. However for Kafka 0.11.x and 0.10.x versions, we recommend
using dedicated flink-connector-kafka-0.11 and link-connector-kafka-0.10
respectively.
-
+
+This universal Kafka connector attempts to track the latest version of
the Kafka client.
+The version of the client it uses may change between Flink releases.
+Modern Kafka clients are backwards compatible with broker versions
0.10.0 or later.
+However for Kafka 0.11.x and 0.10.x versions, we recommend using
dedicated
+flink-connector-kafka-0.11 and link-connector-kafka-0.10 respectively.
+
+ Attention: as of Flink 1.7 the universal Kafka
connector is considered to be
+ in a BETA status and might not be as stable as the
0.11 connector.
+ In case of problems with the universal connector, you can try to use
flink-connector-kafka-0.11{{ site.scala_version_suffix }}
+ which should be compatible with all of the Kafka versions starting
from 0.11.
Review comment:
This is a good point, but as far as I know this is related only if someone
would want to upgrade `kafka.version` of the `flink-connector-kafka`. There
were couple of users that were successfully using `flink-connector-kafka-0.11`
against Kafka 1.0 and it should work for the same reasons why `universal`
connector should work with any Kafka version `0.11+`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Mark Kafka 2.0 connector as beta feature
>
>
> Key: FLINK-10900
> URL: https://issues.apache.org/jira/browse/FLINK-10900
> Project: Flink
> Issue Type: Task
> Components: Kafka Connector
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Piotr Nowojski
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> Given the test problems with the Kafka 2.0 connector we should mark this
> connector as a beta feature until we have fully understood why so many tests
> deadlock.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10900) Mark Kafka 2.0 connector as beta feature
[
https://issues.apache.org/jira/browse/FLINK-10900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688236#comment-16688236
]
ASF GitHub Bot commented on FLINK-10900:
yanghua commented on a change in pull request #7109: [FLINK-10900][kafka][docs]
Mark universal Kafka connector as beta
URL: https://github.com/apache/flink/pull/7109#discussion_r233892269
##
File path: docs/dev/connectors/kafka.md
##
@@ -73,21 +73,33 @@ For most users, the `FlinkKafkaConsumer08` (part of
`flink-connector-kafka`) is
This connector supports https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+timestamps+to+Kafka+message;>Kafka
messages with timestamps both for producing and consuming.
-flink-connector-kafka-0.11_2.11
+flink-connector-kafka-0.11{{ site.scala_version_suffix }}
1.4.0
FlinkKafkaConsumer011
FlinkKafkaProducer011
0.11.x
Since 0.11.x Kafka does not support scala 2.10. This connector
supports https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging;>Kafka
transactional messaging to provide exactly once semantic for the
producer.
-flink-connector-kafka_2.11
+flink-connector-kafka{{ site.scala_version_suffix }}
1.7.0
FlinkKafkaConsumer
FlinkKafkaProducer
>= 1.0.0
-This Kafka connector attempts to track the latest version of the
Kafka client. The version of the client it uses may change between Flink
releases. Modern Kafka clients are backwards compatible with broker versions
0.10.0 or later. However for Kafka 0.11.x and 0.10.x versions, we recommend
using dedicated flink-connector-kafka-0.11 and link-connector-kafka-0.10
respectively.
-
+
+This universal Kafka connector attempts to track the latest version of
the Kafka client.
+The version of the client it uses may change between Flink releases.
+Modern Kafka clients are backwards compatible with broker versions
0.10.0 or later.
+However for Kafka 0.11.x and 0.10.x versions, we recommend using
dedicated
+flink-connector-kafka-0.11 and link-connector-kafka-0.10 respectively.
+
+ Attention: as of Flink 1.7 the universal Kafka
connector is considered to be
+ in a BETA status and might not be as stable as the
0.11 connector.
+ In case of problems with the universal connector, you can try to use
flink-connector-kafka-0.11{{ site.scala_version_suffix }}
+ which should be compatible with all of the Kafka versions starting
from 0.11.
Review comment:
Here, maybe not. Do you remember the producer transaction uses reflect to
access the inner private field ([sequenceNumbers 0.11 -> nextSequence
1.0+](https://github.com/apache/kafka/commit/5d2422258cb975a137a42a4e08f03573c49a387e#diff-f4ef1afd8792cd2a2e9069cd7ddea630))
?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Mark Kafka 2.0 connector as beta feature
>
>
> Key: FLINK-10900
> URL: https://issues.apache.org/jira/browse/FLINK-10900
> Project: Flink
> Issue Type: Task
> Components: Kafka Connector
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Priority: Blocker
> Labels: pull-request-available
>
> Given the test problems with the Kafka 2.0 connector we should mark this
> connector as a beta feature until we have fully understood why so many tests
> deadlock.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10900) Mark Kafka 2.0 connector as beta feature
[
https://issues.apache.org/jira/browse/FLINK-10900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688234#comment-16688234
]
ASF GitHub Bot commented on FLINK-10900:
yanghua commented on a change in pull request #7109: [FLINK-10900][kafka][docs]
Mark universal Kafka connector as beta
URL: https://github.com/apache/flink/pull/7109#discussion_r233888491
##
File path: docs/dev/connectors/kafka.md
##
@@ -73,21 +73,33 @@ For most users, the `FlinkKafkaConsumer08` (part of
`flink-connector-kafka`) is
This connector supports https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+timestamps+to+Kafka+message;>Kafka
messages with timestamps both for producing and consuming.
-flink-connector-kafka-0.11_2.11
+flink-connector-kafka-0.11{{ site.scala_version_suffix }}
1.4.0
FlinkKafkaConsumer011
FlinkKafkaProducer011
0.11.x
Since 0.11.x Kafka does not support scala 2.10. This connector
supports https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging;>Kafka
transactional messaging to provide exactly once semantic for the
producer.
-flink-connector-kafka_2.11
+flink-connector-kafka{{ site.scala_version_suffix }}
1.7.0
FlinkKafkaConsumer
FlinkKafkaProducer
>= 1.0.0
-This Kafka connector attempts to track the latest version of the
Kafka client. The version of the client it uses may change between Flink
releases. Modern Kafka clients are backwards compatible with broker versions
0.10.0 or later. However for Kafka 0.11.x and 0.10.x versions, we recommend
using dedicated flink-connector-kafka-0.11 and link-connector-kafka-0.10
respectively.
-
+
+This universal Kafka connector attempts to track the latest version of
the Kafka client.
+The version of the client it uses may change between Flink releases.
+Modern Kafka clients are backwards compatible with broker versions
0.10.0 or later.
+However for Kafka 0.11.x and 0.10.x versions, we recommend using
dedicated
+flink-connector-kafka-0.11 and link-connector-kafka-0.10 respectively.
Review comment:
Shall we also add `{{ site.scala_version_suffix }}` suffix here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Mark Kafka 2.0 connector as beta feature
>
>
> Key: FLINK-10900
> URL: https://issues.apache.org/jira/browse/FLINK-10900
> Project: Flink
> Issue Type: Task
> Components: Kafka Connector
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Priority: Blocker
> Labels: pull-request-available
>
> Given the test problems with the Kafka 2.0 connector we should mark this
> connector as a beta feature until we have fully understood why so many tests
> deadlock.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10900) Mark Kafka 2.0 connector as beta feature
[ https://issues.apache.org/jira/browse/FLINK-10900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688235#comment-16688235 ] ASF GitHub Bot commented on FLINK-10900: yanghua commented on a change in pull request #7109: [FLINK-10900][kafka][docs] Mark universal Kafka connector as beta URL: https://github.com/apache/flink/pull/7109#discussion_r233893714 ## File path: docs/dev/connectors/kafka.md ## @@ -110,13 +123,17 @@ Note that the streaming connectors are currently not part of the binary distribu ## Kafka 1.0.0+ Connector -Starting with Flink 1.7, there is a new Kafka connector that does not track a specific Kafka major version. Rather, it tracks the latest version of Kafka at the time of the Flink release. +Starting with Flink 1.7, there is a new Kafka connector that does not track a specific Kafka major version. +Rather, it tracks the latest version of Kafka at the time of the Flink release. -If your Kafka broker version is 1.0.0 or newer, you should use this Kafka connector. If you use an older version of Kafka (0.11, 0.10, 0.9, or 0.8), you should use the connector corresponding to the broker version. +If your Kafka broker version is 1.0.0 or newer, you should use this Kafka connector. +If you use an older version of Kafka (0.11, 0.10, 0.9, or 0.8), you should use the connector corresponding to the broker version. ### Compatibility -The modern Kafka connector is compatible with older and newer Kafka brokers through the compatibility guarantees of the Kafka client API and broker. The modern Kafka connector is compatible with broker versions 0.11.0 or later, depending on the features used. For details on Kafka compatibility, please refer to the [Kafka documentation](https://kafka.apache.org/protocol.html#protocol_compatibility). +The modern Kafka connector is compatible with older and newer Kafka brokers through the compatibility guarantees of the Kafka client API and broker. Review comment: I think we should give this connector a consistent name, either `universal ` or `modern`. Not both. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Mark Kafka 2.0 connector as beta feature > > > Key: FLINK-10900 > URL: https://issues.apache.org/jira/browse/FLINK-10900 > Project: Flink > Issue Type: Task > Components: Kafka Connector >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Priority: Blocker > Labels: pull-request-available > > Given the test problems with the Kafka 2.0 connector we should mark this > connector as a beta feature until we have fully understood why so many tests > deadlock. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10900) Mark Kafka 2.0 connector as beta feature
[ https://issues.apache.org/jira/browse/FLINK-10900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688156#comment-16688156 ] ASF GitHub Bot commented on FLINK-10900: pnowojski commented on issue #7109: [FLINK-10900][kafka][docs] Mark universal Kafka connector as beta URL: https://github.com/apache/flink/pull/7109#issuecomment-439061104 CC @tzulitai @yanghua This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Mark Kafka 2.0 connector as beta feature > > > Key: FLINK-10900 > URL: https://issues.apache.org/jira/browse/FLINK-10900 > Project: Flink > Issue Type: Task > Components: Kafka Connector >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Priority: Blocker > Labels: pull-request-available > > Given the test problems with the Kafka 2.0 connector we should mark this > connector as a beta feature until we have fully understood why so many tests > deadlock. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10900) Mark Kafka 2.0 connector as beta feature
[ https://issues.apache.org/jira/browse/FLINK-10900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688132#comment-16688132 ] ASF GitHub Bot commented on FLINK-10900: pnowojski opened a new pull request #7109: [FLINK-10900][kafka][docs] Mark universal Kafka connector as beta URL: https://github.com/apache/flink/pull/7109 Because of frequent deadlocks and other failures in tests the connector is marked as beta. Those errors are probably only test issues or Kafka bugs but as for now it's not confirmed. ## Verifying this change This is a change in documentation only. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Mark Kafka 2.0 connector as beta feature > > > Key: FLINK-10900 > URL: https://issues.apache.org/jira/browse/FLINK-10900 > Project: Flink > Issue Type: Task > Components: Kafka Connector >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Priority: Blocker > Labels: pull-request-available > > Given the test problems with the Kafka 2.0 connector we should mark this > connector as a beta feature until we have fully understood why so many tests > deadlock. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
