[jira] [Updated] (SPARK-25731) Spark Structured Streaming Support for Kafka 2.0
[ https://issues.apache.org/jira/browse/SPARK-25731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandan updated SPARK-25731: Labels: beginner features (was: ) > Spark Structured Streaming Support for Kafka 2.0 > > > Key: SPARK-25731 > URL: https://issues.apache.org/jira/browse/SPARK-25731 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming >Affects Versions: 2.3.2 >Reporter: Chandan >Priority: Major > Labels: beginner, features > > [https://github.com/apache/spark/tree/master/external] > As far as I can see, > This doesn't have support for newly release *kafka2.0,* > support is available only till *kafka-0-10.* > If we use the > "org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.3.0" > for kafka2.0, below is the error I get > 11:46:18.061 [stream execution thread for [id = > e393ea37-8009-4ce0-b996-94f767994fb8, runId = > bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG > org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2, > groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] > *Completed connection to node -1. Fetching API versions.* > 11:46:18.061 [stream execution thread for [id = > e393ea37-8009-4ce0-b996-94f767994fb8, runId = > bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG > org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2, > groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] > *Initiating API versions fetch from node -1.* > 11:46:18.452 [stream execution thread for [id = > e393ea37-8009-4ce0-b996-94f767994fb8, runId = > bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG > org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-2, > groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] > Connection with *kafka-muhammad-45e0.aivencloud.com/18.203.67.147 > disconnected* > *java.io.EOFException: null* > at > org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:119) > at > org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:335) > at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:296) > > > I might be wrong, but this is the best option I thought to open an issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-25731) Spark Structured Streaming Support for Kafka 2.0
[ https://issues.apache.org/jira/browse/SPARK-25731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandan updated SPARK-25731: Description: [https://github.com/apache/spark/tree/master/external] As far as I can see, This doesn't have support for newly release *kafka2.0,* support is available only till *kafka-0-10.* If we use the "org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.3.0" for kafka2.0, below is the error I get 11:46:18.061 [stream execution thread for [id = e393ea37-8009-4ce0-b996-94f767994fb8, runId = bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2, groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] *Completed connection to node -1. Fetching API versions.* 11:46:18.061 [stream execution thread for [id = e393ea37-8009-4ce0-b996-94f767994fb8, runId = bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2, groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] *Initiating API versions fetch from node -1.* 11:46:18.452 [stream execution thread for [id = e393ea37-8009-4ce0-b996-94f767994fb8, runId = bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-2, groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] Connection with *kafka-muhammad-45e0.aivencloud.com/18.203.67.147 disconnected* *java.io.EOFException: null* at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:119) at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:335) at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:296) I might be wrong, but this is the best option I thought to open an issue. was: [https://github.com/apache/spark/tree/master/external] As far as I can see, This doesn't have support for newly release *kafka2.0,* support is available only till *kafka-0-10.* If we use the "org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.3.0" for kafka2.0, below is the error I get 11:46:18.061 [stream execution thread for [id = e393ea37-8009-4ce0-b996-94f767994fb8, runId = bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2, groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] *Completed connection to node -1. Fetching API versions.* 11:46:18.061 [stream execution thread for [id = e393ea37-8009-4ce0-b996-94f767994fb8, runId = bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2, groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] *Initiating API versions fetch from node -1.* 11:46:18.452 [stream execution thread for [id = e393ea37-8009-4ce0-b996-94f767994fb8, runId = bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-2, groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] Connection with *kafka-muhammad-45e0.aivencloud.com/18.203.67.147 disconnected* *java.io.EOFException: null* at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:119) at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:335) at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:296) > Spark Structured Streaming Support for Kafka 2.0 > > > Key: SPARK-25731 > URL: https://issues.apache.org/jira/browse/SPARK-25731 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming >Affects Versions: 2.3.2 >Reporter: Chandan >Priority: Major > > [https://github.com/apache/spark/tree/master/external] > As far as I can see, > This doesn't have support for newly release *kafka2.0,* > support is available only till *kafka-0-10.* > If we use the > "org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.3.0" > for kafka2.0, below is the error I get > 11:46:18.061 [stream execution thread for [id = > e393ea37-8009-4ce0-b996-94f767994fb8, runId = > bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG > org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2, > groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] > *Completed connection to node -1. Fetching API versions.* > 11:46:18.061 [stream execution thread for [id = > e393ea37-8009-4ce0-b996-94f767994fb8, runId = > bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG > org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2, >
[jira] [Updated] (SPARK-25731) Spark Structured Streaming Support for Kafka 2.0
[ https://issues.apache.org/jira/browse/SPARK-25731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandan updated SPARK-25731: Description: [https://github.com/apache/spark/tree/master/external] As far as I can see, This doesn't have support for newly release *kafka2.0,* support is available only till *kafka-0-10.* If we use the "org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.3.0" for kafka2.0, below is the error I get 11:46:18.061 [stream execution thread for [id = e393ea37-8009-4ce0-b996-94f767994fb8, runId = bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2, groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] *Completed connection to node -1. Fetching API versions.* 11:46:18.061 [stream execution thread for [id = e393ea37-8009-4ce0-b996-94f767994fb8, runId = bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2, groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] *Initiating API versions fetch from node -1.* 11:46:18.452 [stream execution thread for [id = e393ea37-8009-4ce0-b996-94f767994fb8, runId = bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-2, groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] Connection with *kafka-muhammad-45e0.aivencloud.com/18.203.67.147 disconnected* *java.io.EOFException: null* at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:119) at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:335) at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:296) was: [https://github.com/apache/spark/tree/master/external] As far as I can see, This doesn't have support for newly release kafka2.0, support is available only till kafka-0-10. If we use the "org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.3.0" for kafka2.0, below is the error I get 11:46:18.061 [stream execution thread for [id = e393ea37-8009-4ce0-b996-94f767994fb8, runId = bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2, groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] *Completed connection to node -1. Fetching API versions.* 11:46:18.061 [stream execution thread for [id = e393ea37-8009-4ce0-b996-94f767994fb8, runId = bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2, groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] *Initiating API versions fetch from node -1.* 11:46:18.452 [stream execution thread for [id = e393ea37-8009-4ce0-b996-94f767994fb8, runId = bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-2, groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] Connection with *kafka-muhammad-45e0.aivencloud.com/18.203.67.147 disconnected* *java.io.EOFException: null* at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:119) at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:335) at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:296) > Spark Structured Streaming Support for Kafka 2.0 > > > Key: SPARK-25731 > URL: https://issues.apache.org/jira/browse/SPARK-25731 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming >Affects Versions: 2.3.2 >Reporter: Chandan >Priority: Major > > [https://github.com/apache/spark/tree/master/external] > As far as I can see, > This doesn't have support for newly release *kafka2.0,* > support is available only till *kafka-0-10.* > If we use the > "org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.3.0" > for kafka2.0, below is the error I get > 11:46:18.061 [stream execution thread for [id = > e393ea37-8009-4ce0-b996-94f767994fb8, runId = > bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG > org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2, > groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] > *Completed connection to node -1. Fetching API versions.* > 11:46:18.061 [stream execution thread for [id = > e393ea37-8009-4ce0-b996-94f767994fb8, runId = > bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG > org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2, > groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] > *Initiating API versions fetch from node -1.* >
