[
https://issues.apache.org/jira/browse/SPARK-30169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991196#comment-16991196
]
Jungtaek Lim commented on SPARK-30169:
--------------------------------------
Note that this is not just replacing old deprecated method with new one. Kafka
made a huge change between old poll(long) and new poll(Duration).
Former doesn't count the duration for metadata update into account - it's
blocking though we provide timeout (that's how poll(0) works and the intention
of why we call poll(0)). Latter takes metadata update into account of timeout,
so we should give higher enough timeout to new poll. Simply migrating long to
Duration would result in unexpected behavior.
That's why we haven't dealt with "deprecated" in spark-sql-kafka module. We've
already filed couple of JIRA issues to deal with `poll` and `ZkUtils`. (Btw, it
would be great for anyone to deal with ZkUtils as we got lost and don't work
for such issue for now.)
> Eliminate warnings in Kafka connector
> -------------------------------------
>
> Key: SPARK-30169
> URL: https://issues.apache.org/jira/browse/SPARK-30169
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Maxim Gekk
> Priority: Minor
>
> *Eliminate compilation warnings (10) in module external/kafka-0-10*:
> 1.
> /home/root1/spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala
> {code:java}
> Line:108
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> Line:162
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 2.
> /home/root1/spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaDataConsumer.scala
> {code:java}
> Line:206
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 3.
> /home/root1/spark/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala
> {code:java}
> Line:258
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> Line:314
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> Line:476
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 4.
> /home/root1/spark/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
> {code:java}
> Line:60
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:88
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:100
> object ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:178
> method createTopic in object AdminUtils is deprecated (since 1.1.0): This
> method is deprecated and will be replaced by kafka.zk.AdminZkClient.
> {code}
> *Eliminate compilation warnings (13) in module external/kafka-0-10-sql*:
> 1.
> /home/root1/spark/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
> {code}
> Line:74
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 2.
> /home/root1/spark/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala
> {code}
> Line:119
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> Line:382
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> {code}
> 3.
> /home/root1/spark/external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala
> {code}
> Line:84
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:118
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:246
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line: 458
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> Line: 472
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> Line: 581
> object ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line: 582
> object ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line: 583
> object ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line: 583
> object ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line: 607
> object ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
