LvDongrong created SPARK-19863: ---------------------------------- Summary: Whether or not use CachedKafkaConsumer need to be configured, when you use DirectKafkaInputDStream to connect the kafka in a Spark Streaming application Key: SPARK-19863 URL: https://issues.apache.org/jira/browse/SPARK-19863 Project: Spark Issue Type: Bug Components: DStreams, Input/Output Affects Versions: 2.1.0 Reporter: LvDongrong
Whether or not use CachedKafkaConsumer need to be configured, when you use DirectKafkaInputDStream to connect the kafka in a Spark Streaming application. In Spark 2.x, the kafka consumer was replaced by CachedKafkaConsumer (some KafkaConsumer will keep establishing the kafka cluster), and cannot change the way. In fact ,The KafkaRDD(used by DirectKafkaInputDStream to connect kafka) provide the parameter useConsumerCache to choose Whether to use the CachedKafkaConsumer, but the DirectKafkaInputDStream set the parameter true. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org