eaton created SPARK-22606:
-----------------------------
Summary: There may be two or more tasks in one executor will use
the same kafka consumer at the same time, then it will throw an exception:
"KafkaConsumer is not safe for multi-threaded access"
Key: SPARK-22606
URL: https://issues.apache.org/jira/browse/SPARK-22606
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 2.2.0
Reporter: eaton
Priority: Minor
If the value of param 'spark.streaming.concurrentJobs' is more than one, and
the value of param 'spark.executor.cores' is more than one, there may be two or
more tasks in one executor will use the same kafka consumer at the same time,
then it will throw an exception: "KafkaConsumer is not safe for multi-threaded
access"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
