gaborgsomogyi commented on a change in pull request #27146:
[SPARK-21869][SS][DOCS][FOLLOWUP] Document Kafka producer pool configuration
URL: https://github.com/apache/spark/pull/27146#discussion_r365151930
##########
File path: docs/structured-streaming-kafka-integration.md
##########
@@ -802,6 +802,31 @@ df.selectExpr("topic", "CAST(key AS STRING)", "CAST(value
AS STRING)") \
</div>
</div>
+### Producer Caching
+
+Given Kafka producer instance is designed to be thread-safe, Spark initializes
a Kafka producer instance and co-use across tasks for same caching key.
+
+The caching key is built up from the following information:
+
+* Kafka producer configuration
+
+The following properties are available to configure the producer pool:
+
+<table class="table">
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+<tr>
+ <td>spark.kafka.producer.cache.timeout</td>
+ <td>The minimum amount of time a producer may sit idle in the pool before it
is eligible for eviction by the evictor.</td>
+ <td>10m (10 minutes)</td>
+</tr>
+<tr>
+ <td>spark.kafka.producer.cache.evictorThreadRunInterval</td>
+ <td>The interval of time between runs of the idle evictor thread for
producer pool. When non-positive, no idle evictor thread will be run.</td>
+ <td>1m (1 minutes)</td>
Review comment:
s/1 minutes/1 minute
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
