Repository: spark Updated Branches: refs/heads/branch-2.0 3e9840f1d -> d3c78c4f3
[STREAMING][KAFKA][DOC] clarify kafka settings needed for larger batches ## What changes were proposed in this pull request? Minor doc change to mention kafka configuration for larger spark batches. ## How was this patch tested? Doc change only, confirmed via jekyll. The configuration issue was discussed / confirmed with users on the mailing list. Author: cody koeninger <[email protected]> Closes #15570 from koeninger/kafka-doc-heartbeat. (cherry picked from commit c9720b2195a465653690b3e221ce789142217b0d) Signed-off-by: Shixiong Zhu <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d3c78c4f Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d3c78c4f Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d3c78c4f Branch: refs/heads/branch-2.0 Commit: d3c78c4f379a6ce3d055f935a6018d866606ebaf Parents: 3e9840f Author: cody koeninger <[email protected]> Authored: Fri Oct 21 16:27:19 2016 -0700 Committer: Shixiong Zhu <[email protected]> Committed: Fri Oct 21 16:27:26 2016 -0700 ---------------------------------------------------------------------- docs/streaming-kafka-0-10-integration.md | 1 + 1 file changed, 1 insertion(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/d3c78c4f/docs/streaming-kafka-0-10-integration.md ---------------------------------------------------------------------- diff --git a/docs/streaming-kafka-0-10-integration.md b/docs/streaming-kafka-0-10-integration.md index 456b845..de95ea9 100644 --- a/docs/streaming-kafka-0-10-integration.md +++ b/docs/streaming-kafka-0-10-integration.md @@ -48,6 +48,7 @@ Each item in the stream is a [ConsumerRecord](http://kafka.apache.org/0100/javad </div> For possible kafkaParams, see [Kafka consumer config docs](http://kafka.apache.org/documentation.html#newconsumerconfigs). +If your Spark batch duration is larger than the default Kafka heartbeat session timeout (30 seconds), increase heartbeat.interval.ms and session.timeout.ms appropriately. For batches larger than 5 minutes, this will require changing group.max.session.timeout.ms on the broker. Note that the example sets enable.auto.commit to false, for discussion see [Storing Offsets](streaming-kafka-0-10-integration.html#storing-offsets) below. ### LocationStrategies --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
