Github user koeninger commented on a diff in the pull request:
https://github.com/apache/spark/pull/15679#discussion_r85642029
--- Diff: docs/streaming-kafka-0-10-integration.md ---
@@ -165,6 +240,36 @@ For data stores that support transactions, saving
offsets in the same transactio
}
</div>
<div data-lang="java" markdown="1">
+ // The details depend on your data store, but the general idea looks
like this
+
+ // begin from the the offsets committed to the database
+ Map<TopicPartition, Long> fromOffsets = new HashMap<>();
+ for (resultSet: selectOffsetsFromYourDatabase)
+ fromOffsets.put(new TopicPartition(resultSet.string("topic"),
resultSet.int("partition")), resultSet.long("offset"));
+ }
+
+ JavaInputDStream<ConsumerRecord<String, String>> stream =
KafkaUtils.<String, String>createDirectStream(
+ streamingContext,
+ LocationStrategies.PreferConsistent(),
+ ConsumerStrategies.<String, String>Assign(fromOffsets.keySet(),
kafkaParams, fromOffsets)
+ );
+
+ stream.foreachRDD(new VoidFunction<JavaRDD<ConsumerRecord<String,
String>>>() {
+ @Override
+ public void call(JavaRDD<ConsumerRecord<String, String>> rdd) {
+ OffsetRange[] offsetRanges = ((HasOffsetRanges)
rdd.rdd()).offsetRanges();
+
+ Object results = yourCalculation(rdd);
+
+ yourTransactionBlock {
--- End diff --
I agree with Sean, this would probably be clearer if it was changed to a
comment like
// begin your transaction
...
// end your transaction
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
