HeartSaVioR commented on a change in pull request #32609: URL: https://github.com/apache/spark/pull/32609#discussion_r638541266
########## File path: docs/structured-streaming-kafka-integration.md ########## @@ -512,6 +526,17 @@ The following configurations are optional: </tr> </table> +### Details on timestamp offset options + +The returned offset for each partition is the earliest offset whose timestamp is greater than or equal to the given timestamp in the corresponding partition. +The behavior varies across options if the matched offset doesn't exist - check the description of each option. + +Spark simply passes the timestamp information to <code>KafkaConsumer.offsetsForTimes</code>, and doesn't interpret or reason about the value. +For more details on <code>KafkaConsumer.offsetsForTimes</code>, please refer <a href="https://kafka.apache.org/21/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#offsetsForTimes-java.util.Map-";>javadoc</a> for details. Review comment: That makes sense. Hopefully Kafka community looks to maintain all versions of doc so the risk of broken link is relatively low. I'll update the link to 0.10.1.x. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
