gaborgsomogyi commented on a change in pull request #23747: [SPARK-26848][SQL] Introduce new option to Kafka source: offset by timestamp (starting/ending) URL: https://github.com/apache/spark/pull/23747#discussion_r256321947
########## File path: docs/structured-streaming-kafka-integration.md ########## @@ -310,6 +310,23 @@ The following configurations are optional: <table class="table"> <tr><th>Option</th><th>value</th><th>default</th><th>query type</th><th>meaning</th></tr> +<tr> Review comment: I agree too detailed explanation is bad/redundant and that's the reason why I've suggested `compacted limitations`. Stating `The start point of timestamp when a query is started` is just not true because producer can overwrite this field under some circumstances (not Spark is the only producer). I would refer to Kafka configuration and mention that this field is depending on how Kafka is configured / data produced without mentioning all the details. I think based on the field users can look for KIP in need. Yeah, offsetsForTimes is not over-documented but that's a Kafka question. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
