koeninger commented on issue #23747: [SPARK-26848][SQL] Introduce new option to Kafka source: offset by timestamp (starting/ending) URL: https://github.com/apache/spark/pull/23747#issuecomment-463247595 Why not just link to e.g. https://kafka.apache.org/21/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#offsetsForTimes-java.util.Map- The main kafka documentation at https://kafka.apache.org/documentation/ has plenty of discussion of timestamp imho On Wed, Feb 13, 2019 at 4:01 AM Gabor Somogyi <[email protected]> wrote: > > @gaborgsomogyi commented on this pull request. > > ________________________________ > > In docs/structured-streaming-kafka-integration.md: > > > @@ -310,6 +310,23 @@ The following configurations are optional: > > <table class="table"> > <tr><th>Option</th><th>value</th><th>default</th><th>query type</th><th>meaning</th></tr> > +<tr> > > I agree too detailed explanation is bad/redundant and that's the reason why I've suggested compacted limitations. > > Stating The start point of timestamp when a query is started is just not true because producer can overwrite this field under some circumstances (not Spark is the only producer). I would refer to Kafka configuration and mention that this field is depending on how Kafka is configured / data produced without mentioning all the details. I think based on the field users can look for KIP in need. > > Yeah, offsetsForTimes is not over-documented but that's a Kafka question. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub, or mute the thread.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
