Kavpreet Grewal created SPARK-55300:
---------------------------------------
Summary: Visualize the behavior of startingOffset/endingOffset
option in Kafka connector
Key: SPARK-55300
URL: https://issues.apache.org/jira/browse/SPARK-55300
Project: Spark
Issue Type: Improvement
Components: Documentation
Affects Versions: 4.2
Reporter: Kavpreet Grewal
The guide doc of Kafka connector describes the behavior of
startingTimestamp/endingTimestamp and related timestamp offset options as
follows:
[Structured Streaming + Kafka Integration Guide (Kafka broker version 0.10.0 or
higher) - Spark 4.1.0
Documentation|https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html]
{quote}The start point of timestamp when a query is started, a string
specifying a starting timestamp for all partitions in topics being subscribed.
Please refer the details on timestamp offset options below. If Kafka doesn't
return the matched offset, the behavior will follow to the value of the option
{{startingOffsetsByTimestampStrategy}}
{quote}
which can be illustrated in (ASCII style).
We got some feedback from users that the text explanation in the guide doc may
not be sufficient to come up with an illustration in their mind. Their feedback
is something like “A picture is worth a thousand words”. So it would be great
to visualize this and update the guide doc.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
