Jungtaek Lim created SPARK-29223:
------------------------------------
Summary: Kafka source: offset by timestamp - allow specifying
timestamp for "all partitions"
Key: SPARK-29223
URL: https://issues.apache.org/jira/browse/SPARK-29223
Project: Spark
Issue Type: Improvement
Components: SQL, Structured Streaming
Affects Versions: 3.0.0
Reporter: Jungtaek Lim
This issue is a follow-up of SPARK-26848.
In SPARK-26848, we decided to open possibility to let end users set individual
timestamp per partition. But in many cases, specifying timestamp represents the
intention that we would want to go back to specific timestamp and reprocess
records, which should be applied to all topics and partitions.
According to the format ofÂ
`startingOffsetsByTimestamp`/`endingOffsetsByTimestamp`, while it's not
intuitive to provide an option to set a global timestamp across topic, it's
still intuitive to provide an option to set a global timestamp across
partitions in a topic.
This issue tracks the efforts to deal with this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
