[jira] [Assigned] (SPARK-26848) Introduce new option to Kafka source - specify timestamp to start and end offset
[ https://issues.apache.org/jira/browse/SPARK-26848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-26848: - Assignee: Jungtaek Lim > Introduce new option to Kafka source - specify timestamp to start and end > offset > > > Key: SPARK-26848 > URL: https://issues.apache.org/jira/browse/SPARK-26848 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Major > > Kafka source (for SQL/SS) provides options to set specific offset per topic > partition to let source starts reading from start offsets and ends reading > until end offsets. ("startingOffsets" and "endingOffsets" in below document.) > http://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html > I'd like to introduce new options "startingOffsetsByTimestamp" and > "endingOffsetsByTimestamp" to set specific timestamp per topic (since we're > unlikely to set the different value per partition) to let source starts > reading from offsets which have equal of greater timestamp, and ends reading > until offsets which have equal of greater timestamp. > The option would be optional of course, and have a preference to apply the > options 1) timestamp option applies first, and if it doesn't exist, offset > option applies. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-26848) Introduce new option to Kafka source - specify timestamp to start and end offset
[ https://issues.apache.org/jira/browse/SPARK-26848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26848: Assignee: Apache Spark > Introduce new option to Kafka source - specify timestamp to start and end > offset > > > Key: SPARK-26848 > URL: https://issues.apache.org/jira/browse/SPARK-26848 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Jungtaek Lim >Assignee: Apache Spark >Priority: Major > > Kafka source (for SQL/SS) provides options to set specific offset per topic > partition to let source starts reading from start offsets and ends reading > until end offsets. ("startingOffsets" and "endingOffsets" in below document.) > http://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html > I'd like to introduce new options "startingOffsetsByTimestamp" and > "endingOffsetsByTimestamp" to set specific timestamp per topic (since we're > unlikely to set the different value per partition) to let source starts > reading from offsets which have equal of greater timestamp, and ends reading > until offsets which have equal of greater timestamp. > The option would be optional of course, and have a preference to apply the > options 1) timestamp option applies first, and if it doesn't exist, offset > option applies. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-26848) Introduce new option to Kafka source - specify timestamp to start and end offset
[ https://issues.apache.org/jira/browse/SPARK-26848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26848: Assignee: (was: Apache Spark) > Introduce new option to Kafka source - specify timestamp to start and end > offset > > > Key: SPARK-26848 > URL: https://issues.apache.org/jira/browse/SPARK-26848 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Jungtaek Lim >Priority: Major > > Kafka source (for SQL/SS) provides options to set specific offset per topic > partition to let source starts reading from start offsets and ends reading > until end offsets. ("startingOffsets" and "endingOffsets" in below document.) > http://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html > I'd like to introduce new options "startingOffsetsByTimestamp" and > "endingOffsetsByTimestamp" to set specific timestamp per topic (since we're > unlikely to set the different value per partition) to let source starts > reading from offsets which have equal of greater timestamp, and ends reading > until offsets which have equal of greater timestamp. > The option would be optional of course, and have a preference to apply the > options 1) timestamp option applies first, and if it doesn't exist, offset > option applies. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org