HeartSaVioR commented on issue #23747: [SPARK-26848][SQL] Introduce new option to Kafka source: offset by timestamp (starting/ending) URL: https://github.com/apache/spark/pull/23747#issuecomment-476828021 Requiring all partitions to be specified is not only having flexibility, but also having limitation. If end users don't write query all the time and build its app jar, the query wouldn't work when they repartition the topic. It's not true for streaming query with checkpoint, but true for batch query and streaming query without existing checkpoint. If we are seeing benefit of both side, we may have special representation of `all partitions`, and still support per-partition. I can file as a follow-up issue and provide a new PR, given we all agree for per-partition and I don't want to drag this for adding new possibility. Will update to make per-partition be available.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
