HeartSaVioR commented on issue #23749: [SPARK-26841][SQL] Kafka timestamp pushdown URL: https://github.com/apache/spark/pull/23749#issuecomment-572824425 As I commented in #25911 , this is the ideal way to go for dealing with use cases on Kafka batch query (especially interactive query). Unlike other sources, in most cases we can't leverage partition pruning from Kafka, and timestamp is the only one indexed by Kafka. Offset is definitely indexed, but would the value have any meaning in business logic? This patch adds lots of lines (which we tend to say "complexity" whereas I'm not sure I agree) but now it can be reduced as #23747 added similar functionality and now it's only the matter of pushdown. I'd take this over if we agree about the value.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
