arunmahadevan edited a comment on issue #23749: [SPARK-26841][SQL] Kafka timestamp pushdown URL: https://github.com/apache/spark/pull/23749#issuecomment-473358007 IMO, we could adopt the more generic solution. If its just timestamp based filtering of start and end offsets maybe the timestamp offset approach proposed in #23747 looks straightforward. However if there are more cases that the filter pushdown might be able to handle we should go with that. I am assuming the filter condition can appear anywhere in the query and get pushed down to filter the rows and if so looks more generic and avoids having to add extra options to the kafka source. I am not sure providing two different options for timestamp based filtering is necessary. If we support both, the user can provide different values via the options and the filter and it gets very confusing.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
