HyukjinKwon commented on PR #59943: URL: https://github.com/apache/spark/pull/59943#issuecomment-6646798947
Thank you, @dongjoon-hyun. 160427c addresses the seven comments; the replies are in the threads. - A cast of a foldable string with only a time to a timestamp with a local time zone, such as `ts < '12:00:00'`, becomes a placeholder at startup, as the casts of the special strings do, so it is not pushed down and does not decide the columns of a scan. - The filters of a read are those up to the first one that is not pushed down, and with several reads the longest common prefix of their filters, so no pushed filter sees a row that a filter written before it drops, also for the filters that fail with ANSI. - The scan builder only gets, and each micro-batch only applies first, the filters that `PushDownUtils.pushFilters` gives it in some form. The others are evaluated after them, as for a batch scan. - `StreamingScan.toRelation` builds the scan relation for both executions, `PushDownUtils` has the checks of the builder interfaces next to `pushFilters`, and `passes` moved after the table properties. The suites of the PR pass locally: `StreamingV2ScanPushDownSuite`, the streaming, continuous, watermark and aggregation suites, `DataSourceV2Suite` and `ColumnarDataSourceSuite`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
