HyukjinKwon commented on PR #59943: URL: https://github.com/apache/spark/pull/59943#issuecomment-6643592634
Thank you, @dongjoon-hyun. 1873787 addresses the sixth round; the replies are in the threads. The casts of special strings such as `'today'` and `'now'` are placeholders at the start now, as the time values are, also behind `to_date`, so neither the pushed filters nor the columns of a scan depend on the day that the query started. The filters of a read are selected as `PushPredicateThroughNonJoin` splits them, so a filter that may throw is not pushed after one that is not: your `raise_error` query failed with `USER_RAISED_EXCEPTION` with the previous selection and runs now. The test tables can drop the rows that fail the pushed filters, in micro-batch and continuous streams, so the answers and `numInputRows` of the new tests depend on the pushdown. The pushdown suite and the streaming and DSv2 suites pass locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
