HyukjinKwon commented on PR #59943: URL: https://github.com/apache/spark/pull/59943#issuecomment-6626865743
Thank you, @gengliangwang and @dongjoon-hyun. 9ef62c1 addresses this round; the replies are in the threads. In short: - **Time and session values**: the plan optimized at the start replaces every `CURRENT_LIKE` expression (including `now()` and `current_time()`) and the casts of times to timestamps with a placeholder that keeps their children and that the optimizer does not take as deterministic, so no filter on them is pushed down, in both modes. - **Pushed filters are checked**: `StreamingDataSourceV2ScanRelation.pushedFilters` records them, and the planner fails with an internal error if a micro-batch does not apply them, as for the columns. So that this holds for every valid micro-batch, the startup optimization no longer infers filters from constraints: a filter inferred from another source is not there in a micro-batch where that source has no new data. - **CTE**: I could not reproduce the copies. `DeduplicateRelations.deduplicateRight`, which `InlineCTE` and `ReplaceCTERefWithRepartition` use, returns streaming plans unchanged, so every reference has the same relation, and both are combined as before. Both examples are now tests and pass. If a later rule ever copies a streaming relation, nothing is pushed down. - **Python data sources** get no filter, and no longer run the startup optimization. - **Cost**: batch relations are placeholders at the start, so no batch scan, Python worker or partition listing happens there; the rest of the optimizer stays, since the checks need what the micro-batches compute. - **Fallback**: the read schema is checked before the stream is created, also when it throws, so a source creates a single stream. - **`ColumnarScan`**: its read schema no longer depends on the call order, and its streams only output the required columns. #59931 gets the same change. - Migration guide (input row metrics), string of the relation, and the test cleanups. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
