HyukjinKwon commented on PR #59943: URL: https://github.com/apache/spark/pull/59943#issuecomment-6676160122
Thank you, @zikangh and @dongjoon-hyun. 0072b09 addresses the latest comments; the replies are in the threads. - The subtrees without a streaming relation, such as a static side of a join, are placeholders of their outputs when the query starts, so nothing of them folds into the plan, whatever is cached then or later. - The scan of all the columns of a table with `STREAMING_COLUMN_PRUNING` fails the query when it starts with `STREAMING_SCAN_INVALID_READ_SCHEMA` if its read schema does not follow the relation, rather than falling back. - `reads` is `requiredColumnsOfSources`, with a shorter doc, and `toRelation` is gone. - `spark.sql.streaming.v2ColumnPruning.enabled` is public, and the scaladoc of `scanOutput` says that a planner must use it. - A new `StreamingV2ScanPushDownRealTimeSuite` covers the column pruning of a real-time query. `StreamingV2ScanPushDownSuite`, the new real-time suite, and the streaming, continuous, real-time, DSv2 and columnar suites pass locally, as does `SparkThrowableSuite`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
