HyukjinKwon commented on PR #59943: URL: https://github.com/apache/spark/pull/59943#issuecomment-6671822924
Thank you, @dongjoon-hyun. c8491d3 addresses the ten comments; the replies are in the threads. In short: - Every expression whose type has a date or a timestamp at any depth, and one of whose children has a string, a time or a variant at any depth, becomes a placeholder when the query starts. That covers the nested casts of times, the casts of variants, `from_json` and `variant_get`, and it replaces the list of folds, with no more calls to `ComputeCurrentTime` or `SpecialDatetimeValues`. - Every scan of a table with the capability is read in the order of its read schema, also a scan of all the columns. A read schema with a column of another type falls back to a scan of all the columns, and the scan node keeps the types and the nullability of the relation. - The stream is created outside of the fallback, so a failure to create it fails the query. - A micro-batch that reads another column, as after a change of the configuration of the optimizer in `foreachBatch`, fails with the new `STREAMING_SCAN_MISSING_COLUMNS`, which says what to do, and the docs mention it. - The in-memory tables only declare the capability with a boolean property, which only the tables of this suite set. --- <!-- GITHUB_MCP_FOOTER: This attribution is automatically appended by GitHub MCP. --> _This comment was generated with [GitHub MCP](http://go/mcps)._ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
