dbtsai commented on PR #55017: URL: https://github.com/apache/spark/pull/55017#issuecomment-4569347704
Sorry for the delay — the changes are now pushed (commit da6ae80f23b on `dfCache`). To summarise what's in place: **Config toggle**: `spark.sql.inMemoryColumnarStorage.enableDatasourceV2` (default `true`). Setting it to `false` falls back to the pre-DSv2 `InMemoryScans` planner path, which is unchanged from before this PR. **Migration strategy**: Both the DSv2 path and the fallback path ultimately produce `InMemoryTableScanExec` as the physical node, so existing workloads are transparent to the change at the execution level. The main user-visible difference is in `EXPLAIN` / `withCachedData` output, which shows `DataSourceV2ScanRelation` instead of `InMemoryRelation` under the new path. Users or tooling that inspect plan structure can set the config to `false` to restore the old shape. I'm happy to flip the default to `false` (opt-in) if you and @viirya prefer a more conservative rollout — just let me know. --- _This comment was generated with [GitHub MCP](http://go/mcps)._ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
