ulysses you created SPARK-29966:
-----------------------------------
Summary: Add version method in TableCatalog to avoid load table
twice
Key: SPARK-29966
URL: https://issues.apache.org/jira/browse/SPARK-29966
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.0.0
Reporter: ulysses you
Now resolve logic plan will load table twice which are in ResolveTables and
ResolveRelations. The ResolveRelations is old code path, and ResolveTables is
v2 code path, and the reason why load table twice is that ResolveTables will
load table and rollback v1 table to ResolveRelations code path.
The same scene also exists in ResolveSessionCatalog.
It affect that execute command will cost double time than spark 2.4.
Here is the idea that add a table version method in TableCatalog, and rules
should always get table version firstly without load table.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
