Qiang Yang created SPARK-45205:
----------------------------------
Summary: Since version 3.2.0, Spark SQL has taken longer to
execute "show paritions",probably because of changes introduced by SPARK-35278
Key: SPARK-45205
URL: https://issues.apache.org/jira/browse/SPARK-45205
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.2.0
Reporter: Qiang Yang
After SPARK-35378 was changed, I noticed that the execution of statements such
as ‘show parititions test' became slower.
The change point is that the execution process changes from ExecutedCommandEnec
to CommandResultExec, but ExecutedCommandExec originally implemented the
following method
override def executeToIterator(): Iterator[InternalRow] =
sideEffectResult.iterator
CommandResultExec is not rewritten, so when the hasNext method is executed, a
job process is created, resulting in increased time-consuming
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
