Asif created SPARK-59160:
----------------------------
Summary: The currentStageId initialized to 0, in the
AdaptiveSparkPlanExec constructor, may clash with existing stageIDs in the
AdaptiveExecutionContext
Key: SPARK-59160
URL: https://issues.apache.org/jira/browse/SPARK-59160
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.3, 4.3.0
Reporter: Asif
The counter is initialized in the constructor of AdaptiveSparkPlanExec as
private var currentStageId = 0
but the previous stages encountered in the subplans and cached in stageCache
field of AdaptiveExecutionContext are bound to clash with IDs generated in the
AdaptiveSparkPlanExec.
As of now in the stock spark code, those Ids are not used for lookups ( instead
its the canonicalized plans which are used), so there are no issues. ( atleast
there are no failing tests and possibly no scope of bug).
But as seen in my private fork due to my changes, the clashing IDs result in
bug.
May be to make the code safer for future , the currentStageId should be moved
as a var field in AdaptiveExecutionContext , to guarantee uniqueness among all
the stages seen within an AdaptiveExecutionContext.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
