Naveen Kumar Puppala created SPARK-60610:
--------------------------------------------
Summary: FileSourceScanExec.bucketedScan re-reads
spark.sql.sources.bucketing.enabled on the copy made at stage creation
Key: SPARK-60610
URL: https://issues.apache.org/jira/browse/SPARK-60610
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.0
Reporter: Naveen Kumar Puppala
{{FileSourceScanExec.bucketedScan}} is a lazy val that reads the live session
config via {{{}getSqlConf(relation.sparkSession).bucketingEnabled{}}}, and
{{outputPartitioning}} depends on it. When the node is copied after planning
(for example the copy {{PlanAdaptiveDynamicPruningFilters}} makes at stage
creation), the lazy val re-evaluates on the copy and reads the config again. If
{{spark.sql.sources.bucketing.enabled}} is turned off between planning and
execution, the rebuilt scan reports a different output partitioning than the
plan was built with.
This is the V1 file-source analogue of SPARK-60260 (DataSource V2
storage-partitioned join) and SPARK-59252. Found by inspection during the
SPARK-60260 review ([apache/spark#59789 · [SPARK-60260][SQL] Keep
storage…|https://github.com/apache/spark/pull/59789], Peter Toth). A
reproduction confirming the resulting wrong results or failure is still to be
written.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
