Stephen0421 opened a new issue, #11121: URL: https://github.com/apache/paimon/issues/11121
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version 2.2-SNAPSHOT, built from the unmodified Apache Paimon master branch. ### Compute Engine Apache Spark 3.2.4, Scala 2.12, Java 8. The reproduction uses stock Apache Spark and community Paimon code. ### Minimal reproduce step The following reproduces an ordinary `INSERT INTO` failure using three partitioned Parquet inputs and two chained `LEFT JOIN`s. The destination is a partitioned primary-key table with dynamic buckets. Start a fresh Spark shell with the Paimon Spark 3.2 bundle built from the commit above: ```bash spark-shell --master 'local[2]' \ --jars /path/to/paimon-spark-3.2_2.12-2.2-SNAPSHOT.jar \ --conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions ``` Run the following Scala code. For multiline input in `spark-shell`, use `:paste` and finish with Ctrl-D. ```scala import java.io.File import java.nio.file.Files import org.apache.spark.sql.catalyst.expressions.DynamicPruningSubquery import spark.implicits._ val tempDir = Files.createTempDirectory("paimon-dpp-repro").toFile spark.conf.set( "spark.sql.catalog.paimon", "org.apache.paimon.spark.SparkCatalog") spark.conf.set( "spark.sql.catalog.paimon.warehouse", new File(tempDir, "warehouse").toURI.toString) Seq( "spark.paimon.write.use-v2-write" -> "false", "spark.sql.optimizer.dynamicPartitionPruning.enabled" -> "true", "spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly" -> "false", "spark.sql.optimizer.dynamicPartitionPruning.useStats" -> "false", "spark.sql.optimizer.dynamicPartitionPruning.fallbackFilterRatio" -> "1.0", "spark.sql.adaptive.enabled" -> "false", "spark.sql.shuffle.partitions" -> "2" ).foreach { case (key, value) => spark.conf.set(key, value) } spark.sql("CREATE DATABASE paimon.dpp_repro") def createTarget(name: String): Unit = { spark.sql(s""" CREATE TABLE paimon.dpp_repro.$name ( id INT, pt STRING, left_value INT, right_value INT ) PARTITIONED BY (pt) TBLPROPERTIES ( 'primary-key' = 'id,pt', 'bucket' = '-1', 'dynamic-bucket.assigner-parallelism' = '2' ) """) } createTarget("dpp_target") val rows = Seq((1, "p1", 10), (2, "p2", 20), (3, "p3", 30)) val extraRows = (4 until 104).map(i => (i, "p4", i)) Seq("dpp_source", "dpp_left", "dpp_right").foreach { name => val input = name match { case "dpp_source" => rows case "dpp_left" => rows ++ extraRows case _ => rows.filterNot(_._1 == 2) ++ extraRows } val path = new File(tempDir, name).getCanonicalPath input.toDF("id", "pt", "metric") .write.partitionBy("pt").parquet(path) spark.read.parquet(path).createOrReplaceTempView(name) } val query = """ SELECT a.id, a.pt, b.metric AS left_value, c.metric AS right_value FROM dpp_source a LEFT JOIN dpp_left b ON a.id = b.id AND a.pt = b.pt LEFT JOIN dpp_right c ON a.id = c.id AND a.pt = c.pt WHERE a.id < 3 """ val plan = spark.sql(query).queryExecution.optimizedPlan val pruning = plan.flatMap(_.expressions).flatMap { expression => expression.collect { case s: DynamicPruningSubquery => s } } assert(pruning.nonEmpty, s"Expected DPP in the input plan:\n$plan") // Fails on the unmodified baseline. spark.sql(s"INSERT INTO paimon.dpp_repro.dpp_target $query") ``` After the failure, disable DPP and run the same insert into a fresh target: ```scala spark.conf.set("spark.sql.optimizer.dynamicPartitionPruning.enabled", "false") createTarget("dpp_target_without_dpp") spark.sql(s"INSERT INTO paimon.dpp_repro.dpp_target_without_dpp $query") spark.sql(""" SELECT * FROM paimon.dpp_repro.dpp_target_without_dpp ORDER BY id """).show() ``` The DPP-disabled control succeeds and returns the expected rows. The data, SQL, and key settings above were verified in regression tests. The standalone Spark shell adaptation has not been separately executed. ### What doesn't meet your expectations? Expected: the insert succeeds with DPP enabled and writes the same rows as the DPP-disabled execution: ```text +---+---+----------+-----------+ | id| pt|left_value|right_value| +---+---+----------+-----------+ | 1| p1| 10| 10| | 2| p2| 20| null| +---+---+----------+-----------+ ``` Actual: with DPP enabled, the insert fails during writer-side analysis: ```text org.apache.spark.sql.AnalysisException: unresolved operator 'Filter dynamicpruning#276 [pt#228] ``` Relevant stack frames: ```text org.apache.spark.sql.Dataset.withColumn(Dataset.scala:2384) org.apache.paimon.spark.commands.PaimonSparkWriter.write(PaimonSparkWriter.scala:165) org.apache.paimon.spark.commands.WriteIntoPaimonTable.run(WriteIntoPaimonTable.scala:74) org.apache.paimon.spark.write.PaimonWrite.$anonfun$toInsertableRelation$1(PaimonWrite.scala:36) org.apache.spark.sql.execution.datasources.v2.AppendDataExecV1.writeWithV1(V1FallbackWriters.scala:34) ``` Expression IDs may differ between runs. This failure was reproduced on Spark 3.2.4 with the chained `LEFT JOIN` query above. A simpler `INNER JOIN` query containing DPP did not reproduce it, so this report does not imply that every DPP-enabled insert fails. ### Anything else? _No response_ ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
