Stephen0421 opened a new issue, #11121:
URL: https://github.com/apache/paimon/issues/11121

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   
   ### Paimon version
   
   2.2-SNAPSHOT, built from the unmodified Apache Paimon master branch.
   
   ### Compute Engine
   
   Apache Spark 3.2.4, Scala 2.12, Java 8.
   
   The reproduction uses stock Apache Spark and community Paimon code.
   
   ### Minimal reproduce step
   
   The following reproduces an ordinary `INSERT INTO` failure using three 
partitioned Parquet inputs and two chained `LEFT JOIN`s. The destination is a 
partitioned primary-key table with dynamic buckets.
   
   Start a fresh Spark shell with the Paimon Spark 3.2 bundle built from the 
commit above:
   
   ```bash
   spark-shell --master 'local[2]' \
     --jars /path/to/paimon-spark-3.2_2.12-2.2-SNAPSHOT.jar \
     --conf 
spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions
   ```
   
   Run the following Scala code. For multiline input in `spark-shell`, use 
`:paste` and finish with Ctrl-D.
   
   ```scala
   import java.io.File
   import java.nio.file.Files
   import org.apache.spark.sql.catalyst.expressions.DynamicPruningSubquery
   import spark.implicits._
   
   val tempDir = Files.createTempDirectory("paimon-dpp-repro").toFile
   
   spark.conf.set(
     "spark.sql.catalog.paimon",
     "org.apache.paimon.spark.SparkCatalog")
   spark.conf.set(
     "spark.sql.catalog.paimon.warehouse",
     new File(tempDir, "warehouse").toURI.toString)
   
   Seq(
     "spark.paimon.write.use-v2-write" -> "false",
     "spark.sql.optimizer.dynamicPartitionPruning.enabled" -> "true",
     "spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly" -> 
"false",
     "spark.sql.optimizer.dynamicPartitionPruning.useStats" -> "false",
     "spark.sql.optimizer.dynamicPartitionPruning.fallbackFilterRatio" -> "1.0",
     "spark.sql.adaptive.enabled" -> "false",
     "spark.sql.shuffle.partitions" -> "2"
   ).foreach { case (key, value) => spark.conf.set(key, value) }
   
   spark.sql("CREATE DATABASE paimon.dpp_repro")
   
   def createTarget(name: String): Unit = {
     spark.sql(s"""
       CREATE TABLE paimon.dpp_repro.$name (
         id INT,
         pt STRING,
         left_value INT,
         right_value INT
       )
       PARTITIONED BY (pt)
       TBLPROPERTIES (
         'primary-key' = 'id,pt',
         'bucket' = '-1',
         'dynamic-bucket.assigner-parallelism' = '2'
       )
     """)
   }
   
   createTarget("dpp_target")
   
   val rows = Seq((1, "p1", 10), (2, "p2", 20), (3, "p3", 30))
   val extraRows = (4 until 104).map(i => (i, "p4", i))
   
   Seq("dpp_source", "dpp_left", "dpp_right").foreach { name =>
     val input = name match {
       case "dpp_source" => rows
       case "dpp_left" => rows ++ extraRows
       case _ => rows.filterNot(_._1 == 2) ++ extraRows
     }
     val path = new File(tempDir, name).getCanonicalPath
     input.toDF("id", "pt", "metric")
       .write.partitionBy("pt").parquet(path)
     spark.read.parquet(path).createOrReplaceTempView(name)
   }
   
   val query = """
     SELECT a.id, a.pt,
            b.metric AS left_value,
            c.metric AS right_value
     FROM dpp_source a
     LEFT JOIN dpp_left b ON a.id = b.id AND a.pt = b.pt
     LEFT JOIN dpp_right c ON a.id = c.id AND a.pt = c.pt
     WHERE a.id < 3
   """
   
   val plan = spark.sql(query).queryExecution.optimizedPlan
   val pruning = plan.flatMap(_.expressions).flatMap { expression =>
     expression.collect { case s: DynamicPruningSubquery => s }
   }
   assert(pruning.nonEmpty, s"Expected DPP in the input plan:\n$plan")
   
   // Fails on the unmodified baseline.
   spark.sql(s"INSERT INTO paimon.dpp_repro.dpp_target $query")
   ```
   
   After the failure, disable DPP and run the same insert into a fresh target:
   
   ```scala
   spark.conf.set("spark.sql.optimizer.dynamicPartitionPruning.enabled", 
"false")
   createTarget("dpp_target_without_dpp")
   spark.sql(s"INSERT INTO paimon.dpp_repro.dpp_target_without_dpp $query")
   spark.sql("""
     SELECT * FROM paimon.dpp_repro.dpp_target_without_dpp ORDER BY id
   """).show()
   ```
   
   The DPP-disabled control succeeds and returns the expected rows.
   
   The data, SQL, and key settings above were verified in regression tests. The 
standalone Spark shell adaptation has not been separately executed.
   
   
   ### What doesn't meet your expectations?
   
   Expected: the insert succeeds with DPP enabled and writes the same rows as 
the DPP-disabled execution:
   
   ```text
   +---+---+----------+-----------+
   | id| pt|left_value|right_value|
   +---+---+----------+-----------+
   |  1| p1|        10|         10|
   |  2| p2|        20|       null|
   +---+---+----------+-----------+
   ```
   
   Actual: with DPP enabled, the insert fails during writer-side analysis:
   
   ```text
   org.apache.spark.sql.AnalysisException:
   unresolved operator 'Filter dynamicpruning#276 [pt#228]
   ```
   
   Relevant stack frames:
   
   ```text
   org.apache.spark.sql.Dataset.withColumn(Dataset.scala:2384)
   
org.apache.paimon.spark.commands.PaimonSparkWriter.write(PaimonSparkWriter.scala:165)
   
org.apache.paimon.spark.commands.WriteIntoPaimonTable.run(WriteIntoPaimonTable.scala:74)
   
org.apache.paimon.spark.write.PaimonWrite.$anonfun$toInsertableRelation$1(PaimonWrite.scala:36)
   
org.apache.spark.sql.execution.datasources.v2.AppendDataExecV1.writeWithV1(V1FallbackWriters.scala:34)
   ```
   
   Expression IDs may differ between runs.
   
   This failure was reproduced on Spark 3.2.4 with the chained `LEFT JOIN` 
query above. A simpler `INNER JOIN` query containing DPP did not reproduce it, 
so this report does not imply that every DPP-enabled insert fails.
   
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to