JingsongLi commented on code in PR #11190:
URL: https://github.com/apache/paimon/pull/11190#discussion_r4594455380


##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/PaimonSparkTableBase.scala:
##########
@@ -49,10 +49,7 @@ abstract class PaimonSparkTableBase(val table: Table)
 
   lazy val coreOptions = new CoreOptions(table.options())
 
-  lazy val useV2Write: Boolean = {
-    val v2WriteConfigured = OptionUtils.useV2Write()
-    v2WriteConfigured && supportsV2Write
-  }
+  lazy val useV2Write: Boolean = OptionUtils.useV2Write(table.options()) && 
supportsV2Write

Review Comment:
   [P2] Honor the per-write V1 override in saveAsTable
   
   For a new table, `df.write.format("paimon").option("write.use-v2-write", 
"false").saveAsTable(...)` still executes V2 after this default change. Spark's 
DataFrame CTAS carries the option in `writeOptions`, while 
`PaimonTableAsSelectHelper.splitTableAndWriteOptions` only promotes 
Core/Iceberg options into table properties. Consequently, `table.options()` 
does not contain this override, and `newWriteBuilder` also selects the writer 
using this cached flag rather than the operation options. On Spark 3.4.4/JDK 8, 
the actual plan contains `AppendData ... PaimonWrite` despite the explicit 
`false`; the same input succeeds through V1 when the session flag is false. 
This leaves callers unable to opt out of the newly enabled default for this 
write. Please merge the per-write override before selecting table capabilities 
and the writer, and add a new-table `saveAsTable` regression test.



##########
paimon-spark/paimon-spark-3.4/src/main/scala/org/apache/spark/sql/execution/shim/PaimonCreateTableAsSelectStrategy.scala:
##########
@@ -58,19 +58,31 @@ case class PaimonCreateTableAsSelectStrategy(spark: 
SparkSession)
         parts.nonEmpty,
         qualifiedSpec.properties.asJava)
 
-      CreateTableAsSelectExec(
-        catalog.asTableCatalog,
-        ident,
-        parts,
-        analyzedQuery.get,
-        planLater(query),
-        qualifiedSpec,
-        new CaseInsensitiveStringMap(
-          (writeOptions +
-            (PaimonWriteOptions.OPERATION_OPTION -> 
Snapshot.Operation.CREATE_TABLE_AS_SELECT
-              .name())).asJava),
-        ifNotExists
-      ) :: Nil
+      PaimonTableAsSelectWriteExec.prepare(
+        CreateTableAsSelectExec(
+          catalog.asTableCatalog,
+          ident,
+          parts,
+          analyzedQuery.get,
+          planLater(query),
+          qualifiedSpec,
+          new CaseInsensitiveStringMap(
+            (writeOptions +
+              (PaimonWriteOptions.OPERATION_OPTION -> 
Snapshot.Operation.CREATE_TABLE_AS_SELECT
+                .name())).asJava),
+          ifNotExists
+        )) :: Nil
+    case create @ CreateTableAsSelect(
+          ResolvedIdentifier(_: SparkGenericCatalog, _),
+          _,
+          _,
+          tableSpec: TableSpec,
+          _,
+          _,
+          _) if 
tableSpec.provider.exists(_.equalsIgnoreCase(SparkSource.NAME)) =>

Review Comment:
   [P2] Prepare GenericCatalog CTAS writes with an omitted provider
   
   With `spark.sql.sources.default=paimon` and 
`spark.sql.legacy.createHiveTableByDefault=false`, 
`df.writeTo("spark_catalog.db.t").tableProperty(...).create()` can reach this 
strategy with `tableSpec.provider=None` when `.using("paimon")` is omitted. 
`SparkGenericCatalog.stageCreate` accepts that case as Paimon through 
`usePaimon(null)`, but this `exists` guard skips the wrapper, leaving the write 
on Spark's native legacy CTAS executor, which bypasses distribution 
preparation. On Spark 3.4.4/JDK 8, writing 40 rows from four input partitions 
into a one-bucket append table produces one data file through V1 versus four 
with the new default V2 path; both read back the same rows. This increases 
small files and subsequent compaction work. Please align the provider check 
with the catalog's accepted default-provider cases, including the corresponding 
legacy-version CTAS/RTAS strategies, and cover an API call without `.using()`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to