JingsongLi commented on code in PR #11190:
URL: https://github.com/apache/paimon/pull/11190#discussion_r4594455380
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/PaimonSparkTableBase.scala:
##########
@@ -49,10 +49,7 @@ abstract class PaimonSparkTableBase(val table: Table)
lazy val coreOptions = new CoreOptions(table.options())
- lazy val useV2Write: Boolean = {
- val v2WriteConfigured = OptionUtils.useV2Write()
- v2WriteConfigured && supportsV2Write
- }
+ lazy val useV2Write: Boolean = OptionUtils.useV2Write(table.options()) &&
supportsV2Write
Review Comment:
[P2] Honor the per-write V1 override in saveAsTable
For a new table, `df.write.format("paimon").option("write.use-v2-write",
"false").saveAsTable(...)` still executes V2 after this default change. Spark's
DataFrame CTAS carries the option in `writeOptions`, while
`PaimonTableAsSelectHelper.splitTableAndWriteOptions` only promotes
Core/Iceberg options into table properties. Consequently, `table.options()`
does not contain this override, and `newWriteBuilder` also selects the writer
using this cached flag rather than the operation options. On Spark 3.4.4/JDK 8,
the actual plan contains `AppendData ... PaimonWrite` despite the explicit
`false`; the same input succeeds through V1 when the session flag is false.
This leaves callers unable to opt out of the newly enabled default for this
write. Please merge the per-write override before selecting table capabilities
and the writer, and add a new-table `saveAsTable` regression test.
##########
paimon-spark/paimon-spark-3.4/src/main/scala/org/apache/spark/sql/execution/shim/PaimonCreateTableAsSelectStrategy.scala:
##########
@@ -58,19 +58,31 @@ case class PaimonCreateTableAsSelectStrategy(spark:
SparkSession)
parts.nonEmpty,
qualifiedSpec.properties.asJava)
- CreateTableAsSelectExec(
- catalog.asTableCatalog,
- ident,
- parts,
- analyzedQuery.get,
- planLater(query),
- qualifiedSpec,
- new CaseInsensitiveStringMap(
- (writeOptions +
- (PaimonWriteOptions.OPERATION_OPTION ->
Snapshot.Operation.CREATE_TABLE_AS_SELECT
- .name())).asJava),
- ifNotExists
- ) :: Nil
+ PaimonTableAsSelectWriteExec.prepare(
+ CreateTableAsSelectExec(
+ catalog.asTableCatalog,
+ ident,
+ parts,
+ analyzedQuery.get,
+ planLater(query),
+ qualifiedSpec,
+ new CaseInsensitiveStringMap(
+ (writeOptions +
+ (PaimonWriteOptions.OPERATION_OPTION ->
Snapshot.Operation.CREATE_TABLE_AS_SELECT
+ .name())).asJava),
+ ifNotExists
+ )) :: Nil
+ case create @ CreateTableAsSelect(
+ ResolvedIdentifier(_: SparkGenericCatalog, _),
+ _,
+ _,
+ tableSpec: TableSpec,
+ _,
+ _,
+ _) if
tableSpec.provider.exists(_.equalsIgnoreCase(SparkSource.NAME)) =>
Review Comment:
[P2] Prepare GenericCatalog CTAS writes with an omitted provider
With `spark.sql.sources.default=paimon` and
`spark.sql.legacy.createHiveTableByDefault=false`,
`df.writeTo("spark_catalog.db.t").tableProperty(...).create()` can reach this
strategy with `tableSpec.provider=None` when `.using("paimon")` is omitted.
`SparkGenericCatalog.stageCreate` accepts that case as Paimon through
`usePaimon(null)`, but this `exists` guard skips the wrapper, leaving the write
on Spark's native legacy CTAS executor, which bypasses distribution
preparation. On Spark 3.4.4/JDK 8, writing 40 rows from four input partitions
into a one-bucket append table produces one data file through V1 versus four
with the new default V2 path; both read back the same rows. This increases
small files and subsequent compaction work. Please align the provider check
with the catalog's accepted default-provider cases, including the corresponding
legacy-version CTAS/RTAS strategies, and cover an API call without `.using()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]