Zouxxyy opened a new pull request, #10013: URL: https://github.com/apache/paimon/pull/10013
### Purpose Format tables with `spark.paimon.format-table.implementation=paimon` reject writes under `spark.sql.storeAssignmentPolicy=LEGACY` because Spark's generic V2 analyzer disallows that policy. For example, inserting the string `'123'` into a `BIGINT` column fails before the write. Resolve these writes through Spark's `TableOutputResolver`, allowing LEGACY casts and column alignment without changing the SQL. Keep the table schema fixed: missing, extra, or incompatible columns remain errors, and existing writer-side NOT NULL checks are retained. Invalid string-to-number casts produce NULL under LEGACY. Only Paimon format tables under LEGACY opt into this resolution. ANSI/STRICT assignments and the `engine` implementation retain their existing paths. Spark 3.2 uses a scoped configuration override so LEGACY casts remain independent of ANSI expression evaluation. ### Tests `FormatTableTest` passed with standard Maven `verify`: - Spark 3.5.8 / Java 8: 33 passed. - Spark 3.3.4 / Java 8: 31 passed; 2 existing tests skipped because they require Spark 3.4 or later. - Spark 3.2.4 / Java 8: 31 passed; the same 2 existing tests skipped. - Spark 4.1.2 / Java 17: 33 passed. The generated Hive Metastore test resource used an available local port because 9092 was occupied. The eight added tests cover string casts with ANSI evaluation enabled and disabled, fixed schemas and by-name alignment, nested types, append and partition overwrite, ANSI/STRICT behavior, the engine V1 write path, NOT NULL enforcement and analyzer convergence, and the writer schema guard. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
