sablejade opened a new pull request, #9167:
URL: https://github.com/apache/paimon/pull/9167
### Purpose
This PR fixes a Spark dynamic partition write issue where positional writes
may be incorrectly interpreted as Hive-style dynamic partition writes.
When writing to a partitioned Paimon table with Spark SQL dynamic partition
syntax, the query output may already follow the target table schema order.
However, the current logic may still treat it as Hive-style order and move
dynamic partition columns from the tail. This can misalign columns, especially
for UNION queries whose output names are inherited from the first branch.
For example, a UNION query may output columns in the correct table schema
order, but one expression name, such as `detail_ratio`, does not match the
target column name `value`. The write is positional, so the value should still
be written to the `value` column. Relying only on output names can misclassify
the query order and cause silent column misalignment when column types are
compatible.
This PR adds a configurable dynamic partition column order mode:
- `AUTO`: preserve compatible behavior and automatically detect table-order
or Hive-style-order writes.
- `TABLE`: always interpret positional dynamic partition writes using the
target table schema order.
- `HIVE`: interpret dynamic partition writes using Hive-style order when
applicable.
### Changes
- Add `spark.paimon.sql.dynamic-partition-column-order`.
- Support `AUTO`, `TABLE`, and `HIVE` modes.
- Avoid unnecessary Hive-style reordering when the query already follows
table schema order.
- Make dynamic overwrite without explicit `PARTITION (...)` respect the
configured column order when Hive-style output is applicable.
- Add Spark SQL tests for:
- UNION output whose expression names differ from target column names.
- AUTO table-order detection.
- AUTO/HIVE Hive-style detection.
- Explicit TABLE mode.
- Explicit HIVE mode.
- INSERT BY NAME behavior.
- Invalid config handling.
### Tests
- `PaimonDynamicPartitionColumnOrderTest`
Closes #9156
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
