sundapeng opened a new pull request, #9353:
URL: https://github.com/apache/paimon/pull/9353
### Purpose
`INSERT OVERWRITE` on a format table replaced only the partitions the commit
happened to write,
because the files to delete were derived from the files written. Two
consequences:
- **A statement whose query returns no rows deletes nothing.** On an
unpartitioned table every row
it had stays readable, so a reader afterwards sees data the statement said
it had replaced. This
is the visible one: the table is not empty after `INSERT OVERWRITE t
SELECT ... WHERE <false>`.
- **A Spark `STATIC` overwrite behaves dynamically.** `INSERT OVERWRITE t
VALUES (...)` on a
partitioned table under the default
`spark.sql.sources.partitionOverwriteMode=STATIC` left the
partitions the query did not write in place, merging with them instead of
replacing the table.
A data table does neither: its commit reads `dynamic-partition-overwrite` to
decide which partitions
an overwrite replaces, and dynamic selection only applies when the table has
partition keys, so an
overwrite that names no partition otherwise means the table. The format
table path never consulted
the option, and the Spark write builder dropped the mode Spark had already
resolved
(`overwriteDynamic`) before reaching the commit, so nothing downstream could
tell `STATIC` from
`DYNAMIC`.
### Changes
- `FormatTableCommit` applies the same rule a data table commit applies: an
overwrite that names no
static partition replaces only the written partitions when the table has
partition keys **and**
`dynamic-partition-overwrite` is on, and replaces the whole table
otherwise. The whole-table case
walks down every partition level from the table root, and the partitions
it clears flow into the
existing reporting so a partition emptied but not rewritten still reports
an exact zero.
- `PaimonFormatTableWriterBuilder` carries the resolved mode into
`dynamic-partition-overwrite`, the
way `PaimonV2WriteBuilder` already does for data tables.
A named static partition keeps its behaviour, and so does dynamic overwrite:
a commit that writes
nothing selects no partition, which is not the same as selecting all of
them. Staging trees
(`_temporary`) are left alone, as before.
### Tests
`FormatTableCommitTest`: an overwrite with no messages on an unpartitioned
table removes the table's
data files and leaves another writer's staging tree; the same statement on a
partitioned table
follows `dynamic-partition-overwrite` in both positions.
`FormatTableTestBase` (Spark SQL): an empty `INSERT OVERWRITE` empties an
unpartitioned table; a
`STATIC` overwrite replaces every partition, including one the query does
not write, and empties the
table when the query returns nothing; a `DYNAMIC` overwrite replaces only
the partitions it writes
and leaves everything alone when it writes nothing.
### API and Format
No public API or file format change. Behaviour changes for `INSERT
OVERWRITE` without a `PARTITION`
clause on a format table, which is the point of the fix;
`dynamic-partition-overwrite` (default
`true`) restores the previous behaviour for partitioned tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
