dev-donghwan opened a new pull request, #10389:
URL: https://github.com/apache/paimon/pull/10389
### Purpose
**Problem.** On master, a table that already has both
`changelog-producer=lookup` and
`full-compaction.delta-commits`, for example one created before 2.0.0,
cannot be used at all.
#7999 added a check rejecting this combination in
`SchemaValidation.validateTableSchema`, and
that method also runs whenever a table is loaded
(`AbstractFileStoreTable#copyInternal` and
`#copyWithLatestSchema`). So getting, reading, writing and dropping the
table all fail with the
same exception. The table cannot be fixed with
`ALTER TABLE ... RESET ('full-compaction.delta-commits')` either, as the
error message
suggests, because the catalog loads the table first. I reproduced this with
a table created
and written by 1.4.2, then opened by 2.0.0 and by current master.
**Fix.** Following #10187, loading a table now skips this check when the
table already has the
combination with the same values. The table keeps its options and behaves as
before.
**Kept as is.**
- Creating a table and altering its schema still reject the combination, so
other ALTERs keep
failing until the option is removed.
- Setting up the combination, or changing `full-compaction.delta-commits`,
with dynamic options
is still rejected.
**Now works.**
- Such a table can be loaded, read, written and dropped, and the option can
be removed with
`ALTER TABLE ... RESET`.
- This includes Flink SQL, which passes the stored options again as dynamic
options.
- A job that loaded the table before the option was removed can still
refresh its schema.
For reference, the Flink sink has been running periodic full compactions for
this combination,
since `StoreSinkWrite` chooses `GlobalFullCompactionSinkWrite` whenever
`full-compaction.delta-commits` is set. This PR does not change the rule
itself.
#7999 is already in 2.0.0, so this should be cherry-picked to the release
branches too.
### Tests
-
`FileSystemCatalogTest#testLookupTableWithFullCompactionDeltaCommitsStillLoads`
covers each
point above, using a table whose schema file has both options.
- I also checked a table created by 1.4.2 with Flink SQL: `SELECT`, `INSERT`
and
`ALTER TABLE ... RESET` work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
