ZZZxDong opened a new pull request, #10983:
URL: https://github.com/apache/paimon/pull/10983
### Problem
`CREATE TABLE` accepts an `aggregate-function` that is incompatible with the
column type, and the failure only surfaces later on the data path:
```sql
CREATE TABLE t (
k STRING,
tags STRING, -- not ARRAY
PRIMARY KEY (k) NOT ENFORCED
) WITH (
'merge-engine' = 'aggregation',
'fields.tags.aggregate-function' = 'collect'
);
-- succeeds
```
Any read, write, or compaction on that table then fails deterministically
with:
```
Data type for collect column must be 'Array' but was 'STRING'.
```
The cause is that `SchemaValidation#validateMergeFunctionFactory` only calls
`FactoryUtil.discoverFactory`, i.e. it verifies that the aggregation function
*name* exists, but never calls `FieldAggregatorFactory#create`, which is where
the type checks live. So DDL only proves the merge engine is valid and the
function name is spelled correctly.
The same gap applies to the other type-checking aggregators (`sum`,
`product`, `listagg`, `merge_map`, `hll_sketch`, `theta_sketch`,
`roaringbitmap32/64`, `bool_and`/`bool_or`, `nested_update`,
`nested_partial_update`). It is also present for `partial-update`:
`PartialUpdateMergeFunction` wraps each aggregator in a `Supplier`, so even
instantiating the merge function does not trigger the check.
Because the resulting `IllegalArgumentException` is deterministic and maps
to HTTP 400 in REST catalogs, an invalid table is effectively unreadable until
someone fixes its options or drops it.
### Fix
Instantiate each aggregation factory once during schema validation, so
type-incompatible configurations are rejected at `CREATE TABLE` / `ALTER TABLE`
time, reusing the existing factory error messages:
- `aggregation`: resolve the function through
`AggregateMergeFunction#getAggFuncName` (primary key columns map to
`primary_key`, sequence fields to `last_value`, otherwise field-level option
then default), then instantiate.
- `partial-update` / `versioned-partial-update`: skip sequence fields, map
primary key columns to `primary_key`, otherwise use the field-level option or
the default. Fields that never get an aggregator are skipped: sequence-group
ordering fields for `partial-update`, and multi-version columns for
`versioned-partial-update`.
No behavior change for valid tables; the check only constructs the object
that the read/write path would have constructed anyway.
### Tests
- `collect` / `sum` on a `STRING` column now fails at DDL time for both
`aggregation` and `partial-update`.
- Compatible types (`collect` on `ARRAY<STRING>`, `sum` on a numeric column)
still pass.
- A `partial-update` sequence-group ordering field configured with an
incompatible aggregation function is not rejected, since it never gets an
aggregator.
- `SchemaManagerTest#testResetSequenceGroupForAggregateFunction` used a
`VARCHAR` column with `sum`; the column type is corrected so the fixture is
valid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]