ZZZxDong opened a new pull request, #10983:
URL: https://github.com/apache/paimon/pull/10983

   ### Problem
   
   `CREATE TABLE` accepts an `aggregate-function` that is incompatible with the 
column type, and the failure only surfaces later on the data path:
   
   ```sql
   CREATE TABLE t (
     k    STRING,
     tags STRING,   -- not ARRAY
     PRIMARY KEY (k) NOT ENFORCED
   ) WITH (
     'merge-engine' = 'aggregation',
     'fields.tags.aggregate-function' = 'collect'
   );
   -- succeeds
   ```
   
   Any read, write, or compaction on that table then fails deterministically 
with:
   
   ```
   Data type for collect column must be 'Array' but was 'STRING'.
   ```
   
   The cause is that `SchemaValidation#validateMergeFunctionFactory` only calls 
`FactoryUtil.discoverFactory`, i.e. it verifies that the aggregation function 
*name* exists, but never calls `FieldAggregatorFactory#create`, which is where 
the type checks live. So DDL only proves the merge engine is valid and the 
function name is spelled correctly.
   
   The same gap applies to the other type-checking aggregators (`sum`, 
`product`, `listagg`, `merge_map`, `hll_sketch`, `theta_sketch`, 
`roaringbitmap32/64`, `bool_and`/`bool_or`, `nested_update`, 
`nested_partial_update`). It is also present for `partial-update`: 
`PartialUpdateMergeFunction` wraps each aggregator in a `Supplier`, so even 
instantiating the merge function does not trigger the check.
   
   Because the resulting `IllegalArgumentException` is deterministic and maps 
to HTTP 400 in REST catalogs, an invalid table is effectively unreadable until 
someone fixes its options or drops it.
   
   ### Fix
   
   Instantiate each aggregation factory once during schema validation, so 
type-incompatible configurations are rejected at `CREATE TABLE` / `ALTER TABLE` 
time, reusing the existing factory error messages:
   
   - `aggregation`: resolve the function through 
`AggregateMergeFunction#getAggFuncName` (primary key columns map to 
`primary_key`, sequence fields to `last_value`, otherwise field-level option 
then default), then instantiate.
   - `partial-update` / `versioned-partial-update`: skip sequence fields, map 
primary key columns to `primary_key`, otherwise use the field-level option or 
the default. Fields that never get an aggregator are skipped: sequence-group 
ordering fields for `partial-update`, and multi-version columns for 
`versioned-partial-update`.
   
   No behavior change for valid tables; the check only constructs the object 
that the read/write path would have constructed anyway.
   
   ### Tests
   
   - `collect` / `sum` on a `STRING` column now fails at DDL time for both 
`aggregation` and `partial-update`.
   - Compatible types (`collect` on `ARRAY<STRING>`, `sum` on a numeric column) 
still pass.
   - A `partial-update` sequence-group ordering field configured with an 
incompatible aggregation function is not rejected, since it never gets an 
aggregator.
   - `SchemaManagerTest#testResetSequenceGroupForAggregateFunction` used a 
`VARCHAR` column with `sum`; the column type is corrected so the fixture is 
valid.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to