wombatu-kun opened a new issue, #19743: URL: https://github.com/apache/hudi/issues/19743
### Task Description **What needs to be done:** Add an end-to-end integration test that produces a real Hudi type evolution with a Spark write and then reads it back through the Trino connector, so the connector's evolved-column predicate handling is covered against a table an actual write path created rather than a hand-built fixture. Two pieces, both modelled on files that already exist: - `docker/demo/sparksql-schema-evolution-trino.commands`, on the shape of `sparksql-blob-type-df.commands`: write a v1 schema with `metric FloatType` into `dt='2024-01-01'` with `SaveMode.Overwrite`, then append a v2 schema with `metric DoubleType` into `dt='2024-01-02'`, both through one shared `applyWriteOpts` so hive sync is configured identically, and print a `SCHEMA_EVOLUTION_TEST_SUCCESS` marker. - `hudi-integ-test/src/test/java/org/apache/hudi/integ2/testcontainers/trino/ITTestTrinoSchemaEvolution.java`, on the `ITTestTrinoCustomType` skeleton: `SELECT count(*)` as the anchor, `SELECT id ... WHERE metric > 1.5 ORDER BY id` returning ids from both partitions as the regression assertion, and `WHERE id > 4` as a positive control that pushdown is still alive. No workflow change is needed: `hudi_trino_e2e.yml` already runs `-Dit.test='ITTestTrino*'` against the spark402 compose stack, which is the only pair defining the `trinocoordinator` service. **Why this task is needed:** #19467 fixes a query failure (#19457) where a predicate on a type-widened column either failed the split with `Corrupted statistics for column ...` or, with parquet bloom filters present, silently dropped the row groups holding the matching rows. Its tests build the post-evolution state directly: a base file written with the pre-evolution physical types, plus handles carrying the widened types the metastore reports. That is exactly the state an unrewritten base file is in, but it is asserted rather than produced. What that leaves uncovered is the write side. Two assumptions the connector-side fix rests on are never exercised: that a DataFrame append carrying a widened schema really does leave earlier base files unrewritten, and that hive sync really does report the widened type for the whole table afterwards. Only a Spark write can establish either. The schema-on-read route (`ALTER TABLE ... ALTER COLUMN ... TYPE` under `hoodie.schema.on.read.enable=true`) is deliberately out of scope: `hudi-trino` carries no reference to `InternalSchema` at all, so it is a separate gap, tracked in #19381. ### Task Type Test enhancement ### Related Issues **Parent feature issue:** #19457 **Related issues:** #19467, #19381 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
