JingsongLi commented on PR #9822: URL: https://github.com/apache/paimon/pull/9822#issuecomment-5954440402
Reviewed current head `2b5f23f8289ddc7b07c6deec9a728791da452b21`. The Cassandra event-version use case has end-to-end value. The immutable prefix, source-field-ID storage names, rename handling, and shared write-side filtering address the earlier problems. I found two new historical-schema issues and one production contract gap. **[P1] Dropping an ordinary column silently replaces event metadata with that column's historical value.** `ChangelogEventMetadata.metadataValueFields` allocates IDs above the remaining row type's maximum (line 253), which can reuse a dropped physical field ID. Reproduction with real Parquet and Avro tables: physical schema `(id INT, data INT, event_ts BIGINT, extra BIGINT)`, expose `event_ts`; insert `(1,10,50,777)`, then update `(1,20,100,888)`. Before `ALTER ... DROP extra`, metadata is `50/100/100` for `+I/-U/+U`. After the successful DROP, it is `777/777/888`; latest batch reads return physical `event_ts=100` but metadata `888`. Physical-only reads remain correct. The table's historical highest field ID remains 3, but the new metadata ID also becomes 3. A control allocating non-colliding metadata IDs preserves the correct values in both formats. Please ensure synthetic IDs/read mappings cannot alias historical physical fields, and add a real DROP/read regression. This can sen d an incorrect WRITETIME to the downstream store without an exception. **[P2] Avro historical metadata loses the source-type conversion after a legal ALTER.** For an Avro lookup table exposing `event_ts BIGINT`, write values 50 and 100, then ALTER it to `DECIMAL(20,0)`. Historical physical values still read correctly, but metadata reads throw `ClassCastException: Long cannot be cast to Decimal`. `KeyValueFileReaderFactory` lines 434–441 appends the current `extraFields` to each historical schema, so the metadata is treated as DECIMAL on disk and misses the BIGINT→DECIMAL cast. Constructing the historical metadata fields using their historical source-field type makes the same reproduction pass. Please cover this in both raw and merged reader mappings, with an actual historical-file test. **[P2] The documented pass-through support boundary must include automatic sink materialization.** A plain `INSERT INTO event_sink SELECT data,writetime FROM source_table`, with source PK `id` and an external upsert sink PK `data`, inserts Flink's default `AUTO` upsert materializer. For `(1,10,50)` followed by `(1,20,100)`, it cannot match `-U(10,100)` to stored `(10,50)`, so old sink key 10 remains. There is no metadata filter or aggregation. I reproduced this against Flink's values upsert sink, with a later marker proving catch-up. `NONE` produces the correct final sink; keeping `AUTO` but projecting physical `event_ts` also correctly removes old key 10. Flink `TRY_RESOLVE` rejects the unsafe metadata plan. The warning at docs lines 118–120 currently excludes only filters/aggregations; please state the compatible key/plan requirements and necessary safeguards, and add an actual downstream-sink regression. Flink's event-oriented CDC metadata API is a defensible contract, so I a m not repeating the previous metadata-filter example as a new blocker. Validation: 147 Core tests passed on JDK 8; all four existing Flink integration cases passed on Flink 1.20.1 and 2.2.0 after aligning the local Avro runtime; Spark 3.5.8 CDC suite passed 6/6. Spark 4.1.2 CDC suite passed 6/6 with normal Maven checks. Independent actual persisted tests passed scalar rename, ordinary-column addition, metadata-field addition/reordering, and nine ROW/ARRAY/MAP × Parquet/Avro/ORC full-row and metadata-only projections. Exact-head CI is green, but the new schema-evolution probes above fail on this head and pass with isolated controls. No shared source changes were retained. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
