LuciferYang opened a new pull request, #10092: URL: https://github.com/apache/paimon/pull/10092
### Purpose close #10091 `FileIndexProcessor` rebuilds the file index of an existing data file by reading the file with a projection and feeding the rows to an index writer. The writer projection holds positions into the file's own schema, but the same file-schema positions were also passed to `ReadBuilder.withProjection`, which indexes into the current table schema. Once the schema evolves the two drift apart: after a `DROP COLUMN` that shrinks the arity a file-schema position can exceed the current schema and the rewrite crashes with `IndexOutOfBoundsException` (every file written before the drop fails); with same-arity drift (a drop plus a later add) the position lands on a different current column, so the rewrite reads that column's values and builds the index under the original column's name, silently corrupting predicate pruning. This resolves the read projection against the current schema by column name, keeping it positionally aligned with the writer projection. Both are collected in one pass and skip columns absent from either schema. The `currentSchema` is taken from the `table` object the read projects against rather than `schemaManager.latest()`, which could be a newer schema under concurrent DDL and re-introduce the same drift. When the file schema is the current schema the two projections are identical and nothing changes. ### Tests `FileIndexProcessorTest#testProcessAfterDroppingColumnBeforeIndexedColumn`: file written at `[k, a, v]` with a bloom filter on `v`, then `a` is dropped so the current schema is `[k, v]`. Pre-fix the read projects file position 2 into a size-2 schema and throws `IndexOutOfBoundsException`; the test asserts the rewrite succeeds and writes the `v` index. `FileIndexProcessorTest#testProcessAfterColumnShiftIndexesTheRightColumnValues`: pins the same-arity silent-corruption mode. Dropping `a` and adding `b` keeps the arity at three but shifts `v` from file position 2 to current position 1, with the all-null `b` now at position 2. The test reads the `v` bloom filter back and asserts it holds `v`'s real values (10, 20) rather than `b`'s nulls. It fails against the pre-fix code, which indexes `b` under the name `v`. ### API and Format no ### Documentation no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
