LuciferYang opened a new issue, #10091: URL: https://github.com/apache/paimon/issues/10091
### Search before asking - [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master ### Compute Engine Spark / Flink (`CALL rewrite_file_index`) ### Minimal reproduce step Create a table with a file index on a column, for example a bloom filter on `v`, with schema `[k, a, v]`. Write some rows so a data file is committed at schema 0. Then `ALTER TABLE ... DROP COLUMN a`. Run `rewrite_file_index` over the table. The rewrite reads the pre-drop file (still at schema 0) to rebuild its index. ### What doesn't meet your expectations? `FileIndexProcessor` reads the file with a projection and feeds the rows to the index writer. The writer projection holds positions into the file's own schema, but the same file-schema positions were also passed to `ReadBuilder.withProjection`, which indexes into the current table schema. Once the schema evolves the two drift apart: - After a drop that shrinks the arity, a file-schema position can exceed the current schema. The read projects an out-of-range position and the rewrite fails with `IndexOutOfBoundsException`. Every file written before the drop fails. - With same-arity drift (a drop plus a later add), the file-schema position lands on a different current column. The rewrite reads that column's values and builds the index under the original column's name, silently corrupting predicate pruning with no error. Expected: `rewrite_file_index` rebuilds the index over the intended column's values for files written before a schema change. ### Anything else? Fix direction: resolve the read projection against the current schema by column name, keeping it positionally aligned with the writer projection (both collected in one pass, skipping columns absent from either schema). When the file schema is the current schema the two projections are identical and nothing changes. ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
