shashank11yadav opened a new pull request, #4439: URL: https://github.com/apache/iceberg-python/pull/4439
Fixes #4438 # Rationale for this change When a scan projects `_row_id` and has a row filter, `_task_to_record_batches` adds a temporary column of row positions to each batch. That column always gets the fixed name `_POSITION_COLUMN` (`__iceberg_file_position`). If the data file already has a column with that name, the table ends up with two fields of the same name. The lookup then fails with `KeyError: 'Field "__iceberg_file_position" exists 2 times in schema'`. The fix picks a free name for the temporary column by putting `_` in front until no data column has that name. `_determine_partitions` already avoids clashes with its `_partition_<name>` columns this way. When no data column has that name, the behaviour is the same as before. `_POSITION_COLUMN` itself is unchanged. ## Are these changes tested? Yes. The new test is `test_task_to_record_batches_row_ids_with_position_column_name` in `tests/io/test_pyarrow.py`. It writes a file with an `id` column and a data column named `__iceberg_file_position`, then reads it with `_row_id` projected and the filter `id > 1`. It checks that: - the filter is applied, - the data column comes back unchanged, - the row IDs are 101 and 102. Without the fix, the test fails with the KeyError above. With the fix, it passes. What was run: - `pytest tests/io/test_pyarrow.py -m unit`: 256 passed, 1 skipped. - `ruff check` (ruff 0.14.3): pass. - `ruff format --check` (ruff 0.14.3): pass. - `mypy --config=pyproject.toml pyiceberg tests` (mypy 1.18.2): pass. Not run: - [ ] `make lint` (prek): its hooks need network access, which my environment doesn't have. - The full `make test` suite. ## Are there any user-facing changes? Scans that project `_row_id` with a row filter no longer fail when a data file has a column named `__iceberg_file_position`. No API changes. The bug only exists on `main`, and no released version has it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
