shashank11yadav opened a new pull request, #4439:
URL: https://github.com/apache/iceberg-python/pull/4439

   Fixes #4438
   
   # Rationale for this change
   
   When a scan projects `_row_id` and has a row filter, 
`_task_to_record_batches` adds a temporary column of row positions to each 
batch. That column always gets the fixed name `_POSITION_COLUMN` 
(`__iceberg_file_position`). If the data file already has a column with that 
name, the table ends up with two fields of the same name. The lookup then fails 
with `KeyError: 'Field "__iceberg_file_position" exists 2 times in schema'`.
   
   The fix picks a free name for the temporary column by putting `_` in front 
until no data column has that name. `_determine_partitions` already avoids 
clashes with its `_partition_<name>` columns this way. When no data column has 
that name, the behaviour is the same as before. `_POSITION_COLUMN` itself is 
unchanged.
   
   ## Are these changes tested?
   
   Yes. The new test is 
`test_task_to_record_batches_row_ids_with_position_column_name` in 
`tests/io/test_pyarrow.py`. It writes a file with an `id` column and a data 
column named `__iceberg_file_position`, then reads it with `_row_id` projected 
and the filter `id > 1`. It checks that:
   - the filter is applied,
   - the data column comes back unchanged,
   - the row IDs are 101 and 102.
   
   Without the fix, the test fails with the KeyError above. With the fix, it 
passes.
   
   What was run:
   - `pytest tests/io/test_pyarrow.py -m unit`: 256 passed, 1 skipped.
   - `ruff check` (ruff 0.14.3): pass.
   - `ruff format --check` (ruff 0.14.3): pass.
   - `mypy --config=pyproject.toml pyiceberg tests` (mypy 1.18.2): pass.
   
   Not run:
   - [ ] `make lint` (prek): its hooks need network access, which my 
environment doesn't have.
   - The full `make test` suite.
   
   ## Are there any user-facing changes?
   
   Scans that project `_row_id` with a row filter no longer fail when a data 
file has a column named `__iceberg_file_position`. No API changes. The bug only 
exists on `main`, and no released version has it.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to