shashank11yadav opened a new issue, #4438:
URL: https://github.com/apache/iceberg-python/issues/4438

   ### Apache Iceberg version
   
   main (development)
   
   ### Please describe the bug
   
   When a scan projects `_row_id` and has a row filter, 
`_task_to_record_batches` appends a helper column named 
`__iceberg_file_position` (`_POSITION_COLUMN`, `pyiceberg/io/pyarrow.py` line 
219) to the batch and later looks it up by name (lines 1846-1850). If the data 
file already has a column with that name, the lookup is ambiguous and the scan 
fails.
   
   ```python
   import os
   import sys
   import tempfile
   
   import pyarrow as pa
   import pyarrow.parquet as pq
   
   from pyiceberg.expressions import GreaterThan
   from pyiceberg.io.pyarrow import ArrowScan, PyArrowFileIO, schema_to_pyarrow
   from pyiceberg.manifest import DataFile, DataFileContent, FileFormat
   from pyiceberg.partitioning import PartitionSpec
   from pyiceberg.schema import Schema
   from pyiceberg.table import FileScanTask
   from pyiceberg.table.metadata import TableMetadataV3
   from pyiceberg.table.metadata_columns import ROW_ID
   from pyiceberg.typedef import Record
   from pyiceberg.types import LongType, NestedField
   
   COL = "__iceberg_file_position"
   schema = Schema(NestedField(1, "id", LongType(), required=False), 
NestedField(2, COL, LongType(), required=False))
   
   with tempfile.TemporaryDirectory() as tmp:
       path = os.path.join(tmp, "data.parquet")
       pq.write_table(
           pa.table({"id": pa.array([1, 2, 3], pa.int64()), COL: pa.array([10, 
20, 30], pa.int64())}, schema=schema_to_pyarrow(schema)),
           path,
       )
       data_file = DataFile.from_args(
           content=DataFileContent.DATA,
           file_path=path,
           file_format=FileFormat.PARQUET,
           partition=Record(),
           record_count=3,
           file_size_in_bytes=os.path.getsize(path),
           first_row_id=100,
       )
       data_file.spec_id = 0
       scan = ArrowScan(
           table_metadata=TableMetadataV3(
               location=tmp, last_column_id=2, schemas=[schema], 
current_schema_id=0, partition_specs=[PartitionSpec()], next_row_id=103
           ),
           io=PyArrowFileIO(),
           projected_schema=Schema(*schema.fields, ROW_ID),
           row_filter=GreaterThan("id", 1),
       )
       print("EXPECTED: 2 rows (id 2,3) with _row_id [101, 102]")
       try:
           result = scan.to_table([FileScanTask(data_file)])
       except Exception as e:
           print(f"ACTUAL: {type(e).__name__}: {e}")
           sys.exit(1)
       print(f"ACTUAL: {result.to_pydict()}")
   ```
   
   Output on `main` (b84df8d):
   
   ```
   EXPECTED: 2 rows (id 2,3) with _row_id [101, 102]
   ACTUAL: KeyError: 'Field "__iceberg_file_position" exists 2 times in schema'
   ```
   
   Expected: the two matching rows with `_row_id` 101 and 102. Without a data 
column named `__iceberg_file_position`, the same scan returns the filtered rows.
   
   The 0.12.0 release has no `pyiceberg.table.metadata_columns`, so only main 
is affected.
   
   ### Additional context
   
   Same fixed-name collision that #4344 fixed in `_determine_partitions`. A fix 
could pick a helper name not in `table.column_names`.
   
   ### Willingness to contribute
   
   I can contribute a fix for this bug independently.
   
   I have a fix with a regression test ready and will open a pull request for 
it right after filing this issue; it will link back here.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to