shashank11yadav opened a new issue, #4438:
URL: https://github.com/apache/iceberg-python/issues/4438
### Apache Iceberg version
main (development)
### Please describe the bug
When a scan projects `_row_id` and has a row filter,
`_task_to_record_batches` appends a helper column named
`__iceberg_file_position` (`_POSITION_COLUMN`, `pyiceberg/io/pyarrow.py` line
219) to the batch and later looks it up by name (lines 1846-1850). If the data
file already has a column with that name, the lookup is ambiguous and the scan
fails.
```python
import os
import sys
import tempfile
import pyarrow as pa
import pyarrow.parquet as pq
from pyiceberg.expressions import GreaterThan
from pyiceberg.io.pyarrow import ArrowScan, PyArrowFileIO, schema_to_pyarrow
from pyiceberg.manifest import DataFile, DataFileContent, FileFormat
from pyiceberg.partitioning import PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.table import FileScanTask
from pyiceberg.table.metadata import TableMetadataV3
from pyiceberg.table.metadata_columns import ROW_ID
from pyiceberg.typedef import Record
from pyiceberg.types import LongType, NestedField
COL = "__iceberg_file_position"
schema = Schema(NestedField(1, "id", LongType(), required=False),
NestedField(2, COL, LongType(), required=False))
with tempfile.TemporaryDirectory() as tmp:
path = os.path.join(tmp, "data.parquet")
pq.write_table(
pa.table({"id": pa.array([1, 2, 3], pa.int64()), COL: pa.array([10,
20, 30], pa.int64())}, schema=schema_to_pyarrow(schema)),
path,
)
data_file = DataFile.from_args(
content=DataFileContent.DATA,
file_path=path,
file_format=FileFormat.PARQUET,
partition=Record(),
record_count=3,
file_size_in_bytes=os.path.getsize(path),
first_row_id=100,
)
data_file.spec_id = 0
scan = ArrowScan(
table_metadata=TableMetadataV3(
location=tmp, last_column_id=2, schemas=[schema],
current_schema_id=0, partition_specs=[PartitionSpec()], next_row_id=103
),
io=PyArrowFileIO(),
projected_schema=Schema(*schema.fields, ROW_ID),
row_filter=GreaterThan("id", 1),
)
print("EXPECTED: 2 rows (id 2,3) with _row_id [101, 102]")
try:
result = scan.to_table([FileScanTask(data_file)])
except Exception as e:
print(f"ACTUAL: {type(e).__name__}: {e}")
sys.exit(1)
print(f"ACTUAL: {result.to_pydict()}")
```
Output on `main` (b84df8d):
```
EXPECTED: 2 rows (id 2,3) with _row_id [101, 102]
ACTUAL: KeyError: 'Field "__iceberg_file_position" exists 2 times in schema'
```
Expected: the two matching rows with `_row_id` 101 and 102. Without a data
column named `__iceberg_file_position`, the same scan returns the filtered rows.
The 0.12.0 release has no `pyiceberg.table.metadata_columns`, so only main
is affected.
### Additional context
Same fixed-name collision that #4344 fixed in `_determine_partitions`. A fix
could pick a helper name not in `table.column_names`.
### Willingness to contribute
I can contribute a fix for this bug independently.
I have a fix with a regression test ready and will open a pull request for
it right after filing this issue; it will link back here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]