rich7420 opened a new issue, #4137:
URL: https://github.com/apache/iceberg-python/issues/4137
### Apache Iceberg version
0.12.0 (latest release)
### Please describe the bug 🐞
Also reproduces on main (d37149c).
`table.scan(selected_fields=()).to_arrow()` returns an empty table instead
of a table with the scan's row count and no columns. `to_arrow_batch_reader()`
does the same, with or without a `row_filter`. A zero-column result is how
callers ask for a row count without reading data: Ray's `Dataset.count()`
pushes an empty projection into `read_iceberg` and got 0
(ray-project/ray#63913), which Ray worked around in ray-project/ray#65175 by
reading one column and dropping it.
```python
import tempfile
import pyarrow as pa
import pyiceberg
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.schema import Schema
from pyiceberg.types import LongType, NestedField
wh = tempfile.mkdtemp()
cat = SqlCatalog("c", uri=f"sqlite:///{wh}/cat.db", warehouse=f"file://{wh}")
cat.create_namespace("db")
t = cat.create_table("db.t", schema=Schema(NestedField(1, "id", LongType(),
required=False)))
t.append(pa.table({"id": pa.array(range(100), pa.int64())}))
print(pyiceberg.__version__)
print(t.scan(selected_fields=()).to_arrow().num_rows) # expected 100
print(t.scan(selected_fields=()).to_arrow_batch_reader().read_all().num_rows)
# expected 100
print(t.scan(selected_fields=(), row_filter="id >= 50").to_arrow().num_rows)
# expected 50
```
```
0.12.0
0
0
0
```
Two places drop the row count, both in `pyiceberg/io/pyarrow.py`:
- `_to_requested_schema` builds each batch with
`pa.RecordBatch.from_struct_array(...)`. A struct array with no children has
length 0, so every batch comes back with 0 rows.
- `ArrowScan.to_table` combines batches with `pa.concat_tables`, which also
returns 0 rows for tables without columns (pyarrow 25.0.1), so fixing the first
place alone is not enough for `to_arrow()`.
I have a fix: return `batch.select([])` from `_to_requested_schema` when the
requested schema is empty, and build the result with `pa.Table.from_batches` in
`to_table` for an empty projection. Tests cover `to_arrow()`, the batch reader,
a row filter, and a table with deleted rows. I can open a PR.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
