wangzhigang1999 opened a new issue, #10053:
URL: https://github.com/apache/paimon/issues/10053
### Search before asking
- [x] I searched existing issues for these projection failures and found no
matching report.
### Paimon version
Master at `bb5498ff3`, using `paimon-python` from the source checkout.
### Compute Engine
PyPaimon, Python 3.12.6, PyArrow 19.0.1, macOS. The example uses a local
filesystem table with Parquet files.
### Minimal reproduce step
Read two distinct MAP keys, `foo` and `.foo`, from a one-row table:
```python
import tempfile
import pyarrow as pa
from pypaimon import CatalogFactory, Schema
with tempfile.TemporaryDirectory() as warehouse:
catalog = CatalogFactory.create({"warehouse": warehouse})
catalog.create_database("default", False)
data = pa.table(
{
"attrs": pa.array(
[[("foo", 100), (".foo", 107)]],
type=pa.map_(pa.string(), pa.int64()),
)
}
)
catalog.create_table(
"default.repro",
Schema.from_pyarrow_schema(
data.schema, options={"bucket": "-1", "file.format": "parquet"}
),
False,
)
table = catalog.get_table("default.repro")
wb = table.new_batch_write_builder()
writer = wb.new_write()
try:
writer.write_arrow(data)
wb.new_commit().commit(writer.prepare_commit())
finally:
writer.close()
rb = table.new_read_builder().with_projection(["attrs['foo']",
"attrs['.foo']"])
result =
rb.new_read().to_arrow(rb.new_scan().plan().splits()).to_pydict()
print(result)
assert result == {"attrs_foo": [100], "attrs__foo": [107]}
```
### What doesn't meet your expectations?
Expected:
```python
{'attrs_foo': [100], 'attrs__foo': [107]}
```
Actual:
```python
{'attrs_foo': [100], 'attrs__foo': [100]}
```
The read returns the value for `foo` in both columns without reporting an
error. Reading only `attrs['.foo']` raises `ArrowInvalid`. I reproduced both
cases with Parquet and row files.
### Anything else?
I also found two projection failures in the same checkout:
- **MAP selector prefix matching:** with MAP columns named `attrs` and
`attrs['x`, and `attrs` containing `('x[0]', 42)`, `attrs['x[0]']` drops the
requested column, while `attrs["x[0]"]` returns 42. A longer field-name prefix
wins before the remaining selector is validated.
- **ROW nullability:** for a nullable `r: ROW<x BIGINT NOT NULL>`, writing
`r = NULL` and `r = {x: 7}` then projecting `r.x` returns `[None, 7]` with a
`not null` output field. Writing this Arrow result to Parquet raises `Column
'r_x' is declared non-nullable but contains nulls`.
For the first example, the selected MAP keys become struct fields during
reading. Passing `.foo` as a string to `pyarrow.compute.struct_field`
interprets it as a field path. The projection reader needs to preserve the
literal key name.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
