b-phi commented on issue #1623:
URL:
https://github.com/apache/iceberg-python/issues/1623#issuecomment-2648535596
Hey Kevin, thanks for the suggestion. Although that didn't work, I did some
further investigation and believe I found the root issue. It seems the write
behavior depends on whether metadata is present in the provided arrow table.
When I create a "plain" arrow table, I'm able to write it without issue
```python
arrow_schema = pa.schema([pa.field('my_map', pa.map_(pa.large_string(),
pa.large_string()))])
data = {'my_map': [[{'key': 'symbol', 'value': 'BTC'}]]}
pa_table = pa.Table.from_pydict(data, schema=arrow_schema)
table.append(pa_table)
```
The schema in this case is
```
my_map: map<large_string, large_string>
child 0, entries: struct<key: large_string not null, value: large_string>
not null
child 0, key: large_string not null
child 1, value: large_string
```
However if I use the converted iceberg schema as the arrow table schema, I
get the missing field error in the original post
```python
pa_table = pa.Table.from_pydict(data, schema=iceberg_schema.as_arrow())
table.append(pa_table) # errors
```
The schema in this case looks like this.
```
my_map: map<large_string, large_string>
child 0, entries: struct<key: large_string not null, value: large_string>
not null
child 0, key: large_string not null
-- field metadata --
PARQUET:field_id: '1001'
child 1, value: large_string
-- field metadata --
PARQUET:field_id: '1002'
-- field metadata --
PARQUET:field_id: '1'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
