Re: [I] Create iceberg table from existsing parquet files with slightly different schemas (schemas merge is possible). [iceberg-python]

2024-04-16 Thread via GitHub
sergun commented on issue #601: URL: https://github.com/apache/iceberg-python/issues/601#issuecomment-2058470428 BTW: Found some explaination why merge of Arrow tables with different schemas is not possible: https://github.com/apache/arrow/issues/35424 The reason looks weired, but

Re: [I] Create iceberg table from existsing parquet files with slightly different schemas (schemas merge is possible). [iceberg-python]

2024-04-16 Thread via GitHub
sergun commented on issue #601: URL: https://github.com/apache/iceberg-python/issues/601#issuecomment-2058456159 > One thing I wonder is if PyIceberg can handle schema evolution of nested structs. Looks like it can. From https://py.iceberg.apache.org/api/#add-column: ```

Re: [I] Create iceberg table from existsing parquet files with slightly different schemas (schemas merge is possible). [iceberg-python]

2024-04-16 Thread via GitHub
sergun commented on issue #601: URL: https://github.com/apache/iceberg-python/issues/601#issuecomment-2058445375 @kevinjqliu It is strange to me that in PyArrow there is [pa.unify_schemas(...)](https://arrow.apache.org/docs/python/generated/pyarrow.unify_schemas.html) which is able (I

Re: [I] Create iceberg table from existsing parquet files with slightly different schemas (schemas merge is possible). [iceberg-python]

2024-04-15 Thread via GitHub
kevinjqliu commented on issue #601: URL: https://github.com/apache/iceberg-python/issues/601#issuecomment-2058220348 Looks like your schema is nested, which makes things more complicated. It's pretty difficult to deal with merging nested schemas. I'm not sure if there's an out-of-the-box

Re: [I] Create iceberg table from existsing parquet files with slightly different schemas (schemas merge is possible). [iceberg-python]

2024-04-14 Thread via GitHub
sergun commented on issue #601: URL: https://github.com/apache/iceberg-python/issues/601#issuecomment-2054129400 Thank you @kevinjqliu ! Do you know how to read parquet file with unified schema in pyarrow? I successfully merged schemas: ``` t1 =

Re: [I] Create iceberg table from existsing parquet files with slightly different schemas (schemas merge is possible). [iceberg-python]

2024-04-12 Thread via GitHub
kevinjqliu commented on issue #601: URL: https://github.com/apache/iceberg-python/issues/601#issuecomment-2052628807 There's a [`Table.add_files` API](https://github.com/apache/iceberg-python/blob/5039b5d70644bc06c98349090912c6e9066d3ea1/mkdocs/docs/api.md#add-files) which supports