Re: [I] Manifest entry partition filtering on struct fields [iceberg-go]
kevinjqliu commented on issue #404: URL: https://github.com/apache/iceberg-go/issues/404#issuecomment-2851952779 the only time we need to sanitize column names is when writing https://github.com/apache/iceberg-python/blob/34c89494c39916b9b1aa7e6da2c24c34c4d7f058/pyiceberg/io/pyarrow.py#L2418-L2422 https://github.com/apache/iceberg-python/issues/584 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [I] Manifest entry partition filtering on struct fields [iceberg-go]
kevinjqliu commented on issue #404: URL: https://github.com/apache/iceberg-go/issues/404#issuecomment-2851945226 this is a good test case from the pyiceberg side https://github.com/apache/iceberg-python/blob/main/tests/integration/test_writes/test_writes.py#L1158-L1177 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [I] Manifest entry partition filtering on struct fields [iceberg-go]
kevinjqliu commented on issue #404: URL: https://github.com/apache/iceberg-go/issues/404#issuecomment-2851941593 I think the real culprit here is looking up the partition fields by name and not by id https://github.com/apache/iceberg-go/blob/091352672b4191a4bb11b603c1fb9bd2ab6c2aaf/table/scanner.go#L116-L125 pyiceberg uses the partition's source_id https://github.com/apache/iceberg-python/blob/34c89494c39916b9b1aa7e6da2c24c34c4d7f058/pyiceberg/partitioning.py#L412-L423 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [I] Manifest entry partition filtering on struct fields [iceberg-go]
loicalleyne commented on issue #404: URL: https://github.com/apache/iceberg-go/issues/404#issuecomment-2845357483 @GabrielM98 do you think you'd be able to provide one of the avro files being decoded? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [I] Manifest entry partition filtering on struct fields [iceberg-go]
GabrielM98 commented on issue #404: URL: https://github.com/apache/iceberg-go/issues/404#issuecomment-2842644898 Hey @zeroshade, apologies for not doing this on Monday as promised but I tried querying the table with PyIceberg and was able to do so successfully. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [I] Manifest entry partition filtering on struct fields [iceberg-go]
GabrielM98 commented on issue #404:
URL: https://github.com/apache/iceberg-go/issues/404#issuecomment-2830909201
Thanks for the quick response @zeroshade!
> can you share the full schema
The schema looks like so...
```json
{
"type": "struct",
"schema-id": 0,
"fields": [
{
"id": 1,
"name": "event_metadata",
"required": false,
"type": {
"type": "struct",
"fields": [
{
"id": 6,
"name": "uuid",
"required": false,
"type": "string"
},
{
"id": 7,
"name": "source",
"required": false,
"type": {
"type": "struct",
"fields": [
{
"id": 12,
"name": "type",
"required": false,
"type": "string"
},
{
"id": 13,
"name": "id",
"required": false,
"type": "string"
},
{
"id": 14,
"name": "service_id",
"required": false,
"type": "string"
}
]
}
},
{
"id": 8,
"name": "subjects",
"required": false,
"type": {
"type": "list",
"element-id": 15,
"element": {
"type": "struct",
"fields": [
{
"id": 16,
"name": "type",
"required": false,
"type": "string"
},
{
"id": 17,
"name": "id",
"required": false,
"type": "string"
},
{
"id": 18,
"name": "service_id",
"required": false,
"type": "string"
}
]
},
"element-required": false
}
},
{
"id": 9,
"name": "timing",
"required": false,
"type": {
"type": "struct",
"fields": [
{
"id": 19,
"name": "created_at",
"required": false,
"type": "timestamptz"
},
{
"id": 20,
"name": "emitted_at",
"required": false,
"type": "timestamptz"
}
]
}
},
{
"id": 10,
"name": "collection",
"required": false,
"type": {
"type": "struct",
"fields": [
{
"id": 21,
"name": "collections",
"required": false,
"type": {
"type": "list",
"element-id": 22,
"element": {
"type": "struct",
"fields": [
{
"id": 23,
"name": "type",
"required": false,
"type": "string"
},
{
"id": 24,
"name": "id",
"required": false,
"type": "string"
},
{
"id": 25,
"name": "service_id",
"required": false,
"type": "string"
}
]
},
"element-required": false
}
}
]
}
},
{
"id": 11,
"name": "parent_event_uuid",
"required": false,
