twuebi opened a new pull request, #1299: URL: https://github.com/apache/iceberg-go/pull/1299
compacting a pyiceberg created table fails because of this Summary - coerce LIST/LARGE_LIST records to the variant useLargeTypes requests in arrowProjectionVisitor.List, converting offsets rather than relabeling - add a regression covering large_list->list downcast and list->large_list Why ReadTasks returns a writer schema built with useLargeTypes (small list by default), but the visitor copied the reader's list variant verbatim. The arrow-go bump in #1238 taught getNestedFactory to reconstruct a parquet file's embedded ARROW:schema, so list columns from pyiceberg-written files (which declare large_list) now read back as large_list instead of list. The projected records no longer match the small-list writer schema, so the parquet file writer rejects every batch with "record schema does not match writer's" -- surfacing in compaction as a group that writes no files and a table that never gets its replace snapshot. Primitives dodge this because castIfNeeded already downcasts large_string/large_binary. Testing - go test ./table -run TestToRequestedSchema -count=1 - go test ./table -count=1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
