Currently the Arrow Spec has Large* types that support u64 offsets instead of u32, however this doesn’t extend to the map type. This is problematic with libraries like polars that exclusively rely on Large* types internally, but could easily hit cast issues when trying to encode processed data as an arrow map. I found this request <https://github.com/apache/arrow/issues/31022> from back in 2022 to add support, but I guess they never posted as I didn’t see any messages in the archive that mention LargeMap. This feels “relatively” simple to add as it would just be defined the same as Map but bucked by a LargeList instead of a List.
Erik Brinkman [email protected]
