js8544 commented on issue #36158: URL: https://github.com/apache/arrow/issues/36158#issuecomment-1598374887
Hi, this is expected because in FixedSizeBinary arrays, null elements still have to occupy the same amount of space as valid elements, so here you would need 1M * 1001 memory. FixedSizeBinary is a fixed-size primitive array and its memory layout is specified [here](https://arrow.apache.org/docs/format/Columnar.html#fixed-size-primitive-layout). For your use case a workaround is to use variable size binary arrays in which null elements won't take any space. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
