mapleFU commented on PR #38784: URL: https://github.com/apache/arrow/pull/38784#issuecomment-1819106152
> Also, I don't understand why this would fix dictionary decoding, and why this would be ok for non-dictionary decoding. This lacks a serious analysis IMHO. This is easy because: 1. `len_` is the size of `Page`'s data payload 2. `PlainByteArrayDecoder`'s `len_` is similiar to the final result(each record might have a length). But for dictionary, the `len_` might be unrelated to the ByteArray size, which could make the Program reserve unneccessary memory. Evenmore, it doesn't decrease `len_` after each decode. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
