pitrou commented on issue #50548: URL: https://github.com/apache/arrow/issues/50548#issuecomment-5022452452
> for RLE batches, every level is the same value I think you may be misunderstanding how this encoding works: https://github.com/apache/parquet-format/blob/master/Encodings.md#RLE > for bit-packed, even the full scan can't really guarantee catching corruption... a corrupted value could still land in the valid range. and we already have page CRC for actual corruption detection. so the O(n) scan was just burning cpu without adding much real safety To decide that the check was useless, you have to check whether any use of the decoded levels relies on the values lying between 0 and max_level - 1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
