emkornfield commented on issue #50548: URL: https://github.com/apache/arrow/issues/50548#issuecomment-5029963045
> For the record, the min-max check was added by @emkornfield in https://github.com/apache/arrow/pull/8177/changes#diff-05802abb678d7f865de06497651d9ca057bf0b44ec37ceeae2a5d720ffe98e28R160-R168 I believe this replaced existing checks: https://github.com/apache/arrow/pull/8177/changes#diff-05802abb678d7f865de06497651d9ca057bf0b44ec37ceeae2a5d720ffe98e28L1173 > To decide that the check was useless, you have to check whether any use of the decoded levels relies on the values lying between 0 and max_level - 1. I would need to double check but I think negative values might be impossible, so we could at least get rid of that check. > even the full scan can't really guarantee catching corruption... a corrupted value could still land in the valid range. and we already have page CRC for actual corruption detection. This isn't just about corruption necessarily but making sure the reader is hardened against bugs, or malicious values. Valid optimization I can see: 1. Push down the check for at least RLE runs 2. If we guarantee only non-negative values are possible in decoding, remove the min check. 3. remove the check if max repetition/max definition level is out a power of 2 or zero. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
