LuciferYang commented on PR #55924: URL: https://github.com/apache/spark/pull/55924#issuecomment-5512397365
One residual edge from the buffer rework: the old code kept `previous` as null until the first value was decoded, so a corrupt file whose very first value already carries a non-zero `prefixLength` failed fast with an NPE. `prevBuf` is now pre-zeroed, so the same input with a fake prefix of 64 bytes or fewer silently assembles zero bytes as the shared prefix and returns wrong data (larger prefixes still throw). Legal writers never emit this (PARQUET-246 files only affect the first value of pages after the first), so this only turns the corrupt-input failure mode from loud into silent. Would it be possible to throw `ParquetDecodingException` in `readValues`, `skipBinary` and `readGeoData` when `prevLen < prefixLength`, so corrupt files still fail fast? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
