schenksj commented on issue #12330: URL: https://github.com/apache/arrow-rs/issues/12330#issuecomment-7062416792
Yes. A malformed or malicious Parquet file can make the reader request an allocation of attacker-chosen size before it has decoded any data. One page of a few bytes is enough. It needs a DELTA_LENGTH_BYTE_ARRAY or DELTA_BYTE_ARRAY column whose length header declares a huge value count. What a user sees depends on the declared count: - **About 2^31 values:** the reader asks for 8 GiB at once. If that fails, Rust aborts the whole process; this is not a recoverable error or panic. On a container or executor with a memory limit, the OOM killer usually kills the process first. Either way, every query in that process is taken down, not only the one reading the file. - **2^60 values:** it panics with `capacity overflow`. An application that catches panics survives this; anything embedding the reader over FFI/JNI usually does not. Who is affected: any service that reads Parquet it doesn't fully trust, such as user uploads, shared lakehouse tables or files from another team. A single bad file can crash a shared query server or executor, which makes it a denial of service. What it is not: there is no out-of-bounds read or write and no memory unsafety, and correctly written files are unaffected. The fix in #12331 refuses the count with a `ParquetError` when it exceeds the page's value count or what the page's bytes could encode. Every such count would have failed later anyway, so valid files decode as before. We found it while bounding memory in a search engine that reads Parquet from object storage on behalf of many users. One file could take down an executor shared by many queries. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
