schenksj commented on issue #12330:
URL: https://github.com/apache/arrow-rs/issues/12330#issuecomment-7062416792

   Yes. A malformed or malicious Parquet file can make the reader request an 
allocation of attacker-chosen size before it has decoded any data. One page of 
a few bytes is enough. It needs a DELTA_LENGTH_BYTE_ARRAY or DELTA_BYTE_ARRAY 
column whose length header declares a huge value count.
   
   What a user sees depends on the declared count:
   - **About 2^31 values:** the reader asks for 8 GiB at once. If that fails, 
Rust aborts the whole process; this is not a recoverable error or panic. On a 
container or executor with a memory limit, the OOM killer usually kills the 
process first. Either way, every query in that process is taken down, not only 
the one reading the file.
   - **2^60 values:** it panics with `capacity overflow`. An application that 
catches panics survives this; anything embedding the reader over FFI/JNI 
usually does not.
   
   Who is affected: any service that reads Parquet it doesn't fully trust, such 
as user uploads, shared lakehouse tables or files from another team. A single 
bad file can crash a shared query server or executor, which makes it a denial 
of service.
   
   What it is not: there is no out-of-bounds read or write and no memory 
unsafety, and correctly written files are unaffected. The fix in #12331 refuses 
the count with a `ParquetError` when it exceeds the page's value count or what 
the page's bytes could encode. Every such count would have failed later anyway, 
so valid files decode as before.
   
   We found it while bounding memory in a search engine that reads Parquet from 
object storage on behalf of many users. One file could take down an executor 
shared by many queries.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to