tustvold commented on code in PR #4434:
URL: https://github.com/apache/arrow-rs/pull/4434#discussion_r1242061745
##########
arrow-ipc/src/reader.rs:
##########
@@ -647,9 +649,18 @@ impl<R: Read + Seek> FileReader<R> {
reader.seek(SeekFrom::End(-10 - footer_len as i64))?;
reader.read_exact(&mut footer_data)?;
- let footer = crate::root_as_footer(&footer_data[..]).map_err(|err| {
- ArrowError::IoError(format!("Unable to get root as footer:
{err:?}"))
- })?;
+ // construct verifier options that reflect actual number of columns
+ // in file and avoid an error if the file contains more than 1M rows
+ let verifier_options = VerifierOptions {
Review Comment:
I wonder if we should create a FileReaderOptions, that contains both the
projection, and also `VerifierOptions`, to allow people to set these as they
deem fit
##########
arrow-ipc/src/reader.rs:
##########
@@ -647,9 +649,18 @@ impl<R: Read + Seek> FileReader<R> {
reader.seek(SeekFrom::End(-10 - footer_len as i64))?;
reader.read_exact(&mut footer_data)?;
- let footer = crate::root_as_footer(&footer_data[..]).map_err(|err| {
- ArrowError::IoError(format!("Unable to get root as footer:
{err:?}"))
- })?;
+ // construct verifier options that reflect actual number of columns
+ // in file and avoid an error if the file contains more than 1M rows
+ let verifier_options = VerifierOptions {
+ max_depth: 128,
+ max_tables: footer_len as usize * 8,
Review Comment:
I think some explanation of where this number is coming from is probably not
remiss
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
