This is an automated email from the ASF dual-hosted git repository.
npr pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new 7b68bcf ARROW-10056: [C++] Increase flatbuffers max_tables parameter
in order to read wide tables
7b68bcf is described below
commit 7b68bcffd2354dfe457c837d304b3e9fcda0f78f
Author: Gert Hulselmans <[email protected]>
AuthorDate: Fri Jan 29 12:03:00 2021 -0800
ARROW-10056: [C++] Increase flatbuffers max_tables parameter in order to
read wide tables
PyArrow fails to read (valid) Feather v2 files which it created as default
flatbuffers::Verifier verifier number of columns argument is to low when
creating a Feather file from a dataframe with 499999 columns
Increase the number of maximum allowed tables when verifying a FlatBuffer
from 1_000_000 to 10_000_000.
This allows reading back Feather v2 files with more than 1_000_000
columns (499_999 columns from Pandas dataframe ==> Feather v2).
Closes #9349 from ghuls/ARROW-10056
Authored-by: Gert Hulselmans <[email protected]>
Signed-off-by: Neal Richardson <[email protected]>
---
cpp/src/arrow/ipc/reader.cc | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/cpp/src/arrow/ipc/reader.cc b/cpp/src/arrow/ipc/reader.cc
index c4cc84e..57bf57a 100644
--- a/cpp/src/arrow/ipc/reader.cc
+++ b/cpp/src/arrow/ipc/reader.cc
@@ -1053,7 +1053,9 @@ class RecordBatchFileReaderImpl : public
RecordBatchFileReader {
file_->ReadAt(footer_offset_ - footer_length - file_end_size,
footer_length));
auto data = footer_buffer_->data();
- flatbuffers::Verifier verifier(data, footer_buffer_->size(), 128);
+ flatbuffers::Verifier verifier(data, footer_buffer_->size(),
/*max_depth=*/128,
+ /*max_tables=*/UINT_MAX);
+
if (!flatbuf::VerifyFooterBuffer(verifier)) {
return Status::IOError("Verification of flatbuffer-encoded Footer
failed.");
}
