This is an automated email from the ASF dual-hosted git repository.
Jefffrey pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
The following commit(s) were added to refs/heads/main by this push:
new b06b9c12c1 fix(arrow-ipc): return an error for a DictionaryBatch
without its data (#11020)
b06b9c12c1 is described below
commit b06b9c12c17dca4f575c9c3e5a091fdb9412bab7
Author: Marco Bonamente <[email protected]>
AuthorDate: Wed Sep 9 02:19:46 2026 +0200
fix(arrow-ipc): return an error for a DictionaryBatch without its data
(#11020)
# Which issue does this PR close?
Closes #11019.
# Rationale for this change
`DictionaryBatch.data` is optional in the flatbuffer grammar and
required by the format. `get_dictionary_values` opened it with `unwrap`,
so a file that omits it panicked instead of producing an `ArrowError`.
This is the same class of field, in the same file, that the footer's
custom metadata was just fixed for:
```rust
let (Some(key), Some(value)) = (kv.key(), kv.value()) else {
return Err(ArrowError::ParseError(
"Custom metadata in the IPC footer is missing a key or a
value".to_string(),
));
};
```
The patch uses that shape. The proposal is essentially "finish the
round".
A panic matters more than usual on this path: under `libfuzzer-sys` a
caught panic still becomes `abort()` before unwinding, so a fuzz target
reading untrusted IPC goes into quarantine rather than reporting a
rejected input.
# What changes are included in this PR?
One `let ... else` in `get_dictionary_values`, plus a regression test.
Measured before and after on `main`: the fixture panics without the
change and returns `Parser error: Dictionary batch is missing its data`
with it. `cargo test -p arrow-ipc` passes — 132, 7 and 11.
# Are these changes tested?
Yes. The fixture is the original 2169-byte input from the fuzzing corpus
of [plenora-IO-tools](https://github.com/PlenoraETL/plenora-IO-tools),
not a reduced one, added under `arrow-ipc/test/data/`.
# Are there any user-facing changes?
A file that used to panic now returns an `ArrowError::ParseError`. No
API change.
Co-authored-by: PlenoraETL <[email protected]>
---
arrow-ipc/src/reader.rs | 51 ++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 50 insertions(+), 1 deletion(-)
diff --git a/arrow-ipc/src/reader.rs b/arrow-ipc/src/reader.rs
index 6c0a6749d2..52a026863b 100644
--- a/arrow-ipc/src/reader.rs
+++ b/arrow-ipc/src/reader.rs
@@ -923,9 +923,14 @@ fn get_dictionary_values(
let value = value_type.as_ref().clone();
let schema = Schema::new(vec![Field::new("", value, true)]);
// Read a single column
+ let Some(data) = batch.data() else {
+ return Err(ArrowError::ParseError(
+ "Dictionary batch is missing its data".to_string(),
+ ));
+ };
let record_batch = RecordBatchDecoder::try_new(
buf,
- batch.data().unwrap(),
+ data,
Arc::new(schema),
dictionaries_by_id,
metadata,
@@ -2197,6 +2202,50 @@ mod tests {
);
}
+ #[test]
+ fn test_invalid_dictionary_batch_without_data() {
+ use crate::r#gen::Message::*;
+ use flatbuffers::FlatBufferBuilder;
+
+ let schema = Schema::new(vec![Field::new(
+ "col",
+ DataType::Dictionary(Box::new(DataType::Int8),
Box::new(DataType::Utf8)),
+ true,
+ )]);
+
+ // DictionaryBatch.data is optional in the flatbuffer grammar and
+ // required by the format
+ let mut fbb = FlatBufferBuilder::new();
+ let batch_offset = DictionaryBatch::create(
+ &mut fbb,
+ &DictionaryBatchArgs {
+ id: 0,
+ data: None,
+ isDelta: false,
+ },
+ );
+ fbb.finish_minimal(batch_offset);
+ let batch_bytes = fbb.finished_data().to_vec();
+ let batch =
flatbuffers::root::<DictionaryBatch>(&batch_bytes).unwrap();
+
+ let data_buffer = Buffer::from(vec![0u8; 0]);
+ let mut dictionaries: HashMap<i64, ArrayRef> = HashMap::new();
+
+ let err = read_dictionary(
+ &data_buffer,
+ batch,
+ &schema,
+ &mut dictionaries,
+ &MetadataVersion::V5,
+ )
+ .unwrap_err();
+
+ assert_eq!(
+ err.to_string(),
+ "Parser error: Dictionary batch is missing its data"
+ );
+ }
+
#[test]
fn test_missing_buffer_metadata_error() {
use crate::r#gen::Message::*;
