HippoBaro opened a new issue, #11837:
URL: https://github.com/apache/arrow-rs/issues/11837
### Describe the bug
The Parquet Arrow writer assumes that batch columns, Arrow schema fields,
Parquet leaf columns, and physical column writers have matching shapes, but
some entry points do not validate those assumptions.
Consequently:
- ArrowWriter::write can accept a batch with extra top-level columns and
silently ignore those columns.
- A batch with missing top-level columns is not rejected immediately.
- `ArrowRowGroupWriterFactory::create_column_writers` can panic when the
Arrow schema requires more leaves than the supplied Parquet schema contains.
The opposite mismatch can also be accepted rather than returning an error.
- Supplying an explicit Parquet schema bypasses Arrow-to-Parquet conversion
and can therefore bypass validation of the Arrow map-key schema. An Arrow
schema with nullable map keys can be accepted through this path even though the
normal conversion path rejects it.
These cases should produce descriptive errors rather than silently
truncating work, panicking, or accepting invalid schema combinations.
### To Reproduce
```rust
use std::sync::Arc;
use arrow_array::{Int32Array, RecordBatch};
use arrow_schema::{DataType, Field, Schema};
use parquet::arrow::ArrowWriter;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let writer_schema = Arc::new(Schema::new(vec![
Field::new("a", DataType::Int32, false),
]));
let batch_schema = Arc::new(Schema::new(vec![
Field::new("a", DataType::Int32, false),
Field::new("b", DataType::Int32, false),
]));
let batch = RecordBatch::try_new(
batch_schema,
vec![
Arc::new(Int32Array::from(vec![1])),
Arc::new(Int32Array::from(vec![2])),
],
)?;
let mut writer =
ArrowWriter::try_new(Vec::new(), writer_schema, None)?;
let result = writer.write(&batch);
assert!(
result.is_err(),
"expected a column-count mismatch error, got {result:?}"
);
Ok(())
}
```
The assertion fails because write returns `Ok(())`: iteration over the
writer fields and batch columns stops after column `a`, ignoring column `b`.
The reverse case (constructing a writer with fields `a` and `b`, then passing
`a` batch containing only` a`) also lacks an immediate column-count error.
### Expected behavior
- Reject missing or extra top-level batch columns with a descriptive error
before updating row-group row accounting.
- Reject mismatches between the Arrow schema’s expected leaf count and the
supplied Parquet schema when creating column writers.
- Validate each field’s produced leaf count before dispatching its leaf
writes.
- Apply map-key schema validation consistently, including when an explicit
Parquet schema is supplied or Arrow metadata is disabled. Invalid map-key
schemas should be rejected before file initialization.
- Apply the corresponding structural checks to ordinary and
content-defined-chunking write paths, including physical-writer and chunker
counts.
- Continue accepting logically compatible dense, dictionary, and run-end
encoded representations; this should not become a strict Arrow
datatype-equality check.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
