sdf-jkl opened a new issue, #11994:
URL: https://github.com/apache/arrow-rs/issues/11994
## Describe the bug
Writing 28 distinct 76 MiB strings followed by 996 nulls in one batch fails
with:
```text
Error: General("Page uncompressed size overflow: 2231369840")
```
The column is nullable `LargeUtf8`. Dictionary encoding is enabled, with the
default 1 MiB page limits and 1,024-row write batch size. Each individual value
fits well below the format's 2 GiB page limit, but the writer accumulates all
28 values into the dictionary before checking fallback.
This remains reproducible after the large-value page budgeting changes in
[#9972](https://github.com/apache/arrow-rs/pull/9972) and the exact-value
batching changes in @adriangb's
[#10554](https://github.com/apache/arrow-rs/pull/10554).
## To Reproduce
Add this test as `parquet/tests/clustered_null_dictionary_overflow.rs` in an
Arrow Rust checkout, then run:
```sh
cargo test -p parquet --no-default-features --features arrow,zstd --test
clustered_null_dictionary_overflow -- --nocapture
```
It uses approximately 4.4 GiB of RAM. The values and trailing nulls are
synthetic.
```rust
use arrow_array::{RecordBatch, builder::LargeStringBuilder};
use arrow_schema::{DataType, Field, Schema};
use parquet::arrow::arrow_writer::{ArrowRowGroupWriterFactory,
compute_leaves};
use parquet::arrow::{ArrowSchemaConverter,
add_encoded_arrow_schema_to_metadata};
use parquet::basic::{Compression, ZstdLevel};
use parquet::file::properties::{WriterProperties, WriterVersion};
use parquet::file::writer::SerializedFileWriter;
use std::sync::Arc;
#[test]
fn clustered_null_dictionary_overflow() -> Result<(), Box<dyn
std::error::Error>> {
let rows = 28;
let trailing_nulls = 996;
let value_bytes = 76 * 1024 * 1024;
let version = WriterVersion::PARQUET_1_0;
let mut values = LargeStringBuilder::with_capacity(rows +
trailing_nulls, rows * value_bytes);
for row in 0..rows {
let mut json =
format!("{{\"row\":{row},\"payload\":\"").into_bytes();
json.resize(value_bytes - 2, b'x');
json.extend_from_slice(b"\"}");
values.append_value(std::str::from_utf8(&json)?);
}
values.append_nulls(trailing_nulls);
let field = Field::new("jsonb", DataType::LargeUtf8, true);
let schema = Arc::new(Schema::new(vec![field]));
let batch = RecordBatch::try_new(Arc::clone(&schema),
vec![Arc::new(values.finish())])?;
let mut props = WriterProperties::builder()
.set_writer_version(version)
.set_compression(Compression::ZSTD(ZstdLevel::try_new(9)?))
.build();
let parquet_schema = ArrowSchemaConverter::new().convert(&schema)?;
add_encoded_arrow_schema_to_metadata(&schema, &mut props);
let mut file = SerializedFileWriter::new(
std::io::sink(),
parquet_schema.root_schema_ptr(),
Arc::new(props),
)?;
let factory = ArrowRowGroupWriterFactory::new(&file,
Arc::clone(&schema));
let mut columns = factory.create_column_writers(0)?;
for (field, array) in schema.fields().iter().zip(batch.columns()) {
for leaf in compute_leaves(field, array)? {
columns[0].write(&leaf)?;
}
}
let mut row_group = file.next_row_group()?;
for column in columns {
column.close()?.append_to_row_group(&mut row_group)?;
}
row_group.close()?;
file.close()?;
Ok(())
}
```
The test above fails on upstream `c1ce2b9cdccf50ccd826108027b59892bccb426e`
with the error shown above. The original parameterized reproducer, which also
attached the JSON extension to the column, produced:
| Version | Writer version | 996 trailing nulls |
| --- | --- | --- |
| 59.3.2 | V1 and V2 | Same overflow |
| 60.1.0 | V1 and V2 | Same overflow |
| Upstream `e600f5a22d3883d4edb19df8be8730f1b5ad45ed` | V1 and V2 | Same
overflow |
As a control, the same 28 values without trailing nulls write and round-trip
successfully on 59.3.2 with both writer versions.
## Expected behavior
The writer should respect the byte budget closely enough to avoid a
multi-value dictionary page exceeding the format's size limit, regardless of
where nulls occur in the batch. Exceeding the configured 1 MiB target to
accommodate one indivisible 76 MiB value is expected; accumulating all 28
values is not.
## Additional context
This appears to be the remaining ratio-based batching path discussed in
[#10538](https://github.com/apache/arrow-rs/issues/10538) and
[#10554](https://github.com/apache/arrow-rs/pull/10554). The latter applies
exact value counts to data pages for encodings that compress against the
preceding value, while leaving dictionary batching ratio-based.
In the tested upstream source, [dictionary encoding selects `value_exact =
false`](https://github.com/apache/arrow-rs/blob/e600f5a22d3883d4edb19df8be8730f1b5ad45ed/parquet/src/column/writer/byte_budget_chunker.rs#L170-L186).
The [conversion from a value budget to a level
count](https://github.com/apache/arrow-rs/blob/e600f5a22d3883d4edb19df8be8730f1b5ad45ed/parquet/src/column/writer/byte_budget_chunker.rs#L244-L261)
then gives:
```text
values_per_subbatch = 1
levels_per_subbatch = ceil(1 * 1024 / 28) = 37
```
The first 37 rows contain all 28 non-null values because the nulls are
clustered at the end. Their dictionary entries occupy `28 * (76 * 1024 * 1024 +
4) = 2231369840` bytes, matching the error. The average null ratio therefore
does not bound the number of values in an individual window. In particular,
this arrangement exceeds the two-value bound described in #10554's “What the
other paths give up” section.
This case was found while investigating large-value page overflows. The
clustered nulls were introduced during testing; they were not reported in the
original customer data.
AI assistance was used to prepare this report and reproducer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
