charliec05 opened a new pull request, #24424: URL: https://github.com/apache/datafusion/pull/24424
## Which issue does this PR close? - Part of #17498. ## Rationale for this change `SET datafusion.execution.parquet.maximum_parallel_row_group_writers = 0` currently succeeds. When single-file parallelism is enabled, that value is later used as the capacity of an internal Tokio channel, which requires a capacity greater than zero. Rejecting the value when the configuration is set turns a deferred writer panic into an actionable configuration error. ## What changes are included in this PR? - Store `maximum_parallel_row_group_writers` as `ConfigNonZeroUsize` and unwrap it only at the Parquet writer and protobuf boundaries. - Preserve the documented default when an older protobuf payload omits the scalar field, which proto3 decodes as zero. - Add SQLLogicTest and protobuf coverage, and regenerate the configuration documentation. ## Are these changes tested? Yes. The new `set_variable.slt` case first failed against `main` because setting the option to zero succeeded. It passes with this change. The following checks pass: - `cargo fmt --all` - `cargo clippy --all-targets --all-features -- -D warnings` - `./dev/rust_lint.sh` - `cargo test -p datafusion-common -p datafusion-proto-common -p datafusion-proto-models -p datafusion-datasource-parquet` - `cargo test --profile=ci --test sqllogictests -- set_variable.slt information_schema.slt` - `cargo test -p datafusion -p datafusion-cli` - `RUST_BACKTRACE=1 cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli --workspace --lib --tests --bins --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption` - `./ci/scripts/doc_prettier_check.sh --write --allow-dirty` ## Are there any user-facing changes? Yes. Setting `datafusion.execution.parquet.maximum_parallel_row_group_writers` to zero now returns `Invalid or Unsupported Configuration: value must be greater than 0` instead of being accepted and potentially causing a writer panic. Programmatic users constructing `ParquetOptions` use `ConfigNonZeroUsize` for this field. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
