liwuhen opened a new pull request, #602:
URL: https://github.com/apache/paimon-rust/pull/602
### Purpose
`TableWrite::write_arrow_batch` currently accepts an Arrow `RecordBatch`
without validating that its schema matches the table's write contract.
Malformed batches can therefore reach downstream writers. In particular, the
dedicated BLOB and VECTOR writer assumes that input columns match the table
schema and indexes them directly. A batch with missing fields can panic
instead
of returning a normal Paimon error.
The empty-batch fast path also returns successfully before any downstream
component can notice an invalid schema.
This change validates the batch schema at the public table-write entry before
the empty-batch fast path, row-kind enrichment, partition routing, or writer
dispatch.
Validation is performed in the following order:
- Field count.
- Field names and order.
- Arrow data types.
The accepted schemas are:
- Normal writes accept exactly the table fields.
- `changelog-producer=input` accepts either the table fields or the table
fields followed by `_VALUE_KIND: Int8`.
- Cross-partition and `rowkind.field` writes continue to accept only the
table
fields. Their internal `_VALUE_KIND` generation remains unchanged.
Every schema mismatch returns `Error::DataInvalid` with expected and actual
context for the first mismatch. No casting, field reordering, column
insertion,
or schema coercion is performed.
### Brief change log
- Cache the table's expected Arrow write schema in `TableWrite` using the
existing `build_target_arrow_schema` conversion.
- Add a private schema validator to `TableWrite::write_arrow_batch`.
- Run validation before the empty-batch fast path and writer dispatch.
- Validate field count before names/order, and names/order before Arrow data
types.
- Allow an optional trailing `_VALUE_KIND: Int8` only for
`changelog-producer=input`.
- Reuse the cached Arrow schema when creating the dedicated-format writer.
- Add regression coverage for:
- Missing BLOB fields.
- Missing VECTOR fields.
- Reordered fields.
- Incorrect field names.
- Incorrect Arrow data types.
- Unexpected extra columns.
- `_VALUE_KIND` on normal writes.
- Misplaced or incorrectly typed `_VALUE_KIND` on input changelog writes.
- Invalid schemas on empty batches.
- Caller-provided `_VALUE_KIND` in cross-partition and `rowkind.field`
modes.
- Valid normal, input changelog, cross-partition, and `rowkind.field`
writes.
### Tests
- Unit test `test_write_arrow_batch_rejects_missing_blob_field`
(`table_write.rs`): verifies that a BLOB batch with a missing field returns
`DataInvalid` instead of reaching the dedicated writer and panicking.
**Fails on the pre-fix code.**
- Unit test `test_write_arrow_batch_rejects_missing_vector_field`
(`table_write.rs`): verifies the same behavior for a dedicated VECTOR
writer. **Fails on the pre-fix code.**
- Unit test `test_write_arrow_batch_rejects_mismatched_table_fields`
(`table_write.rs`): covers field reordering, incorrect names, incorrect
Arrow types, and unexpected columns.
- Unit test `test_write_arrow_batch_rejects_invalid_empty_batch_schema`
(`table_write.rs`): verifies that schema validation occurs before the empty
batch fast path. **Fails on the pre-fix code.**
- Unit test `test_normal_write_rejects_appended_value_kind`
(`table_write.rs`): verifies that normal writes reject a caller-provided
`_VALUE_KIND`.
- Unit test
`test_input_changelog_rejects_misplaced_or_wrong_type_value_kind`
(`table_write.rs`): verifies that input changelog writes accept
`_VALUE_KIND` only as the final `Int8` field.
- Unit test
`test_rowkind_field_accepts_table_schema_and_rejects_caller_value_kind`
(`table_write.rs`): verifies that `rowkind.field` retains its existing
caller schema contract and internal value-kind generation.
- Unit test `test_cross_partition_write_rejects_caller_value_kind`
(`table_write.rs`): verifies that cross-partition writes continue to accept
only table fields.
- Existing table-write tests verify valid normal, input changelog, and
cross-partition writes.
- Commands run locally:
- `cargo test -p paimon --lib table_write`
- `cargo test -p paimon --lib`
- `cargo fmt --all --check`
- `cargo check -p paimon --all-targets`
- `cargo clippy -p paimon --all-targets -- -D warnings`
- `git diff --check`
### API and Format
No public API or persisted data format changes.
The change only rejects Arrow batch schemas that do not satisfy the existing
table-write contract. It does not cast, reorder, pad, or coerce input
columns.
Top-level Arrow field nullability and field metadata are not added to the
validation contract. Existing schema evolution behavior is unchanged.
Changelog, cross-partition, and `rowkind.field` semantics remain unchanged.
The dedicated BLOB and VECTOR formats and their downstream writing logic are
not modified.
### Documentation
No documentation changes required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
