liujiwen-up opened a new pull request, #611:
URL: https://github.com/apache/paimon-rust/pull/611
### Purpose
Linked issue: close #xxx
`TableWrite::write_arrow_batch` is the public entry point for writing Arrow
data into a Paimon table, but it forwarded the caller's `RecordBatch` to the
per-(partition, bucket) writers without validating its schema. When a caller
supplied a malformed schema — e.g. a missing column — the downstream
dedicated
BLOB/VECTOR writer (`AppendDedicatedFormatFileWriter`) indexes columns
positionally via `batch.column(idx)` and panics instead of returning a clean
error (finding F5 / P2).
This change validates the incoming batch schema at the core write entry, so
malformed input is rejected uniformly with `Error::DataInvalid` before it can
reach — and crash — any downstream writer.
### Brief change log
- Add a private `TableWrite::validate_write_batch_schema` and call it at the
very start of `write_arrow_batch`, before the empty-batch fast path and
before writer dispatch.
- Validation runs in order: field count → field names/order → Arrow data type
(reusing the existing `paimon_type_to_arrow` conversion and the
`VALUE_KIND_FIELD_NAME` constant; no new public API).
- Legal input contract enforced per write mode:
- Normal write: schema must match the table fields exactly (count, order,
name, type).
- `changelog-producer=input`: table fields, optionally followed by a single
trailing `_VALUE_KIND: Int8` column.
- cross-partition / `rowkind.field`: only the table fields are accepted;
`_VALUE_KIND` is generated internally, so a caller-supplied `_VALUE_KIND`
is rejected.
- Illegal input returns `Error::DataInvalid` with a message carrying the
expected/actual context of the first mismatching field.
- No change to the downstream `dedicated_format_file_writer.rs`
indexing/write
logic, and no cast / reorder-by-name / column-filling / schema coercion.
### Tests
Added regression tests in the existing `table_write` test module (assertions
key on the error type + context rather than the full message string):
- BLOB dedicated writer with a missing field returns `DataInvalid` (no
panic).
- VECTOR dedicated writer with a missing field returns `DataInvalid` (no
panic).
- Reordered columns, wrong field name, wrong Arrow type, unexpected extra
column.
- Illegal appended `_VALUE_KIND` on a plain write.
- input-changelog `_VALUE_KIND` in the wrong position / with the wrong type.
- `rowkind.field` rejects a caller-supplied `_VALUE_KIND`.
- A schema-invalid empty batch is still rejected (validation runs before the
empty-batch fast path).
- A VECTOR batch whose FixedSizeList inner field diverges from the canonical
form is rejected up front.
- Existing legal inputs (normal write, input changelog with/without
`_VALUE_KIND`, cross-partition, `rowkind.field`) continue to pass.
Verified locally:
- `cargo test -p paimon --lib table_write` — 71 passed
- `cargo test -p paimon --lib` — 1743 passed, 0 failed
- `cargo fmt --all --check`, `cargo check -p paimon --all-targets`,
`cargo clippy -p paimon --all-targets -- -D warnings` — all clean
### API and Format
No public API or storage format change. The added validation is a private
method; the accepted-input contract is unchanged for all existing legal write
modes, and previously-accepted batches keep working.
### Documentation
No documentation changes required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
