sundapeng opened a new pull request, #867: URL: https://github.com/apache/paimon-rust/pull/867
### Purpose `DataType` derived `Deserialize` on a `#[serde(untagged)]` enum. serde offers every field's type string to each of the 23 variants in turn, and every variant that rejects it formats an error (the `NullableType` variants also allocate a `String` and format an "expected X or X NOT NULL" message; the `FromStr` variants run substring searches and format their own error). For a table with 11,374 columns this made parsing the schema JSON the dominant cost of opening a table through the C bindings: `paimon_table_from_schema_json` took about 87 ms per call in a release build, with `serde_json::error::make_error`, formatting and allocation at the top of the profile. An engine that opens the table once per split pays that for every split. This replaces the derive with a visitor that dispatches once on the type keyword: - plain keywords (`BOOLEAN`, `INT`, `DOUBLE`, `DATE`, `VARIANT`, ... with an optional `NOT NULL`) map straight to their nullable type; - bracketed types (`DECIMAL(p, s)`, `CHAR(n)`, `VARCHAR(n)`, `BINARY(n)`, `VARBINARY(n)`, `BYTES`, `STRING`, `TIME(p)`, `TIMESTAMP(p)`, `TIMESTAMP(p) WITH LOCAL TIME ZONE`, `TIMESTAMP_LTZ(p)`) go to their existing `FromStr` parser; - constructed types (JSON objects) are chosen by their `type` key before the object is deserialized, so key order does not matter. Acceptance is unchanged: the same strings parse to the same values, including the `TIMESTAMP_LTZ` alias and `NOT NULL` suffixes, and unknown keywords, malformed strings and objects without a `type` key are still rejected. Serialization is untouched (`Serialize` keeps the untagged shape). Measured with a 1.08 MB schema JSON of 11,374 columns (60% DOUBLE, 30% STRING, 10% BIGINT) through the C bindings, release build, median of 30 calls on the same host: | step | before | after | |---|---:|---:| | `paimon_table_from_schema_json` | 87.1 ms | 10.6 ms | | read builder + projection (12 columns) + filter + `new_read` + free | 11.4 ms | 9.7 ms | | total per open | 98.5 ms | 20.3 ms | ### Tests - `spec::types::tests::test_datatype_deserialize_atomic_by_keyword` (new): every atomic keyword with and without `NOT NULL`, bracketed forms, `BYTES`/`STRING` and the `TIMESTAMP_LTZ` alias parse to the expected variant and nullability and round-trip through `Serialize`. - `spec::types::tests::test_datatype_deserialize_constructed_by_name` (new): `ARRAY`, `MAP`, `MULTISET`, `ROW` (with nested constructed fields) and `VECTOR` objects, including `NOT NULL` on the object and a `type` key that is not first. - `spec::types::tests::test_datatype_deserialize_rejects_unknown_shapes` (new): unknown keywords, `INT NULLABLE`, a bare `VARCHAR`, an unknown object type, an object without `type`, and a number are rejected. - Existing `spec::types` tests, including `test_timestamp_ltz_alias_preserves_type_precision_and_nullability` and the vector/array round trips, pass unchanged; `cargo test -p paimon --lib` (2868 tests) and `cargo test -p paimon-c` pass. ### API and Format No API or format change. Error messages for invalid type strings now name the offending string instead of serde's "did not match any variant of untagged enum DataType". ### Documentation None needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
