wangyong9999 opened a new pull request, #194: URL: https://github.com/apache/paimon-cpp/pull/194
### Purpose Linked issue: part of #192 First of three stacked PRs adding read support for the source-backed global scalar indexes that Paimon release-2.0.0 introduced for primary-key tables (`pk-btree` / `pk-bitmap`). This part adds the core types with no behavior change to existing scans: - `PrimaryKeyIndexSourceMeta` v1 decoding, byte-compatible with the Java implementation: big-endian layout, Java modified UTF-8 file names (backed by a shared `JavaModifiedUtf8` utility), defensive source-file count cap, rejection of unknown versions and trailing bytes. - `PrimaryKeyIndexSourcePolicy`: only `COMPACT` files with `level > 0` participate in coverage. - `PrimaryKeyIndexDefinitions`: parses `pk-btree.index.columns` / `pk-bitmap.index.columns` / `pk-vector.index.columns` / `pk-full-text.index.columns` and the field-scoped JSON options with the same validation and scalar-value coercion semantics as Java `CoreOptions` (algorithm-prefix qualification, conflict rejection, one index family per column). - `PkSortedIndexGroup` / `PkSortedBucketIndexState`: the exact per-data-level coverage validation that decides whether a payload can be trusted — exactly one payload per level whose ordered source files match the level's active `COMPACT` files, matching index type / field id, `rowRange == [0, sum(rowCount) - 1]`, and payload row count equal to the source sum; anything else is rejected so the affected files keep a normal scan. The metadata carrier (`GlobalIndexMeta` `_SOURCE_META`, commit message v12) already landed in #179; this change decodes and validates what it carries. The batch-scan planner and read path follow in the second PR, the payload builder and end-to-end integration tests in the third. ### Tests - `java_modified_utf8_test`: ASCII / CJK round trips, `U+0000` as `C0 80`, supplementary characters as CESU-8 surrogate pairs, malformed-sequence rejection for both encode and decode, and Java `readUTF` bit-pattern leniency. - `primary_key_index_source_meta_test`: byte-exact golden vector asserted against the serialized form, int64 big-endian row counts, and a rejection matrix for unknown versions, invalid counts, the defensive cap, truncation and trailing bytes. - `primary_key_index_definitions_test`: column parsing, JSON option qualification, scalar (number / boolean) value coercion like Java's `parseJsonMap(..., String.class)`, and rejection of duplicate columns, cross-family columns, null and nested values. - `pk_sorted_bucket_index_state_test`: the coverage rejection matrix — misordered sources, altered row counts, missing / extra files, two payloads per level, wrong field id / index type / row range / row count, corrupt source metadata — plus multi-level partial coverage. ### API and Format - New option key constants on `Options` (`pk-btree.index.columns`, `pk-bitmap.index.columns`, `pk-vector.index.columns`, `pk-full-text.index.columns`), names identical to Java `CoreOptions`. - No storage format change: this reads the source metadata carried by the manifest structures introduced in #179, byte-compatible with Java release-2.0.0 `PrimaryKeyIndexSourceMeta` v1. ### Documentation A user guide page for the feature arrives with the final PR of the series. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
