xiangfu0 commented on issue #16619: URL: https://github.com/apache/pinot/issues/16619#issuecomment-5334575556
UUID v1 is now complete on `master`, and the superseded monolithic implementation #18140 has been closed. Closing this umbrella tracker as completed. ## Implementation status Core UUID work: - [x] #18869 — logical UUID type foundation (`pinot-spi`) - [x] #18870 — ingestion and segment storage - [x] #18871 — `DataSchema` plus JSON/Arrow result rendering - [x] #18872 — server-side equality, `IN`, and range predicates - [x] #18873 — aggregation, group-by, `HAVING`, distinct, and supported sketch functions - [x] #18874 — multi-stage planner, UUID literal bridge, and runtime support - [x] #18876 — focused end-to-end coverage for a multi-stage UUID equality join and realtime UUID-primary-key upsert - [x] #19091 — UUID scalar functions and multi-stage UDF wrappers - [x] #19155 — UUID literals over the existing binary request-literal encoding Review-driven split-outs: - [x] #19181 — logical UUID `CAST` - [x] #19182 — UUID bloom-filter support - [x] #19183 — UUID `CASE`, `IN`, and comparison transforms #18875 was closed without merge intentionally. A dedicated UUID segment partition function is not required by the OSS v1 logical-type contract. It would create a new raw-16-byte producer/Pinot hashing convention without a demonstrated interoperability requirement. Existing partitioning remains unchanged; a specialized contract can be proposed separately if a concrete OSS use case emerges. ## Supporting merged work - [x] #16471 — keep `UUID` and `VARIANT` non-reserved - [x] #18400 — Avro/Parquet record-extractor logical-type contract - [x] #18927 — external `java.util.UUID` / internal fixed-width byte conversion - [x] #18952 — raw `BYTES` min/max correctness - [x] #18969 — MV `BYTES`/`BIG_DECIMAL` `GenericRow` SerDe - [x] #18970 — recommender logical-type mapping - [x] #18973 — UUID sample generation in the schema recommender - [x] #19029 — logical scalar rendering in `jsonPathString` - [x] #19073 — Avro schema generation from Pinot logical types Existing UUID byte/string helpers remain available through #11988, and the separate UUID hash option for upsert primary keys remains available through #12538. ## Final UUID v1 contract - `UUID` is a first-class Pinot logical type backed by the existing fixed-width 16-byte `BYTES` representation. - Persisted values and main execution paths use `byte[16]` or `ByteArray`; `java.util.UUID` is used at typed Java and Calcite boundaries. - Query results render as canonical lowercase dashed UUID strings. Plain `BYTES` columns remain unchanged and render as hexadecimal. - Single-value and multi-value UUID dimensions are supported, including dictionary/raw storage, offline/realtime ingestion, Avro UUID interop, and JSON/Arrow result encoding, subject to the same operator-specific restrictions as other MV/array types. - UUID columns support equality/inequality, `IN`/`NOT IN`, range predicates, `CAST`, `CASE`, comparisons, ordering, grouping, `HAVING`, exact distinct, and supported distinct-count sketches. - The multi-stage engine preserves UUID literals through the existing binary literal field and uses generic stored-`BYTES` paths for grouping and equality/hash joins. - UUID scalar functions cover validation, string/byte conversion, v4/v7 generation, version extraction, and timestamp extraction. - Focused integration coverage now includes the previously missing MSE UUID equality-join and realtime UUID-primary-key upsert paths. ## Documentation The canonical user documentation was landed in `pinot-contrib/pinot-docs` rather than duplicated in the Apache Pinot root README: - [x] pinot-contrib/pinot-docs#917 — schema recommender UUID support - [x] pinot-contrib/pinot-docs#936 — UUID ingestion and storage - [x] pinot-contrib/pinot-docs#940 — UUID result rendering and upgrade guidance - [x] pinot-contrib/pinot-docs#952 — UUID scalar functions - [x] pinot-contrib/pinot-docs#964 and pinot-contrib/pinot-docs#968 — UUID casts and rendering - [x] pinot-contrib/pinot-docs#969 — UUID literals in `CASE`, `IN`, and comparisons - [x] pinot-contrib/pinot-docs#970 — UUID predicates - [x] pinot-contrib/pinot-docs#989 — UUID aggregation/grouping behavior - [x] pinot-contrib/pinot-docs#991 — multi-stage UUID queries The older upsert UUID hash function is documented by pinot-contrib/pinot-docs#428. ## Intentional limitations and focused follow-ups - **Rolling upgrades:** segment storage and request literals reuse existing byte formats, but `DataSchema` carries new `UUID`/`UUID_ARRAY` logical tokens. Older brokers and servers cannot parse them. Upgrade brokers and servers together before enabling UUID result traffic; rollback to a pre-UUID build is unsafe while UUID-typed results are in flight. - **Null sentinel:** without column-based null handling, the nil UUID (`00000000-0000-0000-0000-000000000000`) is the default null sentinel. Enable column-based null handling when nil UUID must be preserved as a value. - **Schema migration:** Pinot does not support changing an existing `STRING` or `BYTES` column to `UUID` in place. Adopt UUID through a new column/table and reingest or backfill. - **Multi-value behavior:** MV UUID is supported, subject to generic operator restrictions for MV/array values. - **Partitioning:** v1 does not add a dedicated raw-16-byte UUID segment partitioning contract; #18875 was closed intentionally. This is separate from the UUID hash option for upsert primary keys. - **Dimension-table lookup:** regular equality/hash joins are covered. The generic dimension-table `BYTES` primary-key lookup bug also affects BYTES-backed UUID keys and is tracked independently in #19228. That issue should be fixed and tested there rather than keeping this UUID umbrella open. - **Benchmarks:** #18140 remains the historical design and benchmark record. Stale standalone JMH classes were deliberately not landed as ongoing production-tree maintenance burden. All implementation layers, focused end-to-end coverage, and user documentation for the agreed UUID v1 scope have landed. Future UUID enhancements or defects can now be tracked as focused follow-up issues. <!-- uuid-v1-issue-16619-completion-summary --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
