andygrove opened a new pull request, #5034: URL: https://github.com/apache/datafusion-comet/pull/5034
## Which issue does this PR close? There is no dedicated tracking issue; `uuid` was previously listed as planned (🔜) in the expression support guide. Closes #. ## Rationale for this change Spark's `uuid()` fell back to Spark because Comet had no native implementation. Spark generates RFC 4122 version 4 UUID strings via `org.apache.spark.sql.catalyst.util.RandomUUIDGenerator`, which seeds a Commons Math3 `MersenneTwister` with `randomSeed + partitionIndex` and, per row, draws two `nextLong()`s, masks in the version and variant bits, and formats via `java.util.UUID.toString`. Comet already ports that exact `MersenneTwister` for native `shuffle` (`SparkMersenneTwister`), so `uuid()` can run natively and produce results that match Spark bit for bit, keeping the enclosing operator on the native pipeline instead of falling back. ## What changes are included in this PR? This change was scaffolded with the project's `implement-comet-expression` skill (and audited with `audit-comet-expression`). - Native `UuidExpr` (`native/spark-expr/src/nondetermenistic_funcs/uuid.rs`), modeled on `ShuffleExpr`: it keeps a per-partition `SparkMersenneTwister` in a `Mutex` so state advances across batches, and produces a `Utf8` column. Added `SparkMersenneTwister::next_long()` (a port of Commons Math3 `BitsStreamGenerator.nextLong()`). - Proto `Uuid` message and oneof field, wired through the expression registry with a `UuidBuilder` that combines the seed with the partition index (matching `rand`/`randn`/`shuffle`). - `CometUuid` serde emitting the resolved random seed, registered in `QueryPlanSerde`. - Support-status doc flipped from planned to supported, and a `misc_funcs` audit entry recording the cross-version findings. The audit confirmed `RandomUUIDGenerator` is identical across Spark 3.4.3 / 3.5.8 / 4.0.1 / 4.1.1, so no shim is needed. The `uuid(seed)` SQL form exists only in Spark 4.0+, so the seeded tests are gated with `MinSparkVersion: 4.0`. ## How are these changes tested? - Rust unit tests in `uuid.rs`: RFC 4122 v4 format/version/variant, determinism per seed, and per-batch state continuity. - `expressions/misc/uuid.sql` (all versions): no-seed `uuid()` runs natively and satisfies deterministic properties (length, RFC 4122 layout, version/variant nibbles) that hold identically on Comet and Spark. - `expressions/misc/uuid_with_seed.sql` (Spark 4.0+): seeded `uuid(seed)` asserts bit-for-bit equality with Spark across single and multiple rows, negative/zero/boundary seeds, and combined with other expressions. Verified natively on Spark 4.1.2 (default) and 3.5.8 (seeded file correctly skipped). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
