pchintar opened a new issue, #3840:
URL: https://github.com/apache/datafusion-comet/issues/3840
Description
The following Spark expressions currently fall back to JVM execution in
Comet:
- 'SEC'
- 'CSC'
- 'CBRT'
- 'HYPOT'
- 'FACTORIAL'
- 'ShiftRightUnsigned'
This reduces native execution coverage and introduces avoidable performance
overhead for math-heavy and bitwise workloads.
---
Proposed Solution
Add native support using Comet’s existing scalar-function pathway:
1. Map Catalyst expressions in QueryPlanSerde.scala:
- Sec -> "sec"
- Csc -> "csc"
- Cbrt -> "cbrt"
- Hypot -> "hypot"
- Factorial -> "factorial"
- ShiftRightUnsigned -> "shiftrightunsigned"
2. Implement corresponding native functions in Rust using ScalarUDFImpl
3. Register functions in comet_scalar_funcs.rs
This approach requires:
- No protobuf changes
- No planner changes
- Reuses existing scalar-function infrastructure
---
Implementation Notes
- Vectorized Arrow-based implementation
- Null propagation preserved
- Factorial bounded (e.g., 0–20)
- ShiftRightUnsigned supports:
- Int32 >> Int32
- Int64 >> Int32
---
Expected Impact
- Eliminates fallback
- Improves performance
- Expands expression coverage
---
Related
#240
---
Planned Changes (High-Level)
The implementation touches the following areas:
- native/spark-expr/src/math_funcs/math_expr.rs (new file created)
- native/spark-expr/src/math_funcs/mod.rs
- native/spark-expr/src/lib.rs
- native/spark-expr/src/comet_scalar_funcs.rs
- spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
No changes required in:
- protobuf definitions
- native planner
The implementation follows the existing scalar-function pathway.
I have already implemented and validated this locally (native execution
confirmed).
I will open a PR once aligned.
---
cc @andygrove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
