L. C. Hsieh created SPARK-61774:
-----------------------------------
Summary: Add columnar scalar UDFs, evaluated by an operator that
native engines can replace
Key: SPARK-61774
URL: https://issues.apache.org/jira/browse/SPARK-61774
Project: Spark
Issue Type: New Feature
Components: SQL
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
Spark's scalar UDFs are evaluated row by row. Over a columnar source this costs
a conversion per row, most visibly for strings, where each value is decoded to
a Java String and encoded back. It also leaves native engines that replace
Spark's operators without a function boundary they can take over: an engine
either runs the row-based function itself or defines a private mechanism that
plain Spark cannot run.
This proposes a columnar scalar UDF:
* {{ColumnarScalarFunction}}, a JVM interface that takes the arguments as a
{{ColumnarBatch}} and returns a {{ColumnVector}}.
* A function definition that holds an optional JVM implementation and a list of
opaque engine implementations (a kind and properties), which Spark does not
interpret.
* A logical and a physical operator, {{EvalColumnarUDFs}}, that the calls are
extracted into, as Python UDFs are. The physical operator is columnar, passes
its input columns through and appends one column per call.
Spark runs the JVM implementation. An engine that replaces the physical
operator can run its own implementation of the same function instead, and the
query falls back to the JVM implementation when it does not. A function with no
JVM implementation is allowed and fails at execution if no engine has taken the
operator.
The operator targets columnar input. Over a row-based child, Spark inserts its
usual row to columnar transition.
In a benchmark over Parquet, the JVM columnar implementation is 1.2x to 2.6x
faster than the equivalent Scala UDF, with the larger gains on string functions.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]