L. C. Hsieh created SPARK-61774:
-----------------------------------

             Summary: Add columnar scalar UDFs, evaluated by an operator that 
native engines can replace
                 Key: SPARK-61774
                 URL: https://issues.apache.org/jira/browse/SPARK-61774
             Project: Spark
          Issue Type: New Feature
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: L. C. Hsieh


Spark's scalar UDFs are evaluated row by row. Over a columnar source this costs 
a conversion per row, most visibly for strings, where each value is decoded to 
a Java String and encoded back. It also leaves native engines that replace 
Spark's operators without a function boundary they can take over: an engine 
either runs the row-based function itself or defines a private mechanism that 
plain Spark cannot run.

This proposes a columnar scalar UDF:

* {{ColumnarScalarFunction}}, a JVM interface that takes the arguments as a 
{{ColumnarBatch}} and returns a {{ColumnVector}}.
* A function definition that holds an optional JVM implementation and a list of 
opaque engine implementations (a kind and properties), which Spark does not 
interpret.
* A logical and a physical operator, {{EvalColumnarUDFs}}, that the calls are 
extracted into, as Python UDFs are. The physical operator is columnar, passes 
its input columns through and appends one column per call.

Spark runs the JVM implementation. An engine that replaces the physical 
operator can run its own implementation of the same function instead, and the 
query falls back to the JVM implementation when it does not. A function with no 
JVM implementation is allowed and fails at execution if no engine has taken the 
operator.

The operator targets columnar input. Over a row-based child, Spark inserts its 
usual row to columnar transition.

In a benchmark over Parquet, the JVM columnar implementation is 1.2x to 2.6x 
faster than the equivalent Scala UDF, with the larger gains on string functions.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to