pedrumj2 opened a new issue, #13014:
URL: https://github.com/apache/gluten/issues/13014
### Description
### Description
**Proposal: Infer the Java signature from the Velox C++ UDF**
**Why?** The velox C++ UDF implementation already contains the necessary
information needed to derive the signature, defining the java class introduces
an extra step for the user that can be avoided. From a user experience point of
view, user can then write a Velox C++ udf once and have it available to
different backends out of the box (i.e they get spark out of the box for free).
The following section shares the current steps needed to define a UDF. After
this proposal:
- Step1 would no longer be needed
- Step4 would no longer be needed
- Step5 would no longer need the `CREATE TEMPORARY FUNCTION my_new_udf AS
'com.example.udf.MyNewUdf';`
- Step6 would no longer need the `--jars "$BUNDLE",out/mynewudf.jar ` and
` --driver-class-path "$BUNDLE:out/mynewudf.jar" `
## Current State
- A UDF that only ever runs on Gluten needs a Java class written, compiled,
jarred and shipped purely as a name.
- The Java body and the C++ body must be kept in sync by hand; nothing
enforces it.
- Any change to the signature has to be made twice, in two languages.
### [step1] Write the Java UDF
> Not needed after this proposal.
```java
package com.example.udf;
import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.io.Text;
public class MyNewUdf extends UDF {
public Text evaluate(Text a, Text b) {
throw new UnsupportedOperationException(
"com.example.udf.MyNewUdf is implemented natively in Velox; "
+ "reaching this method means the query fell back to the JVM.");
}
}
```
### [step2] Write the C++ Velox UDF
Put `MyNewUdf.cc` in `cpp/velox/udf/examples/` so it inherits the flags the
shipped examples use.
```cpp
#include <velox/functions/Macros.h>
#include <velox/functions/Registerer.h>
#include "udf/Udf.h"
#include "udf/examples/UdfCommon.h" // gluten::UdfRegisterer
using namespace facebook::velox;
namespace {
static const char* kVarChar = "varchar";
template <typename T>
struct MyNewUdfFunction {
VELOX_DEFINE_FUNCTION_TYPES(T);
FOLLY_ALWAYS_INLINE void call(
out_type<Varchar>& result,
const arg_type<Varchar>& a,
const arg_type<Varchar>& b) {
result.append(a.data());
result.append(" ");
result.append(b.data());
}
};
class MyNewUdfRegisterer final : public gluten::UdfRegisterer {
public:
int getNumUdf() override { return 1; }
void populateUdfEntries(int& i, gluten::UdfEntry* e) override {
e[i++] = {name_.c_str(), kVarChar, 2, arg_, false, true};
}
void registerSignatures() override {
registerFunction<MyNewUdfFunction, Varchar, Varchar, Varchar>({name_});
}
private:
const std::string name_ = "com.example.udf.MyNewUdf";
const char* arg_[2] = {kVarChar, kVarChar};
};
} // namespace
...
// globalRegisters(), setupRegisterers(), and the DEFINE_GET_NUM_UDF /
// DEFINE_GET_UDF_ENTRIES / DEFINE_REGISTER_UDF macros: copy verbatim from
MyUDF.cc.
```
`name_` must equal the Java class name — that is the binding this proposal
removes.
References:
[MyUDF.cc](https://github.com/apache/gluten/blob/main/cpp/velox/udf/examples/MyUDF.cc),
[UdfCommon.h](https://github.com/apache/gluten/blob/main/cpp/velox/udf/examples/UdfCommon.h),
[Udf.h](https://github.com/apache/gluten/blob/main/cpp/velox/udf/Udf.h),
[VeloxUDF.md](https://github.com/apache/gluten/blob/main/docs/developers/VeloxUDF.md)
### [step3] Build the `.so`
Append to
[cpp/velox/udf/examples/CMakeLists.txt](https://github.com/apache/gluten/blob/main/cpp/velox/udf/examples/CMakeLists.txt):
```cmake
add_library(mynewudf SHARED "MyNewUdf.cc")
target_link_libraries(mynewudf velox)
```
```bash
docker run --rm -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c
'
source /opt/rh/gcc-toolset-12/enable
export VELOX_BUILD_SHARED=ON
cd /work/gluten && ./dev/builddeps-veloxbe.sh --build_examples=ON
'
```
`--build_examples=ON` is what pulls `udf/examples` into the build.
Incremental on an already-built tree: 24 seconds. Produces
`cpp/build/velox/udf/examples/libmynewudf.so`.
Use the `centos-9` image; `centos-8` ships an fbthrift predating
`readLEFromBuffer` and Velox's Parquet reader will not compile against it.
### [step4] Compile the Java UDF into a jar
> Not needed after this proposal.
```bash
javac -cp "$SPARK_HOME/jars/*" -d out/classes MyNewUdf.java
jar cf out/mynewudf.jar -C out/classes .
```
### [step5] Write the query
> After this proposal the `CREATE TEMPORARY FUNCTION` line would no longer
be needed.
```sql
CREATE TEMPORARY FUNCTION my_new_udf AS 'com.example.udf.MyNewUdf';
CREATE TABLE IF NOT EXISTS greetings AS
SELECT * FROM VALUES ('hello'), ('bonjour'), ('hola') AS t(word);
SELECT word, my_new_udf(word, 'world') AS greeting FROM greetings ORDER BY
word;
EXPLAIN SELECT my_new_udf(word, 'world') FROM greetings;
```
### [step6] Run the query
> After this proposal the bundle jar would no longer need to go on the
classpath.
```bash
$SPARK_HOME/bin/spark-sql \
--master 'local[2]' \
--jars "$BUNDLE",out/mynewudf.jar \
--driver-class-path "$BUNDLE:out/mynewudf.jar" \
--conf spark.plugins=org.apache.gluten.GlutenPlugin \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=4g \
--conf
spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
--conf
spark.gluten.sql.columnar.backend.velox.udfLibraryPaths=file:///abs/path/libmynewudf.so
\
--conf
spark.gluten.sql.columnar.backend.velox.driver.udfLibraryPaths=file:///abs/path/libmynewudf.so
\
--conf spark.sql.catalogImplementation=hive \
-f demo.sql
```
#### Results
Verified end to end on 2026-09-13 against `main` at commit `84a2b7a`, Spark
3.5.9.
```
bonjour bonjour world
hello hello world
hola hola world
Time taken: 1.789 seconds, Fetched 3 row(s)
```
```
== Physical Plan ==
VeloxColumnarToRow
+- ^(1) ProjectExecTransformer
[HiveSimpleUDF#com.example.udf.MyNewUdf(word#12,world) AS my_new_udf(word,
world)#13]
+- ^(1) InputIteratorTransformer[word#12]
+- RowToVeloxColumnar
+- Scan hive spark_catalog.default.greetings [word#12],
HiveTableRelation [`spark_catalog`.`default`.`greetings`,
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Data Cols: [word#12],
Partition Cols: []]
```
### Gluten version
main branch
### Gluten version
main branch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
