Letian Jiang created DRILL-8557:
-----------------------------------
Summary: Velox-based native execution for Apache Drill
Key: DRILL-8557
URL: https://issues.apache.org/jira/browse/DRILL-8557
Project: Apache Drill
Issue Type: Improvement
Components: Execution - Flow
Reporter: Letian Jiang
Attachments: 01-homogeneous.png, 02-heterogeneous.png, 03-sidecar.png,
04-execution-model.png
h2. Motivation
Explore Velox-based native execution to improve CPU efficiency and performance
for analytical workloads while retaining Drill’s SQL planning, distributed
scheduling, and storage plugin ecosystem.
Velox already supports native execution in [Presto
Native|https://github.com/prestodb/presto/blob/master/presto-native-execution/README.md]
and [Spark SQL through
Gluten|https://github.com/apache/gluten/blob/main/docs/get-started/Velox.md].
[Gluten for
Flink|https://github.com/apache/gluten/blob/main/gluten-flink/docs/Flink.md]
also provides an experimental integration. These projects offer useful
references for bringing native execution to Drill.
h2. Proposed design
Use a *homogeneous Drillbit architecture*: each Java Drillbit can embed the
same C++ NativeEngine.
* *Node discovery:* Each Drillbit retains one ZooKeeper registration and
advertises optional native execution endpoints. The Foreman submits native
fragments directly through RPC.
* *Coordination and transport:* Java retains planning, query coordination, and
the complete root fragment. Non-root fragments execute in C++ and exchange data
through native RPC; results return to the Java root through Drill Data RPC.
* *Deployment:* The same architecture supports single-node and multi-node
deployments. The Foreman’s Drillbit may also execute native non-root fragments.
!01-homogeneous.png|width=900!
* *Execution model:* Each non-root minor fragment maps to one Velox Task, which
runs pipelines and drivers using shared CPU and scan/I/O pools.
* *Scan paths:* Existing Java plugin readers are reused through JNI in the
current JVM. Optional native readers retain the plugin’s metadata and split
planning while keeping non-root scanning, computation, and exchange in C++,
avoiding Java/native FFI on that data processing path.
!04-execution-model.png|width=900!
h2. Dynamic schema boundary
The experimental native path supports initial runtime schema discovery, but
requires the schema to remain stable after native plan binding. Subsequent
schema changes currently fail the query. The existing Java execution path
remains available for workloads requiring dynamic schema handling.
h2. Current status
I am currently working on a PoC to validate this design, including Java plugin
reuse, optional native scans, fragment execution and exchange, correctness,
cancellation, and performance comparisons with Java execution.
h2. Alternative designs
*Heterogeneous Drillbits:* Java Drillbits handle query coordination and root
fragments; separate native Drillbits execute non-root fragments. Both node
types register independently and communicate through RPC. Maintaining two node
types increases deployment and lifecycle management complexity.
!02-heterogeneous.png|width=900!
*Native executor sidecar:* Each Java Drillbit runs alongside a separate C++
executor, sharing one logical node identity and communicating through RPC. This
adds process management overhead; reusing Java readers through JNI also
requires a JVM inside the sidecar, since JNI cannot cross process boundaries.
!03-sidecar.png|width=900!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
