[
https://issues.apache.org/jira/browse/DRILL-8557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Letian Jiang updated DRILL-8557:
--------------------------------
Summary: Velox-based native execution (was: Velox-based native execution
for Apache Drill)
> Velox-based native execution
> ----------------------------
>
> Key: DRILL-8557
> URL: https://issues.apache.org/jira/browse/DRILL-8557
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
> Reporter: Letian Jiang
> Priority: Major
> Attachments: 01-homogeneous.png, 02-heterogeneous.png,
> 03-sidecar.png, 04-execution-model.png
>
>
> h2. Motivation
> Explore Velox-based native execution to improve CPU efficiency and
> performance for analytical workloads while retaining Drill’s SQL planning,
> distributed scheduling, and storage plugin ecosystem.
> Velox already supports native execution in [Presto
> Native|https://github.com/prestodb/presto/blob/master/presto-native-execution/README.md]
> and [Spark SQL through
> Gluten|https://github.com/apache/gluten/blob/main/docs/get-started/Velox.md].
> [Gluten for
> Flink|https://github.com/apache/gluten/blob/main/gluten-flink/docs/Flink.md]
> also provides an experimental integration. These projects offer useful
> references for bringing native execution to Drill.
> h2. Proposed design
> Use a *homogeneous Drillbit architecture*: each Java Drillbit can embed the
> same C++ NativeEngine.
> * *Node discovery:* Each Drillbit retains one ZooKeeper registration and
> advertises optional native execution endpoints. The Foreman submits native
> fragments directly through RPC.
> * *Coordination and transport:* Java retains planning, query coordination,
> and the complete root fragment. Non-root fragments execute in C++ and
> exchange data through native RPC; results return to the Java root through
> Drill Data RPC.
> * *Deployment:* The same architecture supports single-node and multi-node
> deployments. The Foreman’s Drillbit may also execute native non-root
> fragments.
> !01-homogeneous.png|width=900!
> * *Execution model:* Each non-root minor fragment maps to one Velox Task,
> which runs pipelines and drivers using shared CPU and scan/I/O pools.
> * *Scan paths:* Existing Java plugin readers are reused through JNI in the
> current JVM. Optional native readers retain the plugin’s metadata and split
> planning while keeping non-root scanning, computation, and exchange in C++,
> avoiding Java/native FFI on that data processing path.
> !04-execution-model.png|width=900!
> h2. Dynamic schema boundary
> The experimental native path supports initial runtime schema discovery, but
> requires the schema to remain stable after native plan binding. Subsequent
> schema changes currently fail the query. The existing Java execution path
> remains available for workloads requiring dynamic schema handling.
> h2. Current status
> I am currently working on a PoC to validate this design, including Java
> plugin reuse, optional native scans, fragment execution and exchange,
> correctness, cancellation, and performance comparisons with Java execution.
> h2. Alternative designs
> *Heterogeneous Drillbits:* Java Drillbits handle query coordination and root
> fragments; separate native Drillbits execute non-root fragments. Both node
> types register independently and communicate through RPC. Maintaining two
> node types increases deployment and lifecycle management complexity.
> !02-heterogeneous.png|width=900!
> *Native executor sidecar:* Each Java Drillbit runs alongside a separate C++
> executor, sharing one logical node identity and communicating through RPC.
> This adds process management overhead; reusing Java readers through JNI also
> requires a JVM inside the sidecar, since JNI cannot cross process boundaries.
> !03-sidecar.png|width=900!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
