viirya commented on PR #6883:
URL: 
https://github.com/apache/datafusion-comet/pull/6883#issuecomment-6923958697

   Thanks @andygrove for the approval and the careful passes.
   
   On sharing: now that #7299 is in, I think most of it can be shared as is. 
`assembly::build_plan` turns a `QueryRequest` into a DataFusion plan of 
`CometFragmentExec` stages joined by `RepartitionExec` and 
`CoalescePartitionsExec`, with nothing from Ballista in it. `CometFragmentExec` 
already does what local execution needs from a stage. It plans each partition's 
block when the partition runs, on a blocking thread. It also reserves from the 
memory pool and spills through the disk manager of the `TaskContext` it runs 
with, which for a local query are the Spark task's. So the plan I have in mind:
   
   1. **Driver.** Move #7289's stage-graph builder and its strict-native check 
out of `CometNativeExecutorsExec` into a shared object that returns a 
`QueryRequest`. `CometLocalRule` admits a plan when the builder accepts it, and 
when the builder refuses one, the query keeps ordinary execution instead of 
failing. Admission keeps its own checks for what the builder does not cover, 
such as the local master and partition-dependent expressions.
   2. **Native.** Move `build_plan` and `exchange` from the `native-executors` 
workspace into core next to `CometFragmentExec`, so libcomet can build the 
graph. The JNI call then takes the `QueryRequest` instead of one block, and 
`native/local` runs the root stage's partitions as it does now.
   3. **What local needs on top.**
      - **Root stage.** `native/local` requires that the root's partitions not 
share an exchange. A root partition waiting for its turn would otherwise leave 
the exchange buffering for the others. A root stage that reads an exchange 
needs to drain its partitions concurrently into spillable buffers, or end in a 
merge.
      - **Range exchanges.** Range exchanges are read as one partition today. 
Local can sample in process, as #7289's scheduler does.
      - **Shared stages.** A stage that several stages read runs once for each. 
In one process its output could be shared, which matters most for broadcasts.
      - **Partition plans.** Each file-reading partition's copy of the block 
sits in the partition descriptor for the whole query. That is the per-partition 
copy sunchao raised here, and it would be good to make it lazy for both modes.
   
   Both modes would then share the driver's stage graph, the wire format and 
the stage assembly. They would differ only in what runs the plan: Ballista's 
scheduler and shuffles, or `native/local` in the Spark task.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to