karuppayya opened a new issue, #5116:
URL: https://github.com/apache/datafusion-comet/issues/5116
### What is the problem the feature request solves?
Provisioning executor memory for Comet is one of the longest-standing
operational pain points.
Native execution wants large **off-heap** memory; JVM (fallback) execution
wants large **on-heap** memory. Today both are set on the same executor for the
whole application, so you either
- over-provision both (wasted memory per executor) or
- pick a compromise that starves whichever path a given query actually uses.
Until recently, a single query stage could interleave native (off-heap) and
JVM (on-heap) operators, so no single memory configuration could ever be
correct for it.
As of #4519 (stage-based fallback), a query stage is **homogeneous** — it
either runs fully native or is reverted entirely to JVM row-based execution at
the shuffle boundary. Native and JVM operators no longer interleave within a
stage. This makes per-stage memory tuning meaningful.
### Describe the potential solution
A Spark stage maps cleanly onto a Spark **ResourceProfile** (profiles attach
to RDDs, and a stage is a set of RDDs bounded by shuffles). Now that stages are
homogeneous, Comet can attach a ResourceProfile to the RDD it generates per
stage, based on the native-vs-JVM decision:
- **Native stages** → profile with large off-heap, minimal heap
- **Reverted JVM stages** → profile with large heap, minimal off-heap
Each stage then gets memory shaped for how it actually executes, instead of
one compromise config for the whole application.
Example profiles:
```scala
// Native stages: off-heap heavy
val nativeProfile = new ResourceProfileBuilder()
.require(new ExecutorResourceRequests()
.memory("2g") // small heap
.offHeap("14g") // large off-heap for native execution
.cores(4))
.build()
// JVM stages: on-heap heavy
val jvmProfile = new ResourceProfileBuilder()
.require(new ExecutorResourceRequests()
.memory("14g") // large heap for JVM row execution
.offHeap("2g") // minimal off-heap
.cores(4))
.build()
```
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
