FelixYBW commented on code in PR #8908:
URL: https://github.com/apache/incubator-gluten/pull/8908#discussion_r1982390019
##########
docs/get-started/VeloxFeature.md:
##########
@@ -0,0 +1,79 @@
+---
+layout: page
+title: Stage-Level Resource Adjustment in Velox Backend
+nav_order: 3
+parent: Getting-Started
+---
+## Using Stage-Level Resource Adjustment to Avoid OOM(Experimental)
+---
+
+### **Overview**
+Apache Gluten introduces a stage-level resource auto-adjustment framework to
mitigate heap Out-of-Memory (OOM) issues caused by varying memory demands
across stages in Spark applications. This feature dynamically adjusts task and
executor resource profiles (e.g., heap/off-heap memory allocation) based on
stage characteristics, such as the presence of fallback operators or heavy
shuffle workloads(to be supported).
+
+This document explains how to configure and use this feature to avoid OOM
errors.
+
+---
+
+### **Prerequisites**
+1. **Enable Adaptive Query Execution (AQE)**:
+ ```properties
+ spark.sql.adaptive.enabled=true
+ ```
+2. **Enable Executor Dynamic Allocation**:
+ ```properties
+ spark.dynamicAllocation.enabled=true
+ ```
+3. **Resource Scheduler Compatibility**:
+ Ensure the underlying cluster resource manager (e.g., YARN, Kubernetes)
supports dynamic resource allocation.
+
+---
+
+### **Key Configurations**
+Add the following configurations to your Spark application:
+
+
+| Parameters |
Description
|
Default |
+|-------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|---------|
+| spark.gluten.auto.adjustStageResource.enabled |
Experimental: If enabled, gluten will try to set the stage resource according
to stage execution plan. Only worked when aqe is enabled at the same time!! |
false |
Review Comment:
"NOTE: Only works when aqe is enabled at the same time." avoid to use "!!"
in document
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
