jinchengchenghh commented on code in PR #8882:
URL: https://github.com/apache/incubator-gluten/pull/8882#discussion_r1977769255
##########
docs/get-started/Velox.md:
##########
@@ -545,6 +545,46 @@ I20231121 10:19:42.348845 90094332
WholeStageResultIterator.cc:220] Native Plan
queuedWallNanos sum: 2.00us, count: 1, min: 2.00us, max:
2.00us
```
+
+## Broadcast Build Relations to Off-Heap(Experimental)
+
+The experimental feature **Off-Heap Broadcast Build Relations** aims to
mitigate out-of-memory (OOM) issues caused by heap memory consumption during
broadcast operations. Detailed design
+can be found
[here](https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?tab=t.0)
+
+### Purpose & how it works
+- **Avoid OOM**: Prevent OOM errors when broadcasting large datasets.
+- **Reduce Heap Memory Usage**: Store broadcast build relations in Spark
off-heap memory instead of on-heap memory
+
+### Configuration
+
+### Enable Off-Heap Broadcast
+Set the following configuration in your Spark session to enable the feature:
+
+| Property | Default |
Description |
Review Comment:
I don't think we need the detailed information to say how to set the config.
Can you describe the potential shortage,so we set it default false
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
