lgbo-ustc commented on code in PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#discussion_r1687698636
##########
backends-clickhouse/src/main/java/org/apache/gluten/vectorized/BatchIterator.java:
##########
@@ -77,8 +80,17 @@ public IMetrics getMetricsInternal() {
return new NativeMetrics(nativeFetchMetrics(handle));
}
+ public void updatePeakMemory() {
+ if (TaskContext.get() != null) {
+
TaskContext.get().taskMetrics().setPeakExecutionMemory(nativeCurrentPeakMemory());
+ }
+ }
+
@Override
public void closeInternal() {
+ // CH backend uses Clickhouse to manage off heap and updates memory
statistics at the end of the
Review Comment:
只在结束时更新这个信息会有什么影响吗？
##########
cpp-ch/local-engine/Shuffle/ShuffleCommon.h:
##########
@@ -45,16 +45,10 @@ struct SplitOptions
std::string out_exprs;
std::string compress_method = "zstd";
int compress_level;
- size_t spill_threshold = 300 * 1024 * 1024;
+ size_t spill_threshold = 500 * 1024 * 1024;
Review Comment:
调大默认值可能有风险。shuffle占据更大的内存空间，会对上游的聚合/join/sort造成内存压力，引发更多的磁盘spill
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
