wForget commented on issue #9938:
URL:
https://github.com/apache/incubator-gluten/issues/9938#issuecomment-2979286426
`HashBuild` reserved a large amount of memory and disabled spilling memory
pool, which causes the `SortBuffer` of OrderBy to generate a large number of
spill files.
```
W20250617 13:41:24.180392 190049 HashBuild.cpp:223] Exceeded spill level
limit: -1, and disable spilling for memory pool: op.3.1.0.HashBuild
I20250617 13:41:28.797995 190049 VeloxMemoryManager.cc:255]
Shrink[root/root]: Trying to shrink 47290775 bytes of data...
I20250617 13:41:28.798067 190049 VeloxMemoryManager.cc:256]
Shrink[root/root]: Pool has reserved
1572860160/1583349760/1583349760/9223372036854775807 bytes.
I20250617 13:41:28.798090 190049 VeloxMemoryManager.cc:258]
Shrink[root/root]: Velox memory usage stats...
I20250617 13:41:28.798102 190049 VeloxMemoryManager.cc:259]
Shrink[root/root]: peak: 1600126976
current: 1572860160
children {
key: "default_leaf"
value {
}
}
children {
key: "task.Gluten_Stage_5_TID_9347_VTID_10"
value {
peak: 1600126976
current: 1572860160
children {
key: "node.0"
value {
children {
key: "op.0.0.0.ValueStream"
value {
}
}
}
}
children {
key: "node.1"
value {
peak: 134217728
current: 33751040
children {
key: "op.1.0.0.Aggregation"
value {
peak: 103482112
current: 33751040
}
}
}
}
children {
key: "node.2"
value {
children {
key: "op.2.1.0.ValueStream"
value {
}
}
}
}
children {
key: "node.3"
value {
peak: 1561329664
current: 1538322688
children {
key: "op.3.0.0.HashProbe"
value {
peak: 16859520
current: 8970496
}
}
children {
key: "op.3.1.0.HashBuild"
value {
peak: 1546129408
current: 1529352192
}
}
}
}
children {
key: "node.4"
value {
children {
key: "op.4.0.0.FilterProject"
value {
}
}
}
}
children {
key: "node.5"
value {
peak: 2097152
current: 786432
children {
key: "op.5.0.0.FilterProject"
value {
peak: 1597440
current: 786432
}
}
}
}
children {
key: "node.6"
value {
peak: 1048576
children {
key: "op.6.0.0.FilterProject"
value {
peak: 24576
}
}
}
}
children {
key: "node.7"
value {
peak: 9437184
children {
key: "op.7.0.0.OrderBy"
value {
peak: 4718592
}
}
}
}
children {
key: "node.8"
value {
children {
key: "op.8.0.0.FilterProject"
value {
}
}
}
}
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
