seanfulton commented on issue #29378:
URL: https://github.com/apache/doris/issues/29378#issuecomment-1875411260
I am seeing the same thing. We have a six-node cluster, three BE and three
FE. All are 64G of RAM. We are loading a series of MySQL tables from files. The
largest table is 1.5G. The files are a series of INSERT statements.
The BE nodes had 64G of RAM and we consistently ran into the limit of 52 G.
I increased the RAM to 96G and now we are getting the error below. Restarting
the BE process causes the RAM to drop, but once we restart the load, the RAM
creeps up until it exceeds the limit.
ON the
FreeTopOvercommitMemoryQuery:
WorkloadGroup:
I0103 08:02:27.237190 19832 daemon.cpp:262] [MemoryGC] start minor GC,
process memory used 72.75 GB exceed soft limit 69.12 GB or sys available memory
18.70 GB less than warning water mark 3.20 GB..
I0103 08:02:27.237272 19832 mem_tracker_limiter.cpp:522] [MemoryGC] GC free
process top memory overcommit query, , start seek all query, running query and
load num: 63
I0103 08:02:27.237390 19832 mem_tracker_limiter.cpp:560] [MemoryGC] GC free
process top memory overcommit query, seek finished, seek 63 tasks. among them,
0 tasks can be canceled; 63 small tasks that were skipped; 0 tasks is being
canceled and has not been completed yet;
I0103 08:02:27.237396 19832 mem_tracker_limiter.cpp:568] [MemoryGC] GC free
process top memory overcommit query, finished, no task need be canceled.
I0103 08:02:27.237417 19832 mem_info.cpp:128] [MemoryGC] end minor GC, free
memory 0. cost(us): 192, details: :
FreeTopOvercommitMemoryQuery:
WorkloadGroup:
I0103 08:02:27.793373 20963 olap_server.cpp:1065] cooldown producer get
tablet num: 0
I0103 08:02:28.338559 19832 daemon.cpp:262] [MemoryGC] start minor GC,
process memory used 72.75 GB exceed soft limit 69.12 GB or sys available memory
18.70 GB less than warning water mark 3.20 GB..
I0103 08:02:28.338682 19832 mem_tracker_limiter.cpp:522] [MemoryGC] GC free
process top memory overcommit query, , start seek all query, running query and
load num: 63
I0103 08:02:28.338913 19832 mem_tracker_limiter.cpp:560] [MemoryGC] GC free
process top memory overcommit query, seek finished, seek 63 tasks. among them,
0 tasks can be canceled; 63 small tasks that were skipped; 0 tasks is being
canceled and has not been completed yet;
I0103 08:02:28.338925 19832 mem_tracker_limiter.cpp:568] [MemoryGC] GC free
process top memory overcommit query, finished, no task need be canceled.
I0103 08:02:28.338955 19832 mem_info.cpp:128] [MemoryGC] end minor GC, free
memory 0. cost(us): 339, details: :
FreeTopOvercommitMemoryQuery:
WorkloadGroup:
I0103 08:02:29.440176 19832 daemon.cpp:262] [MemoryGC] start minor GC,
process memory used 72.75 GB exceed soft limit 69.12 GB or sys available memory
18.70 GB less than warning water mark 3.20 GB..
I0103 08:02:29.440274 19832 mem_tracker_limiter.cpp:522] [MemoryGC] GC free
process top memory overcommit query, , start seek all query, running query and
load num: 63
I0103 08:02:29.440418 19832 mem_tracker_limiter.cpp:560] [MemoryGC] GC free
process top memory overcommit query, seek finished, seek 63 tasks. among them,
0 tasks can be canceled; 63 small tasks that were skipped; 0 tasks is being
canceled and has not been completed yet;
I0103 08:02:29.440428 19832 mem_tracker_limiter.cpp:568] [MemoryGC] GC free
process top memory overcommit query, finished, no task need be canceled.
I0103 08:02:29.440454 19832 mem_info.cpp:128] [MemoryGC] end minor GC, free
memory 0. cost(us): 237, details: :
FreeTopOvercommitMemoryQuery:
WorkloadGroup:
I0103 08:02:30.061405 21622 heartbeat_server.cpp:61] get heartbeat from
FE.host:10.10.21.11, port:9020, cluster id:1280404001, counter:9637, BE start
time: 1704238503931
I0103 08:02:30.541870 19832 daemon.cpp:262] [MemoryGC] start minor GC,
process memory used 72.75 GB exceed soft limit 69.12 GB or sys available memory
18.70 GB less than warning water mark 3.20 GB..
I0103 08:02:30.541988 19832 mem_tracker_limiter.cpp:522] [MemoryGC] GC free
process top memory overcommit query, , start seek all query, running query and
load num: 63
I0103 08:02:30.542172 19832 mem_tracker_limiter.cpp:560] [MemoryGC] GC free
process top memory overcommit query, seek finished, seek 63 tasks. among them,
0 tasks can be canceled; 63 small tasks that were skipped; 0 tasks is being
canceled and has not been completed yet;
I0103 08:02:30.542183 19832 mem_tracker_limiter.cpp:568] [MemoryGC] GC free
process top memory overcommit query, finished, no task need be canceled.
I0103 08:02:30.542214 19832 mem_info.cpp:128] [MemoryGC] end minor GC, free
memory 0. cost(us): 292, details: :
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
