TangSiyang2001 opened a new pull request, #21611:
URL: https://github.com/apache/doris/pull/21611
## Proposed changes
close:#21606
### Debug
Using jemalloc prof tool and found there is unreasonable memory overhead
within multi_table_put_result obj.
Found that there are loop refs during processing each single routine load
task, i.e. StreamLoadContext holds a shared ptr ref to MultiTablePipe, and
MultitablePipe holds shared ptr ref to StreamLoadContext as well.
Furthermore, during related loading process, StreamLoadContext's obj cnt is
always growing and do not decrease when load stream stopped.
### Resolve
Using a naked ptr rather than shared ptr in MultiTablePipe to hold the
StreamLoadContext, so that both the StreamLoadContext and the MultiTablePipe
will be released normally.
Why not use weak_ptr?
1. Cuz introducing weak_ptr is not so elegant and lower performance.
2. MultiTablePipe's life cycle is under control of the StreamLoadContext, it
is reasonable to use naked ptr to provide MultiTablePipe with an access to the
current ctx.
### Fix result
Memory usage during load process:
Obj cnt:
Observing the obj cnt
## Further comments
If this is a relatively large or complex change, kick off the discussion at
[[email protected]](mailto:[email protected]) by explaining why you
chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
