Hello Surya Hebbar, Michael Smith, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24486
to look at the new patch set (#7).
Change subject: IMPALA-14178: Track profile output memory
......................................................................
IMPALA-14178: Track profile output memory
Add daemon-level memory trackers for temporary runtime profile
representations generated by profile output requests. The JSON path
tracks the full RapidJSON allocator capacity while the temporary
profile document is live, including reused allocator chunks.
Web profile handlers build the profile in a tracker-scoped document
and deep-copy it into the response document. Internal timeline JSON
metadata is built in that same document so the response keeps the
existing profile_json contract.
Name the trackers RuntimeProfileTemporaryJson and
RuntimeProfileTemporaryThrift to make their lifetime explicit. The
Webserver-owned response document and serialization buffers are
intentionally outside this scope and are documented at the copy
boundary.
What the trackers do not cover is the peak. A web handler builds the
profile under the tracker and deep-copies it into the Webserver-owned
response document, so both copies are live at the copy boundary and only
one of them is counted. Counting the other means holding the response
document under a tracker the Webserver owns, which is a wider change
than this one.
The Thrift estimate walks the fields of RuntimeProfile.thrift by hand,
so a field added there and not here goes uncounted and the tracker
reports low. The two structs now say so where the fields are declared,
which is where someone adding one will be looking.
The Thrift path estimates heap storage owned by the generated
TRuntimeProfileTree containers, including ordered-container nodes and
external string storage. Scoped accounting remains active until the
caller-owned temporary output has been destroyed, including partially
populated Thrift output on a deserialization error.
Testing:
- git diff --check
- python3 bin/jenkins/critique-gerrit-review.py --dryrun
- Focused Linux build: ninja java unifiedbetests; 10 focused profile
tests passed, including timeline JSON ownership, tracker lifecycle,
and Thrift estimate coverage
Assisted-by: gpt-5.6-sol (OpenAI Codex)
Change-Id: I7114794cb8d26ef0aef4d7d620f041cccffc3bde
---
M be/src/service/impala-http-handler-test.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-http-handler.h
M be/src/service/impala-server-test.cc
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/util/runtime-profile-test.cc
M be/src/util/runtime-profile.cc
M be/src/util/runtime-profile.h
M common/thrift/RuntimeProfile.thrift
10 files changed, 541 insertions(+), 12 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/24486/7
--
To view, visit http://gerrit.cloudera.org:8080/24486
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7114794cb8d26ef0aef4d7d620f041cccffc3bde
Gerrit-Change-Number: 24486
Gerrit-PatchSet: 7
Gerrit-Owner: Aleksandr Efimov <[email protected]>
Gerrit-Reviewer: Aleksandr Efimov <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Surya Hebbar <[email protected]>
