[
https://issues.apache.org/jira/browse/IMPALA-14641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18048497#comment-18048497
]
Arnab Karmakar commented on IMPALA-14641:
-----------------------------------------
As [~csringhofer] pointed out some superb points in the patch
[https://gerrit.cloudera.org/#/c/23811/|http://example.com/]
1. The storage of the thrift requests could be a problem when the request size
is too large (with several partitions/files). While most requests would be
small, few of them could have substantial size of requests.
2. I was serialising the requests immediately which is wasteful as most of the
json format will not be requested.
Considering the solutions that Csaba proposed, we could do this:
1. We'll keep a size limit of *1KB* when *archiving* the finished operations to
prevent memory consumption and with default
{*}catalog_operation_log_size=100{*}, this would keep extra mem usage under
{*}100KB{*}.
2. Full request details remain available for {*}in-flight operations{*}.
3. Request objects are now stored as references (not serialised) for in-flight
operations.
4. Json serialisation happens *lazily* only when the web page is accessed.
[~stigahuang] Could you please provide some pointers and let us know if any
changes are needed to the above?
> Details link for DDL/DML operations
> -----------------------------------
>
> Key: IMPALA-14641
> URL: https://issues.apache.org/jira/browse/IMPALA-14641
> Project: IMPALA
> Issue Type: Improvement
> Reporter: Quanlong Huang
> Assignee: Arnab Karmakar
> Priority: Major
> Attachments: Screenshot 2025-12-29 at 3.53.11 PM.png, Screenshot
> 2025-12-29 at 3.53.26 PM.png, Screenshot 2025-12-29 at 3.53.33 PM.png
>
>
> IMPALA-9118 shows the in-flight and finished DDL/DMLs in /operations page of
> Catalogd. The details column is used to show some critical query options.
> It'd be more useful to replace that with a link which can open a page to show
> more details:
> * The formatted thrift request sent to catalogd
> * The timeline of the execution in catalogd
> * (Optional) byte sizes of the request and response
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
