[
https://issues.apache.org/jira/browse/KNOX-3444?focusedWorklogId=1040849&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-1040849
]
ASF GitHub Bot logged work on KNOX-3444:
----------------------------------------
Author: ASF GitHub Bot
Created on: 11/Sep/26 07:19
Start Date: 11/Sep/26 07:19
Worklog Time Spent: 10m
Work Description: bonampak opened a new pull request, #1392:
URL: https://github.com/apache/knox/pull/1392
[KNOX-3444](https://issues.apache.org/jira/browse/KNOX-3444) - Exclude role
types that are not referenced by ServiceModelGenerators
## What changes were proposed in this pull request?
During CM discovery only cache role configurations that
ServiceModelGenerators actually use.
## How was this patch tested?
Updated unit tests and did manual test on a cluster.
Verified that the ClusterConfiguration cache persisted after discovery is
the same as before the changes were applied and the topologies have the same
generated service URLs.
Acquired a heap dump using jmap:
```bash
jmap -dump:live,format=b,file="/tmp/knox-heap-jmap-${knox_pid}".hprof
"${knox_pid}";
```
Used Eclipse Memory Analyzer Tool and Calcite SQL plugin to analyze the heap
usage of the ApiRoleConfig objects cached in
`ClouderaManagerServiceDiscoveryRepository`
```sql
SELECT toString(r.this['roleType']) roleType, COUNT(r.this) as cnt,
AVG(retainedSize(r.this)) as avg_retained_size, SUM(retainedSize(r.this))
sum_retained_size
FROM "com.cloudera.api.swagger.model.ApiRoleConfig" r
GROUP BY toString(r.this['roleType'])
ORDER BY sum_retained_size desc, roleType
```
The unused roles (e.g. NODEMANAGER, REGIONSERVER, GATEWAY, DATANODE,
KNOX_GATEWAY, KAFKA_BROKER, OZONE_DATANODE, etc.) are not cached. For each
unused role, the memory savings are 60K - 150K by role instance not cached
(with an average of roughly 100K).
## Integration Tests
N/A. Integration tests would require a live CM dependency. Setting up a real
(or faithfully simulated) Cloudera Manager server with representative
services/roles is heavy infrastructure that the project's test suite does not
provide, and a mock-based "integration" test would exercise the same code paths
the unit tests already cover.
Issue Time Tracking
-------------------
Worklog Id: (was: 1040849)
Remaining Estimate: 0h
Time Spent: 10m
> Optimize CM discovery memory usage by only caching role configurations that
> ServiceModelGenerators actually use
> ---------------------------------------------------------------------------------------------------------------
>
> Key: KNOX-3444
> URL: https://issues.apache.org/jira/browse/KNOX-3444
> Project: Apache Knox
> Issue Type: Task
> Components: cm-discovery
> Reporter: Tamás Marcinkovics
> Assignee: Tamás Marcinkovics
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently CM auto discovery fetches role configurations for every service
> role that is not explicitly excluded, using full view.
> The response role config items include a lot of metadata, such as description
> and validation, for every role instance in the service.
> For some role types that currently no ServiceModelGenerators use (DATANODE,
> GATEWAY, NODEMANAGER, REGIONSERVER) the role config items list in the
> response takes up around 200KB memory; for large clusters with 500+ nodes,
> this will result in a heap usage of 40-100 MB per role type, and the
> in-memory discovery cache can exceed 1GB.
> We should not cache the role config items not used by ServiceModelGenerators.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
