[
https://issues.apache.org/jira/browse/HDDS-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-11660:
----------------------------------
Labels: pull-request-available (was: )
> Recon List Key API: Reduce object creation and buffering memory
> ---------------------------------------------------------------
>
> Key: HDDS-11660
> URL: https://issues.apache.org/jira/browse/HDDS-11660
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Recon
> Reporter: Stephen O'Donnell
> Assignee: Stephen O'Donnell
> Priority: Major
> Labels: pull-request-available
>
> The current ListKeysAPI implementation, pulls a batch of OMKeyInfo objects
> from the File / Key table and buffers them in an array.
> Then it converts each item in that array to a KeyEntityInfo object, which has
> JSON annotations allowing the data to be serialized over HTTP.
> This results in two copies of the objects buffered in memory at once.
> Additionally, the OMKeyInfo object is a fairly heavyweight object, and
> creating it from protobuf also deserialized the block information, which is
> not needed for the ListKey purposes.
> Further taking the protobuffer bytes from RocksDB creates a KeyInfo object
> (deserialized proto) and then this object is copied into OmKeyInfo.
> So for each key retrieved, we start with:
> 1. A deserialized proto object (KeyInfo)
> 2. A copy of the proto fields (OmKeyInfo)
> 3. A json annotated object (KeyEntityInfo).
> By deserializing into the 3rd object directly, and having it wrap a keyInfo
> object and delegate calls to it when appropriate, we avoid needing the
> heavyweight OmKeyInfo object, and make the KeyEntityInfo object "slimmer". We
> can also avoid the double buffering, creating only a single list of objects
> for the batch.
> This PR introduces a KeyEntityInfoProtoWrapper object, rather than changing
> the existing KeyEntityInfo object. The reason, is that other APIs outside of
> ListKeys use it. I am not ready to change those APIs now, however after this
> change gets committed and we are happy with it, it should be possible to
> apply similar changes to those other API and eventually remove KeyEntityInfo
> from the system.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
