[
https://issues.apache.org/jira/browse/HDDS-7715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kohei Sugihara updated HDDS-7715:
---------------------------------
Description:
keyManagerImpl#listStatus for FSO buckets serves a KeyLocationList but might be
large size of response (> 100 MB) and exceed the limit. The problem is
KeyLocation field is large enough (> 1MB) so just listing some multi-part
uploaded keys exceeds the limit. A possible solution is to reduce KeyLocation
size by omitting pipeline fields.
{quote}% ozone sh key ls /s3v/bucket --prefix=path/to/prefix
RPC response exceeds maximum data length
{quote}
We checked a List<OzoneFileStatus> in our environment and it turned out that
protobufs corresponding to some multi-part keys are larger than 1MB. AFAIK, the
maximum IPC length is 100MB by default so it easily exceeds the limit.
!large-protos.png!
was:
keyManagerImpl#listStatus for FSO buckets serves a KeyLocationList but might be
large size of response (> 100 MB) and exceed the limit. The problem is
KeyLocation field is large enough (> 1MB) so just listing some multi-part
uploaded keys exceeds the limit. A possible solution is to reduce KeyLocation
size by omitting pipeline fields.
{quote}% ozone sh key ls /s3v/bucket --prefix=path/to/prefix
RPC response exceeds maximum data length
{quote}
> keyManagerImpl#listStatus exceeds the maximum RPC length
> --------------------------------------------------------
>
> Key: HDDS-7715
> URL: https://issues.apache.org/jira/browse/HDDS-7715
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM, Ozone Manager
> Affects Versions: 1.3.0
> Environment: Version: Our forked version based ozone-1.3.0 branch.
> commit hash is
> [https://github.com/apache/ozone/commit/9c61a8aa497ab96c014ad3bb7b1ee4f731ebfaf8]
> FSO-enabled bucket
> Reporter: Kohei Sugihara
> Priority: Major
> Attachments: large-protos.png
>
>
> keyManagerImpl#listStatus for FSO buckets serves a KeyLocationList but might
> be large size of response (> 100 MB) and exceed the limit. The problem is
> KeyLocation field is large enough (> 1MB) so just listing some multi-part
> uploaded keys exceeds the limit. A possible solution is to reduce KeyLocation
> size by omitting pipeline fields.
> {quote}% ozone sh key ls /s3v/bucket --prefix=path/to/prefix
> RPC response exceeds maximum data length
> {quote}
>
> We checked a List<OzoneFileStatus> in our environment and it turned out that
> protobufs corresponding to some multi-part keys are larger than 1MB. AFAIK,
> the maximum IPC length is 100MB by default so it easily exceeds the limit.
> !large-protos.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
