[
https://issues.apache.org/jira/browse/HDDS-12882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Andika updated HDDS-12882:
-------------------------------
Description:
We have a situation where OM crash with heap space OOM because S3 user keeps
sending a lot listObjectsV2 with max-keys=1 parameter as a way to check the
object's owner (although GetObjectAcl should be a better alternative).
We see that the KeyIterator implementation always send listStatus with maxKeys
specified by ozone.client.list.cache (default is 1000). The default
ozone.client.list.cache of 1000 is also the same as the "max-keys" specified by
S3G, so normally it will be ok. If there are 1000 files inside this directory,
the S3 client will send 1000 S3 listObjectsV2 (with max-keys=1), and OM has to
load 1000 * 1000 = 1,000,000 OmKeyInfo entries to its memory. This causes OM
heap space to be OOM.
We need to take into account this max-keys sent by S3. One possible idea is to
use the S3 max-keys if it's smaller than the configured
ozone.client.list.cache. This way, OM will only load the necessary OmKeyInfo to
memory.
was:
We have a situation where OM crash with heap space OOM because S3 user keeps
sending a lot listObjectsV2 with max-keys=1 parameter.
The code looks something like this:
{code:java}
s3_parquet_path = f"s3a://bucket/dir/*.parquet"
parquet_tab = spark.read.format("parquet").load(s3_parquet_path)
parquet_count = parquet_tab.count(){code}
We see that the KeyIterator implementation always send listStatus with maxKeys
specified by ozone.client.list.cache (default is 1000). The default
ozone.client.list.cache of 1000 is also the same as the "max-keys" specified by
S3G, so normally it will be ok. If there are 1000 files inside this directory,
the S3 client will send 1000 S3 listObjectsV2 (with max-keys=1), and OM has to
load 1000 * 1000 = 1,000,000 OmKeyInfo entries to its memory. This causes OM
heap space to be OOM.
We need to take into account this max-keys sent by S3. One possible idea is to
use the S3 max-keys if it's smaller than the configured
ozone.client.list.cache. This way, OM will only load the necessary OmKeyInfo to
memory.
> S3G should consider S3 listObjects max-keys parameter
> -----------------------------------------------------
>
> Key: HDDS-12882
> URL: https://issues.apache.org/jira/browse/HDDS-12882
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> We have a situation where OM crash with heap space OOM because S3 user keeps
> sending a lot listObjectsV2 with max-keys=1 parameter as a way to check the
> object's owner (although GetObjectAcl should be a better alternative).
> We see that the KeyIterator implementation always send listStatus with
> maxKeys specified by ozone.client.list.cache (default is 1000). The default
> ozone.client.list.cache of 1000 is also the same as the "max-keys" specified
> by S3G, so normally it will be ok. If there are 1000 files inside this
> directory, the S3 client will send 1000 S3 listObjectsV2 (with max-keys=1),
> and OM has to load 1000 * 1000 = 1,000,000 OmKeyInfo entries to its memory.
> This causes OM heap space to be OOM.
> We need to take into account this max-keys sent by S3. One possible idea is
> to use the S3 max-keys if it's smaller than the configured
> ozone.client.list.cache. This way, OM will only load the necessary OmKeyInfo
> to memory.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org
For additional commands, e-mail: issues-h...@ozone.apache.org
