[
https://issues.apache.org/jira/browse/HDDS-15228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Andika updated HDDS-15228:
-------------------------------
Description:
We encountered the following NegativeArraySizeException issue which triggered
unnecessary failovers which increases the latency.
{code:java}
com.google.protobuf.ServiceException: java.lang.NegativeArraySizeException:
-1273201896, while invoking $Proxy34.send over
nodeId=scm4,nodeAddress=<redacted> after 256 failover attempts. Trying to
failover after sleeping for 2000ms.
{code}
Currently, KeyDeletingService would send deletions based on the number of keys
(ozone.key.deleting.limit.per.task). However, some keys can have a large number
of blocks especially keys with EC where one block is assigned per shard (e.g.
EC 6+3 will have 9 different BlockID per KeyLocationInfo compare to RATIS/THREE
only have 1 BlockID).
This can cause issues where a large SCM deleteKeyBlocks response causes Integer
overflow which triggers java.lang.NegativeArraySizeException. Even when we set
the ipc.maximum.data.length (512MB) and ipc.maximum.response.length (640MB) to
higher value, it seems to still trigger the issue.
To prevent this, we can batch the deletions based on the number of blocks.
However, we need ensure that at least a single key is sent to deletion (even if
breaches the number of blocks) so that the OM deletion still proceeds.
HDDS-13517 already divided the key based on the Ratis limit.
was:
We encountered the following NegativeArraySizeException issue which triggered
unnecessary failovers which increases the latency.
{code:java}
com.google.protobuf.ServiceException: java.lang.NegativeArraySizeException:
-1273201896, while invoking $Proxy34.send over
nodeId=scm4,nodeAddress=<redacted> after 256 failover attempts. Trying to
failover after sleeping for 2000ms.
{code}
Currently, KeyDeletingService would send deletions based on the number of keys
(ozone.key.deleting.limit.per.task). However, some keys can have a large number
of blocks especially keys with EC where one block is assigned per shard (e.g.
EC 6+3 will have 9 different BlockID per KeyLocationInfo compare to RATIS/THREE
only have 1 BlockID).
This can cause issues where a large SCM deleteKeyBlocks response causes Integer
overflow which triggers java.lang.NegativeArraySizeException. Even when we set
the ipc.maximum.data.length (512MB) and ipc.maximum.response.length (640MB) to
higher value, it seems to still trigger the issue.
To prevent this, we can batch the deletions based on the number of blocks.
However, we need ensure that at least a single key is sent to deletion (even if
breaches the number of blocks) so that the OM deletion still proceeds.
> KeyDeletingService limit batch deletions based on number of blocks
> ------------------------------------------------------------------
>
> Key: HDDS-15228
> URL: https://issues.apache.org/jira/browse/HDDS-15228
> Project: Apache Ozone
> Issue Type: Improvement
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> We encountered the following NegativeArraySizeException issue which
> triggered unnecessary failovers which increases the latency.
> {code:java}
> com.google.protobuf.ServiceException: java.lang.NegativeArraySizeException:
> -1273201896, while invoking $Proxy34.send over
> nodeId=scm4,nodeAddress=<redacted> after 256 failover attempts. Trying to
> failover after sleeping for 2000ms.
> {code}
> Currently, KeyDeletingService would send deletions based on the number of
> keys (ozone.key.deleting.limit.per.task). However, some keys can have a large
> number of blocks especially keys with EC where one block is assigned per
> shard (e.g. EC 6+3 will have 9 different BlockID per KeyLocationInfo compare
> to RATIS/THREE only have 1 BlockID).
> This can cause issues where a large SCM deleteKeyBlocks response causes
> Integer overflow which triggers java.lang.NegativeArraySizeException. Even
> when we set the ipc.maximum.data.length (512MB) and
> ipc.maximum.response.length (640MB) to higher value, it seems to still
> trigger the issue.
> To prevent this, we can batch the deletions based on the number of blocks.
> However, we need ensure that at least a single key is sent to deletion (even
> if breaches the number of blocks) so that the OM deletion still proceeds.
> HDDS-13517 already divided the key based on the Ratis limit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
