[
https://issues.apache.org/jira/browse/HDDS-8195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17704080#comment-17704080
]
Devesh Kumar Singh commented on HDDS-8195:
------------------------------------------
dbUpdatesWrapper.addWriteBatch(result.writeBatch().data(),
result.sequenceNumber());
Above code is in while loop for all log files getting iterated and each log
file batch allocates a byte[] array which results in accumulating the data in
dbUpdatesWrapper in the form of dataList. This will increase JVM heap due to
dataList growing with each allocated byte[] array getting added in dataList for
dbUpdatesWrapper object and may further fail to allocate any byte[] array in
the log iterator loop on calling "result.writeBatch().data()" code. If Recon
has limited heap memory, this may fail frequently and may even fall into worse
situation where a first call to "result.writeBatch().data()" may fail to
allocate byte[] array and throw "{*}org.rocksdb.RocksDBException: Requested
array size exceeds VM limit{*}".
So to reduce the chance of this byte[] array allocation failure, we need to
ensure following 3 points:
# Recon max memory -Xmx configuration should allocate 80G as recommended in a
cluster of
# Need to fix and avoid increase in recon heap due to increase in size of
dataList wrapped in dbUpdatesWrapper object more than configured limit of
default 1 GB. If it crosses more than 1 GB, break the loop and wait for delta
updates since last sequence number in next periodic task run of delta updates.
# Throw *RocksDBException* to client, so. that Recon client catches the
exception and set *fullSnapshot* as *True*
> RDBStore.getUpdatesSince() throws RocksDBException: Requested array size
> exceeds VM limit
> -----------------------------------------------------------------------------------------
>
> Key: HDDS-8195
> URL: https://issues.apache.org/jira/browse/HDDS-8195
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 1.4.0
> Reporter: Wei-Chiu Chuang
> Assignee: Devesh Kumar Singh
> Priority: Major
>
> It appears that this operations attempted request an array larger than
> Integer.MAX_INT.
> {noformat}
> 2023-03-15 22:27:13,988 ERROR org.apache.hadoop.hdds.utils.db.RDBStore:
> Unable to get delta updates since sequenceNumber 1329458. This exception will
> not be thrown to
> the client
> org.rocksdb.RocksDBException: Requested array size exceeds VM limit
> at org.rocksdb.WriteBatch.data(Native Method)
> at org.rocksdb.WriteBatch.data(WriteBatch.java:75)
> at
> org.apache.hadoop.hdds.utils.db.RDBStore.getUpdatesSince(RDBStore.java:405)
> at
> org.apache.hadoop.ozone.om.OzoneManager.getDBUpdates(OzoneManager.java:3975)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.getOMDBUpdates(OzoneManagerRequestHandler.java:354)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleReadRequest(OzoneManagerRequestHandler.java:233)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:223)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:177)
> at
> org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:147)
> at
> org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:989)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:917)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2894)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
