devmadhuu opened a new pull request, #4459:
URL: https://github.com/apache/ozone/pull/4459
Observations about below code in
`org.apache.hadoop.hdds.utils.db.RDBStore#getUpdatesSince(long, long)`
```
dbUpdatesWrapper.addWriteBatch(result.writeBatch().data(),
result.sequenceNumber());
```
Above code is in while loop for all log files getting iterated and each log
file batch allocates a byte[] array which results in accumulating the data in
dbUpdatesWrapper in the form of dataList. This will increase JVM heap due to
dataList growing with each allocated byte[] array getting added in dataList for
dbUpdatesWrapper object and may further fail to allocate any byte[] array in
the log iterator loop on calling "result.writeBatch().data()" code. If Recon
has limited heap memory, this may fail frequently and may even fall into worse
situation where a first call to "result.writeBatch().data()" may fail to
allocate byte[] array and throw "org.rocksdb.RocksDBException: Requested array
size exceeds VM limit".
So to reduce the chance of this byte[] array allocation failure, we need to
ensure following 3 points:
1. Recon max memory -Xmx configuration should allocate sufficient heap.
2. Need to fix and avoid increase in recon heap due to increase in size of
dataList wrapped in dbUpdatesWrapper object more than configured limit of
default 1 GB. If it crosses more than 1 GB as default, break the loop and wait
for delta updates since last sequence number in next periodic task run of delta
updates.
3. Throw RocksDBException to client, so. that Recon client catches the
exception and set fullSnapshot as True
https://issues.apache.org/jira/browse/HDDS-8195
Patch was tested locally on docker setup by creating very frequent writes
and generated keys using freon tool and waited for recon to get OM DB sync
updates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
