sadanand48 opened a new pull request, #6954: URL: https://github.com/apache/ozone/pull/6954
## What changes were proposed in this pull request? Currently the max_open_files rocksdb config is -1 by default and not set anywhere via Ozone code. However in situations where the number of SST Files in the rocksdb instance becomes greater than the ulimit (open file limit of the OS), Rocksdb will not respect the OS config and try to open all files at once thereby leading to "Too many open files" error. Introducing a config in Ozone to set this is helpful to avoid/recover from such crashes. Default is still kept unchanged as -1 as [RocksDB recommends](https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB) -1 for best performance > If you are certain that your ulimit will always be bigger than number of files in the database, we recommend setting max_open_files to -1, which means infinity This patch adds an option to reconfigure the value if we hit the issue thereby preserving existing behaviour and adding a recovery mechanism in case of this issue. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-11191 ## How was this patch tested? Verified that the config takes effect via docker In docker-config ```bash OZONE-SITE.XML_ozone.om.db.max.open.files=2048 ``` Checking the rocksdb LOG ```bash bash-4.2$ cat /data/metadata/om.db/LOG | grep "Options.max_open_file" 2024/07/16-11:56:59.830522 139896182388480 Options.max_open_files: 2048 ``` Without any config change: ```bash bash-4.2$ cat /data/metadata/om.db/LOG | grep "Options.max_open_file" 2024/07/16-11:51:54.934483 140099612952320 Options.max_open_files: -1 bash-4.2$ exit ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
