empiredan opened a new pull request, #1009: URL: https://github.com/apache/incubator-pegasus/pull/1009
# 1. Motivation This PR is to solve the problem of https://github.com/apache/incubator-pegasus/issues/994. The replica server can run for a very long without any fault. Each instance of rocksdb will generate a lot of info logs that occupies considerable disk spaces. Thus each instance of rocksdb should clear out-dated logs themselves periodically. # 2. Implementation We can use 3 options provided by `rocksdb::DBOptions` to restrict the size of info logs: - `max_log_file_size` - `log_file_time_to_roll` - `keep_log_file_num` ## 2.1 `max_log_file_size` This option specifies the maximal size of the info log file. Once the log file is larger than this option, a new info log file will be created. If this option is set to 0, all logs will be written to one log file."); ## 2.2 `log_file_time_to_roll` This options specifies time for the info log file to roll (in seconds). If this option is specified with non-zero value, log file will be rolled if it has been active longer than this option. Otherwise, if this options is set to 0, log file will never be rolled by life time. ## 2.3 `keep_log_file_num` This options specifies the maximal numbers of info log files to be kept. Once the number of info logs goes beyond this option, stale log files will be cleaned. ## 2.4 default values In production environment, it has been observed that an instance of rocksdb about 20GB in total size which has run beyond 199 days, generated a big log file sized 96MB, with 492KB for each day. Accordingly, default value for `rocksdb_log_file_time_to_roll` can be set to one day, and that for `rocksdb_keep_log_file_num` can be set to 32, which means log files for recent one month will be reserved. On the other hand, the max size of a log file is also be restricted to 8MB. In practice, the size of logs over a day tends to be less than 1MB; however, once errors are reported very frequently, the log file will grow larger and go far beyond several hundreds of KB. # 3. Configurations The added configurations are listed below, which have been set to the default values: ```diff + rocksdb_max_log_file_size = 8388608 + rocksdb_log_file_time_to_roll = 86400 + rocksdb_keep_log_file_num = 32 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
