GitHub user byteayan added a comment to the discussion: [Question] sync hugegraph data daily with rocksdb, how to handle lots of g/logs
Following up on @imbajin's suggestion to check `RocksDBOptions`, here's what it exposes on master today, for anyone who lands here with the same problem. The files in `g`, `m` and `s` are three different things: - `*.sst`: the data itself. Leave these alone. - `000063.log` and other numbered `.log` files: the WAL. RocksDB deletes a WAL file on its own once the memtables it covers have been flushed. Don't delete them while the server is running, or you can lose recent writes. - `LOG` and `LOG.old.<timestamp>`: RocksDB's own info log. The current `LOG` is rotated into a `LOG.old.*` file on every restart. These are safe to delete. What you can set in `conf/graphs/hugegraph.properties`: - `rocksdb.max_total_wal_size`: caps the total WAL size in bytes. Once it's exceeded, RocksDB forces a flush of the column families holding the oldest WAL so that file can be deleted. The default is `0`, which leaves the limit to RocksDB (it derives one from the memtable sizes). With a sync every 10 minutes, something like `268435456` (256 MB) keeps the WAL small. - `rocksdb.log_level`: defaults to `INFO`. `WARN` or `ERROR` makes the `LOG` files much smaller. - `rocksdb.wal_path`: puts the WAL on a different disk from the data, if disk space is the main concern. What isn't exposed: RocksDB's `keep_log_file_num` and `max_log_file_size`, which control how many `LOG.old.*` files are kept and how large each one gets. For those, a cron job is still the simplest option, for example: ```sh find /path/to/rocksdb-data -name 'LOG.old.*' -mtime +7 -delete ``` Adding those two to `RocksDBOptions` would be a small change, and I'm happy to open a PR for it if that's useful. GitHub link: https://github.com/apache/hugegraph/discussions/2267#discussioncomment-18990272 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
