GitHub user byteayan added a comment to the discussion: [Question] sync 
hugegraph data daily with rocksdb, how to handle lots of g/logs

Following up on @imbajin's suggestion to check `RocksDBOptions`, here's what it 
exposes on master today, for anyone who lands here with the same problem.

The files in `g`, `m` and `s` are three different things:

- `*.sst`: the data itself. Leave these alone.
- `000063.log` and other numbered `.log` files: the WAL. RocksDB deletes a WAL 
file on its own once the memtables it covers have been flushed. Don't delete 
them while the server is running, or you can lose recent writes.
- `LOG` and `LOG.old.<timestamp>`: RocksDB's own info log. The current `LOG` is 
rotated into a `LOG.old.*` file on every restart. These are safe to delete.

What you can set in `conf/graphs/hugegraph.properties`:

- `rocksdb.max_total_wal_size`: caps the total WAL size in bytes. Once it's 
exceeded, RocksDB forces a flush of the column families holding the oldest WAL 
so that file can be deleted. The default is `0`, which leaves the limit to 
RocksDB (it derives one from the memtable sizes). With a sync every 10 minutes, 
something like `268435456` (256 MB) keeps the WAL small.
- `rocksdb.log_level`: defaults to `INFO`. `WARN` or `ERROR` makes the `LOG` 
files much smaller.
- `rocksdb.wal_path`: puts the WAL on a different disk from the data, if disk 
space is the main concern.

What isn't exposed: RocksDB's `keep_log_file_num` and `max_log_file_size`, 
which control how many `LOG.old.*` files are kept and how large each one gets. 
For those, a cron job is still the simplest option, for example:

```sh
find /path/to/rocksdb-data -name 'LOG.old.*' -mtime +7 -delete
```

Adding those two to `RocksDBOptions` would be a small change, and I'm happy to 
open a PR for it if that's useful.

GitHub link: 
https://github.com/apache/hugegraph/discussions/2267#discussioncomment-18990272

----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Reply via email to