[
https://issues.apache.org/jira/browse/FLINK-15068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao updated FLINK-15068:
-----------------------------
Release Note: Logging in RocksDB (e.g., logging related to flush,
compaction, memtable creation, etc.) has been disabled by default to prevent
disk space from being filled up unexpectedly. Users that need to enable logging
should implement their own RocksDBOptionsFactory that creates DBOptions
instances with InfoLogLevel set to INFO_LEVEL. (was: After FLINK-15068, the
RocksDB local log (logging for flush/compaction/memtable-creation etc. events)
is disabled by default, to prevent the disk space from being filled up
unexpectedly. If there's a need to enable the log, please define your own
OptionsFactory and create a DBOptions with InfoLogLevel set to INFO_LEVEL)
> Disable RocksDB's local LOG by default
> --------------------------------------
>
> Key: FLINK-15068
> URL: https://issues.apache.org/jira/browse/FLINK-15068
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
> Affects Versions: 1.7.2, 1.8.2, 1.9.1
> Reporter: Nico Kruber
> Assignee: Nico Kruber
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> With Flink's default settings for RocksDB, it will write a log file (not the
> WAL, but pure logging statements) into the data folder. Besides periodic
> statistics, it will log compaction attempts, new memtable creations, flushes,
> etc.
> A few things to note about this practice:
> # *this LOG file is growing over time with no limit (!)*
> # the default logging level is INFO
> # the statistics in there may help looking into performance and/or disk
> space problems (but maybe you should be looking and monitoring metrics
> instead)
> # this file is not useful for debugging errors since it will be deleted
> along with the local dir when the TM goes down
> With a custom \{{OptionsFactory}}, the user can change the behaviour like the
> following:
> {code:java}
> @Override
> public DBOptions createDBOptions(DBOptions currentOptions) {
> currentOptions = super.createDBOptions(currentOptions);
> currentOptions.setKeepLogFileNum(10);
> currentOptions.setInfoLogLevel(InfoLogLevel.WARN_LEVEL);
> currentOptions.setStatsDumpPeriodSec(0);
> currentOptions.setMaxLogFileSize(1024 * 1024); // 1 MB each
> return currentOptions;
> }{code}
> However, the rotating logger does currently not work (it will not delete old
> log files - see [https://github.com/dataArtisans/frocksdb/pull/12]). Also,
> the user should not have to write his own {{OptionsFactory}} to get a
> sensible default.
> To prevent this file from filling up the disk, I propose to change Flink's
> default RocksDB settings so that the LOG file is effectively disabled
> (nothing is written to it by default).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
