[
https://issues.apache.org/jira/browse/FLINK-15068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-15068:
-----------------------------------
Labels: pull-request-available (was: )
> Disable RocksDB's local LOG by default
> --------------------------------------
>
> Key: FLINK-15068
> URL: https://issues.apache.org/jira/browse/FLINK-15068
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
> Affects Versions: 1.7.2, 1.8.2, 1.9.1
> Reporter: Nico Kruber
> Assignee: Nico Kruber
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
>
> With Flink's default settings for RocksDB, it will write a log file (not the
> WAL, but pure logging statements) into the data folder. Besides periodic
> statistics, it will log compaction attempts, new memtable creations, flushes,
> etc.
> A few things to note about this practice:
> # *this LOG file is growing over time with no limit (!)*
> # the default logging level is INFO
> # the statistics in there may help looking into performance and/or disk
> space problems (but maybe you should be looking and monitoring metrics
> instead)
> # this file is not useful for debugging errors since it will be deleted
> along with the local dir when the TM goes down
> With a custom \{{OptionsFactory}}, the user can change the behaviour like the
> following:
> {code:java}
> @Override
> public DBOptions createDBOptions(DBOptions currentOptions) {
> currentOptions = super.createDBOptions(currentOptions);
> currentOptions.setKeepLogFileNum(10);
> currentOptions.setInfoLogLevel(InfoLogLevel.WARN_LEVEL);
> currentOptions.setStatsDumpPeriodSec(0);
> currentOptions.setMaxLogFileSize(1024 * 1024); // 1 MB each
> return currentOptions;
> }{code}
> However, the rotating logger does currently not work (it will not delete old
> log files - see [https://github.com/dataArtisans/frocksdb/pull/12]). Also,
> the user should not have to write his own {{OptionsFactory}} to get a
> sensible default.
> To prevent this file from filling up the disk, I propose to change Flink's
> default RocksDB settings so that the LOG file is effectively disabled
> (nothing is written to it by default).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
