[jira] [Commented] (FLINK-6647) Fail-fast on invalid RocksDBStateBackend configuration
[ https://issues.apache.org/jira/browse/FLINK-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17336748#comment-17336748 ] Flink Jira Bot commented on FLINK-6647: --- This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Fail-fast on invalid RocksDBStateBackend configuration > -- > > Key: FLINK-6647 > URL: https://issues.apache.org/jira/browse/FLINK-6647 > Project: Flink > Issue Type: Bug > Components: Runtime / State Backends >Reporter: Andrey >Priority: Major > Labels: stale-major > > Currently: > * setup "state.backend.rocksdb.checkpointdir=hdfs:///some/base/path/hdfs" > * setup backend: state.backend: > "org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory" > * rocksdb doesn't support hdfs backend so in logs: > {code} > 2017-05-19 15:42:33,737 ERROR > org.apache.flink.contrib.streaming.state.RocksDBStateBackend - Local DB files > directory '/some/base/path/hdfs' does not exist and cannot be created. > {code} > * however job continue execution and IOManager temp directory will be picked > up for rocksdb files. > There are several issues with such approach: > * after "ERROR" message printed and before developer fixes configuration, > /tmp directory/partition might run out of disk space. > * if hdfs base path is the same as local path, then no errors in logs and > rocksdb files will be written into an incorrect location. For example: > "hdfs:///home/flink/data" will cause an issue. > Expected: > * validate URI and throw IllegalArgumentException like already implemented in > "RocksDBStateBackend.setDbStoragePaths" method. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-6647) Fail-fast on invalid RocksDBStateBackend configuration
[ https://issues.apache.org/jira/browse/FLINK-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17328767#comment-17328767 ] Flink Jira Bot commented on FLINK-6647: --- This major issue is unassigned and itself and all of its Sub-Tasks have not been updated for 30 days. So, it has been labeled "stale-major". If this ticket is indeed "major", please either assign yourself or give an update. Afterwards, please remove the label. In 7 days the issue will be deprioritized. > Fail-fast on invalid RocksDBStateBackend configuration > -- > > Key: FLINK-6647 > URL: https://issues.apache.org/jira/browse/FLINK-6647 > Project: Flink > Issue Type: Bug > Components: Runtime / State Backends >Reporter: Andrey >Priority: Major > Labels: stale-major > > Currently: > * setup "state.backend.rocksdb.checkpointdir=hdfs:///some/base/path/hdfs" > * setup backend: state.backend: > "org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory" > * rocksdb doesn't support hdfs backend so in logs: > {code} > 2017-05-19 15:42:33,737 ERROR > org.apache.flink.contrib.streaming.state.RocksDBStateBackend - Local DB files > directory '/some/base/path/hdfs' does not exist and cannot be created. > {code} > * however job continue execution and IOManager temp directory will be picked > up for rocksdb files. > There are several issues with such approach: > * after "ERROR" message printed and before developer fixes configuration, > /tmp directory/partition might run out of disk space. > * if hdfs base path is the same as local path, then no errors in logs and > rocksdb files will be written into an incorrect location. For example: > "hdfs:///home/flink/data" will cause an issue. > Expected: > * validate URI and throw IllegalArgumentException like already implemented in > "RocksDBStateBackend.setDbStoragePaths" method. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-6647) Fail-fast on invalid RocksDBStateBackend configuration
[ https://issues.apache.org/jira/browse/FLINK-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16017687#comment-16017687 ] Helder Pereira commented on FLINK-6647: --- The {{File.pathSeparator}} seems a bit dangerous here, as it will strip out the schema and consider it an alternative path: https://github.com/apache/flink/blob/master/flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackendFactory.java#L64 And there seems to be a mismatch with what is expected here, which is prepared to get paths with schema: https://github.com/apache/flink/blob/master/flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackend.java#L367 > Fail-fast on invalid RocksDBStateBackend configuration > -- > > Key: FLINK-6647 > URL: https://issues.apache.org/jira/browse/FLINK-6647 > Project: Flink > Issue Type: Bug >Reporter: Andrey > > Currently: > * setup "state.backend.rocksdb.checkpointdir=hdfs:///some/base/path/hdfs" > * setup backend: state.backend: > "org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory" > * rocksdb doesn't support hdfs backend so in logs: > {code} > 2017-05-19 15:42:33,737 ERROR > org.apache.flink.contrib.streaming.state.RocksDBStateBackend - Local DB files > directory '/some/base/path/hdfs' does not exist and cannot be created. > {code} > * however job continue execution and IOManager temp directory will be picked > up for rocksdb files. > There are several issues with such approach: > * after "ERROR" message printed and before developer fixes configuration, > /tmp directory/partition might run out of disk space. > * if hdfs base path is the same as local path, then no errors in logs and > rocksdb files will be written into an incorrect location. For example: > "hdfs:///home/flink/data" will cause an issue. > Expected: > * validate URI and throw IllegalArgumentException like already implemented in > "RocksDBStateBackend.setDbStoragePaths" method. -- This message was sent by Atlassian JIRA (v6.3.15#6346)