I submit Flink SQL jobs on native Kubernetes operator in Application Mode (Flink 2.2.0, Kubernetes operator 1.4.0)
I regularly run into these issues (Stack trace below) and I cannot figure out how to solve them apart from using short and cryptic app names. As you can see I already try to keep the base dir as short as possible. I already tried not setting the state.backend.rocksdb.log.dir & state.backend.rocksdb.localdir at all (and letting the system choose), choosing something invalid (e.g. /dev/null) like the configuration documentation suggested, reducing rocksdb log level (unfortunately unable to completely turn it off) and investigated. I found FLINK-31743 <https://issues.apache.org/jira/browse/FLINK-31743> that seems to address the issue but since I am on a later version, only does so incompletely. Is there a way of reliably fixing this rather than trying to come up with shorter and shorter app names? [...] java.io.IOException: Error while opening RocksDB instance. at org.apache.flink.state.rocksdb.RocksDBOperationUtils.openDB( RocksDBOperationUtils.java:101) at org.apache.flink.state.rocksdb.restore.RestoredDBInstance .restoreTempDBInstanceFromLocalState(RestoredDBInstance.java:121) at org.apache.flink.state.rocksdb.restore.RocksDBIncrementalRestoreOperation .copyToBaseDBUsingTempDBs(RocksDBIncrementalRestoreOperation.java:788) at org.apache.flink.state.rocksdb.restore.RocksDBIncrementalRestoreOperation .mergeStateHandlesWithCopyFromTemporaryInstance( RocksDBIncrementalRestoreOperation.java:628) at org.apache.flink.state.rocksdb.restore.RocksDBIncrementalRestoreOperation .restoreFromMultipleStateHandles(RocksDBIncrementalRestoreOperation.java:446 ) at org.apache.flink.state.rocksdb.restore.RocksDBIncrementalRestoreOperation .restoreFromLocalState(RocksDBIncrementalRestoreOperation.java:326) at org.apache.flink.state.rocksdb.restore.RocksDBIncrementalRestoreOperation .lambda$restore$1(RocksDBIncrementalRestoreOperation.java:253) at org.apache.flink.state.rocksdb.restore.RocksDBIncrementalRestoreOperation .runAndReportDuration(RocksDBIncrementalRestoreOperation.java:893) at org.apache.flink.state.rocksdb.restore.RocksDBIncrementalRestoreOperation .restore(RocksDBIncrementalRestoreOperation.java:252) at org.apache.flink.state.rocksdb.RocksDBKeyedStateBackendBuilder.build( RocksDBKeyedStateBackendBuilder.java:390) ... 19 more Caused by: org.rocksdb.RocksDBException: While open a file for appending: /tmp/rdb/tmp_tm_hydra-sql-adr-assoc-device-and-login-features-taskmanager-1- 10 _tmp_job_41471278f6601d1a7ab05da6958d83f7_op_KeyedProcessOperator_d4d5e8c74c3d05d8a9a53a9c312a6161__1_5__uuid_aadf2786-a3dd- 4fa9-acaa-59d560e05ce3_b5ea62d0-713f-46c4-bd4e-a4526f117f33_LOG: File name too long at org.rocksdb.RocksDB.open(Native Method) at org.rocksdb.RocksDB.open(RocksDB.java:315) at org.apache.flink.state.rocksdb.RocksDBOperationUtils.openDB( RocksDBOperationUtils.java:89)
