Zakelly Lan created FLINK-37439:
-----------------------------------
Summary: Missing file creation with overwrite semantic in ForSt
wrapped file system
Key: FLINK-37439
URL: https://issues.apache.org/jira/browse/FLINK-37439
Project: Flink
Issue Type: Bug
Components: Runtime / State Backends
Affects Versions: 2.0.0
Reporter: Zakelly Lan
Assignee: Zakelly Lan
Fix For: 2.0.0
ForSt may create LOG files twice with overwrite, however, the ForSt wrapped
file system does not support overwriting files. Exception thrown in this case:
{code:java}
Caused by: java.lang.IllegalStateException: Try to add a file that is already
in mappingTable, but with inconsistent entry. Key:
oss://xxxxxxxxxxxxxxxxxx/ba3bcc9061a3858257e3119f0a6493ab/shared/op_AsyncKeyedProcessOperator_7261c9d38047b376863833deae26476f__4_8__attempt_0/db/LOG,
source:
FileBackedSource{filePath=/tmp/ba3bcc9061a3858257e3119f0a6493ab/op_AsyncKeyedProcessOperator_7261c9d38047b376863833deae26476f__4_8__attempt_0/db/1b72a4ee-9d4f-4d37-b1d4-781774c4da78},
fileOwnership: PRIVATE_OWNED_BY_DB. Entry in table:
MappingEntry{source=FileBackedSource{filePath=/tmp/ba3bcc9061a3858257e3119f0a6493ab/op_AsyncKeyedProcessOperator_7261c9d38047b376863833deae26476f__4_8__attempt_0/db/f60883d6-355e-491b-82d9-1875e5cb151e},
fileOwnership=PRIVATE_OWNED_BY_DB, isDirectory= false}
at
org.apache.flink.util.Preconditions.checkState(Preconditions.java:193)
~[flink-dist-2.0-SNAPSHOT.jar:2.0-SNAPSHOT]
at
org.apache.flink.state.forst.fs.filemapping.FileMappingManager.getExistingMappingEntry(FileMappingManager.java:107)
~[flink-dist-2.0-SNAPSHOT.jar:2.0-SNAPSHOT]
at
org.apache.flink.state.forst.fs.filemapping.FileMappingManager.addFileToMappingTable(FileMappingManager.java:97)
~[flink-dist-2.0-SNAPSHOT.jar:2.0-SNAPSHOT]
at
org.apache.flink.state.forst.fs.filemapping.FileMappingManager.createNewFile(FileMappingManager.java:70)
~[flink-dist-2.0-SNAPSHOT.jar:2.0-SNAPSHOT]
at
org.apache.flink.state.forst.fs.ForStFlinkFileSystem.create(ForStFlinkFileSystem.java:189)
~[flink-dist-2.0-SNAPSHOT.jar:2.0-SNAPSHOT]
at
org.apache.flink.state.forst.fs.ForStFlinkFileSystem.create(ForStFlinkFileSystem.java:182)
~[flink-dist-2.0-SNAPSHOT.jar:2.0-SNAPSHOT]
at
org.apache.flink.state.forst.fs.StringifiedForStFileSystem.create(StringifiedForStFileSystem.java:74)
~[flink-dist-2.0-SNAPSHOT.jar:2.0-SNAPSHOT]
at org.forstdb.RocksDB.write0(Native Method)
~[flink-dist-2.0-SNAPSHOT.jar:2.0-SNAPSHOT]
at org.forstdb.RocksDB.write(RocksDB.java:1843)
~[flink-dist-2.0-SNAPSHOT.jar:2.0-SNAPSHOT]
at
org.apache.flink.state.forst.ForStDBWriteBatchWrapper.flush(ForStDBWriteBatchWrapper.java:136)
~[flink-dist-2.0-SNAPSHOT.jar:2.0-SNAPSHOT]
at
org.apache.flink.state.forst.ForStWriteBatchOperation.lambda$process$0(ForStWriteBatchOperation.java:72)
~[flink-dist-2.0-SNAPSHOT.jar:2.0-SNAPSHOT]
... 29 more
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
