zlzhang0122 created FLINK-22447:
-----------------------------------
Summary: when job failed and restore from rocksdb, it produced a
exception "Could not load the native RocksDB library"
Key: FLINK-22447
URL: https://issues.apache.org/jira/browse/FLINK-22447
Project: Flink
Issue Type: Bug
Components: Runtime / State Backends
Affects Versions: 1.11.1
Reporter: zlzhang0122
Have someone The full stack trace is:
{code:java}
// code placeholder
java.lang.Exception: Exception while creating StreamOperatorStateContext. at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:204)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:274)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:291)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:473)
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:469)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:522)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) at
java.lang.Thread.run(Thread.java:748) Caused by:
org.apache.flink.util.FlinkException: Could not restore keyed state backend for
AdMetaJoinOperator_b6fdad9df33c017e430813a04b752f9a_(66/512) from any of the 1
provided restore options. at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:317)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:144)
... 9 more Caused by: java.io.IOException: Could not load the native RocksDB
library at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:1007)
at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:538)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:301)
at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)
at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)
... 11 more Caused by: java.lang.UnsatisfiedLinkError:
/home/work/hdd1/yarn/azmbcommonprc-hadoop/nodemanager/20800/usercache/s_global_ad/appcache/application_1617086162049_443089/rocksdb-lib-5280b3bf65c39fcc2c0c717a7bd6706f/librocksdbjni-linux64.so:
/home/work/hdd1/yarn/azmbcommonprc-hadoop/nodemanager/20800/usercache/s_global_ad/appcache/application_1617086162049_443089/rocksdb-lib-5280b3bf65c39fcc2c0c717a7bd6706f/librocksdbjni-linux64.so:
ELF load command past end of file at
java.lang.ClassLoader$NativeLibrary.load(Native Method) at
java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) at
java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) at
java.lang.Runtime.load0(Runtime.java:809) at
java.lang.System.load(System.java:1086) at
org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)
at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56) at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:982)
... 15 more{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
