Yun Tang created FLINK-24932:
--------------------------------
Summary: Frocksdb cannot run on Apple M1
Key: FLINK-24932
URL: https://issues.apache.org/jira/browse/FLINK-24932
Project: Flink
Issue Type: Bug
Components: Runtime / State Backends
Reporter: Yun Tang
After we bump up RocksDB version to 6.20.3, we support to run RocksDB on linux
arm cluster. However, according to the feedback from Robert, Apple M1 machines
cannot run FRocksDB yet:
{code:java}
java.lang.Exception: Exception while creating StreamOperatorStateContext.
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:255)
~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:268)
~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
at
org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:109)
~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:711)
~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:687)
~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:654)
~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
at
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
~[flink-runtime-1.14.0.jar:1.14.0]
at
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927)
~[flink-runtime-1.14.0.jar:1.14.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
~[flink-runtime-1.14.0.jar:1.14.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
~[flink-runtime-1.14.0.jar:1.14.0]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_312]
Caused by: org.apache.flink.util.FlinkException: Could not restore keyed state
backend for StreamFlatMap_c21234bcbf1e8eb4c61f1927190efebd_(1/1) from any of
the 1 provided restore options.
at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:160)
~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:346)
~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:164)
~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
... 11 more
Caused by: java.io.IOException: Could not load the native RocksDB library
at
org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.ensureRocksDBIsLoaded(EmbeddedRocksDBStateBackend.java:882)
~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
at
org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:402)
~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:345)
~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:87)
~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:329)
~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)
~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:346)
~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:164)
~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
... 11 more
Caused by: java.lang.UnsatisfiedLinkError:
/private/var/folders/js/yfk_y2450q7559kygttykwk00000gn/T/rocksdb-lib-5783c058ce68d31d371327abc9b51cac/librocksdbjni-osx.jnilib:
dlopen(/private/var/folders/js/yfk_y2450q7559kygttykwk00000gn/T/rocksdb-lib-5783c058ce68d31d371327abc9b51cac/librocksdbjni-osx.jnilib,
0x0001): tried:
'/private/var/folders/js/yfk_y2450q7559kygttykwk00000gn/T/rocksdb-lib-5783c058ce68d31d371327abc9b51cac/librocksdbjni-osx.jnilib'
(mach-o file, but is an incompatible architecture (have 'x86_64', need
'arm64e')), '/usr/lib/librocksdbjni-osx.jnilib' (no such file)
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
~[?:1.8.0_312]
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1950)
~[?:1.8.0_312]
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1832)
~[?:1.8.0_312]
at java.lang.Runtime.load0(Runtime.java:811) ~[?:1.8.0_312]
at java.lang.System.load(System.java:1088) ~[?:1.8.0_312]
at
org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:79)
~[frocksdbjni-6.20.3-ververica-1.0.jar:?]
at
org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:57)
~[frocksdbjni-6.20.3-ververica-1.0.jar:?]
at
org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.ensureRocksDBIsLoaded(EmbeddedRocksDBStateBackend.java:856)
~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
at
org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:402)
~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:345)
~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:87)
~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:329)
~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)
~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:346)
~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:164)
~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
... 11 more
{code}
This issue is tracked by RocksDB community:
[rocksdb/issues/7720|https://github.com/facebook/rocksdb/issues/7720]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
