Hi Tobias

First of all, I think you would not need to ADD the flink-statebackend-rocksdb 
jar package into your docker image's lib folder, as the flink-dist jar package 
within lib folder already include all classes of flink-statebackend-rocksdb.

I think the root cause is that you might assemble the rocksdbjni jar package in 
your user application jar which was rocksdbjni-5.7.5.jar in Flink-1.7. As Flink 
would load classes first from the user code jar [1], however, method 
org.rocksdb.ColumnFamilyHandle.getDescriptor() is not existed in 
rocksdbjni-5.7.5.jar but in rocksdbjni-5.17.2 (or we can say 
frocksdbjni-5.17.2-artisans-1.0 in Flink-1.8). That's why you come across this 
NoSuchMethodError exception.

If no necessary, please do not assemble rocksdbjni package in your user code 
jar as flink-dist already provide all needed classes. Moreover, adding 
dependency of flink-statebackend-rocksdb_2.11 in your pom.xml should be enough 
as it already includes the dependency of rocksdbjni.

[1] 
https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#classloader-resolve-order

Best
Yun Tang

________________________________
From: Kaymak, Tobias <[email protected]>
Sent: Tuesday, August 13, 2019 21:20
To: [email protected] <[email protected]>
Subject: Flink 1.8: Using the RocksDB state backend causes "NoSuchMethodError" 
when trying to stop a pipeline

Hi,

I am using Apache Beam 2.14.0 with Flink 1.8.0 and I have included the RocksDb 
dependency in my projects pom.xml as well as baked it into the Dockerfile like 
this:

FROM flink:1.8.0-scala_2.11

ADD --chown=flink:flink 
http://central.maven.org/maven2/org/apache/flink/flink-statebackend-rocksdb_2.11/1.8.0/flink-statebackend-rocksdb_2.11-1.8.0.jar
 /opt/flink/lib/flink-statebackend-rocksdb_2.11-1.8.0.jar


Everything seems to be normal up to the point when I try to stop and cleanly 
shutdown my pipeline. I get the following error:

java.lang.NoSuchMethodError: 
org.rocksdb.ColumnFamilyHandle.getDescriptor()Lorg/rocksdb/ColumnFamilyDescriptor;
at 
org.apache.flink.contrib.streaming.state.RocksDBOperationUtils.addColumnFamilyOptionsToCloseLater(RocksDBOperationUtils.java:160)
at 
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.dispose(RocksDBKeyedStateBackend.java:331)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator.dispose(AbstractStreamOperator.java:362)
at 
org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.dispose(DoFnOperator.java:470)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.tryDisposeAllOperators(StreamTask.java:454)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:337)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
at java.lang.Thread.run(Thread.java:748)

I can cancel my pipeline and snapshotting in general works, however. Flink 
1.7.2 with Beam 2.12.0 did not have any problem, could it be that this is 
caused by the switch to FRocksDb?[0]

Best,
Tobias

[0] 
https://ci.apache.org/projects/flink/flink-docs-stable/release-notes/flink-1.8.html#rocksdb-version-bump-and-switch-to-frocksdb-flink-10471

Reply via email to