You are right, my bad. We had a company internal java dependency that was referring to an older version of RocksDB. I've spotted it via running
mvn dependency:tree while investigating with a colleague. Thank you! On Tue, Aug 13, 2019 at 8:01 PM Yun Tang <myas...@live.com> wrote: > Hi Tobias > > First of all, I think you would not need to ADD the > flink-statebackend-rocksdb jar package into your docker image's lib folder, > as the flink-dist jar package within lib folder already include all classes > of flink-statebackend-rocksdb. > > I think the root cause is that you might assemble the rocksdbjni jar > package in your user application jar which was rocksdbjni-5.7.5.jar in > Flink-1.7. As Flink would load classes first from the user code jar [1], > however, method org.rocksdb.ColumnFamilyHandle.getDescriptor() is not > existed in rocksdbjni-5.7.5.jar but in rocksdbjni-5.17.2 (or we can say > frocksdbjni-5.17.2-artisans-1.0 in Flink-1.8). That's why you come across > this NoSuchMethodError exception. > > If no necessary, please do not assemble rocksdbjni package in your user > code jar as flink-dist already provide all needed classes. Moreover, adding > dependency of flink-statebackend-rocksdb_2.11 in your pom.xml should be > enough as it already includes the dependency of rocksdbjni. > > [1] > https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#classloader-resolve-order > > Best > Yun Tang > > ------------------------------ > *From:* Kaymak, Tobias <tobias.kay...@ricardo.ch> > *Sent:* Tuesday, August 13, 2019 21:20 > *To:* user@flink.apache.org <user@flink.apache.org> > *Subject:* Flink 1.8: Using the RocksDB state backend causes > "NoSuchMethodError" when trying to stop a pipeline > > Hi, > > I am using Apache Beam 2.14.0 with Flink 1.8.0 and I have included the > RocksDb dependency in my projects pom.xml as well as baked it into the > Dockerfile like this: > > FROM flink:1.8.0-scala_2.11 > > ADD --chown=flink:flink > http://central.maven.org/maven2/org/apache/flink/flink-statebackend-rocksdb_2.11/1.8.0/flink-statebackend-rocksdb_2.11-1.8.0.jar > /opt/flink/lib/flink-statebackend-rocksdb_2.11-1.8.0.jar > > > Everything seems to be normal up to the point when I try to stop and > cleanly shutdown my pipeline. I get the following error: > > java.lang.NoSuchMethodError: > org.rocksdb.ColumnFamilyHandle.getDescriptor()Lorg/rocksdb/ColumnFamilyDescriptor; > at > org.apache.flink.contrib.streaming.state.RocksDBOperationUtils.addColumnFamilyOptionsToCloseLater(RocksDBOperationUtils.java:160) > at > org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.dispose(RocksDBKeyedStateBackend.java:331) > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator.dispose(AbstractStreamOperator.java:362) > at > org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.dispose(DoFnOperator.java:470) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.tryDisposeAllOperators(StreamTask.java:454) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:337) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711) > at java.lang.Thread.run(Thread.java:748) > > I can cancel my pipeline and snapshotting in general works, however. Flink > 1.7.2 with Beam 2.12.0 did not have any problem, could it be that this is > caused by the switch to FRocksDb?[0] > > Best, > Tobias > > [0] > https://ci.apache.org/projects/flink/flink-docs-stable/release-notes/flink-1.8.html#rocksdb-version-bump-and-switch-to-frocksdb-flink-10471 >
