PanTheMan opened a new pull request #1275: SAMZA-2456: Upgrade Rocksdb to 6.5.3 URL: https://github.com/apache/samza/pull/1275 Symptom: Currently Samza is using RocksDB 5.7.3 version. This would cause a segment fault if the dependency order is not properly provisioned. This was reported in https://github.com/tensorflow/tensorflow/issues/22307 Cause: For example, if RockDB library is loaded before Tensorflow library, segment fault is triggered. In contrast, if Tensorflow library is loaded first, then there is no run-time error. Please see the following experiments for more details. public class RocksDBTest{ public static void main(String[] args) throws InterruptedException { //System.load("/home/yangzhan/libs/librocksdbjni7259590262328036607.so"); // RocksDB v=5.0.1 => Failed // System.load("/home/yangzhan/libs/librocksdbjni575105474288895339.so"); // RocksDB v=5.7.3 => Failed System.load("/home/yangzhan/libs/librocksdbjni6484631531885492614.so"); // RockdsDB v=5.15.10 => Passed System.load("/home/yangzhan/libs/libtensorflow_framework.so"); } } Changes: I upgraded RocksDB to the latest (6.5.3) to fix this issue. I tested (see below) to ensure that this upgrade doesn't change any current RocksDB use. A list of RocksDB changes since 5.7.3 can be found here https://github.com/facebook/rocksdb/releases. Tests: Ran the script above to verify this fixes the issue. Also tested to verify that this upgrade doesn't require any code changes to our RocksDbKeyValueStore by running: ./gradlew clean test ./gradlew build test bin/integration-tests.sh for standalone and yarn mint build && mint pcl in samza-li Ran Brett's store workload job in samza release tests
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
