[
https://issues.apache.org/jira/browse/NIFI-9810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Doran updated NIFI-9810:
------------------------------
Description:
For arm64 / Apple M1 compatibility, RocksDB needs to be bumped to >= v6.29.4.1
(see below for runtime errors on an Apple Silicon M1 Mac with the current
version).
RocksDB is used only by the RocksDBFlowFileRepository implementation. After
wider discussion with the community and who could help test and verify this
change, we came to the conclusion it makes the most sense to focus maintenance
efforts on the default FlowFileRepository implementation and deprecate the
RocksDB variant.
For now, we will bump the version of RocksDB for arm64 compatibility on macOS,
but this ticket is also to extract it to its own module to isolate it, and also
mark it as deprecated with an eye on dropping it in a future major version of
NiFi.
*MIGRATION GUIDANCE NOTE*
Users of the {{RocksDBFlowFileRepository}} will have to download the new
{{org.apache.nifi::nifi-rocksdb-nar::x.y.z}} and add it to the lib directory
after upgrading past 1.16.3
was:
For arm64 / Apple M1 compatibility, RocksDB needs to be bumped to >= v6.29.4.1
(see below for runtime errors on an Apple Silicon M1 Mac with the current
version).
RocksDB is used only by the RocksDBFlowFileRepository implementation. After
wider discussion with the community and who could help test and verify this
change, we came to the conclusion it makes the most sense to focus maintenance
efforts on the default FlowFileRepository implementation and deprecate the
RocksDB variant.
For now, we will bump the version of RocksDB for arm64 compatibility on macOS,
but this ticket is also to extract it to its own module to isolate it, and also
mark it as deprecated with an eye on dropping it in a future major version of
NiFi.
*MIGRATION GUIDANCE NOTE*
Users of the {{RocksDBFlowFileRepository}} will have to download the new
{{org.apache.nifi::nifi-rocksdb-nar::x.y.z}} and add it to the lib directory
after upgrading past 1.16.3
----
{noformat}
[INFO] -----------------< org.apache.nifi:nifi-rocksdb-utils >-----------------
[INFO] Building nifi-rocksdb-utils 1.16.0-SNAPSHOT [35/642]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0:enforce (enforce-maven-version) @
nifi-rocksdb-utils ---
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0:enforce (enforce-java-version) @
nifi-rocksdb-utils ---
[INFO]
[INFO] --- maven-remote-resources-plugin:1.7.0:process
(process-resource-bundles) @ nifi-rocksdb-utils ---
[INFO] Preparing remote bundle org.apache:apache-jar-resource-bundle:1.4
[INFO] Copying 3 resources from 1 bundle.
[INFO]
[INFO] --- maven-resources-plugin:3.2.0:resources (default-resources) @
nifi-rocksdb-utils ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Using 'UTF-8' encoding to copy filtered properties files.
[INFO] skip non existing resourceDirectory
/Users/kdoran/dev/code/nifi/nifi-commons/nifi-rocksdb-utils/src/main/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.9.0:compile (default-compile) @
nifi-rocksdb-utils ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:3.2.0:testResources (default-testResources) @
nifi-rocksdb-utils ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Using 'UTF-8' encoding to copy filtered properties files.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.9.0:testCompile (default-testCompile) @
nifi-rocksdb-utils ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-compiler-plugin:3.9.0:testCompile (groovy-tests) @
nifi-rocksdb-utils ---
[INFO] Changes detected - recompiling the module!
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-surefire-plugin:3.0.0-M5:test (default-test) @
nifi-rocksdb-utils ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.nifi.rocksdb.TestRocksDBMetronome
[ERROR] Tests run: 10, Failures: 2, Errors: 7, Skipped: 0, Time elapsed: 0.097
s <<< FAILURE! - in org.apache.nifi.rocksdb.TestRocksDBMetronome
[ERROR] org.apache.nifi.rocksdb.TestRocksDBMetronome.testColumnFamilies(Path)
Time elapsed: 0.058 s <<< ERROR!
java.lang.UnsatisfiedLinkError:
/private/var/folders/dj/1c85sd0d6dvcp1fltmwr5nl40000gn/T/librocksdbjni1540031708884427750.jnilib:
dlopen(/private/var/folders/dj/1c85sd0d6dvcp1fltmwr5nl40000gn/T/librocksdbjni1540031708884427750.jnilib,
0x0001): tried:
'/private/var/folders/dj/1c85sd0d6dvcp1fltmwr5nl40000gn/T/librocksdbjni1540031708884427750.jnilib'
(mach-o file, but is an incompatible architecture (have 'x86_64', need
'arm64e')), '/usr/lib/librocksdbjni1540031708884427750.jnilib' (no such file)
at
org.apache.nifi.rocksdb.TestRocksDBMetronome.testColumnFamilies(TestRocksDBMetronome.java:170)
[ERROR] org.apache.nifi.rocksdb.TestRocksDBMetronome.testWaitForSync(Path)
Time elapsed: 0.003 s <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class org.rocksdb.RocksDB
at
org.apache.nifi.rocksdb.TestRocksDBMetronome.testWaitForSync(TestRocksDBMetronome.java:267)
[ERROR]
org.apache.nifi.rocksdb.TestRocksDBMetronome.testWaitForSyncWithValue(Path)
Time elapsed: 0.001 s <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class org.rocksdb.RocksDB
at
org.apache.nifi.rocksdb.TestRocksDBMetronome.testWaitForSyncWithValue(TestRocksDBMetronome.java:299)
[ERROR] org.apache.nifi.rocksdb.TestRocksDBMetronome.testCounterIncrement(Path)
Time elapsed: 0.001 s <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class org.rocksdb.RocksDB
at
org.apache.nifi.rocksdb.TestRocksDBMetronome.testCounterIncrement(TestRocksDBMetronome.java:247)
[ERROR] org.apache.nifi.rocksdb.TestRocksDBMetronome.testPutGetDelete(Path)
Time elapsed: 0.001 s <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class org.rocksdb.RocksDB
at
org.apache.nifi.rocksdb.TestRocksDBMetronome.testPutGetDelete(TestRocksDBMetronome.java:105)
[ERROR] org.apache.nifi.rocksdb.TestRocksDBMetronome.testPutClosed(Path) Time
elapsed: 0.003 s <<< FAILURE!
org.opentest4j.AssertionFailedError: Unexpected exception type thrown ==>
expected: <java.lang.IllegalStateException> but was:
<java.lang.NoClassDefFoundError>
at
org.apache.nifi.rocksdb.TestRocksDBMetronome.testPutClosed(TestRocksDBMetronome.java:152)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class
org.rocksdb.RocksDB
at
org.apache.nifi.rocksdb.TestRocksDBMetronome.lambda$testPutClosed$1(TestRocksDBMetronome.java:155)
at
org.apache.nifi.rocksdb.TestRocksDBMetronome.testPutClosed(TestRocksDBMetronome.java:152)
[ERROR]
org.apache.nifi.rocksdb.TestRocksDBMetronome.testPutGetConfiguration(Path)
Time elapsed: 0 s <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class org.rocksdb.RocksDB
at
org.apache.nifi.rocksdb.TestRocksDBMetronome.testPutGetConfiguration(TestRocksDBMetronome.java:128)
[ERROR] org.apache.nifi.rocksdb.TestRocksDBMetronome.testPutBeforeInit(Path)
Time elapsed: 0.001 s <<< FAILURE!
org.opentest4j.AssertionFailedError: Unexpected exception type thrown ==>
expected: <java.lang.IllegalStateException> but was:
<java.lang.NoClassDefFoundError>
at
org.apache.nifi.rocksdb.TestRocksDBMetronome.testPutBeforeInit(TestRocksDBMetronome.java:141)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class
org.rocksdb.RocksDB
at
org.apache.nifi.rocksdb.TestRocksDBMetronome.lambda$testPutBeforeInit$0(TestRocksDBMetronome.java:144)
at
org.apache.nifi.rocksdb.TestRocksDBMetronome.testPutBeforeInit(TestRocksDBMetronome.java:141)
[ERROR] org.apache.nifi.rocksdb.TestRocksDBMetronome.testIterator(Path) Time
elapsed: 0 s <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class org.rocksdb.RocksDB
at
org.apache.nifi.rocksdb.TestRocksDBMetronome.testIterator(TestRocksDBMetronome.java:220)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestRocksDBMetronome.testPutBeforeInit:141 Unexpected exception type
thrown ==> expected: <java.lang.IllegalStateException> but was:
<java.lang.NoClassDefFoundError>
[ERROR] TestRocksDBMetronome.testPutClosed:152 Unexpected exception type
thrown ==> expected: <java.lang.IllegalStateException> but was:
<java.lang.NoClassDefFoundError>
[ERROR] Errors:
[ERROR] TestRocksDBMetronome.testColumnFamilies:170 » UnsatisfiedLink
/private/var/fol...
[ERROR] TestRocksDBMetronome.testCounterIncrement:247 » NoClassDefFound Could
not init...
[ERROR] TestRocksDBMetronome.testIterator:220 » NoClassDefFound Could not
initialize c...
[ERROR] TestRocksDBMetronome.testPutGetConfiguration:128 » NoClassDefFound
Could not i...
[ERROR] TestRocksDBMetronome.testPutGetDelete:105 » NoClassDefFound Could not
initiali...
[ERROR] TestRocksDBMetronome.testWaitForSync:267 » NoClassDefFound Could not
initializ...
[ERROR] TestRocksDBMetronome.testWaitForSyncWithValue:299 » NoClassDefFound
Could not ...
[INFO]
[ERROR] Tests run: 10, Failures: 2, Errors: 7, Skipped: 0
{noformat}
and
{noformat}
[INFO] Running
org.apache.nifi.controller.repository.TestRocksDBFlowFileRepository
[ERROR] Tests run: 12, Failures: 0, Errors: 11, Skipped: 0, Time elapsed: 0.103
s <<< FAILURE! - in
org.apache.nifi.controller.repository.TestRocksDBFlowFileRepository
[ERROR]
org.apache.nifi.controller.repository.TestRocksDBFlowFileRepository.testRecoveryModeWithContinuedLoading
Time elapsed: 0.072 s <<< ERROR!
java.lang.UnsatisfiedLinkError:
/private/var/folders/dj/1c85sd0d6dvcp1fltmwr5nl40000gn/T/librocksdbjni8333131115704997159.jnilib:
dlopen(/private/var/folders/dj/1c85sd0d6dvcp1fltmwr5nl40000gn/T/librocksdbjni8333131115704997159.jnilib,
0x0001): tried:
'/private/var/folders/dj/1c85sd0d6dvcp1fltmwr5nl40000gn/T/librocksdbjni8333131115704997159.jnilib'
(mach-o file, but is an incompatible architecture (have 'x86_64', need
'arm64e')), '/usr/lib/librocksdbjni8333131115704997159.jnilib' (no such file)
at
org.apache.nifi.controller.repository.TestRocksDBFlowFileRepository.testRecoveryModeWithContinuedLoading(TestRocksDBFlowFileRepository.java:516)
[ERROR]
org.apache.nifi.controller.repository.TestRocksDBFlowFileRepository.testUnknownVersion
Time elapsed: 0 s <<< ERROR!
java.lang.Exception: Unexpected exception,
expected<java.lang.IllegalStateException> but
was<java.lang.NoClassDefFoundError>
at
org.apache.nifi.controller.repository.TestRocksDBFlowFileRepository.testUnknownVersion(TestRocksDBFlowFileRepository.java:436)
[ERROR]
org.apache.nifi.controller.repository.TestRocksDBFlowFileRepository.testDoNotRemoveOrphans
Time elapsed: 0.001 s <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class org.rocksdb.RocksDB
at
org.apache.nifi.controller.repository.TestRocksDBFlowFileRepository.testDoNotRemoveOrphans(TestRocksDBFlowFileRepository.java:354)
[ERROR]
org.apache.nifi.controller.repository.TestRocksDBFlowFileRepository.testStallStop
Time elapsed: 0.001 s <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class org.rocksdb.RocksDB
at
org.apache.nifi.controller.repository.TestRocksDBFlowFileRepository.testStallStop(TestRocksDBFlowFileRepository.java:598)
[ERROR]
org.apache.nifi.controller.repository.TestRocksDBFlowFileRepository.testSwapLocationsRestored
Time elapsed: 0 s <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class org.rocksdb.RocksDB
at
org.apache.nifi.controller.repository.TestRocksDBFlowFileRepository.testSwapLocationsRestored(TestRocksDBFlowFileRepository.java:124)
[ERROR]
org.apache.nifi.controller.repository.TestRocksDBFlowFileRepository.testRecoveryMode
Time elapsed: 0.001 s <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class org.rocksdb.RocksDB
at
org.apache.nifi.controller.repository.TestRocksDBFlowFileRepository.testRecoveryMode(TestRocksDBFlowFileRepository.java:451)
[ERROR]
org.apache.nifi.controller.repository.TestRocksDBFlowFileRepository.testSwapLocationsUpdatedOnRepoUpdate
Time elapsed: 0.001 s <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class org.rocksdb.RocksDB
at
org.apache.nifi.controller.repository.TestRocksDBFlowFileRepository.testSwapLocationsUpdatedOnRepoUpdate(TestRocksDBFlowFileRepository.java:168)
[ERROR]
org.apache.nifi.controller.repository.TestRocksDBFlowFileRepository.testResourceClaimsIncremented
Time elapsed: 0.02 s <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class org.rocksdb.RocksDB
at
org.apache.nifi.controller.repository.TestRocksDBFlowFileRepository.testResourceClaimsIncremented(TestRocksDBFlowFileRepository.java:223)
[ERROR]
org.apache.nifi.controller.repository.TestRocksDBFlowFileRepository.testRestartWithOneRecord
Time elapsed: 0 s <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class org.rocksdb.RocksDB
at
org.apache.nifi.controller.repository.TestRocksDBFlowFileRepository.testRestartWithOneRecord(TestRocksDBFlowFileRepository.java:284)
[ERROR]
org.apache.nifi.controller.repository.TestRocksDBFlowFileRepository.testKnownVersion
Time elapsed: 0.001 s <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class org.rocksdb.RocksDB
at
org.apache.nifi.controller.repository.TestRocksDBFlowFileRepository.testKnownVersion(TestRocksDBFlowFileRepository.java:422)
[ERROR]
org.apache.nifi.controller.repository.TestRocksDBFlowFileRepository.testRemoveOrphans
Time elapsed: 0.001 s <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class org.rocksdb.RocksDB
at
org.apache.nifi.controller.repository.TestRocksDBFlowFileRepository.testRemoveOrphans(TestRocksDBFlowFileRepository.java:390)
{noformat}
> Extract RocksDBFlowFileRepository to its own module and deprecate
> -----------------------------------------------------------------
>
> Key: NIFI-9810
> URL: https://issues.apache.org/jira/browse/NIFI-9810
> Project: Apache NiFi
> Issue Type: Sub-task
> Reporter: Kevin Doran
> Priority: Minor
> Time Spent: 1h
> Remaining Estimate: 0h
>
> For arm64 / Apple M1 compatibility, RocksDB needs to be bumped to >=
> v6.29.4.1 (see below for runtime errors on an Apple Silicon M1 Mac with the
> current version).
> RocksDB is used only by the RocksDBFlowFileRepository implementation. After
> wider discussion with the community and who could help test and verify this
> change, we came to the conclusion it makes the most sense to focus
> maintenance efforts on the default FlowFileRepository implementation and
> deprecate the RocksDB variant.
> For now, we will bump the version of RocksDB for arm64 compatibility on
> macOS, but this ticket is also to extract it to its own module to isolate it,
> and also mark it as deprecated with an eye on dropping it in a future major
> version of NiFi.
> *MIGRATION GUIDANCE NOTE*
> Users of the {{RocksDBFlowFileRepository}} will have to download the new
> {{org.apache.nifi::nifi-rocksdb-nar::x.y.z}} and add it to the lib directory
> after upgrading past 1.16.3
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
