[
https://issues.apache.org/jira/browse/HBASE-19387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16272275#comment-16272275
]
Yuqi Gu commented on HBASE-19387:
---------------------------------
Everything is ok for running hbase-spark unit tests both on x86 and Arm64.
> HBase-spark snappy.SnappyError on Arm64
> ---------------------------------------
>
> Key: HBASE-19387
> URL: https://issues.apache.org/jira/browse/HBASE-19387
> Project: HBase
> Issue Type: Bug
> Components: spark, test
> Affects Versions: 3.0.0
> Reporter: Yuqi Gu
> Priority: Minor
> Attachments: HBASE-19387.patch
>
>
> When running the hbase-spark Unit tests on Arm64, the failures are shown as
> follows:
>
> {code:java}
> scalatest-maven-plugin:1.0:test (test) @ hbase-spark ---
> Discovery starting.
> Discovery completed in 2 seconds, 837 milliseconds.
> Run starting. Expected test count is: 79
> HBaseDStreamFunctionsSuite:
> Formatting using clusterid: testClusterID
> - bulkput to test HBase client *** FAILED ***
> java.lang.reflect.InvocationTargetException:
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
> at
> org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65)
> at
> org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
> at
> org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80)
> at
> org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
> at
> org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
> ...
> Cause: java.lang.IllegalArgumentException: org.xerial.snappy.SnappyError:
> [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux
> and os.arch=aarch64
> at
> org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:156)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
> at
> org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65)
> at
> org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
> at
> org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80)
> at
> org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
> ...
> Cause: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no
> native library is found for os.name=Linux and os.arch=aarch64
> at org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:331)
> at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:171)
> at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:152)
> at org.xerial.snappy.Snappy.<clinit>(Snappy.java:46)
> at
> org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:154)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
> ...
> Formatting using clusterid: testClusterID
> PartitionFilterSuite:
> *** RUN ABORTED ***
> java.lang.reflect.InvocationTargetException:
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
> at
> org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65)
> at
> org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
> at
> org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80)
> at
> org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
> at
> org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
> ...
> Cause: java.lang.IllegalArgumentException: java.lang.NoClassDefFoundError:
> Could not initialize class org.xerial.snappy.Snappy
> at
> org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:156)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
> at
> org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65)
> at
> org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
> at
> org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80)
> at
> org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
> ...
> Cause: java.lang.NoClassDefFoundError: Could not initialize class
> org.xerial.snappy.Snappy
> at
> org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:154)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
> at
> org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65)
> at
> org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
> at
> org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80)
> at
> org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
> {code}
> Root Cause:
> The spark version in hbase is 1.6.0.
> And the snappy version based on hbase-1.6.0 do not support the Arm64.
> Fix it by upgrading snappy-java to 1.1.4 in hbase-spark.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
