the linux box roughly as capable as your laptop?
On Tue, Oct 3, 2017 at 12:51 PM, Stack <[email protected]> wrote: > Tests passed eventually for me: > > real 47m6.498s > user 5m29.671s > sys 0m41.885s > > ... which is a big diff from macosx run. > > Need to look into this. > > St.Ack > > > On Tue, Oct 3, 2017 at 10:03 AM, Stack <[email protected]> wrote: > >> The below gets us further but now I see that the spark tests take a really >> long time to run on linux but complete promptly on macosx (2m 55s). >> Looking.... >> >> St.Ack >> >> On Tue, Oct 3, 2017 at 9:13 AM, Stack <[email protected]> wrote: >> >>> This seems to work for me. Does it work for you? >>> >>> >>> diff --git a/hbase-spark/pom.xml b/hbase-spark/pom.xml >>> index 594aa2a..6d191e3 100644 >>> --- a/hbase-spark/pom.xml >>> +++ b/hbase-spark/pom.xml >>> @@ -568,6 +568,9 @@ >>> <junitxml>.</junitxml> >>> <filereports>WDF TestSuite.txt</filereports> >>> <parallel>false</parallel> >>> + <systemProperties> >>> + <org.apache.hadoop.hbase.shaded.io.netty.packagePrefix>org. >>> apache.hadoop.hbase.shaded.</org.apache.hadoop.hbase. >>> shaded.io.netty.packagePrefix> >>> + </systemProperties> >>> </configuration> >>> <executions> >>> <execution> >>> >>> St.Ack >>> >>> On Tue, Oct 3, 2017 at 8:45 AM, Amit Kabra <[email protected]> >>> wrote: >>> >>>> Thanks Stack / Sean Busbey for replying. >>>> >>>> OS : Ubuntu 16.04.2 , 64 bit. >>>> Eclipse : Version: Neon.3 Release (4.6.3) >>>> HBase branch : branch-2 >>>> Command line test to reproduce : mvn clean package >>>> -Dtest=TestIncrementalBackup >>>> Reproduce from eclipse , right click on TestIncBackupRestore and run as >>>> junit from test class TestIncrementalBackup. >>>> No I am not embedding hbase in my application. I have just checked out >>>> hbase , switched to branch-2 and run the unit test from command line or >>>> from eclipse. Failing with same error in both cases. >>>> Yes the trailing period is also present. >>>> >>>> Thanks, >>>> Amit Kabra. >>>> >>>> >>>> >>>> >>>> >>>> On Tue, Oct 3, 2017 at 8:53 PM, Stack <[email protected]> wrote: >>>> >>>> > Thank you for the detail. >>>> > >>>> > Pardon the questions below asking for yet more detail. I am unable to >>>> > reproduce locally or on another os (though we see this issue up on our >>>> > build box). >>>> > >>>> > What is your OS when you see the below? >>>> > >>>> > On Tue, Oct 3, 2017 at 2:06 AM, Amit Kabra <[email protected]> >>>> > wrote: >>>> > >>>> > > Hello, >>>> > > >>>> > > I am using "branch-2" branch of hbase, when I run unit test I get >>>> > following >>>> > > error for netty "java.lang.UnsatisfiedLinkError: failed to load the >>>> > > required native library" >>>> > > >>>> > > >>>> > > This is running a unit test in your eclipse environment? >>>> > >>>> > You are trying to run an hbase-spark unit test when you see the above? >>>> > >>>> > >>>> > >>>> > >>>> > > *I already have following set in "maven-surefire-plugin" in pom.xml >>>> as >>>> > > per http://hbase.apache.org/book.html#thirdparty >>>> > > <http://hbase.apache.org/book.html#thirdparty>* >>>> > > >>>> > > >>>> > > >>>> > >>>> > Are you embedding hbase into your application? >>>> > >>>> > >>>> > >>>> > > <systemPropertyVariables> >>>> > > <!-- >>>> > > <test.build.classes>${test.build.classes}</test.build. >>>> > classe >>>> > > s> >>>> > > --> >>>> > > <!--For shaded netty, to find the relocated .so. >>>> > > Trick from >>>> > > https://stackoverflow.com/questions/33825743/rename- >>>> > > files-inside-a-jar-using-some-maven-plugin >>>> > > <https://stackoverflow.com/questions/33825743/rename- >>>> > files-inside-a-jar-using-some-maven-plugin> >>>> > > >>>> > > The netty jar has a .so in it. Shading requires >>>> rename of >>>> > > the .so and then passing a system >>>> > > property so netty finds the renamed .so and >>>> associates it >>>> > > w/ the relocated netty files. >>>> > > >>>> > > The relocated netty is in hbase-thirdparty >>>> dependency. >>>> > Just >>>> > > set this propery globally rather >>>> > > than per module. >>>> > > --> >>>> > > <org.apache.hadoop.hbase.shad >>>> ed.io.netty.packagePrefix> >>>> > > org.apache.hadoop.hbase.shaded.</org.apache.hadoop. >>>> > hbase.shaded.io.netty. >>>> > > packagePrefix> >>>> > > </systemPropertyVariables> >>>> > > >>>> > > >>>> > > >>>> > > *And I see in the code as per HBASE-18271, all io.netty is already >>>> > replaced >>>> > > with org.apache.hadoop.hbase.shaded.io.netty* >>>> > > >>>> > > >>>> > The trailing period is also present? >>>> > >>>> > >>>> > >>>> > > >>>> > > If I run a test from eclipse , I see the error immediately and my >>>> test >>>> > > doesn't run, but when I run from command line , the test runs but I >>>> get >>>> > the >>>> > > error at the end when the mvn command finishes. >>>> > > >>>> > > >>>> > > Is it any eclipse test? >>>> > >>>> > Thank you. Let me try and fix this this morning. >>>> > >>>> > S >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > > *Here is the complete error output.* >>>> > > >>>> > > >>>> > > >>>> > > [INFO] >>>> > > [INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @ >>>> > hbase-spark >>>> > > --- >>>> > > [INFO] >>>> > > [INFO] --- scalatest-maven-plugin:1.0:test (test) @ hbase-spark --- >>>> > > Discovery starting. >>>> > > Discovery completed in 1 second, 558 milliseconds. >>>> > > Run starting. Expected test count is: 79 >>>> > > HBaseDStreamFunctionsSuite: >>>> > > Formatting using clusterid: testClusterID >>>> > > *** RUN ABORTED *** >>>> > > java.io.IOException: Shutting down >>>> > > at org.apache.hadoop.hbase.MiniHBaseCluster.init( >>>> > > MiniHBaseCluster.java:232) >>>> > > at org.apache.hadoop.hbase.MiniHBaseCluster.<init>( >>>> > > MiniHBaseCluster.java:94) >>>> > > at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCl >>>> uster( >>>> > > HBaseTestingUtility.java:1124) >>>> > > at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster( >>>> > > HBaseTestingUtility.java:1078) >>>> > > at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster( >>>> > > HBaseTestingUtility.java:949) >>>> > > at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster( >>>> > > HBaseTestingUtility.java:943) >>>> > > at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster( >>>> > > HBaseTestingUtility.java:872) >>>> > > at org.apache.hadoop.hbase.spark.HBaseDStreamFunctionsSuite.bef >>>> oreAll( >>>> > > HBaseDStreamFunctionsSuite.scala:41) >>>> > > at org.scalatest.BeforeAndAfterAll$class.beforeAll(BeforeAndAft >>>> erAll. >>>> > > scala:187) >>>> > > at org.apache.hadoop.hbase.spark.HBaseDStreamFunctionsSuite.bef >>>> oreAll( >>>> > > HBaseDStreamFunctionsSuite.scala:30) >>>> > > ... >>>> > > Cause: java.lang.RuntimeException: Failed construction of Master: >>>> class >>>> > > org.apache.hadoop.hbase.master.HMasterorg.apache. >>>> > > hadoop.hbase.shaded.io.netty.channel.epoll. >>>> > NativeStaticallyReferencedJniM >>>> > > ethods.epollin()I >>>> > > at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread( >>>> > > JVMClusterUtil.java:145) >>>> > > at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster( >>>> > > LocalHBaseCluster.java:217) >>>> > > at org.apache.hadoop.hbase.LocalHBaseCluster.<init>( >>>> > > LocalHBaseCluster.java:152) >>>> > > at org.apache.hadoop.hbase.MiniHBaseCluster.init( >>>> > > MiniHBaseCluster.java:214) >>>> > > at org.apache.hadoop.hbase.MiniHBaseCluster.<init>( >>>> > > MiniHBaseCluster.java:94) >>>> > > at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCl >>>> uster( >>>> > > HBaseTestingUtility.java:1124) >>>> > > at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster( >>>> > > HBaseTestingUtility.java:1078) >>>> > > at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster( >>>> > > HBaseTestingUtility.java:949) >>>> > > at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster( >>>> > > HBaseTestingUtility.java:943) >>>> > > at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster( >>>> > > HBaseTestingUtility.java:872) >>>> > > ... >>>> > > Cause: java.lang.UnsatisfiedLinkError: failed to load the required >>>> > > native >>>> > > library >>>> > > at org.apache.hadoop.hbase.shaded.io.netty.channel.epoll. >>>> > > Epoll.ensureAvailability(Epoll.java:78) >>>> > > at org.apache.hadoop.hbase.shaded.io.netty.channel.epoll. >>>> > > EpollEventLoopGroup.<clinit>(EpollEventLoopGroup.java:38) >>>> > > at org.apache.hadoop.hbase.util.NettyEventLoopGroupConfig.<init>( >>>> > > NettyEventLoopGroupConfig.java:61) >>>> > > at org.apache.hadoop.hbase.regionserver.HRegionServer.< >>>> > > init>(HRegionServer.java:552) >>>> > > at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:475) >>>> > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>>> > Method) >>>> > > at sun.reflect.NativeConstructorAccessorImpl.newInstance( >>>> > > NativeConstructorAccessorImpl.java:62) >>>> > > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance( >>>> > > DelegatingConstructorAccessorImpl.java:45) >>>> > > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) >>>> > > at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread( >>>> > > JVMClusterUtil.java:140) >>>> > > ... >>>> > > Cause: java.lang.UnsatisfiedLinkError: org.apache.hadoop.hbase. >>>> > > shaded.io.netty.channel.epoll.NativeStaticallyReferencedJniM >>>> > > ethods.epollin()I >>>> > > at org.apache.hadoop.hbase.shaded.io.netty.channel.epoll. >>>> > > NativeStaticallyReferencedJniMethods.epollin(Native Method) >>>> > > at org.apache.hadoop.hbase.shaded.io.netty.channel.epoll. >>>> > > Native.<clinit>(Native.java:66) >>>> > > at org.apache.hadoop.hbase.shaded.io.netty.channel.epoll. >>>> > > Epoll.<clinit>(Epoll.java:33) >>>> > > at org.apache.hadoop.hbase.shaded.io.netty.channel.epoll. >>>> > > EpollEventLoopGroup.<clinit>(EpollEventLoopGroup.java:38) >>>> > > at org.apache.hadoop.hbase.util.NettyEventLoopGroupConfig.<init>( >>>> > > NettyEventLoopGroupConfig.java:61) >>>> > > at org.apache.hadoop.hbase.regionserver.HRegionServer.< >>>> > > init>(HRegionServer.java:552) >>>> > > at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:475) >>>> > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>>> > Method) >>>> > > at sun.reflect.NativeConstructorAccessorImpl.newInstance( >>>> > > NativeConstructorAccessorImpl.java:62) >>>> > > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance( >>>> > > DelegatingConstructorAccessorImpl.java:45) >>>> > > ... >>>> > > [INFO] ------------------------------------------------------------ >>>> > > ------------ >>>> > > [INFO] Reactor Summary: >>>> > > [INFO] >>>> > > [INFO] Apache HBase ....................................... SUCCESS >>>> [ >>>> > > 1.575 s] >>>> > > [INFO] Apache HBase - Checkstyle .......................... SUCCESS [ >>>> > > 0.317 s] >>>> > > [INFO] Apache HBase - Annotations ......................... SUCCESS [ >>>> > > 0.537 s] >>>> > > [INFO] Apache HBase - Build Configuration ................. SUCCESS [ >>>> > > 0.053 s] >>>> > > [INFO] Apache HBase - Shaded Protocol ..................... SUCCESS [ >>>> > > 15.410 s] >>>> > > [INFO] Apache HBase - Common .............................. SUCCESS [ >>>> > > 4.603 s] >>>> > > [INFO] Apache HBase - Metrics API ......................... SUCCESS [ >>>> > > 1.213 s] >>>> > > [INFO] Apache HBase - Hadoop Compatibility ................ SUCCESS [ >>>> > > 0.985 s] >>>> > > [INFO] Apache HBase - Metrics Implementation .............. SUCCESS [ >>>> > > 0.863 s] >>>> > > [INFO] Apache HBase - Hadoop Two Compatibility ............ SUCCESS [ >>>> > > 1.750 s] >>>> > > [INFO] Apache HBase - Protocol ............................ SUCCESS [ >>>> > > 4.880 s] >>>> > > [INFO] Apache HBase - Client .............................. SUCCESS [ >>>> > > 5.233 s] >>>> > > [INFO] Apache HBase - Replication ......................... SUCCESS [ >>>> > > 1.040 s] >>>> > > [INFO] Apache HBase - Prefix Tree ......................... SUCCESS [ >>>> > > 1.121 s] >>>> > > [INFO] Apache HBase - Procedure ........................... SUCCESS [ >>>> > > 1.084 s] >>>> > > [INFO] Apache HBase - Resource Bundle ..................... SUCCESS [ >>>> > > 0.092 s] >>>> > > [INFO] Apache HBase - Server .............................. SUCCESS [ >>>> > > 19.849 s] >>>> > > [INFO] Apache HBase - MapReduce ........................... SUCCESS [ >>>> > > 4.221 s] >>>> > > [INFO] Apache HBase - Testing Util ........................ SUCCESS [ >>>> > > 3.273 s] >>>> > > [INFO] Apache HBase - Thrift .............................. SUCCESS [ >>>> > > 5.519 s] >>>> > > [INFO] Apache HBase - RSGroup ............................. SUCCESS [ >>>> > > 3.408 s] >>>> > > [INFO] Apache HBase - Shell ............................... SUCCESS >>>> [ >>>> > > 3.859 s] >>>> > > [INFO] Apache HBase - Coprocessor Endpoint ................ SUCCESS [ >>>> > > 4.038 s] >>>> > > [INFO] Apache HBase - Backup .............................. SUCCESS >>>> > [01:13 >>>> > > min] >>>> > > [INFO] Apache HBase - Integration Tests ................... SUCCESS [ >>>> > > 4.229 s] >>>> > > [INFO] Apache HBase - Examples ............................ SUCCESS [ >>>> > > 3.471 s] >>>> > > [INFO] Apache HBase - Rest ................................ SUCCESS >>>> [ >>>> > > 4.448 s] >>>> > > [INFO] Apache HBase - External Block Cache ................ SUCCESS [ >>>> > > 2.040 s] >>>> > > [INFO] Apache HBase - Spark ............................... FAILURE >>>> [ >>>> > > 32.833 s] >>>> > > [INFO] Apache HBase - Spark Integration Tests ............. SKIPPED >>>> > > [INFO] Apache HBase - Assembly ............................ SKIPPED >>>> > > [INFO] Apache HBase - Shaded .............................. SKIPPED >>>> > > [INFO] Apache HBase - Shaded - Client ..................... SKIPPED >>>> > > [INFO] Apache HBase - Shaded - MapReduce .................. SKIPPED >>>> > > [INFO] Apache HBase Shaded Packaging Invariants ........... SKIPPED >>>> > > [INFO] Apache HBase - Archetypes .......................... SKIPPED >>>> > > [INFO] Apache HBase - Exemplar for hbase-client archetype . SKIPPED >>>> > > [INFO] Apache HBase - Exemplar for hbase-shaded-client archetype >>>> SKIPPED >>>> > > [INFO] Apache HBase - Archetype builder ................... SKIPPED >>>> > > [INFO] ------------------------------------------------------------ >>>> > > ------------ >>>> > > [INFO] BUILD FAILURE >>>> > > [INFO] ------------------------------------------------------------ >>>> > > ------------ >>>> > > [INFO] Total time: 03:26 min >>>> > > [INFO] Finished at: 2017-09-27T19:34:35+05:30 >>>> > > [INFO] Final Memory: 345M/6055M >>>> > > [INFO] ------------------------------------------------------------ >>>> > > ------------ >>>> > > [ERROR] Failed to execute goal org.scalatest:scalatest-maven- >>>> > > plugin:1.0:test >>>> > > (test) on project hbase-spark: There are test failures -> [Help 1] >>>> > > [ERROR] >>>> > > [ERROR] To see the full stack trace of the errors, re-run Maven with >>>> the >>>> > -e >>>> > > switch. >>>> > > [ERROR] Re-run Maven using the -X switch to enable full debug >>>> logging. >>>> > > [ERROR] >>>> > > [ERROR] For more information about the errors and possible solutions, >>>> > > please read the following articles: >>>> > > [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ >>>> > > MojoFailureException >>>> > > [ERROR] >>>> > > [ERROR] After correcting the problems, you can resume the build with >>>> the >>>> > > command >>>> > > [ERROR] mvn <goals> -rf :hbase-spark >>>> > > >>>> > >>>> >>> >>> >>
