+all user
-------- Forwarding messages -------- From: "Young" <[email protected]> Date: 2015-06-26 10:31:19 To: "Ted Yu" <[email protected]> Subject: Re:Re: Spark1.4.0 compiling error with java1.6.0_20： sun.misc.Unsafe cannot be applied to (java.lang.Object,long,java.lang.Object,long,long) Thanks for replay：） I can compile spark1.4.0 with jre-1.7.0-openjdk successfully， but since my hadoop cluster（hadoop2.2） is using java6， it seems that spark1.4.0 compiled with java7, which is under yarn mode, cannot start ApplicationMaster. Logs are as followed ( command: spark-shell --master yarn-client): dirver log: ... 2015-06-26 10:17:53,962 INFO [main] yarn.Client: Application report for application_1430702413439_215318 (state: ACCEPTED) 2015-06-26 10:17:53,966 INFO [main] yarn.Client: client token: N/A diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: 0 queue: root.yarn start time: 1435285072944 final status: UNDEFINED tracking URL: RMhost:8088/proxy/application_1430702413439_215318/ user: yarn 2015-06-26 10:17:54,967 INFO [main] yarn.Client: Application report for application_1430702413439_215318 (state: ACCEPTED) 2015-06-26 10:17:55,969 INFO [main] yarn.Client: Application report for application_1430702413439_215318 (state: ACCEPTED) 2015-06-26 10:17:56,971 INFO [main] yarn.Client: Application report for application_1430702413439_215318 (state: ACCEPTED) 2015-06-26 10:17:57,973 INFO [main] yarn.Client: Application report for application_1430702413439_215318 (state: FAILED) 2015-06-26 10:17:57,973 INFO [main] yarn.Client: client token: N/A diagnostics: Application application_1430702413439_215318 failed 2 times due to AM Container for appattempt_1430702413439_215318_000002 exited with exitCode: 1 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) at org.apache.hadoop.util.Shell.run(Shell.java:379) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) .Failing this attempt.. Failing the application. ApplicationMaster host: N/A ApplicationMaster RPC port: 0 queue: root.yarn start time: 1435285072944 final status: FAILED tracking URL:RMhost:8088/cluster/app/application_1430702413439_215318 user: yarn 2015-06-26 10:17:57,974 ERROR [main] spark.SparkContext: Error initializing SparkContext. org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master. at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:114) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:59) at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141) at org.apache.spark.SparkContext.<init>(SparkContext.scala:497) at org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1017) at $line3.$read$$iwC$$iwC.<init>(<console>:9) at $line3.$read$$iwC.<init>(<console>:18) at $line3.$read.<init>(<console>:20) at $line3.$read$.<init>(<console>:24) at $line3.$read$.<clinit>(<console>) at $line3.$eval$.<init>(<console>:7) at $line3.$eval$.<clinit>(<console>) at $line3.$eval.$print(<console>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method .... container log: ... LogType: stderr LogLength: 661 Log Contents: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/deploy/yarn/ExecutorLauncher Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.ExecutorLauncher at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) Could not find the main class: org.apache.spark.deploy.yarn.ExecutorLauncher. Program will exit. ... At 2015-06-26 10:06:29, "Ted Yu" <[email protected]> wrote: Looks like the Java 1.6 version of copyMemory doesn't support specification of offsets. This means extra memory copy. Can you upgrade your Java version ? Thanks On Jun 25, 2015, at 6:35 PM, Young <[email protected]> wrote: Hi ,all: When compiling spark1.4.0 with java1.6.0_20 (maven version: 3.2.5, scala vertion: 2.10.4), I always encounter the compiling error as followed: ****/spark-1.4.0/unsafe/src/main/java/org/apache/spark/unsafe/PlatformDependent.java:149: copyMemory(long,long,long) in sun.misc.Unsafe cannot be applied to (java.lang.Object,long,java.lang.Object,long,long) [ERROR] _UNSAFE.copyMemory(src, srcOffset, dst, dstOffset, size); [ERROR] ^ [ERROR] 1 error [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Spark Project Parent POM ........................... SUCCESS [ 7.304 s] [INFO] Spark Launcher Project ............................. SUCCESS [ 13.355 s] [INFO] Spark Project Networking ........................... SUCCESS [ 11.703 s] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 5.817 s] [INFO] Spark Project Unsafe ............................... FAILURE [ 1.048 s] [INFO] Spark Project Core ................................. SKIPPED [INFO] Spark Project Bagel ................................ SKIPPED [INFO] Spark Project GraphX ............................... SKIPPED [INFO] Spark Project Streaming ............................ SKIPPED [INFO] Spark Project Catalyst ............................. SKIPPED [INFO] Spark Project SQL .................................. SKIPPED [INFO] Spark Project ML Library ........................... SKIPPED [INFO] Spark Project Tools ................................ SKIPPED [INFO] Spark Project Hive ................................. SKIPPED [INFO] Spark Project REPL ................................. SKIPPED [INFO] Spark Project YARN ................................. SKIPPED [INFO] Spark Project Assembly ............................. SKIPPED [INFO] Spark Project External Twitter ..................... SKIPPED [INFO] Spark Project External Flume Sink .................. SKIPPED [INFO] Spark Project External Flume ....................... SKIPPED [INFO] Spark Project External MQTT ........................ SKIPPED [INFO] Spark Project External ZeroMQ ...................... SKIPPED [INFO] Spark Project External Kafka ....................... SKIPPED [INFO] Spark Project Examples ............................. SKIPPED [INFO] Spark Project External Kafka Assembly .............. SKIPPED [INFO] Spark Project YARN Shuffle Service ................. SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 40.705 s [INFO] Finished at: 2015-06-26T09:15:33+08:00 [INFO] Final Memory: 74M/838M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.1:compile (scala-compile-first) on project spark-unsafe_2.10: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.1:compile failed. CompileFailed -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :spark-unsafe_2.10 According to the error info “/PlatformDependent.java:149: copyMemory(long,long,long) in sun.misc.Unsafe cannot be applied to (java.lang.Object,long,java.lang.Object,long,long)”，I find this in PlatformDependent.java： 141 static public void copyMemory( 142 Object src, 143 long srcOffset, 144 Object dst, 145 long dstOffset, 146 long length) { 147 while (length > 0) { 148 long size = Math.min(length, UNSAFE_COPY_THRESHOLD); 149 _UNSAFE.copyMemory(src, srcOffset, dst, dstOffset, size); 150 length -= size; 151 srcOffset += size; 152 dstOffset += size; 153 } 154 } It seems that copyMemory (java.lang.Object,long,java.lang.Object,long,long) is introduced since Java7？ But According to the official document， "Spark runs on Java 6+, Python 2.6+ and R 3.1+. For the Scala API, Spark 1.4.0 uses Scala 2.10. You will need to use a compatible Scala version (2.10.x)." So, how can I compile spark1.4.0 successfully with java1.6.0_20? Does anybody have the same error? Thank you very much for help：） Sincerely， Young
