I also ran your example locally and it worked with 0.8.1 and
0.9.0-rc1. So it's possible somehow you are pulling in an older
version if Spark or an incompatible version of Hadoop.
- Patrick
On Thu, Jan 16, 2014 at 9:39 AM, Patrick Wendell <pwend...@gmail.com> wrote:
> Hey Alex,
>
> Thanks for testing out this rc. Would you mind forking this into a different
> thread so we can discuss there?
>
> Also, does your application build and run correctly with spark 0.8.1? That
> would determine whether the problem is specifically with this rc...
>
> Patrick
>
> ---
> sent from my phone
>
> On Jan 15, 2014 11:44 PM, "Alex Cozzi" <alexco...@gmail.com> wrote:
>>
>> Oh, I forgot: I am using the “yarn” maven profile to target yarn 2.2
>>
>> Alex Cozzi
>> alexco...@gmail.com
>> On Jan 15, 2014, at 11:41 PM, Alex Cozzi <alexco...@gmail.com> wrote:
>>
>> > Just testing out the rc1. I create a dependent project (using maven) and
>> > I copied the HdfsTest.scala test, but I added a single line to save the
>> > file
>> > back to disk:
>> >
>> > package org.apache.spark.examples
>> >
>> > import org.apache.spark._
>> >
>> > object HdfsTest {
>> > def main(args: Array[String]) {
>> > val sc = new SparkContext(args(0), "HdfsTest",
>> > System.getenv("SPARK_HOME"),
>> > SparkContext.jarOfClass(this.getClass))
>> > val file = sc.textFile(args(1))
>> > val mapped = file.map(s => s.length).cache()
>> > for (iter <- 1 to 10) {
>> > val start = System.currentTimeMillis()
>> > for (x <- mapped) { x + 2 }
>> > // println("Processing: " + x)
>> > val end = System.currentTimeMillis()
>> > println("Iteration " + iter + " took " + (end-start) + " ms")
>> > mapped.saveAsTextFile("out")
>> > }
>> > System.exit(0)
>> > }
>> > }
>> >
>> > and this my pom file:
>> > <project xmlns="http://maven.apache.org/POM/4.0.0";
>> > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
>> > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
>> > http://maven.apache.org/maven-v4_0_0.xsd";>
>> > <modelVersion>4.0.0</modelVersion>
>> > <groupId>my.examples</groupId>
>> > <artifactId>spark-samples</artifactId>
>> > <version>0.0.1-SNAPSHOT</version>
>> > <inceptionYear>2014</inceptionYear>
>> >
>> > <properties>
>> > <maven.compiler.source>1.6</maven.compiler.source>
>> > <maven.compiler.target>1.6</maven.compiler.target>
>> > <encoding>UTF-8</encoding>
>> > <scala.tools.version>2.10</scala.tools.version>
>> > <scala.version>2.10.0</scala.version>
>> > </properties>
>> >
>> > <repositories>
>> > <repository>
>> > <id>spark staging</id>
>> >
>> > <url>https://repository.apache.org/content/repositories/orgapachespark-1001</url>
>> > </repository>
>> > </repositories>
>> >
>> > <dependencies>
>> > <dependency>
>> > <groupId>org.scala-lang</groupId>
>> > <artifactId>scala-library</artifactId>
>> > <version>${scala.version}</version>
>> > </dependency>
>> >
>> > <dependency>
>> > <groupId>org.apache.spark</groupId>
>> >
>> > <artifactId>spark-core_${scala.tools.version}</artifactId>
>> > <version>0.9.0-incubating</version>
>> > </dependency>
>> >
>> > <!-- Test -->
>> > <dependency>
>> > <groupId>junit</groupId>
>> > <artifactId>junit</artifactId>
>> > <version>4.11</version>
>> > <scope>test</scope>
>> > </dependency>
>> > <dependency>
>> > <groupId>org.specs2</groupId>
>> >
>> > <artifactId>specs2_${scala.tools.version}</artifactId>
>> > <version>1.13</version>
>> > <scope>test</scope>
>> > </dependency>
>> > <dependency>
>> > <groupId>org.scalatest</groupId>
>> >
>> > <artifactId>scalatest_${scala.tools.version}</artifactId>
>> > <version>2.0.M6-SNAP8</version>
>> > <scope>test</scope>
>> > </dependency>
>> > </dependencies>
>> >
>> > <build>
>> > <sourceDirectory>src/main/scala</sourceDirectory>
>> > <testSourceDirectory>src/test/scala</testSourceDirectory>
>> > <plugins>
>> > <plugin>
>> > <!-- see
>> > http://davidb.github.com/scala-maven-plugin -->
>> > <groupId>net.alchim31.maven</groupId>
>> >
>> > <artifactId>scala-maven-plugin</artifactId>
>> > <version>3.1.6</version>
>> > <configuration>
>> >
>> > <scalaCompatVersion>2.10</scalaCompatVersion>
>> > <jvmArgs>
>> > <jvmArg>-Xms128m</jvmArg>
>> > <jvmArg>-Xmx2048m</jvmArg>
>> > </jvmArgs>
>> > </configuration>
>> > <executions>
>> > <execution>
>> > <goals>
>> >
>> > <goal>compile</goal>
>> >
>> > <goal>testCompile</goal>
>> > </goals>
>> > <configuration>
>> > <args>
>> >
>> > <arg>-make:transitive</arg>
>> >
>> > <arg>-dependencyfile</arg>
>> >
>> > <arg>${project.build.directory}/.scala_dependencies</arg>
>> > </args>
>> > </configuration>
>> > </execution>
>> > </executions>
>> > </plugin>
>> > <plugin>
>> >
>> > <groupId>org.apache.maven.plugins</groupId>
>> >
>> > <artifactId>maven-surefire-plugin</artifactId>
>> > <version>2.13</version>
>> > <configuration>
>> > <useFile>false</useFile>
>> >
>> > <disableXmlReport>true</disableXmlReport>
>> > <!-- If you have classpath issue
>> > like NoDefClassError,... -->
>> > <!--
>> > useManifestOnlyJar>false</useManifestOnlyJar -->
>> > <includes>
>> >
>> > <include>**/*Test.*</include>
>> >
>> > <include>**/*Suite.*</include>
>> > </includes>
>> > </configuration>
>> > </plugin>
>> > <plugin>
>> > <groupId>org.codehaus.mojo</groupId>
>> > <artifactId>exec-maven-plugin</artifactId>
>> > <version>1.2.1</version>
>> > <executions>
>> > <execution>
>> > <goals>
>> > <goal>exec</goal>
>> > </goals>
>> > </execution>
>> > </executions>
>> > <configuration>
>> >
>> > <mainClass>org.apache.spark.examples.HdfsTest</mainClass>
>> > <arguments>
>> > <argument>local</argument>
>> >
>> > <argument>pom.xml</argument>
>> > </arguments>
>> > </configuration>
>> > </plugin>
>> > </plugins>
>> > </build>
>> > </project>
>> >
>> >
>> > now, when I run it either in eclipse or using "mvn exec:java" I get the
>> > following error:
>> > [INFO]
>> > [INFO] --- exec-maven-plugin:1.2.1:java (default-cli) @ spark-samples
>> > ---
>> > SLF4J: Class path contains multiple SLF4J bindings.
>> > SLF4J: Found binding in
>> > [jar:file:/Users/acozzi/.m2/repository/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> > SLF4J: Found binding in
>> > [jar:file:/Users/acozzi/.m2/repository/org/slf4j/slf4j-simple/1.6.1/slf4j-simple-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>> > explanation.
>> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>> > 14/01/15 23:37:57 INFO slf4j.Slf4jLogger: Slf4jLogger started
>> > 14/01/15 23:37:57 INFO Remoting: Starting remoting
>> > 14/01/15 23:37:57 INFO Remoting: Remoting started; listening on
>> > addresses :[akka.tcp://spark@10.0.1.10:53682]
>> > 14/01/15 23:37:57 INFO Remoting: Remoting now listens on addresses:
>> > [akka.tcp://spark@10.0.1.10:53682]
>> > 14/01/15 23:37:57 INFO spark.SparkEnv: Registering BlockManagerMaster
>> > 14/01/15 23:37:57 INFO storage.DiskBlockManager: Created local directory
>> > at
>> > /var/folders/mm/4qxz27w91p96v2zp5f9ncmqm38ychm/T/spark-local-20140115233757-7a41
>> > 14/01/15 23:37:57 INFO storage.MemoryStore: MemoryStore started with
>> > capacity 1218.8 MB.
>> > 14/01/15 23:37:57 INFO network.ConnectionManager: Bound socket to port
>> > 53683 with id = ConnectionManagerId(10.0.1.10,53683)
>> > 14/01/15 23:37:57 INFO storage.BlockManagerMaster: Trying to register
>> > BlockManager
>> > 14/01/15 23:37:57 INFO storage.BlockManagerMasterActor$BlockManagerInfo:
>> > Registering block manager 10.0.1.10:53683 with 1218.8 MB RAM
>> > 14/01/15 23:37:57 INFO storage.BlockManagerMaster: Registered
>> > BlockManager
>> > 14/01/15 23:37:57 INFO spark.HttpServer: Starting HTTP Server
>> > 14/01/15 23:37:57 INFO server.Server: jetty-7.6.8.v20121106
>> > 14/01/15 23:37:57 INFO server.AbstractConnector: Started
>> > SocketConnector@0.0.0.0:53684
>> > 14/01/15 23:37:57 INFO broadcast.HttpBroadcast: Broadcast server started
>> > at http://10.0.1.10:53684
>> > 14/01/15 23:37:57 INFO spark.SparkEnv: Registering MapOutputTracker
>> > 14/01/15 23:37:57 INFO spark.HttpFileServer: HTTP File server directory
>> > is
>> > /var/folders/mm/4qxz27w91p96v2zp5f9ncmqm38ychm/T/spark-e9304513-3714-430f-aa14-1a430a915d98
>> > 14/01/15 23:37:57 INFO spark.HttpServer: Starting HTTP Server
>> > 14/01/15 23:37:57 INFO server.Server: jetty-7.6.8.v20121106
>> > 14/01/15 23:37:57 INFO server.AbstractConnector: Started
>> > SocketConnector@0.0.0.0:53685
>> > 14/01/15 23:37:57 INFO server.Server: jetty-7.6.8.v20121106
>> > 14/01/15 23:37:57 INFO handler.ContextHandler: started
>> > o.e.j.s.h.ContextHandler{/storage/rdd,null}
>> > 14/01/15 23:37:57 INFO handler.ContextHandler: started
>> > o.e.j.s.h.ContextHandler{/storage,null}
>> > 14/01/15 23:37:57 INFO handler.ContextHandler: started
>> > o.e.j.s.h.ContextHandler{/stages/stage,null}
>> > 14/01/15 23:37:57 INFO handler.ContextHandler: started
>> > o.e.j.s.h.ContextHandler{/stages/pool,null}
>> > 14/01/15 23:37:57 INFO handler.ContextHandler: started
>> > o.e.j.s.h.ContextHandler{/stages,null}
>> > 14/01/15 23:37:57 INFO handler.ContextHandler: started
>> > o.e.j.s.h.ContextHandler{/environment,null}
>> > 14/01/15 23:37:57 INFO handler.ContextHandler: started
>> > o.e.j.s.h.ContextHandler{/executors,null}
>> > 14/01/15 23:37:57 INFO handler.ContextHandler: started
>> > o.e.j.s.h.ContextHandler{/metrics/json,null}
>> > 14/01/15 23:37:57 INFO handler.ContextHandler: started
>> > o.e.j.s.h.ContextHandler{/static,null}
>> > 14/01/15 23:37:57 INFO handler.ContextHandler: started
>> > o.e.j.s.h.ContextHandler{/,null}
>> > 14/01/15 23:37:57 INFO server.AbstractConnector: Started
>> > SelectChannelConnector@0.0.0.0:4040
>> > 14/01/15 23:37:57 INFO ui.SparkUI: Started Spark Web UI at
>> > http://10.0.1.10:4040
>> > 2014-01-15 23:37:57.929 java[34819:1020b] Unable to load realm mapping
>> > info from SCDynamicStore
>> > 14/01/15 23:37:58 INFO storage.MemoryStore: ensureFreeSpace(35456)
>> > called with curMem=0, maxMem=1278030643
>> > 14/01/15 23:37:58 INFO storage.MemoryStore: Block broadcast_0 stored as
>> > values to memory (estimated size 34.6 KB, free 1218.8 MB)
>> > 14/01/15 23:37:58 WARN util.NativeCodeLoader: Unable to load
>> > native-hadoop library for your platform... using builtin-java classes where
>> > applicable
>> > 14/01/15 23:37:58 WARN snappy.LoadSnappy: Snappy native library not
>> > loaded
>> > 14/01/15 23:37:58 INFO mapred.FileInputFormat: Total input paths to
>> > process : 1
>> > 14/01/15 23:37:58 INFO spark.SparkContext: Starting job: foreach at
>> > HdfsTest.scala:30
>> > 14/01/15 23:37:58 INFO scheduler.DAGScheduler: Got job 0 (foreach at
>> > HdfsTest.scala:30) with 1 output partitions (allowLocal=false)
>> > 14/01/15 23:37:58 INFO scheduler.DAGScheduler: Final stage: Stage 0
>> > (foreach at HdfsTest.scala:30)
>> > 14/01/15 23:37:58 INFO scheduler.DAGScheduler: Parents of final stage:
>> > List()
>> > 14/01/15 23:37:58 INFO scheduler.DAGScheduler: Missing parents: List()
>> > 14/01/15 23:37:58 INFO scheduler.DAGScheduler: Submitting Stage 0
>> > (MappedRDD[2] at map at HdfsTest.scala:27), which has no missing parents
>> > 14/01/15 23:37:58 INFO scheduler.DAGScheduler: Submitting 1 missing
>> > tasks from Stage 0 (MappedRDD[2] at map at HdfsTest.scala:27)
>> > 14/01/15 23:37:58 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0
>> > with 1 tasks
>> > 14/01/15 23:37:58 INFO scheduler.TaskSetManager: Starting task 0.0:0 as
>> > TID 0 on executor localhost: localhost (PROCESS_LOCAL)
>> > 14/01/15 23:37:58 INFO scheduler.TaskSetManager: Serialized task 0.0:0
>> > as 1778 bytes in 5 ms
>> > 14/01/15 23:37:58 INFO executor.Executor: Running task ID 0
>> > 14/01/15 23:37:58 INFO storage.BlockManager: Found block broadcast_0
>> > locally
>> > 14/01/15 23:37:58 INFO spark.CacheManager: Partition rdd_2_0 not found,
>> > computing it
>> > 14/01/15 23:37:58 INFO rdd.HadoopRDD: Input split:
>> > file:/Users/acozzi/Documents/workspace/spark-samples/pom.xml:0+4092
>> > 14/01/15 23:37:58 INFO storage.MemoryStore: ensureFreeSpace(2853) called
>> > with curMem=35456, maxMem=1278030643
>> > 14/01/15 23:37:58 INFO storage.MemoryStore: Block rdd_2_0 stored as
>> > values to memory (estimated size 2.8 KB, free 1218.8 MB)
>> > 14/01/15 23:37:58 INFO storage.BlockManagerMasterActor$BlockManagerInfo:
>> > Added rdd_2_0 in memory on 10.0.1.10:53683 (size: 2.8 KB, free: 1218.8 MB)
>> > 14/01/15 23:37:58 INFO storage.BlockManagerMaster: Updated info of block
>> > rdd_2_0
>> > 14/01/15 23:37:58 INFO executor.Executor: Serialized size of result for
>> > 0 is 525
>> > 14/01/15 23:37:58 INFO executor.Executor: Sending result for 0 directly
>> > to driver
>> > 14/01/15 23:37:58 INFO executor.Executor: Finished task ID 0
>> > 14/01/15 23:37:58 INFO scheduler.TaskSetManager: Finished TID 0 in 61 ms
>> > on localhost (progress: 0/1)
>> > 14/01/15 23:37:58 INFO scheduler.DAGScheduler: Completed ResultTask(0,
>> > 0)
>> > 14/01/15 23:37:58 INFO scheduler.TaskSchedulerImpl: Remove TaskSet 0.0
>> > from pool
>> > 14/01/15 23:37:58 INFO scheduler.DAGScheduler: Stage 0 (foreach at
>> > HdfsTest.scala:30) finished in 0.071 s
>> > 14/01/15 23:37:58 INFO spark.SparkContext: Job finished: foreach at
>> > HdfsTest.scala:30, took 0.151199 s
>> > Iteration 1 took 189 ms
>> > [WARNING]
>> > java.lang.reflect.InvocationTargetException
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> > org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297)
>> > at java.lang.Thread.run(Thread.java:695)
>> > Caused by: java.lang.IncompatibleClassChangeError: Implementing class
>> > at java.lang.ClassLoader.defineClass1(Native Method)
>> > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:637)
>> > at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
>> > at
>> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
>> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
>> > at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
>> > at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
>> > at java.security.AccessController.doPrivileged(Native Method)
>> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>> > at java.lang.Class.forName0(Native Method)
>> > at java.lang.Class.forName(Class.java:171)
>> > at
>> > org.apache.hadoop.mapred.SparkHadoopMapRedUtil$class.firstAvailableClass(SparkHadoopMapRedUtil.scala:48)
>> > at
>> > org.apache.hadoop.mapred.SparkHadoopMapRedUtil$class.newJobContext(SparkHadoopMapRedUtil.scala:23)
>> > at
>> > org.apache.hadoop.mapred.SparkHadoopWriter.newJobContext(SparkHadoopWriter.scala:40)
>> > at
>> > org.apache.hadoop.mapred.SparkHadoopWriter.getJobContext(SparkHadoopWriter.scala:149)
>> > at
>> > org.apache.hadoop.mapred.SparkHadoopWriter.preSetup(SparkHadoopWriter.scala:64)
>> > at
>> > org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:713)
>> > at
>> > org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:686)
>> > at
>> > org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:572)
>> > at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:894)
>> > at
>> > org.apache.spark.examples.HdfsTest$$anonfun$main$1.apply$mcVI$sp(HdfsTest.scala:34)
>> > at
>> > scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:142)
>> > at org.apache.spark.examples.HdfsTest$.main(HdfsTest.scala:28)
>> > at org.apache.spark.examples.HdfsTest.main(HdfsTest.scala)
>> > ... 6 more
>> > [INFO]
>> > ------------------------------------------------------------------------
>> > [INFO] BUILD FAILURE
>> > [INFO]
>> > ------------------------------------------------------------------------
>> > [INFO] Total time: 3.224s
>> > [INFO] Finished at: Wed Jan 15 23:37:58 PST 2014
>> > [INFO] Final Memory: 12M/81M
>> > [INFO]
>> > ------------------------------------------------------------------------
>> > [ERROR] Failed to execute goal
>> > org.codehaus.mojo:exec-maven-plugin:1.2.1:java (default-cli) on project
>> > spark-samples: An exception occured while executing the Java class. null:
>> > InvocationTargetException: Implementing class -> [Help 1]
>> >
>> >
>> > Alex Cozzi
>> > alexco...@gmail.com
>> > On Jan 15, 2014, at 5:48 PM, Patrick Wendell <pwend...@gmail.com> wrote:
>> >
>> >> Please vote on releasing the following candidate as Apache Spark
>> >> (incubating) version 0.9.0.
>> >>
>> >> A draft of the release notes along with the changes file is attached
>> >> to this e-mail.
>> >>
>> >> The tag to be voted on is v0.9.0-incubating (commit 7348893):
>> >>
>> >> https://git-wip-us.apache.org/repos/asf?p=incubator-spark.git;a=commit;h=7348893f0edd96dacce2f00970db1976266f7008
>> >>
>> >> The release files, including signatures, digests, etc can be found at:
>> >> http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc1/
>> >>
>> >> Release artifacts are signed with the following key:
>> >> https://people.apache.org/keys/committer/pwendell.asc
>> >>
>> >> The staging repository for this release can be found at:
>> >> https://repository.apache.org/content/repositories/orgapachespark-1001/
>> >>
>> >> The documentation corresponding to this release can be found at:
>> >> http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc1-docs/
>> >>
>> >> Please vote on releasing this package as Apache Spark 0.9.0-incubating!
>> >>
>> >> The vote is open until Sunday, January 19, at 02:00 UTC
>> >> and passes if a majority of at least 3 +1 PPMC votes are cast.
>> >>
>> >> [ ] +1 Release this package as Apache Spark 0.9.0-incubating
>> >> [ ] -1 Do not release this package because ...
>> >>
>> >> To learn more about Apache Spark, please see
>> >> http://spark.incubator.apache.org/
>> >
>>
>
