The instance type is "ti.micro" with size as 7.9GB out of which 4.3GB is still available. For running spark (and later on hadoop), should i use a Storage Optimized instance or it can work on this as well?
On Nov 20, 2013, at 11:39 AM, Prashant Sharma <[email protected]> wrote: > What is the instance type ?. Use an instance with atleast 4Gb+ RAM. I don't > think it is possible to build on less than that. Other option would be to use > prebuilt binary. > > > On Wed, Nov 20, 2013 at 8:56 PM, Pankhuri Gupta <[email protected]> wrote: > Hi, > I am new to Spark and Scala. As a part of one of my projects, I am > trying to build and locally publish spark-0.8.0-incubating on an Amazon ec2 > cluster. > After setting up all the java class paths and options, when I run : > ** sbt/sbt compile , OR > ** sbt/sbt assembly, OR > ** sbt/sbt publish-local > The command runs for some time (approx 10 mins) and after that the java > command simply gets killed. No error message is thrown out. Here is a > small snapshot of the messages: > > [info] Updating {file:/home/ec2-user/spark-0.8.0-incubating/}bagel... > [info] Resolving cglib#cglib-nodep;2.2.2 … > [info] Done updating. > [info] Compiling 258 Scala sources and 16 Java sources to > /home/ec2-user/spark-0.8.0-incubating/core/target/scala-2.9.3/classes... > sbt/sbt: line 30: 21454 Killed java -Xmx1200m -XX:MaxPermSize=350m > -XX:ReservedCodeCacheSize=256m $EXTRA_ARGS $SBT_OPTS -jar > "$SPARK_HOME"/sbt/sbt-launch-*.jar "$@" > > When I run the jstat, I get the following output: > Timestamp S0 S1 E O P YGC YGCT FGC FGCT GCT LGCC GCC > 257.3 100.00 0.00 50.66 82.97 99.59 156 5.163 10 12.076 17.239 unknown > GCCause No GC > 365.2 0.00 100.00 53.13 88.96 99.94 157 7.563 10 12.076 19.639 unknown > GCCause No GC > 386.6 0.00 0.00 1.97 60.00 99.51 157 7.563 11 25.281 32.844 Permanent > Generation Full No GC > 407.9 0.00 0.00 29.53 60.00 99.97 157 7.563 11 25.281 32.844 Permanent > Generation Full No GC > 578.8 64.82 0.00 41.90 77.75 99.68 162 10.896 11 25.281 36.178 unknown > GCCause No GC > 600.1 64.82 0.00 91.90 77.75 99.72 162 10.896 11 25.281 36.178 unknown > GCCause No GC > 664.2 77.92 70.32 100.00 99.94 99.71 168 12.451 11 25.281 37.732 > unknown GCCause Allocation Failure > > I changed the memory limits as : -XX:MaxPermSize=720m and > -XX:ReservedCodeCacheSize=512m, but still the problem persists. > I am not able to figure out the reason why the command is getting > killed. Please let me know if I need to do some other checks. I read through > many links on google and spark site as well but was not able to get any > insight into this problem. > > I am using the following- > Java version: 6 > Jvm : 1.6.0-openjdk.x86_64 > Scala Version : 2.9.3 installed > > Any help would be deeply appreciated. > > Thanks, > > > > -- > s
