Re: spark-shell 1.5 doesn't seem to work in local mode

Thanks guys. I do have HADOOP_INSTALL set, but Spark 1.4.1 did not seem to mind. Seems like there's a difference in behavior between 1.5.0 and 1.4.1 for some reason. To the best of my knowledge, I just downloaded each tgz and untarred them in /opt I adjusted my PATH to point to one or the other,

Re: spark-shell 1.5 doesn't seem to work in local mode

It sounds a lot like you have some local Hadoop config pointing to a cluster, and you're picking that up when you run the shell. Look for HADOOP_* env variables and clear them, and use --master local[*] On Sat, Sep 19, 2015 at 5:14 PM, Madhu wrote: > I downloaded

Re: RDD API patterns

Hi Sim, I understand that what you propose is defining a trait SparkIterable (and also PairSparkIterable for RDDs of pairs) that encapsulates the methods in RDDs, and then program using that trait instead of RDD. That is similar to programming using scala.collection.GenSeq to abstract from using

SparkR installation not working

Hi All, I've built spark 1.5.0 with hadoop 2.6 with a fresh download : build/mvn -Phadoop-2.6 -Dhadoop.version=2.6.0 -DskipTests clean package I try to run SparkR it launches the normal R without the spark addons: ./bin/sparkR --master local[*] Picked up JAVA_TOOL_OPTIONS:

Re: SparkR installation not working

Looks like you didn't specify sparkr profile when building. Cheers On Sat, Sep 19, 2015 at 12:30 PM, Devl Devel wrote: > Hi All, > > I've built spark 1.5.0 with hadoop 2.6 with a fresh download : > > build/mvn -Phadoop-2.6 -Dhadoop.version=2.6.0 -DskipTests clean

Re: spark-shell 1.5 doesn't seem to work in local mode

It does not matter whether you start your spark with local or other mode. If you have hdfs-site.xml somewhere and spark configuration pointing to that config, you will read/write to HDFS. Thanks. Zhan Zhang From: Madhu Sent: Saturday,

Re: BUILD SYSTEM: fire and power event at UC berkeley's IST colo, jenkins offline

Great! Jon / Shane: Thanks for handling this. On Saturday, September 19, 2015, shane knapp wrote: > we're up and building! time for breakfast... :) > > https://amplab.cs.berkeley.edu/jenkins/ > > On Sat, Sep 19, 2015 at 7:35 AM, shane knapp

Re: RDD API patterns

Juan, I wouldn't go as far as suggesting we switch from programming using RDDs to using SparkIterable. For example, all methods involving context, jobs or partitions should only be part of the RDD API and not part of SparkIterable. That said, the Spark community would benefit from a consistent set

Re: spark-shell 1.5 doesn't seem to work in local mode

Maybe you have a hdfs-site.xml lying around somewhere? On Sat, Sep 19, 2015 at 9:14 AM, Madhu wrote: > I downloaded spark-1.5.0-bin-hadoop2.6.tgz recently and installed on > CentOS. > All my local Spark code works fine locally. > > For some odd reason, spark-shell doesn't work

BUILD SYSTEM: fire and power event at UC berkeley's IST colo, jenkins offline

TL; DR: jenkins is currently down and will probably not be brought back up until monday morning. a machine caught fire in the colo this evening, and this tripped the halon, and now IST is overheating... it looks like it may have been one of our servers that popped and caused the event, and

Re: BUILD SYSTEM: fire and power event at UC berkeley's IST colo, jenkins offline

> On 19 Sep 2015, at 08:28, shane knapp wrote: > > TL; DR: jenkins is currently down and will probably not be brought > back up until monday morning. > > a machine caught fire in the colo this evening, and this tripped the > halon, and now IST is overheating... it looks

[SparkSQL]How does spark handle a parquet file in parallel?

Hi all, I'm using Spark (1.4.1) + Hive (0.13.1), I found that a large number of network IO appeared when query a parquet table *with only one part file* use SparkSQL. The SQL is: SELECT concat(year(fkbb5855f0), "-", month(fkbb5855f0), "-", day(fkbb5855f0), " 00:00:00"),COUNT(fk919b1d80) FROM

Re: BUILD SYSTEM: fire and power event at UC berkeley's IST colo, jenkins offline

we're up and building! time for breakfast... :) https://amplab.cs.berkeley.edu/jenkins/ On Sat, Sep 19, 2015 at 7:35 AM, shane knapp wrote: > it was definitely one of our servers... we have no ETA on when > jenkins will be back online. we will need to inspect the rack

Re: AMP JENKINS - unplanned outage at 1845, ongoing

we're up and building! time for breakfast... :) https://amplab.cs.berkeley.edu/jenkins/ On Fri, Sep 18, 2015 at 9:30 PM, jon kuroda wrote: > Starting tonight at about 6:45PM, the AMP Jenkins instance, which is > hosted at the main UC Berkeley Campus Datacenter, went

spark-shell 1.5 doesn't seem to work in local mode

I downloaded spark-1.5.0-bin-hadoop2.6.tgz recently and installed on CentOS. All my local Spark code works fine locally. For some odd reason, spark-shell doesn't work in local mode. It looks like it want's to connect to HDFS, even if I use --master local or specify local mode in the conf. Even