Re: EC2 clusters ready in launch time + 30 seconds

Hey All, Just a couple notes. I recently posted a shell script for creating the AMI's from a clean Amazon Linux AMI. https://github.com/mesos/spark-ec2/blob/v3/create_image.sh I think I will update the AMI's soon to get the most recent security updates. For spark-ec2's purpose this is probably s

Re: Fwd: Breeze Library usage in Spark

You dont have to include breeze jar which is already in spark assembly jar. For native one, its optional. Sent from my Google Nexus 5 On Oct 3, 2014 8:04 PM, "Priya Ch" wrote: > > > > yes. I have included breeze-0.9 in build.sbt file. I ll change this to > 0.7. Apart from this, do we need to inc

Re: EC2 clusters ready in launch time + 30 seconds

FYI: There is an existing issue -- SPARK-3314 -- about scripting the creation of Spark AMIs. With Packer, it looks like we may be able to script the creation of multiple image types (VMWare, GCE, AMI, Docker, etc...) at once from a single Packer t

Parquet schema migrations

Wondering if anyone has thoughts on a path forward for parquet schema migrations, especially for people (like us) that are using raw parquet files rather than Hive. So far we've gotten away with reading old files, converting, and writing to new directories, but that obviously becomes problematic a

Fwd: Breeze Library usage in Spark

yes. I have included breeze-0.9 in build.sbt file. I ll change this to 0.7. Apart from this, do we need to include breeze jars explicitly in the spark context as sc.addJar() ? and what about the dependencies netlib-native_ref-linux- x86_64-1.1-natives.jar, netlib-native_system-linux-x86_64-1.

Re: emergency jenkins restart -- massive security patch released

update complete. i'm retriggering builds now. On Fri, Oct 3, 2014 at 10:51 AM, shane knapp wrote: > > https://wiki.jenkins-ci.org/display/SECURITY/Jenkins+Security+Advisory+2014-10-01 > > there's some pretty big stuff that's been identified and we need to get > this upgraded asap. > > i'll be k

emergency jenkins restart -- massive security patch released

https://wiki.jenkins-ci.org/display/SECURITY/Jenkins+Security+Advisory+2014-10-01 there's some pretty big stuff that's been identified and we need to get this upgraded asap. i'll be killing off what's currently running, and will retrigger them all once we're done. sorry for the inconvenience. s

Re: Breeze Library usage in Spark

yeah, breeze.storage.Zero was introduced in either 0.8 or 0.9. On Fri, Oct 3, 2014 at 9:45 AM, Xiangrui Meng wrote: > Did you add a different version of breeze to the classpath? In Spark > 1.0, we use breeze 0.7, and in Spark 1.1 we use 0.9. If the breeze > version you used is different from the

Re: HiveContext: cache table not supported for partitioned table?

Thanks for your explanation. From: Cheng Lian mailto:lian.cs@gmail.com>> Date: Thursday, October 2, 2014 at 8:01 PM To: Du Li mailto:l...@yahoo-inc.com.INVALID>>, "dev@spark.apache.org" mailto:dev@spark.apache.org>> Cc: "u...@spark.apache.org

Re: Breeze Library usage in Spark

Did you add a different version of breeze to the classpath? In Spark 1.0, we use breeze 0.7, and in Spark 1.1 we use 0.9. If the breeze version you used is different from the one comes with Spark, you might see class not found. -Xiangrui On Fri, Oct 3, 2014 at 4:22 AM, Priya Ch wrote: > Hi Team,

Breeze Library usage in Spark

Hi Team, When I am trying to use DenseMatrix of breeze library in spark, its throwing me the following error: java.lang.noclassdeffounderror: breeze/storage/Zero Can someone help me on this ? Thanks, Padma Ch