Re: [PySpark]: reading arbitrary Hadoop InputFormats

2014-03-18 Thread Nick Pentreath
Hi Matei I'm afraid I haven't had enough time to focus on this as work has just been crazy. It's still something I want to get to a mergeable status.  Actually it was working fine it was just a bit rough and needs to be updated to HEAD. I'll absolutely try my utmost to get something rea

Re: Announcing the official Spark Job Server repo

2014-03-18 Thread Evan Chan
Matei, Maybe it's time to explore the spark-contrib idea again? Should I start a JIRA ticket? -Evan On Tue, Mar 18, 2014 at 4:04 PM, Matei Zaharia wrote: > Cool, glad to see this posted! I've added a link to it at > https://cwiki.apache.org/confluence/display/SPARK/Powered+By+Spark. > > Mat

Re: Announcing the official Spark Job Server repo

2014-03-18 Thread Evan Chan
Andy, Yeah, we've thought of deploying this on Marathon ourselves, but we're not sure how much Mesos we're going to use yet. (Indeed if you look at bin/server_start.sh, I think I set up the PORT environment var specifically for Marathon.)This is also why we have deploy scripts which package

Re: Announcing the official Spark Job Server repo

2014-03-18 Thread Matei Zaharia
Cool, glad to see this posted! I’ve added a link to it at https://cwiki.apache.org/confluence/display/SPARK/Powered+By+Spark. Matei On Mar 18, 2014, at 1:51 PM, Evan Chan wrote: > Dear Spark developers, > > Ooyala is happy to announce that we have pushed our official, Spark > 0.9.0 / Scala 2.

Re: Announcing the official Spark Job Server repo

2014-03-18 Thread andy petrella
tad! That's awesome. A quick question, does someone has insights regarding having such JobServers deployed using Marathon on Mesos? I'm thinking about an arch where Marathon would deploy and keep the Job Servers running along with part of the whole set of apps deployed on it regarding the res

Re: Announcing the official Spark Job Server repo

2014-03-18 Thread yao
+1 On Tue, Mar 18, 2014 at 3:39 PM, Henry Saputra wrote: > W00t! > > Thanks for releasing this, Evan. > > - Henry > > On Tue, Mar 18, 2014 at 1:51 PM, Evan Chan wrote: > > Dear Spark developers, > > > > Ooyala is happy to announce that we have pushed our official, Spark > > 0.9.0 / Scala 2.10-c

Re: Announcing the official Spark Job Server repo

2014-03-18 Thread Henry Saputra
W00t! Thanks for releasing this, Evan. - Henry On Tue, Mar 18, 2014 at 1:51 PM, Evan Chan wrote: > Dear Spark developers, > > Ooyala is happy to announce that we have pushed our official, Spark > 0.9.0 / Scala 2.10-compatible, job server as a github repo: > > https://github.com/ooyala/spark-job

Re: [PySpark]: reading arbitrary Hadoop InputFormats

2014-03-18 Thread Matei Zaharia
Hey Nick, I’m curious, have you been doing any further development on this? It would be good to get expanded InputFormat support in Spark 1.0. To start with we don’t have to do SequenceFiles in particular, we can do stuff like Avro (if it’s easy to read in Python) or some kind of WholeFileInputF

Announcing the official Spark Job Server repo

2014-03-18 Thread Evan Chan
Dear Spark developers, Ooyala is happy to announce that we have pushed our official, Spark 0.9.0 / Scala 2.10-compatible, job server as a github repo: https://github.com/ooyala/spark-jobserver Complete with unit tests, deploy scripts, and examples. The original PR (#222) on incubator-spark is n