Owen so...@cloudera.com
Cc: dev dev@spark.apache.org, jay vyas jayunit100.apa...@gmail.com,
Paolo Platter
paolo.plat...@agilelab.it, Nicholas Chammas
nicholas.cham...@gmail.com, Will Benton wi...@redhat.com
Sent: Wednesday, January 21, 2015 2:09:35 AM
Subject: Re: Standardized Spark dev
If the goal is a reproducible test environment then I think that is what
Jenkins is. Granted you can only ask it for a test. But presumably you get
the same result if you start from the same VM image as Jenkins and run the
same steps.
But the issue is when users can't reproduce Jenkins
Great suggestion.
On Jan 20, 2015 7:14 PM, Nicholas Chammas nicholas.cham...@gmail.com
wrote:
What do y'all think of creating a standardized Spark development
environment, perhaps encoded as a Vagrantfile, and publishing it under
`dev/`?
The goal would be to make it easier for new developers
How many profiles (hadoop / hive /scala) would this development environment
support ?
Cheers
On Tue, Jan 20, 2015 at 4:13 PM, Nicholas Chammas
nicholas.cham...@gmail.com wrote:
What do y'all think of creating a standardized Spark development
environment, perhaps encoded as a Vagrantfile, and
How many profiles (hadoop / hive /scala) would this development environment
support ?
As many as we want. We probably want to cover a good chunk of the build
matrix https://issues.apache.org/jira/browse/SPARK-2004 that Spark
officially supports.
What does this provide, concretely?
It provides a
I can comment on both... hi will and nate :)
1) Will's Dockerfile solution is the most simple direct solution to the
dev environment question : its a efficient way to build and develop spark
environments for dev/test.. It would be cool to put that Dockerfile
(and/or maybe a shell script
if any folks are interested.
Nate
-Original Message-
From: Sean Owen [mailto:so...@cloudera.com]
Sent: Tuesday, January 20, 2015 5:09 PM
To: Nicholas Chammas
Cc: dev
Subject: Re: Standardized Spark dev environment
My concern would mostly be maintenance. It adds to an already very complex
Hey Nick,
I did something similar with a Docker image last summer; I haven't updated the
images to cache the dependencies for the current Spark master, but it would be
trivial to do so:
http://chapeau.freevariable.com/2014/08/jvm-test-docker.html
best,
wb
- Original Message -
: Will Bentonmailto:wi...@redhat.com; Spark dev
listmailto:dev@spark.apache.org
Oggetto: Re: Standardized Spark dev environment
I can comment on both... hi will and nate :)
1) Will's Dockerfile solution is the most simple direct solution to the
dev environment question : its a efficient way
