Thanks for the info and good luck with 1.0.
Regards,
Art
On Fri, Apr 25, 2014 at 9:48 AM, Andrew Or and...@databricks.com wrote:
Hi Art,
First of all thanks a lot for your PRs. We are currently in the middle of
all the Spark 1.0 release so most of us are swamped with the more core
features. To answer your questions:
1. Neither. We welcome changes from developers for all components of Spark,
including the EC2 scripts. Once the release is out we will have more time
to review the many PRs that we missed on the ride.
2. We prefer to keep the EC2 scripts within Spark, at least for now.
Cheers,
Andrew
On Friday, April 25, 2014, Art Peel found...@gmail.com wrote:
I've been setting up Spark cluster on EC2 using the provided
ec2/spark_ec2.py script and am very happy I didn't have to write it from
scratch. Thanks for providing it.
There have been some issues, though, and I have had to make some
additions.
So far, they are all additions of command-line options. For example,
the
original script allows access from anywhere to the various ports. I've
added an option to specify what net/mask should be allowed to access
those
ports.
I've filed a couple of pull requests, but they are not going anywhere.
Given what I've seen of the traffic on this list, I don't feel that a
lot
of the developers are thinking about EC2 setup. I totally agree that it
is
not as important as improving the guts of Spark itself; nevertheless, I
feel that being able to run Spark on EC2 smartly and easily is valuable.
So, I have 2 questions for the committers:
1. Is ec2/spark_ec2.py something the committers
a. are not thinking about?
b. are planning to replace?
c. other
2. Should I just start a new project based on ec2/spark_ec2.py but
without
all the other stuff and make (and share) my changes there?
Regards,
Art
