Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/926#issuecomment-44899788
Hey Sean,
I'm still a little confused about what it is you're doing. What is the
javadoc you refer to? I've looked at a few classes in org.apache.spark.examples
and I don't really see a lot of comments.
If you're using spark-submit or run-example, you shouldn't be running into
this issue ("spark.master" not set) at all. You'd only run into it if using
spark-class directly.
In 0.9 the examples (or at least a few of them) used to take the master as
the first argument. In 1.0.0 it seems the approach taken was to just ignore
that argument since it's provided automatically by spark-submit. Your command
line is "backwards compatible" although that one argument is just ignored. I
don't know how important that is for examples, though - I'd rather have the
examples as an example of how to write an app, and having that old argument
there kinda defeats that purpose.
So, if there are docs telling people to run the examples directly with java
or spark-class, we should fix those to use run-example / spark-submit instead.
Perhaps a future approach could be to have a "SparkApp" base class that
calls a "run(SparkContext)" method so that initializing the context clearly
becomes the job of the framework. But that approach has all sorts of other
issues.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
