Github user aarondav commented on the pull request:
https://github.com/apache/spark/pull/116#issuecomment-38129776
@berngp Thanks for the clarifications! Spark has historically leaned
towards providing a simpler, leaner API in order to be easier to learn and
develop. With this goal in mind, I think we should avoid the -X style options
for now until we get clear use-cases for each. I understand that this mechanism
is very convenient for testing out different options, but users can always
achieve this goal by writing some harness code like you have here, but I don't
think it should intersect with the tools we want all spark users to be using.
I like the inclusion of --master, the renaming of
--driver-memory/--executor-memory, and the reorganization of the script. I
think before merging, though, the following changes should be made:
- Removal of all options currently under "Other options" except --master
and --log-conf. Since --log-conf would be the only thing in "Other options" we
should probably just go ahead and merge it into "Basics" and rename "Basics" to
just "Options".
- Removal of DRY_RUN variable.
- Remove VERBOSE but allow us to log the base directory, Spark Master, and
REPL options if --log-conf is set.
- Revert --executor-cores to --cores. Otherwise this may confuse users,
since it's not at all related to --executor-memory (which sets per-executor
memory) while --executor-cores sets the total number of cores that the
spark-shell asks for from the scheduler.
- If no MASTER is defined, default to "local" to avoid a behavioral
regression.
I also like the inclusion of --jvm-debug for the same reason as --log-conf,
I think debugging options are useful. However, we can save that for a future
patch, I think.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
