Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3737#issuecomment-67604433
The deprecation warning idea was good; it turns out that there's a pending
deprecation which will change the semantics of one of the methods we use, so I
upgraded the code according to the documentation's suggestion.
For obtaining code coverage metrics, I applied the following change to the
Bash script:
```diff
diff --git a/ec2/spark-ec2 b/ec2/spark-ec2
index 3abd3f3..4e3fc69 100755
--- a/ec2/spark-ec2
+++ b/ec2/spark-ec2
@@ -22,4 +22,6 @@
#+ the underlying Python script.
SPARK_EC2_DIR="$(dirname $0)"
-python -Wdefault "${SPARK_EC2_DIR}/spark_ec2.py" "$@"
+
+export PYTHONWARNINGS="default"
+coverage run -a "${SPARK_EC2_DIR}/spark_ec2.py" "$@"
```
The `-a` option tells `coverage` to accumulate information across multiple
runs. I performed an iterative process where I interactively ran `spark-ec2`,
used `coverage html` to generate a report, then went back and ran more commands
to exercise the code areas that I missed.
With the workloads that I ran (launching spot clusters, stopping and
starting a cluster, destroying / creating security groups, logging in,
canceling spot instance requests), I got to 80% line coverage; most of the
lines that I missed were error-handling code.
Here's a link to an ASCII coverage report, produced with `coverage annotate
spark_ec2.py`: https://gist.github.com/JoshRosen/c09a742805bae3503185
According to the docs:
> Usage: coverage annotate [options] [modules]
> Make annotated copies of the given files, marking statements that are
executed
> with > and statements that are missed with !.
As you can see, the coverage is pretty good. Therefore, I'd be comfortable
merging this PR now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
