[
https://issues.apache.org/jira/browse/SPARK-11155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14980789#comment-14980789
]
Imran Rashid commented on SPARK-11155:
--------------------------------------
Hi [~iamshrek],
First start by reading
https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark#ContributingtoSpark-PreparingtoContributeCodeChanges
(especialy the section on "Pull Requests") and
https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools.
Of course everyone has their own favorite development environment / techniques,
but I'll tell you what I do. I've found debugging from within IDEs to not
work very well for spark (or scala codebases in general), so I just don't
bother anymore. (But I know others do debug from within intellij, so its not
impossible.) Instead, I just navigate through code in Intellij, and I use sbt,
unit tests, & printlns to test out code.
1) run {{build/sbt -Pyarn -Phadoop-2.4 -Dhadoop.version=2.5.0 -Pscala-2.10}}.
This will start an sbt shell
2) inside the sbt shell, run {{project core}} (since you only care about the
"spark-core" project)
3) run {{~test:compile}}. This will compile all the main & test code for the
"spark-core" project and its dependencies. Also it'll watch all the src files
-- anytime you change them it'll *incrementally* recompile. (the "~" prefix
makes sbt run the command in the incremental watch mode.)
4) leave that sbt-shell open, and then go to Intellij and do some coding. Eg.,
try adding some garbage syntax, and then you'll see the sbt shell recompile and
tell you about the compile error (and it should recompile quickly this time).
5) you can run some tests with {{~test-only <some pattern>}}. In your case,
you probably want {{~test-only *.HistoryServerSuite}}. As before, it'll run in
watch mode, but this time it'll rerun the tests as you change your code. The
tests you are most interested in are these:
https://github.com/apache/spark/blob/master/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala#L102
which reference expected results here:
https://github.com/apache/spark/tree/master/core/src/test/resources/HistoryServerExpectations
While debugging, you can add printlns, which will show up in the sbt shell, or
you can also add logging statements, which will appear in
core/target/unit-tests.log (along with all the existing logging statements).
6) before submitting your PR, run {{scalastyle}} and {{test:scalastyle}} from
within sbt (or just run {{dev/scalastyle}} from bash). that'll help you track
down style violations locally. (jenkins would do this for you, but its a lot
faster if you fix them locally -- that said, I often forget to do this myself.)
7) re-read the wiki guidelines, then submit your PR. Jenkins will then run the
full set of tests for you, and reviewers will comment.
For the HistoryServer in particular, you can also just run it locally, navigate
to some endpoints in your browser and see what happens.
hope this helps!
> Stage summary json should include stage duration
> -------------------------------------------------
>
> Key: SPARK-11155
> URL: https://issues.apache.org/jira/browse/SPARK-11155
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
> Reporter: Imran Rashid
> Assignee: Xin Ren
> Priority: Minor
> Labels: Starter
>
> The json endpoint for stages doesn't include information on the stage
> duration that is present in the UI. This looks like a simple oversight, they
> should be included. eg., the metrics should be included at
> {{api/v1/applications/<appId>/stages}}. The missing metrics are
> {{submissionTime}} and {{completionTime}} (and whatever other metrics come
> out of the discussion on SPARK-10930)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
