Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25641491
--- Diff:
core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
---
@@ -89,16 +105,18 @@ class FsHistoryProviderSuite extends
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25647732
--- Diff:
core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
---
@@ -89,16 +105,18 @@ class FsHistoryProviderSuite extends
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25647694
--- Diff:
core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
---
@@ -89,16 +105,18 @@ class FsHistoryProviderSuite extends
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76848440
LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76846594
[Test build #28198 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28198/consoleFull)
for PR 4821 at commit
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25647830
--- Diff:
core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
---
@@ -89,16 +105,18 @@ class FsHistoryProviderSuite extends
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76831455
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25649521
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -574,6 +583,11 @@ private[spark] object JsonProtocol {
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76831439
[Test build #28182 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28182/consoleFull)
for PR 4821 at commit
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25648384
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -574,6 +583,11 @@ private[spark] object JsonProtocol {
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76850938
[Test build #28201 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28201/consoleFull)
for PR 4821 at commit
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25648636
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -574,6 +583,11 @@ private[spark] object JsonProtocol {
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76860619
[Test build #28198 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28198/consoleFull)
for PR 4821 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76860628
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76864451
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76855580
Okay I took a close look through this and it LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76864438
[Test build #28201 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28201/consoleFull)
for PR 4821 at commit
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/4821
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76815711
By the way I gained first-hand experience of the flakiness of
`FsHistoryProviderSuite` from this patch. The tests passed locally but failed
here, and I suspect that
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76815718
[Test build #28182 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28182/consoleFull)
for PR 4821 at commit
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76821354
flakiness of FsHistoryProviderSuite
Do you have any theories for why this would be related to the log's
modification time? The only place where it's used in
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76673355
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76673346
[Test build #28159 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28159/consoleFull)
for PR 4821 at commit
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25625032
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -279,52 +253,17 @@ private[spark] object EventLoggingListener
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25624317
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -320,30 +320,24 @@ private[history] class FsHistoryProvider(conf:
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25624492
--- Diff: core/src/main/scala/org/apache/spark/io/CompressionCodec.scala ---
@@ -71,6 +74,21 @@ private[spark] object CompressionCodec {
sConsider
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25624653
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -111,19 +119,17 @@ private[spark] class EventLoggingListener(
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25625518
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -111,19 +119,17 @@ private[spark] class EventLoggingListener(
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25625161
--- Diff:
core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
---
@@ -45,18 +44,26 @@ class FsHistoryProviderSuite extends
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25625447
--- Diff:
core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
---
@@ -45,18 +44,26 @@ class FsHistoryProviderSuite extends
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25626186
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -111,19 +119,17 @@ private[spark] class EventLoggingListener(
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25626361
--- Diff:
core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
---
@@ -45,18 +44,26 @@ class FsHistoryProviderSuite extends
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76782633
@vanzin so just to be clear - you are anticipating a future case where we
need to read the version to correctly parse the logs? Is that the argument for
it? I am only
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25628232
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -111,19 +119,17 @@ private[spark] class EventLoggingListener(
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25628566
--- Diff: core/src/main/scala/org/apache/spark/io/CompressionCodec.scala ---
@@ -71,6 +74,21 @@ private[spark] object CompressionCodec {
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76787929
[Test build #28180 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28180/consoleFull)
for PR 4821 at commit
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25628525
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -111,19 +119,17 @@ private[spark] class EventLoggingListener(
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76780546
@pwendell the header is needed because it contains potentially useful
information for the code parsing the logs. For example, now it contains the
Spark version, which
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76785685
@pwendell yes, that's what I'm trying to say. Even today, logs generated by
Spark 1.0, 1.1 and 1.2 (and soon 1.3) are not exactly the same - events have
different
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25573801
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -217,57 +223,67 @@ private[spark] object EventLoggingListener
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25573931
--- Diff:
core/src/main/scala/org/apache/spark/deploy/ApplicationDescription.scala ---
@@ -23,7 +23,9 @@ private[spark] class ApplicationDescription(
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25573776
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -217,57 +223,67 @@ private[spark] object EventLoggingListener
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25574126
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala ---
@@ -99,6 +99,12 @@ case class SparkListenerExecutorRemoved(time: Long,
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76641955
I took a pass on this with some feedback. Overall, it would be good to
really minimize the scope of the changes since this is so late in the game.
There is some
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25579333
--- Diff:
core/src/main/scala/org/apache/spark/deploy/ApplicationDescription.scala ---
@@ -23,7 +23,9 @@ private[spark] class ApplicationDescription(
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76663276
[Test build #28157 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28157/consoleFull)
for PR 4821 at commit
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25579888
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -322,28 +322,22 @@ private[history] class
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25579899
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -322,28 +322,22 @@ private[history] class
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25579955
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -322,28 +322,22 @@ private[history] class
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25579983
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala ---
@@ -39,10 +39,9 @@ private[spark] class ReplayListenerBus extends
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76663334
As of the latest commit I have removed the Spark version and the metadata.
Please have a look.
---
If your project is set up for it, you can reply to this email and
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76663956
[Test build #28158 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28158/consoleFull)
for PR 4821 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-7392
[Test build #28159 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28159/consoleFull)
for PR 4821 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76668879
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76668874
[Test build #28157 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28157/consoleFull)
for PR 4821 at commit
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25577271
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -217,57 +223,67 @@ private[spark] object EventLoggingListener
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25577300
--- Diff:
core/src/main/scala/org/apache/spark/deploy/ApplicationDescription.scala ---
@@ -23,7 +23,9 @@ private[spark] class ApplicationDescription(
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76669704
[Test build #28158 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28158/consoleFull)
for PR 4821 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76669712
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76505881
[Test build #28102 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28102/consoleFull)
for PR 4821 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76505885
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25552458
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -110,17 +117,12 @@ private[spark] class EventLoggingListener(
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25552104
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -217,53 +219,60 @@ private[spark] object EventLoggingListener
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76507612
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76507606
[Test build #28103 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28103/consoleFull)
for PR 4821 at commit
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25552366
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -217,53 +219,60 @@ private[spark] object EventLoggingListener
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76503962
[Test build #28103 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28103/consoleFull)
for PR 4821 at commit
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25552048
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -217,53 +219,60 @@ private[spark] object EventLoggingListener
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76502888
[Test build #28102 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28102/consoleFull)
for PR 4821 at commit
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25552069
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -202,11 +204,11 @@ private[spark] object EventLoggingListener
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76494032
Tests will fail to compile. I'm fixing this in the mean time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25549140
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -110,17 +117,12 @@ private[spark] class EventLoggingListener(
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25549234
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -202,11 +204,11 @@ private[spark] object EventLoggingListener
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4821#discussion_r25549402
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -217,53 +219,60 @@ private[spark] object EventLoggingListener
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76496914
Looks reasonable, will wait for tests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76497550
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76497543
[Test build #28098 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28098/consoleFull)
for PR 4821 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4821#issuecomment-76493745
[Test build #28098 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28098/consoleFull)
for PR 4821 at commit
78 matches
Mail list logo
