GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/4633
[SPARK-1600] Refactor FileInputStream tests to remove Thread.sleep() calls
and SystemClock usage (branch-1.2 backport)
(This PR backports #3801 into `branch-1.2` (1.2.2))
This patch
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4633#issuecomment-74577468
[Test build #27577 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27577/consoleFull)
for PR 4633 at commit
Github user JoshRosen closed the pull request at:
https://github.com/apache/spark/pull/4633
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4633#issuecomment-74589724
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4633#issuecomment-74589708
[Test build #27577 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27577/consoleFull)
for PR 4633 at commit
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/4633#issuecomment-74591552
Since this backport passes tests, I'm going to merge it to `branch-1.2`
(1.2.2) to see if this reduces flakiness there.
---
If your project is set up for it, you can
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/3801#issuecomment-68838737
I am merging this. Thanks @JoshRosen for this humongous effort!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/3801
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3801#issuecomment-68688402
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3801#issuecomment-68688395
[Test build #25052 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25052/consoleFull)
for PR 3801 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3801#issuecomment-68683294
[Test build #25052 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25052/consoleFull)
for PR 3801 at commit
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3801#issuecomment-68684233
Pushed some commits addressing most of the feedback, but I'm still
struggling to remove that last `Thread.sleep(1000)`. I think that the problem
here is that the
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3801#discussion_r22491068
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -319,102 +318,141 @@ class CheckpointSuite extends TestSuiteBase
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3801#issuecomment-68813139
[Test build #25071 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25071/consoleFull)
for PR 3801 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3801#issuecomment-68813146
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3801#issuecomment-68806849
[Test build #25071 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25071/consoleFull)
for PR 3801 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3801#issuecomment-68824923
[Test build #25086 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25086/consoleFull)
for PR 3801 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3801#issuecomment-68824928
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3801#issuecomment-68821073
[Test build #25086 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25086/consoleFull)
for PR 3801 at commit
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/3801#issuecomment-68756791
Right, that is probably the case. The checkpointing is asynchronous, and
there isnt any callback hook for that, which we can use to wait on. We could
wait for the expected
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3801#discussion_r22449971
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -281,102 +279,130 @@ class CheckpointSuite extends TestSuiteBase
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3801#discussion_r22449114
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala ---
@@ -104,6 +106,77 @@ class TestOutputStreamWithPartitions[T:
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3801#discussion_r22449487
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala ---
@@ -104,6 +106,77 @@ class TestOutputStreamWithPartitions[T:
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3801#discussion_r22447683
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -281,102 +279,130 @@ class CheckpointSuite extends TestSuiteBase
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3801#discussion_r22447710
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -281,102 +279,130 @@ class CheckpointSuite extends TestSuiteBase
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3801#discussion_r22447721
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -281,102 +279,130 @@ class CheckpointSuite extends TestSuiteBase
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3801#discussion_r22447733
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala ---
@@ -104,6 +106,77 @@ class TestOutputStreamWithPartitions[T:
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3801#discussion_r22447724
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -281,102 +279,130 @@ class CheckpointSuite extends TestSuiteBase
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/3801#discussion_r22448279
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -281,102 +279,130 @@ class CheckpointSuite extends TestSuiteBase {
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3801#discussion_r22449238
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala ---
@@ -104,6 +106,77 @@ class TestOutputStreamWithPartitions[T:
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3801#discussion_r22448937
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -281,102 +279,130 @@ class CheckpointSuite extends TestSuiteBase
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/3801#discussion_r22436035
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -281,102 +279,130 @@ class CheckpointSuite extends TestSuiteBase {
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/3801#discussion_r22436087
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -281,102 +279,130 @@ class CheckpointSuite extends TestSuiteBase {
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/3801#discussion_r22436093
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -281,102 +279,130 @@ class CheckpointSuite extends TestSuiteBase {
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/3801#discussion_r22436099
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -281,102 +279,130 @@ class CheckpointSuite extends TestSuiteBase {
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/3801#discussion_r22436103
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -281,102 +279,130 @@ class CheckpointSuite extends TestSuiteBase {
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/3801#discussion_r22436115
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala ---
@@ -104,6 +106,77 @@ class TestOutputStreamWithPartitions[T:
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/3801#discussion_r22436357
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -281,102 +279,130 @@ class CheckpointSuite extends TestSuiteBase {
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/3801#issuecomment-68619619
@JoshRosen I did a pass and I had some comments. Overall it looks pretty
good. Addressing those comments should make it good enough.
Could you tell me how much is
GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/3801
[SPARK-1600] Refactor FileInputStream tests to remove Thread.sleep() calls
and SystemClock usage
This PR refactors Spark Streaming's FileInputStream tests to remove uses of
Thread.sleep() and
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3801#issuecomment-68097807
These changes are split off from #3687, a larger PR of mine which tried to
remove all uses of Thread.sleep() in the streaming tests.
It may look like there are
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3801#discussion_r22271205
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -281,102 +279,130 @@ class CheckpointSuite extends TestSuiteBase
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3801#issuecomment-68097905
[Test build #24820 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24820/consoleFull)
for PR 3801 at commit
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3801#discussion_r22271345
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -281,102 +279,130 @@ class CheckpointSuite extends TestSuiteBase
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3801#discussion_r22271355
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -281,102 +279,130 @@ class CheckpointSuite extends TestSuiteBase
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3801#issuecomment-68100834
[Test build #24820 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24820/consoleFull)
for PR 3801 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3801#issuecomment-68100836
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
47 matches
Mail list logo
