Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154273968
**[Test build #1994 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1994/consoleFull)**
for PR 9477 at commit
[`b367daf`](https://gi
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/9477
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enab
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154270321
LGTM, so I'm going to merge this into master and 1.6. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154261887
**[Test build #1993 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1993/consoleFull)**
for PR 9477 at commit
[`b367daf`](https://gi
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154248203
**[Test build #1994 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1994/consoleFull)**
for PR 9477 at commit
[`b367daf`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154228478
**[Test build #1993 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1993/consoleFull)**
for PR 9477 at commit
[`b367daf`](https://git
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154228222
The failed test is not related, will re-run it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your pr
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154223721
**[Test build #1989 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1989/consoleFull)**
for PR 9477 at commit
[`b367daf`](https://gi
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154212667
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154212555
**[Test build #45127 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45127/consoleFull)**
for PR 9477 at commit
[`c35f512`](https://git
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154212662
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154210332
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154210333
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154210258
**[Test build #45125 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45125/consoleFull)**
for PR 9477 at commit
[`854a99f`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154188619
**[Test build #1989 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1989/consoleFull)**
for PR 9477 at commit
[`b367daf`](https://git
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154170426
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154170431
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154165565
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154165525
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154140207
**[Test build #45127 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45127/consoleFull)**
for PR 9477 at commit
[`c35f512`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154139271
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154139243
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154137615
**[Test build #45125 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45125/consoleFull)**
for PR 9477 at commit
[`854a99f`](https://gith
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9477#discussion_r44043737
--- Diff:
core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java
---
@@ -293,9 +292,10 @@ private void growPointerArrayI
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154135434
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154135406
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154134910
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154054818
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154054716
**[Test build #45110 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45110/consoleFull)**
for PR 9477 at commit
[`854a99f`](https://git
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154054815
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154048718
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154048664
**[Test build #45108 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45108/consoleFull)**
for PR 9477 at commit
[`b42e7db`](https://git
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-154048716
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153996532
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153996536
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153995771
**[Test build #45096 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45096/consoleFull)**
for PR 9477 at commit
[`89319e0`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153992590
**[Test build #45110 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45110/consoleFull)**
for PR 9477 at commit
[`854a99f`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153991140
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153991110
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153990700
@JoshRosen I should had addressed your comments, will merge this once pass
the tests.
---
If your project is set up for it, you can reply to this email and have your
rep
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153987965
**[Test build #45108 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45108/consoleFull)**
for PR 9477 at commit
[`b42e7db`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153987324
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153987350
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153983073
**[Test build #1983 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1983/consoleFull)**
for PR 9477 at commit
[`3cb22d4`](https://gi
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153978941
LGTM overall, but I'd like to address one concern before merging: I'm
worried that passing both the `MemoryConsumer` and `TaskMemoryManager` to the
sorter components w
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9477#discussion_r43983728
--- Diff:
core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
---
@@ -78,22 +81,33 @@ public int compare(RecordPointe
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9477#discussion_r43983708
--- Diff:
core/src/main/java/org/apache/spark/shuffle/sort/ShuffleInMemorySorter.java ---
@@ -32,24 +37,39 @@ public int compare(PackedRecordPointer left,
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9477#discussion_r43983565
--- Diff:
core/src/main/java/org/apache/spark/shuffle/sort/ShuffleInMemorySorter.java ---
@@ -96,14 +111,12 @@ public long getMemoryUsage() {
*/
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9477#discussion_r43983352
--- Diff:
core/src/main/java/org/apache/spark/shuffle/sort/ShuffleInMemorySorter.java ---
@@ -96,14 +111,12 @@ public long getMemoryUsage() {
*/
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9477#discussion_r43983322
--- Diff:
core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSortDataFormat.java
---
@@ -44,37 +47,43 @@ public RecordPointerAndKeyPrefi
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9477#discussion_r43983319
--- Diff:
core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSortDataFormat.java
---
@@ -44,37 +47,43 @@ public RecordPointerAndKeyPrefi
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9477#discussion_r43983106
--- Diff:
core/src/main/java/org/apache/spark/shuffle/sort/ShuffleInMemorySorter.java ---
@@ -96,14 +111,12 @@ public long getMemoryUsage() {
*/
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9477#discussion_r43983047
--- Diff:
core/src/main/java/org/apache/spark/shuffle/sort/ShuffleInMemorySorter.java ---
@@ -32,24 +37,39 @@ public int compare(PackedRecordPointer left,
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9477#discussion_r43983033
--- Diff:
core/src/main/java/org/apache/spark/shuffle/sort/ShuffleInMemorySorter.java ---
@@ -32,24 +37,39 @@ public int compare(PackedRecordPointer left,
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9477#discussion_r43982895
--- Diff:
core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java ---
@@ -321,9 +320,10 @@ private void growPointerArrayIfNecessary() th
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153963379
**[Test build #45096 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45096/consoleFull)**
for PR 9477 at commit
[`89319e0`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153962209
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153962196
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153962004
**[Test build #1983 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1983/consoleFull)**
for PR 9477 at commit
[`3cb22d4`](https://git
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153961923
@JoshRosen This is ready for review
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does n
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153953917
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153953921
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153953782
**[Test build #45070 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45070/consoleFull)**
for PR 9477 at commit
[`862b38f`](https://git
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153950136
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153950106
**[Test build #45076 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45076/consoleFull)**
for PR 9477 at commit
[`3cb22d4`](https://git
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153950138
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153941335
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153941334
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153941271
**[Test build #45072 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45072/consoleFull)**
for PR 9477 at commit
[`77555e1`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153925753
**[Test build #45076 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45076/consoleFull)**
for PR 9477 at commit
[`3cb22d4`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153924545
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153924523
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153917831
**[Test build #45072 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45072/consoleFull)**
for PR 9477 at commit
[`77555e1`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153917595
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153917623
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153914518
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153914523
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153913274
**[Test build #45070 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45070/consoleFull)**
for PR 9477 at commit
[`862b38f`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153912804
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153912836
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153911937
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9477#issuecomment-153911915
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/9477
[SPARK-7542] [SQL] Support off-heap index/sort buffer
This brings the support of off-heap memory for array inside BytesToBytesMap
and InMemorySorter, then we could allocate all the memory from off-he
83 matches
Mail list logo
