Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/1481
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enab
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51390403
thanks Patrick
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
en
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51390354
hahaha I am filled with jubilation
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51389935
Okay I'm merging this - thanks Sandy!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project doe
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51389902
lol
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled an
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51389141
wooh!
On Wed, Aug 6, 2014 at 1:04 PM, Apache Spark QA
wrote:
> QA results for PR 1481:
> - This patch PASSES unit tests.
>
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51389072
QA results for PR 1481:- This patch PASSES unit tests.- This patch
merges cleanly- This patch adds no public classesFor more
information see test
ouptut:https://amplab.c
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51382561
QA tests have started for PR 1481. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18039/consoleFull
---
If
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51381773
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51378628
QA results for PR 1481:- This patch FAILED unit tests.- This patch
merges cleanly- This patch adds no public classesFor more
information see test
ouptut:https://amplab.c
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51372235
QA tests have started for PR 1481. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18030/consoleFull
---
If
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51363900
QA results for PR 1481:- This patch FAILED unit tests.- This patch
merges cleanly- This patch adds no public classesFor more
information see test
ouptut:https://amplab.c
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51358315
QA tests have started for PR 1481. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18027/consoleFull
---
If
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51304501
QA results for PR 1481:- This patch FAILED unit tests.- This patch
merges cleanly- This patch adds no public classesFor more
information see test
ouptut:https://amplab.c
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51300902
QA tests have started for PR 1481. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18012/consoleFull
---
If
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51288063
QA results for PR 1481:- This patch FAILED unit tests.For more
information see test
ouptut:https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17964/consol
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51285295
QA results for PR 1481:- This patch FAILED unit tests.- This patch
merges cleanly- This patch adds no public classesFor more
information see test
ouptut:https://amplab.c
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51282138
QA tests have started for PR 1481. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17965/consoleFull
---
If
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51281836
QA tests have started for PR 1481. This patch DID NOT merge cleanly!
View progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17964/consoleFull
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51233023
Jenkins, retest this please. I have access to the jenkins logs, but it's
not an open system right now - however I might be able to grab the logs for
you. Let's keep test
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51168899
Looking into it. I ran the test that it was hanging on and things
completed fine. I also combed the code and didn't see anywhere where this
patch had changed how things a
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51158962
Okay here's the deal - this patch is causing some type of non-deterministic
failure which seems related to the shuffle write path. It looks like the test
is hanging, but
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51151464
QA tests have started for PR 1481. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17921/consoleFull
---
If
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51151290
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not hav
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51140953
For some reason the tests are always hanging hanging in
`GeneratedMutableEvaluationSuite`... not sure what's up here
---
If your project is set up for it, you can reply
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51138180
QA tests have started for PR 1481. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17901/consoleFull
---
If
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51137891
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not hav
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51133558
QA tests have started for PR 1481. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17890/consoleFull
---
If
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51133352
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not hav
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51132873
QA tests have started for PR 1481. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17888/consoleFull
---
If
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51132885
QA results for PR 1481:- This patch FAILED unit tests.- This patch
merges cleanly- This patch adds no public classesFor more
information see test
ouptut:https://amplab.c
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51130186
QA tests have started for PR 1481. This patch DID NOT merge cleanly!
View progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17882/consoleFull
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51129863
@sryza I think there is a minor merge conflict going on here...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51129768
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51026321
Cool - thanks Sandy. Let's see if tests pass. Can likely merge this
tomorrow and fix any remaining issues (if they exist).
---
If your project is set up for it, you can
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51025769
QA tests have started for PR 1481. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17848/consoleFull
---
If
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51025499
Updated patch addresses @pwendell and @kayousterhout 's comments and adds
tests.
---
If your project is set up for it, you can reply to this email and have your
reply appe
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1481#discussion_r15742765
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -29,7 +29,7 @@ import akka.actor.{ActorSystem, Cancellable, Props}
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51008685
I tool a pass. I like the new approach, with this approach you actually net
remove code from Spark, which is great. Left comments inline. This needs some
unit tests as w
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1481#discussion_r15737693
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala ---
@@ -79,22 +70,13 @@ private[spark] class DiskBlockObjectWriter(
s
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1481#discussion_r15737688
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -567,10 +567,12 @@ private[spark] class BlockManager(
blockId: Bl
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1481#discussion_r15737640
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala ---
@@ -184,19 +171,28 @@ private[spark] class DiskBlockObjectWriter(
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1481#discussion_r15737644
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala ---
@@ -184,19 +171,28 @@ private[spark] class DiskBlockObjectWriter(
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1481#discussion_r15737617
--- Diff:
core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleWriter.scala ---
@@ -39,10 +39,13 @@ private[spark] class HashShuffleWriter[K, V](
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1481#discussion_r15737608
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala ---
@@ -111,7 +93,12 @@ private[spark] class DiskBlockObjectWriter(
pri
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1481#discussion_r15737311
--- Diff:
core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala
---
@@ -191,7 +193,9 @@ class ExternalAppendOnlyMap[K, V, C](
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1481#discussion_r15737305
--- Diff:
core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala
---
@@ -89,6 +90,8 @@ class ExternalAppendOnlyMap[K, V, C](
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1481#discussion_r15737300
--- Diff:
core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala
---
@@ -89,6 +90,8 @@ class ExternalAppendOnlyMap[K, V, C](
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51001886
QA tests have started for PR 1481. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17828/consoleFull
---
If
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51001772
QA results for PR 1481:- This patch FAILED unit tests.- This patch
merges cleanly- This patch adds no public classesFor more
information see test
ouptut:https://amplab.c
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51001765
Updated patch keeps it as ShuffleWriteMetrics for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If yo
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51001744
QA tests have started for PR 1481. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17827/consoleFull
---
If
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51000600
I guess the current name isn't _that_ bad because technically this external
aggregation does occur during a shuffle, although it's not exactly the write of
the shuffle d
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-50978548
The best way might be to do something like this.
```
/**
* Alias for WriterMetrics for compatibility reasons
*/
@DeveloperApi
class ShuffleW
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-50978354
@sryza Can't the `ExeternalSorter` and `ExternalAppendOnlyMap` just pass
their own `ShuffleWriteMetrics` when they create a disk writer and then read
back the bytes writ
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-50977756
I hadn't noticed this before, but DiskObjectWriter is used for tracking
bytes spilled by ExternalSorter and ExternalAppendOnlyMap in addition to
shuffle bytes written. So
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-50935680
I'm not totally finished with this but a couple high level comments:
1. I think this entirely subsumes some existing code that tracks timing
only in order to bub
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-50934577
QA results for PR 1481:- This patch FAILED unit tests.- This patch
merges cleanly- This patch adds no public classesFor more
information see test
ouptut:https://amplab.c
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/1481#discussion_r15718797
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala ---
@@ -168,6 +174,7 @@ private[spark] class DiskBlockObjectWriter(
/
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1481#discussion_r15718740
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala ---
@@ -176,7 +183,9 @@ private[spark] class DiskBlockObjectWriter(
i
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1481#discussion_r15718037
--- Diff:
core/src/main/scala/org/apache/spark/storage/ShuffleBlockManager.scala ---
@@ -136,7 +139,8 @@ class ShuffleBlockManager(blockManager: BlockManager
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1481#discussion_r15717899
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -567,10 +567,12 @@ private[spark] class BlockManager(
blockId: Bl
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-50929222
QA tests have started for PR 1481. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17696/consoleFull
---
If
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-49496103
QA results for PR 1481:- This patch PASSES unit tests.- This patch
merges cleanly- This patch adds no public classesFor more
information see test
ouptut:https://amplab.c
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-49492607
QA tests have started for PR 1481. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16839/consoleFull
---
If
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-49415768
QA results for PR 1481:- This patch FAILED unit tests.- This patch
merges cleanly- This patch adds no public classesFor more
information see test
ouptut:https://amplab.c
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-49407558
QA tests have started for PR 1481. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16822/consoleFull
---
If
GitHub user sryza opened a pull request:
https://github.com/apache/spark/pull/1481
SPARK-2566. Update ShuffleWriteMetrics incrementally
I haven't tested this out on a cluster yet, but wanted to make sure the
approach (passing ShuffleWriteMetrics down to DiskBlockObjectWriter) was ok
68 matches
Mail list logo
