Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-57676144
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-57676219
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21214/consoleFull)
for PR 2366 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-57678711
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-57692138
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21214/consoleFull)
for PR 2366 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-57692143
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-57703944
Thanks. Merging in master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/2366
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r18200248
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +791,110 @@ private[spark] class BlockManager(
}
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r18200286
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -111,6 +112,9 @@ private[spark] class BlockManager(
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r18200328
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +791,110 @@ private[spark] class BlockManager(
}
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r18201177
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +791,110 @@ private[spark] class BlockManager(
}
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r18201208
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +791,110 @@ private[spark] class BlockManager(
}
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r18201230
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +791,110 @@ private[spark] class BlockManager(
}
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r18201286
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +791,110 @@ private[spark] class BlockManager(
}
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r18233118
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala
---
@@ -59,6 +59,8 @@ class BlockManagerId private (
def port: Int =
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r18233139
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +789,88 @@ private[spark] class BlockManager(
}
/**
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-57351691
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-57352608
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21044/consoleFull)
for PR 2366 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-57363077
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21044/consoleFull)
for PR 2366 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-57363086
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-57402676
@rxin This is good to go!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r18259626
--- Diff:
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
@@ -1228,4 +1244,314 @@ class BlockManagerSuite extends FunSuite with
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r18117922
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +790,111 @@ private[spark] class BlockManager(
}
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-57032133
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20888/consoleFull)
for PR 2366 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-57035730
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20888/consoleFull)
for PR 2366 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-57035733
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-57042546
@rxin I dont think it is getting more complicated that the status quo. The
complexity of fetching and caching of peers is contained in this one method
`getPeers`, so it
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-57042584
No I was responding to Mridul's rebalancing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r18122027
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +791,110 @@ private[spark] class BlockManager(
}
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r18122030
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -111,6 +112,9 @@ private[spark] class BlockManager(
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r18122034
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +791,110 @@ private[spark] class BlockManager(
}
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r18122036
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +789,88 @@ private[spark] class BlockManager(
}
/**
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r18122040
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +791,110 @@ private[spark] class BlockManager(
}
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r18122042
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala
---
@@ -59,6 +59,8 @@ class BlockManagerId private (
def port: Int =
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r18122059
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +791,110 @@ private[spark] class BlockManager(
}
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17902630
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +789,88 @@ private[spark] class BlockManager(
}
/**
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17904273
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +789,88 @@ private[spark] class BlockManager(
}
/**
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17906928
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +789,88 @@ private[spark] class BlockManager(
}
/**
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17907153
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +789,88 @@ private[spark] class BlockManager(
}
/**
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17916865
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +789,88 @@ private[spark] class BlockManager(
}
/**
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17921361
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +789,88 @@ private[spark] class BlockManager(
}
/**
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-56553233
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20711/consoleFull)
for PR 2366 at commit
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17924427
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +789,88 @@ private[spark] class BlockManager(
}
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-56561115
@rxin, @mridulm I have slightly changed the logic to select one peer at
random to make it more efficient.
---
If your project is set up for it, you can reply to this
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-56563234
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20711/consoleFull)
for PR 2366 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-56563250
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20711/
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17927862
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +790,111 @@ private[spark] class BlockManager(
}
Github user mridulm commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-56566367
@tdas In case I did not mention it before :-) this is definitely a great
improvement over what existed earlier !
I would love it if we could (sometime soon I hope)
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-56573235
If this is getting more complicated, we should consider standardizing the
internal api and then buliding a separate service that properly handles all
these issues. That
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17833383
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +789,88 @@ private[spark] class BlockManager(
}
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17833419
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +789,88 @@ private[spark] class BlockManager(
}
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17833483
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +789,88 @@ private[spark] class BlockManager(
}
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-56293506
@mridulm Very good thoughts! I totally agree that replication is not only
for streaming, and the implications of this patch in other scenarios is
important to understand.
Github user mridulm commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-56293724
@tdas handling (1) deterministically will make (2) in line with what we
currently have.
And that should be sufficient imo.
(3) was not in context of this
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17825042
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala ---
@@ -403,16 +402,20 @@ class BlockManagerMasterActor(val isLocal:
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17825062
--- Diff:
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
@@ -1228,4 +1240,212 @@ class BlockManagerSuite extends FunSuite with
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17825064
--- Diff:
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
@@ -62,6 +63,7 @@ class BlockManagerSuite extends FunSuite with Matchers
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17825538
--- Diff:
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
@@ -1228,4 +1240,212 @@ class BlockManagerSuite extends FunSuite with
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17825562
--- Diff:
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
@@ -1228,4 +1240,212 @@ class BlockManagerSuite extends FunSuite with
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17825572
--- Diff:
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
@@ -1228,4 +1240,212 @@ class BlockManagerSuite extends FunSuite with
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-56311396
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20634/consoleFull)
for PR 2366 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-56313463
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20634/consoleFull)
for PR 2366 at commit
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-56317365
@mridulm I implemented (1) and also added an unit test for testing that
behavior.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-56317539
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20636/consoleFull)
for PR 2366 at commit
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17829791
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +789,88 @@ private[spark] class BlockManager(
}
/**
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-56319636
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20636/consoleFull)
for PR 2366 at commit
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17829854
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +789,88 @@ private[spark] class BlockManager(
}
/**
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17829869
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -111,6 +111,8 @@ private[spark] class BlockManager(
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17829908
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -111,6 +111,8 @@ private[spark] class BlockManager(
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17829919
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +789,88 @@ private[spark] class BlockManager(
}
/**
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17829945
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +789,88 @@ private[spark] class BlockManager(
}
/**
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17829956
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +789,88 @@ private[spark] class BlockManager(
}
/**
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17829979
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +789,88 @@ private[spark] class BlockManager(
}
/**
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17830017
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +789,88 @@ private[spark] class BlockManager(
}
/**
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17830025
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +789,88 @@ private[spark] class BlockManager(
}
/**
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17831655
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +789,88 @@ private[spark] class BlockManager(
}
/**
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17833363
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +789,88 @@ private[spark] class BlockManager(
}
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-55372429
Isn't 1s cache span too low? How often will we get a cache hit if they
expire in 1 sec?
---
If your project is set up for it, you can reply to this email and have your
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17493889
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala
---
@@ -59,6 +59,8 @@ class BlockManagerId private (
def port:
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-55447441
Yeah, 1 second is probably too low (though there will be 4/5 cache hits for
every miss for streaming). Better to have it be a minute. Its a tradeoff
between how fast we
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17497968
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala ---
@@ -403,16 +402,20 @@ class BlockManagerMasterActor(val isLocal:
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17498010
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala ---
@@ -403,16 +402,20 @@ class BlockManagerMasterActor(val isLocal:
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17498022
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala ---
@@ -403,16 +402,20 @@ class BlockManagerMasterActor(val isLocal:
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17498056
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala ---
@@ -403,16 +402,20 @@ class BlockManagerMasterActor(val isLocal:
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17498083
--- Diff:
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
@@ -62,6 +63,7 @@ class BlockManagerSuite extends FunSuite with
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17498146
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala ---
@@ -403,16 +402,20 @@ class BlockManagerMasterActor(val isLocal:
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17498158
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala ---
@@ -403,16 +402,20 @@ class BlockManagerMasterActor(val isLocal:
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17498282
--- Diff:
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
@@ -1228,4 +1240,212 @@ class BlockManagerSuite extends FunSuite with
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17498486
--- Diff:
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
@@ -1228,4 +1240,212 @@ class BlockManagerSuite extends FunSuite with
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17498517
--- Diff:
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
@@ -1228,4 +1240,212 @@ class BlockManagerSuite extends FunSuite with
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17498587
--- Diff:
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
@@ -1228,4 +1240,212 @@ class BlockManagerSuite extends FunSuite with
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17498639
--- Diff:
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
@@ -1228,4 +1240,212 @@ class BlockManagerSuite extends FunSuite with
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-55451730
I looked at the logic in `BlockManager` in detail and it looks reasonable.
I left a few minor comments all over the place, but in general this LGTM.
---
If your
Github user mridulm commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-55472830
What happens when there is recomputation which results in same blockId
getting regenerated (unpersist followed by recomputation/persist or block drop
followed by
GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/2366
[SPARK-3495] Block replication fails continuously when the replication
target node is dead
If a block manager (say, A) wants to replicate a block and the node chosen
for replication (say, B) is dead,
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-55351240
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-55351493
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20190/consoleFull)
for PR 2366 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-55352316
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20193/consoleFull)
for PR 2366 at commit
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2366#discussion_r17460098
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -787,31 +789,87 @@ private[spark] class BlockManager(
}
/**
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2366#issuecomment-55354916
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20190/consoleFull)
for PR 2366 at commit
1 - 100 of 102 matches
Mail list logo
