attilapiros commented on pull request #28708:
URL: https://github.com/apache/spark/pull/28708#issuecomment-641993718
I checked the refactoring and it is fine. I would like to still read the PR
and test a bit more.
This is
attilapiros commented on pull request #28708:
URL: https://github.com/apache/spark/pull/28708#issuecomment-641294356
Hi @holdenk!
I have opened a new PR for your branch:
https://github.com/holdenk/spark/pull/6
I know this seams to be a lot of change but it is really just
attilapiros commented on pull request #28708:
URL: https://github.com/apache/spark/pull/28708#issuecomment-639990452
It is in the `unit-tests.log` on the jenkins too (this is the appender
target see
attilapiros commented on pull request #28708:
URL: https://github.com/apache/spark/pull/28708#issuecomment-639679228
> > > Oh also I think I understand some of our disagreement over the
threads. I thought you were asking me to stop the Spark executor because I’ve
started doing some
attilapiros commented on pull request #28708:
URL: https://github.com/apache/spark/pull/28708#issuecomment-639642946
> Oh also I think I understand some of our disagreement over the threads. I
thought you were asking me to stop the Spark executor because I’ve started
doing some separate
attilapiros commented on pull request #28708:
URL: https://github.com/apache/spark/pull/28708#issuecomment-639553419
I have created one unit test and opened a PR with it:
https://github.com/holdenk/spark/pull/5
With `sbt` it is passing as expected:
```
$ ./build/sbt -Pyarn
attilapiros commented on pull request #28708:
URL: https://github.com/apache/spark/pull/28708#issuecomment-639004563
> So @attilapiros looking at the Jenkins console logs we aren't leaking any
threads during testing (nor would I expect us to). But I'll add something to
more aggressively
attilapiros commented on pull request #28708:
URL: https://github.com/apache/spark/pull/28708#issuecomment-638950902
> Hey @attilapiros can you explain to my why you think we need to test the
different kinds of block fetches? When we migrate we're always migrating to
disk so I'm not
attilapiros commented on pull request #28708:
URL: https://github.com/apache/spark/pull/28708#issuecomment-638916984
I just found a bug in my `BlockManagerSuite` test which is caused by this
line:
attilapiros commented on pull request #28708:
URL: https://github.com/apache/spark/pull/28708#issuecomment-638684678
> So I don't want to stop the executor directly once the block migration is
done. Instead, I have a follow-up JIRA which I've started working on that
shutdowns the executor
attilapiros commented on pull request #28708:
URL: https://github.com/apache/spark/pull/28708#issuecomment-638174017
I think we should test the offloading of the shuffle blocks at the unit test
level so I suggest to add new test to the `BlockManagerSuite`:
```scala
test("test
11 matches
Mail list logo
