attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network
when shuffle blocks are fetched from the same host
URL: https://github.com/apache/spark/pull/25299#issuecomment-558654601
The error is:
```
[ERROR] Failed to execute goal on project spark-catalyst_2.12:
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network
when shuffle blocks are fetched from the same host
URL: https://github.com/apache/spark/pull/25299#issuecomment-558624810
jenkins retest this please
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network
when shuffle blocks are fetched from the same host
URL: https://github.com/apache/spark/pull/25299#issuecomment-558214745
@tgravescs The block content reading will occur at the client side of the
'ShuffleBlockFetc
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network
when shuffle blocks are fetched from the same host
URL: https://github.com/apache/spark/pull/25299#issuecomment-557687439
jenkins retest this please
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network
when shuffle blocks are fetched from the same host
URL: https://github.com/apache/spark/pull/25299#issuecomment-554339093
jenkins retest this please
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network
when shuffle blocks are fetched from the same host
URL: https://github.com/apache/spark/pull/25299#issuecomment-553580712
With my previous commit accessing host local directories became
asynchronous. And it is use
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network
when shuffle blocks are fetched from the same host
URL: https://github.com/apache/spark/pull/25299#issuecomment-539100344
I am working on a solution where the local directories for the executor
(which are not cach
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network
when shuffle blocks are fetched from the same host
URL: https://github.com/apache/spark/pull/25299#issuecomment-528886783
The current state only covers the original idea to have communication with
the driver: no f
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network
when shuffle blocks are fetched from the same host
URL: https://github.com/apache/spark/pull/25299#issuecomment-522217408
jenkins retest this please
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network
when shuffle blocks are fetched from the same host
URL: https://github.com/apache/spark/pull/25299#issuecomment-522207781
The last three builds were failing for different reasons and all seams to me
unrelated for
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network
when shuffle blocks are fetched from the same host
URL: https://github.com/apache/spark/pull/25299#issuecomment-521998645
retest this please.
---
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network
when shuffle blocks are fetched from the same host
URL: https://github.com/apache/spark/pull/25299#issuecomment-521748111
retest this please.
---
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network
when shuffle blocks are fetched from the same host
URL: https://github.com/apache/spark/pull/25299#issuecomment-521709371
ok to test
This is an auto
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network
when shuffle blocks are fetched from the same host
URL: https://github.com/apache/spark/pull/25299#issuecomment-521670791
retest this please.
---
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network
when shuffle blocks are fetched from the same host
URL: https://github.com/apache/spark/pull/25299#issuecomment-521650177
jenkins retest this please
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network
when shuffle blocks are fetched from the same host
URL: https://github.com/apache/spark/pull/25299#issuecomment-521633221
jenkins retest this please
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network
when shuffle blocks are fetched from the same host
URL: https://github.com/apache/spark/pull/25299#issuecomment-520345813
@maropu I think the performance gains would be significant if the block size
is over "spark
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network
when shuffle blocks are fetched from the same host
URL: https://github.com/apache/spark/pull/25299#issuecomment-518320725
I can revert the metric changes and can count the host-local related bytes
and number of bl
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network
when shuffle blocks are fetched from the same host
URL: https://github.com/apache/spark/pull/25299#issuecomment-517737974
@tgravescs unfortunately haven't run any as for this case I have no good
performance tests.
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network
when shuffle blocks are fetched from the same host
URL: https://github.com/apache/spark/pull/25299#issuecomment-516867434
No. To fetch a block from the external shuffle service the network must be
used.
Here t
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network
when shuffle blocks are fetched from the same host
URL: https://github.com/apache/spark/pull/25299#issuecomment-516843077
Yes.
Basically external shuffle service basically does the same: I mean it reads
the
21 matches
Mail list logo