[GitHub] [spark] attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host

2019-10-07 Thread GitBox
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#issuecomment-539100344 I am working on a solution where the local directories for the executor (which are not

[GitHub] [spark] attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host

2019-09-06 Thread GitBox
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#issuecomment-528886783 The current state only covers the original idea to have communication with the driver: no

[GitHub] [spark] attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host

2019-08-17 Thread GitBox
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#issuecomment-522217408 jenkins retest this please

[GitHub] [spark] attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host

2019-08-16 Thread GitBox
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#issuecomment-522207781 The last three builds were failing for different reasons and all seams to me unrelated for

[GitHub] [spark] attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host

2019-08-16 Thread GitBox
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#issuecomment-521998645 retest this please.

[GitHub] [spark] attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host

2019-08-15 Thread GitBox
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#issuecomment-521748111 retest this please.

[GitHub] [spark] attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host

2019-08-15 Thread GitBox
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#issuecomment-521709371 ok to test This is an

[GitHub] [spark] attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host

2019-08-15 Thread GitBox
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#issuecomment-521670791 retest this please.

[GitHub] [spark] attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host

2019-08-15 Thread GitBox
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#issuecomment-521650177 jenkins retest this please

[GitHub] [spark] attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host

2019-08-15 Thread GitBox
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#issuecomment-521633221 jenkins retest this please

[GitHub] [spark] attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host

2019-08-12 Thread GitBox
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#issuecomment-520345813 @maropu I think the performance gains would be significant if the block size is over

[GitHub] [spark] attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host

2019-08-05 Thread GitBox
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#issuecomment-518320725 I can revert the metric changes and can count the host-local related bytes and number of

[GitHub] [spark] attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host

2019-08-02 Thread GitBox
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#issuecomment-517737974 @tgravescs unfortunately haven't run any as for this case I have no good performance

[GitHub] [spark] attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host

2019-07-31 Thread GitBox
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#issuecomment-516867434 No. To fetch a block from the external shuffle service the network must be used. Here

[GitHub] [spark] attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host

2019-07-31 Thread GitBox
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#issuecomment-516843077 Yes. Basically external shuffle service basically does the same: I mean it reads