Github user szhem commented on the issue:
https://github.com/apache/spark/pull/19410
Hi @mallman!
In case of
```
StorageLevel.MEMORY_AND_DISK
StorageLevel.MEMORY_AND_DISK_SER_2
```
... tests pass.
They still fail in case of
```
StorageLevel.MEMORY_ONLY
StorageLevel.MEMORY_ONLY_SER
```
Although it works, I'm not sure that changing the caching level of the
graph is really a good option to go with as Spark starts complaining
[here](https://github.com/apache/spark/blob/f830bb9170f6b853565d9dd30ca7418b93a54fe3/graphx/src/main/scala/org/apache/spark/graphx/impl/VertexPartitionBaseOps.scala#L111)
and
[here](https://github.com/apache/spark/blob/f830bb9170f6b853565d9dd30ca7418b93a54fe3/graphx/src/main/scala/org/apache/spark/graphx/impl/VertexPartitionBaseOps.scala#L131)
```
18/06/27 16:08:46.802 Executor task launch worker for task 3 WARN
ShippableVertexPartitionOps: Diffing two VertexPartitions with different
indexes is slow.
18/06/27 16:08:47.000 Executor task launch worker for task 4 WARN
ShippableVertexPartitionOps: Diffing two VertexPartitions with different
indexes is slow.
18/06/27 16:08:47.164 Executor task launch worker for task 5 WARN
ShippableVertexPartitionOps: Diffing two VertexPartitions with different
indexes is slow.
18/06/27 16:08:48.724 Executor task launch worker for task 18 WARN
ShippableVertexPartitionOps: Joining two VertexPartitions with different
indexes is slow.
18/06/27 16:08:48.749 Executor task launch worker for task 18 WARN
ShippableVertexPartitionOps: Diffing two VertexPartitions with different
indexes is slow.
18/06/27 16:08:48.868 Executor task launch worker for task 19 WARN
ShippableVertexPartitionOps: Joining two VertexPartitions with different
indexes is slow.
18/06/27 16:08:48.899 Executor task launch worker for task 19 WARN
ShippableVertexPartitionOps: Diffing two VertexPartitions with different
indexes is slow.
18/06/27 16:08:49.008 Executor task launch worker for task 20 WARN
ShippableVertexPartitionOps: Joining two VertexPartitions with different
indexes is slow.
18/06/27 16:08:49.028 Executor task launch worker for task 20 WARN
ShippableVertexPartitionOps: Diffing two VertexPartitions with different
indexes is slow.
```
P.S. To emulate the lack of memory I just set the following options like
[here](https://github.com/apache/spark/pull/19410/files?utf8=%E2%9C%93&diff=unified#diff-c2823ca69af75fc6cdfd1ebbf25c2aefR85)
to emulate lack of memory resources.
```
spark.testing.reservedMemory
spark.testing.memory
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
