Github user srowen commented on the issue:
https://github.com/apache/spark/pull/9124
@andrewor14 looks like we still have some failures of the form...
```
- spilling with compression *** FAILED ***
java.lang.Exception: Test failed with compression using codec
org.apache.spark.io.LZ4CompressionCodec:
assertion failed: expected groupByKey to spill, but did not
at scala.Predef$.assert(Predef.scala:170)
at org.apache.spark.TestUtils$.assertSpilled(TestUtils.scala:170)
at
org.apache.spark.util.collection.ExternalAppendOnlyMapSuite.org$apache$spark$util$collection$ExternalAppendOnlyMapSuite$$testSimpleSpilling(ExternalAppendOnlyMapSuite.scala:253)
at
org.apache.spark.util.collection.ExternalAppendOnlyMapSuite$$anonfun$10$$anonfun$apply$mcV$sp$8.apply(ExternalAppendOnlyMapSuite.scala:218)
at
org.apache.spark.util.collection.ExternalAppendOnlyMapSuite$$anonfun$10$$anonfun$apply$mcV$sp$8.apply(ExternalAppendOnlyMapSuite.scala:216)
at scala.collection.immutable.Stream.foreach(Stream.scala:594)
...
```
Do you think it's possible that somehow there's a race here in reporting
the metrics? so that things are spilling but not reported to the listener by
the time the number of spilled tasks is checked?
I could try putting in an `eventually` to let it check repeatedly for a
short time? any better ideas, or reasons to think that's not it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
