Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/21977
@squito, I updated `YarnClusterSuite` in d58ad7a to capture the output of
the child processes to find out what is causing the test failures. I think it
is related to the commit you reviewed,
15fc2372269159ea2556b028d4eb8860c4108650, where the accumulator was updated.
Here's the error:
```
18/08/13 14:29:33 INFO SparkContext: Successfully stopped SparkContext
Traceback (most recent call last):
File
"/home/blue-unencrypted/workspace/spark/resource-managers/yarn/target/tmp/spark-8370887e-9415-4f51-b08d-a6285e0384f5/test.py",
line 11, in <module>
sc = SparkContext(conf=SparkConf())
File
"/home/blue-unencrypted/workspace/spark/python/lib/pyspark.zip/pyspark/context.py",
line 118, in __init__
File
"/home/blue-unencrypted/workspace/spark/python/lib/pyspark.zip/pyspark/context.py",
line 188, in _do_init
File
"/home/blue-unencrypted/workspace/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py",
line 1525, in __call__
File
"/home/blue-unencrypted/workspace/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py",
line 332, in get_return_value
py4j.protocol.Py4JError: An error occurred while calling
None.org.apache.spark.api.python.PythonAccumulatorV2. Trace:
py4j.Py4JException: Constructor
org.apache.spark.api.python.PythonAccumulatorV2([class java.lang.String, class
java.lang.Integer]) does not exist
at
py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:179)
at
py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:196)
at py4j.Gateway.invoke(Gateway.java:237)
at
py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at
py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
```
Can you tell why that is failing after the changes I've made here? Is this
a bug with this PR or is it an existing problem from that commit?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
