Github user bersprockets commented on the issue:
https://github.com/apache/spark/pull/20909
Thanks @felixcheung . Turns out HiveSparkSubmitTests will fail if Spark is
not built with the hive profile (AssertionError: 0 != 1).
In addition, at least one pyspark.sql.readwriter docstring test fails.
This PR uses pyspark.sql.tests as the "leader of the pack" (run-tests.py
gives it priortity 0 amongst the pyspark.sql tests) to check for prerequisites
for its own tests as well as the sql docstring tests. The docstring tests can't
make these checks.
I modeled this after pyspark/streaming/tests.py, which checks for prereqs
and raises exceptions with a useful message so one can get past the error
(although pyspark/streaming/tests.py only checks for its own prereqs, not
those required by streaming docstring tests).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
