Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21007#discussion_r180383755 --- Diff: python/pyspark/sql/tests.py --- @@ -3062,6 +3062,73 @@ def test_sparksession_with_stopped_sparkcontext(self): sc.stop() +class SQLTests3(unittest.TestCase): --- End diff -- I manually tested different conditions with this test for sure: Before: ``` test_query_execution_listener_on_collect (pyspark.sql.tests.SQLTests3) ... FAIL test_query_execution_listener_on_collect_with_arrow (pyspark.sql.tests.SQLTests3) ... FAIL ====================================================================== FAIL: test_query_execution_listener_on_collect (pyspark.sql.tests.SQLTests3) ---------------------------------------------------------------------- Traceback (most recent call last): File "/.../spark/python/pyspark/sql/tests.py", line 3105, in test_query_execution_listener_on_collect "The callback from the query execution listener should be called after 'collect'") AssertionError: The callback from the query execution listener should be called after 'collect' ====================================================================== FAIL: test_query_execution_listener_on_collect_with_arrow (pyspark.sql.tests.SQLTests3) ---------------------------------------------------------------------- Traceback (most recent call last): File "/.../spark/python/pyspark/sql/tests.py", line 3122, in test_query_execution_listener_on_collect_with_arrow "The callback from the query execution listener should be called after 'toPandas'") AssertionError: The callback from the query execution listener should be called after 'toPandas' ``` After: ``` test_query_execution_listener_on_collect (pyspark.sql.tests.SQLTests3) ... ok test_query_execution_listener_on_collect_with_arrow (pyspark.sql.tests.SQLTests3) ... ok ``` Missing 'org.apache.spark.sql.TestQueryExecutionListener' ``` skipped "'org.apache.spark.sql.TestQueryExecutionListener' is not available. Skipping the related tests." ``` Missing Pandas ``` test_query_execution_listener_on_collect (pyspark.sql.tests.SQLTests3) ... ok test_query_execution_listener_on_collect_with_arrow (pyspark.sql.tests.SQLTests3) ... skipped 'Pandas >= 0.19.2 must be installed; however, it was not found.' ``` Missing PyArrow ``` test_query_execution_listener_on_collect (pyspark.sql.tests.SQLTests3) ... ok test_query_execution_listener_on_collect_with_arrow (pyspark.sql.tests.SQLTests3) ... skipped 'PyArrow >= 0.8.0 must be installed; however, it was not found.' ```
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org