HyukjinKwon opened a new pull request #23870: [SPARK-26945][PYTHON][TESTS] Fix flaky PySpark tests in StreamingTests URL: https://github.com/apache/spark/pull/23870 ## What changes were proposed in this pull request? This PR proposes to make sure process all available data before stopping and delete the temp directory. See https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/102518/console ``` ERROR: test_query_manager_await_termination (pyspark.sql.tests.test_streaming.StreamingTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/sql/tests/test_streaming.py", line 259, in test_query_manager_await_termination shutil.rmtree(tmpPath) File "/home/anaconda/lib/python2.7/shutil.py", line 256, in rmtree onerror(os.rmdir, path, sys.exc_info()) File "/home/anaconda/lib/python2.7/shutil.py", line 254, in rmtree os.rmdir(path) OSError: [Errno 39] Directory not empty: '/home/jenkins/workspace/SparkPullRequestBuilder/python/target/072153bd-f981-47be-bda2-e2b657a16f65/tmp4WGp7n' ``` See https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/102311/console ``` ERROR: test_stream_await_termination (pyspark.sql.tests.test_streaming.StreamingTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jenkins/workspace/SparkPullRequestBuilder@2/python/pyspark/sql/tests/test_streaming.py", line 202, in test_stream_await_termination shutil.rmtree(tmpPath) File "/usr/lib64/pypy-2.5.1/lib-python/2.7/shutil.py", line 256, in rmtree onerror(os.rmdir, path, sys.exc_info()) File "/usr/lib64/pypy-2.5.1/lib-python/2.7/shutil.py", line 254, in rmtree os.rmdir(path) OSError: [Errno 39] Directory not empty: '/home/jenkins/workspace/SparkPullRequestBuilder@2/python/target/7244f4ff-6b60-4f6c-b787-de4f15922bf5/tmpQbMZSo' ``` ## How was this patch tested? Jenkins tests
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
