[GitHub] spark pull request #19997: [SPARK-22811][pyspark][ml] Fix pyspark.ml.tests f...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/19997


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19997: [SPARK-22811][pyspark][ml] Fix pyspark.ml.tests f...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/19997#discussion_r157330796
  
--- Diff: python/pyspark/ml/tests.py ---
@@ -44,6 +44,7 @@
 import numpy as np
 from numpy import abs, all, arange, array, array_equal, inf, ones, tile, 
zeros
 import inspect
+import py4j
--- End diff --

Ah, it was my bad. Yup, you are right.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19997: [SPARK-22811][pyspark][ml] Fix pyspark.ml.tests f...

[MrBago]

Github user MrBago commented on a diff in the pull request:

https://github.com/apache/spark/pull/19997#discussion_r157325083
  
--- Diff: python/pyspark/ml/tests.py ---
@@ -44,6 +44,7 @@
 import numpy as np
 from numpy import abs, all, arange, array, array_equal, inf, ones, tile, 
zeros
 import inspect
+import py4j
--- End diff --

On the line below, we catch `except py4j.protocol.Py4JError` errors so that 
we can then raise SkipTest instead, but if we don't import `py4j` we get a 
`NameError` instead of skipping the test. Furthermore, because we don't trigger 
`tearDownClass()` on the following line we leave behind stale state which 
causes other tests to fail. The `except` line is only ever triggered in 
environments that don't have Hive.


https://github.com/apache/spark/pull/19997/files#diff-4a75aace12688903bc8f97e7930622f4R1868


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19997: [SPARK-22811][pyspark][ml] Fix pyspark.ml.tests f...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/19997#discussion_r157318220
  
--- Diff: python/pyspark/ml/tests.py ---
@@ -44,6 +44,7 @@
 import numpy as np
 from numpy import abs, all, arange, array, array_equal, inf, ones, tile, 
zeros
 import inspect
+import py4j
--- End diff --

BTW, mind elaborating how importing this fixes an issue? It sounds 
orthogonal to me.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19997: [SPARK-22811][pyspark][ml] Fix pyspark.ml.tests f...

[MrBago]

GitHub user MrBago opened a pull request:

https://github.com/apache/spark/pull/19997

[SPARK-22811][pyspark][ml] Fix pyspark.ml.tests failure when Hive is not 
available.

## What changes were proposed in this pull request?

pyspark.ml.tests is missing a py4j import. I've added the import and fixed 
the test that uses it. This test was only failing when testing without Hive.

## How was this patch tested?

Existing tests.

Please review http://spark.apache.org/contributing.html before opening a 
pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MrBago/spark fix-ImageReaderTest2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/19997.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #19997


commit 46410f164aa668a95d60c07a8f8d66938a6bd5cf
Author: Bago Amirbekian 
Date:   2017-12-15T22:16:41Z

Fix pyspark.ml.tests failure when Hive is not available.




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org