[GitHub] spark pull request #20892: [SPARK-23700][PYTHON] Cleanup imports in pyspark....

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/20892


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20892: [SPARK-23700][PYTHON] Cleanup imports in pyspark....

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/20892#discussion_r176897639
  
--- Diff: python/pyspark/sql/functions.py ---
@@ -28,10 +27,10 @@
 
 from pyspark import since, SparkContext
 from pyspark.rdd import ignore_unicode_prefix, PythonEvalType
-from pyspark.serializers import PickleSerializer, AutoBatchedSerializer
 from pyspark.sql.column import Column, _to_java_column, _to_seq
 from pyspark.sql.dataframe import DataFrame
 from pyspark.sql.types import StringType, DataType
+# Keep UserDefinedFunction import for backwards compatible import; moved 
in SPARK-22409
 from pyspark.sql.udf import UserDefinedFunction, _create_udf
--- End diff --

yea, I think we should better keep this import and the comment looks good.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20892: [SPARK-23700][PYTHON] Cleanup imports in pyspark....

[BryanCutler]

Github user BryanCutler commented on a diff in the pull request:

https://github.com/apache/spark/pull/20892#discussion_r176829827
  
--- Diff: python/pyspark/sql/functions.py ---
@@ -28,10 +27,10 @@
 
 from pyspark import since, SparkContext
 from pyspark.rdd import ignore_unicode_prefix, PythonEvalType
-from pyspark.serializers import PickleSerializer, AutoBatchedSerializer
 from pyspark.sql.column import Column, _to_java_column, _to_seq
 from pyspark.sql.dataframe import DataFrame
 from pyspark.sql.types import StringType, DataType
+# Keep UserDefinedFunction import for backwards compatible import; moved 
in SPARK-22409
 from pyspark.sql.udf import UserDefinedFunction, _create_udf
--- End diff --

Not sure if there is a better way to do this other than importing 
`UserDefinedFunction` here, but hopefully the note will show the intent.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20892: [SPARK-23700][PYTHON] Cleanup imports in pyspark....

[BryanCutler]

GitHub user BryanCutler opened a pull request:

https://github.com/apache/spark/pull/20892

[SPARK-23700][PYTHON] Cleanup imports in pyspark.sql

## What changes were proposed in this pull request?

This cleans up unused imports, mainly from pyspark.sql module.  Added a 
note in function.py that imports `UserDefinedFunction` only to maintain 
backwards compatibility for using `from pyspark.sql.function import 
UserDefinedFunction`. 

## How was this patch tested?

Existing tests.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/BryanCutler/spark 
pyspark-cleanup-imports-SPARK-23700

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/20892.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #20892


commit 5da8c3d39cbbe8e78df2fad4c04d7a7ab1d9db9d
Author: Bryan Cutler 
Date:   2018-03-22T00:09:01Z

tests passing

commit 5214f411d28a19b244a97ffe25f8be5852e273c1
Author: Bryan Cutler 
Date:   2018-03-23T18:25:28Z

change note description




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org