[GitHub] spark pull request #20658: [SPARK-23488][python] Add missing catalog methods...

[drboyer]

Github user drboyer commented on a diff in the pull request:

https://github.com/apache/spark/pull/20658#discussion_r170158422
  
--- Diff: python/pyspark/sql/catalog.py ---
@@ -28,7 +28,7 @@
 Database = namedtuple("Database", "name description locationUri")
 Table = namedtuple("Table", "name database description tableType 
isTemporary")
 Column = namedtuple("Column", "name description dataType nullable 
isPartition isBucket")
-Function = namedtuple("Function", "name description className isTemporary")
+Function = namedtuple("Function", "name database description className 
isTemporary")
--- End diff --

Ah yes, `database` is in the [Scala 
api](http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.catalog.Function)
 so I added it in for the sake of completeness, but I'm happy to remove it if 
there's a concern.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20658: [SPARK-23488][python] Add missing catalog methods...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/20658#discussion_r170003442
  
--- Diff: python/pyspark/sql/catalog.py ---
@@ -28,7 +28,7 @@
 Database = namedtuple("Database", "name description locationUri")
 Table = namedtuple("Table", "name database description tableType 
isTemporary")
 Column = namedtuple("Column", "name description dataType nullable 
isPartition isBucket")
-Function = namedtuple("Function", "name description className isTemporary")
+Function = namedtuple("Function", "name database description className 
isTemporary")
--- End diff --

Hm, wouldn't this break backward compatibility?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20658: [SPARK-23488][python] Add missing catalog methods...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/20658#discussion_r170003647
  
--- Diff: python/pyspark/sql/catalog.py ---
@@ -137,6 +138,78 @@ def listColumns(self, tableName, dbName=None):
 isBucket=jcolumn.isBucket()))
 return columns
 
+@ignore_unicode_prefix
+# TODO: @since() decorator?
--- End diff --

I think `@since(2.4)`.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20658: [SPARK-23488][python] Add missing catalog methods...

[drboyer]

GitHub user drboyer opened a pull request:

https://github.com/apache/spark/pull/20658

[SPARK-23488][python] Add missing catalog methods to python API

## What changes were proposed in this pull request?

As noted in SPARK-23488, the Python Catalog API was missing some methods 
that are present in the Scala API. Both for the sake of consistency, and 
because of their utility, I've added the missing methods (the 
database/table/functionExists() and getDatabase/Table/Function() methods).

I modeled these methods off of how the existing ones were written in 
catalog.py.

## How was this patch tested?

manually tested the added methods and compared functionality to the scala 
API counterparts

## Questions

I wasn't sure whether to set the `@since(x.y)` decorators on the new 
functions, and if so, what value to set them to?


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/drboyer/spark add-other-catalog-methods

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/20658.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #20658


commit c82394a87d1a36b3fc899553433119d5ca8bc28f
Author: Devin Boyer 
Date:   2018-02-22T05:06:37Z

add missing catalog methods to python API




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org