HyukjinKwon commented on code in PR #57079:
URL: https://github.com/apache/spark/pull/57079#discussion_r3556084785
##########
sql/api/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -8986,6 +10385,8 @@ object functions {
*
* @group collection_funcs
* @since 2.4.0
+ * @return
+ * Returns a column of the element type of the input array.
Review Comment:
`element_at` also accepts a map, where it returns the **value type**
(`ElementAt.dataType`: `case MapType(_, valueType, _) => valueType`; the prose
above says "Returns value for the given key ... if column is map"). The
`@return` currently covers only the array case:
```suggestion
* Returns a column of the element type of the input array, or the value
type of the input map.
```
##########
sql/api/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -9435,6 +10890,8 @@ object functions {
*
* @group generator_funcs
* @since 1.3.0
+ * @return
+ * Returns a column of the element type of the input array.
Review Comment:
`explode` accepts a map too (`TypeCollection(ArrayType, MapType)`); for a
map it produces `key` and `value` columns, not a single element type.
```suggestion
* Returns a column of the element type of the input array, or the key
and value columns of the input map.
```
##########
sql/api/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -8999,6 +10400,8 @@ object functions {
*
* @group collection_funcs
* @since 3.5.0
+ * @return
+ * Returns a column of the element type of the input array.
Review Comment:
Same as `element_at` (`try_element_at` wraps `ElementAt`, same `dataType`):
also returns the map value type.
```suggestion
* Returns a column of the element type of the input array, or the value
type of the input map.
```
##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -19653,14 +20249,17 @@ def element_at(col: "ColumnOrName", extraction: Any)
-> Column:
Parameters
----------
col : :class:`~pyspark.sql.Column` or str
- name of column containing array or map
+ name of column containing array or map.
+ A column that evaluates to an array or map.
extraction :
- index to check for in array or key to check for in map
+ index to check for in array or key to check for in map.
+ A column that evaluates to an integer for an array, or the key type
for a map.
Returns
-------
:class:`~pyspark.sql.Column`
value at given position.
+ Returns a column of the element type of the input array.
Review Comment:
`element_at` also accepts a map, returning the value type (the docstring
above says "Returns value for given key in `extraction` if col is map").
```suggestion
Returns a column of the element type of the input array, or the
value type of the input map.
```
##########
sql/api/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -9445,6 +10902,8 @@ object functions {
*
* @group generator_funcs
* @since 2.2.0
+ * @return
+ * Returns a column of the element type of the input array.
Review Comment:
Same as `explode`: `explode_outer` on a map yields `key`/`value` columns.
```suggestion
* Returns a column of the element type of the input array, or the key
and value columns of the input map.
```
##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -20779,6 +21414,7 @@ def explode(col: "ColumnOrName") -> Column:
-------
:class:`~pyspark.sql.Column`
One row per array item or map key value.
+ Returns a column of the element type of the input array.
Review Comment:
`explode` accepts a map too — the docstring says "Returns a new row for each
element in the given array or map" and has "Example 2: Exploding a map column"
(yielding `key`/`value`).
```suggestion
Returns a column of the element type of the input array, or the key
and value columns of the input map.
```
##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -21093,6 +21729,7 @@ def explode_outer(col: "ColumnOrName") -> Column:
-------
:class:`~pyspark.sql.Column`
one row per array item or map key value.
+ Returns a column of the element type of the input array.
Review Comment:
Same as `explode`: `explode_outer` on a map yields `key`/`value` columns.
```suggestion
Returns a column of the element type of the input array, or the key
and value columns of the input map.
```
##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -19749,9 +20348,16 @@ def try_element_at(col: "ColumnOrName", extraction:
"ColumnOrName") -> Column:
Parameters
----------
col : :class:`~pyspark.sql.Column` or str
- name of column containing array or map
+ name of column containing array or map.
+ A column that evaluates to an array or map.
extraction :
- index to check for in array or key to check for in map
+ index to check for in array or key to check for in map.
+ A column that evaluates to an integer for an array, or the key type
for a map.
+
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ Returns a column of the element type of the input array.
Review Comment:
Same as `element_at`: `try_element_at` also returns the map value type.
```suggestion
Returns a column of the element type of the input array, or the
value type of the input map.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
