Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/20211#discussion_r161119843
--- Diff: python/pyspark/sql/group.py ---
@@ -233,6 +233,27 @@ def apply(self, udf):
| 2| 1.1094003924504583|
+---+-------------------+
+ Notes on grouping column:
--- End diff --
@felixcheung, WDYT?
To cut the context short, it's a Pandas map group API like `gapply` (not
Pandas scalar udf).
Its current implementation is as follows
```python
def foo(pdf):
pdf # this is the Pandas DataFrame
pudf = pandas_udf(f=foo, returnType="id int, v double",
functionType=GROUP_MAP)
df.groupby(group_column).apply(pudf)
```
First `'id int, v double'` describes the output schema and input `pdf` is
the grouped Pandas's DataFrame.
As @icexelloss described above as a new proposal, looking at `gapply` in R
at a glance again, seems making sense that we do:
```python
def foo(key, pdf):
key # this is a grouping key.
pdf # this is the Pandas DataFrame
pudf = pandas_udf(f=foo, returnType="id int, v double",
functionType=GROUP_MAP)
df.groupby(group_column).apply(pudf)
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
