santosh-d3vpl3x commented on code in PR #39902:
URL: https://github.com/apache/spark/pull/39902#discussion_r1102879097
##########
python/pyspark/worker.py:
##########
@@ -208,6 +208,41 @@ def wrapped(left_key_series, left_value_series,
right_key_series, right_value_se
return lambda kl, vl, kr, vr: [(wrapped(kl, vl, kr, vr),
to_arrow_type(return_type))]
+def wrap_multi_cogrouped_map_pandas_udf(f, return_type, runner_conf, argspec):
+ def wrapped(key_series, value_series):
+ import pandas as pd
+
+ dfs = [pd.concat(series, axis=1) for series in value_series]
+
+ if runner_conf.get("pass_key") == "true":
Review Comment:
Yes indeed, if we don't allow users to pass var-args in UDF.
I would like to have
- capability to pass var-args as input in UDFs
- explicitness if we don't follow your last suggestion
I am waiting for more opinions in this regard. The last experimental
implementation from v3.3.1 is somewhat soft blocking this PR and I would prefer
to go with your last suggestion that we always have key as first arg if we can.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
