Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/21427
I would prefer if we could do this without a config because while the
current behavior can work if the user knows what they are doing, it can also
fail very easily and not obviously. So to me that seems like a bug and we
should just fix it so the feature can not be used in a potentially dangerous
way.
If we need to make a config though, can it be such that it falls back to
the current behavior (to use position) only in there is a `KeyError` and the
switch is set to be backwards compatible? Otherwise it would raise the
`KeyError`.. If we did this, then (1) and (2) from
https://github.com/apache/spark/pull/21427#issuecomment-392070950 could
continue to work but the following would no longer work (this seems pretty
silly though):
```
@pandas_udf("a string, b float", GROUPED_MAP)
def foo(pdf):
return pd.DataFrame({'b': ['hi'], 'a': [1.0]})
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
