Chuck Connell created SPARK-37183:
-------------------------------------
Summary: pyspark.pandas.DataFrame.map() should support .fillna()
Key: SPARK-37183
URL: https://issues.apache.org/jira/browse/SPARK-37183
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 3.2.0
Reporter: Chuck Connell
In regular pandas you can say
{quote}DF["new_column"] = DF["column"].map(some_map).fillna(DF["column"])
{quote}
In order to use the existing value if the mapping key is not found.
But this does not work in pyspark.pandas.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
