Maciej Szymkiewicz created SPARK-19454:
------------------------------------------
Summary: Improve DataFrame.replace API
Key: SPARK-19454
URL: https://issues.apache.org/jira/browse/SPARK-19454
Project: Spark
Issue Type: Improvement
Components: PySpark, SQL
Affects Versions: 2.1.0, 2.0.0, 1.6.0, 1.5.0, 2.2.0
Reporter: Maciej Szymkiewicz
Current implementation suffers from following issues:
- It is possible to use {{dict}} as {{to_replace}}, but we cannot skip or use
{{None}} as the value {{value}} (although it is ignored). This requires passing
"magic" values:
{code}
df = sc.parallelize([("Alice", 1, 3.0)]).toDF()
df.replace({"Alice": "Bob"}, 1)
{code}
- Code doesn't check if provided types are correct. This can lead to exception
in Py4j (harder to diagnose):
{code}
df.replace({"Alice": 1}, 1)
{code}
or silent failures (with bundled Py4j version):
{code}
df.replace({1: 2, 3.0: 4.1, "a": "b"}, 1)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
