Jacek Laskowski created SPARK-20912:
---------------------------------------
Summary: map function with columns as strings
Key: SPARK-20912
URL: https://issues.apache.org/jira/browse/SPARK-20912
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 2.3.0
Reporter: Jacek Laskowski
Priority: Trivial
There's only {{map}} function that accepts {{Column}} values only. It'd be very
helpful to have a variant that accepted {{String}} for columns like {{array}}
or {{struct}}.
{code}
scala> val kvs = Seq(("key", "value")).toDF("k", "v")
kvs: org.apache.spark.sql.DataFrame = [k: string, v: string]
scala> kvs.printSchema
root
|-- k: string (nullable = true)
|-- v: string (nullable = true)
scala> kvs.withColumn("map", map("k", "v")).show
<console>:26: error: type mismatch;
found : String("k")
required: org.apache.spark.sql.Column
kvs.withColumn("map", map("k", "v")).show
^
<console>:26: error: type mismatch;
found : String("v")
required: org.apache.spark.sql.Column
kvs.withColumn("map", map("k", "v")).show
^
// note $ to create Columns per string
// not very dev-friendly
scala> kvs.withColumn("map", map($"k", $"v")).show
+---+-----+-----------------+
| k| v| map|
+---+-----+-----------------+
|key|value|Map(key -> value)|
+---+-----+-----------------+
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
