Cheng Lian created SPARK-4202:
---------------------------------
Summary: DSL support for Scala UDF
Key: SPARK-4202
URL: https://issues.apache.org/jira/browse/SPARK-4202
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 1.1.1
Reporter: Cheng Lian
Using Scala UDF with current DSL API is quite verbose, e.g.:
{code}
case class KeyValue(key: Int, value: String)
val schemaRDD = sc.parallelize(1 to 10).map(i => KeyValue(i,
i.toString)).toSchemaRDD
def foo = (a: Int, b: String) => a.toString + b
schemaRDD.select( // SELECT
Star(None), // *,
ScalaUdf( //
foo, // foo(
StringType, //
'key.attr :: 'value.attr :: Nil) // key, value
).collect() // ) FROM ...
{code}
It would be good to add a DSL syntax to simplify UDF invocation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
