Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/22379#discussion_r224843696
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3854,6 +3854,38 @@ object functions {
@scala.annotation.varargs
def map_concat(cols: Column*): Column = withExpr {
MapConcat(cols.map(_.expr)) }
+ /**
+ * Parses a column containing a CSV string into a `StructType` with the
specified schema.
+ * Returns `null`, in the case of an unparseable string.
+ *
+ * @param e a string column containing CSV data.
+ * @param schema the schema to use when parsing the CSV string
+ * @param options options to control how the CSV is parsed. accepts the
same options and the
+ * CSV data source.
+ *
+ * @group collection_funcs
+ * @since 3.0.0
+ */
+ def from_csv(e: Column, schema: StructType, options: Map[String,
String]): Column = withExpr {
+ CsvToStructs(schema, options, e.expr)
+ }
+
+ /**
+ * (Java-specific) Parses a column containing a CSV string into a
`StructType`
+ * with the specified schema. Returns `null`, in the case of an
unparseable string.
+ *
+ * @param e a string column containing CSV data.
+ * @param schema the schema to use when parsing the CSV string
+ * @param options options to control how the CSV is parsed. accepts the
same options and the
+ * CSV data source.
+ *
+ * @group collection_funcs
+ * @since 3.0.0
+ */
+ def from_csv(e: Column, schema: String, options: java.util.Map[String,
String]): Column = {
--- End diff --
What's stopped me to do that was I didn't know how to support the `Column`
type in R. I even opened JIRA ticket for a similar issue related to
`schema_of_json`: https://issues.apache.org/jira/browse/SPARK-25446 . The
`from_json()` accepts the schema as `characterOrstructType` and how to extend
to support the `Column` type as well not clear to me:
https://github.com/apache/spark/blob/17781d75308c328b11cab3658ca4f358539414f2/R/pkg/R/functions.R#L2186
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
