Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/19339#discussion_r140771519
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -456,6 +456,40 @@ class DataFrameReader private[sql](sparkSession:
SparkSession) extends Logging {
}
/**
+ * Loads a `JavaRDD[String]` storing storing CSV rows and returns the
result as a `DataFrame`.
+ *
+ * If the schema is not specified using `schema` function and
`inferSchema` option is enabled,
+ * this function goes through the input once to determine the input
schema.
+ *
+ * If the schema is not specified using `schema` function and
`inferSchema` option is disabled,
+ * it determines the columns as string types and it reads only the first
line to determine the
+ * names and the number of fields.
+ *
+ * @param csvRDD input RDD with one CSV row per record
+ * @since 2.2.0
+ */
+ @deprecated("Use csv(Dataset[String]) instead.", "2.2.0")
+ def csv(csvRDD: JavaRDD[String]): DataFrame = csv(csvRDD.rdd)
+
+ /**
+ * Loads a `RDD[String]` storing storing CSV rows and returns the result
as a `DataFrame`.
+ *
+ * If the schema is not specified using `schema` function and
`inferSchema` option is enabled,
+ * this function goes through the input once to determine the input
schema.
+ *
+ * If the schema is not specified using `schema` function and
`inferSchema` option is disabled,
+ * it determines the columns as string types and it reads only the first
line to determine the
+ * names and the number of fields.
+ *
+ * @param csvRDD input RDD with one CSV row per record
+ * @since 2.2.0
+ */
+ @deprecated("Use csv(Dataset[String]) instead.", "2.2.0")
+ def csv(csvRDD: RDD[String]): DataFrame = {
--- End diff --
Wait ... I think we shouldn't introduce an RDD API in Scala side. I was
thinking doing this within Python-side, or _maybe_ adding a private wrapper in
Scala side if required .. Will takr a closer look tomorrow (KTS).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
