Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/13727#discussion_r67572068
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -368,6 +397,63 @@ class DataFrameReader private[sql](sparkSession:
SparkSession) extends Logging {
* <li>`maxCharsPerColumn` (default `1000000`): defines the maximum
number of characters allowed
* for any given value being read.</li>
* <li>`mode` (default `PERMISSIVE`): allows a mode for dealing with
corrupt records
+ * during parsing.</li>
+ * <ul>
+ * <li>`PERMISSIVE` : sets other fields to `null` when it meets a
corrupted record. When
+ * a schema is set by user, it sets `null` for extra fields.</li>
+ * <li>`DROPMALFORMED` : ignores the whole corrupted records.</li>
+ * <li>`FAILFAST` : throws an exception when it meets corrupted
records.</li>
+ * </ul>
+ *
+ * @since 2.0.0
+ */
+ def csv(path: String): DataFrame = {
+ // This method ensures that calls that explicit need single argument
works, see SPARK-16009
+ csv(Seq(path): _*)
+ }
+
+ /**
+ * Loads a CSV file and returns the result as a [[DataFrame]].
+ *
+ * This function will go through the input once to determine the input
schema if `inferSchema`
+ * is enabled. To avoid going through the entire data once, disable
`inferSchema` option or
+ * specify the schema explicitly using [[schema]].
+ *
+ * You can set the following CSV-specific options to deal with CSV files:
+ * <li>`sep` (default `,`): sets the single character as a separator for
each
+ * field and value.</li>
+ * <li>`encoding` (default `UTF-8`): decodes the CSV files by the given
encoding
+ * type.</li>
+ * <li>`quote` (default `"`): sets the single character used for
escaping quoted values where
+ * the separator can be part of the value. If you would like to turn off
quotations, you need to
+ * set not `null` but an empty string. This behaviour is different form
+ * `com.databricks.spark.csv`.</li>
+ * <li>`escape` (default `\`): sets the single character used for
escaping quotes inside
+ * an already quoted value.</li>
+ * <li>`comment` (default empty string): sets the single character used
for skipping lines
+ * beginning with this character. By default, it is disabled.</li>
+ * <li>`header` (default `false`): uses the first line as names of
columns.</li>
+ * <li>`inferSchema` (default `false`): infers the input schema
automatically from data. It
+ * requires one extra pass over the data.</li>
+ * <li>`ignoreLeadingWhiteSpace` (default `false`): defines whether or
not leading whitespaces
+ * from values being read should be skipped.</li>
+ * <li>`ignoreTrailingWhiteSpace` (default `false`): defines whether or
not trailing
+ * whitespaces from values being read should be skipped.</li>
+ * <li>`nullValue` (default empty string): sets the string
representation of a null value.</li>
+ * <li>`nanValue` (default `NaN`): sets the string representation of a
non-number" value.</li>
+ * <li>`positiveInf` (default `Inf`): sets the string representation of
a positive infinity
+ * value.</li>
+ * <li>`negativeInf` (default `-Inf`): sets the string representation of
a negative infinity
+ * value.</li>
+ * <li>`dateFormat` (default `null`): sets the string that indicates a
date format. Custom date
+ * formats follow the formats at `java.text.SimpleDateFormat`. This
applies to both date type
+ * and timestamp type. By default, it is `null` which means trying to
parse times and date by
+ * `java.sql.Timestamp.valueOf()` and `java.sql.Date.valueOf()`.</li>
+ * <li>`maxColumns` (default `20480`): defines a hard limit of how many
columns
+ * a record can have.</li>
+ * <li>`maxCharsPerColumn` (default `1000000`): defines the maximum
number of characters allowed
+ * for any given value being read.</li>
+ * <li>`mode` (default `PERMISSIVE`): allows a mode for dealing with
corrupt records
--- End diff --
This can be addressed in a follow up, but I don't think we should duplicate
docs cause they are going to get out of sync. I'd have one canonical one and
the other link to it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
