Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/21435#discussion_r191107787
--- Diff: docs/sql-programming-guide.md ---
@@ -1827,6 +1827,10 @@ working with timestamps in `pandas_udf`s to get the
best performance, see
- Since Spark 2.0, Spark converts Parquet Hive tables by default for
better performance. Since Spark 2.4, Spark converts ORC Hive tables by default,
too. It means Spark uses its own ORC support by default instead of Hive SerDe.
As an example, `CREATE TABLE t(id int) STORED AS ORC` would be handled with
Hive SerDe in Spark 2.3, and in Spark 2.4, it would be converted into Spark's
ORC data source table and ORC vectorization would be applied. To set `false` to
`spark.sql.hive.convertMetastoreOrc` restores the previous behavior.
- In version 2.3 and earlier, CSV rows are considered as malformed if at
least one column value in the row is malformed. CSV parser dropped such rows in
the DROPMALFORMED mode or outputs an error in the FAILFAST mode. Since Spark
2.4, CSV row is considered as malformed only when it contains malformed column
values requested from CSV datasource, other values can be ignored. As an
example, CSV file contains the "id,name" header and one row "1234". In Spark
2.4, selection of the id column consists of a row with one column value 1234
but in Spark 2.3 and earlier it is empty in the DROPMALFORMED mode. To restore
the previous behavior, set `spark.sql.csv.parser.columnPruning.enabled` to
`false`.
+## Upgrading From Spark SQL 2.3.0 to 2.3.1 and Above
--- End diff --
sure
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
