AngersZhuuuu commented on a change in pull request #29421:
URL: https://github.com/apache/spark/pull/29421#discussion_r475370426
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala
##########
@@ -783,7 +783,9 @@ class SparkSqlAstBuilder(conf: SQLConf) extends
AstBuilder(conf) {
// Use default (serde) format.
val name = conf.getConfString("hive.script.serde",
"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe")
- val props = Seq("field.delim" -> "\t")
+ val props = Seq(
+ "field.delim" -> "\t",
+ "serialization.last.column.takes.rest" -> "true")
Review comment:
> We don't respect `WITH SERDEPROPERTIES`?
```
rowFormat
: ROW FORMAT SERDE name=STRING (WITH SERDEPROPERTIES
props=tablePropertyList)? #rowFormatSerde
| ROW FORMAT DELIMITED
(FIELDS TERMINATED BY fieldsTerminatedBy=STRING (ESCAPED BY
escapedBy=STRING)?)?
(COLLECTION ITEMS TERMINATED BY collectionItemsTerminatedBy=STRING)?
(MAP KEYS TERMINATED BY keysTerminatedBy=STRING)?
(LINES TERMINATED BY linesSeparatedBy=STRING)?
(NULL DEFINED AS nullDefinedAs=STRING)?
#rowFormatDelimited
;
```
`WITH SERDEPROPERTIES` must use with ROW FORMAT SERDE, and here is for case
without `ROW FORMAT SERDE` or `ROW FORMAT DELIMITED`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
