AngersZhuuuu commented on a change in pull request #29421:
URL: https://github.com/apache/spark/pull/29421#discussion_r469855638
##########
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationExec.scala
##########
@@ -274,6 +275,14 @@ object HiveScriptIOSchema extends HiveInspectors {
var propsMap = serdeProps.toMap + (serdeConstants.LIST_COLUMNS ->
columns.mkString(","))
propsMap = propsMap + (serdeConstants.LIST_COLUMN_TYPES ->
columnTypesNames)
+ if
(!propsMap.contains(serdeConstants.SERIALIZATION_LAST_COLUMN_TAKES_REST)) {
Review comment:
> This fix works only in hive? How about spark-native transform?
As I have show in pr desc, spark no-serde mode is same as hive
no-serde(ROW FORMAT DELIMITED) mode.
In other words, hive serde mode(Such as LazySimpleSerde) is different from
no-serde(ROW FORMAT DELIMITED) mode.
Since in hive serde(support this config) mode this config's value default is
true
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
