MaxGekk commented on pull request #32087: URL: https://github.com/apache/spark/pull/32087#issuecomment-816451921
I am trying to enable ANSI intervals in `ThriftServerQueryTestSuite`, see https://github.com/apache/spark/pull/32099. I made the following changes at Spark side: ```diff diff --git a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala index 8ca0ab91a7..4b080a99f6 100644 --- a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala +++ b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala @@ -120,7 +120,8 @@ private[hive] class SparkExecuteStatementOperation( (from.getAs[CalendarInterval](ordinal), CalendarIntervalType), false, timeFormatters) - case _: ArrayType | _: StructType | _: MapType | _: UserDefinedType[_] => + case _: ArrayType | _: StructType | _: MapType | _: UserDefinedType[_] | + YearMonthIntervalType | DayTimeIntervalType => to += toHiveString((from.get(ordinal), dataTypes(ordinal)), false, timeFormatters) } } @@ -377,6 +378,8 @@ object SparkExecuteStatementOperation { val attrTypeString = field.dataType match { case NullType => "void" case CalendarIntervalType => StringType.catalogString + case YearMonthIntervalType => "INTERVAL_YEAR_MONTH" + case DayTimeIntervalType => "INTERVAL_DAY_TIME" case other => other.catalogString } new FieldSchema(field.name, attrTypeString, field.getComment.getOrElse("")) ``` but the interval values that sent via JDBC are not recognized by Hive lib: ```java java.lang.IllegalArgumentException: Interval string does not match day-time format of 'd h:m:s.n': INTERVAL '0 16:00:00' DAY TO SECOND at org.apache.hadoop.hive.common.type.HiveIntervalDayTime.valueOf(HiveIntervalDayTime.java:234) at org.apache.hive.jdbc.HiveBaseResultSet.evaluate(HiveBaseResultSet.java:452) at org.apache.hive.jdbc.HiveBaseResultSet.getColumnValue(HiveBaseResultSet.java:424) at org.apache.hive.jdbc.HiveBaseResultSet.getObject(HiveBaseResultSet.java:464) ``` It seems we have to especially format intervals for Thrift server (and Hive libs) in the format: https://github.com/apache/hive/blob/7b3ecf617a6d46f48a3b6f77e0339fd4ad95a420/storage-api/src/java/org/apache/hadoop/hive/common/type/HiveIntervalDayTime.java#L244-L245 cc @cloud-fan @HyukjinKwon @@srielau -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
