dongjoon-hyun commented on a change in pull request #23466: [SPARK-26547][SQL]
Remove duplicate toHiveString from HiveUtils
URL: https://github.com/apache/spark/pull/23466#discussion_r245491587
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -444,52 +444,6 @@ private[spark] object HiveUtils extends Logging {
propMap.toMap
}
- protected val primitiveTypes =
- Seq(StringType, IntegerType, LongType, DoubleType, FloatType, BooleanType,
ByteType,
- ShortType, DateType, TimestampType, BinaryType)
-
- protected[sql] def toHiveString(a: (Any, DataType)): String = a match {
- case (struct: Row, StructType(fields)) =>
- struct.toSeq.zip(fields).map {
- case (v, t) => s""""${t.name}":${toHiveStructString((v,
t.dataType))}"""
- }.mkString("{", ",", "}")
- case (seq: Seq[_], ArrayType(typ, _)) =>
- seq.map(v => (v, typ)).map(toHiveStructString).mkString("[", ",", "]")
- case (map: Map[_, _], MapType(kType, vType, _)) =>
- map.map {
- case (key, value) =>
- toHiveStructString((key, kType)) + ":" + toHiveStructString((value,
vType))
- }.toSeq.sorted.mkString("{", ",", "}")
- case (null, _) => "NULL"
- case (d: Int, DateType) => new DateWritable(d).toString
- case (t: Timestamp, TimestampType) => new TimestampWritable(t).toString
- case (bin: Array[Byte], BinaryType) => new String(bin,
StandardCharsets.UTF_8)
- case (decimal: java.math.BigDecimal, DecimalType()) =>
- // Hive strips trailing zeros so use its toString
- HiveDecimal.create(decimal).toString
- case (other, _ : UserDefinedType[_]) => other.toString
- case (other, tpe) if primitiveTypes contains tpe => other.toString
- }
Review comment:
Ur, actually, the deleted one has the same functionality, but has better
structures.
- `primitiveTypes` is declared outside of `toHiveString`.
- `toHiveStructString` is declared outside of `toHiveString`.
- A useful comment `// Hive strips trailing zeros so use its toString`
What about refactoring the survived one to have these better structure?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
