Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/3208#discussion_r20705901
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala ---
@@ -339,18 +339,15 @@ class SqlParser extends AbstractSparkSQLParser {
| floatLit ^^ { f => Literal(f.toDouble) }
)
- private val longMax = BigDecimal(s"${Long.MaxValue}")
- private val longMin = BigDecimal(s"${Long.MinValue}")
- private val intMax = BigDecimal(s"${Int.MaxValue}")
- private val intMin = BigDecimal(s"${Int.MinValue}")
-
private def toNarrowestIntegerType(value: String) = {
val bigIntValue = BigDecimal(value)
bigIntValue match {
- case v if v < longMin || v > longMax => v
- case v if v < intMin || v > intMax => v.toLong
- case v => v.toInt
+ case v if bigIntValue.isValidByte => v.toByteExact
+ case v if bigIntValue.isValidShort => v.toShortExact
+ case v if bigIntValue.isValidInt => v.toIntExact
+ case v if bigIntValue.isValidLong => v.toLongExact
+ case v => v
--- End diff --
I have no idea if any other SQL systems do like this. The only extra
complexity it adds is the type casting in `GetItem` and `Substring` as I see.
But I can not tell if the memory benefits is worth doing this. It may depend on
how often the type casting is happened. If the time complexity issue is a big
concern here, I can remove the byte and short types. Please let me know that.
Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
