metanil commented on code in PR #59760:
URL: https://github.com/apache/spark/pull/59760#discussion_r4445705151
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TransformExpression.scala:
##########
@@ -202,4 +324,62 @@ object TransformExpression {
case t: TransformExpression => t.reducedWith.isDefined
case _ => false
}
+
+ /**
+ * `argument` at `declaredType`, or `None` when the function could not take
it. See
+ * [[TransformExpression.evaluableFunction]].
+ *
+ * An argument goes as it is when it is stored the way the declared type is
(`storedAlike`),
+ * which is how a function read it before this check. A numeric argument
narrower than declared
+ * (`UpCastRule.legalNumericPrecedence`), e.g. a `SMALLINT` against an `INT`
or an `INT` against a
+ * `DOUBLE`, is held as another JVM type, so it is cast, which gives the
value a magic `invoke`
+ * got from Java's widening before this check, and needs no time zone.
Anything else is refused,
+ * a `DATE` against a `TIMESTAMP` in particular, whose cast takes the
session time zone.
+ */
+ private def atDeclaredType(argument: Expression, declaredType: DataType):
Option[Expression] =
+ (argument.dataType, declaredType) match {
+ case (argumentType, _) if storedAlike(argumentType, declaredType, nested
= false) =>
+ Some(argument)
+ case (argumentType, _) if
UpCastRule.legalNumericPrecedence(argumentType, declaredType) =>
+ Some(Cast(argument, declaredType))
+ case _ =>
+ None
+ }
+
+ /**
+ * Whether a value of type `from` is stored the way one of type `to` is, so
that a function
+ * declaring `to` reads it as it is:
+ * - two types with one physical type, e.g. a `VARCHAR` and a string of the
same collation, a
+ * `DATE` and an `INT`, or a `TIMESTAMP` and a `BIGINT`. A user-defined
type is stored as its
+ * `sqlType`;
+ * - two structs, arrays or maps whose fields, elements, keys and values
are stored alike, by
+ * position. Field names and nullability do not change how a value is
stored. A function
+ * declaring a non-nullable field gets the values a nullable one holds;
+ * - two decimals of one scale, so that the unscaled digits a function
reads are the same. At the
+ * top level the function gets the `Decimal` itself, whatever precision
it declares, e.g. a
+ * `DECIMAL(12,2)` key of a cogroup against a function declaring
`DECIMAL(10,2)`, which a
+ * cogroup does not coerce. Nested in an unsafe row, a decimal is read at
the declared
+ * precision: it is stored as a `long` up to `Decimal.MAX_LONG_DIGITS`
digits and as bytes
+ * above, so there `to` needs no less precision, and both on one side of
that.
+ */
+ private def storedAlike(from: DataType, to: DataType, nested: Boolean):
Boolean =
+ (from, to) match {
+ case _ if from == to => true
+ case (udt: UserDefinedType[_], _) => storedAlike(udt.sqlType, to, nested)
+ case (_, udt: UserDefinedType[_]) => storedAlike(from, udt.sqlType,
nested)
+ case (f: StructType, t: StructType) =>
+ f.length == t.length && f.fields.zip(t.fields).forall { case (ff, tf)
=>
+ storedAlike(ff.dataType, tf.dataType, nested = true)
+ }
+ case (f: ArrayType, t: ArrayType) => storedAlike(f.elementType,
t.elementType, nested = true)
+ case (f: MapType, t: MapType) =>
+ storedAlike(f.keyType, t.keyType, nested = true) &&
+ storedAlike(f.valueType, t.valueType, nested = true)
+ case (f: DecimalType, t: DecimalType) =>
+ f.scale == t.scale && (!nested || (f.precision <= t.precision &&
+ DecimalType.isByteArrayDecimalType(f) ==
DecimalType.isByteArrayDecimalType(t)))
+ case _ =>
+ val physical = PhysicalDataType(from)
+ physical != UninitializedPhysicalType && physical ==
PhysicalDataType(to)
Review Comment:
On master, a years declaring `TIMESTAMP` over `TIME` '10:00' gives year 1,
and a months declaring `DATE` over a 30-year `YEAR TO MONTH` interval gives
month 11. The fix will follow your rule: two `datetime` or `interval` types go
as they are only when equal, with `TIMESTAMP_NTZ/TIMESTAMP` allowed. I'm
thinking opening it on top of this PR or should I wait for this PR to get
merged first?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
