HeartSaVioR commented on a change in pull request #27747: [SPARK-30993][SQL]
Use its sql type for UDT when checking the type of length (fixed/var) or mutable
URL: https://github.com/apache/spark/pull/27747#discussion_r386157157
##########
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeRowJoinerSuite.scala
##########
@@ -99,6 +101,23 @@ class GenerateUnsafeRowJoinerSuite extends SparkFunSuite {
testConcatOnce(N, N, variable)
}
+ test("SPARK-30993: UserDefinedType matched to fixed length SQL type
shouldn't be corrupted") {
+ val schema1 = new StructType(Array(
+ StructField("date", new WrappedDateTimeUDT),
+ StructField("s", StringType),
+ StructField("i", IntegerType)))
+ val proj1 = UnsafeProjection.create(schema1.fields.map(_.dataType))
+ val intRow1 = new GenericInternalRow(Array[Any](
+ LocalDateTime.now().toEpochSecond(ZoneOffset.UTC),
+ UTF8String.fromString("hello"), 1))
+
+ val schema2 = new StructType(Array(StructField("i", IntegerType)))
+ val proj2 = UnsafeProjection.create(schema2.fields.map(_.dataType))
+ val intRow2 = new GenericInternalRow(Array[Any](2))
+
+ testConcat(schema1, proj1.apply(intRow1), schema2, proj2.apply(intRow2))
Review comment:
I've left comment regarding this.
https://github.com/apache/spark/pull/27747#discussion_r386154074
It's no-brainer to have E2E test with SS mapWithState as we already got
report from actual code (and I confirmed it fails), but if we would like to
have simpler batch query to reproduce I may need to try out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
