chris-twiner commented on code in PR #50023:
URL: https://github.com/apache/spark/pull/50023#discussion_r1971221056
##########
sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala:
##########
@@ -2841,6 +2860,265 @@ class DatasetSuite extends QueryTest
checkDataset(Seq(seqMutableSet).toDS(), seqMutableSet)
checkDataset(Seq(mapMutableSet).toDS(), mapMutableSet)
}
+
+ // below tests are related to SPARK-49960 and TransformingEncoder usage
+ test("Incorrect derived nullability with TransformingEncoder - non
nullable") {
+ val sparkI = spark
+ type T = Tuple2[Seq[Seq[Int]], Seq[Int]]
+ val data: Seq[T] = Seq( ( Seq( Seq(1, 2, 3) ), Seq(1, 2, 3) ) )
+ // for reference only
+ val sparkDataTypeOG = {
+ import sparkI.implicits._
Review Comment:
I've removed all of the "for reference only" sections, they were only for
debug and I should have removed the beforehand
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
