HeartSaVioR commented on code in PR #49751:
URL: https://github.com/apache/spark/pull/49751#discussion_r1946034847
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StatefulProcessorHandleImpl.scala:
##########
@@ -363,18 +363,36 @@ class DriverStatefulProcessorHandleImpl(timeMode:
TimeMode, keyExprEnc: Expressi
addTimerColFamily()
}
+ /**
+ * This method returns all column family schemas, and checks and enforces
nullability
+ * if need be. The nullability check and set is only set to true when Avro
is enabled.
+ * @param shouldCheckNullable Whether we need to check the nullability. This
is set to
+ * true when using Python, as this is the only
avenue through
+ * which users can set nullability
+ * @param shouldSetNullable Whether we need to set the fields as nullable.
This is set to
+ * true when using Scala, as case classes are set to
Review Comment:
Maybe it is only true for primitive type? If then it might make sense, like
an optimization for the type which could never have null. If you see
non-nullable to String or so, this should be a bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
