HeartSaVioR commented on code in PR #56061:
URL: https://github.com/apache/spark/pull/56061#discussion_r3301924493
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -3440,6 +3440,23 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val STATEFUL_OPERATOR_ALWAYS_NULLABLE_OUTPUT =
+
buildConf("spark.sql.streaming.statefulOperator.alwaysNullableOutput.enabled")
+ .internal()
+ .doc("When true, every streaming stateful operator reports its output
schema with " +
+ "nullable=true on all columns (including nested struct fields, array
elements, and " +
+ "map values), the state schema is widened at every construction site,
and the state " +
+ "schema is widened at every construction site, so the existing state
schema " +
+ "compatibility check trivially passes regardless of input nullability.
" +
+ "This prevents query-optimizer decisions (e.g., PropagateEmptyRelation
dropping a " +
+ "Union branch) from flipping the state schema nullability across
microbatches or " +
+ "restarts. The effective value is pinned per query via the offset log
at batch 0, " +
+ "so pre-existing queries keep their original behavior; only newly
started queries " +
+ "pick this up.")
+ .version("4.1.0")
Review Comment:
self-review: This should be 4.3.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
