HeartSaVioR commented on code in PR #42940:
URL: https://github.com/apache/spark/pull/42940#discussion_r1329415427
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -2180,6 +2180,17 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val STREAMING_TRIGGER_AVAILABLE_NOW_WRAPPER_ENABLED =
+ buildConf("spark.sql.streaming.triggerAvailableNowWrapper.enabled")
+ .internal()
+ .doc("Whether to use the wrapper implementation of Trigger.AvailableNow
if the source " +
+ "does not support Trigger.AvailableNow. Enabling this allows the
benefits of " +
+ "Trigger.AvailableNow with sources which don't support it, but some
sources " +
+ "may show unexpected behavior including duplication, data loss, etc.
So use with " +
+ "extreme care! The ideal direction is to persuade developers of
source(s) to " +
+ "support Trigger.AvailableNow.")
Review Comment:
I'd say we are going to break a bunch of 3rd party data sources then, which
is what we have been avoided so far.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
