Github user joseph-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/19926#discussion_r156468624
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala
---
@@ -71,27 +68,29 @@ class StreamExecution(
import org.apache.spark.sql.streaming.StreamingQueryListener._
- private val pollingDelayMs =
sparkSession.sessionState.conf.streamingPollingDelay
+ protected val pollingDelayMs: Long =
sparkSession.sessionState.conf.streamingPollingDelay
- private val minBatchesToRetain =
sparkSession.sessionState.conf.minBatchesToRetain
+ protected val minBatchesToRetain: Int =
sparkSession.sessionState.conf.minBatchesToRetain
--- End diff --
We may want to tweak the variable name, but continuous processing will
still need to know how long it should retain commit and offset log entries.
Unfortunately we're stuck with the config name, and I don't think it makes
sense to introduce a second parallel one doing the same thing.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
