AnishMahto commented on code in PR #51507:
URL: https://github.com/apache/spark/pull/51507#discussion_r2211741560
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/PipelineUpdateContextImpl.scala:
##########
@@ -24,10 +24,17 @@ import
org.apache.spark.sql.pipelines.logging.{FlowProgressEventLogger, Pipeline
* An implementation of the PipelineUpdateContext trait used in production.
* @param unresolvedGraph The graph (unresolved) to be executed in this update.
* @param eventCallback A callback function to be called when an event is
added to the event buffer.
+ * @param refreshTables Filter for which tables should be refreshed when
performing this update.
+ * @param fullRefreshTables Filter for which tables should be full refreshed
+ * when performing this update.
+ * @param resetCheckpointFlows Filter for which flows should be reset.
*/
class PipelineUpdateContextImpl(
override val unresolvedGraph: DataflowGraph,
- override val eventCallback: PipelineEvent => Unit
+ override val eventCallback: PipelineEvent => Unit,
+ override val refreshTables: TableFilter = AllTables,
+ override val fullRefreshTables: TableFilter = NoTables,
+ override val resetCheckpointFlows: FlowFilter = NoFlows
Review Comment:
I see, so we do a union of this argument and the deduced flow refreshes from
`refreshTables` and `fullRefreshTables`.
Why do we need `resetCheckpointFlows`? Is it possible it was added before
solely for testing, and its redundant now because we're propagating
`refreshTables` and `fullRefreshTables`? Or is there any non-test use case for
this field? If not, we should remove it and refactor current references
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
