JiaqiWang18 commented on code in PR #51507:
URL: https://github.com/apache/spark/pull/51507#discussion_r2211686949
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/PipelineUpdateContextImpl.scala:
##########
@@ -24,10 +24,17 @@ import
org.apache.spark.sql.pipelines.logging.{FlowProgressEventLogger, Pipeline
* An implementation of the PipelineUpdateContext trait used in production.
* @param unresolvedGraph The graph (unresolved) to be executed in this update.
* @param eventCallback A callback function to be called when an event is
added to the event buffer.
+ * @param refreshTables Filter for which tables should be refreshed when
performing this update.
+ * @param fullRefreshTables Filter for which tables should be full refreshed
+ * when performing this update.
+ * @param resetCheckpointFlows Filter for which flows should be reset.
*/
class PipelineUpdateContextImpl(
override val unresolvedGraph: DataflowGraph,
- override val eventCallback: PipelineEvent => Unit
+ override val eventCallback: PipelineEvent => Unit,
+ override val refreshTables: TableFilter = AllTables,
+ override val fullRefreshTables: TableFilter = NoTables,
+ override val resetCheckpointFlows: FlowFilter = NoFlows
Review Comment:
`resetCheckpointFlows` was accidentally moved into the constructor. I will
move it back.
It being set to `NoTables` won't conflict with the `fullRefreshTables`
argument because [this
logic](https://github.com/apache/spark/blob/master/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/PipelineUpdateContext.scala#L41)
will include the flows for the full refresh tables directly so we don't need
to set it manually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
