AnishMahto commented on code in PR #56042:
URL: https://github.com/apache/spark/pull/56042#discussion_r3291510420
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/CoreDataflowNodeProcessor.scala:
##########
@@ -199,9 +199,18 @@ private class FlowResolver(rawGraph: DataflowGraph) {
}
}
- private def convertResolvedToTypedFlow(
+ private def transformUnresolvedFlowToResolvedFlow(
flow: UnresolvedFlow,
funcResult: FlowFunctionResult): ResolvedFlow = {
+ flow match {
+ case acf: AutoCdcFlow => new AutoCdcMergeFlow(acf, funcResult)
Review Comment:
So this is a good question, but we don't actually need to do anything here
because:
1. It's not possible to create an MV with an AutoCDC flow input because (a)
MV's must be defined with an inline flow function, (b) AutoCDC flows must be
defined as standalone flows with a target, and (c) MVs are not allowed to have
multiple input flows. That means if an AutoCDC flow targets an MV, the MV
necessarily has at least 2 flows, and would be invalidated
2. AutoCDC flows are unique in that users don't actually get to define their
flow functions. SDP will define the AutoCDC flow function at flow registration
time, and we will define it in such a way that forces it to be streaming by
construction (i.e `spark.read_stream(source)`)
Eventually we will support AutoCDC once flows which would indeed be batch
flows, but that's not supported today - `once = false` always by construction.
As a middle ground though I'll also introduce a test demonstrating that
AutoCDC flows cannot write to MVs in either the flow execution or flow
registration PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
