uros-b commented on code in PR #57412:
URL: https://github.com/apache/spark/pull/57412#discussion_r3631587610
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala:
##########
@@ -478,17 +478,39 @@ private[connect] object PipelinesHandler extends Logging {
case proto.PipelineCommand.DefineFlow.SCDType.SCD_TYPE_1 |
proto.PipelineCommand.DefineFlow.SCDType.SCD_TYPE_UNSPECIFIED =>
ScdType.Type1
+ case proto.PipelineCommand.DefineFlow.SCDType.SCD_TYPE_2 =>
+ ScdType.Type2
case other =>
throw new UnsupportedOperationException(s"Unsupported AutoCDC SCD
type: $other")
}
+ // SCD2-only history-tracking column selection. Mirrors the
column_list/except_column_list
+ // handling above: at most one side may be non-empty, and an empty
selection means "track
+ // every eligible column" (None).
+ val trackHistorySelection: Option[ColumnSelection] = {
Review Comment:
Note: Server-side validation is asymmetric: the handler re-checks the
both-lists case (AUTOCDC_BOTH_TRACK_HISTORY_...) but does NOT mirror the
"track-history requires SCD2" check. A raw/non-Python Connect client sending
SCD_TYPE_1 (a currently-functional path) with a populated track-history list
reaches ChangeArgs, whose validateTrackHistoryOnlyForScd2 guard throws
SparkException.internalError (INTERNAL_ERROR) rather than a clean user-facing
AnalysisException (verified against master: the guard's own scaladoc designates
the API layer as the validation point). Python clients are shielded by the new
api.py check, so this is a defense-in-depth/UX gap, not a functional bug; but
since the sibling both-lists check IS mirrored here, mirroring the
requires-SCD2 check too would be consistent and is worth deciding before de-WIP.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
