anew commented on code in PR #57412:
URL: https://github.com/apache/spark/pull/57412#discussion_r3632279011
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala:
##########
@@ -478,17 +478,39 @@ private[connect] object PipelinesHandler extends Logging {
case proto.PipelineCommand.DefineFlow.SCDType.SCD_TYPE_1 |
proto.PipelineCommand.DefineFlow.SCDType.SCD_TYPE_UNSPECIFIED =>
ScdType.Type1
+ case proto.PipelineCommand.DefineFlow.SCDType.SCD_TYPE_2 =>
+ ScdType.Type2
case other =>
throw new UnsupportedOperationException(s"Unsupported AutoCDC SCD
type: $other")
}
+ // SCD2-only history-tracking column selection. Mirrors the
column_list/except_column_list
+ // handling above: at most one side may be non-empty, and an empty
selection means "track
+ // every eligible column" (None).
+ val trackHistorySelection: Option[ColumnSelection] = {
Review Comment:
Good catch — you're right that the validation was asymmetric. I've mirrored
the "track-history requires SCD2" check server-side in buildAutoCdcFlow (new
commit), so a raw Connect client now gets a clean
AUTOCDC_TRACK_HISTORY_REQUIRES_SCD2 AnalysisException instead of the
INTERNAL_ERROR from the ChangeArgs guard. Added raw-proto tests in
SparkDeclarativePipelinesServerSuite covering both this
guard and the both-lists case (the Python client rejects these client-side,
so the raw stub is needed to reach the server path). The Python-side check
stays as INVALID_MULTIPLE_ARGUMENT_CONDITIONS since it's a different exception
hierarchy, with a parallel message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
