rangadi commented on code in PR #40834:
URL: https://github.com/apache/spark/pull/40834#discussion_r1171485025
##########
connector/connect/common/src/main/protobuf/spark/connect/relations.proto:
##########
@@ -68,6 +68,7 @@ message Relation {
WithWatermark with_watermark = 33;
ApplyInPandasWithState apply_in_pandas_with_state = 34;
HtmlString html_string = 35;
+ Deduplicate deduplicate_within_watermark = 36;
Review Comment:
This is not required, right? The flag in Deduplicate will indicate this.
##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##########
@@ -90,6 +90,8 @@ class SparkConnectPlanner(val session: SparkSession) {
case proto.Relation.RelTypeCase.TAIL => transformTail(rel.getTail)
case proto.Relation.RelTypeCase.JOIN => transformJoin(rel.getJoin)
case proto.Relation.RelTypeCase.DEDUPLICATE =>
transformDeduplicate(rel.getDeduplicate)
+ case proto.Relation.RelTypeCase.DEDUPLICATE_WITHIN_WATERMARK =>
Review Comment:
Connected to the proto comment. We don't need this field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
