dongjoon-hyun opened a new pull request, #586: URL: https://github.com/apache/spark-connect-swift/pull/586
### What changes were proposed in this pull request? This PR aims to fix `SparkConnectClient.getDropDuplicates` to set the `within_watermark` field of the `Deduplicate` relation by adding `deduplicate.withinWatermark = withinWatermark`. ### Why are the changes needed? Previously, `getDropDuplicates` accepted the `withinWatermark` parameter but never propagated it to `Spark_Connect_Deduplicate`. As a result, the public `DataFrame.dropDuplicatesWithinWatermark(...)` API silently behaved like `dropDuplicates(...)`. For streaming queries, this gives the wrong semantics: the server keeps unbounded deduplication state instead of the watermark-bounded state of `DeduplicateWithinWatermark`. ### Does this PR introduce _any_ user-facing change? Yes, this is a bug fix. `dropDuplicatesWithinWatermark` now sends `within_watermark = true` to the Spark Connect server, consistent with PySpark and Scala Spark Connect clients. For batch `DataFrame`s, the result is unchanged because Spark treats `DeduplicateWithinWatermark` like `Deduplicate` in batch queries. ### How was this patch tested? Pass the CIs with the newly added plan-level test case, `DataFrameInternalTests.dropDuplicatesWithinWatermarkPlan`, which does not require a Spark Connect server. It fails without this fix and passes with it. ``` $ swift test --no-parallel --filter DataFrameInternalTests/dropDuplicatesWithinWatermarkPlan ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
