adiranjan25 opened a new pull request, #59951: URL: https://github.com/apache/spark/pull/59951
## What changes were proposed in this pull request? Fix pandas-on-Spark `DataFrame.drop_duplicates()` and `DataFrame.duplicated()` when column names contain periods, such as `customer.id`. Previously, the shared `_mark_duplicates` helper passed raw column names into `Window.partitionBy()`, causing Spark to interpret dotted column names as nested field references and raise `UNRESOLVED_COLUMN.WITH_SUGGESTION`. The fix uses `scol_for()` to correctly reference literal column names. A regression test was added for both methods with `keep="first"`, `keep="last"`, and `keep=False`. ## How was this patch tested? - Apache Spark Maven build: SUCCESS - Original bug reproduced before the fix - Bug verified as fixed afterward - New regression test: PASS - Existing `test_drop_duplicates`: PASS - Existing `test_duplicated`: PASS - Final result: 3 tests passed - `git diff --check`: PASS JIRA: https://issues.apache.org/jira/browse/SPARK-60643 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
