GitHub user mgaido91 opened a pull request:
https://github.com/apache/spark/pull/21737
[SPARK-24208][SQL] Fix attribute deduplication for FlatMapGroupsInPandas
## What changes were proposed in this pull request?
A self-join on a dataset which contains a `FlatMapGroupsInPandas` fails
because of duplicate attributes. This happens because we are not dealing with
this specific case in our `dedupAttr` rules.
The PR fix the issue by adding the management of the specific case
## How was this patch tested?
added UT + manual tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mgaido91/spark SPARK-24208
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21737.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21737
----
commit 5f325a4d4c420800c08293a135c935bb03c48c9e
Author: Marco Gaido <marcogaido91@...>
Date: 2018-07-09T15:11:18Z
[SPARK-24208][SQL] Fix attribute deduplication for FlatMapGroupsInPandas
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
