GitHub user NarineK opened a pull request:
https://github.com/apache/spark/pull/14431
[SPARK-16258][SparkR][WIP] Gapply add key attach option
## What changes were proposed in this pull request?
The following pull request addresses the new feature request described in
SPARK-16258.
It automatically('by default') appends grouping keys to output `DataFrame`.
I've also tried to solve the problem by adding an optional flag in `gapply`
that states if the key is required or not. However, the optional flag needs to
be passed as an argument through a number of methods which is not necessarily
elegant and leads to some issues such as "The number of parameters should not
exceed 10" in '..../logical/object.scala:290'
Since this pull request already appends the grouping key automatically, I
was thinking if we really need to pass 'key' as R functions input argument -
function(key, x) {....} Isn't it superfluous ?
I'd be happy to hear your thoughts on that.
Thanks!
## How was this patch tested?
Test cases in R.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/NarineK/spark gapply-add-key-attach-option
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14431.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14431
----
commit 29d8a5c6c22202cdf7d6cc44f1d6cbeca5946918
Author: Narine Kokhlikyan <[email protected]>
Date: 2016-06-20T22:12:11Z
Fixed duplicated documentation problem + separated documentation for dapply
and dapplyCollect
commit 60491b98d5cea46fc752e09fcc8306bcfc9423d0
Author: Narine Kokhlikyan <[email protected]>
Date: 2016-07-21T05:54:10Z
merge with master
commit 575fcf82ff01f14b1186981c4188b6f9094e6bbc
Author: Narine Kokhlikyan <[email protected]>
Date: 2016-08-01T00:40:07Z
gapply: prepend key to output
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
