GitHub user saurfang opened a pull request:
https://github.com/apache/spark/pull/9234
[SPARK-8277][SPARKR] Faster createDataFrame using mapply
With a single loop using `mapply`, I'm able to create DataFrame much faster
from R data.frame.
Please see benchmark results and code:
```r
Unit: milliseconds
expr min lq mean median uq max neval
old 278.842059 316.711990 338.708011 327.762371 345.78249 549.76514 100
new 6.545662 7.993431 9.596966 8.636927 10.16429 22.94926 100
```
```r
library(nycflights13)
library(microbenchmark)
data <- head(flights, n = 1000)
# get rid of factor type
dropFactor <- function(x) {
if (is.factor(x)) {
as.character(x)
} else {
x
}
}
createDataFrameNew <- function(data) {
data <- data.frame(lapply(data, dropFactor), stringsAsFactors = FALSE)
do.call(mapply, c(list, as.list(data), SIMPLIFY = FALSE))
}
createDataFrameOld <- function(data) {
n <- nrow(data)
m <- ncol(data)
lapply(1:n, function(i) {
lapply(1:m, function(j) { dropFactor(data[i,j]) })
})
}
microbenchmark(old = createDataFrameOld(data), new =
createDataFrameNew(data))
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/saurfang/spark createdataframe
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9234.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9234
----
commit 0bf8b7c932b609252a76a69d63aff8fb705730c3
Author: Forest Fang <[email protected]>
Date: 2015-10-22T21:59:42Z
use mapply for faster createDataFrame
commit 397438d02d3f8c540e9e45b7a355f813f1970870
Author: Forest Fang <[email protected]>
Date: 2015-10-22T22:03:43Z
drop no longer used m,n
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
