Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/16739#discussion_r98527967
--- Diff: R/pkg/R/DataFrame.R ---
@@ -680,14 +680,45 @@ setMethod("storageLevel",
storageLevelToString(callJMethod(x@sdf, "storageLevel"))
})
+#' Coalesce
+#'
+#' Returns a new SparkDataFrame that has exactly \code{numPartitions}
partitions.
+#' This operation results in a narrow dependency, e.g. if you go from 1000
partitions to 100
+#' partitions, there will not be a shuffle, instead each of the 100 new
partitions will claim 10 of
+#' the current partitions.
+#'
+#' @param numPartitions the number of partitions to use.
+#'
+#' @family SparkDataFrame functions
+#' @rdname coalesce
+#' @name coalesce
+#' @aliases coalesce,SparkDataFrame-method
+#' @seealso \link{repartition}
+#' @export
+#' @examples
+#'\dontrun{
+#' sparkR.session()
+#' path <- "path/to/file.json"
+#' df <- read.json(path)
+#' newDF <- coalesce(df, 1L)
+#'}
+#' @note coalesce(SparkDataFrame) since 2.1.1
+setMethod("coalesce",
+ signature(x = "SparkDataFrame"),
+ function(x, numPartitions) {
+ stopifnot(is.numeric(numPartitions))
--- End diff --
it's being coerce into integer - the reason we don't want this to be
integer is to allow calls like
```
coalesce(df, 3)
```
in which `3` is a numeric by default. (vs `3L` is integer) IMO, forcing the
user to call with `3L` is a bit too much
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
