Repository: spark Updated Branches: refs/heads/master 6e268b9ee -> 454ba4d67
[SPARK-12479][SPARKR] sparkR collect on GroupedData throws R error "missing value where TRUE/FALSE needed" ## What changes were proposed in this pull request? This PR is a workaround for NA handling in hash code computation. This PR is on behalf of paulomagalhaes whose PR is https://github.com/apache/spark/pull/10436 ## How was this patch tested? SparkR unit tests. Author: Sun Rui <[email protected]> Author: ray <[email protected]> Closes #12976 from sun-rui/SPARK-12479. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/454ba4d6 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/454ba4d6 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/454ba4d6 Branch: refs/heads/master Commit: 454ba4d67e782369627dfe60261e6648a27b91a0 Parents: 6e268b9 Author: Sun Rui <[email protected]> Authored: Sun May 8 00:17:36 2016 -0700 Committer: Shivaram Venkataraman <[email protected]> Committed: Sun May 8 00:17:36 2016 -0700 ---------------------------------------------------------------------- R/pkg/R/utils.R | 3 +++ R/pkg/inst/tests/testthat/test_utils.R | 4 ++++ 2 files changed, 7 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/454ba4d6/R/pkg/R/utils.R ---------------------------------------------------------------------- diff --git a/R/pkg/R/utils.R b/R/pkg/R/utils.R index bf67e23..784f737 100644 --- a/R/pkg/R/utils.R +++ b/R/pkg/R/utils.R @@ -157,8 +157,11 @@ wrapInt <- function(value) { # Multiply `val` by 31 and add `addVal` to the result. Ensures that # integer-overflows are handled at every step. +# +# TODO: this function does not handle integer overflow well mult31AndAdd <- function(val, addVal) { vec <- c(bitwShiftL(val, c(4, 3, 2, 1, 0)), addVal) + vec[is.na(vec)] <- 0 Reduce(function(a, b) { wrapInt(as.numeric(a) + as.numeric(b)) }, http://git-wip-us.apache.org/repos/asf/spark/blob/454ba4d6/R/pkg/inst/tests/testthat/test_utils.R ---------------------------------------------------------------------- diff --git a/R/pkg/inst/tests/testthat/test_utils.R b/R/pkg/inst/tests/testthat/test_utils.R index 01694ab..54d2eca 100644 --- a/R/pkg/inst/tests/testthat/test_utils.R +++ b/R/pkg/inst/tests/testthat/test_utils.R @@ -164,3 +164,7 @@ test_that("convertToJSaveMode", { expect_error(convertToJSaveMode("foo"), 'mode should be one of "append", "overwrite", "error", "ignore"') #nolint }) + +test_that("hashCode", { + expect_error(hashCode("bc53d3605e8a5b7de1e8e271c2317645"), NA) +}) --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
