Kousuke Saruta created SPARK-56810:
--------------------------------------
Summary: Fix cleanClosure to skip primitive functions for R 4.4+
Key: SPARK-56810
URL: https://issues.apache.org/jira/browse/SPARK-56810
Project: Spark
Issue Type: Sub-task
Components: Spark Core
Affects Versions: 4.2.0, 4.1.2, 3.5.9
Reporter: Kousuke Saruta
With R 4.4+, R/run-tests.sh fails.
{code}
== Failed ======================================================================
-- 1. Error ('test_rdd.R:49:3'): count by values and keys ----------------------
Error in `h(simpleError(msg, call))`: error in evaluating the argument 'x' in
selecting a method for function 'collectRDD': (converted from warning) setting
environment(<primitive function>) is not possible and trying it is deprecated
Backtrace:
x
1. +-SparkR:::countByValue(mods) at test_rdd.R:49:3
2. +-SparkR:::countByValue(mods)
3. | +-SparkR:::collectRDD(reduceByKey(ones, `+`, getNumPartitionsRDD(x)))
4. | +-SparkR:::reduceByKey(ones, `+`, getNumPartitionsRDD(x))
5. | \-SparkR:::reduceByKey(ones, `+`, getNumPartitionsRDD(x))
6. | +-SparkR:::lapplyPartition(x, reduceVals)
7. | \-SparkR:::lapplyPartition(x, reduceVals)
8. | +-SparkR:::lapplyPartitionsWithIndex(...)
9. | \-SparkR:::lapplyPartitionsWithIndex(...)
10. | \-SparkR:::PipelinedRDD(X, FUN)
11. | \-methods::new("PipelinedRDD", prev, func, NULL)
12. | +-methods::initialize(value, ...)
13. | \-SparkR (local) initialize(value, ...)
14. | \-SparkR (local) .local(.Object, ...)
15. | \-SparkR:::cleanClosure(pipelinedFunc)
16. | \-SparkR:::processClosure(...)
17. | \-SparkR:::processClosure(...)
18. | \-SparkR:::processClosure(...)
19. | \-SparkR:::processClosure(...)
20. | \-SparkR:::cleanClosure(obj, checkedFuncs)
21. | \-SparkR:::processClosure(...)
22. | \-SparkR:::processClosure(...)
23. | \-SparkR:::processClosure(...)
24. | \-SparkR:::cleanClosure(obj,
checkedFuncs)
25. | \-SparkR:::processClosure(...)
26. | \-SparkR:::processClosure(...)
27. | \-SparkR:::processClosure(...)
28. | \-SparkR:::processClosure(...)
29. | \-SparkR:::processClosure(...)
30. |
\-SparkR:::processClosure(...)
31. |
\-SparkR:::cleanClosure(obj, checkedFuncs)
32. +-base::.signalSimpleWarning(...)
33. | \-base::withRestarts(...)
34. | \-base (local) withOneRestart(expr, restarts[[1L]])
35. | \-base (local) doWithOneRestart(return(expr), restart)
36. \-base::.handleSimpleError(...)
37. \-base (local) h(simpleError(msg, call))
-- 2. Error ('test_rdd.R:282:3'): maximum() on RDDs ----------------------------
Error in `h(simpleError(msg, call))`: error in evaluating the argument 'x' in
selecting a method for function 'collectRDD': (converted from warning) setting
environment(<primitive function>) is not possible and trying it is deprecated
Backtrace:
x
1. +-SparkR:::maximum(rdd) at test_rdd.R:282:3
2. +-SparkR:::maximum(rdd)
3. | +-SparkR:::reduce(x, max)
4. | \-SparkR:::reduce(x, max)
5. | +-SparkR:::collectRDD(lapplyPartition(x, reducePartition), flatten =
FALSE)
6. | +-SparkR:::lapplyPartition(x, reducePartition)
7. | \-SparkR:::lapplyPartition(x, reducePartition)
8. | +-SparkR:::lapplyPartitionsWithIndex(...)
9. | \-SparkR:::lapplyPartitionsWithIndex(...)
10. | \-SparkR:::PipelinedRDD(X, FUN)
11. | \-methods::new("PipelinedRDD", prev, func, NULL)
12. | +-methods::initialize(value, ...)
13. | \-SparkR (local) initialize(value, ...)
14. | \-SparkR (local) .local(.Object, ...)
15. | \-SparkR:::cleanClosure(func)
16. | \-SparkR:::processClosure(...)
17. | \-SparkR:::processClosure(...)
18. | \-SparkR:::processClosure(...)
19. | \-SparkR:::cleanClosure(obj, checkedFuncs)
20. | \-SparkR:::processClosure(...)
21. | \-SparkR:::processClosure(...)
22. | \-SparkR:::processClosure(...)
23. | \-SparkR:::cleanClosure(obj, checkedFuncs)
24. +-base::.signalSimpleWarning(...)
25. | \-base::withRestarts(...)
26. | \-base (local) withOneRestart(expr, restarts[[1L]])
27. | \-base (local) doWithOneRestart(return(expr), restart)
28. \-base::.handleSimpleError(...)
29. \-base (local) h(simpleError(msg, call))
-- 3. Error ('test_rdd.R:287:3'): minimum() on RDDs ----------------------------
Error in `h(simpleError(msg, call))`: error in evaluating the argument 'x' in
selecting a method for function 'collectRDD': (converted from warning) setting
environment(<primitive function>) is not possible and trying it is deprecated
Backtrace:
x
1. +-SparkR:::minimum(rdd) at test_rdd.R:287:3
2. +-SparkR:::minimum(rdd)
3. | +-SparkR:::reduce(x, min)
4. | \-SparkR:::reduce(x, min)
5. | +-SparkR:::collectRDD(lapplyPartition(x, reducePartition), flatten =
FALSE)
6. | +-SparkR:::lapplyPartition(x, reducePartition)
7. | \-SparkR:::lapplyPartition(x, reducePartition)
8. | +-SparkR:::lapplyPartitionsWithIndex(...)
9. | \-SparkR:::lapplyPartitionsWithIndex(...)
10. | \-SparkR:::PipelinedRDD(X, FUN)
11. | \-methods::new("PipelinedRDD", prev, func, NULL)
12. | +-methods::initialize(value, ...)
13. | \-SparkR (local) initialize(value, ...)
14. | \-SparkR (local) .local(.Object, ...)
15. | \-SparkR:::cleanClosure(func)
16. | \-SparkR:::processClosure(...)
17. | \-SparkR:::processClosure(...)
18. | \-SparkR:::processClosure(...)
19. | \-SparkR:::cleanClosure(obj, checkedFuncs)
20. | \-SparkR:::processClosure(...)
21. | \-SparkR:::processClosure(...)
22. | \-SparkR:::processClosure(...)
23. | \-SparkR:::cleanClosure(obj, checkedFuncs)
24. +-base::.signalSimpleWarning(...)
25. | \-base::withRestarts(...)
26. | \-base (local) withOneRestart(expr, restarts[[1L]])
27. | \-base (local) doWithOneRestart(return(expr), restart)
28. \-base::.handleSimpleError(...)
29. \-base (local) h(simpleError(msg, call))
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
