This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 2b41bc9 [SPARK-32451][R][3.0] Support Apache Arrow 1.0.0
2b41bc9 is described below
commit 2b41bc928199df6d546ef055e233b9c58b019b38
Author: HyukjinKwon <[email protected]>
AuthorDate: Wed Aug 19 07:22:52 2020 -0700
[SPARK-32451][R][3.0] Support Apache Arrow 1.0.0
### What changes were proposed in this pull request?
This PR ports back https://github.com/apache/spark/pull/29252 to support
Arrow 1.0.0.
Currently, SparkR with Arrow tests fails with the latest Arrow version in
branch-3.0, see
https://github.com/apache/spark/pull/29460/checks?check_run_id=996972267
### Why are the changes needed?
To support higher Arrow R version with SparkR.
### Does this PR introduce _any_ user-facing change?
Yes, users will be able to use SparkR with Arrow 1.0.0+.
### How was this patch tested?
Manually tested, GitHub Actions will test it.
Closes #29462 from HyukjinKwon/SPARK-32451-3.0.
Authored-by: HyukjinKwon <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
R/pkg/R/DataFrame.R | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/R/pkg/R/DataFrame.R b/R/pkg/R/DataFrame.R
index f9a07f6..3a60d1c 100644
--- a/R/pkg/R/DataFrame.R
+++ b/R/pkg/R/DataFrame.R
@@ -1231,9 +1231,14 @@ setMethod("collect",
authSecret <- portAuth[[2]]
conn <- socketConnection(
port = port, blocking = TRUE, open = "wb", timeout =
connectionTimeout)
+ version <- packageVersion("arrow")
output <- tryCatch({
doServerAuth(conn, authSecret)
- arrowTable <- arrow::read_arrow(readRaw(conn))
+ if (version$minor >= 17 || version$major >= 1) {
+ arrowTable <- arrow::read_ipc_stream(readRaw(conn))
+ } else {
+ arrowTable <- arrow::read_arrow(readRaw(conn))
+ }
as.data.frame(arrowTable, stringsAsFactors =
stringsAsFactors)
}, finally = {
close(conn)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
