[
https://issues.apache.org/jira/browse/ARROW-11067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17256566#comment-17256566
]
Weston Pace commented on ARROW-11067:
-------------------------------------
Hmm...there might be something more at play here. I'm not able to reproduce
using the latest R package from CRAN...
{code:java}
> x <- arrow::read_csv_arrow("/home/pace/Downloads/demo_data.csv",
> as_data_frame = TRUE)
> mean(is.na(x$json_string))
[1] 0
> x <- arrow::read_csv_arrow("/home/pace/Downloads/demo_data.csv",
> as_data_frame = FALSE)
> mean(is.na(x$json_string))
Scalar
0
{code}
I am using an older version of R (3.6.3) so maybe that matters, I'm also on a
Linux server and not mac.
{code:java}
> sessionInfo()
R version 3.6.3 (2020-02-29)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.1 LTSMatrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C attached base
packages:
[1] stats graphics grDevices utils datasets methods base loaded
via a namespace (and not attached):
[1] tidyselect_1.1.0 bit_4.0.4 compiler_3.6.3 magrittr_2.0.1
[5] assertthat_0.2.1 R6_2.5.0 tools_3.6.3 glue_1.4.2
[9] bit64_4.0.5 vctrs_0.3.6 arrow_2.0.0 rlang_0.4.9
[13] purrr_0.3.4
{code}
I also ran the arrow_sample_data.csv script and got succeeded for all.
> [R] read_csv_arrow silently fails to read some strings and returns nulls
> ------------------------------------------------------------------------
>
> Key: ARROW-11067
> URL: https://issues.apache.org/jira/browse/ARROW-11067
> Project: Apache Arrow
> Issue Type: Bug
> Components: R
> Reporter: John Sheffield
> Priority: Major
> Fix For: 3.0.0
>
> Attachments: arrow_explanation.png, arrow_failure_cases.csv,
> arrow_failure_cases.csv, arrowbug1.png, arrowbug1.png, demo_data.csv
>
>
> A sample file is attached, showing 10 rows each of strings with consistent
> failures (false_na = TRUE) and consistent successes (false_na = FALSE). The
> strings are in the column `json_string` – if relevant, they are geojsons with
> min nchar of 33,229 and max nchar of 202,515.
> When I read this sample file with other R CSV readers (readr and data.table
> shown), the files are imported correctly and there are no NAs in the
> json_string column.
> When I read with arrow::read_csv_arrow, 50% of the sample json_string column
> end up as NAs. as_data_frame TRUE or FALSE does not change the behavior, so
> this might not be limited to the R interface, but I can't help debug much
> further upstream.
>
>
> {code:java}
> aaa1 <- arrow::read_csv_arrow("demo_data.csv", as_data_frame = TRUE)
> aaa2 <- arrow::read_csv_arrow("demo_data.csv", as_data_frame = FALSE)
> bbb <- data.table::fread("demo_data.csv")
> ccc <- readr::read_csv("demo_data.csv")
> mean(is.na(aaa1$json_string)) # 0.5
> mean(is.na(aaa2$column(1))) # Scalar 0.5
> mean(is.na(bbb$json_string)) # 0
> mean(is.na(ccc$json_string)) # 0{code}
>
>
> * arrow 2.0 (latest CRAN)
> * readr 1.4.0
> * data.table 1.13.2
> * R version 4.0.1 (2020-06-06)
> * MacOS Catalina 10.15.7 / x86_64-apple-darwin17.0
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
