[
https://issues.apache.org/jira/browse/ARROW-11067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17256576#comment-17256576
]
John Sheffield edited comment on ARROW-11067 at 12/30/20, 4:07 PM:
-------------------------------------------------------------------
Hm, the plot thickens. I just replicated Weston's results for the
arrow_sample_data.csv script in a few environments that suggest it might be a
Mac-running-R4.0 issue.
* *Success:* In a container (`rocker/geospatial:4.0.2`, container itself is
Ubuntu 20.04LTS running on GCE instance running Debian 10), I also see Weston's
result of all successes, but using R 4.0.2 instead of his 3.6.3.
{code:java}
> sessionInfo()R version 4.0.2 (2020-06-22)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04 LTS
Matrix products: default
BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.8.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8
LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8
[6] LC_MESSAGES=C LC_PAPER=en_US.UTF-8 LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] arrow_2.0.0 forcats_0.5.0 stringr_1.4.0 dplyr_1.0.2 purrr_0.3.4
readr_1.3.1 tidyr_1.1.2 tibble_3.0.4 ggplot2_3.3.2 tidyverse_1.3.0
loaded via a namespace (and not attached):
[1] Rcpp_1.0.5 cellranger_1.1.0 pillar_1.4.7 compiler_4.0.2 dbplyr_1.4.4
tools_4.0.2 digest_0.6.27 bit_4.0.4 jsonlite_1.7.1
[10] lubridate_1.7.9 lifecycle_0.2.0 gtable_0.3.0 pkgconfig_2.0.3 rlang_0.4.9
reprex_0.3.0 cli_2.2.0 DBI_1.1.0 rstudioapi_0.13
[19] haven_2.3.1 withr_2.3.0 xml2_1.3.2 httr_1.4.2 fs_1.5.0 generics_0.0.2
vctrs_0.3.5 hms_0.5.3 bit64_4.0.5
[28] grid_4.0.2 tidyselect_1.1.0 glue_1.4.2 R6_2.5.0 fansi_0.4.1 readxl_1.3.1
farver_2.0.3 modelr_0.1.8 blob_1.2.1
[37] magrittr_2.0.1 backports_1.1.10 scales_1.1.1 ellipsis_0.3.1 rvest_0.3.6
assertthat_0.2.1 colorspace_2.0-0 stringi_1.5.3 munsell_0.5.0
[46] broom_0.7.0 crayon_1.3.4
{code}
* *Failure:* In a fresh Mac R environment running the latest MacOS (Big Sur
11.1 20C69) and R 4.0.3, the alternating success/failure pattern still shows up:
{code:java}
> sessionInfo()
R version 4.0.3 (2020-10-10)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Big Sur 10.16
Matrix products: default
LAPACK:
/Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib
locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages: [1] stats graphics grDevices utils datasets methods
base
other attached packages: [1] arrow_2.0.0 forcats_0.5.0 stringr_1.4.0
dplyr_1.0.2 purrr_0.3.4 readr_1.4.0 tidyr_1.1.2 tibble_3.0.4 ggplot2_3.3.2
tidyverse_1.3.0
loaded via a namespace (and not attached): [1] Rcpp_1.0.5 cellranger_1.1.0
pillar_1.4.7 compiler_4.0.3 dbplyr_2.0.0 tools_4.0.3 digest_0.6.27 bit_4.0.4
jsonlite_1.7.2 [10] lubridate_1.7.9.2 lifecycle_0.2.0 gtable_0.3.0
pkgconfig_2.0.3 rlang_0.4.9 reprex_0.3.0 cli_2.2.0 DBI_1.1.0 rstudioapi_0.13
[19] haven_2.3.1 withr_2.3.0 xml2_1.3.2 httr_1.4.2 fs_1.5.0 generics_0.1.0
vctrs_0.3.5 hms_0.5.3 bit64_4.0.5 [28] grid_4.0.3 tidyselect_1.1.0 glue_1.4.2
R6_2.5.0 fansi_0.4.1 readxl_1.3.1 farver_2.0.3 modelr_0.1.8 magrittr_2.0.1 [37]
backports_1.2.1 scales_1.1.1 ellipsis_0.3.1 rvest_0.3.6 assertthat_0.2.1
colorspace_2.0-0 stringi_1.5.3 munsell_0.5.0 broom_0.7.2 [46] crayon_1.3.4
{code}
was (Author: jms):
Hm, the plot thickens. I just replicated Weston's results for the
arrow_sample_data.csv script in a few environments that suggest it might be a
Mac-running-R4.0 issue.
* *Success:* In a container (`rocker/geospatial:4.0.2`, container itself is
Ubuntu 20.04LTS running on GCE instance running Debian 10), I also see Weston's
result of all successes, but using R 4.0.2 instead of his 3.6.3.
> sessionInfo()R version 4.0.2 (2020-06-22)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04 LTS
Matrix products: default
BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.8.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8
LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8
[6] LC_MESSAGES=C LC_PAPER=en_US.UTF-8 LC_NAME=C
LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] arrow_2.0.0 forcats_0.5.0 stringr_1.4.0 dplyr_1.0.2
purrr_0.3.4 readr_1.3.1 tidyr_1.1.2 tibble_3.0.4 ggplot2_3.3.2
tidyverse_1.3.0
loaded via a namespace (and not attached):
[1] Rcpp_1.0.5 cellranger_1.1.0 pillar_1.4.7 compiler_4.0.2
dbplyr_1.4.4 tools_4.0.2 digest_0.6.27 bit_4.0.4
jsonlite_1.7.1
[10] lubridate_1.7.9 lifecycle_0.2.0 gtable_0.3.0 pkgconfig_2.0.3
rlang_0.4.9 reprex_0.3.0 cli_2.2.0 DBI_1.1.0
rstudioapi_0.13
[19] haven_2.3.1 withr_2.3.0 xml2_1.3.2 httr_1.4.2
fs_1.5.0 generics_0.0.2 vctrs_0.3.5 hms_0.5.3 bit64_4.0.5
[28] grid_4.0.2 tidyselect_1.1.0 glue_1.4.2 R6_2.5.0
fansi_0.4.1 readxl_1.3.1 farver_2.0.3 modelr_0.1.8 blob_1.2.1
[37] magrittr_2.0.1 backports_1.1.10 scales_1.1.1 ellipsis_0.3.1
rvest_0.3.6 assertthat_0.2.1 colorspace_2.0-0 stringi_1.5.3
munsell_0.5.0
[46] broom_0.7.0 crayon_1.3.4
* *Failure:* In a fresh Mac R environment running the latest MacOS (Big Sur
11.1 20C69) and R 4.0.3, the alternating success/failure pattern still shows up:
{code:java}
> sessionInfo()
R version 4.0.3 (2020-10-10)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Big Sur 10.16
Matrix products: default
LAPACK:
/Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib
locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages: [1] stats graphics grDevices utils datasets methods
base
other attached packages: [1] arrow_2.0.0 forcats_0.5.0 stringr_1.4.0
dplyr_1.0.2 purrr_0.3.4 readr_1.4.0 tidyr_1.1.2 tibble_3.0.4 ggplot2_3.3.2
tidyverse_1.3.0
loaded via a namespace (and not attached): [1] Rcpp_1.0.5 cellranger_1.1.0
pillar_1.4.7 compiler_4.0.3 dbplyr_2.0.0 tools_4.0.3 digest_0.6.27 bit_4.0.4
jsonlite_1.7.2 [10] lubridate_1.7.9.2 lifecycle_0.2.0 gtable_0.3.0
pkgconfig_2.0.3 rlang_0.4.9 reprex_0.3.0 cli_2.2.0 DBI_1.1.0 rstudioapi_0.13
[19] haven_2.3.1 withr_2.3.0 xml2_1.3.2 httr_1.4.2 fs_1.5.0 generics_0.1.0
vctrs_0.3.5 hms_0.5.3 bit64_4.0.5 [28] grid_4.0.3 tidyselect_1.1.0 glue_1.4.2
R6_2.5.0 fansi_0.4.1 readxl_1.3.1 farver_2.0.3 modelr_0.1.8 magrittr_2.0.1 [37]
backports_1.2.1 scales_1.1.1 ellipsis_0.3.1 rvest_0.3.6 assertthat_0.2.1
colorspace_2.0-0 stringi_1.5.3 munsell_0.5.0 broom_0.7.2 [46] crayon_1.3.4
{code}
> [R] read_csv_arrow silently fails to read some strings and returns nulls
> ------------------------------------------------------------------------
>
> Key: ARROW-11067
> URL: https://issues.apache.org/jira/browse/ARROW-11067
> Project: Apache Arrow
> Issue Type: Bug
> Components: R
> Reporter: John Sheffield
> Priority: Major
> Fix For: 3.0.0
>
> Attachments: arrow_explanation.png, arrow_failure_cases.csv,
> arrow_failure_cases.csv, arrowbug1.png, arrowbug1.png, demo_data.csv
>
>
> A sample file is attached, showing 10 rows each of strings with consistent
> failures (false_na = TRUE) and consistent successes (false_na = FALSE). The
> strings are in the column `json_string` – if relevant, they are geojsons with
> min nchar of 33,229 and max nchar of 202,515.
> When I read this sample file with other R CSV readers (readr and data.table
> shown), the files are imported correctly and there are no NAs in the
> json_string column.
> When I read with arrow::read_csv_arrow, 50% of the sample json_string column
> end up as NAs. as_data_frame TRUE or FALSE does not change the behavior, so
> this might not be limited to the R interface, but I can't help debug much
> further upstream.
>
>
> {code:java}
> aaa1 <- arrow::read_csv_arrow("demo_data.csv", as_data_frame = TRUE)
> aaa2 <- arrow::read_csv_arrow("demo_data.csv", as_data_frame = FALSE)
> bbb <- data.table::fread("demo_data.csv")
> ccc <- readr::read_csv("demo_data.csv")
> mean(is.na(aaa1$json_string)) # 0.5
> mean(is.na(aaa2$column(1))) # Scalar 0.5
> mean(is.na(bbb$json_string)) # 0
> mean(is.na(ccc$json_string)) # 0{code}
>
>
> * arrow 2.0 (latest CRAN)
> * readr 1.4.0
> * data.table 1.13.2
> * R version 4.0.1 (2020-06-06)
> * MacOS Catalina 10.15.7 / x86_64-apple-darwin17.0
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
