John Sheffield created ARROW-11067:
--------------------------------------
Summary: read_csv_arrow silently fails to read some strings and
returns nulls
Key: ARROW-11067
URL: https://issues.apache.org/jira/browse/ARROW-11067
Project: Apache Arrow
Issue Type: Bug
Components: R
Reporter: John Sheffield
Attachments: demo_data.csv
A sample file is attached, showing 10 rows each of strings with consistent
failures (false_na = TRUE) and consistent successes (false_na = FALSE). The
strings are in the column `json_string` – if relevant, they are geojsons with
min nchar of 33,229 and max nchar of 202,515.
When I read this sample file with other R CSV readers (readr and data.table
shown), the files are imported correctly and there are no NAs in the
json_string column.
When I read with arrow::read_csv_arrow, 50% of the sample json_string column
end up as NAs. as_data_frame TRUE or FALSE does not change the behavior, so
this might not be limited to the R interface, but I can't help debug much
further upstream.
{code:java}
aaa1 <- arrow::read_csv_arrow("demo_data.csv", as_data_frame = TRUE)
aaa2 <- arrow::read_csv_arrow("demo_data.csv", as_data_frame = FALSE)
bbb <- data.table::fread("demo_data.csv")
ccc <- readr::read_csv("demo_data.csv")
mean(is.na(aaa1$json_string)) # 0.5
mean(is.na(aaa2$column(1))) # Scalar 0.5
mean(is.na(bbb$json_string)) # 0
mean(is.na(ccc$json_string)) # 0{code}
* arrow 2.0 (latest CRAN)
* readr 1.4.0
* data.table 1.13.2
* R version 4.0.1 (2020-06-06)
* MacOS Catalina 10.15.7 / x86_64-apple-darwin17.0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
