karldw opened a new issue, #35756:
URL: https://github.com/apache/arrow/issues/35756
### Describe the bug, including details regarding any error messages,
version, and platform.
I noticed that a CSV with header rows can be successfully read with
`read_csv_arrow(..., skip=N)`, but not with `open_csv_dataset` unless a schema
is provided. An example is below.
I expected `open_csv_dataset` to use the skip argument the same way
`read_csv_arrow` does, skipping a fixed number of rows from every CSV, but it
seems to not be skipping any. I think this is likely a bug -- maybe in the
schema parsing?
```R
library(arrow)
lines <- c(
"This line should be skipped",
"This one too, even though it has a comma",
"X,Y,Z",
"1,2,3",
"4,5,6"
)
tmp_dir <- file.path(tempdir(), "arrow_test")
dir.create(tmp_dir, recursive=TRUE)
tmp_csv <- file.path(tmp_dir, "test.csv")
writeLines(lines, tmp_csv)
# This works as expected, skipping the first two lines and reading headers
from the third
read_csv_arrow(tmp_csv, skip=2L)
open_csv_dataset(tmp_dir, skip=2L)
#> ! Invalid: Error creating dataset. Could not read schema from
#> '/tmp/RtmpPRG1yT/arrow_test/test.csv'. Is this a 'csv' file?:
#> Could not open CSV input source '/tmp/RtmpPRG1yT/arrow_test/test.csv':
#> Invalid: CSV parse error: Row #2: Expected 1 columns,
#> got 2: This one too, even though it has a comma
# These generate the same error:
open_csv_dataset(tmp_dir, skip=3L)
open_dataset(tmp_dir, format="csv", skip=2L)
schem <- schema(
field(name="X", type=int32()),
field(name="Y", type=int32()),
field(name="Z", type=int32())
)
# Works when schema is supplied (now also need to skip header row)
open_csv_dataset(tmp_dir, skip=3L, schema=schem) |> dplyr::collect()
```
### Component(s)
R
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]