karldw opened a new issue, #35756:
URL: https://github.com/apache/arrow/issues/35756

   ### Describe the bug, including details regarding any error messages, 
version, and platform.
   
   I noticed that a CSV with header rows can be successfully read with 
`read_csv_arrow(..., skip=N)`, but not with `open_csv_dataset` unless a schema 
is provided. An example is below.
   
   I expected `open_csv_dataset` to use the skip argument the same way 
`read_csv_arrow` does, skipping a fixed number of rows from every CSV, but it 
seems to not be skipping any. I think this is likely a bug -- maybe in the 
schema parsing?
   
   
   ```R
   library(arrow)
   
   lines <- c(
     "This line should be skipped",
     "This one too, even though it has a comma",
     "X,Y,Z",
     "1,2,3",
     "4,5,6"
   )
   tmp_dir <- file.path(tempdir(), "arrow_test")
   dir.create(tmp_dir, recursive=TRUE)
   tmp_csv <- file.path(tmp_dir, "test.csv")
   writeLines(lines, tmp_csv)
   
   # This works as expected, skipping the first two lines and reading headers 
from the third
   read_csv_arrow(tmp_csv, skip=2L)
   
   open_csv_dataset(tmp_dir, skip=2L)
   #> ! Invalid: Error creating dataset. Could not read schema from 
   #> '/tmp/RtmpPRG1yT/arrow_test/test.csv'. Is this a 'csv' file?: 
   #> Could not open CSV input source '/tmp/RtmpPRG1yT/arrow_test/test.csv': 
   #> Invalid: CSV parse error: Row #2: Expected 1 columns, 
   #> got 2: This one too, even though it has a comma
   
   # These generate the same error:
   open_csv_dataset(tmp_dir, skip=3L)
   open_dataset(tmp_dir, format="csv", skip=2L)
   
   
   schem <- schema(
     field(name="X", type=int32()),
     field(name="Y", type=int32()),
     field(name="Z", type=int32())
   )
   
   # Works when schema is supplied (now also need to skip header row)
   open_csv_dataset(tmp_dir, skip=3L, schema=schem) |> dplyr::collect()
   
   ```
   
   
   
   
   
   ### Component(s)
   
   R


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to