Original Message-
From: R-help On Behalf Of Dave Dixon
Sent: Wednesday, April 10, 2024 12:20 PM
To: Rui Barradas ; r-help@r-project.org
Subject: Re: [R] Exceptional slowness with read.csv
That's basically what I did
1. Get text lines using readLines
2. use tryCatch to parse each line
That's basically what I did
1. Get text lines using readLines
2. use tryCatch to parse each line using read.csv(text=...)
3. in the catch, use gregexpr to find any quotes not adjacent to a comma
(gregexpr("[^,]\"[^,]",...)
4. escape any quotes found by adding a second quote (using str_sub from
, ...
-Original Message-
From: R-help On Behalf Of Rui Barradas
Sent: Wednesday, April 10, 2024 9:46 AM
To: Dave Dixon ; r-help@r-project.org
Subject: Re: [R] Exceptional slowness with read.csv
Às 06:47 de 08/04/2024, Dave Dixon escreveu:
> Greetings,
>
> I have a csv file of
Às 06:47 de 08/04/2024, Dave Dixon escreveu:
Greetings,
I have a csv file of 76 fields and about 4 million records. I know that
some of the records have errors - unmatched quotes, specifically.
Reading the file with readLines and parsing the lines with read.csv(text
= ...) is really slow. I
Try reading the lines in (readLines), count the number of both types of
quotes in each line. Find out which are not even and investigate.
On Mon, Apr 8, 2024, 15:24 Dave Dixon wrote:
> I solved the mystery, but not the problem. The problem is that there's
> an unclosed quote somewhere in those
I find QSV very helpful.
el
On 08/04/2024 22:21, Dave Dixon wrote:
> I solved the mystery, but not the problem. The problem is that
> there's an unclosed quote somewhere in those 5 additional records I'm
> trying to access. So read.csv is reading million-character fields.
> It's slow at that.
Right, I meant to add header=FALSE. And, it looks now like the next line
is the one with the unclosed quote, so read.csv is trying to read
million-character headers!
On 4/8/24 12:42, Ivan Krylov wrote:
В Sun, 7 Apr 2024 23:47:52 -0600
Dave Dixon пишет:
> second_records <-
Good suggestion - I'll look into data.table.
On 4/8/24 12:14, CALUM POLWART wrote:
> data.table's fread is also fast. Not sure about error handling. But I
> can merge 300 csvs with a total of 0.5m lines and 50 columns in a
> couple of minutes versus a lifetime with read.csv or readr::read_csv
>
Thanks, yeah, I think scan is more promising. I'll check it out.
On 4/8/24 11:49, Bert Gunter wrote:
> No idea, but have you tried using ?scan to read those next 5 rows? It
> might give you a better idea of the pathologies that are causing
> problems. For example, an unmatched quote might
I solved the mystery, but not the problem. The problem is that there's
an unclosed quote somewhere in those 5 additional records I'm trying to
access. So read.csv is reading million-character fields. It's slow at
that. That mystery solved.
However, the the problem persists: how to fix what is
Às 19:42 de 08/04/2024, Ivan Krylov via R-help escreveu:
В Sun, 7 Apr 2024 23:47:52 -0600
Dave Dixon пишет:
> second_records <- read.csv(file_name, skip = 2459465, nrows = 5)
It may or may not be important that read.csv defaults to header =
TRUE. Having skipped 2459465 lines, it may
В Sun, 7 Apr 2024 23:47:52 -0600
Dave Dixon пишет:
> > second_records <- read.csv(file_name, skip = 2459465, nrows = 5)
It may or may not be important that read.csv defaults to header =
TRUE. Having skipped 2459465 lines, it may attempt to parse the next
one as a header, so the second call
data.table's fread is also fast. Not sure about error handling. But I can
merge 300 csvs with a total of 0.5m lines and 50 columns in a couple of
minutes versus a lifetime with read.csv or readr::read_csv
On Mon, 8 Apr 2024, 16:19 Stevie Pederson,
wrote:
> Hi Dave,
>
> That's rather
No idea, but have you tried using ?scan to read those next 5 rows? It might
give you a better idea of the pathologies that are causing problems. For
example, an unmatched quote might result in some huge number of characters
trying to be read into a single element of a character variable. As your
Hi Dave,
That's rather frustrating. I've found vroom (from the package vroom) to be
helpful with large files like this.
Does the following give you any better luck?
vroom(file_name, delim = ",", skip = 2459465, n_max = 5)
Of course, when you know you've got errors & the files are big like that
15 matches
Mail list logo
