Christian,

In my original example I had an n=1E5 argument in readLines:

lines <- readLines(con, n=1E5)

This ensures that every iteration of the loop only 100000 lines are read (which should usually fit into memory). Without this argument readLines tries to read in the complete file.

Jan


On 08/09/2013 04:43 PM, christian.kame...@astra.admin.ch wrote:
Jan,

Many thanks for your suggestion! The code runs perfectly fine on the test set. 
Applying it to the complete data set, however, results in the following error:

while (TRUE) {
+              lines <- readLines(con, encoding='LATIN1')
+              if (length(lines) == 0) break
+              lines <- sprintf("%-238s", lines)
+              writeLines(lines, out, useBytes=TRUE) }
Error: cannot allocate vector of size 23.2 Mb


Best Regard

Christian Kamenik
Project Manager

Federal Department of the Environment, Transport, Energy and Communications 
DETEC
Federal Roads Office FEDRO
Division Road Traffic
Road Accident Statistics

Mailing Address: 3003 Bern
Location: Weltpoststrasse 5, 3015 Bern

Tel +41 31 323 14 89
Fax +41 31 323 43 21

christian.kame...@astra.admin.ch
www.astra.admin.ch


-----Ursprüngliche Nachricht-----
Von: Jan van der Laan [mailto:rh...@eoos.dds.nl]
Gesendet: Freitag, 9. August 2013 10:01
An: Kamenik Christian ASTRA
Betreff: Re: AW: AW: [R] laf_open_fwf

Christian,

It seems some of the lines in your file have additional characters at the end 
causing the line lengths to vary. The only way I could think of is to first add 
whitespace to the shorter lines to make all line lengths equal:

# Add whitespace to the end of the lines to make all lines the same length con <- file("testdata.txt", 
"rt") out <- file("testdata_2.txt", "wt") while (TRUE) {
    lines <- readLines(con, n=1E5)
    if (length(lines) == 0) break
    lines <- sprintf("%-238s", lines)
    writeLines(lines, out, useBytes=TRUE) }
close(con)
close(out)


I am then able to read you test file using LaF:

library(LaF)

column_widths <- c(3, 28, 4, 30, 28, 6, 3, 30, 10, 26, 25, 30, 2, 5, 5) column_types <- 
rep("string", length(column_widths)) column_types[c(1, 3, 7)] <- "integer"

laf <- laf_open_fwf("testdata_2.txt", column_types = column_types, 
column_widths = column_widths)


HTH,
Jan







christian.kame...@astra.admin.ch schreef:

Hello Jan

I attached an example. Any help is highly appreciated!

Kind Regard

Christian Kamenik
Project Manager

Federal Department of the Environment, Transport, Energy and
Communications DETEC Federal Roads Office FEDRO Division Road Traffic
Road Accident Statistics

Mailing Address: 3003 Bern
Location: Weltpoststrasse 5, 3015 Bern

Tel +41 31 323 14 89
Fax +41 31 323 43 21

christian.kame...@astra.admin.ch
www.astra.admin.ch
-----Ursprüngliche Nachricht-----
Von: Jan van der Laan [mailto:rh...@eoos.dds.nl]
Gesendet: Donnerstag, 8. August 2013 13:58
An: r-help@r-project.org
Cc: Kamenik Christian ASTRA
Betreff: Re: AW: [R] laf_open_fwf


Without example data it is difficult to give suggestions on how you
might read this file.

Are you sure your file is fixed width? Sometimes columns are neatly
aligned using whitespace (tabs/spaces). In that case you could use
read.table with the default settings.

Another possibility might be that the file is encoded in utf8. I
expect that reading it in assuming another encoding (such as latin1)
would lead to varying line sizes. Although I would expect the lengths
to be larger than the sum of your column widths (as one symbol can be
larger than one byte).

Jan



christian.kame...@astra.admin.ch schreef:

Dear Jan

Many thanks for your help. In fact, all lines are shorter than my
column width...

my.column.widths:       238
range(nchar(lines)):    235 237

So, it seems I have an inconsistent file structure...
I guess there is no way to handle this in an automated way?

Best Regard

Christian Kamenik
Project Manager

Federal Department of the Environment, Transport, Energy and
Communications DETEC Federal Roads Office FEDRO Division Road Traffic
Road Accident Statistics

Mailing Address: 3003 Bern
Location: Weltpoststrasse 5, 3015 Bern

Tel +41 31 323 14 89
Fax +41 31 323 43 21

christian.kame...@astra.admin.ch
www.astra.admin.ch
-----Ursprüngliche Nachricht-----
Von: Jan van der Laan [mailto:rh...@eoos.dds.nl]
Gesendet: Mittwoch, 7. August 2013 20:57
An: r-help@r-project.org
Cc: Kamenik Christian ASTRA
Betreff: Re: [R] laf_open_fwf

Dear Christian,

Well... it shouldn't normally do that. The only way I can currently
think of that might cause this problem is that the file has \r\n\r\n,
which would mean that every line is followed by an empty line.

Another cause might be (although I would not really expect the
results you see) that the sum of your column widths is larger than
the actual with of the line.

You can check your line lengths using:

lines <- readLines(my.filename)
nchar(lines)

Each line should have the same length and be equal to (or at least
larger than) sum(my.column.widths)

If this is not the problem: would it be possible that you send me a
small part of your file so that I could try to reproduce the problem?
Or if you cannot share your data: replace the actual values with
nonsense values.

Regards,
Jan

PS I read your mail by chance as I am not a regular r-help reader.
When you have specific LaF problems it is better to also cc me
directly.

On 08/06/2013 12:35 PM, christian.kame...@astra.admin.ch wrote:
Dear all

I was trying the (fairly new) LaF package, and came across the
following problem:

I opened a connection to a fixed width ASCII file using
laf_open_fwf(my.filename, my.column_types, my.column_widths,
my.column_names)

When looking at the data, it turned out that \n (newline) and \r
(carriage return) were considered as characters, thus destroying the
structure in my data (the second column does not include any
numbers):

my.data[1565:1575,1:3]

     MF_FARZ1              Fahrzeugarttext MF_MARKE
1     \n043 Landwirt. Traktor                2140
2     \n043 Landwirt. Traktor                6206
3     \n001 Personenwagen                    2026
4     \n001 Personenwagen                    2026
5    \r\n00 1Personenwagen                    404
6    \r\n02 0Gesellschaftswagen               710
7    \r\n00 1Personenwagen                    505
8    \r\n00 1Personenwagen                    505
9    \r\n00 1Personenwagen                    301
10   \r\n00 1Personenwagen                    553
11   \r\n04 3Landwirt. Traktor                257

I am working on Windows 7 32-bit.

Any help would be highly appreciated.

Best Regard

Christian Kamenik
Project Manager

Federal Department of the Environment, Transport, Energy and
Communications DETEC Federal Roads Office FEDRO Division Road
Traffic Road Accident Statistics

Mailing Address: 3003 Bern
Location: Weltpoststrasse 5, 3015 Bern

Tel +41 31 323 14 89
Fax +41 31 323 43 21

christian.kame...@astra.admin.ch<mailto:christian.kamenik@astra.admin.
ch> www.astra.admin.ch<http://www.astra.admin.ch/>


        [[alternative HTML version deleted]]

______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.





______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Reply via email to