jllipatz opened a new issue, #34923:
URL: https://github.com/apache/arrow/issues/34923
### Describe the usage question you have. Please include as many useful
details as possible.
Hello
This is somehow related to issue 34820.
I am using the arrow package for R in its version 11.0.0.3.
I have a large file (130 Millions of lies, 60 columns, about 3.7 GO of disk
storage with parquet format). My problem is to find the most efficient method
(relative to duration and memory consumption) to add one column that is
computed as recoding an existing one using a small correspondence table.
Althouh I already have some kind of solution with duckdb and a SQL join. I
wanted to expriment some rough method as the ones beginners can think about.
The following code doesn't work. It correctly loads the files and creates
the column; at this point the file uses 58 GO in memory. But when arriving to
the writing point, the R session terminates immediatly without any message.
The strange thing is that if I use an RDS file with the same data the things
come to a normal end, even if it uses 133 GO to process the data which uses 78
GO of memory after loading and recoding.
The available memory on the machine I use is about 250 GO.
My program:
`
dep <- rio::import('V:/PALETTES/IGoR/data/dep2014.dbf')
df <- arrow::read_parquet('V:/PALETTES/parquet/rp68a19.parquet')
df$REGION <-
factor(df$DR,levels=dep$DEP,labels=dep$REGION) |>
as.character()
arrow ::write_parquet(df,'V:/PALETTES/tmp/rp68a19a.parquet')
`
### Component(s)
Benchmarking, Parquet, R
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]