jllipatz opened a new issue, #34923:
URL: https://github.com/apache/arrow/issues/34923

   ### Describe the usage question you have. Please include as many useful 
details as  possible.
   
   
   Hello
   This is somehow related to issue 34820.
   I am using the arrow package for R in its version 11.0.0.3.
   
   I have a large file (130 Millions of lies, 60 columns, about 3.7 GO of disk 
storage with parquet format). My problem is to find the most efficient method  
(relative to duration and memory consumption) to add one column that is 
computed as recoding an existing one using a small correspondence table.
   Althouh I already have some kind of solution with duckdb and a SQL join. I 
wanted to expriment some rough method as the ones beginners can think about.
   
   The following code doesn't work. It correctly loads the files and creates 
the column; at this point the file uses 58 GO in memory. But when arriving to 
the writing point, the R session terminates immediatly without any message. 
   The strange thing is that if I use an RDS file with the same data the things 
come to a normal end, even if it uses 133 GO to process the data which uses 78 
GO of memory after loading and recoding.
   
   The available memory on the machine I use is about 250 GO.
   
   My program:
   `
   dep <- rio::import('V:/PALETTES/IGoR/data/dep2014.dbf')
   
   df <- arrow::read_parquet('V:/PALETTES/parquet/rp68a19.parquet')
   
   df$REGION <- 
     factor(df$DR,levels=dep$DEP,labels=dep$REGION) |>
     as.character()
   
   arrow ::write_parquet(df,'V:/PALETTES/tmp/rp68a19a.parquet')
   `
   
   ### Component(s)
   
   Benchmarking, Parquet, R


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to