Re: [R-SIG-Mac] R 3.2.2 Hangs Reading Files in El Capitan [Solved]

2016-05-07 Thread Charles DiMaggio
Glad the post elicited some discussion.  Haven�t played with feather.  I�ve 
used data.table and it is indeed appreciably faster than base approaches for 
getting big csv�s into R.  I also find dplyr (with say MonetDB) to be a 
solution for out-of-memory approaches to large data sets. But, for native R 
files, I�ve found RDS to be fastest.  


Cheers

Charles






On May 6, 2016, at 9:01 PM, Simon Urbanek  wrote:

> 
> On May 6, 2016, at 6:03 PM, Brandon Hurr  wrote:
> 
>> Simon,
>> 
>> Absolutely was about RDS, but R is all about choices and the
>> underlying issue was time to read in data which fread and feather are
>> quite fast at. I assume when you say efficient you are referring to
>> disk space?
>> 
> 
> No, parsing data is always slower than native formats. Really fastest is 
> readBin (and similar direct I/O approaches), followed by feather and RDS (the 
> only reason RDS is not the fastest is that there is an extra copy in-memory) 
> -- unless you have slow disk, of course.
> 
> 
>> I put together a script to look at this further with and without
>> compression*. If speed is a priority over disk space then Feather and
>> data.table (CSV) are good options**. CSV is portable to any system and
>> feather can be used by python/Julia. RDS/RDA saves a lot of space and,
>> but are slower to write and read due to compression.
>> 
> 
> That's why I said uncompressed RDS [compress=FALSE] - you compress only if 
> you want to save space, not speed :).
> 
> FWIW according to our benchmarks iotools is the fastest for reading CSV if 
> you want to get into that arena, but that's whole another story - my point 
> was that the question was NOT about CSV or anything parsed - and neither 
> about writing - which is why this is getting really OT.
> 
> Cheers,
> Simon
> 
> 
> 
>> I hope that's helpful to those thinking about their priorities for
>> file IO in R.
>> 
>> Brandon
>> 
>> * http://rpubs.com/bhive01/fileioinr
>> **  writing a CSV with data.table is freaky fast if you can get OpenMP
>> working on your machine
>> https://github.com/Rdatatable/data.table/issues/1692 Reading that same
>> CSV is comparable to RDS.
>> 
>> 
>> On Fri, May 6, 2016 at 6:07 AM, Simon Urbanek
>>  wrote:
>>> Brandon,
>>> note that the post was about RDS which is more efficient than all the 
>>> options you list (in particular when not compressed). General advice is to 
>>> avoid strings. Numeric vectors are several orders of magnitude faster than 
>>> strings to load/save.
>>> Cheers,
>>> Simon
>>> 
>>> 
 On May 5, 2016, at 6:49 PM, Brandon Hurr  wrote:
 
 You might be interested in the speed wars that are happening in the
 file reading/writing space currently.
 
 Matt Dowle/Arun Srinivasan's data.table and Hadley Wickham/Wes
 McKinney's Feather have made huge speed advances in reading/writing
 large datasets from disks (mostly csv).
 
 Data Table fread()/fwrite():
 https://github.com/Rdatatable/data.table
 https://stackoverflow.com/questions/35763574/fastest-way-to-read-in-100-000-dat-gz-files
 http://blog.h2o.ai/2016/04/fast-csv-writing-for-r/
 
 
 Feather read_feather()/write_feather()
 https://github.com/wesm/feather
 
 I don't often have big datasets (10s of MBs) so I don't see the
 benefits of these much, but you might.
 
 HTH,
 B
 
 On Thu, May 5, 2016 at 3:16 PM, Charles DiMaggio
  wrote:
> Been a while, but wanted to close the page on a previous post describing 
> R hanging on readRDS() and load() for largish (say 500MB or larger) 
> files. Tried again with recent release (3.3.0).  Am able to read in large 
> files under El Cap.  While the file is reading in, I get a disconcerting 
> spinning pinwheel of death and a check under Force Quit reports R is not 
> responding.  But if I wait it out, it eventually reads in.  Odd.  But I 
> can live with it.
> 
> Cheers
> 
> Charles
> 
> 
> 
> 
> 
> 
> Charles DiMaggio, PhD, MPH
> Professor of Surgery and Population Health
> Director of Injury Research
> Department of Surgery
> New York University School of Medicine
> 462 First Avenue, NBV 15
> New York, NY 10016-9196
> charles.dimag...@nyumc.org
> Office: 212.263.3202
> Mobile: 516.308.6426
> 
> 
> 
> 
> 
> 
>  [[alternative HTML version deleted]]
> 
> ___
> R-SIG-Mac mailing list
> R-SIG-Mac@r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-mac
 
 ___
 R-SIG-Mac mailing list
 R-SIG-Mac@r-project.org
 https://stat.ethz.ch/mailman/listinfo/r-sig-mac
 
>>> 
>> 
> 


[[alternative HTML version deleted]]


Re: [R-SIG-Mac] R 3.2.2 Hangs Reading Files in El Capitan [Solved]

2016-05-06 Thread Simon Urbanek

On May 6, 2016, at 6:03 PM, Brandon Hurr  wrote:

> Simon,
> 
> Absolutely was about RDS, but R is all about choices and the
> underlying issue was time to read in data which fread and feather are
> quite fast at. I assume when you say efficient you are referring to
> disk space?
> 

No, parsing data is always slower than native formats. Really fastest is 
readBin (and similar direct I/O approaches), followed by feather and RDS (the 
only reason RDS is not the fastest is that there is an extra copy in-memory) -- 
unless you have slow disk, of course.


> I put together a script to look at this further with and without
> compression*. If speed is a priority over disk space then Feather and
> data.table (CSV) are good options**. CSV is portable to any system and
> feather can be used by python/Julia. RDS/RDA saves a lot of space and,
> but are slower to write and read due to compression.
> 

That's why I said uncompressed RDS [compress=FALSE] - you compress only if you 
want to save space, not speed :).

FWIW according to our benchmarks iotools is the fastest for reading CSV if you 
want to get into that arena, but that's whole another story - my point was that 
the question was NOT about CSV or anything parsed - and neither about writing - 
which is why this is getting really OT.

Cheers,
Simon



> I hope that's helpful to those thinking about their priorities for
> file IO in R.
> 
> Brandon
> 
> * http://rpubs.com/bhive01/fileioinr
> **  writing a CSV with data.table is freaky fast if you can get OpenMP
> working on your machine
> https://github.com/Rdatatable/data.table/issues/1692 Reading that same
> CSV is comparable to RDS.
> 
> 
> On Fri, May 6, 2016 at 6:07 AM, Simon Urbanek
>  wrote:
>> Brandon,
>> note that the post was about RDS which is more efficient than all the 
>> options you list (in particular when not compressed). General advice is to 
>> avoid strings. Numeric vectors are several orders of magnitude faster than 
>> strings to load/save.
>> Cheers,
>> Simon
>> 
>> 
>>> On May 5, 2016, at 6:49 PM, Brandon Hurr  wrote:
>>> 
>>> You might be interested in the speed wars that are happening in the
>>> file reading/writing space currently.
>>> 
>>> Matt Dowle/Arun Srinivasan's data.table and Hadley Wickham/Wes
>>> McKinney's Feather have made huge speed advances in reading/writing
>>> large datasets from disks (mostly csv).
>>> 
>>> Data Table fread()/fwrite():
>>> https://github.com/Rdatatable/data.table
>>> https://stackoverflow.com/questions/35763574/fastest-way-to-read-in-100-000-dat-gz-files
>>> http://blog.h2o.ai/2016/04/fast-csv-writing-for-r/
>>> 
>>> 
>>> Feather read_feather()/write_feather()
>>> https://github.com/wesm/feather
>>> 
>>> I don't often have big datasets (10s of MBs) so I don't see the
>>> benefits of these much, but you might.
>>> 
>>> HTH,
>>> B
>>> 
>>> On Thu, May 5, 2016 at 3:16 PM, Charles DiMaggio
>>>  wrote:
 Been a while, but wanted to close the page on a previous post describing R 
 hanging on readRDS() and load() for largish (say 500MB or larger) files. 
 Tried again with recent release (3.3.0).  Am able to read in large files 
 under El Cap.  While the file is reading in, I get a disconcerting 
 spinning pinwheel of death and a check under Force Quit reports R is not 
 responding.  But if I wait it out, it eventually reads in.  Odd.  But I 
 can live with it.
 
 Cheers
 
 Charles
 
 
 
 
 
 
 Charles DiMaggio, PhD, MPH
 Professor of Surgery and Population Health
 Director of Injury Research
 Department of Surgery
 New York University School of Medicine
 462 First Avenue, NBV 15
 New York, NY 10016-9196
 charles.dimag...@nyumc.org
 Office: 212.263.3202
 Mobile: 516.308.6426
 
 
 
 
 
 
   [[alternative HTML version deleted]]
 
 ___
 R-SIG-Mac mailing list
 R-SIG-Mac@r-project.org
 https://stat.ethz.ch/mailman/listinfo/r-sig-mac
>>> 
>>> ___
>>> R-SIG-Mac mailing list
>>> R-SIG-Mac@r-project.org
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mac
>>> 
>> 
> 

___
R-SIG-Mac mailing list
R-SIG-Mac@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-mac


Re: [R-SIG-Mac] R 3.2.2 Hangs Reading Files in El Capitan [Solved]

2016-05-06 Thread Brandon Hurr
Simon,

Absolutely was about RDS, but R is all about choices and the
underlying issue was time to read in data which fread and feather are
quite fast at. I assume when you say efficient you are referring to
disk space?

I put together a script to look at this further with and without
compression*. If speed is a priority over disk space then Feather and
data.table (CSV) are good options**. CSV is portable to any system and
feather can be used by python/Julia. RDS/RDA saves a lot of space and,
but are slower to write and read due to compression.

I hope that's helpful to those thinking about their priorities for
file IO in R.

Brandon

* http://rpubs.com/bhive01/fileioinr
**  writing a CSV with data.table is freaky fast if you can get OpenMP
working on your machine
https://github.com/Rdatatable/data.table/issues/1692 Reading that same
CSV is comparable to RDS.


On Fri, May 6, 2016 at 6:07 AM, Simon Urbanek
 wrote:
> Brandon,
> note that the post was about RDS which is more efficient than all the options 
> you list (in particular when not compressed). General advice is to avoid 
> strings. Numeric vectors are several orders of magnitude faster than strings 
> to load/save.
> Cheers,
> Simon
>
>
>> On May 5, 2016, at 6:49 PM, Brandon Hurr  wrote:
>>
>> You might be interested in the speed wars that are happening in the
>> file reading/writing space currently.
>>
>> Matt Dowle/Arun Srinivasan's data.table and Hadley Wickham/Wes
>> McKinney's Feather have made huge speed advances in reading/writing
>> large datasets from disks (mostly csv).
>>
>> Data Table fread()/fwrite():
>> https://github.com/Rdatatable/data.table
>> https://stackoverflow.com/questions/35763574/fastest-way-to-read-in-100-000-dat-gz-files
>> http://blog.h2o.ai/2016/04/fast-csv-writing-for-r/
>>
>>
>> Feather read_feather()/write_feather()
>> https://github.com/wesm/feather
>>
>> I don't often have big datasets (10s of MBs) so I don't see the
>> benefits of these much, but you might.
>>
>> HTH,
>> B
>>
>> On Thu, May 5, 2016 at 3:16 PM, Charles DiMaggio
>>  wrote:
>>> Been a while, but wanted to close the page on a previous post describing R 
>>> hanging on readRDS() and load() for largish (say 500MB or larger) files. 
>>> Tried again with recent release (3.3.0).  Am able to read in large files 
>>> under El Cap.  While the file is reading in, I get a disconcerting spinning 
>>> pinwheel of death and a check under Force Quit reports R is not responding. 
>>>  But if I wait it out, it eventually reads in.  Odd.  But I can live with 
>>> it.
>>>
>>> Cheers
>>>
>>> Charles
>>>
>>>
>>>
>>>
>>>
>>>
>>> Charles DiMaggio, PhD, MPH
>>> Professor of Surgery and Population Health
>>> Director of Injury Research
>>> Department of Surgery
>>> New York University School of Medicine
>>> 462 First Avenue, NBV 15
>>> New York, NY 10016-9196
>>> charles.dimag...@nyumc.org
>>> Office: 212.263.3202
>>> Mobile: 516.308.6426
>>>
>>>
>>>
>>>
>>>
>>>
>>>[[alternative HTML version deleted]]
>>>
>>> ___
>>> R-SIG-Mac mailing list
>>> R-SIG-Mac@r-project.org
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mac
>>
>> ___
>> R-SIG-Mac mailing list
>> R-SIG-Mac@r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-mac
>>
>

___
R-SIG-Mac mailing list
R-SIG-Mac@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-mac


Re: [R-SIG-Mac] R 3.2.2 Hangs Reading Files in El Capitan [Solved]

2016-05-06 Thread Simon Urbanek
Brandon,
note that the post was about RDS which is more efficient than all the options 
you list (in particular when not compressed). General advice is to avoid 
strings. Numeric vectors are several orders of magnitude faster than strings to 
load/save.
Cheers,
Simon


> On May 5, 2016, at 6:49 PM, Brandon Hurr  wrote:
> 
> You might be interested in the speed wars that are happening in the
> file reading/writing space currently.
> 
> Matt Dowle/Arun Srinivasan's data.table and Hadley Wickham/Wes
> McKinney's Feather have made huge speed advances in reading/writing
> large datasets from disks (mostly csv).
> 
> Data Table fread()/fwrite():
> https://github.com/Rdatatable/data.table
> https://stackoverflow.com/questions/35763574/fastest-way-to-read-in-100-000-dat-gz-files
> http://blog.h2o.ai/2016/04/fast-csv-writing-for-r/
> 
> 
> Feather read_feather()/write_feather()
> https://github.com/wesm/feather
> 
> I don't often have big datasets (10s of MBs) so I don't see the
> benefits of these much, but you might.
> 
> HTH,
> B
> 
> On Thu, May 5, 2016 at 3:16 PM, Charles DiMaggio
>  wrote:
>> Been a while, but wanted to close the page on a previous post describing R 
>> hanging on readRDS() and load() for largish (say 500MB or larger) files. 
>> Tried again with recent release (3.3.0).  Am able to read in large files 
>> under El Cap.  While the file is reading in, I get a disconcerting spinning 
>> pinwheel of death and a check under Force Quit reports R is not responding.  
>> But if I wait it out, it eventually reads in.  Odd.  But I can live with it.
>> 
>> Cheers
>> 
>> Charles
>> 
>> 
>> 
>> 
>> 
>> 
>> Charles DiMaggio, PhD, MPH
>> Professor of Surgery and Population Health
>> Director of Injury Research
>> Department of Surgery
>> New York University School of Medicine
>> 462 First Avenue, NBV 15
>> New York, NY 10016-9196
>> charles.dimag...@nyumc.org
>> Office: 212.263.3202
>> Mobile: 516.308.6426
>> 
>> 
>> 
>> 
>> 
>> 
>>[[alternative HTML version deleted]]
>> 
>> ___
>> R-SIG-Mac mailing list
>> R-SIG-Mac@r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-mac
> 
> ___
> R-SIG-Mac mailing list
> R-SIG-Mac@r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-mac
> 

___
R-SIG-Mac mailing list
R-SIG-Mac@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-mac


Re: [R-SIG-Mac] R 3.2.2 Hangs Reading Files in El Capitan [Solved]

2016-05-05 Thread Brandon Hurr
You might be interested in the speed wars that are happening in the
file reading/writing space currently.

Matt Dowle/Arun Srinivasan's data.table and Hadley Wickham/Wes
McKinney's Feather have made huge speed advances in reading/writing
large datasets from disks (mostly csv).

Data Table fread()/fwrite():
https://github.com/Rdatatable/data.table
https://stackoverflow.com/questions/35763574/fastest-way-to-read-in-100-000-dat-gz-files
http://blog.h2o.ai/2016/04/fast-csv-writing-for-r/


Feather read_feather()/write_feather()
https://github.com/wesm/feather

I don't often have big datasets (10s of MBs) so I don't see the
benefits of these much, but you might.

HTH,
B

On Thu, May 5, 2016 at 3:16 PM, Charles DiMaggio
 wrote:
> Been a while, but wanted to close the page on a previous post describing R 
> hanging on readRDS() and load() for largish (say 500MB or larger) files. 
> Tried again with recent release (3.3.0).  Am able to read in large files 
> under El Cap.  While the file is reading in, I get a disconcerting spinning 
> pinwheel of death and a check under Force Quit reports R is not responding.  
> But if I wait it out, it eventually reads in.  Odd.  But I can live with it.
>
> Cheers
>
> Charles
>
>
>
>
>
>
> Charles DiMaggio, PhD, MPH
> Professor of Surgery and Population Health
> Director of Injury Research
> Department of Surgery
> New York University School of Medicine
> 462 First Avenue, NBV 15
> New York, NY 10016-9196
> charles.dimag...@nyumc.org
> Office: 212.263.3202
> Mobile: 516.308.6426
>
>
>
>
>
>
> [[alternative HTML version deleted]]
>
> ___
> R-SIG-Mac mailing list
> R-SIG-Mac@r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-mac

___
R-SIG-Mac mailing list
R-SIG-Mac@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-mac


[R-SIG-Mac] R 3.2.2 Hangs Reading Files in El Capitan [Solved]

2016-05-05 Thread Charles DiMaggio
Been a while, but wanted to close the page on a previous post describing R 
hanging on readRDS() and load() for largish (say 500MB or larger) files. Tried 
again with recent release (3.3.0).  Am able to read in large files under El 
Cap.  While the file is reading in, I get a disconcerting spinning pinwheel of 
death and a check under Force Quit reports R is not responding.  But if I wait 
it out, it eventually reads in.  Odd.  But I can live with it.

Cheers

Charles






Charles DiMaggio, PhD, MPH
Professor of Surgery and Population Health
Director of Injury Research
Department of Surgery
New York University School of Medicine
462 First Avenue, NBV 15
New York, NY 10016-9196
charles.dimag...@nyumc.org
Office: 212.263.3202
Mobile: 516.308.6426 






[[alternative HTML version deleted]]

___
R-SIG-Mac mailing list
R-SIG-Mac@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-mac


Re: [R-SIG-Mac] R 3.2.2 Hangs Reading Files in El Capitan

2015-12-07 Thread Charles DiMaggio
Thanks very much for that, Simon. I will do that and get back to the group.

Charles


Charles DiMaggio, PhD, MPH
Director of Injury Research
Department of Surgery
New York University School of Medicine
550 First Avenue
New York, NY 10016
Direct Line: 1.516.308.6426
Office: 1.212.263.3202
charles.dimag...@nyumc.org

> On Dec 5, 2015, at 4:51 PM, Simon Urbanek  wrote:
> 
> 
> On Dec 5, 2015, at 11:24 AM, Charles DiMaggio  > wrote:
> 
>> Bob and Jeroen, many thanks for your follow up. Unfortunately the files I 
>> was working with are under a data use agreement. But  I couldn�t find any 
>> reference to similar issues online, so agree it may be an idiosyncrasy of my 
>> system.  I ended up rolling back to Mavericks.  I�ll wait for the official 
>> release of 3.2.3 in a few days and try again.
>> 
> 
> You don't need to wait - you can always try the latest build from 
> http://r.research.att.com  but since there were 
> no changes in the area it would be surprising if it behaved any differently. 
> 
> If you really want to dig into it, you could run R using
> R -d lldb
> then
> r
> and do whatever you need to replicate it. Once it seems stuck, hit  
> and then
> bt
> to list the back trace of where it got stuck. That would help us to find out 
> if it's getting stuck in the system itself or if it may have something to do 
> with R.
> 
> Thanks,
> Simon
> 
> 
>> Cheers
>> 
>> Charles
>> 
>> 
>>> On Dec 4, 2015, at 11:39 AM, boB Rudis  wrote:
>>> 
>>> Agreed that I cannot reproduce with generated data sets on el capitan
>>> either but would be glad to test with any real data you have.
>>> 
>>> On Fri, Dec 4, 2015 at 11:23 AM, Jeroen Ooms  
>>> wrote:
 Can you include a reproducible example? This seems to work for me as 
 expected:
 
 x <- rnorm(1e8)
 saveRDS(x, tmp <- tempfile())
 file.info(tmp)$size
 y <- readRDS(tmp)
 identical(x,y)
 
 Could be a hw issue with your disk.
 
 
 
 
 On Sun, Nov 29, 2015 at 4:34 PM, Charles DiMaggio
  wrote:
> Hi. After upgrading to el capitan R hangs on readRDS() and load()  for 
> largish (say 500MB or larger) files, requiring a Force Quit of R.  I am 
> working on a 64 GB Mac Pro machine and had no problem loading these size 
> files prior to upgrade.  Reading and loading smaller files (1MB or less) 
> seems to work fine. I re-installed R 3.2.2 GUI 1.66 Mavericks, XQuartz 
> and CLT after the upgrade. Turned off SIP but experienced the same 
> problem.
> 
> I've looked over recent list posting about R behavior after upgrade to el 
> capitan, and have not seen anything about this.  Am hoping 
> soon-to-be-released R version 3.2.3 (Wooden Christmas-Tree) will address 
> this weirdness, but wondering if anyone else has experienced anything 
> similar?
> 
> First few of lines of Mac Error Report below:
> 
> Date/Time:   2015-11-28 15:45:00 -0500
> OS Version:  Mac OS X 10.11.1 (Build 15B42)
> Architecture:x86_64
> Report Version:  22
> 
> Command: R
> Path:/Applications/R.app/Contents/MacOS/R
> Version: R 3.2.2 GUI 1.66 Mavericks build (6996)
> Parent:  launchd [1]
> PID: 563
> 
> Event:   hang
> Duration:1.70s (process was unresponsive for 25 seconds before 
> sampling)
> Steps:   17 (100ms sampling interval)
> 
> Hardware model:  MacPro6,1
> Active cpus: 8
> 
> ...
> 
> Heaviest stack for the main thread of the target process:
> 17  start + 1 (libdyld.dylib + 13741) [0x7fff88f2f5ad]
> 17  main + 815 (R + 5967) [0x1053e374f]
> 17  -[REngine runREPL] + 138 (R + 75578) [0x1053f473a]
> 17  run_REngineRmainloop + 295 (R + 123751) [0x105400367]
> 17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1243228) [0x10561d85c]
> 17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 675070) [0x105592cfe]
> 17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1044002) [0x1055ece22]
> 17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 674823) [0x105592c07]
> 17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1029000) [0x1055e9388]
> 17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 673910) [0x105592876]
> 17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 750621) [0x1055a541d]
> 17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1601435) [0x105674f9b]
> 17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1592422) [0x105672c66]
> 17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1594038) [0x1056732b6]
> 14  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1596619) [0x105673ccb]
> 7   ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1595584) [0x1056738c0]
> 6   

Re: [R-SIG-Mac] R 3.2.2 Hangs Reading Files in El Capitan

2015-12-05 Thread Charles DiMaggio
Bob and Jeroen, many thanks for your follow up. Unfortunately the files I was 
working with are under a data use agreement. But  I couldn’t find any reference 
to similar issues online, so agree it may be an idiosyncrasy of my system.  I 
ended up rolling back to Mavericks.  I’ll wait for the official release of 
3.2.3 in a few days and try again.

Cheers

Charles


> On Dec 4, 2015, at 11:39 AM, boB Rudis  wrote:
> 
> Agreed that I cannot reproduce with generated data sets on el capitan
> either but would be glad to test with any real data you have.
> 
> On Fri, Dec 4, 2015 at 11:23 AM, Jeroen Ooms  
> wrote:
>> Can you include a reproducible example? This seems to work for me as 
>> expected:
>> 
>>  x <- rnorm(1e8)
>>  saveRDS(x, tmp <- tempfile())
>>  file.info(tmp)$size
>>  y <- readRDS(tmp)
>>  identical(x,y)
>> 
>> Could be a hw issue with your disk.
>> 
>> 
>> 
>> 
>> On Sun, Nov 29, 2015 at 4:34 PM, Charles DiMaggio
>>  wrote:
>>> Hi. After upgrading to el capitan R hangs on readRDS() and load()  for 
>>> largish (say 500MB or larger) files, requiring a Force Quit of R.  I am 
>>> working on a 64 GB Mac Pro machine and had no problem loading these size 
>>> files prior to upgrade.  Reading and loading smaller files (1MB or less) 
>>> seems to work fine. I re-installed R 3.2.2 GUI 1.66 Mavericks, XQuartz and 
>>> CLT after the upgrade. Turned off SIP but experienced the same problem.
>>> 
>>> I've looked over recent list posting about R behavior after upgrade to el 
>>> capitan, and have not seen anything about this.  Am hoping 
>>> soon-to-be-released R version 3.2.3 (Wooden Christmas-Tree) will address 
>>> this weirdness, but wondering if anyone else has experienced anything 
>>> similar?
>>> 
>>> First few of lines of Mac Error Report below:
>>> 
>>> Date/Time:   2015-11-28 15:45:00 -0500
>>> OS Version:  Mac OS X 10.11.1 (Build 15B42)
>>> Architecture:x86_64
>>> Report Version:  22
>>> 
>>> Command: R
>>> Path:/Applications/R.app/Contents/MacOS/R
>>> Version: R 3.2.2 GUI 1.66 Mavericks build (6996)
>>> Parent:  launchd [1]
>>> PID: 563
>>> 
>>> Event:   hang
>>> Duration:1.70s (process was unresponsive for 25 seconds before 
>>> sampling)
>>> Steps:   17 (100ms sampling interval)
>>> 
>>> Hardware model:  MacPro6,1
>>> Active cpus: 8
>>> 
>>> ...
>>> 
>>> Heaviest stack for the main thread of the target process:
>>>  17  start + 1 (libdyld.dylib + 13741) [0x7fff88f2f5ad]
>>>  17  main + 815 (R + 5967) [0x1053e374f]
>>>  17  -[REngine runREPL] + 138 (R + 75578) [0x1053f473a]
>>>  17  run_REngineRmainloop + 295 (R + 123751) [0x105400367]
>>>  17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1243228) [0x10561d85c]
>>>  17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 675070) [0x105592cfe]
>>>  17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1044002) [0x1055ece22]
>>>  17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 674823) [0x105592c07]
>>>  17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1029000) [0x1055e9388]
>>>  17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 673910) [0x105592876]
>>>  17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 750621) [0x1055a541d]
>>>  17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1601435) [0x105674f9b]
>>>  17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1592422) [0x105672c66]
>>>  17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1594038) [0x1056732b6]
>>>  14  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1596619) [0x105673ccb]
>>>  7   ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1595584) [0x1056738c0]
>>>  6   ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1592773) [0x105672dc5]
>>>  6   ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1600194) [0x105674ac2]
>>>  6   ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 383889) [0x10554bb91]
>>>  6   ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 316653) [0x10553b4ed]
>>>  1   inflate + 258 (libz.1.dylib + 20735) [0x7fff9e4e30ff] (running)
>>> 
>>> 
>>> Process: R [563]
>>> Path:/Applications/R.app/Contents/MacOS/R
>>> Architecture:x86_64
>>> Parent:  launchd [1]
>>> UID: 501
>>> Task size:   350330 pages (+14553)
>>> CPU Time:1.604s
>>> Note:Unresponsive for 25 seconds before sampling
>>> Note:2 idle work queue threads omitted
>>> 
>>> 
>>> Cheers
>>> 
>>> Charles
>>> 
>>> 
>>> Charles DiMaggio, PhD, MPH
>>> Director of Injury Research
>>> Department of Surgery
>>> New York University School of Medicine
>>> Division of Trauma, Emergency Surgery and Critical Care Surgery
>>> Bellevue Hosptial Center
>>> 462 First Avenue, NBV 15
>>> New York, NY 10016-9196
>>> charles.dimag...@nyumc.org
>>> Office: 212.263.3202
>>> Mobile: 516.308.6426
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>>[[alternative HTML version deleted]]
>>> 
>>> ___

Re: [R-SIG-Mac] R 3.2.2 Hangs Reading Files in El Capitan

2015-12-05 Thread Simon Urbanek

On Dec 5, 2015, at 11:24 AM, Charles DiMaggio  
wrote:

> Bob and Jeroen, many thanks for your follow up. Unfortunately the files I was 
> working with are under a data use agreement. But  I couldn’t find any 
> reference to similar issues online, so agree it may be an idiosyncrasy of my 
> system.  I ended up rolling back to Mavericks.  I’ll wait for the official 
> release of 3.2.3 in a few days and try again.
> 

You don't need to wait - you can always try the latest build from 
http://r.research.att.com but since there were no changes in the area it would 
be surprising if it behaved any differently. 

If you really want to dig into it, you could run R using
R -d lldb
then
r
and do whatever you need to replicate it. Once it seems stuck, hit  
and then
bt
to list the back trace of where it got stuck. That would help us to find out if 
it's getting stuck in the system itself or if it may have something to do with 
R.

Thanks,
Simon


> Cheers
> 
> Charles
> 
> 
>> On Dec 4, 2015, at 11:39 AM, boB Rudis  wrote:
>> 
>> Agreed that I cannot reproduce with generated data sets on el capitan
>> either but would be glad to test with any real data you have.
>> 
>> On Fri, Dec 4, 2015 at 11:23 AM, Jeroen Ooms  
>> wrote:
>>> Can you include a reproducible example? This seems to work for me as 
>>> expected:
>>> 
>>> x <- rnorm(1e8)
>>> saveRDS(x, tmp <- tempfile())
>>> file.info(tmp)$size
>>> y <- readRDS(tmp)
>>> identical(x,y)
>>> 
>>> Could be a hw issue with your disk.
>>> 
>>> 
>>> 
>>> 
>>> On Sun, Nov 29, 2015 at 4:34 PM, Charles DiMaggio
>>>  wrote:
 Hi. After upgrading to el capitan R hangs on readRDS() and load()  for 
 largish (say 500MB or larger) files, requiring a Force Quit of R.  I am 
 working on a 64 GB Mac Pro machine and had no problem loading these size 
 files prior to upgrade.  Reading and loading smaller files (1MB or less) 
 seems to work fine. I re-installed R 3.2.2 GUI 1.66 Mavericks, XQuartz and 
 CLT after the upgrade. Turned off SIP but experienced the same problem.
 
 I've looked over recent list posting about R behavior after upgrade to el 
 capitan, and have not seen anything about this.  Am hoping 
 soon-to-be-released R version 3.2.3 (Wooden Christmas-Tree) will address 
 this weirdness, but wondering if anyone else has experienced anything 
 similar?
 
 First few of lines of Mac Error Report below:
 
 Date/Time:   2015-11-28 15:45:00 -0500
 OS Version:  Mac OS X 10.11.1 (Build 15B42)
 Architecture:x86_64
 Report Version:  22
 
 Command: R
 Path:/Applications/R.app/Contents/MacOS/R
 Version: R 3.2.2 GUI 1.66 Mavericks build (6996)
 Parent:  launchd [1]
 PID: 563
 
 Event:   hang
 Duration:1.70s (process was unresponsive for 25 seconds before 
 sampling)
 Steps:   17 (100ms sampling interval)
 
 Hardware model:  MacPro6,1
 Active cpus: 8
 
 ...
 
 Heaviest stack for the main thread of the target process:
 17  start + 1 (libdyld.dylib + 13741) [0x7fff88f2f5ad]
 17  main + 815 (R + 5967) [0x1053e374f]
 17  -[REngine runREPL] + 138 (R + 75578) [0x1053f473a]
 17  run_REngineRmainloop + 295 (R + 123751) [0x105400367]
 17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1243228) [0x10561d85c]
 17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 675070) [0x105592cfe]
 17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1044002) [0x1055ece22]
 17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 674823) [0x105592c07]
 17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1029000) [0x1055e9388]
 17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 673910) [0x105592876]
 17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 750621) [0x1055a541d]
 17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1601435) [0x105674f9b]
 17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1592422) [0x105672c66]
 17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1594038) [0x1056732b6]
 14  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1596619) [0x105673ccb]
 7   ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1595584) [0x1056738c0]
 6   ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1592773) [0x105672dc5]
 6   ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1600194) [0x105674ac2]
 6   ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 383889) [0x10554bb91]
 6   ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 316653) [0x10553b4ed]
 1   inflate + 258 (libz.1.dylib + 20735) [0x7fff9e4e30ff] (running)
 
 
 Process: R [563]
 Path:/Applications/R.app/Contents/MacOS/R
 Architecture:x86_64
 Parent:  launchd [1]
 UID: 501
 Task size:   350330 pages (+14553)

Re: [R-SIG-Mac] R 3.2.2 Hangs Reading Files in El Capitan

2015-12-04 Thread Jeroen Ooms
Can you include a reproducible example? This seems to work for me as expected:

  x <- rnorm(1e8)
  saveRDS(x, tmp <- tempfile())
  file.info(tmp)$size
  y <- readRDS(tmp)
  identical(x,y)

Could be a hw issue with your disk.




On Sun, Nov 29, 2015 at 4:34 PM, Charles DiMaggio
 wrote:
> Hi. After upgrading to el capitan R hangs on readRDS() and load()  for 
> largish (say 500MB or larger) files, requiring a Force Quit of R.  I am 
> working on a 64 GB Mac Pro machine and had no problem loading these size 
> files prior to upgrade.  Reading and loading smaller files (1MB or less) 
> seems to work fine. I re-installed R 3.2.2 GUI 1.66 Mavericks, XQuartz and 
> CLT after the upgrade. Turned off SIP but experienced the same problem.
>
> I've looked over recent list posting about R behavior after upgrade to el 
> capitan, and have not seen anything about this.  Am hoping 
> soon-to-be-released R version 3.2.3 (Wooden Christmas-Tree) will address this 
> weirdness, but wondering if anyone else has experienced anything similar?
>
> First few of lines of Mac Error Report below:
>
> Date/Time:   2015-11-28 15:45:00 -0500
> OS Version:  Mac OS X 10.11.1 (Build 15B42)
> Architecture:x86_64
> Report Version:  22
>
> Command: R
> Path:/Applications/R.app/Contents/MacOS/R
> Version: R 3.2.2 GUI 1.66 Mavericks build (6996)
> Parent:  launchd [1]
> PID: 563
>
> Event:   hang
> Duration:1.70s (process was unresponsive for 25 seconds before 
> sampling)
> Steps:   17 (100ms sampling interval)
>
> Hardware model:  MacPro6,1
> Active cpus: 8
>
> ...
>
> Heaviest stack for the main thread of the target process:
>   17  start + 1 (libdyld.dylib + 13741) [0x7fff88f2f5ad]
>   17  main + 815 (R + 5967) [0x1053e374f]
>   17  -[REngine runREPL] + 138 (R + 75578) [0x1053f473a]
>   17  run_REngineRmainloop + 295 (R + 123751) [0x105400367]
>   17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1243228) [0x10561d85c]
>   17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 675070) [0x105592cfe]
>   17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1044002) [0x1055ece22]
>   17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 674823) [0x105592c07]
>   17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1029000) [0x1055e9388]
>   17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 673910) [0x105592876]
>   17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 750621) [0x1055a541d]
>   17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1601435) [0x105674f9b]
>   17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1592422) [0x105672c66]
>   17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1594038) [0x1056732b6]
>   14  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1596619) [0x105673ccb]
>   7   ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1595584) [0x1056738c0]
>   6   ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1592773) [0x105672dc5]
>   6   ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1600194) [0x105674ac2]
>   6   ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 383889) [0x10554bb91]
>   6   ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 316653) [0x10553b4ed]
>   1   inflate + 258 (libz.1.dylib + 20735) [0x7fff9e4e30ff] (running)
>
>
> Process: R [563]
> Path:/Applications/R.app/Contents/MacOS/R
> Architecture:x86_64
> Parent:  launchd [1]
> UID: 501
> Task size:   350330 pages (+14553)
> CPU Time:1.604s
> Note:Unresponsive for 25 seconds before sampling
> Note:2 idle work queue threads omitted
>
>
> Cheers
>
> Charles
>
>
> Charles DiMaggio, PhD, MPH
> Director of Injury Research
> Department of Surgery
> New York University School of Medicine
> Division of Trauma, Emergency Surgery and Critical Care Surgery
> Bellevue Hosptial Center
> 462 First Avenue, NBV 15
> New York, NY 10016-9196
> charles.dimag...@nyumc.org
> Office: 212.263.3202
> Mobile: 516.308.6426
>
>
>
>
>
>
>
> [[alternative HTML version deleted]]
>
> ___
> R-SIG-Mac mailing list
> R-SIG-Mac@r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-mac

___
R-SIG-Mac mailing list
R-SIG-Mac@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-mac


Re: [R-SIG-Mac] R 3.2.2 Hangs Reading Files in El Capitan

2015-12-04 Thread boB Rudis
Agreed that I cannot reproduce with generated data sets on el capitan
either but would be glad to test with any real data you have.

On Fri, Dec 4, 2015 at 11:23 AM, Jeroen Ooms  wrote:
> Can you include a reproducible example? This seems to work for me as expected:
>
>   x <- rnorm(1e8)
>   saveRDS(x, tmp <- tempfile())
>   file.info(tmp)$size
>   y <- readRDS(tmp)
>   identical(x,y)
>
> Could be a hw issue with your disk.
>
>
>
>
> On Sun, Nov 29, 2015 at 4:34 PM, Charles DiMaggio
>  wrote:
>> Hi. After upgrading to el capitan R hangs on readRDS() and load()  for 
>> largish (say 500MB or larger) files, requiring a Force Quit of R.  I am 
>> working on a 64 GB Mac Pro machine and had no problem loading these size 
>> files prior to upgrade.  Reading and loading smaller files (1MB or less) 
>> seems to work fine. I re-installed R 3.2.2 GUI 1.66 Mavericks, XQuartz and 
>> CLT after the upgrade. Turned off SIP but experienced the same problem.
>>
>> I've looked over recent list posting about R behavior after upgrade to el 
>> capitan, and have not seen anything about this.  Am hoping 
>> soon-to-be-released R version 3.2.3 (Wooden Christmas-Tree) will address 
>> this weirdness, but wondering if anyone else has experienced anything 
>> similar?
>>
>> First few of lines of Mac Error Report below:
>>
>> Date/Time:   2015-11-28 15:45:00 -0500
>> OS Version:  Mac OS X 10.11.1 (Build 15B42)
>> Architecture:x86_64
>> Report Version:  22
>>
>> Command: R
>> Path:/Applications/R.app/Contents/MacOS/R
>> Version: R 3.2.2 GUI 1.66 Mavericks build (6996)
>> Parent:  launchd [1]
>> PID: 563
>>
>> Event:   hang
>> Duration:1.70s (process was unresponsive for 25 seconds before 
>> sampling)
>> Steps:   17 (100ms sampling interval)
>>
>> Hardware model:  MacPro6,1
>> Active cpus: 8
>>
>> ...
>>
>> Heaviest stack for the main thread of the target process:
>>   17  start + 1 (libdyld.dylib + 13741) [0x7fff88f2f5ad]
>>   17  main + 815 (R + 5967) [0x1053e374f]
>>   17  -[REngine runREPL] + 138 (R + 75578) [0x1053f473a]
>>   17  run_REngineRmainloop + 295 (R + 123751) [0x105400367]
>>   17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1243228) [0x10561d85c]
>>   17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 675070) [0x105592cfe]
>>   17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1044002) [0x1055ece22]
>>   17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 674823) [0x105592c07]
>>   17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1029000) [0x1055e9388]
>>   17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 673910) [0x105592876]
>>   17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 750621) [0x1055a541d]
>>   17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1601435) [0x105674f9b]
>>   17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1592422) [0x105672c66]
>>   17  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1594038) [0x1056732b6]
>>   14  ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1596619) [0x105673ccb]
>>   7   ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1595584) [0x1056738c0]
>>   6   ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1592773) [0x105672dc5]
>>   6   ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 1600194) [0x105674ac2]
>>   6   ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 383889) [0x10554bb91]
>>   6   ??? (<56AA7B12-7A0D-3F36-8116-218A93BC3CB3> + 316653) [0x10553b4ed]
>>   1   inflate + 258 (libz.1.dylib + 20735) [0x7fff9e4e30ff] (running)
>>
>>
>> Process: R [563]
>> Path:/Applications/R.app/Contents/MacOS/R
>> Architecture:x86_64
>> Parent:  launchd [1]
>> UID: 501
>> Task size:   350330 pages (+14553)
>> CPU Time:1.604s
>> Note:Unresponsive for 25 seconds before sampling
>> Note:2 idle work queue threads omitted
>>
>>
>> Cheers
>>
>> Charles
>>
>>
>> Charles DiMaggio, PhD, MPH
>> Director of Injury Research
>> Department of Surgery
>> New York University School of Medicine
>> Division of Trauma, Emergency Surgery and Critical Care Surgery
>> Bellevue Hosptial Center
>> 462 First Avenue, NBV 15
>> New York, NY 10016-9196
>> charles.dimag...@nyumc.org
>> Office: 212.263.3202
>> Mobile: 516.308.6426
>>
>>
>>
>>
>>
>>
>>
>> [[alternative HTML version deleted]]
>>
>> ___
>> R-SIG-Mac mailing list
>> R-SIG-Mac@r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-mac
>
> ___
> R-SIG-Mac mailing list
> R-SIG-Mac@r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-mac

___
R-SIG-Mac mailing list
R-SIG-Mac@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-mac