FYI, in R.utils (>= 1.28.4) you can use listDirectory() to control how deep the recursion goes, which would give you protection against your problem, e.g.
R.utils::listDirectory("dir", recursive=5L) where recursive=0L is equivalent to recursive=FALSE. Using recursive=TRUE corresponds to recursive=+Inf, that is infinite depth. listDirectory() accepts similar arguments that list.files() does. /Henrik On Sat, Sep 28, 2013 at 5:06 PM, William Dunlap <wdun...@tibco.com> wrote: > The issue is not symbolic links per se, but ones that form loops. > Note that you can detect such loops by running 'find -L ...' and > looking for the error messages. (find by default does not follow > any symbolic links, which can be a problem also.) > > It is a shortcoming of the current version of list.files(). > > Bill Dunlap > Spotfire, TIBCO Software > wdunlap tibco.com > > >> -----Original Message----- >> From: jgrn...@gmail.com [mailto:jgrn...@gmail.com] On Behalf Of Jonathan >> Greenberg >> Sent: Saturday, September 28, 2013 10:51 AM >> To: William Dunlap >> Cc: r-help >> Subject: Re: [R] Error: C stack usage is too close to the limit when using >> list.files() >> >> Thanks all -- ok, so the symbolic link issue is a distinct >> possibility, but fundamentally doesn't solve the issue since most >> users will have symbolic links on their machines SOMEPLACE, so a full >> drive scan will run into these issues -- is list.files calling find, >> or is it using a different algorithm? This seems like a shortcoming >> in the list.files algorithm -- is there a better solution (short of a >> System call, which I'm still not sure will work on Macs without Xcode >> -- a colleague of mine did NOT have Xcode, and reported not being able >> to run find from the command line) -- perhaps a different package? >> >> --j >> >> On Fri, Sep 27, 2013 at 3:08 PM, William Dunlap <wdun...@tibco.com> wrote: >> > Toss a couple of extra files in there and you will see the output grow >> > exponentially. >> > >> > % touch dir/IMPORTANT_1 dir/subdir/IMPORTANT_2 >> > >> > and in R those two new files cause 82 more strings to appear in >> > list.file's output: >> > >> >> nchar(list.files("dir", recursive=TRUE)) >> > [1] 11 18 33 40 55 62 77 84 99 106 121 128 143 150 165 172 187 >> > 194 209 >> > [20] 216 231 238 253 260 275 282 297 304 319 326 341 348 363 370 385 392 >> > 407 414 >> > [39] 429 436 451 458 473 480 495 502 517 524 539 546 561 568 583 590 605 >> > 612 627 >> > [58] 634 649 656 671 678 693 700 715 722 737 744 759 766 781 788 803 810 >> > 825 832 >> > [77] 847 854 869 876 891 898 901 >> > >> > 'find', by default, does not following symbolic links. >> > >> > % find dir >> > dir >> > dir/subdir >> > dir/subdir/IMPORTANT_2 >> > dir/subdir/linkToUpperDir >> > dir/IMPORTANT_1 >> > >> > The -L option makes it follow them, but it won't follow loops: >> > >> > % find -L dir >> > dir >> > dir/subdir >> > dir/subdir/IMPORTANT_2 >> > find: File system loop detected; `dir/subdir/linkToUpperDir' is part of >> > the same file >> system loop as `dir'. >> > dir/IMPORTANT_1 >> > >> > Bill Dunlap >> > Spotfire, TIBCO Software >> > wdunlap tibco.com >> > >> > >> >> -----Original Message----- >> >> From: r-help-boun...@r-project.org [mailto:r-help-boun...@r-project.org] >> >> On >> Behalf >> >> Of William Dunlap >> >> Sent: Friday, September 27, 2013 12:56 PM >> >> To: Jonathan Greenberg; r-help >> >> Subject: Re: [R] Error: C stack usage is too close to the limit when >> >> using list.files() >> >> >> >> Do you have some symbolic links that make loops in your file system? >> >> list.files() has problems with such loops and find does not. E.g., on a >> >> Linux box: >> >> >> >> % cd /tmp >> >> % mkdir dir dir/subdir >> >> % cd dir/subdir >> >> % ln -s ../../dir linkToUpperDir >> >> % cd /tmp >> >> % R --quiet >> >> > list.files("dir", recursive=TRUE, full=TRUE) >> >> [1] >> >> >> "dir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToU >> >> >> pperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkT >> >> >> oUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/li >> >> >> nkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdi >> >> >> r/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/su >> >> >> bdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir >> >> >> /subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpper >> >> >> Dir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUp >> >> >> perDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkTo >> >> >> UpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir/subdir/lin >> >> kToUpperDir/subdir/linkToUpperDir/subdir/linkToUpperDir" >> >> > system("find dir") >> >> dir >> >> dir/subdir >> >> dir/subdir/linkToUpperDir >> >> >> >> Bill Dunlap >> >> Spotfire, TIBCO Software >> >> wdunlap tibco.com >> >> >> >> >> >> > -----Original Message----- >> >> > From: r-help-boun...@r-project.org >> >> > [mailto:r-help-boun...@r-project.org] On >> Behalf >> >> > Of Jonathan Greenberg >> >> > Sent: Friday, September 27, 2013 12:13 PM >> >> > To: r-help >> >> > Subject: [R] Error: C stack usage is too close to the limit when using >> >> > list.files() >> >> > >> >> > R-helpers: >> >> > >> >> > I'm running a file search on my entire drive (Mac OS X) using: >> >> > >> >> > files_found <- >> list.files(dir="/",pattern=somepattern,recursive=TRUE,full.names=TRUE) >> >> > where somepattern is a search pattern (which I have confirmed via a >> >> > unix "find / -name somepattern" only returns ~ 3 results). >> >> > >> >> > I keep getting an error: >> >> > >> >> > Error: C stack usage is too close to the limit >> >> > >> >> > when running this command. Any ideas on 1) how to fix this or 2) if >> >> > there is an alternative to using list.files() to accomplish this >> >> > search without resorting to an external package? >> >> > >> >> > Cheers! >> >> > >> >> > --jonathan >> >> > >> >> > >> >> > -- >> >> > Jonathan A. Greenberg, PhD >> >> > Assistant Professor >> >> > Global Environmental Analysis and Remote Sensing (GEARS) Laboratory >> >> > Department of Geography and Geographic Information Science >> >> > University of Illinois at Urbana-Champaign >> >> > 259 Computing Applications Building, MC-150 >> >> > 605 East Springfield Avenue >> >> > Champaign, IL 61820-6371 >> >> > Phone: 217-300-1924 >> >> > http://www.geog.illinois.edu/~jgrn/ >> >> > AIM: jgrn307, MSN: jgrn...@hotmail.com, Gchat: jgrn307, Skype: jgrn3007 >> >> > >> >> > ______________________________________________ >> >> > R-help@r-project.org mailing list >> >> > https://stat.ethz.ch/mailman/listinfo/r-help >> >> > PLEASE do read the posting guide >> >> > http://www.R-project.org/posting-guide.html >> >> > and provide commented, minimal, self-contained, reproducible code. >> >> >> >> ______________________________________________ >> >> R-help@r-project.org mailing list >> >> https://stat.ethz.ch/mailman/listinfo/r-help >> >> PLEASE do read the posting guide >> >> http://www.R-project.org/posting-guide.html >> >> and provide commented, minimal, self-contained, reproducible code. >> >> >> >> -- >> Jonathan A. Greenberg, PhD >> Assistant Professor >> Global Environmental Analysis and Remote Sensing (GEARS) Laboratory >> Department of Geography and Geographic Information Science >> University of Illinois at Urbana-Champaign >> 259 Computing Applications Building, MC-150 >> 605 East Springfield Avenue >> Champaign, IL 61820-6371 >> Phone: 217-300-1924 >> http://www.geog.illinois.edu/~jgrn/ >> AIM: jgrn307, MSN: jgrn...@hotmail.com, Gchat: jgrn307, Skype: jgrn3007 > > ______________________________________________ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.