On 20 December 2016 at 17:40, Martin Maechler wrote: | >>>>> Steve Bronder <sbron...@stevebronder.com> | >>>>> on Tue, 20 Dec 2016 01:34:31 -0500 writes: | | > Thanks Henrik this is very helpful! I will try this out on our tests and | > see if gcDLLs() has a positive effect. | | > mlr currently has tests broken down by learner type such as classification, | > regression, forecasting, clustering, etc.. There are 83 classifiers alone | > so even when loading and unloading across learner types we can still hit | > the MAX_NUM_DLLS error, meaning we'll have to break them down further (or | > maybe we can be clever with gcDLLs()?). I'm CC'ing Lars Kotthoff and Bernd | > Bischl to make sure I am representing the issue well. | | This came up *here* in May 2015 | and then May 2016 ... did you not find it when googling. | | Hint: Use | site:stat.ethz.ch MAX_NUM_DLLS | as search string in Google, so it will basically only search the | R mailing list archives | | Here's the start of that thread : | | https://stat.ethz.ch/pipermail/r-devel/2016-May/072637.html | | There was not a clear conclusion back then, notably as | Prof Brian Ripley noted that 100 had already been an increase | and that a large number of loaded DLLs decreases look up speed. | | OTOH (I think others have noted that) a large number of DLLs | only penalizes those who *do* load many, and we should probably | increase it. | | Your use case of "hyper packages" which load many others | simultaneously is somewhat convincing to me... in so far as the | general feeling is that memory should be cheap and limits should | not be low. | | (In spite of Brian Ripleys good reasons against it, I'd still | aim for a *dynamic*, i.e. automatically increased list here).
Yes. Start with 10 or 20, add 10 as needed. Still fast in the 'small N' case and no longer a road block for the 'big N' case required by mlr et al. As a C++ programmer, I am now going to hug my std::vector and quietly retreat. Dirk | Martin Maechler | | > Regards, | | > Steve Bronder | > Website: stevebronder.com | > Phone: 412-719-1282 | > Email: sbron...@stevebronder.com | | | > On Tue, Dec 20, 2016 at 1:04 AM, Henrik Bengtsson < | > henrik.bengts...@gmail.com> wrote: | | >> On reason for hitting the MAX_NUM_DLLS (= 100) limit is because some | >> packages don't unload their DLLs when they being unloaded themselves. | >> In other words, there may be left-over DLLs just sitting there doing | >> nothing but occupying space. You can remove these, using: | >> | >> R.utils::gcDLLs() | >> | >> Maybe that will help you get through your tests (as long as you're | >> unloading packages). gcDLLs() will look at base::getLoadedDLLs() and | >> its content and compare to loadedNamespaces() and unregister any | >> "stray" DLLs that remain after corresponding packages have been | >> unloaded. | >> | >> I think it would be useful if R CMD check would also check that DLLs | >> are unregistered when a package is unloaded | >> (https://github.com/HenrikBengtsson/Wishlist-for-R/issues/29), but of | >> course, someone needs to write the code / a patch for this to happen. | >> | >> /Henrik | >> | >> On Mon, Dec 19, 2016 at 6:01 PM, Steve Bronder | >> <sbron...@stevebronder.com> wrote: | >> > This is a request to increase MAX_NUM_DLLS in Rdynload.c in from 100 to | >> 500. | >> > | >> > On line 131 of Rdynload.c, changing | >> > | >> > #define MAX_NUM_DLLS 100 | >> > | >> > to | >> > | >> > #define MAX_NUM_DLLS 500 | >> > | >> > | >> > In development of the mlr package, there have been several episodes in | >> the | >> > past where we have had to break up unit tests because of the "maximum | >> > number of DLLs reached" error. This error has been an inconvenience that | >> is | >> > going to keep happening as the package continues to grow. Is there more | >> > than meets the eye with this error or would everything be okay if the | >> above | >> > line changes? Would that have a larger effect in other parts of R? | >> > | >> > As R grows, we are likely to see more 'meta-packages' such as the | >> > Hadley-verse, caret, mlr, etc. need an increasing amount of DLLs loaded | >> at | >> > any point in time to conduct effective unit tests. If MAX_NUM_DLLS is | >> set | >> > to 100 for a very particular reason than I apologize, but if it is | >> possible | >> > to increase MAX_NUM_DLLS it would at least make the testing at mlr much | >> > easier. | >> > | >> > I understand you are all very busy and thank you for your time. | >> > | >> > | >> > Regards, | >> > | >> > Steve Bronder | >> > Website: stevebronder.com | >> > Phone: 412-719-1282 | >> > Email: sbron...@stevebronder.com | >> > | >> > [[alternative HTML version deleted]] | >> > | >> > ______________________________________________ | >> > R-devel@r-project.org mailing list | >> > https://stat.ethz.ch/mailman/listinfo/r-devel | >> | | > [[alternative HTML version deleted]] | | > ______________________________________________ | > R-devel@r-project.org mailing list | > https://stat.ethz.ch/mailman/listinfo/r-devel | | ______________________________________________ | R-devel@r-project.org mailing list | https://stat.ethz.ch/mailman/listinfo/r-devel -- http://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org ______________________________________________ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
