Re: [Asdf-devel] startup times and initialize-source-registry
I have implemented support for .cl-source-registry.cache in the faster-registry branch, and added a script ./bin/update-cl-source-registry.lisp to create or update cache files, as in: ~/common-lisp/asdf/bin/update-cl-source-registry.lisp ~/common-lisp/ It does significantly speed up the scanning of the source-registry indeed, and thus slash the startup time of cl-launch from noticeably slow to acceptable. Problem: it's only useful if you use the install-asdf-as-module script to override your implementation's ASDF with that from the branch, or else, the old asdf will still spend time scanning the source-registry at startup... except the script seems to be broken in master, unlike in the minimakefile branch. Sigh. So I merged minimakefile into faster-registry. (Note to Robert: that's one more thing broken in master that's working in minimakefile.) There are still more things to speed up in starting cl-launch, though, like the loading (and possibly upgrading) of ASDF, and (on SBCL) priming the CLOS caches for all the ASDF classes and functions. But it looks like they are out of my reach, and they account for most of the .6s of startup left on SBCL. —♯ƒ • François-René ÐVB Rideau •ReflectionCybernethics• http://fare.tunes.org Hi! I'm a signature virus. Copy me into your sig file and help me spread! On Thu, Aug 28, 2014 at 11:17 AM, Faré f...@tunes.org wrote: Any insta-theories for where other half second comes from? If you mean the second half of cl-launch's startup time, I fear it might be a combination of shell and CL compilation overhead, but I admit I haven't tried timing where the time is going. You're welcome to investigate. At some point, I was thinking of translating cl-launch to CL and having it output a bootstrap script to make itself. Then it would have a fast mode that loads code onto the cl-launch image, rather than start a new process (with maybe a different CL implementation) to implement the requested software. At that point, maybe Xach's buildapp is a better starting point. Or not. The other half of cl-launch's startup time is spent in CL, and is definitely a performance bug: % time env SBCL=/bin/true cl -ip 1 env SBCL=/bin/true cl -ip 1 0.03s user 0.02s system 102% cpu 0.045 total % time env CL_SOURCE_REGISTRY=/does/not/exist cl -ip 1 env CL_SOURCE_REGISTRY=/does/not/exist cl -ip 1 0.43s user 0.06s system 98% cpu 0.494 total Inserting a few (time) forms and enabling verbose loading, I find that the cl:load'ing of cl-launch itself is taking most of the time, and the most time in that is loading then upgrading asdf, i.e. loading the fasl's for it. I'm not sure how to speed THAT up; ASDF is big, and not likely to shrink significantly while preserving backward compatibility. SBCL also seems to be spending some notable fraction of the time compiling the cl-launch code itself, though most of the time is the fasl loading. Conclusion: FASt Loading is only so fast. PS: your topology.sh file has a : ; # ... comment as if it were a bilingual script, but it isn't bilingual. Also, the path /usr/local/bin/cl is not standard and neither is the -S argument. —♯ƒ • François-René ÐVB Rideau •ReflectionCybernethics• http://fare.tunes.org The Army is a place where you get up early in the morning to be yelled at by people with short haircuts and tiny brains. — Dave Barry ___ Asdf-devel mailing list Asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
Re: [Asdf-devel] startup times and initialize-source-registry
get’n off topic … scripting isn’t asdf ... should find a room. On Aug 28, 2014, at 11:17 AM, Faré f...@tunes.org wrote: PS: your topology.sh file has a : ; # ... comment as if it were a bilingual script, but it isn't bilingual. right. vistigal. cl-shim has arrived. life is getting better. Also, the path /usr/local/bin/cl is not standard true, i blame the makefile. look'n forward teaching emacs the mode of these files. and neither is the -S argument. not long for this world. cleaning shaved 4 seconds of the run time soon I can remove that damn link farm. standard - snort. tis tacky. thanks for proof reading! new revision[1] - ben [1] https://github.com/bhyde/yed-lisp/blob/master/topology.sh http://enthusiasm.cozy.orgtel:+1-781-791-3054 I forecast sunny weather! The paradox of information retrieval is that a person must describe the information that he or she does not have. -- Christine L. Borgman ___ Asdf-devel mailing list Asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
Re: [Asdf-devel] startup times and initialize-source-registry
On Fri, Aug 29, 2014 at 10:07 AM, Ben Hyde bh...@pobox.com wrote: get’n off topic … scripting isn’t asdf ... should find a room. No, YOU go first! not long for this world. cleaning shaved 4 seconds of the run time soon I can remove that damn link farm. Oh, how did you shave it? Was it a yak? Did you just make the link farm your regularly configured source-registry? Do you have a clever patch to asdf? new revision[1] [1] https://github.com/bhyde/yed-lisp/blob/master/topology.sh I recommend option -sp yed-lisp so you don't need all these package qualifications. —♯ƒ • François-René ÐVB Rideau •ReflectionCybernethics• http://fare.tunes.org To fight a violent enemy, violence is necessary; but to fight violence itself, violence is vain. ___ Asdf-devel mailing list Asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
Re: [Asdf-devel] startup times and initialize-source-registry
Any insta-theories for where other half second comes from? If you mean the second half of cl-launch's startup time, I fear it might be a combination of shell and CL compilation overhead, but I admit I haven't tried timing where the time is going. You're welcome to investigate. At some point, I was thinking of translating cl-launch to CL and having it output a bootstrap script to make itself. Then it would have a fast mode that loads code onto the cl-launch image, rather than start a new process (with maybe a different CL implementation) to implement the requested software. At that point, maybe Xach's buildapp is a better starting point. Or not. The other half of cl-launch's startup time is spent in CL, and is definitely a performance bug: % time env SBCL=/bin/true cl -ip 1 env SBCL=/bin/true cl -ip 1 0.03s user 0.02s system 102% cpu 0.045 total % time env CL_SOURCE_REGISTRY=/does/not/exist cl -ip 1 env CL_SOURCE_REGISTRY=/does/not/exist cl -ip 1 0.43s user 0.06s system 98% cpu 0.494 total Inserting a few (time) forms and enabling verbose loading, I find that the cl:load'ing of cl-launch itself is taking most of the time, and the most time in that is loading then upgrading asdf, i.e. loading the fasl's for it. I'm not sure how to speed THAT up; ASDF is big, and not likely to shrink significantly while preserving backward compatibility. SBCL also seems to be spending some notable fraction of the time compiling the cl-launch code itself, though most of the time is the fasl loading. Conclusion: FASt Loading is only so fast. PS: your topology.sh file has a : ; # ... comment as if it were a bilingual script, but it isn't bilingual. Also, the path /usr/local/bin/cl is not standard and neither is the -S argument. —♯ƒ • François-René ÐVB Rideau •ReflectionCybernethics• http://fare.tunes.org The Army is a place where you get up early in the morning to be yelled at by people with short haircuts and tiny brains. — Dave Barry ___ Asdf-devel mailing list Asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
Re: [Asdf-devel] startup times and initialize-source-registry
My cl-launch scripts can take one second.
I can trim a half second off that with a disk-cache[2] of *source-registry*.
Tedious details follow.
My cl-launch install defaults to using ccl. My ccl install is currently HEAD.
My current setup has 3.8K working directories, which compute-source-registry
scans.
Ok I have three little bash functions[1] to time things.
try-ccl- — run ccl with —no-init
try-cl — run cl-launch, so no init file; but it will load asdf.
try-ccl — run ccl with standard quicklisp init file
Without the disk cache.
try-ccl- 0.03 seconds
try-cl 1 seconds
try-ccl1 seconds
With the disk cache:
try-ccl- 0.03 seconds
try-cl 0.5 seconds
try-ccl 0.5 seconds
These #’s are gleaned from batches of 10 runs so things can settle.
Theories for that remaining half a second need looking into.
- ben
[1] The three bash functions
try-ccl-(){ (time ccl --no-init --eval '(progn (print
(lisp-implementation-type)) (quit))') 21 | grep real ; }
try-ccl(){ (time ccl --eval '(progn (print (lisp-implementation-type))
(quit))') 21 | grep real ; }
try-cl(){ (time cl -ip '(lisp-implementation-type)') 21 | grep real ; }
[2] A proof of concept disk cache...
https://gist.github.com/bhyde/526c5acb38f341d60dba
So, anytime somebody rearranges the set of asdf systems they
are expected to invoke reset-quickasdf.
quickasdf horrible name - but it bemused me at the time.
Calling initialize-source-registry directly from the repl takes
0.47 seconds without the diskcache and .004 seconds with it.
___
Asdf-devel mailing list
Asdf-devel@common-lisp.net
http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
Re: [Asdf-devel] startup times and initialize-source-registry
On Wed, Aug 27, 2014 at 1:11 PM, Ben Hyde bh...@pobox.com wrote: My cl-launch scripts can take one second. I can trim a half second off that with a disk-cache[2] of *source-registry*. I'm jealous. Your machine is almost 25% faster than mine. [2] A proof of concept disk cache... https://gist.github.com/bhyde/526c5acb38f341d60dba The cache should ideally be per source-registry entry; and managed by the same entity that manages said entry. Thus, I was thinking of an optional second form in cl-source-registry.conf file. Or a separate .cl-source-registry.cache file. So, anytime somebody rearranges the set of asdf systems they are expected to invoke reset-quickasdf. quickasdf horrible name - but it bemused me at the time. Ouch. Calling initialize-source-registry directly from the repl takes 0.47 seconds without the diskcache and .004 seconds with it. Yes, that's the general thing I'm thinking about: kind of back to the concept of link farms of ASDF 1, but only as an optimization, without any additional user configuration required, and leaving system administration to system administrators. —♯ƒ • François-René ÐVB Rideau •ReflectionCybernethics• http://fare.tunes.org To surrender to ignorance and call it God has always been premature, and it remains premature today. — Isaac Asimov ___ Asdf-devel mailing list Asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
Re: [Asdf-devel] startup times and initialize-source-registry
On Aug 27, 2014, at 1:35 PM, Faré fah...@gmail.com wrote: On Wed, Aug 27, 2014 at 1:11 PM, Ben Hyde bh...@pobox.com wrote: My cl-launch scripts can take one second. I can trim a half second off that with a disk-cache[2] of *source-registry*. I'm jealous. Your machine is almost 25% faster than mine. weird, it’s nothing special. I bought something I could hold it open with one hand while hitting staff with the other. http://j.mp/1lfhMwY [2] A proof of concept disk cache... https://gist.github.com/bhyde/526c5acb38f341d60dba The cache should ideally be per source-registry entry; and managed by the same entity that manages said entry. Thus, I was thinking of an optional second form in cl-source-registry.conf file. Or a separate .cl-source-registry.cache file. I saw that. i’m ambivalent about the complexity. So, anytime somebody rearranges the set of asdf systems they are expected to invoke reset-quickasdf. quickasdf horrible name - but it bemused me at the time. Ouch. I love achieving my goals. Calling initialize-source-registry directly from the repl takes 0.47 seconds without the diskcache and .004 seconds with it. Yes, that's the general thing I'm thinking about: kind of back to the concept of link farms Any insta-theories for where other half second comes from? - ben Since English is a mess, it maps well onto the problem space, which is also a mess, which we call reality.” - Larry Wall ___ Asdf-devel mailing list Asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
Re: [Asdf-devel] startup times and initialize-source-registry
The cache should ideally be per source-registry entry; and managed by the same entity that manages said entry. Thus, I was thinking of an optional second form in cl-source-registry.conf file. Or a separate .cl-source-registry.cache file. I saw that. i’m ambivalent about the complexity. Checking for .cl-source-registry.cache in collect-sub*directories-asd-files should be straightforward. NB: If you're going to hack these things, please use the faster-registry branch. You don't have an account on common-lisp.net yet, but it looks like, after a year, it's still the same machine that some people were angry about and calling for the replacement of, and I happen to be root there, so I can open you an account and give you write access to the asdf repository (if Robert agrees) — send me your ssh public key. Any insta-theories for where other half second comes from? If you mean the second half of cl-launch's startup time, I fear it might be a combination of shell and CL compilation overhead, but I admit I haven't tried timing where the time is going. You're welcome to investigate. At some point, I was thinking of translating cl-launch to CL and having it output a bootstrap script to make itself. Then it would have a fast mode that loads code onto the cl-launch image, rather than start a new process (with maybe a different CL implementation) to implement the requested software. At that point, maybe Xach's buildapp is a better starting point. Or not. —♯ƒ • François-René ÐVB Rideau •ReflectionCybernethics• http://fare.tunes.org None are more hopelessly enslaved than those who falsely believe they are free. — Johann Wolfgang von Goethe ___ Asdf-devel mailing list Asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
Re: [Asdf-devel] startup times and initialize-source-registry
Faré wrote: The cache should ideally be per source-registry entry; and managed by the same entity that manages said entry. Thus, I was thinking of an optional second form in cl-source-registry.conf file. Or a separate .cl-source-registry.cache file. I saw that. i’m ambivalent about the complexity. Checking for .cl-source-registry.cache in collect-sub*directories-asd-files should be straightforward. NB: If you're going to hack these things, please use the faster-registry branch. I'm having a little trouble keeping track of this discussion. Is there a clear argument for this caching to be a core piece of ASDF instead of a contrib? Could the disk cache be handled by code that massages *system-definition-search-functions* in a contrib? Certainly not as currently defined, but with some rejiggering? Or could we add a means to hook the source registry initialization process? I note also that the quickasdf proposal has no room for people who have multiple *different* lisp configurations. Currently, if you work on different projects, with different library sets, you can change the environment variable to change contexts, you can point your ASDF at different configuration files, or you can use code that changes the value of asdf:*central-registry*. For quickasdf to be an acceptable integration into ASDF proper, it would have to be able to be reconfigurable in exactly the same ways, and guarantee that when users change their current working configuration *in any one of these ways*, they cannot get a some other configuration's disk cache. So: for the moment, no. Quickasdf's disk cache will NOT be integrated into ASDF. There are way too many open questions. See my earlier email manifesto as ASDF maintainer, and new follow on. Best, r ___ Asdf-devel mailing list Asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
Re: [Asdf-devel] startup times and initialize-source-registry
… argument for this caching to be a core piece of ASDF … ? My apologies, I should have known that saying “proof of concept” wasn’t sufficient to signal that - yeah! , it’s a vile hack not ready to be integrated into anything, including my own day to day usage. In fact I put forward code because that’s how I test that my ideas have any grounding in reality. Good point about how people are likely to be switching configs, not one of they many open issues I’d already noticed. ___ Asdf-devel mailing list Asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
Re: [Asdf-devel] startup times and initialize-source-registry
.. access to the asdf repository .. that strikes me as premature :) … meanwhile I’ll be offline for a bit … mad trip to NYC in my immediate future.___ Asdf-devel mailing list Asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
Re: [Asdf-devel] startup times and initialize-source-registry
Would changing (:tree path) so it accepts an optional argument be less adhoc? Possibly (:tree path key (depth nil) (don-not-recure-after-asdf-encountered nil)) That said this would have little if any value in my use case. I point ask it to sweep over my the directory where all my coding lives, and most of the directories in there are not lisp, and so there ain’t any asd files in those. - ben ___ Asdf-devel mailing list Asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
Re: [Asdf-devel] startup times and initialize-source-registry
On Tue, Aug 26, 2014 at 1:40 PM, Ben Hyde bh...@pobox.com wrote: Would changing (:tree path) so it accepts an optional argument be less adhoc? Possibly (:tree path key (depth nil) (don-not-recure-after-asdf-encountered nil)) I of course thought about this, but it's the wrong thing: the person who knows best what's in a tree is not the end-user, but the developer, package-manager or system-administrator, and that's who should be doing this configuring. Therefore, this configuration option is ultimately no help unless the developer, package-manager or system-administrator gets to write configuration files, which is essentially my proposal, at which point this option is not needed, because it can be done better without, with a simple script builds a list of all asd file paths under the tree. Proposal: * :tree checks for cl-source-registry.conf or .cl-source-registry.conf and recurses on that rather than on a directory search, when available. * :file is accepted, in addition to :directory and :tree * a :source-registry entry can have a :cache entry (or be followed by a :cache entry? or have a .cache file equivalent?) that lists all relevant .asd files as a sorted list of relative unix-namestrings. * Some script can regenerate the cache from the registry. That said this would have little if any value in my use case. I point ask it to sweep over my the directory where all my coding lives, and most of the directories in there are not lisp, and so there ain’t any asd files in those. My proposal would solve your issue, at the cost of re-running the script when you install, deinstall or update lisp code. Or you could split your active CL source in a separate directory. —♯ƒ • François-René ÐVB Rideau •ReflectionCybernethics• http://fare.tunes.org It's better to know nothing that to know what ain't so. — Josh Billings ___ Asdf-devel mailing list Asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
Re: [Asdf-devel] startup times and initialize-source-registry
Ben Hyde wrote: Would changing (:tree path) so it accepts an optional argument be less adhoc? Possibly (:tree path key (depth nil) (don-not-recure-after-asdf-encountered nil)) That said this would have little if any value in my use case. I point ask it to sweep over my the directory where all my coding lives, and most of the directories in there are not lisp, and so there ain’t any asd files in those. I think the problem with this is that, as Fare points out, it's the author of the system who has laid out the directory tree, and who knows whether or not there are system definitions floating around in it. Similarly, if the library author makes the modification, that's work for one person. If the callers do, it's work for everyone. cheers, r ___ Asdf-devel mailing list Asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
Re: [Asdf-devel] startup times and initialize-source-registry
On Aug 26, 2014, at 3:22 PM, Robert P. Goldman rpgold...@sift.info wrote: Ben Hyde wrote: Would changing (:tree path) so it accepts an optional argument be less adhoc? Possibly (:tree path key (depth nil) (don-not-recure-after-asdf-encountered nil)) That said this would have little if any value in my use case. I point ask it to sweep over my the directory where all my coding lives, and most of the directories in there are not lisp, and so there ain’t any asd files in those. I think the problem with this is that, as Fare points out, it's the author of the system who has laid out the directory tree your both quite wise, i agree___ Asdf-devel mailing list Asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
Re: [Asdf-devel] startup times and initialize-source-registry
On Aug 26, 2014, at 2:40 PM, Faré fah...@gmail.com wrote: * a :source-registry entry can have a :cache entry (or be followed by a :cache entry? or have a .cache file equivalent?) that lists all relevant .asd files as a sorted list of relative unix-namestrings. * Some script can regenerate the cache from the registry. Something of this kind - A disk cache of *source-registry* - is my preference de jour. It’s trivial to dump the *source-registry* to a file and reload it. My hack here along those lines is almost nothing more than a list of pathnames. The code which builds the source registry fascinates my inner archeologist; I wouldn’t want to write that in bash. It is easier to explain to users that there is one disk cache, v.s. N. … re-running the script when you install, deinstall or update lisp code … I often have very long running lisp processes. So I need to do exactly that via asdf:clear-source-repository etc. Who knows when to update/rebuild the disk cache? Sometimes only the developer. But quicklisp knows when it (un)installes things. I’d be tempted to update the disk cache when that happens. - ben http://enthusiasm.cozy.orgtel:+1-781-791-3054 I forecast sunny weather! Communication rests on a foundation of mutual perspective-taking, the mechanics of which are poorly understood.___ Asdf-devel mailing list Asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
Re: [Asdf-devel] startup times and initialize-source-registry
On Thu, Aug 21, 2014 at 2:10 PM, Robert P. Goldman rpgold...@sift.info wrote: Quick PS: what mechanism do you think should be used to tweak this setting? Should presumably be something easy to specify (i.e., not a config file), so that one can quickly start a lisp script, without messing up one's conventional lisp development environment. Environment variable? Since there's no portable way to do this by command-line argument I'd like to avoid unnecessary environment variables. Once again, where under a hierarchy the .asd files are is ultimately the knowledge and responsibility of the curators of the respective source trees, not of the end-user. Therefore, the absence of recursion should be a matter of said source trees including a file cl-source-registry.conf or .cl-source-registry.conf (visible having priority over hidden), that specifies how (not) to recurse in that tree. A simple script could be provided for system writers to create or update said file. A year after all relevant packages have migrated, we can make it the default and people can drop said configuration file when they follow the default behavior of not having meaningful .asd files except directly under the top directory. Furthermore, with a :file directive, these configuration files can directly list the .asd files without any further filesystem access. —♯ƒ • François-René ÐVB Rideau •ReflectionCybernethics• http://fare.tunes.org The highest of minds / Often have built for themselves / The tallest of jails. ___ Asdf-devel mailing list Asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
Re: [Asdf-devel] startup times and initialize-source-registry
On Mon, Aug 25, 2014 at 12:28 PM, Robert P. Goldman rpgold...@sift.info wrote: Once again, where under a hierarchy the .asd files are is ultimately the knowledge and responsibility of the curators of the respective source trees, not of the end-user. Therefore, the absence of recursion should be a matter of said source trees including a file cl-source-registry.conf or .cl-source-registry.conf (visible having priority over hidden), that specifies how (not) to recurse in that tree. A simple script could be provided for system writers to create or update said file. A year after all relevant packages have migrated, we can make it the default and people can drop said configuration file when they follow the default behavior of not having meaningful .asd files except directly under the top directory. Furthermore, with a :file directive, these configuration files can directly list the .asd files without any further filesystem access. I don't think that this new behavior should ever be the default. Scripting is the edge-case for CL, not building of large, complex systems. Further, for some CL implementations (like ABCL, as Mark points out), scripting use is *never* a good option. Well, we'll see. But faster startup times benefit everyone, not just people using Lisp for scripting. Shaving a second or so from startup time (somewhat less or much more depending on the implementation and the size of your source trees) is useful whether you're (re)starting SLIME, running tests in a batch job, building in a slave or cross-compiling as well as if you're scripting. From a cost tradeoff PoV, there are few scripting configurations now, and fixing them all is easy and cheap, since they are for early-adopters. Fixing all systems that contain nested system definitions is neither easy nor cheap. That's why I proposed a two year migration plan, of the same kind we did for unicode support. I believe it may be worth it. On the other hand, if we support this search for cl-source-registry.conf while recursing, then whichever trees are managed (by e.g. quicklisp or debian or clbuild) can have this cl-source-registry.conf automatically optimized by the same manager, so only unmanaged source code downloaded by the user remains slow, and this should be small, except for power users who are big boys enough to do the system administration step of regenerating that file when they update their repositories. When one wants scripting, it should be easy to specify a scripting lisp. For now, I suggest that people who want to script with CL should build themselves a pre-configured image for the purpose. That image could have a feature or ASDF configuration variable set so to change the default behavior to cut off recursive ASDF search. Configuration files in source trees would complement this behavior. Yes, you can already cut a lot of the time by dumping an image. That approach would serve the purpose of making this behavior easy to specify at the command-line or mouse-click. Later, if CL-based scripting catches on, lisp implementations could ship with versions that are intended for rapid start-up and scripting, avoiding the need for scripters to build their own images. Or separate scripting packages could be provided. The problem is that it's not a simple matter of a scripting lisp vs a non-scripting lisp, since many libraries (and/or managed set of libraries) needs to be modified to keep working in this scheme for (faster) source-registry initialization. —♯ƒ • François-René ÐVB Rideau •ReflectionCybernethics• http://fare.tunes.org To stay young requires unceasing cultivation of the ability to unlearn old falsehoods. — Robert Heinlein, Time Enough For Love ___ Asdf-devel mailing list Asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
Re: [Asdf-devel] startup times and initialize-source-registry
On 21 Aug 2014, at 02:36, Faré fah...@gmail.com wrote: […] The trick here is in this new stop-at-asd flag, which here defaults to t and isn't configurable, but which should default to nil and be configurable, for backward compatibility. Its effect is that recursing into subdirectories stops if a .asd file is found in the toplevel directory. This saves a lot of recursing, and would save even more if a .asd file of symlink to one exists at the top of a git hierarchy. But this is incompatible with a lot of existing code, and so the transition will be long and painful if this is adopted. If you proposing that the stop-at-asd property would be somehow configurable in the DEFSYSTEM form, like: (asdf:defsystem :foo :contains-interior-asdf-defintions :components … then please ensure that this is present when/if you introduce this change to ASDF. But I get the feeling that in order to speed things up, you weren’t intending to parse the DEFSYSTEM form in your search. If you are proposing that the user would have to do explicitly do some sort of configuration “for this instance of a user using asdf with this Lisp implementation”, I won’t be so happy because: 1) This sort of configuration hasn’t been necessary before, so we will introduce complexity in ASDF deployment for efficiency in using CL as a scripting language which is something I don’t currently use (Admittedly because my platform, ABCL, based on the JVM, is just not going to ever have reasonable startup times. Although there are systems that keep a JVM “warmed up” for firing such one-off commands to, and for specialised JVM there are memory mapped solutions for faster startup). 2) I am using a system (lsw2) not in Quicklisp that has many such “interior” ASDF definitions. Usually when systems get in this state it is because they are big enough that nobody has time to package them correctly, so they tend to stay that way. If I can’t put a flag in the top-level system, I’m going to run into problems when users haven’t done the per-user-per-lisp configuration. -- A screaming comes across the sky. It has happened before but there is nothing to compare to it now. ___ Asdf-devel mailing list Asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
Re: [Asdf-devel] startup times and initialize-source-registry
On Thu, Aug 21, 2014 at 2:38 AM, Mark Evenson even...@panix.com wrote: On 21 Aug 2014, at 02:36, Faré fah...@gmail.com wrote: […] The trick here is in this new stop-at-asd flag, which here defaults to t and isn't configurable, but which should default to nil and be configurable, for backward compatibility. Its effect is that recursing into subdirectories stops if a .asd file is found in the toplevel directory. This saves a lot of recursing, and would save even more if a .asd file of symlink to one exists at the top of a git hierarchy. But this is incompatible with a lot of existing code, and so the transition will be long and painful if this is adopted. If you proposing that the stop-at-asd property would be somehow configurable in the DEFSYSTEM form, like: (asdf:defsystem :foo :contains-interior-asdf-defintions :components … then please ensure that this is present when/if you introduce this change to ASDF. But I get the feeling that in order to speed things up, you weren’t intending to parse the DEFSYSTEM form in your search. Indeed, requiring to parse a .asd file is a bad idea — and is even worse when there are hundreds of .asd files in the directory. But maybe we could detect a file called source-registry.conf or something similar, and parse that to look for subdirectories with .asd files in them. In the absence of such a file, the default behavior would for backward compatibility be to always recurse, or maybe for speed in a future version years from now be to recurse only if no .asd file was found. If you are proposing that the user would have to do explicitly do some sort of configuration “for this instance of a user using asdf with this Lisp implementation”, I won’t be so happy because: 1) This sort of configuration hasn’t been necessary before, so we will introduce complexity in ASDF deployment for efficiency in using CL as a scripting language which is something I don’t currently use (Admittedly because my platform, ABCL, based on the JVM, is just not going to ever have reasonable startup times. Although there are systems that keep a JVM “warmed up” for firing such one-off commands to, and for specialised JVM there are memory mapped solutions for faster startup). 2) I am using a system (lsw2) not in Quicklisp that has many such “interior” ASDF definitions. Usually when systems get in this state it is because they are big enough that nobody has time to package them correctly, so they tend to stay that way. If I can’t put a flag in the top-level system, I’m going to run into problems when users haven’t done the per-user-per-lisp configuration. What about supporting a source-registry.conf or .asdf-search.conf or similar file to control recursion of the search, and eventually requiring it when a directory both has a .asd file yet requires recursion? ASDF should do the right thing and require no configuration from users, but minimal configuration of their own directory structures by programmers is acceptable — except that for backward compatibility, it should default to always recurse for now. —♯ƒ • François-René ÐVB Rideau •ReflectionCybernethics• http://fare.tunes.org The problem with Unix ever becoming a widely popular desktop operating system is referred to as the 'guru in the box' problem. To get and keep Unix running smoothly you need a captive guru on site and there just aren't enough gurus to put in the shipping boxes. — Brian Kernighan ___ Asdf-devel mailing list Asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
Re: [Asdf-devel] startup times and initialize-source-registry
If I understand correctly, the proposal is to require configuration only for the special case of wanting faster start up, and absent that, configuration will be as before, since optimization for scripting is the exceptional case. That seems like a benign modification. I'd accept such a patch (with bumping of version for easy detection). We should document it appropriately, of course. On August 21, 2014 2:39:24 AM CDT, Faré fah...@gmail.com wrote: On Thu, Aug 21, 2014 at 2:38 AM, Mark Evenson even...@panix.com wrote: On 21 Aug 2014, at 02:36, Faré fah...@gmail.com wrote: […] The trick here is in this new stop-at-asd flag, which here defaults to t and isn't configurable, but which should default to nil and be configurable, for backward compatibility. Its effect is that recursing into subdirectories stops if a .asd file is found in the toplevel directory. This saves a lot of recursing, and would save even more if a .asd file of symlink to one exists at the top of a git hierarchy. But this is incompatible with a lot of existing code, and so the transition will be long and painful if this is adopted. If you proposing that the stop-at-asd property would be somehow configurable in the DEFSYSTEM form, like: (asdf:defsystem :foo :contains-interior-asdf-defintions :components … then please ensure that this is present when/if you introduce this change to ASDF. But I get the feeling that in order to speed things up, you weren’t intending to parse the DEFSYSTEM form in your search. Indeed, requiring to parse a .asd file is a bad idea — and is even worse when there are hundreds of .asd files in the directory. But maybe we could detect a file called source-registry.conf or something similar, and parse that to look for subdirectories with .asd files in them. In the absence of such a file, the default behavior would for backward compatibility be to always recurse, or maybe for speed in a future version years from now be to recurse only if no .asd file was found. If you are proposing that the user would have to do explicitly do some sort of configuration “for this instance of a user using asdf with this Lisp implementation”, I won’t be so happy because: 1) This sort of configuration hasn’t been necessary before, so we will introduce complexity in ASDF deployment for efficiency in using CL as a scripting language which is something I don’t currently use (Admittedly because my platform, ABCL, based on the JVM, is just not going to ever have reasonable startup times. Although there are systems that keep a JVM “warmed up” for firing such one-off commands to, and for specialised JVM there are memory mapped solutions for faster startup). 2) I am using a system (lsw2) not in Quicklisp that has many such “interior” ASDF definitions. Usually when systems get in this state it is because they are big enough that nobody has time to package them correctly, so they tend to stay that way. If I can’t put a flag in the top-level system, I’m going to run into problems when users haven’t done the per-user-per-lisp configuration. What about supporting a source-registry.conf or .asdf-search.conf or similar file to control recursion of the search, and eventually requiring it when a directory both has a .asd file yet requires recursion? ASDF should do the right thing and require no configuration from users, but minimal configuration of their own directory structures by programmers is acceptable — except that for backward compatibility, it should default to always recurse for now. —♯ƒ • François-René ÐVB Rideau •ReflectionCybernethics• http://fare.tunes.org The problem with Unix ever becoming a widely popular desktop operating system is referred to as the 'guru in the box' problem. To get and keep Unix running smoothly you need a captive guru on site and there just aren't enough gurus to put in the shipping boxes. — Brian Kernighan ___ Asdf-devel mailing list Asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel -- Sent from my Android device with K-9 Mail. Please excuse my brevity.___ Asdf-devel mailing list Asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
Re: [Asdf-devel] startup times and initialize-source-registry
On Thu, Aug 21, 2014 at 9:54 AM, Robert P. Goldman rpgold...@sift.net wrote: If I understand correctly, the proposal is to require configuration only for the special case of wanting faster start up, and absent that, configuration will be as before, since optimization for scripting is the exceptional case. That seems like a benign modification. I'd accept such a patch (with bumping of version for easy detection). We should document it appropriately, of course. Yes, it should all remain backward-compatible, at least unless and until some maintainer leads a two year campaign for migration to a different setting. My plan is as follows: 1- have a special variable tell whether to recurse under a .asd by default, defaulting to t for now 2- adding some keyword argument to :tree to override this variable 3- adding support for source-registry.conf and/or .source-registry.conf as things to detect and heed when recursing into a directory. I modified my previously posted code as a solution for 1, attached. 2 and 3, I'll add to the TODO for now. (And yes, changing startup from 1.45s to .66s with cl-launch (resp. .83s to .33s without) is well worth it. It makes some scripts usable that are otherwise annoyingly slow; the difference is even more dramatic for me on CCL, where it drops from 2.57s to 0.37s with cl-launch (resp. 2.0s to 0.27s without).) PS: while testing my changes, I found a trivial bug in test-program, that failed to rename load-fasl-op to load-bundle-op. Fixed. —♯ƒ • François-René ÐVB Rideau •ReflectionCybernethics• http://fare.tunes.org A flea and a fly in a flue were imprisoned, so what could they do? Said the fly: let us flee!. Said the flea: let us fly!. So they flew thru a flaw in the flue... From 42bdefd38b685bb56f13d483ea763a9b70c345c2 Mon Sep 17 00:00:00 2001 From: Francois-Rene Rideau tu...@google.com Date: Thu, 21 Aug 2014 13:15:27 -0400 Subject: [PATCH] Update a test for the renaming of load-fasl-op to load-bundle-op. --- test/make-hello-world.lisp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/make-hello-world.lisp b/test/make-hello-world.lisp index 11bef3a..bcaa6e0 100644 --- a/test/make-hello-world.lisp +++ b/test/make-hello-world.lisp @@ -28,7 +28,7 @@ (defun make-hello-bundle (operation) - (operate 'load-fasl-op :hello-world-example) + (operate 'load-bundle-op :hello-world-example) (operate operation :hello-world-example) #+mkcl (add-mkcl-dll (asdf::output-file operation :hello-world-example))) -- 2.1.0.rc2.206.gedb03e5 ___ Asdf-devel mailing list Asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
