Hi, tracing through your changes - it looks like: Smalltalk cleanUp: true except: #() confirming: false. Takes care of all the non-unicode changes you proposed (and it seems like its a known cleanup protocol). I wonder if the Unicode change is worth it/risky as many web based services I might connect to with Zinc do support Unicode so maybe I should keep that one in. (I will for now - might verify how much of a difference it really makes)
I think my next port of call is cleanUp for Monticello/Metacello as I see a fair amount of that stuff floating around in my image (after I’ve used it to bootstrap my code). Tim > On 16 Aug 2017, at 02:32, Guillermo Polito <guillermopol...@gmail.com> wrote: > > Actually it happens first that monticello is "nicely" coupled with the > changeset system and logs all the source code loaded in change sets :D :/ ¬¬. > Also, the first two strings in terms of size are related to unicode tables > (we should put them in files instead of in the image and load them on > demand), and the two biggest arrays also to unicode. I just tried the > following in a clean bootstrapped "minimal" image (metacello): > > "Careful, this will make that #isLetter, #isUppercase #isLowercase, > #toLowercase and #toUppercase only work on ascii" > Character characterSet: nil. > Unicode classPool at: #GeneralCategory put: nil. > Unicode classPool at: #DecimalProperty put: nil. > > UnicodeDefinition removeFromSystem. > ChangeSet removeChangeSetsNamedSuchThat: [ :each | true ]. > ChangeSet resetCurrentToNewUnnamedChangeSet. > MCDefinition clearInstances. > Undeclared removeUnreferencedKeys. > Smalltalk garbageCollect. > > like this: > > ./vm/pharo Pharo7.0-metacello-32bit-fa236b7.image eval --save "Character > characterSet: nil. Unicode classPool at: #GeneralCategory put: nil. Unicode > classPool at: #DecimalProperty put: nil. UnicodeDefinitions removeFromSystem. > ChangeSet removeChangeSetsNamedSuchThat: [ :each | true ]. ChangeSet > resetCurrentToNewUnnamedChangeSet. MCDefinition clearInstances. Undeclared > removeUnreferencedKeys. Smalltalk garbageCollect." > > and my image went down from 11MB to 6.6MB (7.0 MB if I don't change back to > ascii with the first three lines) > > Then I tried a tally: > > ./vm/pharo Pharo7.0-metacello-32bit-fa236b7.image save spacetally > > ./vm/pharo spacetally.image eval --save "repo := MCFileTreeRepository new > directory: '../src' asFileReference. version := repo > loadVersionFromFileNamed: 'Tool-Profilers.package'. version load." > > re-clean since i loaded some packages > > ./vm/pharo spacetally.image eval --save "ChangeSet > removeChangeSetsNamedSuchThat: [ :each | true ]. ChangeSet > resetCurrentToNewUnnamedChangeSet. MCDefinition clearInstances. Undeclared > removeUnreferencedKeys. Smalltalk garbageCollect." > > This image is now 6.6MB (7.1MB with the unicode large arrays), 4.1% of > strings (274k) what seems reasonable. Remaining big strings are Pharo's > licence, the buffer of the changes file and then some class comments > (shouldn't they be fetched from disk as any other method source code?). > > Making again a tally shows that ~30% of the space is taken by Arrays and > 21.9% by compiled methods. But, BUT! :) I have ~30k arrays and lots of > collections also: > > "MethodDictionary" 2872 + > "IdentitySet" 12781 + > "OrderedCollection" 4398 + > "Set" 2959 + > "Dictionary" 1997 + > "IdentityDictionary" 454 > ----------------------------------------------- > 25461 > > So there are ~5k arrays that are used outside collections. > > Worth exploring a bit more I think. > > On Wed, Aug 16, 2017 at 1:23 AM, Guillermo Polito <guillermopol...@gmail.com > <mailto:guillermopol...@gmail.com>> wrote: > > > On Tue, Aug 15, 2017 at 11:26 PM, Tim Mackinnon <tim@testit.works > <mailto:tim@testit.works>> wrote: > Hi Guille/Ben - I got a quick moment to try the SpaceTally (aside: it seems > very convoluted to load a single package into the image, I was trying to > avoid having to create a baselineOf for something so simple - I ended up with: > > I know, I also believe we have to simplify this. In any case, baselines are > healthy as they allow to also express dependencies. Otherwise you'll end up > loading dependencies by hand. We'll fix this soon I hope. > > > repo := MCFileTreeRepository new directory: './bootstrap' asFileReference. > version := repo loadVersionFromFileNamed: 'Tool-Profilers.package'. > version load. > > Anyway - in my minimal image, like in the fat image there seems to be a > surprising amount of bytestrings (4mb worth?). I think that might need some > digging into? It seems like a lot somehow. Although Ben’s neat experiment of > zipping strings shows that’s not a real route. > > In a deployed minimal image - maybe I can get rid of some other things like > MethodChangeRecords or MCMethodDefiniion’s (but they are smaller wins - but > noticeable) > > Class code space # instances inst > space percent inst average size > ByteString 2640 37365 > 4823848 21.50 129.10 > Array 3742 53002 > 3961944 17.60 74.75 > CompiledMethod 19159 30481 > 2912968 13.00 95.57 > Association 1148 58348 > 1867136 8.30 32.00 > MethodChangeRecord 431 34312 > 1097984 4.90 32.00 > ByteArray 4605 290 > 908728 4.00 3133.54 > ByteSymbol 1698 22689 > 840168 3.70 37.03 > IdentitySet 408 19076 > 610432 2.70 32.00 > MethodDictionary 3310 3520 > 608688 2.70 172.92 > WeakArray 1758 3024 > 597824 2.70 197.69 > MCMethodDefinition 4318 6659 > 426176 1.90 64.00 > Protocol 1679 8382 > 268224 1.20 32.00 > OrderedCollection 6555 5509 > 220360 1.00 40.00 > > As an aside - my Gitlab project is public, the scripts that load things up > are in ./scripts (build.sh, and minimal.st <http://minimal.st/> and > loadlocal.st <http://loadlocal.st/>) > > Tim > >> On 15 Aug 2017, at 08:02, Guillermo Polito <guillermopol...@gmail.com >> <mailto:guillermopol...@gmail.com>> wrote: >> >> >> >> On Mon, Aug 14, 2017 at 4:42 PM, Tim Mackinnon <tim@testit.works >> <mailto:tim@testit.works>> wrote: >> Hi Guille - just running SpaceTally on my dev image to get a feel for it. It >> turns out that in the minimal images you’ve been creating, its not loaded >> (makes sense). >> >> Yup, it's loaded afterwards. >> >> All packages are loaded through metacello baselines. We should start >> refactoring and making standalone projects, each one with a baseline for >> himself, and his own dependencies described. >> >> I was checking on your gitlab and I have probably no access: how are you >> finally loading packages in the bootstrap image? Can you share that with us >> in text? I'd like to improve that situation. >> >> I’m wondering if there is an easy way to import it in (I guess that package >> should be in the Pharo git tree I cloned to get Fuel loaded right? Or is >> there a separate standalone source?). >> >> Yes it is, you can get the package programatically doing >> >> SpaceTally package name >> >> And furthermore, get the baseline that currently is loading by doing >> >> package := SpaceTally package name. >> BaselineOf subclasses select: [ :e | >> e project version packages anySatisfy: [ :p | p name = package ]]. >> >> >> Thanks for all the support, and your email about why the contexts stack up >> is very well received (I will comment over there). >> >> By the way - it looks like Martin Fowler picked up on this announcement - so >> maybe we might get some interest from his mass of followers. >> >> Tim >> >>> On 14 Aug 2017, at 10:49, Guillermo Polito <guillermopol...@gmail.com >>> <mailto:guillermopol...@gmail.com>> wrote: >>> >>> Hi Tim, >>> >>> On Mon, Aug 14, 2017 at 11:41 AM, Tim Mackinnon <tim@testit.works >>> <mailto:tim@testit.works>> wrote: >>> Hey guys, thanks for your enthusiasm around this - and I cannot stress >>> enough how this was only possible because of the work that has gone into >>> making Pharo (in particular the 64bit image, as well as having a minimal >>> image, and some great blog posts on serialising contexts) as well as the >>> patience from everyone in answering questions and helping me get it all >>> working. >>> >>> I’m still quite keen to get my execution time back down under 800ms and I’d >>> like to actually get back to writing a few skills to automate a few things >>> around my house. >>> >>> To Answer Denis’ question - >>> >>> My final footprint is 30.4mb - thats composed of a 22mb image (with a >>> simple example that pulls in Fuel, ZTimestamp and the S3 Library which >>> depends on XMLParser) and then the VM (from which I removed obvious dll’s). >>> >>> In my original experiments with a 6.0 minimal image - I did manage to get >>> to a 13.4mb image (which started out as 12mb original size, and then loaded >>> in STON and had only a simple clock example). I think the sweet spot is >>> around 20mb total footprint as that seems to get me into the 450ms-900ms >>> range. >>> >>> The 7.0 min image now starts out at 15mb and then I’m not sure why loading >>> Fuel, S3 and XMLParser takes 7mb (it seems big to me - but I’ve not dug >>> into that). >>> >>> You can do further space analysis using the following expression >>> >>> SpaceTally new printSpaceAnalysis >>> >>> You can do that in an eval and check what's taking space. With measures we >>> can iterate and improve :). >>> >>> I’ve also found (and this on the back of unserialising the context in my >>> example) that the way we build images has 15+ saved stack sessions that >>> have saved on top of each other from the way we build up the images. I >>> don’t yet know the implications of size/speed of these - but we need a >>> better way of folding executions when we snapshot headless images. I’m also >>> not clear if there are any other startup tasks that take precious time >>> (this also has implications for our fat development images as they take >>> much longer to appear than they really should). >>> >>> I'm working on this as I'm writing this mail ;) >>> >>> https://pharo.fogbugz.com/f/cases/20309 >>> <https://pharo.fogbugz.com/f/cases/20309> >>> https://github.com/pharo-project/pharo/pull/196 >>> <https://github.com/pharo-project/pharo/pull/196> >>> >>> I'll write down the implications further in a different thread. >>> >>> >>> I’ll be exploring some of these size/speed tradeoff’s in follow on messages. >>> >>> But once again, a big thanks - I’ve not enjoyed programming like this for >>> ages. >>> >>> Tim >>> >>>> On 12 Aug 2017, at 16:26, Ben Coman <b...@openinworld.com >>>> <mailto:b...@openinworld.com>> wrote: >>>> >>>> hi Tim, >>>> >>>> That is..... AWESOME! >>>> >>>> Very nice delivery - it flowed well with great narration. >>>> >>>> I loved @2:17 "this is the interesting piece, because PharoLambda has >>>> serialized the execution context of its application and saved it into [my >>>> S3 bucket] ... [then on the local machine] rematerializes a debugger [on >>>> that context]." >>>> >>>> There is a clarity in your video presentation that really may intrigue >>>> outsiders. As a community we should push this on the usual hacker forums - >>>> ycombinator could be a good starting point (but I'm locked out of my >>>> account there). >>>> An enticing title could be... >>>> "Debugging Lambdas by re-materializing saved execution contexts on your >>>> local machine." >>>> >>>> cheers -ben >>>> >>>> On Fri, Aug 11, 2017 at 3:37 PM, Denis Kudriashov <dionisi...@gmail.com >>>> <mailto:dionisi...@gmail.com>> wrote: >>>> This is cool Tim. >>>> >>>> So what image size you deployed at the end? >>>> >>>> 2017-08-10 15:47 GMT+02:00 Tim Mackinnon <tim@testit.works >>>> <mailto:tim@testit.works>>: >>>> I just wanted to thank everyone for their help in getting my pet project >>>> further along, so that now I can announce that PharoLambda is now working >>>> with the V7 minimal image and also supports post mortem debugging by >>>> saving a zipped fuel context onto S3. >>>> >>>> This latter item is particularly satisfying as at a recent serverless >>>> conference (JeffConf) there was a panel where poor development tools on >>>> serverless platforms was highlighted as a real problem. >>>> >>>> In our community we’ve had these kinds of tools at our fingertips for ages >>>> - but I don’t think the wider development community has really noticed. >>>> Debugging something short lived like a Lambda execution is quite >>>> startling, as the current answer is “add more logging”, and we all know >>>> that sucks. To this end, I’ve created a little screencast showing this in >>>> action - and it was pretty cool because it was a real example I >>>> encountered when I got everything working and was trying my test >>>> application out. >>>> >>>> I’ve also put a bit of work into tuning the excellent GitLab CI tools, so >>>> that I can cache many of the artefacts used between different build runs >>>> (this might also be of interest to others using CI systems). >>>> >>>> The Gitlab project is on: https://gitlab.com/macta/PharoLambda >>>> <https://gitlab.com/macta/PharoLambda> >>>> And the screencast: https://www.youtube.com/watch?v=bNNCT1hLA3E >>>> <https://www.youtube.com/watch?v=bNNCT1hLA3E> >>>> >>>> Tim >>>> >>>> >>>>> On 15 Jul 2017, at 00:39, Tim Mackinnon <tim@testit.works >>>>> <mailto:tim@testit.works>> wrote: >>>>> >>>>> Hi - I’ve been playing around with getting Pharo to run well on AWS >>>>> Lambda. It’s early days, but I though it might be interesting to share >>>>> what I’ve learned so far. >>>>> >>>>> Usage examples and code at https://gitlab.com/macta/PharoLambda >>>>> <https://gitlab.com/macta/PharoLambda> >>>>> >>>>> With help from many of the folks here, I’ve been able to get a simple >>>>> example to run in 500ms-1200ms with a minimal Pharo 6 image. You can >>>>> easily try it out yourself. This seems slightly better than what the >>>>> GoLang folks have been able to do. >>>>> >>>>> Tim >>>> >>>> >>>> >>> >>> >>> >>> >>> -- >>> >>> Guille Polito >>> >>> Research Engineer >>> French National Center for Scientific Research - http://www.cnrs.fr >>> <http://www.cnrs.fr/> >>> >>> >>> Web: http://guillep.github.io <http://guillep.github.io/> >>> Phone: +33 06 52 70 66 13 <tel:+33%206%2052%2070%2066%2013> >> >> >> >> -- >> >> Guille Polito >> >> Research Engineer >> French National Center for Scientific Research - http://www.cnrs.fr >> <http://www.cnrs.fr/> >> >> >> Web: http://guillep.github.io <http://guillep.github.io/> >> Phone: +33 06 52 70 66 13 <tel:+33%206%2052%2070%2066%2013> > > > > -- > > Guille Polito > > Research Engineer > French National Center for Scientific Research - http://www.cnrs.fr > <http://www.cnrs.fr/> > > > Web: http://guillep.github.io <http://guillep.github.io/> > Phone: +33 06 52 70 66 13 <tel:+33%206%2052%2070%2066%2013> > > > -- > > Guille Polito > > Research Engineer > French National Center for Scientific Research - http://www.cnrs.fr > <http://www.cnrs.fr/> > > > Web: http://guillep.github.io <http://guillep.github.io/> > Phone: +33 06 52 70 66 13
