Ah, but then it may be more interesting to have a data image (maybe a lot of these) and a front end image.
Isn't Seamless something that could help us here? No need to bring the data back, just manipulate it through proxies. FWIW, I have 2PB of data. Not going to fit in RAM. But also would takes ages to load anyway, so we work on pieces. FWIW2: one Hadoop cluster I am managing now will grow by, I think, some 100s of servers in the coming months. 64bit Pharo is something I can deploy on the boxes, which are 128GB-256GB things with like 32-48 cores. As I mentioned before, if you want to run something on it with Pharo, be my guest. Phil On Thu, Nov 10, 2016 at 9:12 AM, Igor Stasenko <[email protected]> wrote: > > On 10 November 2016 at 07:27, Tudor Girba <[email protected]> wrote: > >> Hi Igor, >> >> Please refrain from speaking down on people. >> >> > Hi, Doru! > I just wanted to hear you :) > > >> If you have a concrete solution for how to do things, please feel free to >> share it with us. We would be happy to learn from it. >> >> > Well, there's so many solutions, that i even don't know what to offer, and > given the potential of smalltalk, i wonder why > you are not employing any. But in overall it is a quesition of storing > most of your data on disk, and only small portion of it > in image (in most optimal cases - only the portion that user sees/operates > with). > As i said to you before, you will hit this wall inevitably, no matter how > much memory is available. > So, what stops you from digging in that direction? > Because even if you can fit all data in memory, consider how much time it > takes for GC to scan 4+ Gb of memory, comparing to > 100 MB or less. > I don't think you'll find it convenient to work in environment where > you'll have 2-3 seconds pauses between mouse clicks. > So, of course, my tone is not acceptable, but its pain to see how people > remain helpless without even thinking about > doing what they need. We have Fuel for how many years now? > So it can't be as easy as it is, just serialize the data and purge it from > image, till it will be required again. > Sure it will require some effort, but it is nothing comparing to day to > day pain that you have to tolerate because of lack of solution. > > >> Cheers, >> Tudor >> >> >> > On Nov 10, 2016, at 4:11 AM, Igor Stasenko <[email protected]> wrote: >> > >> > Nice progress, indeed. >> > Now i hope at the end of the day, the guys who doing data >> mining/statistical analysis will finally shut up and happily be able >> > to work with more bloat without need of learning a ways to properly >> manage memory & resources, and implement them finally. >> > But i guess, that won't be long silence, before they again start >> screaming in despair: please help, my bloat doesn't fits into memory... :) >> > >> > On 9 November 2016 at 12:06, Sven Van Caekenberghe <[email protected]> >> wrote: >> > OK, I am quite excited about the future possibilities of 64-bit Pharo. >> So I played a bit more with the current test version [1], trying to push >> the limits. In the past, it was only possible to safely allocate about >> 1.5GB of memory even though a 32-bit process' limit is theoretically 4GB >> (the OS and the VM need space too). >> > >> > Allocating a couple of 1GB ByteArrays is one way to push memory use, >> but it feels a bit silly. So I loaded a bunch of projects (including >> Seaside) to push the class/method counts (7K classes, 100K methods) and >> wrote a script [2] that basically copies part of the class/method metadata >> including 2 copies of each's methods source code as well as its AST >> (bypassing the cache of course). This feels more like a real object graph. >> > >> > I had to create no less than 7 (SEVEN) copies (each kept open in an >> inspector) to break through the mythical 4GB limit (real allocated & used >> memory). >> > >> > <Screen Shot 2016-11-09 at 11.25.28.png> >> > >> > I also have the impression that the image shrinking problem is gone >> (closing everything frees memory, saving the image has it return to its >> original size, 100MB in this case). >> > >> > Great work, thank you. Bright future again. >> > >> > Sven >> > >> > PS: Yes, GC is slower; No, I did not yet try to save such a large image. >> > >> > [1] >> > >> > VM here: http://bintray.com/estebanlm/pharo-vm/build#files/ >> > Image here: http://files.pharo.org/get-files/60/pharo-64.zip >> > >> > [2] >> > >> > | meta | >> > ASTCache reset. >> > meta := Dictionary new. >> > Smalltalk allClassesAndTraits do: [ :each | | classMeta methods | >> > (classMeta := Dictionary new) >> > at: #name put: each name asSymbol; >> > at: #comment put: each comment; >> > at: #definition put: each definition; >> > at: #object put: each. >> > methods := Dictionary new. >> > classMeta at: #methods put: methods. >> > each methodsDo: [ :method | | methodMeta | >> > (methodMeta := Dictionary new) >> > at: #name put: method selector; >> > at: #source put: method sourceCode; >> > at: #ast put: method ast; >> > at: #args put: method argumentNames asArray; >> > at: #formatted put: method ast formattedCode; >> > at: #comment put: (method comment ifNotNil: [ :str | str >> withoutQuoting ]); >> > at: #object put: method. >> > methods at: method selector put: methodMeta ]. >> > meta at: each name asSymbol put: classMeta ]. >> > meta. >> > >> > >> > >> > -- >> > Sven Van Caekenberghe >> > Proudly supporting Pharo >> > http://pharo.org >> > http://association.pharo.org >> > http://consortium.pharo.org >> > >> > >> > >> > >> > >> > >> > >> > -- >> > Best regards, >> > Igor Stasenko. >> >> -- >> www.tudorgirba.com >> www.feenk.com >> >> "We can create beautiful models in a vacuum. >> But, to get them effective we have to deal with the inconvenience of >> reality." >> >> >> > > > -- > Best regards, > Igor Stasenko. >
