Tudor, Igor still has a point here. I was talking yesterday with a data science guy and he was indeed more interested in lamenting than working out solutions for his problems.
Which weren't that hard to begin with as all it took is an hour of work to get his results. But I think he felt better complaining and self aggrandizing than actually making things work and move on to the next challenge. Example of his "issues": Him:"I have a looooot of data" Me: "Like what, more or less than 1TB?" Him: "Less than that" Me: "kay, can you give me a sample set of this hard disk?" Him: "Yeah, but no, well, I need to get it first" Me: "Let's sit tomorrow over lunch so that we can ingest it all and work it out" Him: "Let me come back to you..." I think he was more interested in uttering things like "Spark 2.0" "Lots of data" "Star schema" (and saying it loud so that people could hear it) than solving anything real. Overgeneralizing yes, speaking down, heh, not so much. There are indeed super smart/efficient/effective people in data science. But there is also a crowd that is quite, how to say... more interested in the Egyptian-style grand priest status than in the actual problems. Phil On Thu, Nov 10, 2016 at 7:27 AM, Tudor Girba <[email protected]> wrote: > Hi Igor, > > Please refrain from speaking down on people. > > If you have a concrete solution for how to do things, please feel free to > share it with us. We would be happy to learn from it. > > Cheers, > Tudor > > > > On Nov 10, 2016, at 4:11 AM, Igor Stasenko <[email protected]> wrote: > > > > Nice progress, indeed. > > Now i hope at the end of the day, the guys who doing data > mining/statistical analysis will finally shut up and happily be able > > to work with more bloat without need of learning a ways to properly > manage memory & resources, and implement them finally. > > But i guess, that won't be long silence, before they again start > screaming in despair: please help, my bloat doesn't fits into memory... :) > > > > On 9 November 2016 at 12:06, Sven Van Caekenberghe <[email protected]> wrote: > > OK, I am quite excited about the future possibilities of 64-bit Pharo. > So I played a bit more with the current test version [1], trying to push > the limits. In the past, it was only possible to safely allocate about > 1.5GB of memory even though a 32-bit process' limit is theoretically 4GB > (the OS and the VM need space too). > > > > Allocating a couple of 1GB ByteArrays is one way to push memory use, but > it feels a bit silly. So I loaded a bunch of projects (including Seaside) > to push the class/method counts (7K classes, 100K methods) and wrote a > script [2] that basically copies part of the class/method metadata > including 2 copies of each's methods source code as well as its AST > (bypassing the cache of course). This feels more like a real object graph. > > > > I had to create no less than 7 (SEVEN) copies (each kept open in an > inspector) to break through the mythical 4GB limit (real allocated & used > memory). > > > > <Screen Shot 2016-11-09 at 11.25.28.png> > > > > I also have the impression that the image shrinking problem is gone > (closing everything frees memory, saving the image has it return to its > original size, 100MB in this case). > > > > Great work, thank you. Bright future again. > > > > Sven > > > > PS: Yes, GC is slower; No, I did not yet try to save such a large image. > > > > [1] > > > > VM here: http://bintray.com/estebanlm/pharo-vm/build#files/ > > Image here: http://files.pharo.org/get-files/60/pharo-64.zip > > > > [2] > > > > | meta | > > ASTCache reset. > > meta := Dictionary new. > > Smalltalk allClassesAndTraits do: [ :each | | classMeta methods | > > (classMeta := Dictionary new) > > at: #name put: each name asSymbol; > > at: #comment put: each comment; > > at: #definition put: each definition; > > at: #object put: each. > > methods := Dictionary new. > > classMeta at: #methods put: methods. > > each methodsDo: [ :method | | methodMeta | > > (methodMeta := Dictionary new) > > at: #name put: method selector; > > at: #source put: method sourceCode; > > at: #ast put: method ast; > > at: #args put: method argumentNames asArray; > > at: #formatted put: method ast formattedCode; > > at: #comment put: (method comment ifNotNil: [ :str | str > withoutQuoting ]); > > at: #object put: method. > > methods at: method selector put: methodMeta ]. > > meta at: each name asSymbol put: classMeta ]. > > meta. > > > > > > > > -- > > Sven Van Caekenberghe > > Proudly supporting Pharo > > http://pharo.org > > http://association.pharo.org > > http://consortium.pharo.org > > > > > > > > > > > > > > > > -- > > Best regards, > > Igor Stasenko. > > -- > www.tudorgirba.com > www.feenk.com > > "We can create beautiful models in a vacuum. > But, to get them effective we have to deal with the inconvenience of > reality." > > >
