Robert, As far as I can remember, the problem of substitutions at materialziation time was because... as you may have read, Fuel first serializes "objects" and then the references. At materialization, it first creates the objects and then, on a second step, sets the references betwen objects. So the problem was WHERE to place the hook, at which point in time. If we do it just after objects were created, then the substitution code would NOT have access to any of the instVars of that object. If we do it AFTER objects creation and after objects references, then there is no easy way to "substitute" which doesn't involve #become: (because the graph is already constructed) And you know that #become: is still terrible slow because it scans full memory. That will change in Spur.
As for Marea and Ghost, Ghost proxies paper: https://hal.inria.fr/hal-01081236/document Current Ghost repo: http://smalltalkhub.com/#!/~CAR/Ghost Marea paper: http://www.jot.fm/issues/issue_2013_01/article2.pdf Current repo: http://ss3.gemstone.com/ss/Marea.html And finally, my PhD thesis: https://tel.archives-ouvertes.fr/tel-00764991/document In Marea I needed custom clusters for my proxies because the serializer itself sends messages to the objects being serialized. My proxies would bring back graphs from a secondary memory. So if I was serializing a graph that had proxies already, I didn't want that. So I hooked my custom cluster for proxies that send only a few special messages to the proxy that these understand and answer rather than intercept those messages. As for how to extend Fuel for this, I recommend to check the code of Marea. See categories 'Marea-Serializers' and 'Marea-Proxies'. I have a Marea one click here: https://www.dropbox.com/sh/xp8jzyypmz0898j/AACRdHno6V7UfhaJ1ofTPPXva?dl=0 Cheers, On Wed, Oct 14, 2015 at 11:40 AM, Robert Withers <[email protected] > wrote: > Good morning, Max. Thank you for the example. I got a little confused, > between migrations and substitutions. My issue with no-arg blocks, I > believe, is the inability to access my scope object to maintain the object > tables. > > I'm attempting to write my own Materializer, Decoder and Materialization. > At the moment, I'm just going to walk the graph, testing and do #becomes:. > See how well that works when I get a test. > > It's really good to know about that other list. > > thanks so much ^^ > Robert > > On 10/14/2015 04:15 AM, Max Leske wrote: > >> BTW, there is a dedicated Fuel mailing list: >> [email protected] <mailto:[email protected] >> > >> >> Max >> >> >> On 14 Oct 2015, at 09:45, Max Leske <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> >>>> On 14 Oct 2015, at 04:39, Robert Withers <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> >>>> >>>> On 10/13/2015 09:43 PM, Mariano Martinez Peck wrote: >>>> >>>>> >>>>> >>>>> On Tue, Oct 13, 2015 at 10:33 PM, Robert Withers >>>>> <[email protected] <mailto:[email protected]> >>>>> <mailto:[email protected]>> wrote: >>>>> >>>>> Hi Mariano, >>>>> >>>>> This presents me with a big challenge, then. I read the docs and >>>>> explored the code and the only other aspect not mentioned, beyond >>>>> instance creation (#fuelNew #fuelNew:) and postMaterialization >>>>> (#fuelAfterMaterialization), is migrations. However, migration only >>>>> allows for instanceVar mappings, no code blocks. >>>>> >>>>> >>>>> What do you mean that migrations only allows instVar mappings , and no >>>>> code blocks? I mean, what do you mean by code blocks? >>>>> >>>> >>> Sounds to me like this (see FuelOutStackDebuAction): >>> >>> serializeTestFailureContext: aContext toFileNamed: aFilename >>> | serializer | >>> >>> serializer := FLSerializer newDefault. >>> self encodeDebugInformationOn: serializer. >>> serializer addPostMaterializationAction: [ :materialization | >>> Smalltalk tools debugger >>> openOn: Processor activeProcess >>> context: materialization root >>> label: 'External stack' >>> contents: nil >>> fullView: false ]. >>> >>> serializer >>> " use the sender context, generally the current context is not >>> interesting" >>> serialize: aContext >>> toFileNamed: aFilename >>> >>> This stores a block in the serialization which is evaluated after >>> materialization. The only requirement is that it’s a clean block (no >>> closure!). >>> >>> We also support class renames. This is here: >>>>> >>>>> http://rmod.inria.fr/web/software/Fuel/Version1.9/Documentation/Migration?_s=Pvs4DYqPRjfEy8sg&_k=X6ltJu7FDSxumm4X&_n&22 >>>>> >>>>> Which kind of migration example you have in mind that would not be >>>>> supported? An example would help. >>>>> >>>> >>>> Well, my pics will demonstrate. I am interested in doing more than >>>> mappping ivars or a class rename. I want to do a total substitution, >>>> then a further substitution on the receiving, import side: >>>> >>>> Vat1: anObject (Class A) ---> On wire: desc (Descriptor) ---> Vat2: >>>> aProxy (Class FarERef) >>>> >>>> A desc is substituted for a PassByProxy object, then a FarERef is >>>> substituted for the desc. >>>> >>>> >>>>> #fuelAccept: is a serialization side method. >>>>> >>>>> If Fuel supports substitution on serialization, I don't understand >>>>> why no substitution support on materialization. >>>>> >>>>> >>>>> There was a reason, which I cannot remember completely. Maybe Martin or >>>>> Max can remember. >>>>> >>>> >>>> It seems your focus was pickling to disk then back. My focus is >>>> distributed proxy graphs, which has different use cases. >>>> >>>> >>>>> >>>>> I am definitely going to use the world-class Fuel binary >>>>> serialization system. However, I find myself needing to extend Fuel >>>>> to support substitution on materialization. Perhaps the solution is >>>>> a custom decoder. >>>>> >>>>> >>>>> I have made custom clusters for example for my Ghost proxies of Marea >>>>> system. It was a perfect example of how I could extent Fuel besides the >>>>> common hooks. Fuel provides many places for extending , like clusters, >>>>> analyzer, etc >>>>> >>>> >>>> Right on, exactly! Could you tell me more about your Ghost proxies >>>> and Marea, please? As well, could you mention how you select a custom >>>> cluster on the serialization side? >>>> >>>> >>>> thanks so much ^^ >>>> Robert >>>> >>>> >>>> >>>>> No, a bit more. It looks like I need a new >>>>> FLSubstitutePointerObjectCluster, write them on serialization with >>>>> the substitute, then do unsubstitution on materialization, since the >>>>> cluster controls materialization and not the decoder. >>>>> >>>>> Does this approach seem sound to you, from a you know architecture >>>>> and design approach? >>>>> >>>>> >>>>> There was an issue. Hope other can remember. If not, I will try to >>>>> explan what I remember tomorrow. >>>>> >>>>> >>>>> thanks so much ^^ >>>>> Robert >>>>> >>>>> On 10/13/2015 04:49 PM, Mariano Martinez Peck wrote: >>>>> >>>>> No, unfortunately, as far as I can remember, we do not have >>>>> that. There >>>>> are another hooks you may use but only in certain scenarios >>>>> (#fuelNew, >>>>> #fuelAfterMaterialization, global sends, etc). But everything is >>>>> listed >>>>> in >>>>> >>>>> http://rmod.inria.fr/web/software/Fuel/Version1.9/Documentation/CustomizingGraph >>>>> so if you didn't find anything of help in there there are >>>>> chances >>>>> there isn't anything. >>>>> >>>>> Cheers, >>>>> >>>>> On Tue, Oct 13, 2015 at 5:30 PM, Robert Withers >>>>> <[email protected] >>>>> <mailto:[email protected]> <mailto:[email protected] >>>>> > >>>>> <mailto:[email protected] >>>>> <mailto:[email protected]>>> wrote: >>>>> >>>>> Yes, I meant dynamic substitution on materialization, to >>>>> use the >>>>> correct terminology. >>>>> >>>>> thanks, >>>>> Robert >>>>> >>>>> >>>>> On 10/13/2015 11:40 AM, Max Leske wrote: >>>>> >>>>> >>>>> On 13 Oct 2015, at 17:16, Robert Withers >>>>> <[email protected] >>>>> <mailto:[email protected]> >>>>> <mailto:[email protected]> >>>>> <mailto:[email protected] >>>>> <mailto:[email protected]>>> wrote: >>>>> >>>>> Every extra source helps, thank you. I see how to do >>>>> non-stream substitutions on materializations, but >>>>> the >>>>> documentation did not indicate a way to do >>>>> non-stream >>>>> substitutions on serialization. Is it possible? >>>>> >>>>> >>>>> I don’t understand what you mean by “non-stream”. Could >>>>> you give >>>>> an example? >>>>> >>>>> >>>>> thanks, >>>>> Robert >>>>> >>>>> On 10/13/2015 09:00 AM, Mariano Martinez Peck wrote: >>>>> >>>>> Hi Robert, >>>>> >>>>> As for the documentation, you have LOTS of >>>>> tests, you >>>>> have the chapter >>>>> Torsten pasted, you have this documentation: >>>>> http://rmod.inria.fr/web/software/Fuel >>>>> >>>>> But also, as for internals, there is a journal >>>>> paper we >>>>> wrote: >>>>> http://rmod.lille.inria.fr/archives/papers/Dias12a-SPE-Fuel.pdf >>>>> >>>>> Let us know how it goes, >>>>> >>>>> >>>>> On Tue, Oct 13, 2015 at 6:00 AM, Torsten >>>>> Bergmann >>>>> <[email protected] <mailto:[email protected]> >>>>> <mailto:[email protected]> >>>>> <mailto:[email protected] <mailto:[email protected]>> >>>>> <mailto:[email protected] <mailto:[email protected]> >>>>> <mailto:[email protected] <mailto:[email protected]>>>> wrote: >>>>> >>>>> Hi Robert, >>>>> >>>>> Also checkout the chapter on Fuel in Pharo >>>>> Enterprise book: >>>>> >>>>> >>>>> https://ci.inria.fr/pharo-contribution/view/Books/job/EnterprisePharoBook/lastSuccessfulBuild/artifact/book-result/EnterprisePharo-A4.pdf >>>>> >>>>> Bye >>>>> Torsten >>>>> >>>>> Gesendet: Dienstag, 13. Oktober 2015 um >>>>>> >>>>> 09:44 Uhr >>>>> >>>>>> Von: "Robert Withers" >>>>>> >>>>> <[email protected] >>>>> <mailto:[email protected]> >>>>> <mailto:[email protected]> >>>>> <mailto:[email protected] >>>>> <mailto:[email protected]>> >>>>> <mailto:[email protected] >>>>> <mailto:[email protected]> >>>>> <mailto:[email protected] >>>>> <mailto:[email protected]>>>> >>>>> >>>>>> An: [email protected] <mailto:[email protected]> >>>>>> >>>>> <mailto:[email protected]> >>>>> <mailto:[email protected] >>>>> <mailto:[email protected]>> >>>>> <mailto:[email protected] >>>>> <mailto:[email protected]> >>>>> <mailto:[email protected] >>>>> <mailto:[email protected]>>> >>>>> >>>>>> Betreff: Re: [Pharo-dev] binary >>>>>> >>>>> serialization >>>>> >>>>>> >>>>>> Yes, I have to do object substitutions. >>>>>> >>>>> Thanks >>>>> for the link! >>>>> >>>>>> >>>>>> thanks, >>>>>> Robert >>>>>> >>>>>> On 10/13/2015 03:43 AM, Max Leske wrote: >>>>>> >>>>>>> >>>>>>> On 13 Oct 2015, at 09:40, Robert Withers >>>>>>>> >>>>>>> <[email protected] >>>>> <mailto:[email protected]> >>>>> <mailto:[email protected]> >>>>> <mailto:[email protected] >>>>> <mailto:[email protected]>> >>>>> <mailto:[email protected] >>>>> <mailto:[email protected]> >>>>> <mailto:[email protected] >>>>> <mailto:[email protected]>>>> wrote: >>>>> >>>>>> >>>>>>>> Sven and Torsten, that's a binary >>>>>>>> >>>>>>> serialization library! It >>>>> will take time to learn it and how to use >>>>> mappers. >>>>> >>>>>> >>>>>>>> What is the format; is it language >>>>>>>> >>>>>>> neutral? >>>>> >>>>>> >>>>>>> For quick serialization you don’t >>>>>>> >>>>>> need to do >>>>> anything. It works >>>>> for (almost) all objects. Only if you >>>>> want to >>>>> exclude things or >>>>> treat some objects in a special way, you >>>>> will need >>>>> to do some stuff. >>>>> >>>>>> >>>>>>> Documentation: >>>>>>> >>>>>> http://rmod.inria.fr/web/software/Fuel. >>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>>> thanks, >>>>>>>> Robert >>>>>>>> >>>>>>>> On 10/13/2015 01:21 AM, Sven Van >>>>>>>> >>>>>>> Caekenberghe >>>>> wrote: >>>>> >>>>>> Yes, it is called FUEL and it is a >>>>>>>>> >>>>>>>> standard >>>>> part of the >>>>> image. See FLSerializer and FLMaterializer. >>>>> >>>>>> >>>>>>>>> On 13 Oct 2015, at 06:59, Robert >>>>>>>>>> >>>>>>>>> Withers >>>>> <[email protected] >>>>> <mailto:[email protected]> >>>>> <mailto:[email protected]> >>>>> <mailto:[email protected] >>>>> <mailto:[email protected]>> >>>>> <mailto:[email protected] >>>>> <mailto:[email protected]> >>>>> <mailto:[email protected] >>>>> <mailto:[email protected]>>>> wrote: >>>>> >>>>>> >>>>>>>>>> Does Pharo have stream classes to >>>>>>>>>> >>>>>>>>> binary >>>>> de/serialize an >>>>> object, such that the protocol accepts an >>>>> object as >>>>> an argument and >>>>> converts it to a byteArray? >>>>> >>>>>> >>>>>>>>>> -- >>>>>>>>>> thanks, >>>>>>>>>> Robert >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Mariano >>>>> http://marianopeck.wordpress.com >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Mariano >>>>> http://marianopeck.wordpress.com >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Mariano >>>>> http://marianopeck.wordpress.com >>>>> >>>> <Exporting Vat.jpg><Importing Vat.jpg> >>>> >>> >> > -- Mariano http://marianopeck.wordpress.com
