Ship it :) It’s ugly, but will keep the core stable at least through another revision.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: [email protected] WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -----Original Message----- From: Michael Starch <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Friday, November 7, 2014 at 10:39 PM To: "[email protected]" <[email protected]> Subject: Re: Extra Compiler Tools >Chris, Et Al. > >The solution write-up can be found here. Turns out it isn't too >ugly....just has a lot of duplication with the classes it is wrapping. > >https://reviews.apache.org/r/27773/ > >Enjoy, > >-Michael > >On Fri, Nov 7, 2014 at 9:46 AM, Michael Starch <[email protected]> wrote: > >> Sound fine, expect to see this on review board when I get it done and >> passing my test. >> >> Michael >> On Nov 7, 2014 9:44 AM, "Chris Mattmann" <[email protected]> >>wrote: >> >>> Yep I’d like to see the non changing >>> core structs version, in a patch to >>> evaluate it. You said you know how, so let’s see it >>> and then go from there. We can reserve 0.9 to >>> consider changing structs if the code becomes >>> too unwieldily to maintain. >>> >>> Good? >>> >>> Cheers, >>> Chris >>> >>> ------------------------ >>> Chris Mattmann >>> [email protected] >>> >>> >>> >>> >>> -----Original Message----- >>> From: Michael Starch <[email protected]> >>> Reply-To: <[email protected]> >>> Date: Friday, November 7, 2014 at 9:21 AM >>> To: <[email protected]> >>> Subject: Re: Extra Compiler Tools >>> >>> >Chris, >>> > >>> >Did you ever come to a conclusion on this? >>> > >>> >-Michael >>> > >>> >On Wed, Nov 5, 2014 at 7:14 PM, Chris Mattmann >>><[email protected] >>> > >>> >wrote: >>> > >>> >> OK, let me think about this tonight. >>> >> Maybe we can figure this out tomorrow, >>> >> I won’t hold this up longer than that. >>> >> >>> >> ------------------------ >>> >> Chris Mattmann >>> >> [email protected] >>> >> >>> >> >>> >> >>> >> >>> >> -----Original Message----- >>> >> From: Michael Starch <[email protected]> >>> >> Reply-To: <[email protected]> >>> >> Date: Wednesday, November 5, 2014 at 8:23 PM >>> >> To: <[email protected]> >>> >> Subject: Re: Extra Compiler Tools >>> >> >>> >> >According to the specs, any subclass that implements Serializable >>>must >>> >> >manually implement the serialization of the parents' members. I >>> tested >>> >> >this and it fails exactly as expected. The parent's members aren't >>> >> >serialized. >>> >> > >>> >> >Also, JobInput is an interface.... so I would have no way of >>>catching >>> >>all >>> >> >of the possible implementations that could come at me. >>> >> > >>> >> >Michael >>> >> >On Nov 5, 2014 7:06 PM, "Mattmann, Chris A (3980)" < >>> >> >[email protected]> wrote: >>> >> > >>> >> >> Got it, Mike. >>> >> >> >>> >> >> Hmm, how about simply creating SerializableJobSpec and >>> >> >> SerializableJob and SerializableJobInput and then making >>> >> >> them sub-class their parents and implement Serializable. >>> >> >> Then, use these classes in your Mesos implementation. >>> >> >> That seems self-contained, doesn’t change core classes, >>> >> >> and pretty easy, right? >>> >> >> >>> >> >> Cheers, >>> >> >> Chris >>> >> >> >>> >> >> >>> >> >> >>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> >> >> Chris Mattmann, Ph.D. >>> >> >> Chief Architect >>> >> >> Instrument Software and Science Data Systems Section (398) >>> >> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>> >> >> Office: 168-519, Mailstop: 168-527 >>> >> >> Email: [email protected] >>> >> >> WWW: http://sunset.usc.edu/~mattmann/ >>> >> >> >>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> >> >> Adjunct Associate Professor, Computer Science Department >>> >> >> University of Southern California, Los Angeles, CA 90089 USA >>> >> >> >>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> -----Original Message----- >>> >> >> From: Michael Starch <[email protected]> >>> >> >> Reply-To: "[email protected]" <[email protected]> >>> >> >> Date: Wednesday, November 5, 2014 at 8:00 PM >>> >> >> To: "[email protected]" <[email protected]> >>> >> >> Subject: Re: Extra Compiler Tools >>> >> >> >>> >> >> >I need to serialize a JobSpec and children (Job and JobInput) >>>to a >>> >> >>byte[]. >>> >> >> >Java can do this automatically by marking all three as >>> Serializable. >>> >> >> > >>> >> >> >The work around is to manually serialize to a private inner >>>struct >>> >>and >>> >> >> >back >>> >> >> >out again. The inner class will have members for each member in >>> the >>> >> >> >JobSpec and children. Java can the auto-serialize that without >>> >> >>changing >>> >> >> >the other three. >>> >> >> > >>> >> >> >It is ugly, and essentially a reimplementation of those three >>> >> >> >classes....but it is entirely self-contained. >>> >> >> > >>> >> >> >Michael >>> >> >> >On Nov 5, 2014 6:45 PM, "Chris Mattmann" >>><[email protected] >>> > >>> >> >> wrote: >>> >> >> > >>> >> >> >> Hey Mike, >>> >> >> >> >>> >> >> >> Hmm, what’s the work around just so I know >>> >> >> >> what we’re trading against? >>> >> >> >> >>> >> >> >> Cheers, >>> >> >> >> Chris >>> >> >> >> >>> >> >> >> ------------------------ >>> >> >> >> Chris Mattmann >>> >> >> >> [email protected] >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> -----Original Message----- >>> >> >> >> From: Michael Starch <[email protected]> >>> >> >> >> Reply-To: <[email protected]> >>> >> >> >> Date: Wednesday, November 5, 2014 at 6:31 PM >>> >> >> >> To: <[email protected]> >>> >> >> >> Subject: Re: Extra Compiler Tools >>> >> >> >> >>> >> >> >> >That is basically what I did. Regardless, protobuff proves >>>to be >>> >> >> >>overkill. >>> >> >> >> > >>> >> >> >> >If I mark those classes as serializable, the correct >>>solution is >>> >>2 >>> >> >> >>lines >>> >> >> >> >of >>> >> >> >> >code. (protobuff was like 20). Wrote a test case, and it >>>works >>> >> >> >> >perfectly. >>> >> >> >> > >>> >> >> >> >If I cannot make JobSpec Job and JonInput implement >>>Serializable >>> >> >>then >>> >> >> >>the >>> >> >> >> >work around is simple too. >>> >> >> >> > >>> >> >> >> >What do you think? Should I mark them as Serializable, or >>>use a >>> >> >> >> >work-around. Either is a better solution than protobuff. >>> >> >> >> > >>> >> >> >> >Michael >>> >> >> >> >On Nov 5, 2014 4:44 PM, "Chris Mattmann" >>> >><[email protected]> >>> >> >> >> wrote: >>> >> >> >> > >>> >> >> >> >> Mike, have you looked at this yet? >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >>> >> >> >>> >> >>> >> >>> >>>http://techtraits.com/build%20management/maven/2011/09/09/compiling-prot >>>o >>> >> >> >> >>co >>> >> >> >> >> l-buffers-from-maven/ >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> I’m going to play with it tonight and see if >>> >> >> >> >> I can help here. Do you have some files I can test >>> >> >> >> >> with? Can you attach them to JIRA or dropbox them to me >>> >> >> >> >> so I can scope? >>> >> >> >> >> >>> >> >> >> >> Cheers, >>> >> >> >> >> Chris >>> >> >> >> >> >>> >> >> >> >> ------------------------ >>> >> >> >> >> Chris Mattmann >>> >> >> >> >> [email protected] >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> -----Original Message----- >>> >> >> >> >> From: Michael Starch <[email protected]> >>> >> >> >> >> Reply-To: <[email protected]> >>> >> >> >> >> Date: Wednesday, November 5, 2014 at 5:37 PM >>> >> >> >> >> To: <[email protected]> >>> >> >> >> >> Subject: Re: Extra Compiler Tools >>> >> >> >> >> >>> >> >> >> >> >Ok....time for an audible. Protoc needs to be built from >>> >> >>source, no >>> >> >> >> >> >binary >>> >> >> >> >> >distributions available. Thus I am going to purge >>> >>proto-buffers >>> >> >> >>from >>> >> >> >> >>the >>> >> >> >> >> >new code and be done with it. >>> >> >> >> >> > >>> >> >> >> >> >Any problem making the following classes/interfaces >>>implement >>> >> >> >> >> >java.io.Serializable: >>> >> >> >> >> > >>> >> >> >> >> >JobSpec >>> >> >> >> >> >Job >>> >> >> >> >> >JobInput >>> >> >> >> >> > >>> >> >> >> >> >Doing so would allow apache and native java serialization >>>and >>> >> >>thus >>> >> >> >>we >>> >> >> >> >> >wouldn't need something like proto-buffers. >>> >> >> >> >> > >>> >> >> >> >> >-Michael >>> >> >> >> >> >Thanks Mike +1 >>> >> >> >> >> > >>> >> >> >> >> >------------------------ >>> >> >> >> >> >Chris Mattmann >>> >> >> >> >> >[email protected] >>> >> >> >> >> > >>> >> >> >> >> > >>> >> >> >> >> > >>> >> >> >> >> > >>> >> >> >> >> >-----Original Message----- >>> >> >> >> >> >From: Michael Starch <[email protected]> >>> >> >> >> >> >Reply-To: <[email protected]> >>> >> >> >> >> >Date: Wednesday, November 5, 2014 at 12:31 PM >>> >> >> >> >> >To: <[email protected]> >>> >> >> >> >> >Subject: Re: Extra Compiler Tools >>> >> >> >> >> > >>> >> >> >> >> >>Looks like you followed the same reasoning chain that I >>>did. >>> >> >>Yes, >>> >> >> >>I >>> >> >> >> >>came >>> >> >> >> >> >>to the same conclusion that ant-build was best. >>> >> >> >> >> >> >>> >> >> >> >> >>I wasn't sure how to download protoc, but you just >>>answered >>> >> >> >> >>that....so I >>> >> >> >> >> >>think this is a great solution! >>> >> >> >> >> >> >>> >> >> >> >> >>Thanks, >>> >> >> >> >> >> >>> >> >> >> >> >>Michael >>> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >>On Wed, Nov 5, 2014 at 10:23 AM, Chris Mattmann >>> >> >> >> >> >><[email protected]> >>> >> >> >> >> >>wrote: >>> >> >> >> >> >> >>> >> >> >> >> >>> Hi Mike, >>> >> >> >> >> >>> >>> >> >> >> >> >>> Thanks for flushing this out. >>> >> >> >> >> >>> >>> >> >> >> >> >>> My thoughts on the below: >>> >> >> >> >> >>> >>> >> >> >> >> >>> >>> >> >> >> >> >>> -----Original Message----- >>> >> >> >> >> >>> From: Michael Starch <[email protected]> >>> >> >> >> >> >>> Reply-To: <[email protected]> >>> >> >> >> >> >>> Date: Wednesday, November 5, 2014 at 12:12 PM >>> >> >> >> >> >>> To: <[email protected]> >>> >> >> >> >> >>> Subject: Re: Extra Compiler Tools >>> >> >> >> >> >>> >>> >> >> >> >> >>> >I tried this approach. The plugin requires a path to >>>the >>> >> >> >>"protoc" >>> >> >> >> >>tool >>> >> >> >> >> >>>and >>> >> >> >> >> >>> >thus a working installation. This is what prompted >>>the >>> >> >> >>discussion. >>> >> >> >> >> >>> >>> >> >> >> >> >>> Ah - no worries, what you could do is: >>> >> >> >> >> >>> >>> >> >> >> >> >>> 1. only enable to plugin if -Pwith-mesos is enabled; >>>and >>> >> >> >> >> >>> >>> >> >> >> >> >>> > >>> >> >> >> >> >>> >Running the plugin under a profile works. >>> >> >> >> >> >>> >>> >> >> >> >> >>> Yep. >>> >> >> >> >> >>> >>> >> >> >> >> >>> > However, not running the plugin >>> >> >> >> >> >>> >causes compile errors in dependant code. Excluding >>>this >>> >>code >>> >> >> >> >>except >>> >> >> >> >> >>> >within >>> >> >> >> >> >>> >the profile doesn't seem to work, and is considered by >>> >>some >>> >> >>to >>> >> >> >>be >>> >> >> >> >>bad >>> >> >> >> >> >>>form >>> >> >> >> >> >>> >because there is nothing inside the jar file that >>>notes >>> >>which >>> >> >> >> >>profiles >>> >> >> >> >> >>> >were >>> >> >> >> >> >>> >used to compile. >>> >> >> >> >> >>> >>> >> >> >> >> >>> Got it. Suggestion here would be: >>> >> >> >> >> >>> >>> >> >> >> >> >>> 2. create a new module, cas-resource-mesos, and inside >>>of >>> >>that >>> >> >> >> >>module, >>> >> >> >> >> >>> take one of the following approaches, assuming the >>>module >>> >>is >>> >> >> >> >>activated >>> >> >> >> >> >>> when -Pwith-mesos is enabled: >>> >> >> >> >> >>> >>> >> >> >> >> >>> 2a. Maven Antrun like so (in this old example): >>> >> >> >> >> >>> >>> >> >> >> >> >>> >>> >> >> >> >> >>> >> >> >> >>> >> >> >>> >> >>> >>>http://stackoverflow.com/questions/1578456/integrate-protocol-buffers-in >>> >> >> >> >> >>>t >>> >> >> >> >> >>>o- >>> >> >> >> >> >>> maven2-build >>> >> >> >> >> >>> >>> >> >> >> >> >>> (pro: more flexibility in case protoc isn¹t there; to >>>fail >>> >>on >>> >> >> >> >>error; to >>> >> >> >> >> >>> only compile if >>> >> >> >> >> >>> protoc is available >>> >> >> >> >> >>> >>> >> >> >> >> >>> 2b. Maven protobuf plugin >>> >> >> >> >> >>> >>> >>http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html >>> >> >> >> >> >>> >>> >> >> >> >> >>> Here¹s how to enable a module with a profile: >>> >> >> >> >> >>> >>> >> >> >> >> >>> >>> >> >> >> >> >>> >>> >> >> >> >> >>> >> >> >> >>> >> >> >>> >> >>> >>>http://blog.soebes.de/blog/2013/11/09/why-is-it-bad-to-activate-slash-de >>> >> >> >> >> >>>a >>> >> >> >> >> >>>ct >>> >> >> >> >> >>> ive-modules-by-profiles-in-maven/ >>> >> >> >> >> >>> >>> >> >> >> >> >>> >>> >> >> >> >> >>> It seems like that is a bad idea though, based on that >>> >> >> >>discussion. >>> >> >> >> >> >>> >>> >> >> >> >> >>> So, here¹s another option: >>> >> >> >> >> >>> >>> >> >> >> >> >>> 1. Inside of cas-resource (no special new module or >>> >>anything >>> >> >> >>else) >>> >> >> >> >> >>> 2. include some custom Ant magic via a build.xml file >>>and >>> >>the >>> >> >> >>Maven >>> >> >> >> >> >>> AntRun plugin: >>> >> >> >> >> >>> 2a. test if protoc is on the system path, and if not, >>> >> >>download >>> >> >> >>it, >>> >> >> >> >> >>>e.g., >>> >> >> >> >> >>> into the target directory (gets deleted on clean) >>> >> >> >> >> >>> 2b. call protoc and compile after 2a >>> >> >> >> >> >>> >>> >> >> >> >> >>> I would suggest this solution as I think it¹s the most >>> >>robust >>> >> >>and >>> >> >> >> >> >>>ensures >>> >> >> >> >> >>> we always have a cas-resource that includes mesos and >>> >>compiled >>> >> >> >> >> >>>correctly. >>> >> >> >> >> >>> >>> >> >> >> >> >>> Cheers, >>> >> >> >> >> >>> Chris >>> >> >> >> >> >>> >>> >> >> >> >> >>> > >>> >> >> >> >> >>> >Any ideas on how to continue? >>> >> >> >> >> >>> > >>> >> >> >> >> >>> >Michael >>> >> >> >> >> >>> > On Nov 5, 2014 11:04 AM, "Chris Mattmann" >>> >> >> >> >><[email protected]> >>> >> >> >> >> >>> >wrote: >>> >> >> >> >> >>> > >>> >> >> >> >> >>> >> Hi Mike, >>> >> >> >> >> >>> >> >>> >> >> >> >> >>> >> Great discussion. It would be nice if there was >>> >> >> >> >> >>> >> a protoc Maven plugin: >>> >> >> >> >> >>> >> >>> >> >> >> >> >>> >> >>> >> >>http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html >>> >> >> >> >> >>> >> >>> >> >> >> >> >>> >> >>> >> >> >> >> >>> >> Looks like there is. My suggestion: >>> >> >> >> >> >>> >> >>> >> >> >> >> >>> >> 1. use a Profile, something like -Pwith-mesos and >>> >> >> >> >> >>> >> then when activated; >>> >> >> >> >> >>> >> 2. call the above plugin if -Pwith-mesos is >>>activated >>> >> >> >> >> >>> >> in the resource manager >>> >> >> >> >> >>> >> >>> >> >> >> >> >>> >> Sound good? >>> >> >> >> >> >>> >> >>> >> >> >> >> >>> >> Cheers, >>> >> >> >> >> >>> >> Chris >>> >> >> >> >> >>> >> >>> >> >> >> >> >>> >> ------------------------ >>> >> >> >> >> >>> >> Chris Mattmann >>> >> >> >> >> >>> >> [email protected] >>> >> >> >> >> >>> >> >>> >> >> >> >> >>> >> >>> >> >> >> >> >>> >> >>> >> >> >> >> >>> >> >>> >> >> >> >> >>> >> -----Original Message----- >>> >> >> >> >> >>> >> From: Michael Starch <[email protected]> >>> >> >> >> >> >>> >> Reply-To: <[email protected]> >>> >> >> >> >> >>> >> Date: Wednesday, November 5, 2014 at 11:46 AM >>> >> >> >> >> >>> >> To: <[email protected]> >>> >> >> >> >> >>> >> Subject: Extra Compiler Tools >>> >> >> >> >> >>> >> >>> >> >> >> >> >>> >> >All, >>> >> >> >> >> >>> >> > >>> >> >> >> >> >>> >> >I am trying to integrate apache-mesos with our >>> resource >>> >> >> >>manager. >>> >> >> >> >> >>> >>However, >>> >> >> >> >> >>> >> >mesos uses a technology called "protobuff" from >>>Google >>> >>for >>> >> >> >> >> >>> >> >marshaling/unmarshaling data. >>> >> >> >> >> >>> >> > >>> >> >> >> >> >>> >> >This requires running a tool called "protoc" to >>> >>generate a >>> >> >> >> >>source >>> >> >> >> >> >>>file >>> >> >> >> >> >>> >>in >>> >> >> >> >> >>> >> >java. What is the best way to integrate this step >>> into >>> >> >>our >>> >> >> >> >>build >>> >> >> >> >> >>> >>process? >>> >> >> >> >> >>> >> > >>> >> >> >> >> >>> >> >Options I can conceive of: >>> >> >> >> >> >>> >> > -Check in generated java file >>> >> >> >> >> >>> >> > -Require "protoc" installation to build resource >>> >> >>manager >>> >> >> >> >> >>> >> > -Separate extra resource package into new module >>> >> >> >> >> >>> >> > >>> >> >> >> >> >>> >> >None of these ideas are very clean. >>> >> >> >> >> >>> >> > >>> >> >> >> >> >>> >> >Any other ideas? I tried setting up a profile to >>>only >>> >> >> >>compile >>> >> >> >> >> >>>these >>> >> >> >> >> >>> >> >sources when selected, but that turned out not to >>> work. >>> >> >> >> >> >>> >> > >>> >> >> >> >> >>> >> >-Michael Starch >>> >> >> >> >> >>> >> >>> >> >> >> >> >>> >> >>> >> >> >> >> >>> >> >>> >> >> >> >> >>> >>> >> >> >> >> >>> >>> >> >> >> >> >>> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >>> >> >> >>> >> >>> >> >>> >> >>> >>> >>>
