Hey Guys, This made me think of the following issue I filed:
https://issues.apache.org/jira/browse/OODT-147 I think we can deal with a lot of this there. What do you think of my proposal, Michael? Brian, I know what you think :-) Because you and I talked about it on the issue. Cheers, Chris On May 24, 2011, at 7:08 AM, holenoter wrote: > > hey michael, > > if you just need to store this information in the elements.xml file then just > add new tags . . . you don't need to change any code: > > <element id="some-id" name="some-name"> > (current elements here) > .... > <!-- unparsed tags --> > <maxSize>10</maxSize> > </element> > > Also . . . the dublin core stuff is fine for peate . . . if you want to > create a patch for the trunk at apache that is a different story, but peate > will never need the dublin core feature, so the patch as it currently stands > is perfectly okay > > -brian > > On May 24, 2011, at 07:18 AM, "Starch, Michael D (388L)" > <[email protected]> wrote: > >> Brian, >> >> PEATE is busy as always, but doing well. >> >> We need to be able to store this information somewhere so we can just tell >> the DBA to use these values. As it stands now we do a lot of back-and-forth >> to get the right values to support the database, and we need to stream-line >> the process by having this information pre-selected by the project. Due to >> the data's similarity to the other data in the elements file we selected >> this as the place to put it and Chris agreed. >> >> Does this answer your questions? >> >> -Michael >> >> >> >> >> >> >> >> On May 24, 2011, at 1:47 AM, holenoter wrote: >> >>> hey michael, >>> >>> hope everything is going well for peate . . . >>> >>> yup dublin core element was used because of the time crunch before i left . >>> . . being it hasn't been used up to this point anywhere, good odds it >>> probably won't be used anytime in the "peate future" . . . however, dublin >>> core is the wrong place to put this feature because dublin core is used to >>> describe the element itself, not its value's type . . . pretty much chris's >>> recommendation is the way to go, however if you are going to be working >>> towards a patch to the trunk, instead of extending Element as chris >>> recommended just add a "type" member variable to the existing Element class >>> and then modify the XMLValidationLayer and have it read a <type> tag into >>> that "type" member variable in Element . . . i think there is a strong case >>> for having the Element understand to some degree its value's type . . . not >>> sure why you need more information in your elements.xml file than i already >>> added? . . . the database should be doing this validation (otherwise your >>> code will have to become double, float, long, int, etc... aware) . . . the >>> current ColumnBasedDataSourceCatalog handles all these types already via >>> Oracle (just set any double, float, long, int, etc... to a >>> <dcElement>NUMBER</dcElement> in the elements.xml file. . .you have to set >>> Oracle double precision anyway even if you where to treat it as a double in >>> the filemgr) . . . if you want to just keep the information in one spot >>> (i.e. elements.xml) than just add unused tags to the elements.xml file with >>> the information you want (XMLValidationLayer will just ignore these tags) . >>> . . for example you can add a <maxSize> tag to elements.xml without >>> affecting the filemgr. . .as far as validating max size and the like, this >>> really should be happening in your metadata extractors. >>> >>> -brian >>> >>> On May 24, 2011, at 12:59 AM, "Mattmann, Chris A (388J)" >>> <[email protected]> wrote: >>> >>>> Hi Michael, >>>> >>>> Thanks for your email! >>>> >>>> You can find the OODT Apache archives here: >>>> >>>> http://mail-archives.apache.org/mod_mbox/oodt-dev/ >>>> >>>> As for standards, best practices are certainly part of them but we do have >>>> a few standards that we try and stick to including Dublin Core, ISO-11179, >>>> RDF, etc., when we are defining metadata elements. For your purposes in >>>> elements.xml in CAS, the biggest approach is to use unique, namespaced >>>> identifiers for element IDs, product type IDs, etc. >>>> >>>> HTH! >>>> >>>> Cheers, >>>> Chris >>>> >>>> On May 23, 2011, at 1:14 PM, Starch, Michael D (388L) wrote: >>>> >>>> > Chris, >>>> > >>>> > Is there some sort of coding standards document that I should be abiding >>>> > by, or does oodt just follow standard java "best practices"? This is the >>>> > kind of information that seems like it would be easier for me to look up >>>> > rather than ask on the developer list. >>>> > >>>> > Finally, how can I access the archived dev-list mailings? >>>> > >>>> > Thanks, >>>> > >>>> > Michael >>>> > >>>> > >>>> > On May 19, 2011, at 11:06 PM, Mattmann, Chris A (388J) wrote: >>>> > >>>> >> Hi Michael, >>>> >> >>>> >> I'd be happy to reply below, but yes I'd suggest initially CC'ing >>>> >> [email protected]. Giving this information is great, but I prefer the >>>> >> community to benefit from these types of discussions (and it means that >>>> >> I or you or Albert or Cecilia can point others to this question if it >>>> >> is asked again rather than typing the same answer again or fwd'ing >>>> >> something private). Also no worries about researching it -- you can ask >>>> >> questions on that list -- researched or not. We welcome feedback from >>>> >> the community. >>>> >> >>>> >> Answers inline below: >>>> >> >>>> >> On May 19, 2011, at 2:44 PM, Starch, Michael D (388L) wrote: >>>> >> >>>> >>> Chris, >>>> >>> >>>> >>> In order to create tables in the new database architecture that Brian >>>> >>> Foster set up for use, we need the ability to specify some information >>>> >>> beyond that which is stored in elements.xml. We wish to store this >>>> >>> information as part of PCS, rather than continually haggling back and >>>> >>> forth with our DBA in order to get this set in an ad-hoc manner. >>>> >>> >>>> >>> The data we need to store is: >>>> >>> >>>> >>> -Is it a vector type? >>>> >>> -Data-type >>>> >>> -Max data size >>>> >>> and potentially a few more items. >>>> >> >>>> >> Gotcha. >>>> >> >>>> >>> >>>> >>> As it stands now, Brian has used the DCElement tag in elements.xml >>>> >>> file in order to store some of this information but not all of it. >>>> >>> Thus it seems natural that the rest of this information gets stored >>>> >>> along side it. >>>> >>> >>>> >>> There seem to be three ways to accomplish this: >>>> >>> >>>> >>> 1. Add more information to the DCElement tag, >>>> >>> 2. Add additional tags to the element file containing this information >>>> >>> 3. Create a sperate file to store this information (Not part of PCS) >>>> >> >>>> >> I've got a 4th option. >>>> >> >>>> >> 4. Write a new ValidationLayer implementation, that extends >>>> >> XMLValidationLayer, but adds your desired information to elements.xml. >>>> >> For that extra information, I'd favor your proposed option #2 -- I >>>> >> think it's cleaner. >>>> >> >>>> >>> >>>> >>> As head of oodt, we were looking for you input as to which option to >>>> >>> choose, or if you wish this question to be brought to [email protected], >>>> >>> where can I research this information before asking the question on >>>> >>> that list? >>>> >>> >>>> >>> Below is a brief analysis of each option, from our perspective. >>>> >>> >>>> >>> Thanks, >>>> >>> >>>> >>> Michael Starch >>>> >>> >>>> >>> 1. Add more information to the DCElements tag. >>>> >>> -I believe Brain said this tag was unused, so he "borrowed" it to suit >>>> >>> this purpose. >>>> >>> -Adding more information to the tag would "hijack" this tag to a >>>> >>> further extent. >>>> >>> -The changes could be made locally to our ColumnBasedDataSourceCatalog >>>> >>> in the manner Brian already used. >>>> >>> -If the DCElement is used ouside of the new use Brian invented, this >>>> >>> change could affect the intended (original) use >>>> >> >>>> >> I don't like this one (nor do I think others will) since it hijacks >>>> >> dcElement (which is supposed to be a mapping to a Dublin Core element >>>> >> name, rather than used for other information). >>>> >> >>>> >>> >>>> >>> 2. Add more tags to elements.xml >>>> >>> -Most elegant solution, as each data field has a tag specified to hold >>>> >>> exactly it >>>> >>> -No need to use tags which were originally intended for some other >>>> >>> purpose >>>> >>> -It is an architectural change, changing the format of the >>>> >>> elements.xml file >>>> >>> -Must be done carefully to prevent us from deviating from apache-oodt >>>> >>> when it is not necessary to do so. >>>> >> >>>> >> +1 for this option. >>>> >> >>>> >>> >>>> >>> 3. A separate file >>>> >>> -Creates data/configuration duplication, as some of these fields must >>>> >>> stay (in some form) in elements.xml >>>> >>> -Separates similar pieces of data >>>> >>> -Doesn't alter PCS as it exists now >>>> >> >>>> >> This is fine too, but I prefer #2. >>>> >> >>>> >>> >>>> >>> We favor number 1 or number 2 as it keeps similar data together, and >>>> >>> feels like it is the most logical place to store this information. >>>> >>> >>>> >>> I personally favor number 2, as it does not use tags for unintended >>>> >>> purposes, but I do not wish to make a change that affects our >>>> >>> compatibility with apache code. Hence your input. >>>> >>> >>>> >> >>>> >> +1 to your preference. I think the way to accomplish it is to write a >>>> >> YourExtendedValidationLayer extends XMLValidationLayer and that >>>> >> provides a customized: >>>> >> 1. YourElementClass extends Element >>>> >> - provides extra props (getters+setters) that you want to leverage >>>> >> 2. implementations of the SerDe for your specialized elements.xml that >>>> >> includes those extra tags >>>> >> 3. provides the ability to access this ValidationLayer information and >>>> >> Element information in ColumnBasedDataSourceCatalog. >>>> >> >>>> >> BTW, I think it would be great to bake up patches for Apache OODT for >>>> >> what you are working on, including ColumnBasedDataSourceCatalog >>>> >> (already there but in a branch, would be nice to forward port to trunk, >>>> >> and your proposed ValidationLayer and Element extensions). Thoughts? Do >>>> >> you have the time? I think the community at Apache would sincerely >>>> >> appreciate the effort. >>>> >> >>>> >> Cheers, >>>> >> Chris >>>> >> >>>> >> P.S. Would you be OK with me forwarding this thread to [email protected]? I >>>> >> think there's some good info here that I'd hate to be lost in the >>>> >> ether... >>>> >> >>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>> >> Chris Mattmann, Ph.D. >>>> >> Senior Computer Scientist >>>> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>> >> Office: 171-266B, Mailstop: 171-246 >>>> >> Email: [email protected] >>>> >> WWW: http://sunset.usc.edu/~mattmann/ >>>> >> Phone: +1 (818) 354-8810 >>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>> >> Adjunct Assistant Professor, Computer Science Department >>>> >> University of Southern California, Los Angeles, CA 90089 USA >>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>> >> >>>> > >>>> >>>> >>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>> Chris Mattmann, Ph.D. >>>> Senior Computer Scientist >>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>> Office: 171-266B, Mailstop: 171-246 >>>> Email: [email protected] >>>> WWW: http://sunset.usc.edu/~mattmann/ >>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>> Adjunct Assistant Professor, Computer Science Department >>>> University of Southern California, Los Angeles, CA 90089 USA >>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: [email protected] WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
