Thanks! I'm having a great time too! :)
Valerie A. Mallder New Horizons Deputy Mission System Engineer Johns Hopkins University/Applied Physics Laboratory > -----Original Message----- > From: Chris Mattmann [mailto:[email protected]] > Sent: Friday, November 21, 2014 5:02 AM > To: [email protected] > Subject: Re: more info for OODT-751 OPSUI Pages constantly expire > > You are awesome Val, great work! > > ------------------------ > Chris Mattmann > [email protected] > > > > > -----Original Message----- > From: "Mallder, Valerie" <[email protected]> > Reply-To: <[email protected]> > Date: Friday, November 21, 2014 at 4:43 PM > To: "[email protected]" <[email protected]> > Subject: RE: more info for OODT-751 OPSUI Pages constantly expire > > >Oh wait, maybe I misunderstood. I didn't see an actual "fix", just more > >discussion that sparked an idea for me to try something which I have > >been trying to "try" for several days now. I was planning to give an > >update of where I am today. I am going on vacation for two weeks > >starting Monday, so I was definitely planning to give y'all an update > >today before I leave. The good news is, I do have all stages of the > >current JEDI pipeline at APL implemented in OODT (With a couple of > >caveats, of course, but ..). I have raw telemetry from JPL's File > >Exchange Interface (FEI) going in, and daily summary plots of the > >telemetry data coming out - and that in itself deserves a huge YAY!!. I > >haven't haven't had to write any project-specific java code yet. The > >only java code change I made was to make the fix identified OODT-757 > >that prevented an autodetect crawler from running. (I think this fix is > >already > planned for 0.8 and may have > >been implemented already in the trunk already.) I have just one > >workflow and 4 PGE tasks, and I glued together several existing 5 java > >applications by rewriting some perl scripts and adding a few new perl > >scripts. I have verified that all 4 of my product types > >(EngineeringFile, ScienceFile, EDRFile, and SummaryPlot) that are > >generated by the pipeline are in the catalog (using the query-tool), > >and I verified that the the summary plots from my pipeline match the > >summary plots that were generated by the existing JEDI pipeline. All I need > >now > is an operator > >interface to demo this to the folks here at APL. I have kept a detailed > >list of lessons learned which I plan to share (probably after I get > >back from vacation though). > > > >.... more later today > > > >Val > > > > > > > >Valerie A. Mallder > >New Horizons Deputy Mission System Engineer Johns Hopkins > >University/Applied Physics Laboratory > > > >> -----Original Message----- > >> From: Mattmann, Chris A (3980) [mailto:[email protected]] > >> Sent: Friday, November 21, 2014 10:13 AM > >> To: Tom Barber; [email protected] > >> Subject: Re: more info for OODT-751 OPSUI Pages constantly expire > >> > >> You did! Did it work? Can you pass along the thread so I can read it > >>sorry I'm trying to catch up.. > >> > >> > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >> +++++ > >> Chris Mattmann, Ph.D. > >> Chief Architect > >> Instrument Software and Science Data Systems Section (398) NASA Jet > >> Propulsion Laboratory Pasadena, CA 91109 USA > >> Office: 168-519, Mailstop: 168-527 > >> Email: [email protected] > >> WWW: http://sunset.usc.edu/~mattmann/ > >> > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >> +++++ > >> Adjunct Associate Professor, Computer Science Department University > >> of Southern California, Los Angeles, CA 90089 USA > >> > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >> +++++ > >> > >> > >> > >> > >> > >> > >> -----Original Message----- > >> From: Tom Barber <[email protected]> > >> Date: Friday, November 21, 2014 at 4:06 PM > >> To: <[email protected]> > >> Subject: Re: more info for OODT-751 OPSUI Pages constantly expire > >> > >> > > >> > > >> > > >> > Oi Chris, I sent Val a fix! :P > >> > > >> > On 21/11/14 14:46, Mattmann, Chris A (3980) wrote: > >> > > >> > > >> > Hi Val, > >> > > >> >I haven't forgot about this - just haven't had a chance to investigate. > >> >Hope to have some time in the next week. > >> > > >> >Cheers, > >> >Chris > >> > > >> > >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >> ++++++ > >> >Chris Mattmann, Ph.D. > >> >Chief Architect > >> >Instrument Software and Science Data Systems Section (398) NASA Jet > >> >Propulsion Laboratory Pasadena, CA 91109 USA > >> >Office: 168-519, Mailstop: 168-527 > >> >Email: [email protected] > >> >WWW: http://sunset.usc.edu/~mattmann/ > >> > >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >> ++++++ > >> >Adjunct Associate Professor, Computer Science Department University > >> >of Southern California, Los Angeles, CA 90089 USA > >> > >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >> ++++++ > >> > > >> > > >> > > >> > > >> > > >> > > >> >-----Original Message----- > >> >From: <Mallder>, Valerie <[email protected]> > >> ><mailto:[email protected]> > >> >Reply-To: "[email protected]" <mailto:[email protected]> > >> ><[email protected]> <mailto:[email protected]> > >> >Date: Thursday, November 13, 2014 at 2:10 PM > >> >To: "[email protected]" <mailto:[email protected]> > >> ><[email protected]> <mailto:[email protected]> > >> >Subject: RE: more info for OODT-751 OPSUI Pages constantly expire > >> > > >> > > >> > > >> > For those of you who are using the pcs-opsui in an > >> >operational environment without getting expired pages please tell me > >> >what version you are using, the OS you are running it on, and the > >> >browser you are > >>using. > >> > > >> >I need to demonstrate "something" to my project manager to show the > >> >usefulness of OODT in the Jedi instrument science data pipeline here > >> >at APL. And right now, I have nothing to "show" for my last few > >> >months of work. If someone can tell me a configuration using opsui > >> >that is working then maybe I can try to mimic that and get something > >> >useful up and running. So far, for me, versions 0.6, 0.7 and the > >> >current trunk are showing only expired pages. > >> > > >> >Thanks very much! > >> >Val > >> > > >> > > >> > > >> >Sent from my iPhone. > >> >________________________________ > >> >From: Mallder, Valerie <[email protected]> > >> ><mailto:[email protected]> > >> >Sent: Tuesday, November 11, 2014 2:55:12 PM > >> >To: [email protected] > >> >Subject: more info for OODT-751 OPSUI Pages constantly expire > >> > > >> >Hi Chris, > >> > > >> >I know you are working on this, but I wanted to let you know that I > >> >tried to use version 0.6 and am having the same problem in version 0.6. > >> >But, I can't guarantee that I installed version 0.6 correctly so I > >> >would like to run this by you. Based on your email below here's > >> >what > >>I did: > >> > > >> >1. Downloaded pcs-opsui-0.76.war from: > >> >http://repo1.maven.org/maven2/org/apache/oodt/pcs-opsui/0.6/ > >> >and saved it in a new folder named $OODT_HOME/bin/opsui > >> > > >> >2. Created a script called $OODT_HOME/bin/opsui/runopsui that sets > >> >the following variables (this is output from my script): > >> >Using OODT_BASE: > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy > >> >Using OODT_HOME: > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy > >> >Using OODT_TMPDIR: > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/temp > >> >Using FILEMGR_URL: http://localhost:9000 > >> >Using WORKFLOW_URL: http://localhost:9001 > >> >Using RESMGR_URL: http://localhost:9002 > >> >Using WORKFLOW_HOME: > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/workflo > >> >w > >> >Using RESMGR_HOME: > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/resmgr > >> >Using CRAWLER_HOME: > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/crawler > >> >Using TOMCAT_HOME: > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat > >> >Using PCS_HOME: > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pcs > >> >Using PGE_HOME: > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge > >> >Using PGE_JOBS_DIR: > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/pg > >> >e/j > >> >obs > >> >Using FEI_DROP_DIR: > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/te > >> >lem > >> >etr > >> >y > >> >Using JEDI_L0_DIR: > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/l0 > >> >Using JEDI_L2_DIR: > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/l2 > >> >Using ARCHIVE_DIR: > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/ar > >> >chi > >> >ve > >> >Using BACKUP_DIR: > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/me > >> >t > >> >Using FAILURE_DIR: > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/fa > >> >ilu re Using JEDI_PIPELINE_DIR: /homes/malldva1/working/pipeline > >> >Using > >> >SNAPSHOT_DIR: > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/pg > >> >e/j > >> >obs > >> >/snapshot > >> > > >> >(Then, I followed the steps at the bottom of: > >> >https://cwiki.apache.org/confluence/display/OODT/Quick+Start+for+PCS > >> >+OP > >> >SUI > >> >) > >> > > >> >3. Downloaded > >> >http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/webapp > >> >/ME > >> >TA- > >> >INF/context.xml > >> >And saved it as $OODT_HOME/bin/opsui/pcs-opsui.xml > >> > > >> >4. Edited $OODT_HOME/bin/opsui/pcs-opsui.xml and changed the first > >> >line > >> >from: > >> ><Context path="/pcs-opsui"> > >> >To > >> ><Context path="/pcs-opsui" > >> >docBase="[OODT_HOME]/bin/opsui/pcs-opsui-0.6.war"> > >> > > >> >5. Killed any process with "tomcat" in its name to ensure that > >> >tomcat is not running. > >> > > >> >6. Executed the following command to create a symbolic link: > >> >cd $OODT_HOME/bin/opsui/ > >> >ln -s pcs-opsui.xml to > >> >$TOMCAT_HOME/conf/Catalina/localhost/pcs-opsui.xml > >> > > >> >7. Then, in my script that sets the environment variables, I added > >> >the following command to start tomcat. > >> >exec "$OODT_BASE"/tomcat/bin/catalina.sh start > >> > > >> >8. Then, I ran my new $OODT_HOME/bin/opsui/runopsui script > >> > > >> >9. Then, I started firefox and went to "localhost:8080/pcs-opsui" > >> > > >> > > >> > > >> >The home page comes up, and the PCS Status page comes up, but all > >> >other pages are expired. I should note that the "radix" > >> >installation of version 0.7 points me to localhost:8080/opsui, while > >> >step 10 of the instructions from the quick start guide of the wiki > >> >point me to localhost:8080/pcs-opsui. And, no matter which one I go > >> >to, the pages expire. > >> > > >> >I also tried clearing out my cache and browser history and > >> >restarting firefox and topcat, etc. and nothing seems to help. > >> > > >> >If I have gotten the version 0.6 up and running correctly, I thought > >> >you would be interested to know that I am seeing the error. If I did > >> >not get version 0.6 up and running, then I need more hints on how to > >> >do it. :) > >> > > >> >Thanks, > >> >Val > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> >Valerie A. Mallder > >> >New Horizons Deputy Mission System Engineer Johns Hopkins > >> >University/Applied Physics Laboratory > >> > > >> > > >> > > >> > > >> > -----Original Message----- > >> >From: Mattmann, Chris A (3980) > >> >[mailto:[email protected]] > >> >Sent: Friday, October 03, 2014 1:07 PM > >> >To: [email protected] > >> >Subject: Re: Success! RE: how to use MetadataBasedFileVersioner > >> >properly > >> > > >> >Awesome Val! :) > >> > > >> >I think you¹re running into this: > >> > > >> >https://issues.apache.org/jira/browse/OODT-751 > >> > > >> > > >> >In the meanwhile, try the 0.6 OPSUI, which you can grab from here: > >> > > >> >http://repo1.maven.org/maven2/org/apache/oodt/pcs-opsui/0.6/ > >> > > >> > > >> >Grab the WAR file and drop it into your favorite container. > >> >Make sure you have all of these environment variables installed > >> >*before* starting Tomcat or Jetty, etc.: > >> > > >> > > >> >https://cwiki.apache.org/confluence/display/OODT/Quick+Start+for+PCS > >> >+OP > >> >SU > >> >I > >> > > >> > > >> >(see steps at bottom and replace 0.5 with 0.6) > >> > > >> >I¹m working on a fix for OODT-751, at which point RADIX will be > >> >pretty buff. Next steps at that point: > >> > > >> >1. Release 0.7 and then encourage folks to get started by using the > >> >Vagrant build, e.g., > >> > > >> >git clone https://github.com/apache/oodt cd vagrant/radix vagrant up > >> > > >> >2. Fix OODT-491 and remaining workflow manager issues for Wengine 3. > >> >Work > >> >on Streaming OODT API with AMP Stack (via M. Starch et al) and > >> >release in 0.8. > >> >4. Conquer and win. > >> > > >> >Thanks! > >> > > >> >Cheers, > >> >Chris > >> > > >> > >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >> + > >> >+++++ > >> >Chris Mattmann, Ph.D. > >> >Chief Architect > >> >Instrument Software and Science Data Systems Section (398) NASA Jet > >> >Propulsion Laboratory Pasadena, CA 91109 USA > >> >Office: 168-519, Mailstop: 168-527 > >> >Email: [email protected] > >> >WWW: http://sunset.usc.edu/~mattmann/ > >> > >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >> + > >> >+++++ > >> >Adjunct Associate Professor, Computer Science Department University > >> >of Southern California, Los Angeles, CA 90089 USA > >> > >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >> + > >> >+++++ > >> > > >> > > >> > > >> > > >> > > >> > > >> >-----Original Message----- > >> >From: <Mallder>, Valerie <[email protected]> > >> ><mailto:[email protected]> > >> >Reply-To: "[email protected]" <mailto:[email protected]> > >> ><[email protected]> <mailto:[email protected]> > >> >Date: Friday, October 3, 2014 at 9:53 AM > >> >To: "[email protected]" <mailto:[email protected]> > >> ><[email protected]> <mailto:[email protected]> > >> >Subject: Success! RE: how to use MetadataBasedFileVersioner > >> >properly > >> > > >> > > >> > > >> > Hi Chris, > >> > > >> >Yes, that indeed fixed it! Thanks so much! I now have 18 > >> >engineering files ingested. Whoo hoo!! > >> > > >> >Ok, so now, I would like to see what the opsui has to say about my > >> >ingested files. And here comes probably a really stupid question. I > >> >am using Firefox on a Redhat Linux box, and I'm not that familiar > >> >with Firefox and it's settings, so this might be a browser setting issue. > >> >When I startup the opsui and select "File Catalog Browse" it shows > >> >that I have > >> >18 EngineeringFiles. Then, when I select File Catalog Browse from > >> >the strip of options under the logo, I get a message saying the page > >> >has expired. And here's the link that shows up in the address bar: > >> >http://localhost:8080/opsui/?wicket:interface=:7:fmbrowser_link::ILi > >> >nkL > >> >ist > >> >ener:: > >> > > >> >And I get the page expired message for all of the options that I > >> > > >> > > >> > > >> > select. > >> > > >> > > >> > Any idea's on this one?? > >> > > >> >In the meantime though, I will start playing with an action for post > >> >ingestion success that simply makes another copy of all these > >> >engineering files and puts them another folder for Level 0 files. > >> > > >> >Thanks, > >> >Val > >> > > >> > > >> > > >> > > >> >Valerie A. Mallder > >> >New Horizons Deputy Mission System Engineer Johns Hopkins > >> >University/Applied Physics Laboratory > >> > > >> > > >> > > >> > > >> > -----Original Message----- > >> >From: Chris Mattmann [mailto:[email protected]] > >> >Sent: Friday, October 03, 2014 3:03 AM > >> >To: [email protected] > >> >Subject: Re: how to use MetadataBasedFileVersioner properly > >> > > >> >You?re almost there Val! > >> > > >> >Unfortunately Versioners right now aren?t configurable from product > >> >type policy (would be great to capture this in a JIRA issue, here: > >> >https://issues.apache.org/jira/browse/OODT). If they were, it would > >> >have picked up your <property .. declaration of filePathSpec below. > >> >It?s been on my TODO list for a long time. > >> > > >> >Instead I created this: > >> > > >> >https://issues.apache.org/jira/browse/OODT-639 > >> > > >> > > >> >So you can amend your definition below (also note you always have to > >> >include /[Filename] at the end to get the filename you want). > >> > > >> > <type id="urn:oodt:EngineeringFile" name="EngineeringFile"> > >> > <repository path="file://[OODT_HOME]/data/archive/ops/eng" > >> ><file://[OODT_HOME]/data/archive/ops/eng>/> > >> > <versioner > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> >class=?org.apache.oodt.cas.filemgr.versioning.ProductTypeMetVersione > >> >r?/ > >> >> > >> > > >> > > >> > > >> > <description>The default product type for any kind > >> >of file.</description> > >> > <metExtractors> > >> > <extractor > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> >class="org.apache.oodt.cas.filemgr.metadata.extractors.CoreMetExtrac > >> >tor > >> >"> > >> > <configuration> > >> > <!-- you can optionally include the envReplace tag to turn > >> >on/off environment var replacement --> > >> > <property name="nsAware" value="true" /> > >> > <property name="elementNs" value="CAS" /> > >> > <property name="elements" > >> >value="ProductReceivedTime,ProductName,ProductId" /> > >> > </configuration> > >> > </extractor> > >> > </metExtractors> > >> > <metadata> > >> > <keyval> > >> > <key>filePathSpec</key> > >> > <val>/[YearDir]/[DoyDir]/[Filename]</val> > >> > </keyval> > >> > > >> ></metadata> > >> > </type> > >> > > >> > > >> >See if that fixes it! > >> > > >> >Cheers, > >> >Chris > >> > > >> >------------------------ > >> >Chris Mattmann > >> >[email protected] > >> > > >> > > >> > > >> > > >> >-----Original Message----- > >> >From: "Mallder, Valerie" <[email protected]> > >> ><mailto:[email protected]> > >> >Reply-To: <[email protected]> <mailto:[email protected]> > >> >Date: Thursday, October 2, 2014 at 2:56 PM > >> >To: "[email protected]" <mailto:[email protected]> > >> ><[email protected]> <mailto:[email protected]> > >> >Subject: how to use MetadataBasedFileVersioner properly > >> > > >> > > >> > > >> > Hi All, > >> > > >> >I am trying to use the MetadataBasedFileVersioner to store my files > >> >in a subdirectory of the data/archive folder based on the values of > >> >two metadata elements. Has anyone does this before, and if so, can > >> >you give me some hints. > >> > > >> >My files need to be organized in subdirectories by year and day of > >> > > >> > > >> > > >> > year. > >> > > >> > > >> > So, lets say the file name is 'myfile', the year is > >> >2014, and the doy of year is 002. Then, the end result that I am > >> >looking for is to have the final location of my file be: > >> > > >> > > >> > > >> > 'data/archive/ops/eng/2014/002/myfile. > >> > > >> > > >> > However, in my log file, the INFO messages indicate > >> >that the generated final location reference is > >> >'data/archive/ops/eng/myfile'. > >> >And it doesn't include the year and day of year at all. And the > >> >incorrect location leads to other errors. So I want to solve this > >> >one first. Has anyone tried to do something like this before? > >> > > >> >Here is what I have done so far: > >> > > >> >Added two new elements to the .met file 'YearDir' and 'DoyDir'. > >> >Added these new elements to the elements.xml file. > >> >Added these new elements to the product in the > >> >product-type-element-map.xml file. > >> >And, I have made changes to the product-types.xml file, but here's > >> >where I am not sure I've done this properly. > >> > > >> >Here's what product type definition looks like: > >> > <type id="urn:oodt:EngineeringFile" name="EngineeringFile"> > >> > <repository path="file://[OODT_HOME]/data/archive/ops/eng" > >> ><file://[OODT_HOME]/data/archive/ops/eng>/> > >> > <versioner > >> > > >> > > >> > > >> > > >> > > >> > > >> >class="org.apache.oodt.cas.filemgr.versioning.MetadataBasedFileVersi > >> >o > >> >ner > >> >"> > >> > <property name="filePathSpec" > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > value="/[YearDir]/[DoyDir]/" /> > >> > > >> > > >> > > >> > > >> > </versioner> > >> > <description>The default product type for any kind of > >> >file.</description> > >> > <metExtractors> > >> > <extractor > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> >class="org.apache.oodt.cas.filemgr.metadata.extractors.CoreMetExtrac > >> >to > >> >r" > >> > > >> > > >> > > >> > > >> > > >> > <configuration> > >> > <!-- you can optionally include the envReplace tag to turn > >> >on/off environment var replacement --> > >> > <property name="nsAware" value="true" /> > >> > <property name="elementNs" value="CAS" /> > >> > <property name="elements" > >> >value="ProductReceivedTime,ProductName,ProductId" /> > >> > </configuration> > >> > </extractor> > >> > </metExtractors> > >> > <metadata/> > >> > </type> > >> > > >> >Thanks in advance for any help or ideas you might have! > >> > > >> >Valerie > >> > > >> > > >> > > >> >Valerie A. Mallder > >> > > >> >New Horizons Deputy Mission System Engineer The Johns Hopkins > >> >University/Applied Physics Laboratory > >> >11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723 > >> >240-228-7846 (Office) 410-504-2233 (Blackberry) > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > -- > >> > Tom Barber | Technical Director > >> > > >> > meteorite bi > >> > T: +44 20 8133 3730 > >> > W: www.meteorite.bi <http://www.meteorite.bi> | Skype: > >> >meteorite.consulting > >> > A: Surrey Technology Centre, Surrey Research Park, > >> > Guildford, GU2 7YG, UK > >> > > >> > > > >
