Ok, I lied. I just remembered that today is the last day I can get my drivers license renewed so I have to leave work early to take care of that and so I won't be have time to write down more details of my progress. But, when I get back from vacation on Dec 8, I will probably be bugging folks again on what I need to do in order to get the operator interface up and running.
Thanks everyone for all of your help! And I hope you have a great Thanksgiving next week! Val Valerie A. Mallder New Horizons Deputy Mission System Engineer Johns Hopkins University/Applied Physics Laboratory > -----Original Message----- > From: Mallder, Valerie [mailto:[email protected]] > Sent: Friday, November 21, 2014 10:43 AM > To: [email protected] > Subject: RE: more info for OODT-751 OPSUI Pages constantly expire > > Oh wait, maybe I misunderstood. I didn't see an actual "fix", just more > discussion > that sparked an idea for me to try something which I have been trying to > "try" for > several days now. I was planning to give an update of where I am today. I am > going on vacation for two weeks starting Monday, so I was definitely planning > to > give y'all an update today before I leave. The good news is, I do have all > stages > of the current JEDI pipeline at APL implemented in OODT (With a couple of > caveats, of course, but ..). I have raw telemetry from JPL's File Exchange > Interface > (FEI) going in, and daily summary plots of the telemetry data coming out - > and that > in itself deserves a huge YAY!!. I haven't haven't had to write any > project-specific > java code yet. The only java code change I made was to make the fix identified > OODT-757 that prevented an autodetect crawler from running. (I think this fix > is > already planned for 0.8 and may have been implemented already in the trunk > already.) I have just one workflow and 4 PGE tasks, and I glued together > several > existing 5 java applications by rewriting some perl scripts and adding a few > new > perl scripts. I have verified that all 4 of my product types (EngineeringFile, > ScienceFile, EDRFile, and SummaryPlot) that are generated by the pipeline are > in > the catalog (using the query-tool), and I verified that the the summary plots > from > my pipeline match the summary plots that were generated by the existing JEDI > pipeline. All I need now is an operator interface to demo this to the folks > here at > APL. I have kept a detailed list of lessons learned which I plan to share > (probably > after I get back from vacation though). > > .... more later today > > Val > > > > Valerie A. Mallder > New Horizons Deputy Mission System Engineer Johns Hopkins University/Applied > Physics Laboratory > > > -----Original Message----- > > From: Mattmann, Chris A (3980) [mailto:[email protected]] > > Sent: Friday, November 21, 2014 10:13 AM > > To: Tom Barber; [email protected] > > Subject: Re: more info for OODT-751 OPSUI Pages constantly expire > > > > You did! Did it work? Can you pass along the thread so I can read it > > sorry I'm trying to catch up.. > > > > > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > +++++ > > Chris Mattmann, Ph.D. > > Chief Architect > > Instrument Software and Science Data Systems Section (398) NASA Jet > > Propulsion Laboratory Pasadena, CA 91109 USA > > Office: 168-519, Mailstop: 168-527 > > Email: [email protected] > > WWW: http://sunset.usc.edu/~mattmann/ > > > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > +++++ > > Adjunct Associate Professor, Computer Science Department University of > > Southern California, Los Angeles, CA 90089 USA > > > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > +++++ > > > > > > > > > > > > > > -----Original Message----- > > From: Tom Barber <[email protected]> > > Date: Friday, November 21, 2014 at 4:06 PM > > To: <[email protected]> > > Subject: Re: more info for OODT-751 OPSUI Pages constantly expire > > > > > > > > > > > > > > Oi Chris, I sent Val a fix! :P > > > > > > On 21/11/14 14:46, Mattmann, Chris A (3980) wrote: > > > > > > > > > Hi Val, > > > > > >I haven't forgot about this - just haven't had a chance to investigate. > > >Hope to have some time in the next week. > > > > > >Cheers, > > >Chris > > > > > > >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > ++++++ > > >Chris Mattmann, Ph.D. > > >Chief Architect > > >Instrument Software and Science Data Systems Section (398) NASA Jet > > >Propulsion Laboratory Pasadena, CA 91109 USA > > >Office: 168-519, Mailstop: 168-527 > > >Email: [email protected] > > >WWW: http://sunset.usc.edu/~mattmann/ > > > >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > ++++++ > > >Adjunct Associate Professor, Computer Science Department University > > >of Southern California, Los Angeles, CA 90089 USA > > > >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > ++++++ > > > > > > > > > > > > > > > > > > > > >-----Original Message----- > > >From: <Mallder>, Valerie <[email protected]> > > ><mailto:[email protected]> > > >Reply-To: "[email protected]" <mailto:[email protected]> > > ><[email protected]> <mailto:[email protected]> > > >Date: Thursday, November 13, 2014 at 2:10 PM > > >To: "[email protected]" <mailto:[email protected]> > > ><[email protected]> <mailto:[email protected]> > > >Subject: RE: more info for OODT-751 OPSUI Pages constantly expire > > > > > > > > > > > > For those of you who are using the pcs-opsui in an > > >operational environment without getting expired pages please tell me > > >what version you are using, the OS you are running it on, and the browser > > >you > are using. > > > > > >I need to demonstrate "something" to my project manager to show the > > >usefulness of OODT in the Jedi instrument science data pipeline here > > >at APL. And right now, I have nothing to "show" for my last few > > >months of work. If someone can tell me a configuration using opsui > > >that is working then maybe I can try to mimic that and get something > > >useful up and running. So far, for me, versions 0.6, 0.7 and the > > >current trunk are showing only expired pages. > > > > > >Thanks very much! > > >Val > > > > > > > > > > > >Sent from my iPhone. > > >________________________________ > > >From: Mallder, Valerie <[email protected]> > > ><mailto:[email protected]> > > >Sent: Tuesday, November 11, 2014 2:55:12 PM > > >To: [email protected] > > >Subject: more info for OODT-751 OPSUI Pages constantly expire > > > > > >Hi Chris, > > > > > >I know you are working on this, but I wanted to let you know that I > > >tried to use version 0.6 and am having the same problem in version 0.6. > > >But, I can't guarantee that I installed version 0.6 correctly so I > > >would like to run this by you. Based on your email below here's what I > > >did: > > > > > >1. Downloaded pcs-opsui-0.76.war from: > > >http://repo1.maven.org/maven2/org/apache/oodt/pcs-opsui/0.6/ > > >and saved it in a new folder named $OODT_HOME/bin/opsui > > > > > >2. Created a script called $OODT_HOME/bin/opsui/runopsui that sets > > >the following variables (this is output from my script): > > >Using OODT_BASE: > > >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy > > >Using OODT_HOME: > > >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy > > >Using OODT_TMPDIR: > > >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/temp > > >Using FILEMGR_URL: http://localhost:9000 > > >Using WORKFLOW_URL: http://localhost:9001 > > >Using RESMGR_URL: http://localhost:9002 > > >Using WORKFLOW_HOME: > > >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/workflow > > >Using RESMGR_HOME: > > >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/resmgr > > >Using CRAWLER_HOME: > > >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/crawler > > >Using TOMCAT_HOME: > > >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat > > >Using PCS_HOME: > > >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pcs > > >Using PGE_HOME: > > >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge > > >Using PGE_JOBS_DIR: > > >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/pge > > >/j > > >obs > > >Using FEI_DROP_DIR: > > >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/tel > > >em > > >etr > > >y > > >Using JEDI_L0_DIR: > > >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/l0 > > >Using JEDI_L2_DIR: > > >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/l2 > > >Using ARCHIVE_DIR: > > >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/arc > > >hi > > >ve > > >Using BACKUP_DIR: > > >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/met > > >Using FAILURE_DIR: > > >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/fai > > >lu re Using JEDI_PIPELINE_DIR: /homes/malldva1/working/pipeline Using > > >SNAPSHOT_DIR: > > >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/pge > > >/j > > >obs > > >/snapshot > > > > > >(Then, I followed the steps at the bottom of: > > >https://cwiki.apache.org/confluence/display/OODT/Quick+Start+for+PCS+ > > >OP > > >SUI > > >) > > > > > >3. Downloaded > > >http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/webapp/ > > >ME > > >TA- > > >INF/context.xml > > >And saved it as $OODT_HOME/bin/opsui/pcs-opsui.xml > > > > > >4. Edited $OODT_HOME/bin/opsui/pcs-opsui.xml and changed the first > > >line > > >from: > > ><Context path="/pcs-opsui"> > > >To > > ><Context path="/pcs-opsui" > > >docBase="[OODT_HOME]/bin/opsui/pcs-opsui-0.6.war"> > > > > > >5. Killed any process with "tomcat" in its name to ensure that > > >tomcat is not running. > > > > > >6. Executed the following command to create a symbolic link: > > >cd $OODT_HOME/bin/opsui/ > > >ln -s pcs-opsui.xml to > > >$TOMCAT_HOME/conf/Catalina/localhost/pcs-opsui.xml > > > > > >7. Then, in my script that sets the environment variables, I added > > >the following command to start tomcat. > > >exec "$OODT_BASE"/tomcat/bin/catalina.sh start > > > > > >8. Then, I ran my new $OODT_HOME/bin/opsui/runopsui script > > > > > >9. Then, I started firefox and went to "localhost:8080/pcs-opsui" > > > > > > > > > > > >The home page comes up, and the PCS Status page comes up, but all > > >other pages are expired. I should note that the "radix" installation > > >of version 0.7 points me to localhost:8080/opsui, while step 10 of > > >the instructions from the quick start guide of the wiki point me to > > >localhost:8080/pcs-opsui. And, no matter which one I go to, the pages > > >expire. > > > > > >I also tried clearing out my cache and browser history and restarting > > >firefox and topcat, etc. and nothing seems to help. > > > > > >If I have gotten the version 0.6 up and running correctly, I thought > > >you would be interested to know that I am seeing the error. If I did > > >not get version 0.6 up and running, then I need more hints on how to > > >do it. :) > > > > > >Thanks, > > >Val > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >Valerie A. Mallder > > >New Horizons Deputy Mission System Engineer Johns Hopkins > > >University/Applied Physics Laboratory > > > > > > > > > > > > > > > -----Original Message----- > > >From: Mattmann, Chris A (3980) [mailto:[email protected]] > > >Sent: Friday, October 03, 2014 1:07 PM > > >To: [email protected] > > >Subject: Re: Success! RE: how to use MetadataBasedFileVersioner > > >properly > > > > > >Awesome Val! :) > > > > > >I think you¹re running into this: > > > > > >https://issues.apache.org/jira/browse/OODT-751 > > > > > > > > >In the meanwhile, try the 0.6 OPSUI, which you can grab from here: > > > > > >http://repo1.maven.org/maven2/org/apache/oodt/pcs-opsui/0.6/ > > > > > > > > >Grab the WAR file and drop it into your favorite container. > > >Make sure you have all of these environment variables installed > > >*before* starting Tomcat or Jetty, etc.: > > > > > > > > >https://cwiki.apache.org/confluence/display/OODT/Quick+Start+for+PCS+ > > >OP > > >SU > > >I > > > > > > > > >(see steps at bottom and replace 0.5 with 0.6) > > > > > >I¹m working on a fix for OODT-751, at which point RADIX will be > > >pretty buff. Next steps at that point: > > > > > >1. Release 0.7 and then encourage folks to get started by using the > > >Vagrant build, e.g., > > > > > >git clone https://github.com/apache/oodt cd vagrant/radix vagrant up > > > > > >2. Fix OODT-491 and remaining workflow manager issues for Wengine 3. > > >Work > > >on Streaming OODT API with AMP Stack (via M. Starch et al) and > > >release in 0.8. > > >4. Conquer and win. > > > > > >Thanks! > > > > > >Cheers, > > >Chris > > > > > > >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > + > > >+++++ > > >Chris Mattmann, Ph.D. > > >Chief Architect > > >Instrument Software and Science Data Systems Section (398) NASA Jet > > >Propulsion Laboratory Pasadena, CA 91109 USA > > >Office: 168-519, Mailstop: 168-527 > > >Email: [email protected] > > >WWW: http://sunset.usc.edu/~mattmann/ > > > >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > + > > >+++++ > > >Adjunct Associate Professor, Computer Science Department University > > >of Southern California, Los Angeles, CA 90089 USA > > > >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > + > > >+++++ > > > > > > > > > > > > > > > > > > > > >-----Original Message----- > > >From: <Mallder>, Valerie <[email protected]> > > ><mailto:[email protected]> > > >Reply-To: "[email protected]" <mailto:[email protected]> > > ><[email protected]> <mailto:[email protected]> > > >Date: Friday, October 3, 2014 at 9:53 AM > > >To: "[email protected]" <mailto:[email protected]> > > ><[email protected]> <mailto:[email protected]> > > >Subject: Success! RE: how to use MetadataBasedFileVersioner properly > > > > > > > > > > > > Hi Chris, > > > > > >Yes, that indeed fixed it! Thanks so much! I now have 18 > > >engineering files ingested. Whoo hoo!! > > > > > >Ok, so now, I would like to see what the opsui has to say about my > > >ingested files. And here comes probably a really stupid question. I > > >am using Firefox on a Redhat Linux box, and I'm not that familiar > > >with Firefox and it's settings, so this might be a browser setting issue. > > >When I startup the opsui and select "File Catalog Browse" it shows > > >that I have > > >18 EngineeringFiles. Then, when I select File Catalog Browse from the > > >strip of options under the logo, I get a message saying the page has > > >expired. And here's the link that shows up in the address bar: > > >http://localhost:8080/opsui/?wicket:interface=:7:fmbrowser_link::ILin > > >kL > > >ist > > >ener:: > > > > > >And I get the page expired message for all of the options that I > > > > > > > > > > > > select. > > > > > > > > > Any idea's on this one?? > > > > > >In the meantime though, I will start playing with an action for post > > >ingestion success that simply makes another copy of all these > > >engineering files and puts them another folder for Level 0 files. > > > > > >Thanks, > > >Val > > > > > > > > > > > > > > >Valerie A. Mallder > > >New Horizons Deputy Mission System Engineer Johns Hopkins > > >University/Applied Physics Laboratory > > > > > > > > > > > > > > > -----Original Message----- > > >From: Chris Mattmann [mailto:[email protected]] > > >Sent: Friday, October 03, 2014 3:03 AM > > >To: [email protected] > > >Subject: Re: how to use MetadataBasedFileVersioner properly > > > > > >You?re almost there Val! > > > > > >Unfortunately Versioners right now aren?t configurable from product > > >type policy (would be great to capture this in a JIRA issue, here: > > >https://issues.apache.org/jira/browse/OODT). If they were, it would > > >have picked up your <property .. declaration of filePathSpec below. > > >It?s been on my TODO list for a long time. > > > > > >Instead I created this: > > > > > >https://issues.apache.org/jira/browse/OODT-639 > > > > > > > > >So you can amend your definition below (also note you always have to > > >include /[Filename] at the end to get the filename you want). > > > > > > <type id="urn:oodt:EngineeringFile" name="EngineeringFile"> > > > <repository path="file://[OODT_HOME]/data/archive/ops/eng" > > ><file://[OODT_HOME]/data/archive/ops/eng>/> > > > <versioner > > > > > > > > > > > > > > > > > > > > > > > >class=?org.apache.oodt.cas.filemgr.versioning.ProductTypeMetVersioner > > >?/ > > >> > > > > > > > > > > > > <description>The default product type for any kind > > >of file.</description> > > > <metExtractors> > > > <extractor > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >class="org.apache.oodt.cas.filemgr.metadata.extractors.CoreMetExtract > > >or > > >"> > > > <configuration> > > > <!-- you can optionally include the envReplace tag to turn > > >on/off environment var replacement --> > > > <property name="nsAware" value="true" /> > > > <property name="elementNs" value="CAS" /> > > > <property name="elements" > > >value="ProductReceivedTime,ProductName,ProductId" /> > > > </configuration> > > > </extractor> > > > </metExtractors> > > > <metadata> > > > <keyval> > > > <key>filePathSpec</key> > > > <val>/[YearDir]/[DoyDir]/[Filename]</val> > > > </keyval> > > > > > ></metadata> > > > </type> > > > > > > > > >See if that fixes it! > > > > > >Cheers, > > >Chris > > > > > >------------------------ > > >Chris Mattmann > > >[email protected] > > > > > > > > > > > > > > >-----Original Message----- > > >From: "Mallder, Valerie" <[email protected]> > > ><mailto:[email protected]> > > >Reply-To: <[email protected]> <mailto:[email protected]> > > >Date: Thursday, October 2, 2014 at 2:56 PM > > >To: "[email protected]" <mailto:[email protected]> > > ><[email protected]> <mailto:[email protected]> > > >Subject: how to use MetadataBasedFileVersioner properly > > > > > > > > > > > > Hi All, > > > > > >I am trying to use the MetadataBasedFileVersioner to store my files > > >in a subdirectory of the data/archive folder based on the values of > > >two metadata elements. Has anyone does this before, and if so, can > > >you give me some hints. > > > > > >My files need to be organized in subdirectories by year and day of > > > > > > > > > > > > year. > > > > > > > > > So, lets say the file name is 'myfile', the year is > > >2014, and the doy of year is 002. Then, the end result that I am > > >looking for is to have the final location of my file be: > > > > > > > > > > > > 'data/archive/ops/eng/2014/002/myfile. > > > > > > > > > However, in my log file, the INFO messages indicate > > >that the generated final location reference is > > >'data/archive/ops/eng/myfile'. > > >And it doesn't include the year and day of year at all. And the > > >incorrect location leads to other errors. So I want to solve this one > > >first. Has anyone tried to do something like this before? > > > > > >Here is what I have done so far: > > > > > >Added two new elements to the .met file 'YearDir' and 'DoyDir'. > > >Added these new elements to the elements.xml file. > > >Added these new elements to the product in the > > >product-type-element-map.xml file. > > >And, I have made changes to the product-types.xml file, but here's > > >where I am not sure I've done this properly. > > > > > >Here's what product type definition looks like: > > > <type id="urn:oodt:EngineeringFile" name="EngineeringFile"> > > > <repository path="file://[OODT_HOME]/data/archive/ops/eng" > > ><file://[OODT_HOME]/data/archive/ops/eng>/> > > > <versioner > > > > > > > > > > > > > > > > > > > > >class="org.apache.oodt.cas.filemgr.versioning.MetadataBasedFileVersio > > >ner > > >"> > > > <property name="filePathSpec" > > > > > > > > > > > > > > > > > > > > > > > > value="/[YearDir]/[DoyDir]/" /> > > > > > > > > > > > > > > > </versioner> > > > <description>The default product type for any kind of > > >file.</description> > > > <metExtractors> > > > <extractor > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >class="org.apache.oodt.cas.filemgr.metadata.extractors.CoreMetExtract > > >o > > >r" > > > > > > > > > > > > > > > > > > <configuration> > > > <!-- you can optionally include the envReplace tag to turn > > >on/off environment var replacement --> > > > <property name="nsAware" value="true" /> > > > <property name="elementNs" value="CAS" /> > > > <property name="elements" > > >value="ProductReceivedTime,ProductName,ProductId" /> > > > </configuration> > > > </extractor> > > > </metExtractors> > > > <metadata/> > > > </type> > > > > > >Thanks in advance for any help or ideas you might have! > > > > > >Valerie > > > > > > > > > > > >Valerie A. Mallder > > > > > >New Horizons Deputy Mission System Engineer The Johns Hopkins > > >University/Applied Physics Laboratory > > >11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723 > > >240-228-7846 (Office) 410-504-2233 (Blackberry) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Tom Barber | Technical Director > > > > > > meteorite bi > > > T: +44 20 8133 3730 > > > W: www.meteorite.bi <http://www.meteorite.bi> | Skype: > > >meteorite.consulting > > > A: Surrey Technology Centre, Surrey Research Park, > > > Guildford, GU2 7YG, UK > > > > > >
