I agree - this is very good. Dan
On 11/21/14 2:01 AM, "Chris Mattmann" <[email protected]> wrote: >You are awesome Val, great work! > >------------------------ >Chris Mattmann >[email protected] > > > > >-----Original Message----- >From: "Mallder, Valerie" <[email protected]> >Reply-To: <[email protected]> >Date: Friday, November 21, 2014 at 4:43 PM >To: "[email protected]" <[email protected]> >Subject: RE: more info for OODT-751 OPSUI Pages constantly expire > >>Oh wait, maybe I misunderstood. I didn't see an actual "fix", just more >>discussion that sparked an idea for me to try something which I have been >>trying to "try" for several days now. I was planning to give an update of >>where I am today. I am going on vacation for two weeks starting Monday, >>so I was definitely planning to give y'all an update today before I >>leave. The good news is, I do have all stages of the current JEDI >>pipeline at APL implemented in OODT (With a couple of caveats, of course, >>but ..). I have raw telemetry from JPL's File Exchange Interface (FEI) >>going in, and daily summary plots of the telemetry data coming out - and >>that in itself deserves a huge YAY!!. I haven't haven't had to write any >>project-specific java code yet. The only java code change I made was to >>make the fix identified OODT-757 that prevented an autodetect crawler >>from running. (I think this fix is already planned for 0.8 and may have >>been implemented already in the trunk already.) I have just one >>workflow and 4 PGE tasks, and I glued together several existing 5 java >>applications by rewriting some perl scripts and adding a few new perl >>scripts. I have verified that all 4 of my product types (EngineeringFile, >>ScienceFile, EDRFile, and SummaryPlot) that are generated by the pipeline >>are in the catalog (using the query-tool), and I verified that the the >>summary plots from my pipeline match the summary plots that were >>generated by the existing JEDI pipeline. All I need now is an operator >>interface to demo this to the folks here at APL. I have kept a detailed >>list of lessons learned which I plan to share (probably after I get back >>from vacation though). >> >>.... more later today >> >>Val >> >> >> >>Valerie A. Mallder >>New Horizons Deputy Mission System Engineer >>Johns Hopkins University/Applied Physics Laboratory >> >>> -----Original Message----- >>> From: Mattmann, Chris A (3980) [mailto:[email protected]] >>> Sent: Friday, November 21, 2014 10:13 AM >>> To: Tom Barber; [email protected] >>> Subject: Re: more info for OODT-751 OPSUI Pages constantly expire >>> >>> You did! Did it work? Can you pass along the thread so I can read it >>>sorry I'm >>> trying to catch up.. >>> >>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> +++++ >>> Chris Mattmann, Ph.D. >>> Chief Architect >>> Instrument Software and Science Data Systems Section (398) NASA Jet >>> Propulsion Laboratory Pasadena, CA 91109 USA >>> Office: 168-519, Mailstop: 168-527 >>> Email: [email protected] >>> WWW: http://sunset.usc.edu/~mattmann/ >>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> +++++ >>> Adjunct Associate Professor, Computer Science Department University of >>> Southern California, Los Angeles, CA 90089 USA >>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> +++++ >>> >>> >>> >>> >>> >>> >>> -----Original Message----- >>> From: Tom Barber <[email protected]> >>> Date: Friday, November 21, 2014 at 4:06 PM >>> To: <[email protected]> >>> Subject: Re: more info for OODT-751 OPSUI Pages constantly expire >>> >>> > >>> > >>> > >>> > Oi Chris, I sent Val a fix! :P >>> > >>> > On 21/11/14 14:46, Mattmann, Chris A (3980) wrote: >>> > >>> > >>> > Hi Val, >>> > >>> >I haven't forgot about this - just haven't had a chance to >>>investigate. >>> >Hope to have some time in the next week. >>> > >>> >Cheers, >>> >Chris >>> > >>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> ++++++ >>> >Chris Mattmann, Ph.D. >>> >Chief Architect >>> >Instrument Software and Science Data Systems Section (398) NASA Jet >>> >Propulsion Laboratory Pasadena, CA 91109 USA >>> >Office: 168-519, Mailstop: 168-527 >>> >Email: [email protected] >>> >WWW: http://sunset.usc.edu/~mattmann/ >>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> ++++++ >>> >Adjunct Associate Professor, Computer Science Department University of >>> >Southern California, Los Angeles, CA 90089 USA >>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> ++++++ >>> > >>> > >>> > >>> > >>> > >>> > >>> >-----Original Message----- >>> >From: <Mallder>, Valerie <[email protected]> >>> ><mailto:[email protected]> >>> >Reply-To: "[email protected]" <mailto:[email protected]> >>> ><[email protected]> <mailto:[email protected]> >>> >Date: Thursday, November 13, 2014 at 2:10 PM >>> >To: "[email protected]" <mailto:[email protected]> >>> ><[email protected]> <mailto:[email protected]> >>> >Subject: RE: more info for OODT-751 OPSUI Pages constantly expire >>> > >>> > >>> > >>> > For those of you who are using the pcs-opsui in an operational >>> >environment without getting expired pages please tell me what version >>> >you are using, the OS you are running it on, and the browser you are >>>using. >>> > >>> >I need to demonstrate "something" to my project manager to show the >>> >usefulness of OODT in the Jedi instrument science data pipeline here >>>at >>> >APL. And right now, I have nothing to "show" for my last few months of >>> >work. If someone can tell me a configuration using opsui that is >>> >working then maybe I can try to mimic that and get something useful up >>> >and running. So far, for me, versions 0.6, 0.7 and the current trunk >>> >are showing only expired pages. >>> > >>> >Thanks very much! >>> >Val >>> > >>> > >>> > >>> >Sent from my iPhone. >>> >________________________________ >>> >From: Mallder, Valerie <[email protected]> >>> ><mailto:[email protected]> >>> >Sent: Tuesday, November 11, 2014 2:55:12 PM >>> >To: [email protected] >>> >Subject: more info for OODT-751 OPSUI Pages constantly expire >>> > >>> >Hi Chris, >>> > >>> >I know you are working on this, but I wanted to let you know that I >>> >tried to use version 0.6 and am having the same problem in version >>>0.6. >>> >But, I can't guarantee that I installed version 0.6 correctly so I >>> >would like to run this by you. Based on your email below here's what >>>I did: >>> > >>> >1. Downloaded pcs-opsui-0.76.war from: >>> >http://repo1.maven.org/maven2/org/apache/oodt/pcs-opsui/0.6/ >>> >and saved it in a new folder named $OODT_HOME/bin/opsui >>> > >>> >2. Created a script called $OODT_HOME/bin/opsui/runopsui that sets the >>> >following variables (this is output from my script): >>> >Using OODT_BASE: >>> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy >>> >Using OODT_HOME: >>> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy >>> >Using OODT_TMPDIR: >>> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/temp >>> >Using FILEMGR_URL: http://localhost:9000 >>> >Using WORKFLOW_URL: http://localhost:9001 >>> >Using RESMGR_URL: http://localhost:9002 >>> >Using WORKFLOW_HOME: >>> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/workflow >>> >Using RESMGR_HOME: >>> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/resmgr >>> >Using CRAWLER_HOME: >>> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/crawler >>> >Using TOMCAT_HOME: >>> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat >>> >Using PCS_HOME: >>> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pcs >>> >Using PGE_HOME: >>> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge >>> >Using PGE_JOBS_DIR: >>> >>>>/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/pge/j >>> >obs >>> >Using FEI_DROP_DIR: >>> >>>>/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/telem >>> >etr >>> >y >>> >Using JEDI_L0_DIR: >>> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/l0 >>> >Using JEDI_L2_DIR: >>> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/l2 >>> >Using ARCHIVE_DIR: >>> >>>>/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/archi >>> >ve >>> >Using BACKUP_DIR: >>> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/met >>> >Using FAILURE_DIR: >>> >>>>/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/failu >>> >re Using JEDI_PIPELINE_DIR: /homes/malldva1/working/pipeline Using >>> >SNAPSHOT_DIR: >>> >>>>/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/pge/j >>> >obs >>> >/snapshot >>> > >>> >(Then, I followed the steps at the bottom of: >>> >>>>https://cwiki.apache.org/confluence/display/OODT/Quick+Start+for+PCS+OP >>> >SUI >>> >) >>> > >>> >3. Downloaded >>> >>>>http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/webapp/ME >>> >TA- >>> >INF/context.xml >>> >And saved it as $OODT_HOME/bin/opsui/pcs-opsui.xml >>> > >>> >4. Edited $OODT_HOME/bin/opsui/pcs-opsui.xml and changed the first >>>line >>> >from: >>> ><Context path="/pcs-opsui"> >>> >To >>> ><Context path="/pcs-opsui" >>> >docBase="[OODT_HOME]/bin/opsui/pcs-opsui-0.6.war"> >>> > >>> >5. Killed any process with "tomcat" in its name to ensure that tomcat >>> >is not running. >>> > >>> >6. Executed the following command to create a symbolic link: >>> >cd $OODT_HOME/bin/opsui/ >>> >ln -s pcs-opsui.xml to >>> >$TOMCAT_HOME/conf/Catalina/localhost/pcs-opsui.xml >>> > >>> >7. Then, in my script that sets the environment variables, I added the >>> >following command to start tomcat. >>> >exec "$OODT_BASE"/tomcat/bin/catalina.sh start >>> > >>> >8. Then, I ran my new $OODT_HOME/bin/opsui/runopsui script >>> > >>> >9. Then, I started firefox and went to "localhost:8080/pcs-opsui" >>> > >>> > >>> > >>> >The home page comes up, and the PCS Status page comes up, but all >>>other >>> >pages are expired. I should note that the "radix" installation of >>> >version 0.7 points me to localhost:8080/opsui, while step 10 of the >>> >instructions from the quick start guide of the wiki point me to >>> >localhost:8080/pcs-opsui. And, no matter which one I go to, the pages >>> >expire. >>> > >>> >I also tried clearing out my cache and browser history and restarting >>> >firefox and topcat, etc. and nothing seems to help. >>> > >>> >If I have gotten the version 0.6 up and running correctly, I thought >>> >you would be interested to know that I am seeing the error. If I did >>> >not get version 0.6 up and running, then I need more hints on how to >>>do >>> >it. :) >>> > >>> >Thanks, >>> >Val >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> >Valerie A. Mallder >>> >New Horizons Deputy Mission System Engineer Johns Hopkins >>> >University/Applied Physics Laboratory >>> > >>> > >>> > >>> > >>> > -----Original Message----- >>> >From: Mattmann, Chris A (3980) [mailto:[email protected]] >>> >Sent: Friday, October 03, 2014 1:07 PM >>> >To: [email protected] >>> >Subject: Re: Success! RE: how to use MetadataBasedFileVersioner >>> >properly >>> > >>> >Awesome Val! :) >>> > >>> >I think you¹re running into this: >>> > >>> >https://issues.apache.org/jira/browse/OODT-751 >>> > >>> > >>> >In the meanwhile, try the 0.6 OPSUI, which you can grab from here: >>> > >>> >http://repo1.maven.org/maven2/org/apache/oodt/pcs-opsui/0.6/ >>> > >>> > >>> >Grab the WAR file and drop it into your favorite container. >>> >Make sure you have all of these environment variables installed >>> >*before* starting Tomcat or Jetty, etc.: >>> > >>> > >>> >>>>https://cwiki.apache.org/confluence/display/OODT/Quick+Start+for+PCS+OP >>> >SU >>> >I >>> > >>> > >>> >(see steps at bottom and replace 0.5 with 0.6) >>> > >>> >I¹m working on a fix for OODT-751, at which point RADIX will be pretty >>> >buff. Next steps at that point: >>> > >>> >1. Release 0.7 and then encourage folks to get started by using the >>> >Vagrant build, e.g., >>> > >>> >git clone https://github.com/apache/oodt cd vagrant/radix vagrant up >>> > >>> >2. Fix OODT-491 and remaining workflow manager issues for Wengine 3. >>> >Work >>> >on Streaming OODT API with AMP Stack (via M. Starch et al) and release >>> >in 0.8. >>> >4. Conquer and win. >>> > >>> >Thanks! >>> > >>> >Cheers, >>> >Chris >>> > >>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> + >>> >+++++ >>> >Chris Mattmann, Ph.D. >>> >Chief Architect >>> >Instrument Software and Science Data Systems Section (398) NASA Jet >>> >Propulsion Laboratory Pasadena, CA 91109 USA >>> >Office: 168-519, Mailstop: 168-527 >>> >Email: [email protected] >>> >WWW: http://sunset.usc.edu/~mattmann/ >>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> + >>> >+++++ >>> >Adjunct Associate Professor, Computer Science Department University of >>> >Southern California, Los Angeles, CA 90089 USA >>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> + >>> >+++++ >>> > >>> > >>> > >>> > >>> > >>> > >>> >-----Original Message----- >>> >From: <Mallder>, Valerie <[email protected]> >>> ><mailto:[email protected]> >>> >Reply-To: "[email protected]" <mailto:[email protected]> >>> ><[email protected]> <mailto:[email protected]> >>> >Date: Friday, October 3, 2014 at 9:53 AM >>> >To: "[email protected]" <mailto:[email protected]> >>> ><[email protected]> <mailto:[email protected]> >>> >Subject: Success! RE: how to use MetadataBasedFileVersioner properly >>> > >>> > >>> > >>> > Hi Chris, >>> > >>> >Yes, that indeed fixed it! Thanks so much! I now have 18 engineering >>> >files ingested. Whoo hoo!! >>> > >>> >Ok, so now, I would like to see what the opsui has to say about my >>> >ingested files. And here comes probably a really stupid question. I >>>am >>> >using Firefox on a Redhat Linux box, and I'm not that familiar with >>> >Firefox and it's settings, so this might be a browser setting issue. >>> >When I startup the opsui and select "File Catalog Browse" it shows >>>that >>> >I have >>> >18 EngineeringFiles. Then, when I select File Catalog Browse from the >>> >strip of options under the logo, I get a message saying the page has >>> >expired. And here's the link that shows up in the address bar: >>> >>>>http://localhost:8080/opsui/?wicket:interface=:7:fmbrowser_link::ILinkL >>> >ist >>> >ener:: >>> > >>> >And I get the page expired message for all of the options that I >>> > >>> > >>> > >>> > select. >>> > >>> > >>> > Any idea's on this one?? >>> > >>> >In the meantime though, I will start playing with an action for post >>> >ingestion success that simply makes another copy of all these >>> >engineering files and puts them another folder for Level 0 files. >>> > >>> >Thanks, >>> >Val >>> > >>> > >>> > >>> > >>> >Valerie A. Mallder >>> >New Horizons Deputy Mission System Engineer Johns Hopkins >>> >University/Applied Physics Laboratory >>> > >>> > >>> > >>> > >>> > -----Original Message----- >>> >From: Chris Mattmann [mailto:[email protected]] >>> >Sent: Friday, October 03, 2014 3:03 AM >>> >To: [email protected] >>> >Subject: Re: how to use MetadataBasedFileVersioner properly >>> > >>> >You?re almost there Val! >>> > >>> >Unfortunately Versioners right now aren?t configurable from product >>> >type policy (would be great to capture this in a JIRA issue, here: >>> >https://issues.apache.org/jira/browse/OODT). If they were, it would >>> >have picked up your <property .. declaration of filePathSpec below. >>> >It?s been on my TODO list for a long time. >>> > >>> >Instead I created this: >>> > >>> >https://issues.apache.org/jira/browse/OODT-639 >>> > >>> > >>> >So you can amend your definition below (also note you always have to >>> >include /[Filename] at the end to get the filename you want). >>> > >>> > <type id="urn:oodt:EngineeringFile" name="EngineeringFile"> >>> > <repository path="file://[OODT_HOME]/data/archive/ops/eng" >>> ><file://[OODT_HOME]/data/archive/ops/eng>/> >>> > <versioner >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> >>>>class=?org.apache.oodt.cas.filemgr.versioning.ProductTypeMetVersioner?/ >>> >> >>> > >>> > >>> > >>> > <description>The default product type for any kind >>>of >>> >file.</description> >>> > <metExtractors> >>> > <extractor >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> >>>>class="org.apache.oodt.cas.filemgr.metadata.extractors.CoreMetExtractor >>> >"> >>> > <configuration> >>> > <!-- you can optionally include the envReplace tag to turn >>> >on/off environment var replacement --> >>> > <property name="nsAware" value="true" /> >>> > <property name="elementNs" value="CAS" /> >>> > <property name="elements" >>> >value="ProductReceivedTime,ProductName,ProductId" /> >>> > </configuration> >>> > </extractor> >>> > </metExtractors> >>> > <metadata> >>> > <keyval> >>> > <key>filePathSpec</key> >>> > <val>/[YearDir]/[DoyDir]/[Filename]</val> >>> > </keyval> >>> > >>> ></metadata> >>> > </type> >>> > >>> > >>> >See if that fixes it! >>> > >>> >Cheers, >>> >Chris >>> > >>> >------------------------ >>> >Chris Mattmann >>> >[email protected] >>> > >>> > >>> > >>> > >>> >-----Original Message----- >>> >From: "Mallder, Valerie" <[email protected]> >>> ><mailto:[email protected]> >>> >Reply-To: <[email protected]> <mailto:[email protected]> >>> >Date: Thursday, October 2, 2014 at 2:56 PM >>> >To: "[email protected]" <mailto:[email protected]> >>> ><[email protected]> <mailto:[email protected]> >>> >Subject: how to use MetadataBasedFileVersioner properly >>> > >>> > >>> > >>> > Hi All, >>> > >>> >I am trying to use the MetadataBasedFileVersioner to store my files in >>> >a subdirectory of the data/archive folder based on the values of two >>> >metadata elements. Has anyone does this before, and if so, can you >>>give >>> >me some hints. >>> > >>> >My files need to be organized in subdirectories by year and day of >>> > >>> > >>> > >>> > year. >>> > >>> > >>> > So, lets say the file name is 'myfile', the year is >>> >2014, and the doy of year is 002. Then, the end result that I am >>> >looking for is to have the final location of my file be: >>> > >>> > >>> > >>> > 'data/archive/ops/eng/2014/002/myfile. >>> > >>> > >>> > However, in my log file, the INFO messages indicate >>> >that the generated final location reference is >>> >'data/archive/ops/eng/myfile'. >>> >And it doesn't include the year and day of year at all. And the >>> >incorrect location leads to other errors. So I want to solve this one >>> >first. Has anyone tried to do something like this before? >>> > >>> >Here is what I have done so far: >>> > >>> >Added two new elements to the .met file 'YearDir' and 'DoyDir'. >>> >Added these new elements to the elements.xml file. >>> >Added these new elements to the product in the >>> >product-type-element-map.xml file. >>> >And, I have made changes to the product-types.xml file, but here's >>> >where I am not sure I've done this properly. >>> > >>> >Here's what product type definition looks like: >>> > <type id="urn:oodt:EngineeringFile" name="EngineeringFile"> >>> > <repository path="file://[OODT_HOME]/data/archive/ops/eng" >>> ><file://[OODT_HOME]/data/archive/ops/eng>/> >>> > <versioner >>> > >>> > >>> > >>> > >>> > >>> > >>> >class="org.apache.oodt.cas.filemgr.versioning.MetadataBasedFileVersio >>> >ner >>> >"> >>> > <property name="filePathSpec" >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > value="/[YearDir]/[DoyDir]/" /> >>> > >>> > >>> > >>> > >>> > </versioner> >>> > <description>The default product type for any kind of >>> >file.</description> >>> > <metExtractors> >>> > <extractor >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> >class="org.apache.oodt.cas.filemgr.metadata.extractors.CoreMetExtracto >>> >r" >>> > >>> > >>> > >>> > >>> > >>> > <configuration> >>> > <!-- you can optionally include the envReplace tag to turn >>> >on/off environment var replacement --> >>> > <property name="nsAware" value="true" /> >>> > <property name="elementNs" value="CAS" /> >>> > <property name="elements" >>> >value="ProductReceivedTime,ProductName,ProductId" /> >>> > </configuration> >>> > </extractor> >>> > </metExtractors> >>> > <metadata/> >>> > </type> >>> > >>> >Thanks in advance for any help or ideas you might have! >>> > >>> >Valerie >>> > >>> > >>> > >>> >Valerie A. Mallder >>> > >>> >New Horizons Deputy Mission System Engineer The Johns Hopkins >>> >University/Applied Physics Laboratory >>> >11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723 >>> >240-228-7846 (Office) 410-504-2233 (Blackberry) >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > -- >>> > Tom Barber | Technical Director >>> > >>> > meteorite bi >>> > T: +44 20 8133 3730 >>> > W: www.meteorite.bi <http://www.meteorite.bi> | Skype: >>> >meteorite.consulting >>> > A: Surrey Technology Centre, Surrey Research Park, >>> > Guildford, GU2 7YG, UK >>> > >>> > >> > >
