You are awesome Val, great work! ------------------------ Chris Mattmann [email protected]
-----Original Message----- From: "Mallder, Valerie" <[email protected]> Reply-To: <[email protected]> Date: Friday, November 21, 2014 at 4:43 PM To: "[email protected]" <[email protected]> Subject: RE: more info for OODT-751 OPSUI Pages constantly expire >Oh wait, maybe I misunderstood. I didn't see an actual "fix", just more >discussion that sparked an idea for me to try something which I have been >trying to "try" for several days now. I was planning to give an update of >where I am today. I am going on vacation for two weeks starting Monday, >so I was definitely planning to give y'all an update today before I >leave. The good news is, I do have all stages of the current JEDI >pipeline at APL implemented in OODT (With a couple of caveats, of course, >but ..). I have raw telemetry from JPL's File Exchange Interface (FEI) >going in, and daily summary plots of the telemetry data coming out - and >that in itself deserves a huge YAY!!. I haven't haven't had to write any >project-specific java code yet. The only java code change I made was to >make the fix identified OODT-757 that prevented an autodetect crawler >from running. (I think this fix is already planned for 0.8 and may have >been implemented already in the trunk already.) I have just one >workflow and 4 PGE tasks, and I glued together several existing 5 java >applications by rewriting some perl scripts and adding a few new perl >scripts. I have verified that all 4 of my product types (EngineeringFile, >ScienceFile, EDRFile, and SummaryPlot) that are generated by the pipeline >are in the catalog (using the query-tool), and I verified that the the >summary plots from my pipeline match the summary plots that were >generated by the existing JEDI pipeline. All I need now is an operator >interface to demo this to the folks here at APL. I have kept a detailed >list of lessons learned which I plan to share (probably after I get back >from vacation though). > >.... more later today > >Val > > > >Valerie A. Mallder >New Horizons Deputy Mission System Engineer >Johns Hopkins University/Applied Physics Laboratory > >> -----Original Message----- >> From: Mattmann, Chris A (3980) [mailto:[email protected]] >> Sent: Friday, November 21, 2014 10:13 AM >> To: Tom Barber; [email protected] >> Subject: Re: more info for OODT-751 OPSUI Pages constantly expire >> >> You did! Did it work? Can you pass along the thread so I can read it >>sorry I'm >> trying to catch up.. >> >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> +++++ >> Chris Mattmann, Ph.D. >> Chief Architect >> Instrument Software and Science Data Systems Section (398) NASA Jet >> Propulsion Laboratory Pasadena, CA 91109 USA >> Office: 168-519, Mailstop: 168-527 >> Email: [email protected] >> WWW: http://sunset.usc.edu/~mattmann/ >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> +++++ >> Adjunct Associate Professor, Computer Science Department University of >> Southern California, Los Angeles, CA 90089 USA >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> +++++ >> >> >> >> >> >> >> -----Original Message----- >> From: Tom Barber <[email protected]> >> Date: Friday, November 21, 2014 at 4:06 PM >> To: <[email protected]> >> Subject: Re: more info for OODT-751 OPSUI Pages constantly expire >> >> > >> > >> > >> > Oi Chris, I sent Val a fix! :P >> > >> > On 21/11/14 14:46, Mattmann, Chris A (3980) wrote: >> > >> > >> > Hi Val, >> > >> >I haven't forgot about this - just haven't had a chance to investigate. >> >Hope to have some time in the next week. >> > >> >Cheers, >> >Chris >> > >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> ++++++ >> >Chris Mattmann, Ph.D. >> >Chief Architect >> >Instrument Software and Science Data Systems Section (398) NASA Jet >> >Propulsion Laboratory Pasadena, CA 91109 USA >> >Office: 168-519, Mailstop: 168-527 >> >Email: [email protected] >> >WWW: http://sunset.usc.edu/~mattmann/ >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> ++++++ >> >Adjunct Associate Professor, Computer Science Department University of >> >Southern California, Los Angeles, CA 90089 USA >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> ++++++ >> > >> > >> > >> > >> > >> > >> >-----Original Message----- >> >From: <Mallder>, Valerie <[email protected]> >> ><mailto:[email protected]> >> >Reply-To: "[email protected]" <mailto:[email protected]> >> ><[email protected]> <mailto:[email protected]> >> >Date: Thursday, November 13, 2014 at 2:10 PM >> >To: "[email protected]" <mailto:[email protected]> >> ><[email protected]> <mailto:[email protected]> >> >Subject: RE: more info for OODT-751 OPSUI Pages constantly expire >> > >> > >> > >> > For those of you who are using the pcs-opsui in an operational >> >environment without getting expired pages please tell me what version >> >you are using, the OS you are running it on, and the browser you are >>using. >> > >> >I need to demonstrate "something" to my project manager to show the >> >usefulness of OODT in the Jedi instrument science data pipeline here at >> >APL. And right now, I have nothing to "show" for my last few months of >> >work. If someone can tell me a configuration using opsui that is >> >working then maybe I can try to mimic that and get something useful up >> >and running. So far, for me, versions 0.6, 0.7 and the current trunk >> >are showing only expired pages. >> > >> >Thanks very much! >> >Val >> > >> > >> > >> >Sent from my iPhone. >> >________________________________ >> >From: Mallder, Valerie <[email protected]> >> ><mailto:[email protected]> >> >Sent: Tuesday, November 11, 2014 2:55:12 PM >> >To: [email protected] >> >Subject: more info for OODT-751 OPSUI Pages constantly expire >> > >> >Hi Chris, >> > >> >I know you are working on this, but I wanted to let you know that I >> >tried to use version 0.6 and am having the same problem in version 0.6. >> >But, I can't guarantee that I installed version 0.6 correctly so I >> >would like to run this by you. Based on your email below here's what >>I did: >> > >> >1. Downloaded pcs-opsui-0.76.war from: >> >http://repo1.maven.org/maven2/org/apache/oodt/pcs-opsui/0.6/ >> >and saved it in a new folder named $OODT_HOME/bin/opsui >> > >> >2. Created a script called $OODT_HOME/bin/opsui/runopsui that sets the >> >following variables (this is output from my script): >> >Using OODT_BASE: >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy >> >Using OODT_HOME: >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy >> >Using OODT_TMPDIR: >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/temp >> >Using FILEMGR_URL: http://localhost:9000 >> >Using WORKFLOW_URL: http://localhost:9001 >> >Using RESMGR_URL: http://localhost:9002 >> >Using WORKFLOW_HOME: >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/workflow >> >Using RESMGR_HOME: >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/resmgr >> >Using CRAWLER_HOME: >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/crawler >> >Using TOMCAT_HOME: >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat >> >Using PCS_HOME: >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pcs >> >Using PGE_HOME: >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge >> >Using PGE_JOBS_DIR: >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/pge/j >> >obs >> >Using FEI_DROP_DIR: >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/telem >> >etr >> >y >> >Using JEDI_L0_DIR: >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/l0 >> >Using JEDI_L2_DIR: >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/l2 >> >Using ARCHIVE_DIR: >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/archi >> >ve >> >Using BACKUP_DIR: >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/met >> >Using FAILURE_DIR: >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/failu >> >re Using JEDI_PIPELINE_DIR: /homes/malldva1/working/pipeline Using >> >SNAPSHOT_DIR: >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/pge/j >> >obs >> >/snapshot >> > >> >(Then, I followed the steps at the bottom of: >> >https://cwiki.apache.org/confluence/display/OODT/Quick+Start+for+PCS+OP >> >SUI >> >) >> > >> >3. Downloaded >> >http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/webapp/ME >> >TA- >> >INF/context.xml >> >And saved it as $OODT_HOME/bin/opsui/pcs-opsui.xml >> > >> >4. Edited $OODT_HOME/bin/opsui/pcs-opsui.xml and changed the first line >> >from: >> ><Context path="/pcs-opsui"> >> >To >> ><Context path="/pcs-opsui" >> >docBase="[OODT_HOME]/bin/opsui/pcs-opsui-0.6.war"> >> > >> >5. Killed any process with "tomcat" in its name to ensure that tomcat >> >is not running. >> > >> >6. Executed the following command to create a symbolic link: >> >cd $OODT_HOME/bin/opsui/ >> >ln -s pcs-opsui.xml to >> >$TOMCAT_HOME/conf/Catalina/localhost/pcs-opsui.xml >> > >> >7. Then, in my script that sets the environment variables, I added the >> >following command to start tomcat. >> >exec "$OODT_BASE"/tomcat/bin/catalina.sh start >> > >> >8. Then, I ran my new $OODT_HOME/bin/opsui/runopsui script >> > >> >9. Then, I started firefox and went to "localhost:8080/pcs-opsui" >> > >> > >> > >> >The home page comes up, and the PCS Status page comes up, but all other >> >pages are expired. I should note that the "radix" installation of >> >version 0.7 points me to localhost:8080/opsui, while step 10 of the >> >instructions from the quick start guide of the wiki point me to >> >localhost:8080/pcs-opsui. And, no matter which one I go to, the pages >> >expire. >> > >> >I also tried clearing out my cache and browser history and restarting >> >firefox and topcat, etc. and nothing seems to help. >> > >> >If I have gotten the version 0.6 up and running correctly, I thought >> >you would be interested to know that I am seeing the error. If I did >> >not get version 0.6 up and running, then I need more hints on how to do >> >it. :) >> > >> >Thanks, >> >Val >> > >> > >> > >> > >> > >> > >> > >> > >> > >> >Valerie A. Mallder >> >New Horizons Deputy Mission System Engineer Johns Hopkins >> >University/Applied Physics Laboratory >> > >> > >> > >> > >> > -----Original Message----- >> >From: Mattmann, Chris A (3980) [mailto:[email protected]] >> >Sent: Friday, October 03, 2014 1:07 PM >> >To: [email protected] >> >Subject: Re: Success! RE: how to use MetadataBasedFileVersioner >> >properly >> > >> >Awesome Val! :) >> > >> >I think you¹re running into this: >> > >> >https://issues.apache.org/jira/browse/OODT-751 >> > >> > >> >In the meanwhile, try the 0.6 OPSUI, which you can grab from here: >> > >> >http://repo1.maven.org/maven2/org/apache/oodt/pcs-opsui/0.6/ >> > >> > >> >Grab the WAR file and drop it into your favorite container. >> >Make sure you have all of these environment variables installed >> >*before* starting Tomcat or Jetty, etc.: >> > >> > >> >https://cwiki.apache.org/confluence/display/OODT/Quick+Start+for+PCS+OP >> >SU >> >I >> > >> > >> >(see steps at bottom and replace 0.5 with 0.6) >> > >> >I¹m working on a fix for OODT-751, at which point RADIX will be pretty >> >buff. Next steps at that point: >> > >> >1. Release 0.7 and then encourage folks to get started by using the >> >Vagrant build, e.g., >> > >> >git clone https://github.com/apache/oodt cd vagrant/radix vagrant up >> > >> >2. Fix OODT-491 and remaining workflow manager issues for Wengine 3. >> >Work >> >on Streaming OODT API with AMP Stack (via M. Starch et al) and release >> >in 0.8. >> >4. Conquer and win. >> > >> >Thanks! >> > >> >Cheers, >> >Chris >> > >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> + >> >+++++ >> >Chris Mattmann, Ph.D. >> >Chief Architect >> >Instrument Software and Science Data Systems Section (398) NASA Jet >> >Propulsion Laboratory Pasadena, CA 91109 USA >> >Office: 168-519, Mailstop: 168-527 >> >Email: [email protected] >> >WWW: http://sunset.usc.edu/~mattmann/ >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> + >> >+++++ >> >Adjunct Associate Professor, Computer Science Department University of >> >Southern California, Los Angeles, CA 90089 USA >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> + >> >+++++ >> > >> > >> > >> > >> > >> > >> >-----Original Message----- >> >From: <Mallder>, Valerie <[email protected]> >> ><mailto:[email protected]> >> >Reply-To: "[email protected]" <mailto:[email protected]> >> ><[email protected]> <mailto:[email protected]> >> >Date: Friday, October 3, 2014 at 9:53 AM >> >To: "[email protected]" <mailto:[email protected]> >> ><[email protected]> <mailto:[email protected]> >> >Subject: Success! RE: how to use MetadataBasedFileVersioner properly >> > >> > >> > >> > Hi Chris, >> > >> >Yes, that indeed fixed it! Thanks so much! I now have 18 engineering >> >files ingested. Whoo hoo!! >> > >> >Ok, so now, I would like to see what the opsui has to say about my >> >ingested files. And here comes probably a really stupid question. I am >> >using Firefox on a Redhat Linux box, and I'm not that familiar with >> >Firefox and it's settings, so this might be a browser setting issue. >> >When I startup the opsui and select "File Catalog Browse" it shows that >> >I have >> >18 EngineeringFiles. Then, when I select File Catalog Browse from the >> >strip of options under the logo, I get a message saying the page has >> >expired. And here's the link that shows up in the address bar: >> >http://localhost:8080/opsui/?wicket:interface=:7:fmbrowser_link::ILinkL >> >ist >> >ener:: >> > >> >And I get the page expired message for all of the options that I >> > >> > >> > >> > select. >> > >> > >> > Any idea's on this one?? >> > >> >In the meantime though, I will start playing with an action for post >> >ingestion success that simply makes another copy of all these >> >engineering files and puts them another folder for Level 0 files. >> > >> >Thanks, >> >Val >> > >> > >> > >> > >> >Valerie A. Mallder >> >New Horizons Deputy Mission System Engineer Johns Hopkins >> >University/Applied Physics Laboratory >> > >> > >> > >> > >> > -----Original Message----- >> >From: Chris Mattmann [mailto:[email protected]] >> >Sent: Friday, October 03, 2014 3:03 AM >> >To: [email protected] >> >Subject: Re: how to use MetadataBasedFileVersioner properly >> > >> >You?re almost there Val! >> > >> >Unfortunately Versioners right now aren?t configurable from product >> >type policy (would be great to capture this in a JIRA issue, here: >> >https://issues.apache.org/jira/browse/OODT). If they were, it would >> >have picked up your <property .. declaration of filePathSpec below. >> >It?s been on my TODO list for a long time. >> > >> >Instead I created this: >> > >> >https://issues.apache.org/jira/browse/OODT-639 >> > >> > >> >So you can amend your definition below (also note you always have to >> >include /[Filename] at the end to get the filename you want). >> > >> > <type id="urn:oodt:EngineeringFile" name="EngineeringFile"> >> > <repository path="file://[OODT_HOME]/data/archive/ops/eng" >> ><file://[OODT_HOME]/data/archive/ops/eng>/> >> > <versioner >> > >> > >> > >> > >> > >> > >> > >> >class=?org.apache.oodt.cas.filemgr.versioning.ProductTypeMetVersioner?/ >> >> >> > >> > >> > >> > <description>The default product type for any kind of >> >file.</description> >> > <metExtractors> >> > <extractor >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> >class="org.apache.oodt.cas.filemgr.metadata.extractors.CoreMetExtractor >> >"> >> > <configuration> >> > <!-- you can optionally include the envReplace tag to turn >> >on/off environment var replacement --> >> > <property name="nsAware" value="true" /> >> > <property name="elementNs" value="CAS" /> >> > <property name="elements" >> >value="ProductReceivedTime,ProductName,ProductId" /> >> > </configuration> >> > </extractor> >> > </metExtractors> >> > <metadata> >> > <keyval> >> > <key>filePathSpec</key> >> > <val>/[YearDir]/[DoyDir]/[Filename]</val> >> > </keyval> >> > >> ></metadata> >> > </type> >> > >> > >> >See if that fixes it! >> > >> >Cheers, >> >Chris >> > >> >------------------------ >> >Chris Mattmann >> >[email protected] >> > >> > >> > >> > >> >-----Original Message----- >> >From: "Mallder, Valerie" <[email protected]> >> ><mailto:[email protected]> >> >Reply-To: <[email protected]> <mailto:[email protected]> >> >Date: Thursday, October 2, 2014 at 2:56 PM >> >To: "[email protected]" <mailto:[email protected]> >> ><[email protected]> <mailto:[email protected]> >> >Subject: how to use MetadataBasedFileVersioner properly >> > >> > >> > >> > Hi All, >> > >> >I am trying to use the MetadataBasedFileVersioner to store my files in >> >a subdirectory of the data/archive folder based on the values of two >> >metadata elements. Has anyone does this before, and if so, can you give >> >me some hints. >> > >> >My files need to be organized in subdirectories by year and day of >> > >> > >> > >> > year. >> > >> > >> > So, lets say the file name is 'myfile', the year is >> >2014, and the doy of year is 002. Then, the end result that I am >> >looking for is to have the final location of my file be: >> > >> > >> > >> > 'data/archive/ops/eng/2014/002/myfile. >> > >> > >> > However, in my log file, the INFO messages indicate >> >that the generated final location reference is >> >'data/archive/ops/eng/myfile'. >> >And it doesn't include the year and day of year at all. And the >> >incorrect location leads to other errors. So I want to solve this one >> >first. Has anyone tried to do something like this before? >> > >> >Here is what I have done so far: >> > >> >Added two new elements to the .met file 'YearDir' and 'DoyDir'. >> >Added these new elements to the elements.xml file. >> >Added these new elements to the product in the >> >product-type-element-map.xml file. >> >And, I have made changes to the product-types.xml file, but here's >> >where I am not sure I've done this properly. >> > >> >Here's what product type definition looks like: >> > <type id="urn:oodt:EngineeringFile" name="EngineeringFile"> >> > <repository path="file://[OODT_HOME]/data/archive/ops/eng" >> ><file://[OODT_HOME]/data/archive/ops/eng>/> >> > <versioner >> > >> > >> > >> > >> > >> > >> >class="org.apache.oodt.cas.filemgr.versioning.MetadataBasedFileVersio >> >ner >> >"> >> > <property name="filePathSpec" >> > >> > >> > >> > >> > >> > >> > >> > value="/[YearDir]/[DoyDir]/" /> >> > >> > >> > >> > >> > </versioner> >> > <description>The default product type for any kind of >> >file.</description> >> > <metExtractors> >> > <extractor >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> >class="org.apache.oodt.cas.filemgr.metadata.extractors.CoreMetExtracto >> >r" >> > >> > >> > >> > >> > >> > <configuration> >> > <!-- you can optionally include the envReplace tag to turn >> >on/off environment var replacement --> >> > <property name="nsAware" value="true" /> >> > <property name="elementNs" value="CAS" /> >> > <property name="elements" >> >value="ProductReceivedTime,ProductName,ProductId" /> >> > </configuration> >> > </extractor> >> > </metExtractors> >> > <metadata/> >> > </type> >> > >> >Thanks in advance for any help or ideas you might have! >> > >> >Valerie >> > >> > >> > >> >Valerie A. Mallder >> > >> >New Horizons Deputy Mission System Engineer The Johns Hopkins >> >University/Applied Physics Laboratory >> >11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723 >> >240-228-7846 (Office) 410-504-2233 (Blackberry) >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > -- >> > Tom Barber | Technical Director >> > >> > meteorite bi >> > T: +44 20 8133 3730 >> > W: www.meteorite.bi <http://www.meteorite.bi> | Skype: >> >meteorite.consulting >> > A: Surrey Technology Centre, Surrey Research Park, >> > Guildford, GU2 7YG, UK >> > >> > >
