Hi Tom, All of this is very very interesting, and I do appreciate your help. I do have files ingested into the catalog but you've given me some new ideas on things I should be checking each time I run my workflow. I will not be working over the weekend, but I may be able to give you an update on my progress late Monday. Thanks for your help!!
Val Valerie A. Mallder New Horizons Deputy Mission System Engineer Johns Hopkins University/Applied Physics Laboratory > -----Original Message----- > From: Tom Barber [mailto:[email protected]] > Sent: Thursday, November 13, 2014 8:39 PM > To: [email protected] > Subject: Re: more info for OODT-751 OPSUI Pages constantly expire > > Hi Val, > > I tried a 0.7 OpsUI over 0.6 filemgr to see if that made a difference but it > really > doesn't like that. > > I can probably have a prod about over the weekend to see if I can figure out > whats > knackered. > > Don't despair, we'll get you your demo! ;) > > Tom > > > On 14/11/14 01:30, Tom Barber wrote: > > Okay well, I have some good news and some bad news (and its 1:30 am so > > I'm off to bed in a moment, so this is a one shot email :) ) > > > > I have a stock 0.7 radix install with nothing in the filemanager and I > > see what you see, its always been pretty picky and needs replacing > > with something better, I keep prodding people about that, it will > > happen one day ;) > > > > For some reason Wicket has decided a file that has not changed in 4 > > years, now is the worst thing on the planet and it refuses to change > > the Product information into something the browser can render. > > Caused by: java.io.NotSerializableException: > > org.apache.oodt.cas.filemgr.structs.ProductType > > > > My 0.7 install has no products in it. > > > > But, I have a 0.6 Radix install that a used for ApacheCon last year, I > > haven't done much too it apart from put some products into the file > > manager, and that works fine(mostly, some of the OpsUI stuff is just > > endlessly broken). > > > > http://ibin.co/1h9uVtTs0XuX > > http://ibin.co/1h9uge8eqLkU > > http://ibin.co/1h9uowI0RaDO > > > > I've not changed anything in the old one, it just uses the GenericFile > > type, so it does work. Just not all the time. > > > > Out of curiosity have you got any products ingested or are you running > > on an empty repo? > > > > Tom > > > > > > > > On 14/11/14 00:52, Mallder, Valerie wrote: > >> Hi Tom, > >> > >> I am running with the Radix installation of version 0.7. Although, when I > >> tried > running with version 0.6 opsui I did not do a full reinstall of the other oodt > components that come with Radix too. > >> > >> Val > >> > >> > >> > >> Sent from my iPhone. > >> ________________________________ > >> From: Tom Barber<[email protected]> > >> Sent: Thursday, November 13, 2014 10:40:36 AM To:[email protected] > >> Subject: Re: more info for OODT-751 OPSUI Pages constantly expire > >> > >> Hi Valerie, > >> > >> Is this with or without Radix? > >> > >> Tom > >> > >> On 13/11/14 13:10, Mallder, Valerie wrote: > >>> For those of you who are using the pcs-opsui in an operational environment > without getting expired pages please tell me what version you are using, the > OS > you are running it on, and the browser you are using. > >>> > >>> I need to demonstrate "something" to my project manager to show the > usefulness of OODT in the Jedi instrument science data pipeline here at APL. > And > right now, I have nothing to "show" for my last few months of work. If someone > can tell me a configuration using opsui that is working then maybe I can try > to > mimic that and get something useful up and running. So far, for me, versions > 0.6, > 0.7 and the current trunk are showing only expired pages. > >>> > >>> Thanks very much! > >>> Val > >>> > >>> > >>> > >>> Sent from my iPhone. > >>> ________________________________ > >>> From: Mallder, Valerie<[email protected]> > >>> Sent: Tuesday, November 11, 2014 2:55:12 PM To:[email protected] > >>> Subject: more info for OODT-751 OPSUI Pages constantly expire > >>> > >>> Hi Chris, > >>> > >>> I know you are working on this, but I wanted to let you know that I tried > >>> to use > version 0.6 and am having the same problem in version 0.6. But, I can't > guarantee > that I installed version 0.6 correctly so I would like to run this by you. > Based on > your email below here's what I did: > >>> > >>> 1. Downloaded pcs-opsui-0.76.war from: > >>> http://repo1.maven.org/maven2/org/apache/oodt/pcs-opsui/0.6/ > >>> and saved it in a new folder named $OODT_HOME/bin/opsui > >>> > >>> 2. Created a script called $OODT_HOME/bin/opsui/runopsui that sets the > following variables (this is output from my script): > >>> Using OODT_BASE: /homes/malldva1/project/jedi/users/jedi- > pipeline/oodt-deploy > >>> Using OODT_HOME: /homes/malldva1/project/jedi/users/jedi- > pipeline/oodt-deploy > >>> Using OODT_TMPDIR: /homes/malldva1/project/jedi/users/jedi- > pipeline/oodt-deploy/temp > >>> Using FILEMGR_URL:http://localhost:9000 Using > >>> WORKFLOW_URL:http://localhost:9001 > >>> Using RESMGR_URL:http://localhost:9002 > >>> Using WORKFLOW_HOME: /homes/malldva1/project/jedi/users/jedi- > pipeline/oodt-deploy/workflow > >>> Using RESMGR_HOME: /homes/malldva1/project/jedi/users/jedi- > pipeline/oodt-deploy/resmgr > >>> Using CRAWLER_HOME: /homes/malldva1/project/jedi/users/jedi- > pipeline/oodt-deploy/crawler > >>> Using TOMCAT_HOME: /homes/malldva1/project/jedi/users/jedi- > pipeline/oodt-deploy/tomcat > >>> Using PCS_HOME: /homes/malldva1/project/jedi/users/jedi- > pipeline/oodt-deploy/pcs > >>> Using PGE_HOME: /homes/malldva1/project/jedi/users/jedi- > pipeline/oodt-deploy/pge > >>> Using PGE_JOBS_DIR: /homes/malldva1/project/jedi/users/jedi- > pipeline/oodt-deploy/data/pge/jobs > >>> Using FEI_DROP_DIR: /homes/malldva1/project/jedi/users/jedi- > pipeline/oodt-deploy/data/telemetry > >>> Using JEDI_L0_DIR: /homes/malldva1/project/jedi/users/jedi- > pipeline/oodt-deploy/data/l0 > >>> Using JEDI_L2_DIR: /homes/malldva1/project/jedi/users/jedi- > pipeline/oodt-deploy/data/l2 > >>> Using ARCHIVE_DIR: /homes/malldva1/project/jedi/users/jedi- > pipeline/oodt-deploy/data/archive > >>> Using BACKUP_DIR: /homes/malldva1/project/jedi/users/jedi- > pipeline/oodt-deploy/data/met > >>> Using FAILURE_DIR: /homes/malldva1/project/jedi/users/jedi- > pipeline/oodt-deploy/data/failure > >>> Using JEDI_PIPELINE_DIR: /homes/malldva1/working/pipeline > >>> Using SNAPSHOT_DIR: /homes/malldva1/project/jedi/users/jedi- > pipeline/oodt-deploy/data/pge/jobs/snapshot > >>> > >>> (Then, I followed the steps at the bottom > >>> of:https://cwiki.apache.org/confluence/display/OODT/Quick+Start+for+ > >>> PCS+OPSUI) > >>> > >>> 3. > >>> Downloadedhttp://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/m > >>> ain/webapp/META-INF/context.xml And saved it as > >>> $OODT_HOME/bin/opsui/pcs-opsui.xml > >>> > >>> 4. Edited $OODT_HOME/bin/opsui/pcs-opsui.xml and changed the first line > from: > >>> <Context path="/pcs-opsui"> > >>> To > >>> <Context path="/pcs-opsui" > >>> docBase="[OODT_HOME]/bin/opsui/pcs-opsui-0.6.war"> > >>> > >>> 5. Killed any process with "tomcat" in its name to ensure that tomcat is > >>> not > running. > >>> > >>> 6. Executed the following command to create a symbolic link: > >>> cd $OODT_HOME/bin/opsui/ > >>> ln -s pcs-opsui.xml to > >>> $TOMCAT_HOME/conf/Catalina/localhost/pcs-opsui.xml > >>> > >>> 7. Then, in my script that sets the environment variables, I added the > >>> following > command to start tomcat. > >>> exec "$OODT_BASE"/tomcat/bin/catalina.sh start > >>> > >>> 8. Then, I ran my new $OODT_HOME/bin/opsui/runopsui script > >>> > >>> 9. Then, I started firefox and went to "localhost:8080/pcs-opsui" > >>> > >>> > >>> > >>> The home page comes up, and the PCS Status page comes up, but all other > pages are expired. I should note that the "radix" installation of version > 0.7 points > me to localhost:8080/opsui, while step 10 of the instructions from the quick > start > guide of the wiki point me to localhost:8080/pcs-opsui. And, no matter which > one I > go to, the pages expire. > >>> > >>> I also tried clearing out my cache and browser history and restarting > >>> firefox > and topcat, etc. and nothing seems to help. > >>> > >>> If I have gotten the version 0.6 up and running correctly, I thought > >>> you would be interested to know that I am seeing the error. If I did > >>> not get version 0.6 up and running, then I need more hints on how to > >>> do it. :) > >>> > >>> Thanks, > >>> Val > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> Valerie A. Mallder > >>> New Horizons Deputy Mission System Engineer Johns Hopkins > >>> University/Applied Physics Laboratory > >>> > >>> > >>>> -----Original Message----- > >>>> From: Mattmann, Chris A (3980) > >>>> [mailto:[email protected]] > >>>> Sent: Friday, October 03, 2014 1:07 PM To:[email protected] > >>>> Subject: Re: Success! RE: how to use MetadataBasedFileVersioner > >>>> properly > >>>> > >>>> Awesome Val! :) > >>>> > >>>> I think you¹re running into this: > >>>> > >>>> https://issues.apache.org/jira/browse/OODT-751 > >>>> > >>>> > >>>> In the meanwhile, try the 0.6 OPSUI, which you can grab from here: > >>>> > >>>> http://repo1.maven.org/maven2/org/apache/oodt/pcs-opsui/0.6/ > >>>> > >>>> > >>>> Grab the WAR file and drop it into your favorite container. > >>>> Make sure you have all of these environment variables installed > >>>> *before* starting Tomcat or Jetty, etc.: > >>>> > >>>> https://cwiki.apache.org/confluence/display/OODT/Quick+Start+for+PC > >>>> S+OPSUI > >>>> > >>>> > >>>> (see steps at bottom and replace 0.5 with 0.6) > >>>> > >>>> I¹m working on a fix for OODT-751, at which point RADIX will be > >>>> pretty buff. Next steps at that point: > >>>> > >>>> 1. Release 0.7 and then encourage folks to get started by using the > >>>> Vagrant build, e.g., > >>>> > >>>> git clonehttps://github.com/apache/oodt cd vagrant/radix vagrant > >>>> up > >>>> > >>>> 2. Fix OODT-491 and remaining workflow manager issues for Wengine > >>>> 3. Work on Streaming OODT API with AMP Stack (via M. Starch et al) and > release in 0.8. > >>>> 4. Conquer and win. > >>>> > >>>> Thanks! > >>>> > >>>> Cheers, > >>>> Chris > >>>> > >>>> > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >>>> +++++ > >>>> Chris Mattmann, Ph.D. > >>>> Chief Architect > >>>> Instrument Software and Science Data Systems Section (398) NASA Jet > >>>> Propulsion Laboratory Pasadena, CA 91109 USA > >>>> Office: 168-519, Mailstop: 168-527 > >>>> Email:[email protected] > >>>> WWW:http://sunset.usc.edu/~mattmann/ > >>>> > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >>>> +++++ > >>>> Adjunct Associate Professor, Computer Science Department University > >>>> of Southern California, Los Angeles, CA 90089 USA > >>>> > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >>>> +++++ > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> -----Original Message----- > >>>> From: <Mallder>, Valerie<[email protected]> > >>>> Reply-To:"[email protected]" <[email protected]> > >>>> Date: Friday, October 3, 2014 at 9:53 AM To:"[email protected]" > >>>> <[email protected]> > >>>> Subject: Success! RE: how to use MetadataBasedFileVersioner > >>>> properly > >>>> > >>>>> Hi Chris, > >>>>> > >>>>> Yes, that indeed fixed it! Thanks so much! I now have 18 > >>>>> engineering files ingested. Whoo hoo!! > >>>>> > >>>>> Ok, so now, I would like to see what the opsui has to say about my > >>>>> ingested files. And here comes probably a really stupid question. > >>>>> I am using Firefox on a Redhat Linux box, and I'm not that > >>>>> familiar with Firefox and it's settings, so this might be a browser > >>>>> setting > issue. > >>>>> When I startup the opsui and select "File Catalog Browse" it shows > >>>>> that I have > >>>>> 18 EngineeringFiles. Then, when I select File Catalog Browse from > >>>>> the strip of options under the logo, I get a message saying the > >>>>> page has expired. And here's the link that shows up in the address bar: > >>>>> http://localhost:8080/opsui/?wicket:interface=:7:fmbrowser_link::I > >>>>> LinkL > >>>>> ist > >>>>> ener:: > >>>>> > >>>>> And I get the page expired message for all of the options that I select. > >>>>> Any idea's on this one?? > >>>>> > >>>>> In the meantime though, I will start playing with an action for > >>>>> post ingestion success that simply makes another copy of all these > >>>>> engineering files and puts them another folder for Level 0 files. > >>>>> > >>>>> Thanks, > >>>>> Val > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> Valerie A. Mallder > >>>>> New Horizons Deputy Mission System Engineer Johns Hopkins > >>>>> University/Applied Physics Laboratory > >>>>> > >>>>> > >>>>>> -----Original Message----- > >>>>>> From: Chris Mattmann [mailto:[email protected]] > >>>>>> Sent: Friday, October 03, 2014 3:03 AM To:[email protected] > >>>>>> Subject: Re: how to use MetadataBasedFileVersioner properly > >>>>>> > >>>>>> You?re almost there Val! > >>>>>> > >>>>>> Unfortunately Versioners right now aren?t configurable from > >>>>>> product type policy (would be great to capture this in a JIRA issue, > >>>>>> here: > >>>>>> https://issues.apache.org/jira/browse/OODT). If they were, it > >>>>>> would have picked up your <property .. declaration of filePathSpec > below. > >>>>>> It?s been on my TODO list for a long time. > >>>>>> > >>>>>> Instead I created this: > >>>>>> > >>>>>> https://issues.apache.org/jira/browse/OODT-639 > >>>>>> > >>>>>> > >>>>>> So you can amend your definition below (also note you always have > >>>>>> to include /[Filename] at the end to get the filename you want). > >>>>>> > >>>>>> <type id="urn:oodt:EngineeringFile" name="EngineeringFile"> > >>>>>> <repository path="file://[OODT_HOME]/data/archive/ops/eng"/> > >>>>>> <versioner > >>>>>> > class=?org.apache.oodt.cas.filemgr.versioning.ProductTypeMetVersioner?/> > >>>>>> <description>The default product type for any kind of > >>>>>> file.</description> > >>>>>> <metExtractors> > >>>>>> <extractor > >>>>>> > >>>>>> > class="org.apache.oodt.cas.filemgr.metadata.extractors.CoreMetExtractor"> > >>>>>> <configuration> > >>>>>> <!-- you can optionally include the envReplace tag to > >>>>>> turn on/off environment var replacement --> > >>>>>> <property name="nsAware" value="true" /> > >>>>>> <property name="elementNs" value="CAS" /> > >>>>>> <property name="elements" > >>>>>> value="ProductReceivedTime,ProductName,ProductId" /> > >>>>>> </configuration> > >>>>>> </extractor> > >>>>>> </metExtractors> > >>>>>> <metadata> > >>>>>> <keyval> > >>>>>> <key>filePathSpec</key> > >>>>>> <val>/[YearDir]/[DoyDir]/[Filename]</val> > >>>>>> </keyval> > >>>>>> > >>>>>> </metadata> > >>>>>> </type> > >>>>>> > >>>>>> > >>>>>> See if that fixes it! > >>>>>> > >>>>>> Cheers, > >>>>>> Chris > >>>>>> > >>>>>> ------------------------ > >>>>>> Chris Mattmann > >>>>>> [email protected] > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> -----Original Message----- > >>>>>> From: "Mallder, Valerie"<[email protected]> > >>>>>> Reply-To:<[email protected]> > >>>>>> Date: Thursday, October 2, 2014 at 2:56 PM > >>>>>> To:"[email protected]" <[email protected]> > >>>>>> Subject: how to use MetadataBasedFileVersioner properly > >>>>>> > >>>>>>> Hi All, > >>>>>>> > >>>>>>> I am trying to use the MetadataBasedFileVersioner to store my > >>>>>>> files in a subdirectory of the data/archive folder based on the > >>>>>>> values of two metadata elements. Has anyone does this before, > >>>>>>> and if so, can you give me some hints. > >>>>>>> > >>>>>>> My files need to be organized in subdirectories by year and day > >>>>>>> of > >>>>>> year. > >>>>>>> So, lets say the file name is 'myfile', the year is 2014, and > >>>>>>> the doy of year is 002. Then, the end result that I am looking > >>>>>>> for is to have the final location of my file be: > >>>>>> 'data/archive/ops/eng/2014/002/myfile. > >>>>>>> However, in my log file, the INFO messages indicate that the > >>>>>>> generated final location reference is 'data/archive/ops/eng/myfile'. > >>>>>>> And it doesn't include the year and day of year at all. And the > >>>>>>> incorrect location leads to other errors. So I want to solve > >>>>>>> this one first. Has anyone tried to do something like this before? > >>>>>>> > >>>>>>> Here is what I have done so far: > >>>>>>> > >>>>>>> Added two new elements to the .met file 'YearDir' and 'DoyDir'. > >>>>>>> Added these new elements to the elements.xml file. > >>>>>>> Added these new elements to the product in the > >>>>>>> product-type-element-map.xml file. > >>>>>>> And, I have made changes to the product-types.xml file, but > >>>>>>> here's where I am not sure I've done this properly. > >>>>>>> > >>>>>>> Here's what product type definition looks like: > >>>>>>> <type id="urn:oodt:EngineeringFile" name="EngineeringFile"> > >>>>>>> <repository path="file://[OODT_HOME]/data/archive/ops/eng"/> > >>>>>>> <versioner > >>>>>>> class="org.apache.oodt.cas.filemgr.versioning.MetadataBasedFileV > >>>>>>> ersio > >>>>>>> ner > >>>>>>> "> > >>>>>>> <property name="filePathSpec" value="/[YearDir]/[DoyDir]/" > >>>>>>> /> > >>>>>>> </versioner> > >>>>>>> <description>The default product type for any kind of > >>>>>>> file.</description> > >>>>>>> <metExtractors> > >>>>>>> <extractor > >>>>>>> > class="org.apache.oodt.cas.filemgr.metadata.extractors.CoreMetExtractor" > >>>>>>> <configuration> > >>>>>>> <!-- you can optionally include the envReplace tag to > >>>>>>> turn on/off environment var replacement --> > >>>>>>> <property name="nsAware" value="true" /> > >>>>>>> <property name="elementNs" value="CAS" /> > >>>>>>> <property name="elements" > >>>>>>> value="ProductReceivedTime,ProductName,ProductId" /> > >>>>>>> </configuration> > >>>>>>> </extractor> > >>>>>>> </metExtractors> > >>>>>>> <metadata/> > >>>>>>> </type> > >>>>>>> > >>>>>>> Thanks in advance for any help or ideas you might have! > >>>>>>> > >>>>>>> Valerie > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Valerie A. Mallder > >>>>>>> > >>>>>>> New Horizons Deputy Mission System Engineer The Johns Hopkins > >>>>>>> University/Applied Physics Laboratory > >>>>>>> 11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723 > >>>>>>> 240-228-7846 (Office) 410-504-2233 (Blackberry) > >>>>>>> > >> -- > >> *Tom Barber* | Technical Director > >> > >> meteorite bi > >> *T:* +44 20 8133 3730 > >> *W:*www.meteorite.bi<http://www.meteorite.bi> | *Skype:* > >> meteorite.consulting > >> *A:* Surrey Technology Centre, Surrey Research Park, Guildford, GU2 > >> 7YG, UK > >> > > > > > > -- > > *Tom Barber* | Technical Director > > > > meteorite bi > > *T:* +44 20 8133 3730 > > *W:* www.meteorite.bi | *Skype:* meteorite.consulting > > *A:* Surrey Technology Centre, Surrey Research Park, Guildford, GU2 > > 7YG, UK > > > -- > *Tom Barber* | Technical Director > > meteorite bi > *T:* +44 20 8133 3730 > *W:* www.meteorite.bi | *Skype:* meteorite.consulting > *A:* Surrey Technology Centre, Surrey Research Park, Guildford, GU2 7YG, UK
