Awesome job Tom, I implemented this in https://issues.apache.org/jira/browse/OODT-751
Cheers! ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: [email protected] WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -----Original Message----- From: Tom Barber <[email protected]> Date: Friday, November 14, 2014 at 12:39 PM To: <[email protected]> Subject: Re: more info for OODT-751 OPSUI Pages constantly expire > > > > Alright then, my client is going to > kick my ass for not doing any work today, but I think I've > resolved the mystery of the failing top menus, and its > crapola..... > > After doing the worlds largest Diff: > Someone removed BasePage.js in 0.7 because its empty but didn't > remove the dependency in wicket. Wicket is still throwing > serialization exceptions, but that might be because my repository > is empty or just because its loves serializing stuff it can't. > > Either way the page expiry exceptions are gone..... > > So Val, > >https://raw.githubusercontent.com/apache/oodt/92da10aa38fcdcd386d37c9e3e3e >618c9195e224/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/BasePage.js > grab that and stick it in > oodt/tomcat/webapps/opsui/WEB-INF/classes/org/apache/oodt/pcs/opsui/ > and refresh.... > > I'm off to file a jira and get my ass kicked. > > Adios. > > On 14/11/14 10:34, Tom Barber wrote: > > > > So what is happening is something it > causing Wicket to try and serialize everything, ever. I'm not a > wicket guy so I'm not particularly sure whats going on, I did > add serializable headers to everything in the structs package > but that just moved the error further up the chain. > > What I did notice though which is odd, is the menu stuff seems > to work from the homepage, but not from the top toolbar. > > Tom > > On 14/11/14 01:39, Tom Barber wrote: > > > > Hi Val, > > I tried a 0.7 OpsUI over 0.6 filemgr to see if that made a > difference but it really doesn't like that. > > I can probably have a prod about over the weekend to see if I > can figure out whats knackered. > > Don't despair, we'll get you your demo! ;) > > Tom > > > On 14/11/14 01:30, Tom Barber wrote: > > > > Okay well, I have some good news > and some bad news (and its 1:30 am so I'm off to bed in a > moment, so this is a one shot email :) ) > > I have a stock 0.7 radix install with nothing in the > filemanager and I see what you see, its always been pretty > picky and needs replacing with something better, I keep > prodding people about that, it will happen one day ;) > > For some reason Wicket has decided a file that has not > changed in 4 years, now is the worst thing on the planet and > it refuses to change the Product information into something > the browser can render. > Caused by: java.io.NotSerializableException: > org.apache.oodt.cas.filemgr.structs.ProductType > > My 0.7 install has no products in it. > > But, I have a 0.6 Radix install that a used for ApacheCon > last year, I haven't done much too it apart from put some > products into the file manager, and that works fine(mostly, > some of the OpsUI stuff is just endlessly broken). > > http://ibin.co/1h9uVtTs0XuX > http://ibin.co/1h9uge8eqLkU > http://ibin.co/1h9uowI0RaDO > > I've not changed anything in the old one, it just uses the > GenericFile type, so it does work. Just not all the time. > > Out of curiosity have you got any products ingested or are > you running on an empty repo? > > Tom > > > > On 14/11/14 00:52, Mallder, Valerie wrote: > > > Hi Tom, > >I am running with the Radix installation of version 0.7. Although, when >I tried running with version 0.6 opsui I did not do a full reinstall of >the other oodt components that come with Radix too. > >Val > > > >Sent from my iPhone. >________________________________ >From: Tom Barber <[email protected]> ><mailto:[email protected]> >Sent: Thursday, November 13, 2014 10:40:36 AM >To: [email protected] >Subject: Re: more info for OODT-751 OPSUI Pages constantly expire > >Hi Valerie, > >Is this with or without Radix? > >Tom > >On 13/11/14 13:10, Mallder, Valerie wrote: > > > For those of you who are using the pcs-opsui in an >operational environment without getting expired pages please tell me what >version you are using, the OS you are running it on, and the browser you >are using. > >I need to demonstrate "something" to my project manager to show the >usefulness of OODT in the Jedi instrument science data pipeline here at >APL. And right now, I have nothing to "show" for my last few months of >work. If someone can tell me a configuration using opsui that is working >then maybe I can try to mimic that and get something useful up and >running. So far, for me, versions 0.6, 0.7 and the current trunk are >showing only expired pages. > >Thanks very much! >Val > > > >Sent from my iPhone. >________________________________ >From: Mallder, Valerie <[email protected]> ><mailto:[email protected]> >Sent: Tuesday, November 11, 2014 2:55:12 PM >To: [email protected] >Subject: more info for OODT-751 OPSUI Pages constantly expire > >Hi Chris, > >I know you are working on this, but I wanted to let you know that I tried >to use version 0.6 and am having the same problem in version 0.6. But, I >can't guarantee that I installed version 0.6 correctly so I would like to >run this by you. Based on your email below here's what I did: > >1. Downloaded pcs-opsui-0.76.war from: >http://repo1.maven.org/maven2/org/apache/oodt/pcs-opsui/0.6/ >and saved it in a new folder named $OODT_HOME/bin/opsui > >2. Created a script called $OODT_HOME/bin/opsui/runopsui that sets the >following variables (this is output from my script): >Using OODT_BASE: >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy >Using OODT_HOME: >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy >Using OODT_TMPDIR: >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/temp >Using FILEMGR_URL: http://localhost:9000 >Using WORKFLOW_URL: http://localhost:9001 >Using RESMGR_URL: http://localhost:9002 >Using WORKFLOW_HOME: >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/workflow >Using RESMGR_HOME: >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/resmgr >Using CRAWLER_HOME: >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/crawler >Using TOMCAT_HOME: >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat >Using PCS_HOME: >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pcs >Using PGE_HOME: >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge >Using PGE_JOBS_DIR: >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/pge/jobs >Using FEI_DROP_DIR: >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/telemetr >y >Using JEDI_L0_DIR: >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/l0 >Using JEDI_L2_DIR: >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/l2 >Using ARCHIVE_DIR: >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/archive >Using BACKUP_DIR: >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/met >Using FAILURE_DIR: >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/failure >Using JEDI_PIPELINE_DIR: /homes/malldva1/working/pipeline >Using SNAPSHOT_DIR: >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/pge/jobs >/snapshot > >(Then, I followed the steps at the bottom of: >https://cwiki.apache.org/confluence/display/OODT/Quick+Start+for+PCS+OPSUI >) > >3. Downloaded >http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/webapp/META- >INF/context.xml >And saved it as $OODT_HOME/bin/opsui/pcs-opsui.xml > >4. Edited $OODT_HOME/bin/opsui/pcs-opsui.xml and changed the first line >from: ><Context path="/pcs-opsui"> >To ><Context path="/pcs-opsui" >docBase="[OODT_HOME]/bin/opsui/pcs-opsui-0.6.war"> > >5. Killed any process with "tomcat" in its name to ensure that tomcat is >not running. > >6. Executed the following command to create a symbolic link: >cd $OODT_HOME/bin/opsui/ >ln -s pcs-opsui.xml to $TOMCAT_HOME/conf/Catalina/localhost/pcs-opsui.xml > >7. Then, in my script that sets the environment variables, I added the >following command to start tomcat. >exec "$OODT_BASE"/tomcat/bin/catalina.sh start > >8. Then, I ran my new $OODT_HOME/bin/opsui/runopsui script > >9. Then, I started firefox and went to "localhost:8080/pcs-opsui" > > > >The home page comes up, and the PCS Status page comes up, but all other >pages are expired. I should note that the "radix" installation of >version 0.7 points me to localhost:8080/opsui, while step 10 of the >instructions from the quick start guide of the wiki point me to >localhost:8080/pcs-opsui. And, no matter which one I go to, the pages >expire. > >I also tried clearing out my cache and browser history and restarting >firefox and topcat, etc. and nothing seems to help. > >If I have gotten the version 0.6 up and running correctly, I thought you >would be interested to know that I am seeing the error. If I did not get >version 0.6 up and running, then I need more hints on how to do it. :) > >Thanks, >Val > > > > > > > > > >Valerie A. Mallder >New Horizons Deputy Mission System Engineer >Johns Hopkins University/Applied Physics Laboratory > > > > > -----Original Message----- >From: Mattmann, Chris A (3980) [mailto:[email protected]] >Sent: Friday, October 03, 2014 1:07 PM >To: [email protected] >Subject: Re: Success! RE: how to use MetadataBasedFileVersioner properly > >Awesome Val! :) > >I think you¹re running into this: > >https://issues.apache.org/jira/browse/OODT-751 > > >In the meanwhile, try the 0.6 OPSUI, which you can grab from here: > >http://repo1.maven.org/maven2/org/apache/oodt/pcs-opsui/0.6/ > > >Grab the WAR file and drop it into your favorite container. >Make sure you have all of these environment variables installed >*before* starting Tomcat or Jetty, etc.: > >https://cwiki.apache.org/confluence/display/OODT/Quick+Start+for+PCS+OPSUI > > >(see steps at bottom and replace 0.5 with 0.6) > >I¹m working on a fix for OODT-751, at which point RADIX will be pretty >buff. Next >steps at that point: > >1. Release 0.7 and then encourage folks to get started by using the >Vagrant build, >e.g., > >git clone https://github.com/apache/oodt cd vagrant/radix vagrant up > >2. Fix OODT-491 and remaining workflow manager issues for Wengine 3. Work >on Streaming OODT API with AMP Stack (via M. Starch et al) and release in >0.8. >4. Conquer and win. > >Thanks! > >Cheers, >Chris > >+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >+++++ >Chris Mattmann, Ph.D. >Chief Architect >Instrument Software and Science Data Systems Section (398) NASA Jet >Propulsion Laboratory Pasadena, CA 91109 USA >Office: 168-519, Mailstop: 168-527 >Email: [email protected] >WWW: http://sunset.usc.edu/~mattmann/ ><http://sunset.usc.edu/%7Emattmann/> >+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >+++++ >Adjunct Associate Professor, Computer Science Department University of >Southern California, Los Angeles, CA 90089 USA >+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >+++++ > > > > > > >-----Original Message----- >From: <Mallder>, Valerie <[email protected]> ><mailto:[email protected]> >Reply-To: "[email protected]" <mailto:[email protected]> ><[email protected]> <mailto:[email protected]> >Date: Friday, October 3, 2014 at 9:53 AM >To: "[email protected]" <mailto:[email protected]> ><[email protected]> <mailto:[email protected]> >Subject: Success! RE: how to use MetadataBasedFileVersioner properly > > > > Hi Chris, > >Yes, that indeed fixed it! Thanks so much! I now have 18 engineering >files ingested. Whoo hoo!! > >Ok, so now, I would like to see what the opsui has to say about my >ingested files. And here comes probably a really stupid question. I am >using Firefox on a Redhat Linux box, and I'm not that familiar with >Firefox and it's settings, so this might be a browser setting issue. >When I startup the opsui and select "File Catalog Browse" it shows that >I have >18 EngineeringFiles. Then, when I select File Catalog Browse from the >strip of options under the logo, I get a message saying the page has >expired. And here's the link that shows up in the address bar: >http://localhost:8080/opsui/?wicket:interface=:7:fmbrowser_link::ILinkL >ist >ener:: > >And I get the page expired message for all of the options that I select. >Any idea's on this one?? > >In the meantime though, I will start playing with an action for post >ingestion success that simply makes another copy of all these >engineering files and puts them another folder for Level 0 files. > >Thanks, >Val > > > > >Valerie A. Mallder >New Horizons Deputy Mission System Engineer Johns Hopkins >University/Applied Physics Laboratory > > > > > -----Original Message----- >From: Chris Mattmann [mailto:[email protected]] >Sent: Friday, October 03, 2014 3:03 AM >To: [email protected] >Subject: Re: how to use MetadataBasedFileVersioner properly > >You?re almost there Val! > >Unfortunately Versioners right now aren?t configurable from product >type policy (would be great to capture this in a JIRA issue, here: >https://issues.apache.org/jira/browse/OODT). If they were, it would >have picked up your <property .. declaration of filePathSpec below. >It?s been on my TODO list for a long time. > >Instead I created this: > >https://issues.apache.org/jira/browse/OODT-639 > > >So you can amend your definition below (also note you always have to >include /[Filename] at the end to get the filename you want). > > <type id="urn:oodt:EngineeringFile" name="EngineeringFile"> > <repository path="file://[OODT_HOME]/data/archive/ops/eng" ><file://[OODT_HOME]/data/archive/ops/eng>/> > <versioner >class=?org.apache.oodt.cas.filemgr.versioning.ProductTypeMetVersioner?/> > <description>The default product type for any kind of >file.</description> > <metExtractors> > <extractor > >class="org.apache.oodt.cas.filemgr.metadata.extractors.CoreMetExtractor"> > <configuration> > <!-- you can optionally include the envReplace tag to turn >on/off environment var replacement --> > <property name="nsAware" value="true" /> > <property name="elementNs" value="CAS" /> > <property name="elements" >value="ProductReceivedTime,ProductName,ProductId" /> > </configuration> > </extractor> > </metExtractors> > <metadata> > <keyval> > <key>filePathSpec</key> > <val>/[YearDir]/[DoyDir]/[Filename]</val> > </keyval> > ></metadata> > </type> > > >See if that fixes it! > >Cheers, >Chris > >------------------------ >Chris Mattmann >[email protected] > > > > >-----Original Message----- >From: "Mallder, Valerie" <[email protected]> ><mailto:[email protected]> >Reply-To: <[email protected]> <mailto:[email protected]> >Date: Thursday, October 2, 2014 at 2:56 PM >To: "[email protected]" <mailto:[email protected]> ><[email protected]> <mailto:[email protected]> >Subject: how to use MetadataBasedFileVersioner properly > > > > Hi All, > >I am trying to use the MetadataBasedFileVersioner to store my files >in a subdirectory of the data/archive folder based on the values of >two metadata elements. Has anyone does this before, and if so, can >you give me some hints. > >My files need to be organized in subdirectories by year and day of > > > > year. > > > So, lets say the file name is 'myfile', the year is >2014, and the >doy of year is 002. Then, the end result that I am looking for is to >have the final location of my file be: > > > > 'data/archive/ops/eng/2014/002/myfile. > > > However, in my log file, the INFO messages indicate >that the >generated final location reference is 'data/archive/ops/eng/myfile'. >And it doesn't include the year and day of year at all. And the >incorrect location leads to other errors. So I want to solve this >one first. Has anyone tried to do something like this before? > >Here is what I have done so far: > >Added two new elements to the .met file 'YearDir' and 'DoyDir'. >Added these new elements to the elements.xml file. >Added these new elements to the product in the >product-type-element-map.xml file. >And, I have made changes to the product-types.xml file, but here's >where I am not sure I've done this properly. > >Here's what product type definition looks like: > <type id="urn:oodt:EngineeringFile" name="EngineeringFile"> > <repository path="file://[OODT_HOME]/data/archive/ops/eng" ><file://[OODT_HOME]/data/archive/ops/eng>/> > <versioner >class="org.apache.oodt.cas.filemgr.versioning.MetadataBasedFileVersio >ner >"> > <property name="filePathSpec" value="/[YearDir]/[DoyDir]/" /> > </versioner> > <description>The default product type for any kind of >file.</description> > <metExtractors> > <extractor >class="org.apache.oodt.cas.filemgr.metadata.extractors.CoreMetExtractor" > <configuration> > <!-- you can optionally include the envReplace tag to turn >on/off environment var replacement --> > <property name="nsAware" value="true" /> > <property name="elementNs" value="CAS" /> > <property name="elements" >value="ProductReceivedTime,ProductName,ProductId" /> > </configuration> > </extractor> > </metExtractors> > <metadata/> > </type> > >Thanks in advance for any help or ideas you might have! > >Valerie > > > >Valerie A. Mallder > >New Horizons Deputy Mission System Engineer The Johns Hopkins >University/Applied Physics Laboratory >11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723 >240-228-7846 (Office) 410-504-2233 (Blackberry) > > > > > > > > > > > > > -- >*Tom Barber* | Technical Director > >meteorite bi >*T:* +44 20 8133 3730 >*W:* www.meteorite.bi <http://www.meteorite.bi><http://www.meteorite.bi> ><http://www.meteorite.bi> | *Skype:* meteorite.consulting >*A:* Surrey Technology Centre, Surrey Research Park, Guildford, GU2 7YG, >UK > > > > > > > -- > Tom Barber | Technical Director > > meteorite bi > T: +44 20 8133 3730 > W: www.meteorite.bi <http://www.meteorite.bi> | Skype: > meteorite.consulting > A: Surrey Technology Centre, Surrey Research Park, > Guildford, GU2 7YG, UK > > > > > -- > Tom Barber | Technical Director > > meteorite bi > T: +44 20 8133 3730 > W: www.meteorite.bi <http://www.meteorite.bi> | Skype: > meteorite.consulting > A: Surrey Technology Centre, Surrey Research Park, > Guildford, GU2 7YG, UK > > > > > -- > Tom Barber | Technical Director > > meteorite bi > T: +44 20 8133 3730 > W: www.meteorite.bi <http://www.meteorite.bi> | Skype: > meteorite.consulting > A: Surrey Technology Centre, Surrey Research Park, > Guildford, GU2 7YG, UK > > > > > -- > Tom Barber | Technical Director > > meteorite bi > T: +44 20 8133 3730 > W: www.meteorite.bi <http://www.meteorite.bi> | Skype: >meteorite.consulting > A: Surrey Technology Centre, Surrey Research Park, > Guildford, GU2 7YG, UK > >
