Re: more info for OODT-751 OPSUI Pages constantly expire

Fri, 21 Nov 2014 12:02:43 -0800 

Half days! Vacations! Bah some people get all the nice stuff.

Have fun :)

On 21/11/14 19:17, Mallder, Valerie wrote:

Ok, I lied. I just remembered that today is the last day I can get my drivers 
license renewed so I have to leave work early to take care of that and so I 
won't be have time to write down more details of my progress. But, when I get 
back from vacation on Dec 8, I will probably be bugging folks again on what I 
need to do in order to get the operator interface up and running.

Thanks everyone for all of your help! And I hope you have a great Thanksgiving 
next week!

Val





Valerie A. Mallder
New Horizons Deputy Mission System Engineer
Johns Hopkins University/Applied Physics Laboratory



-----Original Message-----
From: Mallder, Valerie [mailto:[email protected]]
Sent: Friday, November 21, 2014 10:43 AM
To: [email protected]
Subject: RE: more info for OODT-751 OPSUI Pages constantly expire

Oh wait, maybe I misunderstood. I didn't see an actual "fix", just more 
discussion
that sparked an idea for me to try something which I have been trying to "try" 
for
several days now. I was planning to give an update of where I am today. I am
going on vacation for two weeks starting Monday, so I was definitely planning to
give y'all an update today before I leave.  The good news is, I do have all 
stages
of the current JEDI pipeline at APL implemented in OODT (With a couple of
caveats, of course, but ..). I have raw telemetry from JPL's File Exchange 
Interface
(FEI) going in, and daily summary plots of the telemetry data coming out - and 
that
in itself deserves a huge YAY!!. I haven't haven't had to write any 
project-specific
java code yet. The only java code change I made was to make the fix identified
OODT-757 that prevented an autodetect crawler from running. (I think this fix is
already planned for 0.8 and may have been implemented already in the trunk
already.)   I have just one workflow and 4 PGE tasks, and I glued together 
several
existing 5 java applications by rewriting some perl scripts and adding a few new
perl scripts. I have verified that all 4 of my product types (EngineeringFile,
ScienceFile, EDRFile, and SummaryPlot) that are generated by the pipeline are in
the catalog (using the query-tool), and I verified that the the summary plots 
from
my pipeline match the summary plots that were generated by the existing JEDI
pipeline.  All I need now is an operator interface to demo this to the folks 
here at
APL.   I have kept a detailed list of lessons learned which I plan to share 
(probably
after I get back from vacation though).

.... more later today

Val



Valerie A. Mallder
New Horizons Deputy Mission System Engineer Johns Hopkins University/Applied
Physics Laboratory


-----Original Message-----
From: Mattmann, Chris A (3980) [mailto:[email protected]]
Sent: Friday, November 21, 2014 10:13 AM
To: Tom Barber; [email protected]
Subject: Re: more info for OODT-751 OPSUI Pages constantly expire

You did! Did it work? Can you pass along the thread so I can read it
sorry I'm trying to catch up..



+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

+++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398) NASA Jet
Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: [email protected]
WWW:  http://sunset.usc.edu/~mattmann/


+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

+++++
Adjunct Associate Professor, Computer Science Department University of
Southern California, Los Angeles, CA 90089 USA


+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

+++++






-----Original Message-----
From: Tom Barber <[email protected]>
Date: Friday, November 21, 2014 at 4:06 PM
To: <[email protected]>
Subject: Re: more info for OODT-751 OPSUI Pages constantly expire




    Oi Chris, I sent Val a fix! :P

      On 21/11/14 14:46, Mattmann, Chris A (3980) wrote:


      Hi Val,

I haven't forgot about this - just haven't had a chance to investigate.
Hope to have some time in the next week.

Cheers,
Chris


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++

Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398) NASA Jet
Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: [email protected]
WWW:  http://sunset.usc.edu/~mattmann/

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++

Adjunct Associate Professor, Computer Science Department University
of Southern California, Los Angeles, CA 90089 USA

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++






-----Original Message-----
From: <Mallder>, Valerie <[email protected]>
<mailto:[email protected]>
Reply-To: "[email protected]" <mailto:[email protected]>
<[email protected]> <mailto:[email protected]>
Date: Thursday, November 13, 2014 at 2:10 PM
To: "[email protected]" <mailto:[email protected]>
<[email protected]> <mailto:[email protected]>
Subject: RE: more info for OODT-751 OPSUI Pages constantly expire



        For those of you who are using the pcs-opsui in an
operational environment without getting expired pages please tell me
what version you are using, the OS you are running it on, and the browser you

are using.

I need to demonstrate "something" to my project manager to show the
usefulness of OODT in the Jedi instrument science data pipeline here
at APL. And right now, I have nothing to "show" for my last few
months of work. If someone can tell me a configuration using opsui
that is working then maybe I can try to mimic that and get something
useful up and running. So far, for me, versions 0.6, 0.7 and the
current trunk are showing only expired pages.

Thanks very much!
Val



Sent from my iPhone.
________________________________
From: Mallder, Valerie <[email protected]>
<mailto:[email protected]>
Sent: Tuesday, November 11, 2014 2:55:12 PM
To: [email protected]
Subject: more info for OODT-751 OPSUI Pages constantly expire

Hi Chris,

I know you are working on this, but I wanted to let you know that I
tried to use version 0.6 and am having the same problem in version 0.6.
But, I can't guarantee that I installed version 0.6 correctly so I
would like to run this by you.  Based on your email below here's what I did:

1. Downloaded pcs-opsui-0.76.war from:
http://repo1.maven.org/maven2/org/apache/oodt/pcs-opsui/0.6/
and saved it in a new folder named $OODT_HOME/bin/opsui

2. Created a script called $OODT_HOME/bin/opsui/runopsui that sets
the following variables (this is output from my script):
Using OODT_BASE:
/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy
Using OODT_HOME:
/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy
Using OODT_TMPDIR:
/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/temp
Using FILEMGR_URL:       http://localhost:9000
Using WORKFLOW_URL:      http://localhost:9001
Using RESMGR_URL:        http://localhost:9002
Using WORKFLOW_HOME:
/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/workflow
Using RESMGR_HOME:
/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/resmgr
Using CRAWLER_HOME:
/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/crawler
Using TOMCAT_HOME:
/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat
Using PCS_HOME:
/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pcs
Using PGE_HOME:
/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge
Using PGE_JOBS_DIR:
/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/pge
/j
obs
Using FEI_DROP_DIR:
/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/tel
em
etr
y
Using JEDI_L0_DIR:
/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/l0
Using JEDI_L2_DIR:
/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/l2
Using ARCHIVE_DIR:
/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/arc
hi
ve
Using BACKUP_DIR:
/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/met
Using FAILURE_DIR:
/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/fai
lu re Using JEDI_PIPELINE_DIR: /homes/malldva1/working/pipeline Using
SNAPSHOT_DIR:
/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/pge
/j
obs
/snapshot

(Then, I followed the steps at the bottom of:
https://cwiki.apache.org/confluence/display/OODT/Quick+Start+for+PCS+
OP
SUI
)

3. Downloaded
http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/webapp/
ME
TA-
INF/context.xml
And saved it as $OODT_HOME/bin/opsui/pcs-opsui.xml

4. Edited $OODT_HOME/bin/opsui/pcs-opsui.xml and changed the first
line
from:
<Context path="/pcs-opsui">
To
<Context path="/pcs-opsui"
docBase="[OODT_HOME]/bin/opsui/pcs-opsui-0.6.war">

5.  Killed any process with "tomcat" in its name to ensure that
tomcat is not running.

6. Executed the following command to create a symbolic link:
cd $OODT_HOME/bin/opsui/
ln -s pcs-opsui.xml to
$TOMCAT_HOME/conf/Catalina/localhost/pcs-opsui.xml

7. Then, in my script that sets the environment variables, I added
the following command to start tomcat.
exec "$OODT_BASE"/tomcat/bin/catalina.sh start

8. Then, I ran my new $OODT_HOME/bin/opsui/runopsui script

9. Then, I started firefox  and went to "localhost:8080/pcs-opsui"



The home page comes up, and the PCS Status page comes up, but all
other pages are expired.  I should note that the "radix" installation
of version 0.7 points me to localhost:8080/opsui, while step 10 of
the instructions from the quick start guide of the wiki point me to
localhost:8080/pcs-opsui. And, no matter which one I go to, the pages
expire.

I also tried clearing out my cache and browser history and restarting
firefox and topcat, etc. and nothing seems to help.

If I have gotten the version 0.6 up and running correctly, I thought
you would be interested to know that I am seeing the error. If I did
not get version 0.6 up and running, then I need more hints on how to
do it.  :)

Thanks,
Val









Valerie A. Mallder
New Horizons Deputy Mission System Engineer Johns Hopkins
University/Applied Physics Laboratory




          -----Original Message-----
From: Mattmann, Chris A (3980) [mailto:[email protected]]
Sent: Friday, October 03, 2014 1:07 PM
To: [email protected]
Subject: Re: Success! RE: how to use MetadataBasedFileVersioner
properly

Awesome Val! :)

I think you¹re running into this:

https://issues.apache.org/jira/browse/OODT-751


In the meanwhile, try the 0.6 OPSUI, which you can grab from here:

http://repo1.maven.org/maven2/org/apache/oodt/pcs-opsui/0.6/


Grab the WAR file and drop it into your favorite container.
Make sure you have all of these environment variables installed
*before* starting Tomcat or Jetty, etc.:


https://cwiki.apache.org/confluence/display/OODT/Quick+Start+for+PCS+
OP
SU
I


(see steps at bottom and replace 0.5 with 0.6)

I¹m working on a fix for OODT-751, at which point RADIX will be
pretty buff. Next steps at that point:

1. Release 0.7 and then encourage folks to get started by using the
Vagrant build, e.g.,

git clone https://github.com/apache/oodt cd vagrant/radix vagrant up

2. Fix OODT-491 and remaining workflow manager issues for Wengine 3.
Work
on Streaming OODT API with AMP Stack (via M. Starch et al) and
release in 0.8.
4. Conquer and win.

Thanks!

Cheers,
Chris


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+

+++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398) NASA Jet
Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: [email protected]
WWW:  http://sunset.usc.edu/~mattmann/

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+

+++++
Adjunct Associate Professor, Computer Science Department University
of Southern California, Los Angeles, CA 90089 USA

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+

+++++






-----Original Message-----
From: <Mallder>, Valerie <[email protected]>
<mailto:[email protected]>
Reply-To: "[email protected]" <mailto:[email protected]>
<[email protected]> <mailto:[email protected]>
Date: Friday, October 3, 2014 at 9:53 AM
To: "[email protected]" <mailto:[email protected]>
<[email protected]> <mailto:[email protected]>
Subject: Success!  RE: how to use MetadataBasedFileVersioner properly



            Hi Chris,

Yes, that indeed fixed it!  Thanks so much!  I now have 18
engineering files ingested. Whoo hoo!!

Ok, so now, I would like to see what the opsui has to say about my
ingested files. And here comes probably a really stupid question.  I
am using Firefox on a Redhat Linux box, and I'm not that familiar
with Firefox and it's settings, so this might be a browser setting issue.
When I startup the opsui and select "File Catalog Browse" it shows
that I have
18 EngineeringFiles. Then, when I select File Catalog Browse from the
strip of options under the logo, I get a message saying the page has
expired. And here's the link that shows up in the address bar:
http://localhost:8080/opsui/?wicket:interface=:7:fmbrowser_link::ILin
kL
ist
ener::

And I get the page expired message for all of the options that I



          select.


            Any idea's on this one??

In the meantime though, I will start playing with an action for post
ingestion success that simply makes another copy of all these
engineering files and puts them another folder for Level 0 files.

Thanks,
Val




Valerie A. Mallder
New Horizons Deputy Mission System Engineer Johns Hopkins
University/Applied Physics Laboratory




              -----Original Message-----
From: Chris Mattmann [mailto:[email protected]]
Sent: Friday, October 03, 2014 3:03 AM
To: [email protected]
Subject: Re: how to use MetadataBasedFileVersioner properly

You?re almost there Val!

Unfortunately Versioners right now aren?t configurable from product
type policy  (would be great to capture this in a JIRA issue, here:
https://issues.apache.org/jira/browse/OODT). If they were, it would
have picked up  your <property .. declaration of filePathSpec below.
It?s been on my TODO list for a long time.

Instead I created this:

https://issues.apache.org/jira/browse/OODT-639


So you can amend your definition below (also note you always have to
include  /[Filename] at the end to get the filename you want).

  <type id="urn:oodt:EngineeringFile" name="EngineeringFile">
    <repository path="file://[OODT_HOME]/data/archive/ops/eng"
<file://[OODT_HOME]/data/archive/ops/eng>/>
    <versioner







class=?org.apache.oodt.cas.filemgr.versioning.ProductTypeMetVersioner
?/


                  <description>The default product type for any kind
of file.</description>
    <metExtractors>
      <extractor










class="org.apache.oodt.cas.filemgr.metadata.extractors.CoreMetExtract
or
">
        <configuration>
          <!-- you can optionally include the envReplace tag to turn
on/off  environment var replacement -->
          <property name="nsAware" value="true" />
          <property name="elementNs" value="CAS" />
          <property name="elements"
value="ProductReceivedTime,ProductName,ProductId" />
        </configuration>
      </extractor>
    </metExtractors>
    <metadata>
<keyval>
    <key>filePathSpec</key>
    <val>/[YearDir]/[DoyDir]/[Filename]</val>
  </keyval>

</metadata>
  </type>


See if that fixes it!

Cheers,
Chris

------------------------
Chris Mattmann
[email protected]




-----Original Message-----
From: "Mallder, Valerie" <[email protected]>
<mailto:[email protected]>
Reply-To: <[email protected]> <mailto:[email protected]>
Date: Thursday, October 2, 2014 at 2:56 PM
To: "[email protected]" <mailto:[email protected]>
<[email protected]> <mailto:[email protected]>
Subject: how to use MetadataBasedFileVersioner properly



                Hi All,

I am trying to use the MetadataBasedFileVersioner to store my files
in a subdirectory of the data/archive folder based on the values of
two metadata elements. Has anyone does this before, and if so, can
you give me some hints.

My files need to be organized in subdirectories by year and day of



              year.


                So, lets say the file name is 'myfile', the year is
2014, and the doy of year is 002. Then, the end result that I am
looking for is to have the final location of my file be:



              'data/archive/ops/eng/2014/002/myfile.


                However, in my log file, the INFO messages indicate
that the generated final location reference is
'data/archive/ops/eng/myfile'.
And it doesn't include the year and day of year at all. And the
incorrect location leads to other errors. So I want to solve this one
first. Has anyone tried to do something like this before?

Here is what I have done so far:

Added two new elements to the .met file 'YearDir' and 'DoyDir'.
Added these new elements to the elements.xml file.
Added these new elements to the product in the
product-type-element-map.xml file.
And, I have made changes to the product-types.xml file, but here's
where I am not sure I've done this properly.

Here's what product type definition looks like:
<type id="urn:oodt:EngineeringFile" name="EngineeringFile">
   <repository path="file://[OODT_HOME]/data/archive/ops/eng"
<file://[OODT_HOME]/data/archive/ops/eng>/>
   <versioner






class="org.apache.oodt.cas.filemgr.versioning.MetadataBasedFileVersio
ner
">
         <property name="filePathSpec"







          value="/[YearDir]/[DoyDir]/" />




                   </versioner>
   <description>The default product type for any kind of
file.</description>
   <metExtractors>
     <extractor













class="org.apache.oodt.cas.filemgr.metadata.extractors.CoreMetExtract
o
r"





                       <configuration>
         <!-- you can optionally include the envReplace tag to turn
on/off environment var replacement -->
         <property name="nsAware" value="true" />
         <property name="elementNs" value="CAS" />
         <property name="elements"
value="ProductReceivedTime,ProductName,ProductId" />
       </configuration>
     </extractor>
   </metExtractors>
   <metadata/>
</type>

Thanks in advance for any help or ideas you might have!

Valerie



Valerie A. Mallder

New Horizons Deputy Mission System Engineer The Johns Hopkins
University/Applied Physics Laboratory
11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723
240-228-7846 (Office) 410-504-2233 (Blackberry)




















    --
      Tom Barber | Technical Director

      meteorite bi
      T: +44 20 8133 3730
      W: www.meteorite.bi <http://www.meteorite.bi> | Skype:
meteorite.consulting
      A: Surrey Technology Centre, Surrey Research Park,
      Guildford, GU2 7YG, UK





--
*Tom Barber* | Technical Director

meteorite bi
*T:* +44 20 8133 3730
*W:* www.meteorite.bi | *Skype:* meteorite.consulting
*A:* Surrey Technology Centre, Surrey Research Park, Guildford, GU2 7YG, UK

Reply via email to