+1, let's chat some more about it. I'll take a look maybe later in the week thanks dude...
Cheers, Chris On Feb 8, 2012, at 10:27 PM, holenoter wrote: > > hey chris, > > this particular thing is a one liner... however, if there are current > workflows using trunk CAS-PGE they may break due to the change... probably > should talk more about maybe making this Java Property configurable or > something > > -brian > > On Feb 08, 2012, at 06:54 PM, "Mattmann, Chris A (388J)" > <[email protected]> wrote: > >> Thanks Bfost. >> >> How hard would it be to just port that part to trunk? Let's coordinate, me >> and you on it. >> >> I'll take a look at the end of this week, but won't have time before then, >> as I'm working >> on Wengine porting, and some cas curator stuff that folks just reported. >> >> Thanks! >> >> Cheers, >> Chris >> >> On Feb 8, 2012, at 9:20 AM, holenoter wrote: >> >> > hey mike, >> > >> > forgot this didn't work with the trunk version... this was something that >> > was changed in the branch... truck CAS-PGE uses PgeMetadata to determine >> > precedence (or query order) when getting metadata values from static, >> > local, or dynamic metadata... PgeMetadata's query order is: Static, >> > Dynamic, then Custom (or Local)... the branched CAS-PGE uses WEngines >> > ControlMetadata whose query order is: Dynamic, Local, then Static... so in >> > the trunk there is no way to override static workflow metadata... >> > something fixed in the branched version, and i guess chris can decide if >> > this is something that he would like ported to the trunk in his porting >> > efforts :) >> > >> > -brian >> > >> > On Feb 08, 2012, at 08:34 AM, "Cayanan, Michael D (388J)" >> > <[email protected]> wrote: >> > >> >> Hey Brian and Chris, >> >> >> >> So, I first re-tried it using Chris' suggestion of using spaces and I >> >> still got an error. Here is what I'm seeing in the log: >> >> >> >> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance >> >> runIngestCrawler >> >> INFO: Executing StdProductCrawler in productPath: >> >> [/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output] >> >> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance >> >> runIngestCrawler >> >> WARNING: Failed while attempting to ingest products while crawling >> >> directory >> >> '/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output' (all >> >> products may not have been ingested) : No bean named >> >> 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined >> >> org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean >> >> named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined >> >> at >> >> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:360) >> >> at >> >> org.springframework.beans.factory.support.AbstractBeanFactory.getMergedBeanDefinition(AbstractBeanFactory.java:914) >> >> at >> >> org.springframework.beans.factorysupport.AbstractBeanFactory.getBean(AbstractBeanFactory.java:241) >> >> at >> >> org.springframework.beans.factory.supportAbstractBeanFactory.getBean(AbstractBeanFactory.java:164) >> >> at >> >> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:761) >> >> at >> >> org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFactory(CrawlerActionRepo.java:68) >> >> at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:87) >> >> at >> >> org.apache.oodt.cas.pge.PGETaskInstance.runIngestCrawler(PGETaskInstance.java:405) >> >> at >> >> org.apache.oodt.cas.pge.PGETaskInstance.ingestProducts(PGETaskInstance.java:349) >> >> at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:454) >> >> at orgapache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102) >> >> at >> >> org.apache.oodtcas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run(XmlRpcBatchStub.java:229) >> >> at java.lang.Thread.run(Thread.java:680) >> >> >> >> >> >> I then modified my tasks.xml file where I specified the 'PCS_ActionsIds' >> >> property with the action IDs: >> >> >> >> <property name="PCS_ActionsIds" >> >> value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" /> >> >> >> >> and added a line in the PgeConfig.xml as Brian suggested: >> >> >> >> <customMetadata> >> >> <metadata key="PCS_ActionsIds" val="[PCS_ActionsIds]"/> >> >> >> >> and still got an error in the log as shown above. >> >> >> >> So, I ended up re-naming the property name in the tasks.xml to some other >> >> name (so as to not reference itself essentially) and that did the trick! >> >> >> >> tasks.xml: >> >> >> >> <property name="Actions" >> >> value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" /> >> >> >> >> PgeConfig.xml: >> >> >> >> <customMetadata> >> >> <metadata key="PCS_ActionsIds" val="[Actions]"/> >> >> >> >> >> >> Also, I found out that you can't have spaces in between the commas like >> >> so, >> >> >> >> <property name="Actions" value="KickoffL1BS0LoResWorkflow, >> >> MoveMetadataToBackupDir" /> >> >> >> >> >> >> Thanks for all your help guys! >> >> >> >> >> >> -Mike >> >> >> >> >> >> From: holenoter <[email protected]> >> >> Reply-To: "[email protected]" <[email protected]> >> >> Date: Tue, 7 Feb 2012 22:06:27 -0800 >> >> To: "[email protected]" <[email protected]> >> >> Subject: Re: CAS PGE Crawler Question >> >> >> >> workflow manager properties (static metadata) are passed to CAS-PGE via >> >> WorkflowTaskConfiguration which only supports scalar metadata... CAS-PGE >> >> doesn't re-parse out these metadata value because you have to allow for >> >> workflow metadata to contain values with commas without always being >> >> split out by CAS-PGE.. so you have to explicitly repoint the metadata >> >> field at itself so it becomes custom (or local) metadata for CAS-PGE and >> >> then the value will be parsed on comma... context: CAS-PGE initially did >> >> the opposite (i.e. always re-parse workflow metadata into multi-values) >> >> but chris later changed this to the way it is now because this broke OCO >> >> since OCO had static workflow metadata which contained metadata with >> >> commas that was not supposed to be parsed into multi-values. >> >> >> >> -brian >> >> >> >> On Feb 07, 2012, at 07:36 PM, "Mattmann, Chris A (388J)" >> >> <[email protected]> wrote: >> >> >> >>> Hi Mike, >> >>> >> >>> Space should work in the property value below. Can you show me >> >>> the log output from CAS-PGE in the workflow log file (or the resource >> >>> manager >> >>> batch stubs?) It should show you what Actions are being executed >> >>> and that can help to debug. >> >>> >> >>> Thanks! >> >>> >> >>> Cheers, >> >>> Chris >> >>> >> >>> On Feb 7, 2012, at 4:06 PM, Cayanan, Michael D (388J) wrote: >> >>> >> >>> > If I wanted to specify more than one PCS_ActionsIds values, how would >> >>> > I do >> >>> > this? >> >>> > >> >>> > I've tried the following with no success: >> >>> > >> >>> > <property name="PCS_ActionIds" value="Action1, Action2" /> >> >>> > >> >>> > >> >>> > and >> >>> > >> >>> > <property name="PCS_ActionIds" value="Action1,Action2" /> >> >>> > >> >>> > >> >>> > and >> >>> > >> >>> > <property name="PCS_ActionIds" value="Action1 Action2" /> >> >>> > >> >>> > >> >>> > >> >>> > -Mike >> >>> > >> >>> > >> >>> > On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)" >> >>> > <[email protected]> wrote: >> >>> > >> >>> >> Hi Mike, >> >>> >> >> >>> >> Glad to hear! >> >>> >> >> >>> >> Cheers, >> >>> >> Chris >> >>> >> >> >>> >> On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote: >> >>> >> >> >>> >>> Hi all, >> >>> >>> >> >>> >>> Disregard my previous message. Turns out I had a typo. I had to set >> >>> >>> 'PCS_ActionsIds' instead of 'PCS_ActionIds'. >> >>> >>> >> >>> >>> -Mike >> >>> >>> >> >>> >>> From: "Cayanan, Michael D (388J)" <[email protected]> >> >>> >>> Reply-To: "[email protected]" <[email protected]> >> >>> >>> Date: Tue, 7 Feb 2012 13:51:12 -0800 >> >>> >>> To: "[email protected]" <[email protected]> >> >>> >>> Cc: "Wong, Cynthia L (388J)" <[email protected]> >> >>> >>> Subject: CAS PGE Crawler Question >> >>> >>> >> >>> >>> Hi all, >> >>> >>> >> >>> >>> I would like CAS PGE to run a post ingest success action that I >> >>> >>> created >> >>> >>> upon a successful ingestion of the output file(s). How do you >> >>> >>> configure >> >>> >>> CAS PGE for this type of scenario? >> >>> >>> >> >>> >>> I have a tasks.xml file in the workflow, where I define my post >> >>> >>> ingest >> >>> >>> action under the 'PCS_ActionIds' property: >> >>> >>> >> >>> >>> <task id="urn:my:Task" name="Task" >> >>> >>> ... >> >>> >>> <property name="PCS_ActionIds" >> >>> >>> value="MyPostIngestSuccessAction" /> >> >>> >>> <property name="PCS_ActionRepoFile" >> >>> >>> value="crawler-config.xml" /> >> >>> >>> </configuration> >> >>> >>> </task> >> >>> >>> >> >>> >>> My crawler-config.xml has the following: >> >>> >>> >> >>> >>> <beans xmlns="http://www.springframework.org/schema/beans"; >> >>> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >> >>> >>> xmlns:p="http://www.springframework.org/schema/p"; >> >>> >>> xsi:schemaLocation="http://www.springframework.org/schema/beans >> >>> >>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd";> >> >>> >>> >> >>> >>> <import >> >>> >>> resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/> >> >>> >>> <import resource="./action-beans.xml"/> >> >>> >>> <import >> >>> >>> resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-be >> >>> >>> ansxml"/> >> >>> >>> <import >> >>> >>> resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-bea >> >>> >>> ns.xml"/> >> >>> >>> </beans> >> >>> >>> >> >>> >>> Within my action-beans.xml, I added the post ingest success action: >> >>> >>> >> >>> >>> <beans xmlns="http://www.springframework.org/schema/beans"; >> >>> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >> >>> >>> xsi:schemaLocation="http://www.springframework.org/schema/beans >> >>> >>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd";> >> >>> >>> >> >>> >>> <bean >> >>> >>> class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcesso >> >>> >>> r"/> >> >>> >>> >> >>> >>> <!-- Crawler Actions --> >> >>> >>> >> >>> >>> <bean id="MyPostIngestSuccessAction" lazy-init="true" >> >>> >>> class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction"> >> >>> >>> <property name="description" value="Kickoff Another Workflow >> >>> >>> Event"/> >> >>> >>> <property name="executeCommand" value="sh >> >>> >>> [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/> >> >>> >>> <property name="phases"> >> >>> >>> <list> >> >>> >>> <value type="java.lang.String">postIngestSuccess</value> >> >>> >>> </list> >> >>> >>> </property> >> >>> >>> </bean> >> >>> >>> >> >>> >>> Not sure if I'm missing anything else. >> >>> >>> >> >>> >>> Thanks in advance, >> >>> >>> Mike >> >>> >>> >> >>> >> >> >>> >> >> >>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >>> >> Chris Mattmann, Ph.D. >> >>> >> Senior Computer Scientist >> >>> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >> >>> >> Office: 171-266B, Mailstop: 171-246 >> >>> >> Email: [email protected] >> >>> >> WWW: http://sunset.usc.edu/~mattmann/ >> >>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >>> >> Adjunct Assistant Professor, Computer Science Department >> >>> >> University of Southern California, Los Angeles, CA 90089 USA >> >>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >>> >> >> >>> > >> >>> >> >>> >> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >>> Chris Mattmann, Ph.D >> >>> Senior Computer Scientist >> >>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >> >>> Office: 171-266B, Mailstop: 171-246 >> >>> Email: [email protected] >> >>> WWW: http://sunset.usc.edu/~mattmann/ >> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >>> Adjunct Assistant Professor, Computer Science Department >> >>> University of Southern California, Los Angeles, CA 90089 USA >> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >>> >> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> Chris Mattmann, Ph.D. >> Senior Computer Scientist >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >> Office: 171-266B, Mailstop: 171-246 >> Email: [email protected] >> WWW: http://sunset.usc.edu/~mattmann/ >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> Adjunct Assistant Professor, Computer Science Department >> University of Southern California, Los Angeles, CA 90089 USA >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: [email protected] WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
