Thanks Mike, you rock! Cheers, Chris
On Feb 9, 2012, at 7:21 AM, Cayanan, Michael D (388J) wrote: > No problemo Mr Mattmann. I've added a FAQ section to the wiki on how to > specify multiple crawler actions. > > Thanks to you and Brian for your help on this, > Mike > > On 2/8/12 7:44 PM, "Mattmann, Chris A (388J)" > <[email protected]> wrote: > >> Great job, Mike! Would you be willing to write up a quick wiki FAQ on >> this, >> here? >> >> https://cwiki.apache.org/confluence/display/OODT/CAS-PGE+Help+and+Document >> ation >> >> Cheers, >> Chris >> >> On Feb 8, 2012, at 8:34 AM, Cayanan, Michael D (388J) wrote: >> >>> Hey Brian and Chris, >>> >>> So, I first re-tried it using Chris' suggestion of using spaces and I >>> still got an error. Here is what I'm seeing in the log: >>> >>> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance >>> runIngestCrawler >>> INFO: Executing StdProductCrawler in productPath: >>> [/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output] >>> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance >>> runIngestCrawler >>> WARNING: Failed while attempting to ingest products while crawling >>> directory >>> '/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output' (all >>> products may not have been ingested) : No bean named >>> 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined >>> org.springframework.beans.factory.NoSuchBeanDefinitionException: No >>> bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined >>> at >>> org.springframework.beans.factory.support.DefaultListableBeanFactory.getB >>> eanDefinition(DefaultListableBeanFactory.java:360) >>> at >>> org.springframework.beans.factory.support.AbstractBeanFactory.getMergedBe >>> anDefinition(AbstractBeanFactory.java:914) >>> at >>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(Abs >>> tractBeanFactory.java:241) >>> at >>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(Abs >>> tractBeanFactory.java:164) >>> at >>> org.springframework.context.support.AbstractApplicationContext.getBean(Ab >>> stractApplicationContext.java:761) >>> at >>> org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFac >>> tory(CrawlerActionRepo.java:68) >>> at >>> org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:87) >>> at >>> org.apache.oodt.cas.pge.PGETaskInstance.runIngestCrawler(PGETaskInstance. >>> java:405) >>> at >>> org.apache.oodt.cas.pge.PGETaskInstance.ingestProducts(PGETaskInstance.ja >>> va:349) >>> at >>> org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:454) >>> at >>> org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102) >>> at >>> org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.ru >>> n(XmlRpcBatchStub.java:229) >>> at java.lang.Thread.run(Thread.java:680) >>> >>> >>> I then modified my tasks.xml file where I specified the >>> 'PCS_ActionsIds' property with the action IDs: >>> >>> <property name="PCS_ActionsIds" >>> value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" /> >>> >>> and added a line in the PgeConfig.xml as Brian suggested: >>> >>> <customMetadata> >>> <metadata key="PCS_ActionsIds" val="[PCS_ActionsIds]"/> >>> >>> and still got an error in the log as shown above. >>> >>> So, I ended up re-naming the property name in the tasks.xml to some >>> other name (so as to not reference itself essentially) and that did the >>> trick! >>> >>> tasks.xml: >>> >>> <property name="Actions" >>> value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" /> >>> >>> PgeConfig.xml: >>> >>> <customMetadata> >>> <metadata key="PCS_ActionsIds" val="[Actions]"/> >>> >>> >>> Also, I found out that you can't have spaces in between the commas like >>> so, >>> >>> <property name="Actions" value="KickoffL1BS0LoResWorkflow, >>> MoveMetadataToBackupDir" /> >>> >>> >>> Thanks for all your help guys! >>> >>> >>> -Mike >>> >>> >>> From: holenoter <[email protected]> >>> Reply-To: "[email protected]" <[email protected]> >>> Date: Tue, 7 Feb 2012 22:06:27 -0800 >>> To: "[email protected]" <[email protected]> >>> Subject: Re: CAS PGE Crawler Question >>> >>> workflow manager properties (static metadata) are passed to CAS-PGE via >>> WorkflowTaskConfiguration which only supports scalar metadata... CAS-PGE >>> doesn't re-parse out these metadata value because you have to allow for >>> workflow metadata to contain values with commas without always being >>> split out by CAS-PGE.. so you have to explicitly repoint the metadata >>> field at itself so it becomes custom (or local) metadata for CAS-PGE and >>> then the value will be parsed on comma... context: CAS-PGE initially did >>> the opposite (i.e. always re-parse workflow metadata into multi-values) >>> but chris later changed this to the way it is now because this broke OCO >>> since OCO had static workflow metadata which contained metadata with >>> commas that was not supposed to be parsed into multi-values. >>> >>> -brian >>> >>> On Feb 07, 2012, at 07:36 PM, "Mattmann, Chris A (388J)" >>> <[email protected]> wrote: >>> >>>> Hi Mike, >>>> >>>> Space should work in the property value below. Can you show me >>>> the log output from CAS-PGE in the workflow log file (or the resource >>>> manager >>>> batch stubs?) It should show you what Actions are being executed >>>> and that can help to debug. >>>> >>>> Thanks! >>>> >>>> Cheers, >>>> Chris >>>> >>>> On Feb 7, 2012, at 4:06 PM, Cayanan, Michael D (388J) wrote: >>>> >>>>> If I wanted to specify more than one PCS_ActionsIds values, how >>>> would I do >>>>> this? >>>>> >>>>> I've tried the following with no success: >>>>> >>>>> <property name="PCS_ActionIds" value="Action1, Action2" /> >>>>> >>>>> >>>>> and >>>>> >>>>> <property name="PCS_ActionIds" value="Action1,Action2" /> >>>>> >>>>> >>>>> and >>>>> >>>>> <property name="PCS_ActionIds" value="Action1 Action2" /> >>>>> >>>>> >>>>> >>>>> -Mike >>>>> >>>>> >>>>> On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)" >>>>> <[email protected]> wrote: >>>>> >>>>>> Hi Mike, >>>>>> >>>>>> Glad to hear! >>>>>> >>>>>> Cheers, >>>>>> Chris >>>>>> >>>>>> On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> Disregard my previous message. Turns out I had a typo. I had to set >>>>>>> 'PCS_ActionsIds' instead of 'PCS_ActionIds'. >>>>>>> >>>>>>> -Mike >>>>>>> >>>>>>> From: "Cayanan, Michael D (388J)" <[email protected]> >>>>>>> Reply-To: "[email protected]" <[email protected]> >>>>>>> Date: Tue, 7 Feb 2012 13:51:12 -0800 >>>>>>> To: "[email protected]" <[email protected]> >>>>>>> Cc: "Wong, Cynthia L (388J)" <[email protected]> >>>>>>> Subject: CAS PGE Crawler Question >>>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I would like CAS PGE to run a post ingest success action that I >>>> created >>>>>>> upon a successful ingestion of the output file(s). How do you >>>> configure >>>>>>> CAS PGE for this type of scenario? >>>>>>> >>>>>>> I have a tasks.xml file in the workflow, where I define my post >>>> ingest >>>>>>> action under the 'PCS_ActionIds' property: >>>>>>> >>>>>>> <task id="urn:my:Task" name="Task" >>>>>>> ... >>>>>>> <property name="PCS_ActionIds" >>>>>>> value="MyPostIngestSuccessAction" /> >>>>>>> <property name="PCS_ActionRepoFile" >>>>>>> value="crawler-config.xml" /> >>>>>>> </configuration> >>>>>>> </task> >>>>>>> >>>>>>> My crawler-config.xml has the following: >>>>>>> >>>>>>> <beans xmlns="http://www.springframework.org/schema/beans"; >>>>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >>>>>>> xmlns:p="http://www.springframework.org/schema/p"; >>>>>>> xsi:schemaLocation="http://www.springframework.org/schema/beans >>>>>>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd";> >>>>>>> >>>>>>> <import >>>>>>> resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/> >>>>>>> <import resource="./action-beans.xml"/> >>>>>>> <import >>>>>>> >>>> resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-b >>>> e >>>>>>> ans.xml"/> >>>>>>> <import >>>>>>> >>>> resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-be >>>> a >>>>>>> ns.xml"/> >>>>>>> </beans> >>>>>>> >>>>>>> Within my action-beans.xml, I added the post ingest success action: >>>>>>> >>>>>>> <beans xmlns="http://www.springframework.org/schema/beans"; >>>>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >>>>>>> xsi:schemaLocation="http://www.springframework.org/schema/beans >>>>>>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd";> >>>>>>> >>>>>>> <bean >>>>>>> >>>> class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcess >>>> o >>>>>>> r"/> >>>>>>> >>>>>>> <!-- Crawler Actions --> >>>>>>> >>>>>>> <bean id="MyPostIngestSuccessAction" lazy-init="true" >>>>>>> class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction"> >>>>>>> <property name="description" value="Kickoff Another Workflow >>>>>>> Event"/> >>>>>>> <property name="executeCommand" value="sh >>>>>>> [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/> >>>>>>> <property name="phases"> >>>>>>> <list> >>>>>>> <value type="java.lang.String">postIngestSuccess</value> >>>>>>> </list> >>>>>>> </property> >>>>>>> </bean> >>>>>>> >>>>>>> Not sure if I'm missing anything else. >>>>>>> >>>>>>> Thanks in advance, >>>>>>> Mike >>>>>>> >>>>>> >>>>>> >>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>> Chris Mattmann, Ph.D. >>>>>> Senior Computer Scientist >>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>>> Office: 171-266B, Mailstop: 171-246 >>>>>> Email: [email protected] >>>>>> WWW: http://sunset.usc.edu/~mattmann/ >>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>> Adjunct Assistant Professor, Computer Science Department >>>>>> University of Southern California, Los Angeles, CA 90089 USA >>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>> >>>>> >>>> >>>> >>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>> Chris Mattmann, Ph.D. >>>> Senior Computer Scientist >>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>> Office: 171-266B, Mailstop: 171-246 >>>> Email: [email protected] >>>> WWW: http://sunset.usc.edu/~mattmann/ >>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>> Adjunct Assistant Professor, Computer Science Department >>>> University of Southern California, Los Angeles, CA 90089 USA >>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>> >> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> Chris Mattmann, Ph.D. >> Senior Computer Scientist >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >> Office: 171-266B, Mailstop: 171-246 >> Email: [email protected] >> WWW: http://sunset.usc.edu/~mattmann/ >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> Adjunct Assistant Professor, Computer Science Department >> University of Southern California, Los Angeles, CA 90089 USA >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: [email protected] WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
