You did great Val, glad you got there! :) You also helped us to flush out some of the issues here and get them on the list. The next step will be to write a new improved get up and running with CAS-PGE guide. If you don¹t beat me to it, I¹ll try and do that over the next few weeks.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: [email protected] WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -----Original Message----- From: <Mallder>, Valerie <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Tuesday, October 21, 2014 at 8:40 AM To: "[email protected]" <[email protected]> Subject: RE: multiple post ingest actions with PGE >Yay Chris! Thanks, adding the "split" attribute and moving the list of >actions to the pgeConfig (without referencing the list from the workflow >task properties) did the trick. Thanks so much. The email thread you >pointed me to is the same email thread I referenced in my previous email. >The concept of why you would need to put the actions in the pgeConfig is >discussed there, but the "split" attribute is not mentioned anywhere. >Hence my confusion. I will be happy to update the wiki page with this >special little tidbit of information. :) > >Ok, so now, I am back to where I was 2-3 weeks ago. I'm crawling for new >files, ingesting them to the archive and copying some of the files to a >new directory for further processing. The only difference is back then I >was using just the crawler and filemgr, and now I'm using CAS-PGE with >crawler, filemgr, workflow, and resmgr. Whew! It's been quite an >adventure. > >I really appreciate everyone's help! Now, I have to start working on the >more complicated aspects of this pipeline. > >Val > > > > >Valerie A. Mallder >New Horizons Deputy Mission System Engineer >Johns Hopkins University/Applied Physics Laboratory > >> -----Original Message----- >> From: Mattmann, Chris A (3980) [mailto:[email protected]] >> Sent: Monday, October 20, 2014 11:38 PM >> To: [email protected] >> Subject: Re: multiple post ingest actions with PGE >> >> Hey Val, >> >> I think the answer is here: >> >> 1. First even though it?s documented on the wiki (incorrectly) that you >>can put the >> met key split ?,? in the tasks.xml file, you can?t: >> http://markmail.org/message/gxnbptf4zamq2qdp >> >> >> 2. The parser for workflow task config doesn?t process the ?split? >> attribute. However the CAS-PGE XML config file reader *does*, so if you >>simply >> move PCS_ActionsIds into your PGE config xml file, and you add the >>split=?true? >> attribute, you can use the comma delimiter to have multiple actions. >> >> <metadata key="PCS_ActionsIds" >> val="TriggerPostIngestWorkflow,DeleteDataFile" split="true"/> >> >> >> Special kudos if you can update the wiki above to have that correct >>info! :) >> >> Cheers, >> Chris >> >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> +++++ >> Chris Mattmann, Ph.D. >> Chief Architect >> Instrument Software and Science Data Systems Section (398) NASA Jet >> Propulsion Laboratory Pasadena, CA 91109 USA >> Office: 168-519, Mailstop: 168-527 >> Email: [email protected] >> WWW: http://sunset.usc.edu/~mattmann/ >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> +++++ >> Adjunct Associate Professor, Computer Science Department University of >> Southern California, Los Angeles, CA 90089 USA >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> +++++ >> >> >> >> >> >> >> -----Original Message----- >> From: <Mallder>, Valerie <[email protected]> >> Reply-To: "[email protected]" <[email protected]> >> Date: Monday, October 20, 2014 at 1:05 PM >> To: "[email protected]" <[email protected]> >> Subject: multiple post ingest actions with PGE >> >> >Hi All, >> >I am trying to run two post ingest success actions after PGE runs the >> >crawler and I have come across something that Cameron also came across >> >back in February. It thinks my two action names separated by a comma is >> >one long bean name. And, despite my best efforts I haven't been able to >> >solve this by following the instructions provided by an email thread in >> >the mailing list archives here: >> > >> >http://markmail.org/search/?q=oodt+ACTION_IDS+order%3Adate- >> backward#query: >> >oodt%20ACTION_IDS%20order%3Adate- >> backward+page:2+mid:gxnbptf4zamq2qdp+s >> >tat >> >e:results >> > >> >or by following Cameron's subsequent documentation on the wiki here: >> > >> >https://cwiki.apache.org/confluence/display/OODT/CAS-PGE+Help+and+Docum >> >ent >> >ation >> > >> >I am sure I am missing something subtle, but I would appreciate your >> >help. Here are the lines from my tasks.xml and PgeConfig.xml files. >> >And the error I am getting (which is similar to the error Cameron >> >received back in February) is below that. >> > >> >Thanks in advance! >> >Val >> > >> >In tasks.xml >> > <property name="PostIngestSuccessActions" >> >value="MoveFileToLevel0Dir,MoveMetFileToBackupDir"/> >> > >> >In PgeConfig.xml >> > <customMetadata> >> > <metadata key="PCS_ActionsIds" val="[PostIngestSuccessActions]" >>/> >> > </customMetadata> >> > >> > >> >INFO: Updating status to workflow as [CRAWLING] Oct 20, 2014 3:56:52 PM >> >org.apache.oodt.cas.pge.PGETaskInstance >> >runIngestCrawler >> >INFO: Crawling for products in >> >[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/stag >> >ing >> >] >> >Oct 20, 2014 3:56:52 PM org.apache.oodt.cas.pge.PGETaskInstance run >> >SEVERE: PGETask FAILED!!! : No bean named >> >'MoveFileToLevel0Dir,MoveMetFileToBackupDir' is defined >> >org.springframework.beans.factory.NoSuchBeanDefinitionException: No >> >bean named 'MoveFileToLevel0Dir,MoveMetFileToBackupDir' is defined >> > at >> >org.springframework.beans.factory.support.DefaultListableBeanFactory.ge >> >tBe >> >anDefinition(DefaultListableBeanFactory.java:360) >> > at >> >org.springframework.beans.factory.support.AbstractBeanFactory.getMerged >> >Bea >> >nDefinition(AbstractBeanFactory.java:914) >> > at >> >org.springframework.beans.factory.support.AbstractBeanFactory.getBean(A >> >bst >> >ractBeanFactory.java:241) >> > at >> >org.springframework.beans.factory.support.AbstractBeanFactory.getBean(A >> >bst >> >ractBeanFactory.java:164) >> > at >> >org.springframework.context.support.AbstractApplicationContext.getBean( >> >Abs >> >tractApplicationContext.java:761) >> > at >> >org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanF >> >act >> >ory(CrawlerActionRepo.java:71) >> > at >> >org.apache.oodt.cas.crawl.ProductCrawler.loadAndValidateActions(Product >> >Cra >> >wler.java:228) >> > at >> >org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:83) >> > at >> >org.apache.oodt.cas.pge.PGETaskInstance.runIngestCrawler(PGETaskInstanc >> >e.j >> >ava:575) >> > at >> >org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:144) >> > at >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.ex >> >ecu >> >teTaskLocally(IterativeWorkflowProcessorThread.java:574) >> > at >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.ru >> >n(I >> >terativeWorkflowProcessorThread.java:321) >> > at >> >EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown >>Source) >> > at java.lang.Thread.run(Thread.java:745) >> > >> >org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstanceExc >> >ept >> >ion: PGETask FAILED!!! : No bean named >> >'MoveFileToLevel0Dir,MoveMetFileToBackupDir' is defined >> > at >> >org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:150) >> > at >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.ex >> >ecu >> >teTaskLocally(IterativeWorkflowProcessorThread.java:574) >> > at >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.ru >> >n(I >> >terativeWorkflowProcessorThread.java:321) >> > at >> >EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown >>Source) >> > at java.lang.Thread.run(Thread.java:745) >> >Caused by: >> >org.springframework.beans.factory.NoSuchBeanDefinitionException: No >> >bean named 'MoveFileToLevel0Dir,MoveMetFileToBackupDir' is defined >> > at >> >org.springframework.beans.factory.support.DefaultListableBeanFactory.ge >> >tBe >> >anDefinition(DefaultListableBeanFactory.java:360) >> > at >> >org.springframework.beans.factory.support.AbstractBeanFactory.getMerged >> >Bea >> >nDefinition(AbstractBeanFactory.java:914) >> > at >> >org.springframework.beans.factory.support.AbstractBeanFactory.getBean(A >> >bst >> >ractBeanFactory.java:241) >> > at >> >org.springframework.beans.factory.support.AbstractBeanFactory.getBean(A >> >bst >> >ractBeanFactory.java:164) >> > at >> >org.springframework.context.support.AbstractApplicationContext.getBean( >> >Abs >> >tractApplicationContext.java:761) >> > at >> >org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanF >> >act >> >ory(CrawlerActionRepo.java:71) >> > at >> >org.apache.oodt.cas.crawl.ProductCrawler.loadAndValidateActions(Product >> >Cra >> >wler.java:228) >> > at >> >org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:83) >> > at >> >org.apache.oodt.cas.pge.PGETaskInstance.runIngestCrawler(PGETaskInstanc >> >e.j >> >ava:575) >> > at >> >org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:144) >> > ... 4 more >> >Oct 20, 2014 3:56:52 PM >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread >> >executeTaskLocally >> >WARNING: Exception executing task: [feiCrawlerTask] locally: Message: >> >PGETask FAILED!!! : No bean named >> >'MoveFileToLevel0Dir,MoveMetFileToBackupDir' is defined >> > >> >Valerie A. Mallder >> > >> >New Horizons Deputy Mission System Engineer The Johns Hopkins >> >University/Applied Physics Laboratory >> >11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723 >> >240-228-7846 (Office) 410-504-2233 (Blackberry) >> > >
