Thanks Val you are close! Looking at what you show below, I think the issue is one of the following:
1. You are using the new style CAS-PGE Keys. Try using the old ones (e.g., the ones present here: http://svn.apache.org/repos/asf/oodt/trunk/pge/src/main/resources/examples/ WorkflowTask/tasks.xml (note the ³_² and not the ³/³ used in keys). These keys are used by CAS-PGE if you see something in your wmgr bin script (or your resource manager batch_stub script) stating ³legacyMode=true²). If you are using RADIX, I believe that¹s the case: http://s.apache.org/hO 2. Regarding better building environments. The m2e plugin for Eclipse is fantastic nowadays and can literally checkout a multi-module Maven project from SVN (when paired with Subversive as a plugin or Subclipse). Another thing to check out is this page on the wiki: https://cwiki.apache.org/confluence/display/OODT/OODT+Eclipse+Developer+Hel p Let me know if that fixes it. Sorry for all the trouble :) Trust me it will be worth it. CAS-PGE is awesome once working. Cheers, Chris ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: [email protected] WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -----Original Message----- From: <Mallder>, Valerie <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Friday, October 10, 2014 at 3:40 PM To: "[email protected]" <[email protected]> Subject: Failed to build PgeConfig, exception in PathUtils.doDynamicReplacement >Hi All, > >I'm still working on this! Still trying to get a CA-PGE Task to run. I'm >almost there. Now it at least trys to build the PGE config file. But the >XmlFilePgeConfigBuilder is failing. I looks like it is picking up a >null string somewhere, but it also looks like it is crashing in a strange >place. It is unable to read the .xml file that contains my ><pgeConfig>...</pgeConfig> but, the correct path and file are shown in >the output log messages I've include below which also shows the stack >trace. I looks like it is failing during a recursive call to >PathUtils.doDynamicReplacement. I saw some chatter about some prior >errors in this code in the mailing list archives from 2010 and 2012. So I >am hoping someone might remember and be able to tell me if I am doing >something that is causing this error. I included my tasks.xml and >fei-crawler-pge-config.xml file after the log messages. > >At this point, I think need to start running these processes in an >environment where I can debug this better. The runtime output and stack >traces simply aren't enough for me to track this down. I am newbie to >using eclipse and mvn, so, does anyone have some notes on the best way to >import oodt-0.7 into an exclipe workspace and tell it to build it using >mvn?? I am hoping I can leverage other peoples knowledge of how to do >this so that I can do it quickly and not waste another week. I imported >the oodt-0.7 directory into eclipse so I could view the files easier, but >I just made eclipse link to the where the files are located rather that >copy them into a workspace folder. If there's a better way, please tell >me. And now, how do I tell exclipse to build stuff? > >Thanks!! >Valerie > > > >Using CATALINA_BASE: >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat >Using CATALINA_HOME: >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat >Using CATALINA_TMPDIR: >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat/temp >Using JRE_HOME: /project/jedi/users/jedi-pipeline/jdk1.7.0_55 >Workflow Manager started PID file >(/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/workflow/run >/cas.workflow.pid). >Oct 10, 2014 5:53:52 PM >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager loadProperties >INFO: Loading Workflow Manager Configuration Properties from: >[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/workflow/etc >/workflow.properties] >Oct 10, 2014 5:53:52 PM >org.apache.oodt.cas.workflow.engine.ThreadPoolWorkflowEngineFactory >getResmgrUrl >INFO: No Resource Manager URL provided or malformed URL: executing jobs >locally. URL: [null] >Oct 10, 2014 5:53:52 PM >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager <init> >INFO: Workflow Manager started by malldva1 >Oct 10, 2014 5:54:12 PM >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager handleEvent >INFO: WorkflowManager: Received event: startJediPipeline >Oct 10, 2014 5:54:12 PM >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager handleEvent >INFO: WorkflowManager: Workflow jediWorkflowName retrieved for event >startJediPipeline >Oct 10, 2014 5:54:13 PM >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread >checkTaskRequiredMetadata >INFO: Task: [feiCrawlerTaskName] has no required metadata fields >Oct 10, 2014 5:54:13 PM >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread >executeTaskLocally >INFO: Executing task: [feiCrawlerTaskName] locally >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >createPgeMetadata >INFO: Converting workflow configuration to static metadata... >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >createPgeMetadata >FINEST: Adding static metadata: key = [PGETask/WorkflowManagerUrl] value >= [http://localhost:9001] >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >createPgeMetadata >FINEST: Adding static metadata: key = [PGETask/Ingest/CrawlerConfigFile] >value = >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/crawler >/policy/crawler-config.xml] >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >createPgeMetadata >FINEST: Adding static metadata: key = [PGETask/Name] value = >[feiCrawlerTaskName] >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >createPgeMetadata >FINEST: Adding static metadata: key = [PGETask/Ingest/ActionIds] value = >[MoveFileToLevel0Dir] >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >createPgeMetadata >FINEST: Adding static metadata: key = [PGETask/DumpMetadata] value = >[true] >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >createPgeMetadata >FINEST: Adding static metadata: key = >[PGETask/Query/ClientTransferServiceFactory] value = >[org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory] >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >createPgeMetadata >FINEST: Adding static metadata: key = [PGE_HOME] value = >[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge] >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >createPgeMetadata >FINEST: Adding static metadata: key = [PGETask/Query/FileManagerUrl] >value = [http://localhost:9000] >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >createPgeMetadata >FINEST: Adding static metadata: key = [PGETask/Ingest/MimeExtractorRepo] >value = >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/extensi >ons/policy/mime-extractor-map.xml] >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >createPgeMetadata >FINEST: Adding static metadata: key = [PGETask/ConfigFilePath] value = >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/extensi >ons/config/fei-crawler-pge-config.xml] >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >createPgeMetadata >FINEST: Adding static metadata: key = [PGETask/Ingest/FileManagerUrl] >value = [http://localhost:9000] >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >createPgeMetadata >INFO: Loading workflow context metadata... >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >createPgeMetadata >FINEST: Adding dynamic metadata: key = [WorkflowInstId] value = >[f8730997-50c7-11e4-b9aa-57625eee7ebd] >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >createPgeMetadata >FINEST: Adding dynamic metadata: key = [JobId] value = >[f8730997-50c7-11e4-b9aa-57625eee7ebd] >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >createPgeMetadata >FINEST: Adding dynamic metadata: key = [WorkflowManagerUrl] value = >[http://slothrop.jhuapl.edu:9001] >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >createPgeMetadata >FINEST: Adding dynamic metadata: key = [TaskId] value = >[urn:oodt:feiCrawlerTaskId] >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >createPgeMetadata >FINEST: Adding dynamic metadata: key = [ProcessingNode] value = >[slothrop.jhuapl.edu] >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >createPgeConfig >INFO: Create PgeConfig... >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >createPgeConfig >INFO: Using default PgeConfigBuilder: >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance run >SEVERE: PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse >value: null >java.io.IOException: Failed to build PgeConfig : Failed to parse value: >null > at >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeCon >figBuilder.java:87) > at >org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.ja >va:230) > at >org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123) > at >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.execu >teTaskLocally(IterativeWorkflowProcessorThread.java:574) > at >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.run(I >terativeWorkflowProcessorThread.java:321) > at >EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Thread.java:745) >Caused by: java.lang.Exception: Failed to parse value: null > at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501) > at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480) > at >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeCon >figBuilder.java:77) > ... 6 more >Caused by: java.lang.NullPointerException > at java.util.regex.Matcher.getTextLength(Matcher.java:1234) > at java.util.regex.Matcher.reset(Matcher.java:308) > at java.util.regex.Matcher.<init>(Matcher.java:228) > at java.util.regex.Pattern.matcher(Pattern.java:1088) > at >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisReplaceme >nt(PathUtils.java:321) > at >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(PathUtils >.java:96) > at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:488) > ... 8 more > >org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstanceExcept >ion: PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse >value: null > at >org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:150) > at >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.execu >teTaskLocally(IterativeWorkflowProcessorThread.java:574) > at >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.run(I >terativeWorkflowProcessorThread.java:321) > at >EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Thread.java:745) >Caused by: java.io.IOException: Failed to build PgeConfig : Failed to >parse value: null > at >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeCon >figBuilder.java:87) > at >org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.ja >va:230) > at >org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123) > ... 4 more >Caused by: java.lang.Exception: Failed to parse value: null > at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501) > at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480) > at >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeCon >figBuilder.java:77) > ... 6 more >Caused by: java.lang.NullPointerException > at java.util.regex.Matcher.getTextLength(Matcher.java:1234) > at java.util.regex.Matcher.reset(Matcher.java:308) > at java.util.regex.Matcher.<init>(Matcher.java:228) > at java.util.regex.Pattern.matcher(Pattern.java:1088) > at >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisReplaceme >nt(PathUtils.java:321) > at >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(PathUtils >.java:96) > at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:488) > ... 8 more >Oct 10, 2014 5:54:13 PM >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread >executeTaskLocally >WARNING: Exception executing task: [feiCrawlerTaskName] locally: Message: >PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse value: >null > > > > >Content of tasks.xml > > ><cas:tasks xmlns:cas="http://oodt.jpl.nasa.gov/1.0/cas";> > <task id="urn:oodt:feiCrawlerTaskId" name="feiCrawlerTaskName" >class="org.apache.oodt.cas.pge.StdPGETaskInstance"> > <conditions/> > > <configuration> > <property name="PGETask/Name" value="feiCrawlerTaskName"/> > <property name="PGETask/ConfigFilePath" >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/e >xtensions/config/fei-crawler-pge-config.xml"/> > <property name="PGETask/DumpMetadata" value="true"/> > <property name="PGETask/WorkflowManagerUrl" >value="http://localhost:9001"; /> > <property name="PGETask/Query/FileManagerUrl" >value="http://localhost:9000"; /> > <property name="PGETask/Ingest/FileManagerUrl" >value="http://localhost:9000"/> > > <property name="PGETask/Query/ClientTransferServiceFactory" >value="org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory"/ >> > <property name="PGETask/Ingest/CrawlerConfigFile" >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/c >rawler/policy/crawler-config.xml"/> > <property name="PGETask/Ingest/MimeExtractorRepo" >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/e >xtensions/policy/mime-extractor-map.xml"/> > <property name="PGETask/Ingest/ActionIds" >value="MoveFileToLevel0Dir"/> > <property name="PGE_HOME" >value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge"/> > </configuration> > > <requiredMetFields/> > </task> ></cas:tasks> > > >Contents of fei-crawler-pge-config.xml > > ><pgeConfig> > > <!-- How to run the PGE --> > <!-just echoing the current directory to a file so I can see if this >thing ever starts --> > <exe >dir="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/pge >/jobs" shell="/bin/sh"> > <cmd>echo "Current Working Directory is `pwd`" > >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/logs/pge.log< >/cmd> > </exe> > > <!-- Files to ingest --> > <output> > <!-- trying this approach. Telling PGE there is output files should >invoke the crawler. --> > ><dir="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/st >aging"/> > </output> > > <!-- Custom metadata to add to output files --> > <customMetadata> > > <!-- each of these directories exist --> > <metadata key="JobDir" val="[OODT_HOME]/data/pge/jobs" >envReplace="true"/> > <metadata key="JobInputDir" val="[JobDir]/input"/> > <metadata key="JobOutputDir" val="[JobDir]/output"/> > <metadata key="JobLogDir" val="[JobDir]/logs"/> > > </customMetadata> ></pgeConfig> > >Valerie A. Mallder > >New Horizons Deputy Mission System Engineer >The Johns Hopkins University/Applied Physics Laboratory >11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723 >240-228-7846 (Office) 410-504-2233 (Blackberry) >
