LOL ------------------------ Chris Mattmann [email protected]
-----Original Message----- From: "Mallder, Valerie" <[email protected]> Reply-To: <[email protected]> Date: Tuesday, October 14, 2014 at 1:20 PM To: "[email protected]" <[email protected]> Subject: RE: Failed to build PgeConfig, exception in PathUtils.doDynamicReplacement >Ok, Thanks! I will look at this more tomorrow while I'm sitting in a long >boring meeting..... > > >Valerie A. Mallder >New Horizons Deputy Mission System Engineer >Johns Hopkins University/Applied Physics Laboratory > > >> -----Original Message----- >> From: Chris Mattmann [mailto:[email protected]] >> Sent: Tuesday, October 14, 2014 4:11 PM >> To: [email protected] >> Subject: Re: Failed to build PgeConfig, exception in >> PathUtils.doDynamicReplacement >> >> Awesome Val, glad it helped and that you are moving further! >> >> Here?s a quick example of how to call your perl script as an extractor >>in CAS-PGE >> (barring yes, the duplication of specifying a regExp to select the type >>to run the >> extractor on, but I?ll get to that later): >> >> Under <output>: >> >> <files regExp=?\\*.hdf" >> >> >>metFileWriterClass="org.apache.oodt.cas.pge.writers.ExternalExtractorMetW >>ri >> ter" >> args=?/path/to/your-extern-config.xml?> >> >> >> Your extern-config.xml file should work like this: >> >> >>http://svn.apache.org/repos/asf/oodt/trunk/metadata/src/main/resources/ex >>am >> ples/extern-config.xml >> >> >> Note that the ExternExtractor is documented here: >> >> http://oodt.apache.org/components/maven/metadata/user/basic.html >> >> >> That will get you your Perl script integrated as an extractor into the >>system. >> >> On to your question about selecting information from the File Manager: >> >> Look up the SQL(.. portions of the DRAT PGE configs to get an idea of >>how to do >> this. In short, yes you can do it, and the way to do it is to use the >>special CAS- >> PGE SQL-like language that you can prefix your met key with (make sure >> envReplace is turned on). You can do like SQL([options]){ SELECT FROM >> <product type name> WHERE <criteria> >> >> The results of your above query are stored into the metadata field name >>in your >> CAS-PGE config file. >> >> I?ll walk you through more of this later, but have to run to a meeting. >> >> Gluck! >> >> Cheers, >> Chris >> >> >> ------------------------ >> Chris Mattmann >> [email protected] >> >> >> >> >> -----Original Message----- >> From: "Mallder, Valerie" <[email protected]> >> Reply-To: <[email protected]> >> Date: Tuesday, October 14, 2014 at 12:58 PM >> To: "[email protected]" <[email protected]> >> Subject: RE: Failed to build PgeConfig, exception in >> PathUtils.doDynamicReplacement >> >> >Hi Chris, >> > >> >Thanks for your emails from Friday and Saturday. They were extremely >> >helpful and I have now worked my way through all of my setup issues so >> >far. I was using the new style CASE-PGE keys because Rishi used those >> >in his working example and at one point I had been trying to mimic that >> >example. I didn't know anything about the legacy mode flag. But, after >> >changing the keys to the old style, everything started to come together >> >a little faster. >> > >> >To start my crawler at the beginning of my pipeline, I chose the >> >approach that Rishi had taken where the pgeConfig starts the >> >crawler_launcher script directly rather than rely on the definition of >> >the output files to trigger the crawler to run. This was because the >> >met extractor I am using is a perl script, and your DRAT system only >> >shows how to specify a class for the metFileWriterClass attribute. I >> >could not find anything to show me how to specify a perl script as the >>met file >> writer instead. >> >Also, having to specify the file types to crawl for by using a regular >> >expression in the config file seemed like an unnecessary duplication of >> >the definition of the files types to crawl for (since the regular >> >expression just specifies a mime-type that is already defined in >> >mime-types.xml). I didn't want to specify the mime-type in two >> >different places. >> > >> >So, now I've been through the process of setting up my CAS-PGE to >> >ingest the raw science and spacecraft housekeeping telemetry files, and >> >run a postIngestSuccess action to copy some of those files to a new >> >directory for further processing. >> > >> >The next step in my pipeline may be a little tricky. I need to query >> >the filemgr and pass the results of the query (it could be a long list) >> >as an input parameter to the first algorithm in the pipeline which, >> >happens to be another perl script. Does anyone know how I might do >> >this in a PGE config file? >> > >> >Thanks!! >> >Val >> > >> > >> > >> > >> > >> >Valerie A. Mallder >> >New Horizons Deputy Mission System Engineer Johns Hopkins >> >University/Applied Physics Laboratory >> > >> > >> >> -----Original Message----- >> >> From: Mattmann, Chris A (3980) [mailto:[email protected]] >> >> Sent: Friday, October 10, 2014 7:59 PM >> >> To: [email protected] >> >> Subject: Re: Failed to build PgeConfig, exception in >> >> PathUtils.doDynamicReplacement >> >> >> >> Thanks Val you are close! >> >> >> >> Looking at what you show below, I think the issue is one of the >> >>following: >> >> >> >> 1. You are using the new style CAS-PGE Keys. Try using the old ones >> >>(e.g., the ones present here: >> >> >> >>http://svn.apache.org/repos/asf/oodt/trunk/pge/src/main/resources/exam >> >>ple >> >>s/ >> >> WorkflowTask/tasks.xml >> >> >> >> (note the ?_? and not the ?/? used in keys). >> >> These keys are used by CAS-PGE if you see something in your wmgr bin >> >>script (or your resource manager batch_stub script) stating >> >>?legacyMode=true?). >> >> If you >> >> are using RADIX, I believe that?s the case: >> >> >> >> http://s.apache.org/hO >> >> >> >> 2. Regarding better building environments. The m2e plugin for Eclipse >> >>is fantastic nowadays and can literally checkout a multi-module Maven >> >>project from SVN (when paired with Subversive as a plugin or >> >>Subclipse). Another thing to check out is this page on the wiki: >> >> >> >> >> >>https://cwiki.apache.org/confluence/display/OODT/OODT+Eclipse+Develope >> >>r+H >> >>el >> >> p >> >> >> >> >> >> Let me know if that fixes it. Sorry for all the trouble :) Trust me >> >>it will be worth it. >> >> CAS-PGE is awesome once working. >> >> >> >> Cheers, >> >> Chris >> >> >> >> >> >> >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> +++++ >> >> Chris Mattmann, Ph.D. >> >> Chief Architect >> >> Instrument Software and Science Data Systems Section (398) NASA Jet >> >> Propulsion Laboratory Pasadena, CA 91109 USA >> >> Office: 168-519, Mailstop: 168-527 >> >> Email: [email protected] >> >> WWW: http://sunset.usc.edu/~mattmann/ >> >> >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> +++++ >> >> Adjunct Associate Professor, Computer Science Department University >> >> of Southern California, Los Angeles, CA 90089 USA >> >> >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> +++++ >> >> >> >> >> >> >> >> >> >> >> >> >> >> -----Original Message----- >> >> From: <Mallder>, Valerie <[email protected]> >> >> Reply-To: "[email protected]" <[email protected]> >> >> Date: Friday, October 10, 2014 at 3:40 PM >> >> To: "[email protected]" <[email protected]> >> >> Subject: Failed to build PgeConfig, exception in >> >>PathUtils.doDynamicReplacement >> >> >> >> >Hi All, >> >> > >> >> >I'm still working on this! Still trying to get a CA-PGE Task to run. >> >> >I'm almost there. Now it at least trys to build the PGE config file. >> >>But the >> >> >XmlFilePgeConfigBuilder is failing. I looks like it is picking up >>a >> >> >null string somewhere, but it also looks like it is crashing in a >> >> >strange place. It is unable to read the .xml file that contains my >> >> ><pgeConfig>...</pgeConfig> but, the correct path and file are shown >> >> >in the output log messages I've include below which also shows the >> >> >stack trace. I looks like it is failing during a recursive call to >> >> >PathUtils.doDynamicReplacement. I saw some chatter about some prior >> >> >errors in this code in the mailing list archives from 2010 and 2012. >> >> >So I am hoping someone might remember and be able to tell me if I am >> >> >doing something that is causing this error. I included my tasks.xml >> >> >and fei-crawler-pge-config.xml file after the log messages. >> >> > >> >> >At this point, I think need to start running these processes in an >> >> >environment where I can debug this better. The runtime output and >> >> >stack traces simply aren't enough for me to track this down. I am >> >> >newbie to using eclipse and mvn, so, does anyone have some notes on >> >> >the best way to import oodt-0.7 into an exclipe workspace and tell >> >> >it to build it using mvn?? I am hoping I can leverage other peoples >> >> >knowledge of how to do this so that I can do it quickly and not >> >> >waste another week. I imported the oodt-0.7 directory into eclipse >> >> >so I could view the files easier, but I just made eclipse link to >> >> >the where the files are located rather that copy them into a >> >> >workspace folder. If there's a better way, please tell me. And now, >> >> >how do I tell exclipse >> >>to build >> >> stuff? >> >> > >> >> >Thanks!! >> >> >Valerie >> >> > >> >> > >> >> > >> >> >Using CATALINA_BASE: >> >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat >> >> >Using CATALINA_HOME: >> >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat >> >> >Using CATALINA_TMPDIR: >> >> >> >>>/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat/t >> >>>emp >> >> >Using JRE_HOME: /project/jedi/users/jedi-pipeline/jdk1.7.0_55 >> >> >Workflow Manager started PID file >> >> >(/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/workfl >> >> >ow/ >> >> >run >> >> >/cas.workflow.pid). >> >> >Oct 10, 2014 5:53:52 PM >> >> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager >> >> >loadProperties >> >> >INFO: Loading Workflow Manager Configuration Properties from: >> >> >[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/workfl >> >> >ow/ >> >> >etc >> >> >/workflow.properties] >> >> >Oct 10, 2014 5:53:52 PM >> >> >org.apache.oodt.cas.workflow.engine.ThreadPoolWorkflowEngineFactory >> >> >getResmgrUrl >> >> >INFO: No Resource Manager URL provided or malformed URL: executing >> >> >jobs locally. URL: [null] Oct 10, 2014 5:53:52 PM >> >> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager <init> >> >> >INFO: Workflow Manager started by malldva1 Oct 10, 2014 5:54:12 PM >> >> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager >> >> >handleEvent >> >> >INFO: WorkflowManager: Received event: startJediPipeline Oct 10, >> >> >2014 >> >> >5:54:12 PM org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager >> >> >handleEvent >> >> >INFO: WorkflowManager: Workflow jediWorkflowName retrieved for event >> >> >startJediPipeline Oct 10, 2014 5:54:13 PM >> >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread >> >> >checkTaskRequiredMetadata >> >> >INFO: Task: [feiCrawlerTaskName] has no required metadata fields Oct >> >> >10, 2014 5:54:13 PM >> >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread >> >> >executeTaskLocally >> >> >INFO: Executing task: [feiCrawlerTaskName] locally Oct 10, 2014 >> >> >5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> >> >createPgeMetadata >> >> >INFO: Converting workflow configuration to static metadata... >> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> >> >createPgeMetadata >> >> >FINEST: Adding static metadata: key = [PGETask/WorkflowManagerUrl] >> >> >value = [http://localhost:9001] Oct 10, 2014 5:54:13 PM >> >> >org.apache.oodt.cas.pge.PGETaskInstance >> >> >createPgeMetadata >> >> >FINEST: Adding static metadata: key = >> >> >[PGETask/Ingest/CrawlerConfigFile] >> >> >value = >> >> >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/c >> >> >raw >> >> >ler >> >> >/policy/crawler-config.xml] >> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> >> >createPgeMetadata >> >> >FINEST: Adding static metadata: key = [PGETask/Name] value = >> >> >[feiCrawlerTaskName] Oct 10, 2014 5:54:13 PM >> >> >org.apache.oodt.cas.pge.PGETaskInstance >> >> >createPgeMetadata >> >> >FINEST: Adding static metadata: key = [PGETask/Ingest/ActionIds] >> >> >value = [MoveFileToLevel0Dir] Oct 10, 2014 5:54:13 PM >> >> >org.apache.oodt.cas.pge.PGETaskInstance >> >> >createPgeMetadata >> >> >FINEST: Adding static metadata: key = [PGETask/DumpMetadata] value = >> >> >[true] Oct 10, 2014 5:54:13 PM >> >> >org.apache.oodt.cas.pge.PGETaskInstance >> >> >createPgeMetadata >> >> >FINEST: Adding static metadata: key = >> >> >[PGETask/Query/ClientTransferServiceFactory] value = >> >> >[org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory] >> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> >> >createPgeMetadata >> >> >FINEST: Adding static metadata: key = [PGE_HOME] value = >> >> >[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge] >> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> >> >createPgeMetadata >> >> >FINEST: Adding static metadata: key = [PGETask/Query/FileManagerUrl] >> >> >value = [http://localhost:9000] Oct 10, 2014 5:54:13 PM >> >> >org.apache.oodt.cas.pge.PGETaskInstance >> >> >createPgeMetadata >> >> >FINEST: Adding static metadata: key = >> >> >[PGETask/Ingest/MimeExtractorRepo] >> >> >value = >> >> >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/e >> >> >xte >> >> >nsi >> >> >ons/policy/mime-extractor-map.xml] >> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> >> >createPgeMetadata >> >> >FINEST: Adding static metadata: key = [PGETask/ConfigFilePath] value >> >> >= >> >> >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/e >> >> >xte >> >> >nsi >> >> >ons/config/fei-crawler-pge-config.xml] >> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> >> >createPgeMetadata >> >> >FINEST: Adding static metadata: key = >> >> >[PGETask/Ingest/FileManagerUrl] value = [http://localhost:9000] Oct >> >> >10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> >> >createPgeMetadata >> >> >INFO: Loading workflow context metadata... >> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> >> >createPgeMetadata >> >> >FINEST: Adding dynamic metadata: key = [WorkflowInstId] value = >> >> >[f8730997-50c7-11e4-b9aa-57625eee7ebd] >> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> >> >createPgeMetadata >> >> >FINEST: Adding dynamic metadata: key = [JobId] value = >> >> >[f8730997-50c7-11e4-b9aa-57625eee7ebd] >> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> >> >createPgeMetadata >> >> >FINEST: Adding dynamic metadata: key = [WorkflowManagerUrl] value = >> >> >[http://slothrop.jhuapl.edu:9001] >> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> >> >createPgeMetadata >> >> >FINEST: Adding dynamic metadata: key = [TaskId] value = >> >> >[urn:oodt:feiCrawlerTaskId] Oct 10, 2014 5:54:13 PM >> >> >org.apache.oodt.cas.pge.PGETaskInstance >> >> >createPgeMetadata >> >> >FINEST: Adding dynamic metadata: key = [ProcessingNode] value = >> >> >[slothrop.jhuapl.edu] Oct 10, 2014 5:54:13 PM >> >> >org.apache.oodt.cas.pge.PGETaskInstance >> >> >createPgeConfig >> >> >INFO: Create PgeConfig... >> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> >> >createPgeConfig >> >> >INFO: Using default PgeConfigBuilder: >> >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder >> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance run >> >> >SEVERE: PGETask FAILED!!! : Failed to build PgeConfig : Failed to >> >> >parse >> >> >value: null >> >> >java.io.IOException: Failed to build PgeConfig : Failed to parse >>value: >> >> >null >> >> > at >> >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFile >> >> >Pge >> >> >Con >> >> >figBuilder.java:87) >> >> > at >> >> >org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInsta >> >> >nce >> >> >.ja >> >> >va:230) >> >> > at >> >> >>>org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123) >> >> > at >> >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread >> >> >.ex >> >> >ecu >> >> >teTaskLocally(IterativeWorkflowProcessorThread.java:574) >> >> > at >> >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread >> >> >.ru >> >> >n(I >> >> >terativeWorkflowProcessorThread.java:321) >> >> > at >> >> >EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown >> >>Source) >> >> > at java.lang.Thread.run(Thread.java:745) >> >> >Caused by: java.lang.Exception: Failed to parse value: null >> >> > at >> >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501) >> >> > at >> >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480) >> >> > at >> >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFile >> >> >Pge >> >> >Con >> >> >figBuilder.java:77) >> >> > ... 6 more >> >> >Caused by: java.lang.NullPointerException >> >> > at java.util.regex.Matcher.getTextLength(Matcher.java:1234) >> >> > at java.util.regex.Matcher.reset(Matcher.java:308) >> >> > at java.util.regex.Matcher.<init>(Matcher.java:228) >> >> > at java.util.regex.Pattern.matcher(Pattern.java:1088) >> >> > at >> >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisRep >> >> >lac >> >> >eme >> >> >nt(PathUtils.java:321) >> >> > at >> >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(Pat >> >> >hUt >> >> >ils >> >> >.java:96) >> >> > at >> >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:488) >> >> > ... 8 more >> >> > >> >> >org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstance >> >> >Exc >> >> >ept >> >> >ion: PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse >> >> >value: null >> >> > at >> >> >>>org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:150) >> >> > at >> >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread >> >> >.ex >> >> >ecu >> >> >teTaskLocally(IterativeWorkflowProcessorThread.java:574) >> >> > at >> >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread >> >> >.ru >> >> >n(I >> >> >terativeWorkflowProcessorThread.java:321) >> >> > at >> >> >EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown >> >>Source) >> >> > at java.lang.Thread.run(Thread.java:745) >> >> >Caused by: java.io.IOException: Failed to build PgeConfig : Failed >> >> >to parse value: null >> >> > at >> >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFile >> >> >Pge >> >> >Con >> >> >figBuilder.java:87) >> >> > at >> >> >org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInsta >> >> >nce >> >> >.ja >> >> >va:230) >> >> > at >> >> >>>org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123) >> >> > ... 4 more >> >> >Caused by: java.lang.Exception: Failed to parse value: null >> >> > at >> >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501) >> >> > at >> >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480) >> >> > at >> >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFile >> >> >Pge >> >> >Con >> >> >figBuilder.java:77) >> >> > ... 6 more >> >> >Caused by: java.lang.NullPointerException >> >> > at java.util.regex.Matcher.getTextLength(Matcher.java:1234) >> >> > at java.util.regex.Matcher.reset(Matcher.java:308) >> >> > at java.util.regex.Matcher.<init>(Matcher.java:228) >> >> > at java.util.regex.Pattern.matcher(Pattern.java:1088) >> >> > at >> >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisRep >> >> >lac >> >> >eme >> >> >nt(PathUtils.java:321) >> >> > at >> >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(Pat >> >> >hUt >> >> >ils >> >> >.java:96) >> >> > at >> >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:488) >> >> > ... 8 more >> >> >Oct 10, 2014 5:54:13 PM >> >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread >> >> >executeTaskLocally >> >> >WARNING: Exception executing task: [feiCrawlerTaskName] locally: >> >>Message: >> >> >PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse >>value: >> >> >null >> >> > >> >> > >> >> > >> >> > >> >> >Content of tasks.xml >> >> > >> >> > >> >> ><cas:tasks xmlns:cas="http://oodt.jpl.nasa.gov/1.0/cas";> >> >> > <task id="urn:oodt:feiCrawlerTaskId" name="feiCrawlerTaskName" >> >> >class="org.apache.oodt.cas.pge.StdPGETaskInstance"> >> >> > <conditions/> >> >> > >> >> > <configuration> >> >> > <property name="PGETask/Name" value="feiCrawlerTaskName"/> >> >> > <property name="PGETask/ConfigFilePath" >> >> >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-de >> >> >plo y/e xtensions/config/fei-crawler-pge-config.xml"/> >> >> > <property name="PGETask/DumpMetadata" value="true"/> >> >> > <property name="PGETask/WorkflowManagerUrl" >> >> >value="http://localhost:9001"; /> >> >> > <property name="PGETask/Query/FileManagerUrl" >> >> >value="http://localhost:9000"; /> >> >> > <property name="PGETask/Ingest/FileManagerUrl" >> >> >value="http://localhost:9000"/> >> >> > >> >> > <property name="PGETask/Query/ClientTransferServiceFactory" >> >> >value="org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFac >> >> >tor >> >> >y"/ >> >> >> >> >> > <property name="PGETask/Ingest/CrawlerConfigFile" >> >> >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-de >> >> >plo >> >> >y/c >> >> >rawler/policy/crawler-config.xml"/> >> >> > <property name="PGETask/Ingest/MimeExtractorRepo" >> >> >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-de >> >> >plo y/e xtensions/policy/mime-extractor-map.xml"/> >> >> > <property name="PGETask/Ingest/ActionIds" >> >> >value="MoveFileToLevel0Dir"/> >> >> > <property name="PGE_HOME" >> >> >> >>>>>value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pg >>>>>e" >> >>>/> >> >> > </configuration> >> >> > >> >> > <requiredMetFields/> >> >> > </task> >> >> ></cas:tasks> >> >> > >> >> > >> >> >Contents of fei-crawler-pge-config.xml >> >> > >> >> > >> >> ><pgeConfig> >> >> > >> >> > <!-- How to run the PGE --> >> >> > <!-just echoing the current directory to a file so I can see if >> >> >this thing ever starts --> >> >> > <exe >> >> >dir="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/da >> >> >ta/ >> >> >pge >> >> >/jobs" shell="/bin/sh"> >> >> > <cmd>echo "Current Working Directory is `pwd`" > >> >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/logs/pg >> >> >e.l >> >> >og< >> >> >/cmd> >> >> > </exe> >> >> > >> >> > <!-- Files to ingest --> >> >> > <output> >> >> > <!-- trying this approach. Telling PGE there is output files >> >> >should invoke the crawler. --> >> >> > >> >> ><dir="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/d >> >> >ata >> >> >/st >> >> >aging"/> >> >> > </output> >> >> > >> >> > <!-- Custom metadata to add to output files --> >> >> > <customMetadata> >> >> > >> >> > <!-- each of these directories exist --> >> >> > <metadata key="JobDir" val="[OODT_HOME]/data/pge/jobs" >> >> >envReplace="true"/> >> >> > <metadata key="JobInputDir" val="[JobDir]/input"/> >> >> > <metadata key="JobOutputDir" val="[JobDir]/output"/> >> >> > <metadata key="JobLogDir" val="[JobDir]/logs"/> >> >> > >> >> > </customMetadata> >> >> ></pgeConfig> >> >> > >> >> >Valerie A. Mallder >> >> > >> >> >New Horizons Deputy Mission System Engineer The Johns Hopkins >> >> >University/Applied Physics Laboratory >> >> >11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723 >> >> >240-228-7846 (Office) 410-504-2233 (Blackberry) >> >> > >> > >> >
