Found it! Ugh! I'm an idiot! Got the sql to work. Now moving on to figuring out why the class in my jar file can't be found. But that's progress! Yay!
Valerie A. Mallder New Horizons Deputy Mission System Engineer Johns Hopkins University/Applied Physics Laboratory > -----Original Message----- > From: Mallder, Valerie [mailto:[email protected]] > Sent: Wednesday, October 15, 2014 5:33 PM > To: [email protected] > Subject: SQL parsing error in PgeConfig > > Hi Chris, > > I played with the SQL statement in my pge config file today and keep getting a > parsing error. I tooked at your PgeConfig_RatAggregator.xml (oh, you have a > cut- > and-paste error on line 31, '[Ampersand]' should be '[Exclamation]' if I > understand > correctly what you are doing there. And this typo propagated its way to the > other > xml files as well), and I have tried to mimic the parts that apply to what I > am trying > to do and I'm stumped by the SQL parsing error. > > I am doing a very simple query just to try to prove that I *can* do a query. > I didn't > really need the FORMAT agreement, but I wasn't sure the right syntax for > leaving it > out. So I just specified the metadata key I'm searching for in the FORMAT > arguement. So here's what I have. > > <metadata key="DirList" val="SQL(FORMAT='FileLocation') { SELECT FileLocation > FROM EngineeringFile }"/> > > One difference between this statement and yours is that you used $ signs in > your > FORMAT specification. I have tried it both ways, with dollar signs and > without > dollar signs and still get the parse error. (What is the significance of the > "$"?) > > Here's the output from oodt.out, and my pge config file and tasks.xml follow > afterwards. The stack trace looks familiar from the standpoint that I've > seen parse > errors before and fixed them. But, this one points out a suspicious looking > exception for a malformed URL for some reason. However, the output from the > log > file indicates all of my urls are fine. No pge metadata is written for this > task and no > script is created. Please let me know if you see anything obvious that I'm > missing. > > Thanks, > Val > > > > Oct 15, 2014 5:15:02 PM > org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread > checkTaskRequiredMetadata > INFO: Task: [DB-Builder-Task] has no required metadata fields Oct 15, 2014 > 5:15:03 PM > org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread > executeTaskLocally > INFO: Executing task: [DB-Builder-Task] locally Oct 15, 2014 5:15:03 PM > org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata > INFO: Converting workflow configuration to static metadata... > Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance > createPgeMetadata > FINEST: Adding static metadata: key = [PGETask_Ingest_CrawlerConfigFile] value > = [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt- > deploy/crawler/policy/crawler-config.xml] > Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance > createPgeMetadata > FINEST: Adding static metadata: key = [PCS_WorkflowManagerUrl] value = > [http://localhost:9001] Oct 15, 2014 5:15:03 PM > org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata > FINEST: Adding static metadata: key = [PGETask_Ingest_FileManagerUrl] value = > [http://localhost:9000] Oct 15, 2014 5:15:03 PM > org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata > FINEST: Adding static metadata: key = [FEI_DROP_DIR] value = > [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/staging] > Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance > createPgeMetadata > FINEST: Adding static metadata: key = [PGE_HOME] value = > [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge] > Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance > createPgeMetadata > FINEST: Adding static metadata: key = > [PGETask_Query_ClientTransferServiceFactory] value = > [org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory] > Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance > createPgeMetadata > FINEST: Adding static metadata: key = [PGETask_Ingest_ActionIds] value = > [MoveFileToLevel0Dir] Oct 15, 2014 5:15:03 PM > org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata > FINEST: Adding static metadata: key = [PGETask_Ingest_MimeExtractorRepo] > value = [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt- > deploy/extensions/policy/mime-extractor-map.xml] > Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance > createPgeMetadata > FINEST: Adding static metadata: key = [PGETask_DumpMetadata] value = [true] > Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance > createPgeMetadata > FINEST: Adding static metadata: key = [PGETask_Query_FileManagerUrl] value = > [http://localhost:9000] Oct 15, 2014 5:15:03 PM > org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata > FINEST: Adding static metadata: key = [PGETask_Name] value = [DB-Builder- > Task] Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance > createPgeMetadata > FINEST: Adding static metadata: key = [PGETask_ConfigFilePath] value = > [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt- > deploy/extensions/config/jedi-conduit-db-builder-config.xml] > Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance > createPgeMetadata > INFO: Loading workflow context metadata... > Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance > createPgeMetadata > FINEST: Adding dynamic metadata: key = [WorkflowInstId] value = [44f791c9- > 54b0-11e4-a032-c9e22d0219d9] > Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance > createPgeMetadata > FINEST: Adding dynamic metadata: key = [JobId] value = [44f791c9-54b0-11e4- > a032-c9e22d0219d9] > Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance > createPgeMetadata > FINEST: Adding dynamic metadata: key = [WorkflowManagerUrl] value = > [http://slothrop.jhuapl.edu:9001] Oct 15, 2014 5:15:03 PM > org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata > FINEST: Adding dynamic metadata: key = [TaskId] value = > [urn:oodt:dbBuilderTask] Oct 15, 2014 5:15:03 PM > org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata > FINEST: Adding dynamic metadata: key = [ProcessingNode] value = > [slothrop.jhuapl.edu] Oct 15, 2014 5:15:03 PM > org.apache.oodt.cas.pge.PGETaskInstance createPgeConfig > INFO: Create PgeConfig... > Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance > createPgeConfig > INFO: Using default PgeConfigBuilder: > org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder > Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance run > SEVERE: PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse value: > SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile } > java.io.IOException: Failed to build PgeConfig : Failed to parse value: > SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile } > at > org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfig > Builder.java:87) > at > org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.ja > va:230) > at > org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123) > at > org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.executeT > askLocally(IterativeWorkflowProcessorThread.java:574) > at > org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.run(Iterati > veWorkflowProcessorThread.java:321) > at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown > Source) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.Exception: Failed to parse value: > SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile } > at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501) > at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480) > at > org.apache.oodt.cas.pge.util.XmlHelper.getMetadataValues(XmlHelper.java:245) > at > org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.loadCustomMetadata(X > mlFilePgeConfigBuilder.java:165) > at > org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.loadConfigFile(XmlFile > PgeConfigBuilder.java:120) > at > org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfig > Builder.java:83) > ... 6 more > Caused by: java.net.MalformedURLException > at java.net.URL.<init>(URL.java:619) > at java.net.URL.<init>(URL.java:482) > at java.net.URL.<init>(URL.java:431) > at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:492) > ... 11 more > Caused by: java.lang.NullPointerException > at java.net.URL.<init>(URL.java:524) > ... 14 more > > org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstanceException > : PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse value: > SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile } > at > org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:150) > at > org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.executeT > askLocally(IterativeWorkflowProcessorThread.java:574) > at > org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.run(Iterati > veWorkflowProcessorThread.java:321) > at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown > Source) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.IOException: Failed to build PgeConfig : Failed to parse > value: > SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile } > at > org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfig > Builder.java:87) > at > org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.ja > va:230) > at > org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123) > ... 4 more > Caused by: java.lang.Exception: Failed to parse value: > SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile } > at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501) > at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480) > at > org.apache.oodt.cas.pge.util.XmlHelper.getMetadataValues(XmlHelper.java:245) > at > org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.loadCustomMetadata(X > mlFilePgeConfigBuilder.java:165) > at > org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.loadConfigFile(XmlFile > PgeConfigBuilder.java:120) > at > org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfig > Builder.java:83) > ... 6 more > Caused by: java.net.MalformedURLException > at java.net.URL.<init>(URL.java:619) > at java.net.URL.<init>(URL.java:482) > at java.net.URL.<init>(URL.java:431) > at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:492) > ... 11 more > Caused by: java.lang.NullPointerException > at java.net.URL.<init>(URL.java:524) > ... 14 more > Oct 15, 2014 5:15:03 PM > org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread > executeTaskLocally > WARNING: Exception executing task: [DB-Builder-Task] locally: Message: > PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse value: > SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile } > > <!-- > ** Filename: jedi-conduit-db-builder-config.xml > --> > <pgeConfig> > > <import file="pge-common-config.xml" /> > > <!-- Note: do not use the "shell" attribute, "shellType is the correct > attribute here"- > -> > <exe dir="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt- > deploy/data/pge/jobs" shellType="/usr/local/bin/bash"> > <cmd>echo "Current Working Directory is `pwd`" > [JobLogDir]/db- > builder.log</cmd> > <cmd>echo "Current Environment is `/bin/env`" >> [JobLogDir]/db- > builder.log</cmd> > > <cmd>[JavaHome] -cp [JediConduitJar] [LoadClass] [DirList] > [Arguements]</cmd> > </exe> > > <!-- Custom metadata to add to output files --> > <customMetadata> > <metadata key="LoadClass" val="conduit.jedi.utils.JEDIFastDBBuilder" /> > <metadata key="TelemetryFileDb" val="[JobDir]/valsTelemetryFile.db" /> > <metadata key="Arguements" val="[TelemetryFileDb]" /> > > <metadata key="DirList" val="SQL(FORMAT='FileLocation') { SELECT > FileLocation FROM EngineeringFile }" /> > > <!-- SQL example statement from the code > SQL > (FORMAT='FileLocation/Filename',SORT_BY='FileSize',FILTER=<StartDateTime,En > dDateTime,DataVersion,TakeHighestPriority) { SELECT > FileLocation,Filename,FileSize FROM IASI_L1C WHERE ProductionDateTime >= > '2007-12-01T00:00:00.000000Z' } > --> > </customMetadata> > </pgeConfig> > > > <!-- > ** Filename: tasks.xml > --> > > <task id="urn:oodt:dbBuilderTask" name="DB-Builder-Task" > class="org.apache.oodt.cas.pge.StdPGETaskInstance"> > <conditions/> > > <configuration> > <property name="PGETask_Name" value="DB-Builder-Task"/> > <property name="PGETask_ConfigFilePath" > value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt- > deploy/extensions/config/jedi-conduit-db-builder-config.xml" > envReplace="true" /> > <property name="PGETask_DumpMetadata" value="true"/> > <property name="PCS_WorkflowManagerUrl" value="http://localhost:9001"; > /> > <property name="PGETask_Query_FileManagerUrl" > value="http://localhost:9000"; /> > <property name="PGETask_Ingest_FileManagerUrl" > value="http://localhost:9000"/> > > <property name="PGETask_Query_ClientTransferServiceFactory" > value="org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory"/> > <property name="PGETask_Ingest_CrawlerConfigFile" > value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt- > deploy/crawler/policy/crawler-config.xml"/> > <property name="PGETask_Ingest_MimeExtractorRepo" > value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt- > deploy/extensions/policy/mime-extractor-map.xml"/> > <property name="PGETask_Ingest_ActionIds" > value="MoveFileToLevel0Dir"/> > <property name="PGE_HOME" > value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge"/> > > <property name="FEI_DROP_DIR" > value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt- > deploy/data/staging"/> > </configuration> > <requiredMetFields/> > </task> > > > Valerie A. Mallder > New Horizons Deputy Mission System Engineer Johns Hopkins University/Applied > Physics Laboratory > > > -----Original Message----- > > From: Chris Mattmann [mailto:[email protected]] > > Sent: Tuesday, October 14, 2014 4:11 PM > > To: [email protected] > > Subject: Re: Failed to build PgeConfig, exception in > > PathUtils.doDynamicReplacement > > > > Awesome Val, glad it helped and that you are moving further! > > > > Here?s a quick example of how to call your perl script as an extractor > > in CAS-PGE (barring yes, the duplication of specifying a regExp to > > select the type to run the extractor on, but I?ll get to that later): > > > > Under <output>: > > > > <files regExp=?\\*.hdf" > > > > metFileWriterClass="org.apache.oodt.cas.pge.writers.ExternalExtractorM > > etWri > > ter" > > args=?/path/to/your-extern-config.xml?> > > > > > > Your extern-config.xml file should work like this: > > > > http://svn.apache.org/repos/asf/oodt/trunk/metadata/src/main/resources > > /exam > > ples/extern-config.xml > > > > > > Note that the ExternExtractor is documented here: > > > > http://oodt.apache.org/components/maven/metadata/user/basic.html > > > > > > That will get you your Perl script integrated as an extractor into the > > system. > > > > On to your question about selecting information from the File Manager: > > > > Look up the SQL(.. portions of the DRAT PGE configs to get an idea of > > how to do this. In short, yes you can do it, and the way to do it is > > to use the special CAS- PGE SQL-like language that you can prefix your > > met key with (make sure envReplace is turned on). You can do like > > SQL([options]){ SELECT FROM <product type name> WHERE <criteria> > > > > The results of your above query are stored into the metadata field > > name in your CAS-PGE config file. > > > > I?ll walk you through more of this later, but have to run to a meeting. > > > > Gluck! > > > > Cheers, > > Chris > > > > > > ------------------------ > > Chris Mattmann > > [email protected] > > > > > > > > > > -----Original Message----- > > From: "Mallder, Valerie" <[email protected]> > > Reply-To: <[email protected]> > > Date: Tuesday, October 14, 2014 at 12:58 PM > > To: "[email protected]" <[email protected]> > > Subject: RE: Failed to build PgeConfig, exception in > > PathUtils.doDynamicReplacement > > > > >Hi Chris, > > > > > >Thanks for your emails from Friday and Saturday. They were extremely > > >helpful and I have now worked my way through all of my setup issues > > >so far. I was using the new style CASE-PGE keys because Rishi used > > >those in his working example and at one point I had been trying to > > >mimic that example. I didn't know anything about the legacy mode > > >flag. But, after changing the keys to the old style, everything > > >started to come together a little faster. > > > > > >To start my crawler at the beginning of my pipeline, I chose the > > >approach that Rishi had taken where the pgeConfig starts the > > >crawler_launcher script directly rather than rely on the definition > > >of the output files to trigger the crawler to run. This was because > > >the met extractor I am using is a perl script, and your DRAT system > > >only shows how to specify a class for the metFileWriterClass > > >attribute. I could not find anything to show me how to specify a perl > > >script as the met file > > writer instead. > > >Also, having to specify the file types to crawl for by using a > > >regular expression in the config file seemed like an unnecessary > > >duplication of the definition of the files types to crawl for (since > > >the regular expression just specifies a mime-type that is already > > >defined in mime-types.xml). I didn't want to specify the mime-type in > > >two different places. > > > > > >So, now I've been through the process of setting up my CAS-PGE to > > >ingest the raw science and spacecraft housekeeping telemetry files, > > >and run a postIngestSuccess action to copy some of those files to a > > >new directory for further processing. > > > > > >The next step in my pipeline may be a little tricky. I need to query > > >the filemgr and pass the results of the query (it could be a long > > >list) as an input parameter to the first algorithm in the pipeline > > >which, happens to be another perl script. Does anyone know how I > > >might do this in a PGE config file? > > > > > >Thanks!! > > >Val > > > > > > > > > > > > > > > > > >Valerie A. Mallder > > >New Horizons Deputy Mission System Engineer Johns Hopkins > > >University/Applied Physics Laboratory > > > > > > > > >> -----Original Message----- > > >> From: Mattmann, Chris A (3980) > > >> [mailto:[email protected]] > > >> Sent: Friday, October 10, 2014 7:59 PM > > >> To: [email protected] > > >> Subject: Re: Failed to build PgeConfig, exception in > > >> PathUtils.doDynamicReplacement > > >> > > >> Thanks Val you are close! > > >> > > >> Looking at what you show below, I think the issue is one of the > > >>following: > > >> > > >> 1. You are using the new style CAS-PGE Keys. Try using the old ones > > >>(e.g., the ones present here: > > >> > > >>http://svn.apache.org/repos/asf/oodt/trunk/pge/src/main/resources/ex > > >>am > > >>ple > > >>s/ > > >> WorkflowTask/tasks.xml > > >> > > >> (note the ?_? and not the ?/? used in keys). > > >> These keys are used by CAS-PGE if you see something in your wmgr > > >>bin script (or your resource manager batch_stub script) stating > > >>?legacyMode=true?). > > >> If you > > >> are using RADIX, I believe that?s the case: > > >> > > >> http://s.apache.org/hO > > >> > > >> 2. Regarding better building environments. The m2e plugin for > > >>Eclipse is fantastic nowadays and can literally checkout a > > >>multi-module Maven project from SVN (when paired with Subversive as > > >>a plugin or Subclipse). Another thing to check out is this page on the > > >>wiki: > > >> > > >> > > >>https://cwiki.apache.org/confluence/display/OODT/OODT+Eclipse+Develo > > >>pe > > >>r+H > > >>el > > >> p > > >> > > >> > > >> Let me know if that fixes it. Sorry for all the trouble :) Trust me > > >>it will be worth it. > > >> CAS-PGE is awesome once working. > > >> > > >> Cheers, > > >> Chris > > >> > > >> > > >> > > > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > >> +++++ > > >> Chris Mattmann, Ph.D. > > >> Chief Architect > > >> Instrument Software and Science Data Systems Section (398) NASA Jet > > >> Propulsion Laboratory Pasadena, CA 91109 USA > > >> Office: 168-519, Mailstop: 168-527 > > >> Email: [email protected] > > >> WWW: http://sunset.usc.edu/~mattmann/ > > >> > > > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > >> +++++ > > >> Adjunct Associate Professor, Computer Science Department University > > >> of Southern California, Los Angeles, CA 90089 USA > > >> > > > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > >> +++++ > > >> > > >> > > >> > > >> > > >> > > >> > > >> -----Original Message----- > > >> From: <Mallder>, Valerie <[email protected]> > > >> Reply-To: "[email protected]" <[email protected]> > > >> Date: Friday, October 10, 2014 at 3:40 PM > > >> To: "[email protected]" <[email protected]> > > >> Subject: Failed to build PgeConfig, exception in > > >>PathUtils.doDynamicReplacement > > >> > > >> >Hi All, > > >> > > > >> >I'm still working on this! Still trying to get a CA-PGE Task to run. > > >> >I'm almost there. Now it at least trys to build the PGE config file. > > >>But the > > >> >XmlFilePgeConfigBuilder is failing. I looks like it is picking up a > > >> >null string somewhere, but it also looks like it is crashing in a > > >> >strange place. It is unable to read the .xml file that contains > > >> >my <pgeConfig>...</pgeConfig> but, the correct path and file are > > >> >shown in the output log messages I've include below which also > > >> >shows the stack trace. I looks like it is failing during a > > >> >recursive call to PathUtils.doDynamicReplacement. I saw some > > >> >chatter about some prior errors in this code in the mailing list > > >> >archives from > 2010 and 2012. > > >> >So I am hoping someone might remember and be able to tell me if I > > >> >am doing something that is causing this error. I included my > > >> >tasks.xml and fei-crawler-pge-config.xml file after the log messages. > > >> > > > >> >At this point, I think need to start running these processes in an > > >> >environment where I can debug this better. The runtime output and > > >> >stack traces simply aren't enough for me to track this down. I am > > >> >newbie to using eclipse and mvn, so, does anyone have some notes > > >> >on the best way to import oodt-0.7 into an exclipe workspace and > > >> >tell it to build it using mvn?? I am hoping I can leverage other > > >> >peoples knowledge of how to do this so that I can do it quickly > > >> >and not waste another week. I imported the oodt-0.7 directory into > > >> >eclipse so I could view the files easier, but I just made eclipse > > >> >link to the where the files are located rather that copy them into > > >> >a workspace folder. If there's a better way, please tell me. And > > >> >now, how do I tell exclipse > > >>to build > > >> stuff? > > >> > > > >> >Thanks!! > > >> >Valerie > > >> > > > >> > > > >> > > > >> >Using CATALINA_BASE: > > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomca > > >> >t > > >> >Using CATALINA_HOME: > > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomca > > >> >t > > >> >Using CATALINA_TMPDIR: > > >> > > >>>/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat > > >>>/t > > >>>emp > > >> >Using JRE_HOME: /project/jedi/users/jedi-pipeline/jdk1.7.0_55 > > >> >Workflow Manager started PID file > > >> >(/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/work > > >> >fl > > >> >ow/ > > >> >run > > >> >/cas.workflow.pid). > > >> >Oct 10, 2014 5:53:52 PM > > >> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager > > >> >loadProperties > > >> >INFO: Loading Workflow Manager Configuration Properties from: > > >> >[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/work > > >> >fl > > >> >ow/ > > >> >etc > > >> >/workflow.properties] > > >> >Oct 10, 2014 5:53:52 PM > > >> >org.apache.oodt.cas.workflow.engine.ThreadPoolWorkflowEngineFactor > > >> >y > > >> >getResmgrUrl > > >> >INFO: No Resource Manager URL provided or malformed URL: executing > > >> >jobs locally. URL: [null] Oct 10, 2014 5:53:52 PM > > >> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager <init> > > >> >INFO: Workflow Manager started by malldva1 Oct 10, 2014 5:54:12 PM > > >> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager > > >> >handleEvent > > >> >INFO: WorkflowManager: Received event: startJediPipeline Oct 10, > > >> >2014 > > >> >5:54:12 PM > > >> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager > > >> >handleEvent > > >> >INFO: WorkflowManager: Workflow jediWorkflowName retrieved for > > >> >event startJediPipeline Oct 10, 2014 5:54:13 PM > > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre > > >> >ad > > >> >checkTaskRequiredMetadata > > >> >INFO: Task: [feiCrawlerTaskName] has no required metadata fields > > >> >Oct 10, 2014 5:54:13 PM > > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre > > >> >ad > > >> >executeTaskLocally > > >> >INFO: Executing task: [feiCrawlerTaskName] locally Oct 10, 2014 > > >> >5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance > > >> >createPgeMetadata > > >> >INFO: Converting workflow configuration to static metadata... > > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance > > >> >createPgeMetadata > > >> >FINEST: Adding static metadata: key = [PGETask/WorkflowManagerUrl] > > >> >value = [http://localhost:9001] Oct 10, 2014 5:54:13 PM > > >> >org.apache.oodt.cas.pge.PGETaskInstance > > >> >createPgeMetadata > > >> >FINEST: Adding static metadata: key = > > >> >[PGETask/Ingest/CrawlerConfigFile] > > >> >value = > > >> >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy > > >> >/c > > >> >raw > > >> >ler > > >> >/policy/crawler-config.xml] > > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance > > >> >createPgeMetadata > > >> >FINEST: Adding static metadata: key = [PGETask/Name] value = > > >> >[feiCrawlerTaskName] Oct 10, 2014 5:54:13 PM > > >> >org.apache.oodt.cas.pge.PGETaskInstance > > >> >createPgeMetadata > > >> >FINEST: Adding static metadata: key = [PGETask/Ingest/ActionIds] > > >> >value = [MoveFileToLevel0Dir] Oct 10, 2014 5:54:13 PM > > >> >org.apache.oodt.cas.pge.PGETaskInstance > > >> >createPgeMetadata > > >> >FINEST: Adding static metadata: key = [PGETask/DumpMetadata] value > > >> >= [true] Oct 10, 2014 5:54:13 PM > > >> >org.apache.oodt.cas.pge.PGETaskInstance > > >> >createPgeMetadata > > >> >FINEST: Adding static metadata: key = > > >> >[PGETask/Query/ClientTransferServiceFactory] value = > > >> >[org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory > > >> >] Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance > > >> >createPgeMetadata > > >> >FINEST: Adding static metadata: key = [PGE_HOME] value = > > >> >[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge] > > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance > > >> >createPgeMetadata > > >> >FINEST: Adding static metadata: key = > > >> >[PGETask/Query/FileManagerUrl] value = [http://localhost:9000] Oct > > >> >10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance > > >> >createPgeMetadata > > >> >FINEST: Adding static metadata: key = > > >> >[PGETask/Ingest/MimeExtractorRepo] > > >> >value = > > >> >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy > > >> >/e > > >> >xte > > >> >nsi > > >> >ons/policy/mime-extractor-map.xml] > > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance > > >> >createPgeMetadata > > >> >FINEST: Adding static metadata: key = [PGETask/ConfigFilePath] > > >> >value = > > >> >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy > > >> >/e > > >> >xte > > >> >nsi > > >> >ons/config/fei-crawler-pge-config.xml] > > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance > > >> >createPgeMetadata > > >> >FINEST: Adding static metadata: key = > > >> >[PGETask/Ingest/FileManagerUrl] value = [http://localhost:9000] > > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance > > >> >createPgeMetadata > > >> >INFO: Loading workflow context metadata... > > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance > > >> >createPgeMetadata > > >> >FINEST: Adding dynamic metadata: key = [WorkflowInstId] value = > > >> >[f8730997-50c7-11e4-b9aa-57625eee7ebd] > > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance > > >> >createPgeMetadata > > >> >FINEST: Adding dynamic metadata: key = [JobId] value = > > >> >[f8730997-50c7-11e4-b9aa-57625eee7ebd] > > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance > > >> >createPgeMetadata > > >> >FINEST: Adding dynamic metadata: key = [WorkflowManagerUrl] value > > >> >= [http://slothrop.jhuapl.edu:9001] Oct 10, 2014 5:54:13 PM > > >> >org.apache.oodt.cas.pge.PGETaskInstance > > >> >createPgeMetadata > > >> >FINEST: Adding dynamic metadata: key = [TaskId] value = > > >> >[urn:oodt:feiCrawlerTaskId] Oct 10, 2014 5:54:13 PM > > >> >org.apache.oodt.cas.pge.PGETaskInstance > > >> >createPgeMetadata > > >> >FINEST: Adding dynamic metadata: key = [ProcessingNode] value = > > >> >[slothrop.jhuapl.edu] Oct 10, 2014 5:54:13 PM > > >> >org.apache.oodt.cas.pge.PGETaskInstance > > >> >createPgeConfig > > >> >INFO: Create PgeConfig... > > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance > > >> >createPgeConfig > > >> >INFO: Using default PgeConfigBuilder: > > >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder > > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance > > >> >run > > >> >SEVERE: PGETask FAILED!!! : Failed to build PgeConfig : Failed to > > >> >parse > > >> >value: null > > >> >java.io.IOException: Failed to build PgeConfig : Failed to parse value: > > >> >null > > >> > at > > >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFi > > >> >le > > >> >Pge > > >> >Con > > >> >figBuilder.java:87) > > >> > at > > >> >org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskIns > > >> >ta > > >> >nce > > >> >.ja > > >> >va:230) > > >> > at > > >> >org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123) > > >> > at > > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre > > >> >ad > > >> >.ex > > >> >ecu > > >> >teTaskLocally(IterativeWorkflowProcessorThread.java:574) > > >> > at > > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre > > >> >ad > > >> >.ru > > >> >n(I > > >> >terativeWorkflowProcessorThread.java:321) > > >> > at > > >> >EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown > > >>Source) > > >> > at java.lang.Thread.run(Thread.java:745) > > >> >Caused by: java.lang.Exception: Failed to parse value: null > > >> > at > > >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501) > > >> > at > > >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480) > > >> > at > > >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFi > > >> >le > > >> >Pge > > >> >Con > > >> >figBuilder.java:77) > > >> > ... 6 more > > >> >Caused by: java.lang.NullPointerException > > >> > at java.util.regex.Matcher.getTextLength(Matcher.java:1234) > > >> > at java.util.regex.Matcher.reset(Matcher.java:308) > > >> > at java.util.regex.Matcher.<init>(Matcher.java:228) > > >> > at java.util.regex.Pattern.matcher(Pattern.java:1088) > > >> > at > > >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisR > > >> >ep > > >> >lac > > >> >eme > > >> >nt(PathUtils.java:321) > > >> > at > > >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(P > > >> >at > > >> >hUt > > >> >ils > > >> >.java:96) > > >> > at > > >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:488) > > >> > ... 8 more > > >> > > > >> >org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstan > > >> >ce > > >> >Exc > > >> >ept > > >> >ion: PGETask FAILED!!! : Failed to build PgeConfig : Failed to > > >> >parse > > >> >value: null > > >> > at > > >> >org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:150) > > >> > at > > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre > > >> >ad > > >> >.ex > > >> >ecu > > >> >teTaskLocally(IterativeWorkflowProcessorThread.java:574) > > >> > at > > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre > > >> >ad > > >> >.ru > > >> >n(I > > >> >terativeWorkflowProcessorThread.java:321) > > >> > at > > >> >EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown > > >>Source) > > >> > at java.lang.Thread.run(Thread.java:745) > > >> >Caused by: java.io.IOException: Failed to build PgeConfig : Failed > > >> >to parse value: null > > >> > at > > >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFi > > >> >le > > >> >Pge > > >> >Con > > >> >figBuilder.java:87) > > >> > at > > >> >org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskIns > > >> >ta > > >> >nce > > >> >.ja > > >> >va:230) > > >> > at > > >> >org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123) > > >> > ... 4 more > > >> >Caused by: java.lang.Exception: Failed to parse value: null > > >> > at > > >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501) > > >> > at > > >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480) > > >> > at > > >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFi > > >> >le > > >> >Pge > > >> >Con > > >> >figBuilder.java:77) > > >> > ... 6 more > > >> >Caused by: java.lang.NullPointerException > > >> > at java.util.regex.Matcher.getTextLength(Matcher.java:1234) > > >> > at java.util.regex.Matcher.reset(Matcher.java:308) > > >> > at java.util.regex.Matcher.<init>(Matcher.java:228) > > >> > at java.util.regex.Pattern.matcher(Pattern.java:1088) > > >> > at > > >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisR > > >> >ep > > >> >lac > > >> >eme > > >> >nt(PathUtils.java:321) > > >> > at > > >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(P > > >> >at > > >> >hUt > > >> >ils > > >> >.java:96) > > >> > at > > >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:488) > > >> > ... 8 more > > >> >Oct 10, 2014 5:54:13 PM > > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre > > >> >ad > > >> >executeTaskLocally > > >> >WARNING: Exception executing task: [feiCrawlerTaskName] locally: > > >>Message: > > >> >PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse value: > > >> >null > > >> > > > >> > > > >> > > > >> > > > >> >Content of tasks.xml > > >> > > > >> > > > >> ><cas:tasks xmlns:cas="http://oodt.jpl.nasa.gov/1.0/cas";> > > >> > <task id="urn:oodt:feiCrawlerTaskId" name="feiCrawlerTaskName" > > >> >class="org.apache.oodt.cas.pge.StdPGETaskInstance"> > > >> > <conditions/> > > >> > > > >> > <configuration> > > >> > <property name="PGETask/Name" value="feiCrawlerTaskName"/> > > >> > <property name="PGETask/ConfigFilePath" > > >> >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt- > > >> >de plo y/e xtensions/config/fei-crawler-pge-config.xml"/> > > >> > <property name="PGETask/DumpMetadata" value="true"/> > > >> > <property name="PGETask/WorkflowManagerUrl" > > >> >value="http://localhost:9001"; /> > > >> > <property name="PGETask/Query/FileManagerUrl" > > >> >value="http://localhost:9000"; /> > > >> > <property name="PGETask/Ingest/FileManagerUrl" > > >> >value="http://localhost:9000"/> > > >> > > > >> > <property name="PGETask/Query/ClientTransferServiceFactory" > > >> >value="org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferF > > >> >ac > > >> >tor > > >> >y"/ > > >> >> > > >> > <property name="PGETask/Ingest/CrawlerConfigFile" > > >> >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt- > > >> >de > > >> >plo > > >> >y/c > > >> >rawler/policy/crawler-config.xml"/> > > >> > <property name="PGETask/Ingest/MimeExtractorRepo" > > >> >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt- > > >> >de plo y/e xtensions/policy/mime-extractor-map.xml"/> > > >> > <property name="PGETask/Ingest/ActionIds" > > >> >value="MoveFileToLevel0Dir"/> > > >> > <property name="PGE_HOME" > > >> > > >>>value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge" > > >>>/> > > >> > </configuration> > > >> > > > >> > <requiredMetFields/> > > >> > </task> > > >> ></cas:tasks> > > >> > > > >> > > > >> >Contents of fei-crawler-pge-config.xml > > >> > > > >> > > > >> ><pgeConfig> > > >> > > > >> > <!-- How to run the PGE --> > > >> > <!-just echoing the current directory to a file so I can see if > > >> >this thing ever starts --> > > >> > <exe > > >> >dir="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/ > > >> >da > > >> >ta/ > > >> >pge > > >> >/jobs" shell="/bin/sh"> > > >> > <cmd>echo "Current Working Directory is `pwd`" > > > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/logs/ > > >> >pg > > >> >e.l > > >> >og< > > >> >/cmd> > > >> > </exe> > > >> > > > >> > <!-- Files to ingest --> > > >> > <output> > > >> > <!-- trying this approach. Telling PGE there is output files > > >> >should invoke the crawler. --> > > >> > > > >> ><dir="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy > > >> >/d > > >> >ata > > >> >/st > > >> >aging"/> > > >> > </output> > > >> > > > >> > <!-- Custom metadata to add to output files --> > > >> > <customMetadata> > > >> > > > >> > <!-- each of these directories exist --> > > >> > <metadata key="JobDir" val="[OODT_HOME]/data/pge/jobs" > > >> >envReplace="true"/> > > >> > <metadata key="JobInputDir" val="[JobDir]/input"/> > > >> > <metadata key="JobOutputDir" val="[JobDir]/output"/> > > >> > <metadata key="JobLogDir" val="[JobDir]/logs"/> > > >> > > > >> > </customMetadata> > > >> ></pgeConfig> > > >> > > > >> >Valerie A. Mallder > > >> > > > >> >New Horizons Deputy Mission System Engineer The Johns Hopkins > > >> >University/Applied Physics Laboratory > > >> >11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723 > > >> >240-228-7846 (Office) 410-504-2233 (Blackberry) > > >> > > > > > >
