You are definitely not an idiot - you are awesome! :) Great work. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: [email protected] WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message----- From: <Mallder>, Valerie <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Thursday, October 16, 2014 at 9:07 AM To: "[email protected]" <[email protected]> Subject: RE: SQL parsing error in PgeConfig >Found it! Ugh! I'm an idiot! Got the sql to work. Now moving on to >figuring out why the class in my jar file can't be found. But that's >progress! Yay! > > >Valerie A. Mallder >New Horizons Deputy Mission System Engineer >Johns Hopkins University/Applied Physics Laboratory > > >> -----Original Message----- >> From: Mallder, Valerie [mailto:[email protected]] >> Sent: Wednesday, October 15, 2014 5:33 PM >> To: [email protected] >> Subject: SQL parsing error in PgeConfig >> >> Hi Chris, >> >> I played with the SQL statement in my pge config file today and keep >>getting a >> parsing error. I tooked at your PgeConfig_RatAggregator.xml (oh, you >>have a cut- >> and-paste error on line 31, '[Ampersand]' should be '[Exclamation]' if >>I understand >> correctly what you are doing there. And this typo propagated its way to >>the other >> xml files as well), and I have tried to mimic the parts that apply to >>what I am trying >> to do and I'm stumped by the SQL parsing error. >> >> I am doing a very simple query just to try to prove that I *can* do a >>query. I didn't >> really need the FORMAT agreement, but I wasn't sure the right syntax >>for leaving it >> out. So I just specified the metadata key I'm searching for in the >>FORMAT >> arguement. So here's what I have. >> >> <metadata key="DirList" val="SQL(FORMAT='FileLocation') { SELECT >>FileLocation >> FROM EngineeringFile }"/> >> >> One difference between this statement and yours is that you used $ >>signs in your >> FORMAT specification. I have tried it both ways, with dollar signs and >>without >> dollar signs and still get the parse error. (What is the significance >>of the "$"?) >> >> Here's the output from oodt.out, and my pge config file and tasks.xml >>follow >> afterwards. The stack trace looks familiar from the standpoint that >>I've seen parse >> errors before and fixed them. But, this one points out a suspicious >>looking >> exception for a malformed URL for some reason. However, the output from >>the log >> file indicates all of my urls are fine. No pge metadata is written for >>this task and no >> script is created. Please let me know if you see anything obvious that >>I'm missing. >> >> Thanks, >> Val >> >> >> >> Oct 15, 2014 5:15:02 PM >> org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread >> checkTaskRequiredMetadata >> INFO: Task: [DB-Builder-Task] has no required metadata fields Oct 15, >>2014 >> 5:15:03 PM >> org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread >> executeTaskLocally >> INFO: Executing task: [DB-Builder-Task] locally Oct 15, 2014 5:15:03 PM >> org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata >> INFO: Converting workflow configuration to static metadata... >> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance >> createPgeMetadata >> FINEST: Adding static metadata: key = >>[PGETask_Ingest_CrawlerConfigFile] value >> = [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt- >> deploy/crawler/policy/crawler-config.xml] >> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance >> createPgeMetadata >> FINEST: Adding static metadata: key = [PCS_WorkflowManagerUrl] value = >> [http://localhost:9001] Oct 15, 2014 5:15:03 PM >> org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata >> FINEST: Adding static metadata: key = [PGETask_Ingest_FileManagerUrl] >>value = >> [http://localhost:9000] Oct 15, 2014 5:15:03 PM >> org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata >> FINEST: Adding static metadata: key = [FEI_DROP_DIR] value = >> >>[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/stagin >>g] >> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance >> createPgeMetadata >> FINEST: Adding static metadata: key = [PGE_HOME] value = >> [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge] >> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance >> createPgeMetadata >> FINEST: Adding static metadata: key = >> [PGETask_Query_ClientTransferServiceFactory] value = >> [org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory] >> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance >> createPgeMetadata >> FINEST: Adding static metadata: key = [PGETask_Ingest_ActionIds] value = >> [MoveFileToLevel0Dir] Oct 15, 2014 5:15:03 PM >> org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata >> FINEST: Adding static metadata: key = [PGETask_Ingest_MimeExtractorRepo] >> value = [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt- >> deploy/extensions/policy/mime-extractor-map.xml] >> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance >> createPgeMetadata >> FINEST: Adding static metadata: key = [PGETask_DumpMetadata] value = >>[true] >> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance >> createPgeMetadata >> FINEST: Adding static metadata: key = [PGETask_Query_FileManagerUrl] >>value = >> [http://localhost:9000] Oct 15, 2014 5:15:03 PM >> org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata >> FINEST: Adding static metadata: key = [PGETask_Name] value = >>[DB-Builder- >> Task] Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance >> createPgeMetadata >> FINEST: Adding static metadata: key = [PGETask_ConfigFilePath] value = >> [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt- >> deploy/extensions/config/jedi-conduit-db-builder-config.xml] >> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance >> createPgeMetadata >> INFO: Loading workflow context metadata... >> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance >> createPgeMetadata >> FINEST: Adding dynamic metadata: key = [WorkflowInstId] value = >>[44f791c9- >> 54b0-11e4-a032-c9e22d0219d9] >> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance >> createPgeMetadata >> FINEST: Adding dynamic metadata: key = [JobId] value = >>[44f791c9-54b0-11e4- >> a032-c9e22d0219d9] >> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance >> createPgeMetadata >> FINEST: Adding dynamic metadata: key = [WorkflowManagerUrl] value = >> [http://slothrop.jhuapl.edu:9001] Oct 15, 2014 5:15:03 PM >> org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata >> FINEST: Adding dynamic metadata: key = [TaskId] value = >> [urn:oodt:dbBuilderTask] Oct 15, 2014 5:15:03 PM >> org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata >> FINEST: Adding dynamic metadata: key = [ProcessingNode] value = >> [slothrop.jhuapl.edu] Oct 15, 2014 5:15:03 PM >> org.apache.oodt.cas.pge.PGETaskInstance createPgeConfig >> INFO: Create PgeConfig... >> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance >> createPgeConfig >> INFO: Using default PgeConfigBuilder: >> org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder >> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance run >> SEVERE: PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse >>value: >> SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile } >> java.io.IOException: Failed to build PgeConfig : Failed to parse value: >> SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile } >> at >> >>org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeCo >>nfig >> Builder.java:87) >> at >> >>org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.j >>a >> va:230) >> at >> org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123) >> at >> >>org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.exec >>uteT >> askLocally(IterativeWorkflowProcessorThread.java:574) >> at >> >>org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.run( >>Iterati >> veWorkflowProcessorThread.java:321) >> at >>EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown >> Source) >> at java.lang.Thread.run(Thread.java:745) >> Caused by: java.lang.Exception: Failed to parse value: >> SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile } >> at >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501) >> at >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480) >> at >> >>org.apache.oodt.cas.pge.util.XmlHelper.getMetadataValues(XmlHelper.java:2 >>45) >> at >> >>org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.loadCustomMetadata >>(X >> mlFilePgeConfigBuilder.java:165) >> at >> >>org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.loadConfigFile(Xml >>File >> PgeConfigBuilder.java:120) >> at >> >>org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeCo >>nfig >> Builder.java:83) >> ... 6 more >> Caused by: java.net.MalformedURLException >> at java.net.URL.<init>(URL.java:619) >> at java.net.URL.<init>(URL.java:482) >> at java.net.URL.<init>(URL.java:431) >> at >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:492) >> ... 11 more >> Caused by: java.lang.NullPointerException >> at java.net.URL.<init>(URL.java:524) >> ... 14 more >> >> >>org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstanceExcep >>tion >> : PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse value: >> SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile } >> at >> org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:150) >> at >> >>org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.exec >>uteT >> askLocally(IterativeWorkflowProcessorThread.java:574) >> at >> >>org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.run( >>Iterati >> veWorkflowProcessorThread.java:321) >> at >>EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown >> Source) >> at java.lang.Thread.run(Thread.java:745) >> Caused by: java.io.IOException: Failed to build PgeConfig : Failed to >>parse value: >> SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile } >> at >> >>org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeCo >>nfig >> Builder.java:87) >> at >> >>org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.j >>a >> va:230) >> at >> org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123) >> ... 4 more >> Caused by: java.lang.Exception: Failed to parse value: >> SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile } >> at >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501) >> at >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480) >> at >> >>org.apache.oodt.cas.pge.util.XmlHelper.getMetadataValues(XmlHelper.java:2 >>45) >> at >> >>org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.loadCustomMetadata >>(X >> mlFilePgeConfigBuilder.java:165) >> at >> >>org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.loadConfigFile(Xml >>File >> PgeConfigBuilder.java:120) >> at >> >>org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeCo >>nfig >> Builder.java:83) >> ... 6 more >> Caused by: java.net.MalformedURLException >> at java.net.URL.<init>(URL.java:619) >> at java.net.URL.<init>(URL.java:482) >> at java.net.URL.<init>(URL.java:431) >> at >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:492) >> ... 11 more >> Caused by: java.lang.NullPointerException >> at java.net.URL.<init>(URL.java:524) >> ... 14 more >> Oct 15, 2014 5:15:03 PM >> org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread >> executeTaskLocally >> WARNING: Exception executing task: [DB-Builder-Task] locally: Message: >> PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse value: >> SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile } >> >> <!-- >> ** Filename: jedi-conduit-db-builder-config.xml >> --> >> <pgeConfig> >> >> <import file="pge-common-config.xml" /> >> >> <!-- Note: do not use the "shell" attribute, "shellType is the >>correct attribute here"- >> -> >> <exe dir="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt- >> deploy/data/pge/jobs" shellType="/usr/local/bin/bash"> >> <cmd>echo "Current Working Directory is `pwd`" > [JobLogDir]/db- >> builder.log</cmd> >> <cmd>echo "Current Environment is `/bin/env`" >> [JobLogDir]/db- >> builder.log</cmd> >> >> <cmd>[JavaHome] -cp [JediConduitJar] [LoadClass] [DirList] >> [Arguements]</cmd> >> </exe> >> >> <!-- Custom metadata to add to output files --> >> <customMetadata> >> <metadata key="LoadClass" >>val="conduit.jedi.utils.JEDIFastDBBuilder" /> >> <metadata key="TelemetryFileDb" >>val="[JobDir]/valsTelemetryFile.db" /> >> <metadata key="Arguements" val="[TelemetryFileDb]" /> >> >> <metadata key="DirList" val="SQL(FORMAT='FileLocation') { SELECT >> FileLocation FROM EngineeringFile }" /> >> >> <!-- SQL example statement from the code >> SQL >> >>(FORMAT='FileLocation/Filename',SORT_BY='FileSize',FILTER=<StartDateTime, >>En >> dDateTime,DataVersion,TakeHighestPriority) { SELECT >> FileLocation,Filename,FileSize FROM IASI_L1C WHERE ProductionDateTime >>>= >> '2007-12-01T00:00:00.000000Z' } >> --> >> </customMetadata> >> </pgeConfig> >> >> >> <!-- >> ** Filename: tasks.xml >> --> >> >> <task id="urn:oodt:dbBuilderTask" name="DB-Builder-Task" >> class="org.apache.oodt.cas.pge.StdPGETaskInstance"> >> <conditions/> >> >> <configuration> >> <property name="PGETask_Name" value="DB-Builder-Task"/> >> <property name="PGETask_ConfigFilePath" >> value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt- >> deploy/extensions/config/jedi-conduit-db-builder-config.xml" >>envReplace="true" /> >> <property name="PGETask_DumpMetadata" value="true"/> >> <property name="PCS_WorkflowManagerUrl" >>value="http://localhost:9001"; >> /> >> <property name="PGETask_Query_FileManagerUrl" >> value="http://localhost:9000"; /> >> <property name="PGETask_Ingest_FileManagerUrl" >> value="http://localhost:9000"/> >> >> <property name="PGETask_Query_ClientTransferServiceFactory" >> >>value="org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory" >>/> >> <property name="PGETask_Ingest_CrawlerConfigFile" >> value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt- >> deploy/crawler/policy/crawler-config.xml"/> >> <property name="PGETask_Ingest_MimeExtractorRepo" >> value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt- >> deploy/extensions/policy/mime-extractor-map.xml"/> >> <property name="PGETask_Ingest_ActionIds" >> value="MoveFileToLevel0Dir"/> >> <property name="PGE_HOME" >> >>value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge"/ >>> >> >> <property name="FEI_DROP_DIR" >> value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt- >> deploy/data/staging"/> >> </configuration> >> <requiredMetFields/> >> </task> >> >> >> Valerie A. Mallder >> New Horizons Deputy Mission System Engineer Johns Hopkins >>University/Applied >> Physics Laboratory >> >> > -----Original Message----- >> > From: Chris Mattmann [mailto:[email protected]] >> > Sent: Tuesday, October 14, 2014 4:11 PM >> > To: [email protected] >> > Subject: Re: Failed to build PgeConfig, exception in >> > PathUtils.doDynamicReplacement >> > >> > Awesome Val, glad it helped and that you are moving further! >> > >> > Here?s a quick example of how to call your perl script as an extractor >> > in CAS-PGE (barring yes, the duplication of specifying a regExp to >> > select the type to run the extractor on, but I?ll get to that later): >> > >> > Under <output>: >> > >> > <files regExp=?\\*.hdf" >> > >> > metFileWriterClass="org.apache.oodt.cas.pge.writers.ExternalExtractorM >> > etWri >> > ter" >> > args=?/path/to/your-extern-config.xml?> >> > >> > >> > Your extern-config.xml file should work like this: >> > >> > http://svn.apache.org/repos/asf/oodt/trunk/metadata/src/main/resources >> > /exam >> > ples/extern-config.xml >> > >> > >> > Note that the ExternExtractor is documented here: >> > >> > http://oodt.apache.org/components/maven/metadata/user/basic.html >> > >> > >> > That will get you your Perl script integrated as an extractor into >>the system. >> > >> > On to your question about selecting information from the File Manager: >> > >> > Look up the SQL(.. portions of the DRAT PGE configs to get an idea of >> > how to do this. In short, yes you can do it, and the way to do it is >> > to use the special CAS- PGE SQL-like language that you can prefix your >> > met key with (make sure envReplace is turned on). You can do like >> > SQL([options]){ SELECT FROM <product type name> WHERE <criteria> >> > >> > The results of your above query are stored into the metadata field >> > name in your CAS-PGE config file. >> > >> > I?ll walk you through more of this later, but have to run to a >>meeting. >> > >> > Gluck! >> > >> > Cheers, >> > Chris >> > >> > >> > ------------------------ >> > Chris Mattmann >> > [email protected] >> > >> > >> > >> > >> > -----Original Message----- >> > From: "Mallder, Valerie" <[email protected]> >> > Reply-To: <[email protected]> >> > Date: Tuesday, October 14, 2014 at 12:58 PM >> > To: "[email protected]" <[email protected]> >> > Subject: RE: Failed to build PgeConfig, exception in >> > PathUtils.doDynamicReplacement >> > >> > >Hi Chris, >> > > >> > >Thanks for your emails from Friday and Saturday. They were extremely >> > >helpful and I have now worked my way through all of my setup issues >> > >so far. I was using the new style CASE-PGE keys because Rishi used >> > >those in his working example and at one point I had been trying to >> > >mimic that example. I didn't know anything about the legacy mode >> > >flag. But, after changing the keys to the old style, everything >> > >started to come together a little faster. >> > > >> > >To start my crawler at the beginning of my pipeline, I chose the >> > >approach that Rishi had taken where the pgeConfig starts the >> > >crawler_launcher script directly rather than rely on the definition >> > >of the output files to trigger the crawler to run. This was because >> > >the met extractor I am using is a perl script, and your DRAT system >> > >only shows how to specify a class for the metFileWriterClass >> > >attribute. I could not find anything to show me how to specify a perl >> > >script as the met file >> > writer instead. >> > >Also, having to specify the file types to crawl for by using a >> > >regular expression in the config file seemed like an unnecessary >> > >duplication of the definition of the files types to crawl for (since >> > >the regular expression just specifies a mime-type that is already >> > >defined in mime-types.xml). I didn't want to specify the mime-type in >> > >two different places. >> > > >> > >So, now I've been through the process of setting up my CAS-PGE to >> > >ingest the raw science and spacecraft housekeeping telemetry files, >> > >and run a postIngestSuccess action to copy some of those files to a >> > >new directory for further processing. >> > > >> > >The next step in my pipeline may be a little tricky. I need to query >> > >the filemgr and pass the results of the query (it could be a long >> > >list) as an input parameter to the first algorithm in the pipeline >> > >which, happens to be another perl script. Does anyone know how I >> > >might do this in a PGE config file? >> > > >> > >Thanks!! >> > >Val >> > > >> > > >> > > >> > > >> > > >> > >Valerie A. Mallder >> > >New Horizons Deputy Mission System Engineer Johns Hopkins >> > >University/Applied Physics Laboratory >> > > >> > > >> > >> -----Original Message----- >> > >> From: Mattmann, Chris A (3980) >> > >> [mailto:[email protected]] >> > >> Sent: Friday, October 10, 2014 7:59 PM >> > >> To: [email protected] >> > >> Subject: Re: Failed to build PgeConfig, exception in >> > >> PathUtils.doDynamicReplacement >> > >> >> > >> Thanks Val you are close! >> > >> >> > >> Looking at what you show below, I think the issue is one of the >> > >>following: >> > >> >> > >> 1. You are using the new style CAS-PGE Keys. Try using the old ones >> > >>(e.g., the ones present here: >> > >> >> > >>http://svn.apache.org/repos/asf/oodt/trunk/pge/src/main/resources/ex >> > >>am >> > >>ple >> > >>s/ >> > >> WorkflowTask/tasks.xml >> > >> >> > >> (note the ?_? and not the ?/? used in keys). >> > >> These keys are used by CAS-PGE if you see something in your wmgr >> > >>bin script (or your resource manager batch_stub script) stating >> > >>?legacyMode=true?). >> > >> If you >> > >> are using RADIX, I believe that?s the case: >> > >> >> > >> http://s.apache.org/hO >> > >> >> > >> 2. Regarding better building environments. The m2e plugin for >> > >>Eclipse is fantastic nowadays and can literally checkout a >> > >>multi-module Maven project from SVN (when paired with Subversive as >> > >>a plugin or Subclipse). Another thing to check out is this page on >>the wiki: >> > >> >> > >> >> > >>https://cwiki.apache.org/confluence/display/OODT/OODT+Eclipse+Develo >> > >>pe >> > >>r+H >> > >>el >> > >> p >> > >> >> > >> >> > >> Let me know if that fixes it. Sorry for all the trouble :) Trust me >> > >>it will be worth it. >> > >> CAS-PGE is awesome once working. >> > >> >> > >> Cheers, >> > >> Chris >> > >> >> > >> >> > >> >> > >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> > >> +++++ >> > >> Chris Mattmann, Ph.D. >> > >> Chief Architect >> > >> Instrument Software and Science Data Systems Section (398) NASA Jet >> > >> Propulsion Laboratory Pasadena, CA 91109 USA >> > >> Office: 168-519, Mailstop: 168-527 >> > >> Email: [email protected] >> > >> WWW: http://sunset.usc.edu/~mattmann/ >> > >> >> > >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> > >> +++++ >> > >> Adjunct Associate Professor, Computer Science Department University >> > >> of Southern California, Los Angeles, CA 90089 USA >> > >> >> > >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> > >> +++++ >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> -----Original Message----- >> > >> From: <Mallder>, Valerie <[email protected]> >> > >> Reply-To: "[email protected]" <[email protected]> >> > >> Date: Friday, October 10, 2014 at 3:40 PM >> > >> To: "[email protected]" <[email protected]> >> > >> Subject: Failed to build PgeConfig, exception in >> > >>PathUtils.doDynamicReplacement >> > >> >> > >> >Hi All, >> > >> > >> > >> >I'm still working on this! Still trying to get a CA-PGE Task to >>run. >> > >> >I'm almost there. Now it at least trys to build the PGE config >>file. >> > >>But the >> > >> >XmlFilePgeConfigBuilder is failing. I looks like it is picking >>up a >> > >> >null string somewhere, but it also looks like it is crashing in a >> > >> >strange place. It is unable to read the .xml file that contains >> > >> >my <pgeConfig>...</pgeConfig> but, the correct path and file are >> > >> >shown in the output log messages I've include below which also >> > >> >shows the stack trace. I looks like it is failing during a >> > >> >recursive call to PathUtils.doDynamicReplacement. I saw some >> > >> >chatter about some prior errors in this code in the mailing list >>archives from >> 2010 and 2012. >> > >> >So I am hoping someone might remember and be able to tell me if I >> > >> >am doing something that is causing this error. I included my >> > >> >tasks.xml and fei-crawler-pge-config.xml file after the log >>messages. >> > >> > >> > >> >At this point, I think need to start running these processes in an >> > >> >environment where I can debug this better. The runtime output and >> > >> >stack traces simply aren't enough for me to track this down. I am >> > >> >newbie to using eclipse and mvn, so, does anyone have some notes >> > >> >on the best way to import oodt-0.7 into an exclipe workspace and >> > >> >tell it to build it using mvn?? I am hoping I can leverage other >> > >> >peoples knowledge of how to do this so that I can do it quickly >> > >> >and not waste another week. I imported the oodt-0.7 directory into >> > >> >eclipse so I could view the files easier, but I just made eclipse >> > >> >link to the where the files are located rather that copy them into >> > >> >a workspace folder. If there's a better way, please tell me. And >> > >> >now, how do I tell exclipse >> > >>to build >> > >> stuff? >> > >> > >> > >> >Thanks!! >> > >> >Valerie >> > >> > >> > >> > >> > >> > >> > >> >Using CATALINA_BASE: >> > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomca >> > >> >t >> > >> >Using CATALINA_HOME: >> > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomca >> > >> >t >> > >> >Using CATALINA_TMPDIR: >> > >> >> > >>>/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat >> > >>>/t >> > >>>emp >> > >> >Using JRE_HOME: >>/project/jedi/users/jedi-pipeline/jdk1.7.0_55 >> > >> >Workflow Manager started PID file >> > >> >(/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/work >> > >> >fl >> > >> >ow/ >> > >> >run >> > >> >/cas.workflow.pid). >> > >> >Oct 10, 2014 5:53:52 PM >> > >> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager >> > >> >loadProperties >> > >> >INFO: Loading Workflow Manager Configuration Properties from: >> > >> >[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/work >> > >> >fl >> > >> >ow/ >> > >> >etc >> > >> >/workflow.properties] >> > >> >Oct 10, 2014 5:53:52 PM >> > >> >org.apache.oodt.cas.workflow.engine.ThreadPoolWorkflowEngineFactor >> > >> >y >> > >> >getResmgrUrl >> > >> >INFO: No Resource Manager URL provided or malformed URL: executing >> > >> >jobs locally. URL: [null] Oct 10, 2014 5:53:52 PM >> > >> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager <init> >> > >> >INFO: Workflow Manager started by malldva1 Oct 10, 2014 5:54:12 PM >> > >> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager >> > >> >handleEvent >> > >> >INFO: WorkflowManager: Received event: startJediPipeline Oct 10, >> > >> >2014 >> > >> >5:54:12 PM >> > >> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager >> > >> >handleEvent >> > >> >INFO: WorkflowManager: Workflow jediWorkflowName retrieved for >> > >> >event startJediPipeline Oct 10, 2014 5:54:13 PM >> > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre >> > >> >ad >> > >> >checkTaskRequiredMetadata >> > >> >INFO: Task: [feiCrawlerTaskName] has no required metadata fields >> > >> >Oct 10, 2014 5:54:13 PM >> > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre >> > >> >ad >> > >> >executeTaskLocally >> > >> >INFO: Executing task: [feiCrawlerTaskName] locally Oct 10, 2014 >> > >> >5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> > >> >createPgeMetadata >> > >> >INFO: Converting workflow configuration to static metadata... >> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> > >> >createPgeMetadata >> > >> >FINEST: Adding static metadata: key = [PGETask/WorkflowManagerUrl] >> > >> >value = [http://localhost:9001] Oct 10, 2014 5:54:13 PM >> > >> >org.apache.oodt.cas.pge.PGETaskInstance >> > >> >createPgeMetadata >> > >> >FINEST: Adding static metadata: key = >> > >> >[PGETask/Ingest/CrawlerConfigFile] >> > >> >value = >> > >> >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy >> > >> >/c >> > >> >raw >> > >> >ler >> > >> >/policy/crawler-config.xml] >> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> > >> >createPgeMetadata >> > >> >FINEST: Adding static metadata: key = [PGETask/Name] value = >> > >> >[feiCrawlerTaskName] Oct 10, 2014 5:54:13 PM >> > >> >org.apache.oodt.cas.pge.PGETaskInstance >> > >> >createPgeMetadata >> > >> >FINEST: Adding static metadata: key = [PGETask/Ingest/ActionIds] >> > >> >value = [MoveFileToLevel0Dir] Oct 10, 2014 5:54:13 PM >> > >> >org.apache.oodt.cas.pge.PGETaskInstance >> > >> >createPgeMetadata >> > >> >FINEST: Adding static metadata: key = [PGETask/DumpMetadata] value >> > >> >= [true] Oct 10, 2014 5:54:13 PM >> > >> >org.apache.oodt.cas.pge.PGETaskInstance >> > >> >createPgeMetadata >> > >> >FINEST: Adding static metadata: key = >> > >> >[PGETask/Query/ClientTransferServiceFactory] value = >> > >> >[org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory >> > >> >] Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> > >> >createPgeMetadata >> > >> >FINEST: Adding static metadata: key = [PGE_HOME] value = >> > >> >[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge] >> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> > >> >createPgeMetadata >> > >> >FINEST: Adding static metadata: key = >> > >> >[PGETask/Query/FileManagerUrl] value = [http://localhost:9000] Oct >> > >> >10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> > >> >createPgeMetadata >> > >> >FINEST: Adding static metadata: key = >> > >> >[PGETask/Ingest/MimeExtractorRepo] >> > >> >value = >> > >> >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy >> > >> >/e >> > >> >xte >> > >> >nsi >> > >> >ons/policy/mime-extractor-map.xml] >> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> > >> >createPgeMetadata >> > >> >FINEST: Adding static metadata: key = [PGETask/ConfigFilePath] >> > >> >value = >> > >> >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy >> > >> >/e >> > >> >xte >> > >> >nsi >> > >> >ons/config/fei-crawler-pge-config.xml] >> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> > >> >createPgeMetadata >> > >> >FINEST: Adding static metadata: key = >> > >> >[PGETask/Ingest/FileManagerUrl] value = [http://localhost:9000] >> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> > >> >createPgeMetadata >> > >> >INFO: Loading workflow context metadata... >> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> > >> >createPgeMetadata >> > >> >FINEST: Adding dynamic metadata: key = [WorkflowInstId] value = >> > >> >[f8730997-50c7-11e4-b9aa-57625eee7ebd] >> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> > >> >createPgeMetadata >> > >> >FINEST: Adding dynamic metadata: key = [JobId] value = >> > >> >[f8730997-50c7-11e4-b9aa-57625eee7ebd] >> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> > >> >createPgeMetadata >> > >> >FINEST: Adding dynamic metadata: key = [WorkflowManagerUrl] value >> > >> >= [http://slothrop.jhuapl.edu:9001] Oct 10, 2014 5:54:13 PM >> > >> >org.apache.oodt.cas.pge.PGETaskInstance >> > >> >createPgeMetadata >> > >> >FINEST: Adding dynamic metadata: key = [TaskId] value = >> > >> >[urn:oodt:feiCrawlerTaskId] Oct 10, 2014 5:54:13 PM >> > >> >org.apache.oodt.cas.pge.PGETaskInstance >> > >> >createPgeMetadata >> > >> >FINEST: Adding dynamic metadata: key = [ProcessingNode] value = >> > >> >[slothrop.jhuapl.edu] Oct 10, 2014 5:54:13 PM >> > >> >org.apache.oodt.cas.pge.PGETaskInstance >> > >> >createPgeConfig >> > >> >INFO: Create PgeConfig... >> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> > >> >createPgeConfig >> > >> >INFO: Using default PgeConfigBuilder: >> > >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder >> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance >> > >> >run >> > >> >SEVERE: PGETask FAILED!!! : Failed to build PgeConfig : Failed to >> > >> >parse >> > >> >value: null >> > >> >java.io.IOException: Failed to build PgeConfig : Failed to parse >>value: >> > >> >null >> > >> > at >> > >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFi >> > >> >le >> > >> >Pge >> > >> >Con >> > >> >figBuilder.java:87) >> > >> > at >> > >> >org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskIns >> > >> >ta >> > >> >nce >> > >> >.ja >> > >> >va:230) >> > >> > at >> > >> >>>org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123) >> > >> > at >> > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre >> > >> >ad >> > >> >.ex >> > >> >ecu >> > >> >teTaskLocally(IterativeWorkflowProcessorThread.java:574) >> > >> > at >> > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre >> > >> >ad >> > >> >.ru >> > >> >n(I >> > >> >terativeWorkflowProcessorThread.java:321) >> > >> > at >> > >> >EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown >> > >>Source) >> > >> > at java.lang.Thread.run(Thread.java:745) >> > >> >Caused by: java.lang.Exception: Failed to parse value: null >> > >> > at >> > >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501) >> > >> > at >> > >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480) >> > >> > at >> > >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFi >> > >> >le >> > >> >Pge >> > >> >Con >> > >> >figBuilder.java:77) >> > >> > ... 6 more >> > >> >Caused by: java.lang.NullPointerException >> > >> > at java.util.regex.Matcher.getTextLength(Matcher.java:1234) >> > >> > at java.util.regex.Matcher.reset(Matcher.java:308) >> > >> > at java.util.regex.Matcher.<init>(Matcher.java:228) >> > >> > at java.util.regex.Pattern.matcher(Pattern.java:1088) >> > >> > at >> > >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisR >> > >> >ep >> > >> >lac >> > >> >eme >> > >> >nt(PathUtils.java:321) >> > >> > at >> > >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(P >> > >> >at >> > >> >hUt >> > >> >ils >> > >> >.java:96) >> > >> > at >> > >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:488) >> > >> > ... 8 more >> > >> > >> > >> >org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstan >> > >> >ce >> > >> >Exc >> > >> >ept >> > >> >ion: PGETask FAILED!!! : Failed to build PgeConfig : Failed to >> > >> >parse >> > >> >value: null >> > >> > at >> > >> >>>org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:150) >> > >> > at >> > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre >> > >> >ad >> > >> >.ex >> > >> >ecu >> > >> >teTaskLocally(IterativeWorkflowProcessorThread.java:574) >> > >> > at >> > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre >> > >> >ad >> > >> >.ru >> > >> >n(I >> > >> >terativeWorkflowProcessorThread.java:321) >> > >> > at >> > >> >EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown >> > >>Source) >> > >> > at java.lang.Thread.run(Thread.java:745) >> > >> >Caused by: java.io.IOException: Failed to build PgeConfig : Failed >> > >> >to parse value: null >> > >> > at >> > >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFi >> > >> >le >> > >> >Pge >> > >> >Con >> > >> >figBuilder.java:87) >> > >> > at >> > >> >org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskIns >> > >> >ta >> > >> >nce >> > >> >.ja >> > >> >va:230) >> > >> > at >> > >> >>>org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123) >> > >> > ... 4 more >> > >> >Caused by: java.lang.Exception: Failed to parse value: null >> > >> > at >> > >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501) >> > >> > at >> > >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480) >> > >> > at >> > >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFi >> > >> >le >> > >> >Pge >> > >> >Con >> > >> >figBuilder.java:77) >> > >> > ... 6 more >> > >> >Caused by: java.lang.NullPointerException >> > >> > at java.util.regex.Matcher.getTextLength(Matcher.java:1234) >> > >> > at java.util.regex.Matcher.reset(Matcher.java:308) >> > >> > at java.util.regex.Matcher.<init>(Matcher.java:228) >> > >> > at java.util.regex.Pattern.matcher(Pattern.java:1088) >> > >> > at >> > >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisR >> > >> >ep >> > >> >lac >> > >> >eme >> > >> >nt(PathUtils.java:321) >> > >> > at >> > >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(P >> > >> >at >> > >> >hUt >> > >> >ils >> > >> >.java:96) >> > >> > at >> > >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:488) >> > >> > ... 8 more >> > >> >Oct 10, 2014 5:54:13 PM >> > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre >> > >> >ad >> > >> >executeTaskLocally >> > >> >WARNING: Exception executing task: [feiCrawlerTaskName] locally: >> > >>Message: >> > >> >PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse >>value: >> > >> >null >> > >> > >> > >> > >> > >> > >> > >> > >> > >> >Content of tasks.xml >> > >> > >> > >> > >> > >> ><cas:tasks xmlns:cas="http://oodt.jpl.nasa.gov/1.0/cas";> >> > >> > <task id="urn:oodt:feiCrawlerTaskId" name="feiCrawlerTaskName" >> > >> >class="org.apache.oodt.cas.pge.StdPGETaskInstance"> >> > >> > <conditions/> >> > >> > >> > >> > <configuration> >> > >> > <property name="PGETask/Name" >>value="feiCrawlerTaskName"/> >> > >> > <property name="PGETask/ConfigFilePath" >> > >> >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt- >> > >> >de plo y/e xtensions/config/fei-crawler-pge-config.xml"/> >> > >> > <property name="PGETask/DumpMetadata" value="true"/> >> > >> > <property name="PGETask/WorkflowManagerUrl" >> > >> >value="http://localhost:9001"; /> >> > >> > <property name="PGETask/Query/FileManagerUrl" >> > >> >value="http://localhost:9000"; /> >> > >> > <property name="PGETask/Ingest/FileManagerUrl" >> > >> >value="http://localhost:9000"/> >> > >> > >> > >> > <property >>name="PGETask/Query/ClientTransferServiceFactory" >> > >> >value="org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferF >> > >> >ac >> > >> >tor >> > >> >y"/ >> > >> >> >> > >> > <property name="PGETask/Ingest/CrawlerConfigFile" >> > >> >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt- >> > >> >de >> > >> >plo >> > >> >y/c >> > >> >rawler/policy/crawler-config.xml"/> >> > >> > <property name="PGETask/Ingest/MimeExtractorRepo" >> > >> >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt- >> > >> >de plo y/e xtensions/policy/mime-extractor-map.xml"/> >> > >> > <property name="PGETask/Ingest/ActionIds" >> > >> >value="MoveFileToLevel0Dir"/> >> > >> > <property name="PGE_HOME" >> > >> >> > >>>>>value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pg >>>>>e" >> > >>>/> >> > >> > </configuration> >> > >> > >> > >> > <requiredMetFields/> >> > >> > </task> >> > >> ></cas:tasks> >> > >> > >> > >> > >> > >> >Contents of fei-crawler-pge-config.xml >> > >> > >> > >> > >> > >> ><pgeConfig> >> > >> > >> > >> > <!-- How to run the PGE --> >> > >> > <!-just echoing the current directory to a file so I can see if >> > >> >this thing ever starts --> >> > >> > <exe >> > >> >dir="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/ >> > >> >da >> > >> >ta/ >> > >> >pge >> > >> >/jobs" shell="/bin/sh"> >> > >> > <cmd>echo "Current Working Directory is `pwd`" > >> > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/logs/ >> > >> >pg >> > >> >e.l >> > >> >og< >> > >> >/cmd> >> > >> > </exe> >> > >> > >> > >> > <!-- Files to ingest --> >> > >> > <output> >> > >> > <!-- trying this approach. Telling PGE there is output files >> > >> >should invoke the crawler. --> >> > >> > >> > >> ><dir="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy >> > >> >/d >> > >> >ata >> > >> >/st >> > >> >aging"/> >> > >> > </output> >> > >> > >> > >> > <!-- Custom metadata to add to output files --> >> > >> > <customMetadata> >> > >> > >> > >> > <!-- each of these directories exist --> >> > >> > <metadata key="JobDir" val="[OODT_HOME]/data/pge/jobs" >> > >> >envReplace="true"/> >> > >> > <metadata key="JobInputDir" val="[JobDir]/input"/> >> > >> > <metadata key="JobOutputDir" val="[JobDir]/output"/> >> > >> > <metadata key="JobLogDir" val="[JobDir]/logs"/> >> > >> > >> > >> > </customMetadata> >> > >> ></pgeConfig> >> > >> > >> > >> >Valerie A. Mallder >> > >> > >> > >> >New Horizons Deputy Mission System Engineer The Johns Hopkins >> > >> >University/Applied Physics Laboratory >> > >> >11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723 >> > >> >240-228-7846 (Office) 410-504-2233 (Blackberry) >> > >> > >> > > >> > >
