+1 billion --Paul
Sent from my iPhone > On Oct 8, 2014, at 5:55 PM, Lewis John Mcgibbney <[email protected]> > wrote: > > Folks, > Is it possible to create a parent issue for defining XSD's for all of the > XML file we need ti OODT? > I do not know them all, but from this thread alone, it is clear that we > could do with setting some kind of restrictions on what can be included > within task and configuration XML within OODT. > Thoughts? > Lewis > > On Wed, Oct 8, 2014 at 5:44 PM, Verma, Rishi (398J) < > [email protected]> wrote: > >> Hi Val, >> >> Yep - here’s a link to the tasks.xml file: >> >> https://github.com/riverma/xdata-jpl-netscan/blob/master/oodt-netscan/workflow/src/main/resources/policy/tasks.xml >> >>> The problem is that the ExternScriptTaskInstance is unable to recognize >> the command line arguments that I want to pass to the crawler_launcher >> script. >> >> >> Hmm.. could you share your workflow manager log, or better yet, the >> batch_stub output? Curious to see what error is thrown. >> >> Is a script file being generated for your PGE? For example, inside your >> [PGE_HOME] directory, and within the particular job directory created for >> your execution of a workflow, you will see some files starting with >> “sciPgeExeScript_…”. You’ll find one for your pgeConfig, and you can check >> to see what the PGE commands actually translate into, with respect to a >> shell script format. If that file is there, take a look at it, and validate >> whether the command works within the script (i.e. copy/paste and run the >> crawler command manually). >> >> Another suggestion is to take a step back, and build up slowly, i.e.: >> 1. Do an “echo” command within your PGE first. (e.g. <cmd> echo “Hello >> APL.” > /tmp/test.txt</cmd>) >> 2. If above works, do a crawler_launcher empty command(e.g. >> <cmd>/path/to/oodt/crawler/bin/crawler_launcher</cmd>) and verify the >> batch_stub or Workflow Manager prints some kind of output when you run the >> workflow. >> 3. Build up your crawler_launcher command piece by piece to see where it >> is failing >> >> Thanks, >> Rishi >> >> On Oct 8, 2014, at 4:24 PM, Mallder, Valerie <[email protected]> >> wrote: >> >>> Hi Rishi, >>> >>> Thank you very much for pointing me to your working example. This is >> very helpful. My pgeConfig looks very similar to yours. So, I commented >> out the resource manager like you suggested and tried running again without >> the resource manager. And my problem still exists. The problem is that the >> ExternScriptTaskInstance is unable to recognize the command line arguments >> that I want to pass to the crawler_launcher script. Could you send me a >> link to your tasks.xml file? I'm curious as to how you defined your task. >> My pgeConfig and tasks.xml are below. >>> >>> Thanks! >>> Val >>> >>> >>> <?xml version="1.0" encoding="UTF-8"?> >>> <pgeConfig> >>> >>> <!-- How to run the PGE --> >>> <exe dir="[JobDir]" shell="/bin/sh" envReplace="true"> >>> <cmd>[CRAWLER_HOME]/bin/crawler_launcher --operation >> --launchAutoCrawler \ >>> --filemgrUrl [FILEMGR_URL] \ >>> --clientTransferer >> org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory \ >>> --productPath [JobInputDir] \ >>> --mimeExtractorRepo >> [OODT_HOME]/extensions/policy/mime-extractor-map.xml \ >>> --actionIds MoveFileToLevel0Dir</cmd> >>> </exe> >>> >>> <!-- Files to ingest --> >>> <output/> >>> </output> >>> >>> <!-- Custom metadata to add to output files --> >>> <customMetadata> >>> <metadata key="JobDir" val="[OODT_HOME]"/> >>> <metadata key="JobInputDir" val="[FEI_DROP_DIR]"/> >>> <metadata key="JobOutputDir" val="[JobDir]/data/pge/jobs"/> >>> <metadata key="JobLogDir" val="[JobDir]/data/pge/logs"/> >>> </customMetadata> >>> >>> </pgeConfig> >>> >>> >>> >>> <!-- tasks.xml **************************************************--> >>> >>> <cas:tasks xmlns:cas="http://oodt.jpl.nasa.gov/1.0/cas";> >>> >>> <task id="urn:oodt:crawlerLauncherId" name="crawlerLauncherName" >> class="org.apache.oodt.cas.workflow.examples.ExternScriptTaskInstance"> >>> <conditions/> <!-- There are no pre execution conditions right now >> --> >>> <configuration> >>> >>> <property name="ShellType" value="/bin/sh" /> >>> <property name="PathToScript" >> value="[CRAWLER_HOME]/bin/crawler_launcher" envReplace="true" /> >>> >>> <property name="PGETask_Name" value="crawler_launcher PGE >> Task"/> >>> <property name="PGETask_ConfigFilePath" >> value="[OODT_HOME]/extensions/config/crawler-pge-config.xml" >> envReplace="true" /> >>> </configuration> >>> </task> >>> >>> </cas:tasks> >>> >>> Valerie A. Mallder >>> New Horizons Deputy Mission System Engineer >>> Johns Hopkins University/Applied Physics Laboratory >>> >>> >>>> -----Original Message----- >>>> From: Verma, Rishi (398J) [mailto:[email protected]] >>>> Sent: Wednesday, October 08, 2014 6:01 PM >>>> To: [email protected] >>>> Subject: Re: what is batch stub? Is it necessary? >>>> >>>> Hi Valerie, >>>> >>>>>>>> All I am trying to do is run "crawler_launcher" as a workflow task >>>>>>>> in the CAS PGE environment. >>>> >>>> Interesting. I have a working example here [1] you can look at that >> does this exact >>>> thing. >>>> >>>>>>>> So, if "batchstub" is necessary in this scenario, pleast tell me >>>>>>>> what it is, why it is necessary, and how to run it (please provide >>>>>>>> exact syntax to put in my startup shell script, because I would >>>>>>>> never be able to figure it out for myself and I don't want to have >>>>>>>> to bother everyone again.) >>>> >>>> Batchstub is only necessary if your Workflow Manger is sending jobs to >> Resource >>>> Manager for execution (where the default execution is to run the job in >> something >>>> called a ?batch stub? executable). Think of batch stubs as a small >> wrapper >>>> program that takes a bundle of executable instructions from Resource >> Manager, >>>> and executes them in a shell environment within a given remote (or >> local) machine. >>>> >>>> Here?s my suggestion: >>>> 1. Like Paul suggested, go to $OODT_HOME/resmgr/bin, and execute the >>>> following command (it?ll start a batch stub in a terminal on port 2001): >>>>> ./batch_stub 2001 >>>> >>>> If the above step doesn?t fix your problem, you can also try having >> Workflow >>>> Manager NOT send jobs to Resource Manager for execution, and instead >> execute >>>> jobs locally through Workflow Manager itself (on localhost only!). To >> disable job >>>> transfer to Resource Manger, you?ll need to modify the Workflow Manager >>>> properties file ($OODT_HOME/wmgr/etc/workflow.properties), and >> specifically >>>> comment out the ?org.apache.oodt.cas.workflow.engine.resourcemgr.url? >> line. >>>> I?ve done this in my example code below, see [2] for an exact example >> of this. >>>> After modifying workflow.properties, make sure to restart workflow >> manager >>>> ($OODT_HOME/wmgr/bin/wmgr stop followed by $OODT_HOME/wmgr/bin/wmgr >>>> start). >>>> >>>> Thanks, >>>> Rishi >>>> >>>> [1] https://github.com/riverma/xdata-jpl-netscan/blob/master/oodt- >> netscan/pge/src/main/resources/policy/netscan-getipv4entriesrandomsample.xml >>>> [2] https://github.com/riverma/xdata-jpl-netscan/blob/master/oodt- >>>> netscan/workflow/src/main/resources/etc/workflow.properties >>>> >>>> On Oct 8, 2014, at 2:31 PM, Ramirez, Paul M (398J) >>>> <[email protected]> wrote: >>>> >>>>> Valerie, >>>>> >>>>> I would have thought it would have just not used a batch stub by >> default. That >>>> said if you go into the $OODT_HOME/resmgr/bin there should be a script >> to start a >>>> batch stub. Right now on my phone I forget the name of the script but >> if you more >>>> the file you will see the Java class name that corresponds to below. >> You should >>>> specify a port when you run the script which from the looks of the >> output below >>>> should be 2001. >>>>> >>>>> HTH, >>>>> Paul R >>>>> >>>>> Sent from my iPhone >>>>> >>>>>> On Oct 8, 2014, at 2:04 PM, Mallder, Valerie < >> [email protected]> >>>> wrote: >>>>>> >>>>>> Well then, I'm proud to be a member :) (I think .... ) >>>>>> >>>>>> >>>>>> Valerie A. Mallder >>>>>> New Horizons Deputy Mission System Engineer Johns Hopkins >>>>>> University/Applied Physics Laboratory >>>>>> >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Bruce Barkstrom [mailto:[email protected]] >>>>>>> Sent: Wednesday, October 08, 2014 4:54 PM >>>>>>> To: [email protected] >>>>>>> Subject: Re: what is batch stub? Is it necessary? >>>>>>> >>>>>>> You have every right to bother everyone. >>>>>>> You won't get what you need unless you do. >>>>>>> >>>>>>> You get one honorary membership in the Society of General Agitators >>>>>>> - at the rank of Major Agitator. >>>>>>> >>>>>>> Bruce B. >>>>>>> >>>>>>> On Wed, Oct 8, 2014 at 4:49 PM, Mallder, Valerie >>>>>>> <[email protected] >>>>>>>> wrote: >>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> I am still having trouble getting my CAS PGE crawler task to run >>>>>>>> due to >>>>>>>> http://localhost:2001 being "down". I have spent the last 2 days >>>>>>>> tracing through the resource manager code and tracked this down to >>>>>>>> line 146 of LRUScheduler where the XmlRpcBatchMgr is failing to >>>>>>>> execute the task remotely, because on line 75 of >>>>>>>> XmlRpcBatchMgrProxy (that was instantiated by XmlRpcBatchMgr on its >>>>>>>> line 74) is trying to call "isAlive" on the webservice named >>>>>>>> "batchstub" which, to my knowledge, is not running because I have >> not done >>>> anything explicitly to run it. >>>>>>>> >>>>>>>> All I am trying to do is run "crawler_launcher" as a workflow task >>>>>>>> in the CAS PGE environment. I had it running perfectly before I >>>>>>>> started trying to make it run as part of a workflow. I really miss >>>>>>>> my crawler and really want it to run again L >>>>>>>> >>>>>>>> So, if "batchstub" is necessary in this scenario, pleast tell me >>>>>>>> what it is, why it is necessary, and how to run it (please provide >>>>>>>> exact syntax to put in my startup shell script, because I would >>>>>>>> never be able to figure it out for myself and I don't want to have >>>>>>>> to bother everyone again.) >>>>>>>> >>>>>>>> Thanks so much! >>>>>>>> >>>>>>>> Val >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Valerie A. Mallder >>>>>>>> >>>>>>>> New Horizons Deputy Mission System Engineer The Johns Hopkins >>>>>>>> University/Applied Physics Laboratory >>>>>>>> 11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723 >>>>>>>> 240-228-7846 (Office) 410-504-2233 (Blackberry) >>>> >>>> --- >>>> Rishi Verma >>>> NASA Jet Propulsion Laboratory >>>> California Institute of Technology >> >> --- >> Rishi Verma >> NASA Jet Propulsion Laboratory >> California Institute of Technology > > > -- > *Lewis*
