Re: Apache Airavata-OODT Integration
Hi Sanjaya, -Original Message- From: Sanjaya Medonsa sanjaya...@gmail.com Reply-To: d...@airavata.apache.org d...@airavata.apache.org Date: Monday, July 8, 2013 12:09 AM To: Airavata Dev d...@airavata.apache.org Cc: dev@oodt.apache.org dev@oodt.apache.org Subject: Re: Apache Airavata-OODT Integration Hi Chris, I have started looking at changing the current implementation to use file Name instead of product id. As per the current PGETask wrapper implementation, it takes two inputs (Product ID or file path at the remote location. If filePath is used force staging should be set. But I am not quite sure what it means by force staging). Force staging I believe controls whether or not the staged files are overwritten. If I am to use the current provisions in PGETaskWrapper, then remote file path (Not the file name) has to be given as input. I am not quite sure whether it is ideal to use file path instead of file name. You can easily generate the file path (which does not have to be remote, in fact, if you think about it, it could easily be local and in Apache OODT, we typically ensure it's local by using distributed filesystems like HDFS or NFS or Gluster to make remote files appear local by pushing that portion down into the distributed filesystem which we think does a better job of data movement :) ). To generate the file path you can use CAS-PGE SQLQuery facility that will allow you to look up e.g., $FileLocation/$Filename based on met fields, which in turn you can then feed into the path. If filename to use as input, then FilesStager needs to be customized to retrieve product references from file name. See above for an alternative. File manager client doesn't have a mechanism to retrieve product by file name. But it has mechanism to retrieve product by product name. I guess typically both are the same. Yeah, or the other easy mechanism is simply to issue a query, e.g., build yourself a Filename query and then query the FM Catalog. One drawback of this approach is that it doesn't support list of product names. The method getProductReferences which returns list of products is based on back end implementation that is based on product id, through actual input is product (Product with just product name set is not possible to as input). Please let me know your thoughts. See above. Cheers, Chris ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ On Mon, Jun 17, 2013 at 5:52 PM, Sanjaya Medonsa sanjaya...@gmail.comwrote: Thanks Chris. I'll update the implementation to use file name instead of OODT product id. Cheers, Sanjaya On Sun, Jun 16, 2013 at 12:51 AM, Mattmann, Chris A (398J) chris.a.mattm...@jpl.nasa.gov wrote: Hey Sanjaya, sure +1 use the Filename. It's not guaranteed to be unique, but you can easily just pop the first one off the top (latest) and take that (since it's sorted by product received time). You may check out the pcs-core module and some of its internal classes like FileManagerUtils to see some cool helper functions that could aid in this regard. Cheers, Chris ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Sanjaya Medonsa sanjaya...@gmail.com Reply-To: d...@airavata.apache.org d...@airavata.apache.org Date: Saturday, June 15, 2013 4:04 AM To: Airavata Dev d...@airavata.apache.org Subject: Re: Apache Airavata-OODT Integration Thanks Chris for your help! Working directory is available in JobExecutionContext in Airavata and directory can easily be retrieved. Issue in my case is that, from XBaya GUI I take product id as input not the file name. Internally file stager query the file manager using product id to retrieve product reference and corresponding file name to stage the file into input dir. Since this product id to file name mapping happens internally during the file staging, my implementation don't have access to filename unless I query the file manager to retrieve the corresponding file name using product id. One of the
Re: Review Request 12308: OODT-619 [sub task of OODT-219] Implement the capability to pull status of resource nodes from ganglia and provide loadvalues to AssignmentMonitor
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/12308/#review22815 --- Ship it! Ship It! - Chris Mattmann On July 7, 2013, 8:44 p.m., Rajith Siriwardana wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/12308/ --- (Updated July 7, 2013, 8:44 p.m.) Review request for oodt and Chris Mattmann. Bugs: OODT-619 https://issues.apache.org/jira/browse/OODT-619 Repository: oodt Description --- Completed the implementation of GangliaResourceMonitor to provide load values normalize between 0 and its capacity. initially added only weightedAverageLoad calculation(load_one, load_five, load_fifteen). decisions: if a particular node is offline/not listed in the nodeMap/any error occurred during getting the load value, then the node capacity will be returned as the load value.one minute interval for updating the loadMap. loadMap moved to GangliaResourceMonitor modifications done to AssignmentMonitor,AssignmentMonitorFactory, GenericResourcemanagementObjectFactory, resource.properties, TestAssignmentMonitor, TestXmlRpcResourceManaber, Diffs - trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/AssignmentMonitor.java 1500522 trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/AssignmentMonitorFactory.java 1500522 trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ResourceMonitor.java PRE-CREATION trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ResourceMonitorFactory.java PRE-CREATION trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/exceptions/GangliaMonitorException.java PRE-CREATION trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/exceptions/MonitoringException.java 1500522 trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/GangliaAdapter.java PRE-CREATION trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/GangliaResourceMonitor.java PRE-CREATION trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/GangliaResourceMonitorFactory.java PRE-CREATION trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/loadcalc/LoadCalculator.java PRE-CREATION trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/loadcalc/LoadCalculatorFactory.java PRE-CREATION trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/loadcalc/WeightedAverageLoadCalc.java PRE-CREATION trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/loadcalc/WeightedAverageLoadCalcFactory.java PRE-CREATION trunk/resource/src/main/java/org/apache/oodt/cas/resource/util/GenericResourceManagerObjectFactory.java 1500522 trunk/resource/src/main/resources/resource.properties 1500522 trunk/resource/src/test/org/apache/oodt/cas/resource/monitor/TestAssignmentMonitor.java 1500522 trunk/resource/src/test/org/apache/oodt/cas/resource/monitor/TestGangliaResourceMonitor.java PRE-CREATION trunk/resource/src/test/org/apache/oodt/cas/resource/monitor/TestGangliaXMLParser.java 1500522 trunk/resource/src/test/org/apache/oodt/cas/resource/monitor/utils/MockGmetad.java PRE-CREATION trunk/resource/src/test/org/apache/oodt/cas/resource/system/TestXmlRpcResourceManager.java 1500522 trunk/resource/src/testdata/gangliaXMLdump.xml 1500522 trunk/resource/src/testdata/resourcemon/gangliaXMLdump.xml PRE-CREATION trunk/resource/src/testdata/resourcemon/nodes.xml PRE-CREATION trunk/resource/src/testdata/test.resource.properties PRE-CREATION Diff: https://reviews.apache.org/r/12308/diff/ Testing --- GangliaResourceMonitor test scenarios added to getLoad, removeNodeById, updateLoad. Other test scenarios modified accordingly. Thanks, Rajith Siriwardana
Re: Review Request 12308: OODT-619 [sub task of OODT-219] Implement the capability to pull status of resource nodes from ganglia and provide loadvalues to AssignmentMonitor
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/12308/#review22814 --- trunk/resource/src/main/java/org/apache/oodt/cas/resource/util/GenericResourceManagerObjectFactory.java https://reviews.apache.org/r/12308/#comment46530 Weird formatting changes? - Chris Mattmann On July 7, 2013, 8:44 p.m., Rajith Siriwardana wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/12308/ --- (Updated July 7, 2013, 8:44 p.m.) Review request for oodt and Chris Mattmann. Bugs: OODT-619 https://issues.apache.org/jira/browse/OODT-619 Repository: oodt Description --- Completed the implementation of GangliaResourceMonitor to provide load values normalize between 0 and its capacity. initially added only weightedAverageLoad calculation(load_one, load_five, load_fifteen). decisions: if a particular node is offline/not listed in the nodeMap/any error occurred during getting the load value, then the node capacity will be returned as the load value.one minute interval for updating the loadMap. loadMap moved to GangliaResourceMonitor modifications done to AssignmentMonitor,AssignmentMonitorFactory, GenericResourcemanagementObjectFactory, resource.properties, TestAssignmentMonitor, TestXmlRpcResourceManaber, Diffs - trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/AssignmentMonitor.java 1500522 trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/AssignmentMonitorFactory.java 1500522 trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ResourceMonitor.java PRE-CREATION trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ResourceMonitorFactory.java PRE-CREATION trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/exceptions/GangliaMonitorException.java PRE-CREATION trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/exceptions/MonitoringException.java 1500522 trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/GangliaAdapter.java PRE-CREATION trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/GangliaResourceMonitor.java PRE-CREATION trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/GangliaResourceMonitorFactory.java PRE-CREATION trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/loadcalc/LoadCalculator.java PRE-CREATION trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/loadcalc/LoadCalculatorFactory.java PRE-CREATION trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/loadcalc/WeightedAverageLoadCalc.java PRE-CREATION trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/loadcalc/WeightedAverageLoadCalcFactory.java PRE-CREATION trunk/resource/src/main/java/org/apache/oodt/cas/resource/util/GenericResourceManagerObjectFactory.java 1500522 trunk/resource/src/main/resources/resource.properties 1500522 trunk/resource/src/test/org/apache/oodt/cas/resource/monitor/TestAssignmentMonitor.java 1500522 trunk/resource/src/test/org/apache/oodt/cas/resource/monitor/TestGangliaResourceMonitor.java PRE-CREATION trunk/resource/src/test/org/apache/oodt/cas/resource/monitor/TestGangliaXMLParser.java 1500522 trunk/resource/src/test/org/apache/oodt/cas/resource/monitor/utils/MockGmetad.java PRE-CREATION trunk/resource/src/test/org/apache/oodt/cas/resource/system/TestXmlRpcResourceManager.java 1500522 trunk/resource/src/testdata/gangliaXMLdump.xml 1500522 trunk/resource/src/testdata/resourcemon/gangliaXMLdump.xml PRE-CREATION trunk/resource/src/testdata/resourcemon/nodes.xml PRE-CREATION trunk/resource/src/testdata/test.resource.properties PRE-CREATION Diff: https://reviews.apache.org/r/12308/diff/ Testing --- GangliaResourceMonitor test scenarios added to getLoad, removeNodeById, updateLoad. Other test scenarios modified accordingly. Thanks, Rajith Siriwardana
[VOTE] Apache OODT 0.6 Release Candidate #1
Hi Guys, A first candidate for the Apache OODT 0.6 release is available at: http://people.apache.org/~mattmann/apache-oodt-0.6/rc1/ The release candidate is a zip archive of the sources in: http://svn.apache.org/repos/asf/oodt/tags/0.6/ The SHA1 checksum of the archive is 32a343a2b23799d39de4d37894b8184b3875b7fd A staged M2 repository can also be found on repository.apache.org here: https://repository.apache.org/content/repositories/orgapacheoodt-114/ Please vote on releasing this package as Apache OODT 0.6. The vote is open for the next 72 hours and passes if a majority of at least three +1 OODT PMC votes are cast. [ ] +1 Release this package as Apache OODT 0.6 [ ] -1 Do not release this package because... Here is my +1 for the release. Cheers, Chris ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++
Build failed in Jenkins: oodt-trunk #1010
See https://builds.apache.org/job/oodt-trunk/1010/ -- [...truncated 3536 lines...] AUworkflow/src/main/resources/examples/timeout.workflow.xml A workflow/src/main/resources/examples/workflow-instance-met.xml A workflow/src/main/resources/examples/longWorkflow.workflow.xml A workflow/src/main/resources/examples/events.xml A workflow/src/main/resources/examples/backwardsTestWorkflow.workflow.xml A workflow/src/main/resources/examples/workflow-lifecycle.xml A workflow/src/main/resources/examples/testMetError.workflow.xml A workflow/src/main/resources/examples/conditions.xml A workflow/src/site A workflow/src/site/site.xml A workflow/src/site/resources A workflow/src/site/resources/images AUworkflow/src/site/resources/images/cas-workflow.psd AUworkflow/src/site/resources/images/warning.png AUworkflow/src/site/resources/images/wm_use_case1.jpg AUworkflow/src/site/resources/images/success.png AUworkflow/src/site/resources/images/error.png AUworkflow/src/site/resources/images/wm_object_model.png AUworkflow/src/site/resources/images/cas-workflow.jpg AUworkflow/src/site/resources/images/info.png AUworkflow/src/site/resources/images/wm_extension_points.png A workflow/src/site/resources/css A workflow/src/site/resources/css/guides.css A workflow/src/site/xdoc A workflow/src/site/xdoc/development A workflow/src/site/xdoc/development/developer.xml AUworkflow/src/site/xdoc/development/maven.xml A workflow/src/site/xdoc/user A workflow/src/site/xdoc/user/basic.xml A workflow/src/site/xdoc/user/advanced.xml A workflow/src/site/xdoc/user/index.xml A workflow/src/testdata A workflow/src/testdata/test.logging.properties A workflow/src/testdata/myScript.sh A workflow/src/testdata/testinstrepo AUworkflow/src/testdata/testinstrepo/_43.cfs AUworkflow/src/testdata/testinstrepo/deletable AUworkflow/src/testdata/testinstrepo/segments A workflow/src/testdata/workflow.sql A workflow/pom.xml A README.txt A KEYS A commons A commons/src A commons/src/test A commons/src/test/org A commons/src/test/org/apache A commons/src/test/org/apache/oodt A commons/src/test/org/apache/oodt/commons A commons/src/test/org/apache/oodt/commons/AbstractTestCase.java A commons/src/test/org/apache/oodt/commons/NaughtyTest.java A commons/src/test/org/apache/oodt/commons/AppTest.java A commons/src/test/org/apache/oodt/commons/ConfiguredTestCaseTest.java A commons/src/test/org/apache/oodt/commons/MultiServerTest.java A commons/src/test/org/apache/oodt/commons/exec AUcommons/src/test/org/apache/oodt/commons/exec/TestEnvUtilities.java A commons/src/test/org/apache/oodt/commons/filter AU commons/src/test/org/apache/oodt/commons/filter/TestTimeEventWeightedHash.java A commons/src/test/org/apache/oodt/commons/net A commons/src/test/org/apache/oodt/commons/net/NetTest.java A commons/src/test/org/apache/oodt/commons/object A commons/src/test/org/apache/oodt/commons/object/jndi A commons/src/test/org/apache/oodt/commons/object/jndi/TestContext.java A commons/src/test/org/apache/oodt/commons/object/jndi/TestContextTest.java A commons/src/test/org/apache/oodt/commons/object/jndi/ObjectContextTest.java A commons/src/test/org/apache/oodt/commons/ConfigurationTest.java A commons/src/test/org/apache/oodt/commons/io A commons/src/test/org/apache/oodt/commons/io/FixedBufferOutputStreamTest.java A commons/src/test/org/apache/oodt/commons/io/NullInputStreamTest.java A commons/src/test/org/apache/oodt/commons/io/NullOutputStreamTest.java AU commons/src/test/org/apache/oodt/commons/io/TestLoggerOutputStream.java A commons/src/test/org/apache/oodt/commons/activity AU commons/src/test/org/apache/oodt/commons/activity/CompositeActivityTest.java AUcommons/src/test/org/apache/oodt/commons/activity/HistoryTest.java AU commons/src/test/org/apache/oodt/commons/activity/ActivityTrackerTest.java AU commons/src/test/org/apache/oodt/commons/activity/ActivityStoppedTest.java AUcommons/src/test/org/apache/oodt/commons/activity/ActivityTest.java AU commons/src/test/org/apache/oodt/commons/activity/DatagramLoggingActivityFactoryTest.java AUcommons/src/test/org/apache/oodt/commons/activity/IncidentTest.java A commons/src/test/org/apache/oodt/commons/util A commons/src/test/org/apache/oodt/commons/util/EnterpriseEntityResolverTest.java A commons/src/test/org/apache/oodt/commons/util/LDAPTest.java A
