Absolutely, here is a simple use case: If you look at CAS-PGE files, they typically include setup to “stage files” (already archived in File Manager) to the local job working directly. Many times this is just handled in the bash script (or Python or whatever) that CAS-PGE generates itself. However it’s such a repetitive task and *part of* the actual workflow for running a job, we added a simple interface to do so. See here:
https://raw.githubusercontent.com/apache/oodt/master/pge/src/test/resources/pge-config.xml And notice this block: <fileStaging dir="[StagingDir]" force="[ForceStaging]"> <stageFiles metadataKey="InputFiles" /> </fileStaging> What the above does is inspect the value of the InputFiles metadata as provided to CAS-PGE (even if it’s a list) and for each of those files, it “stages” them (aka uses the associated and declared DataTransferer for this CAS-PGE instance) to the directory defined by [StagingDir]. Cool huh? Cheers, Chris P.S. It doesn’t work now, but will shortly ;) On 7/19/17, 8:43 AM, "Mallder, Valerie" <valerie.mall...@jhuapl.edu> wrote: Yes, adding a note to CHANGES.txt is fine. The fix to the fileStaging feature of CAS-PGE sounds good, although I've never understood what that feature is supposed to do :) When you have some free time I would love to hear about what you are using it for. Thanks! -----Original Message----- From: Chris Mattmann [mailto:mattm...@apache.org] Sent: Wednesday, July 19, 2017 11:37 AM To: dev@oodt.apache.org Subject: Re: [VOTE] Apache OODT 1.1 RC #1 Thank you Val! I may spin another RC not only because of the below (which I think we should just note in CHANGES.txt (I am happy to add a note on it), but also b/c I found an interesting bug while testing OODT 1.1 RC #1 with DRAT – the bug relates to using fileStaging – it doesn’t explicitly set a Product.PRODUCT_STRUCTURE* and as so gets an NPE if you try and use it. I never encountered this before b/c I never used OODT’s native file staging support in CAS-PGE. It is a simple fix, and I think will add a lot of value (fileStaging is inoperable without it in CAS-PGE). Due to that, I will spin another RC shortly. Thank you! Cheers, Chris On 7/19/17, 7:55 AM, "Mallder, Valerie" <valerie.mall...@jhuapl.edu> wrote: +1 I downloaded the source and built it with "mvn package -skipTests" and it built ok. I did a system test by running some data through my pipeline that uses cas-workflow-1.1, cas-filemgr-1.1 and cas-pge-1.1 and everything worked as I expected. And it even helped find some things that I needed to tweak in my test environment :) The only problem I should note is that had to delete the workflow repository directory and allow cas-workflow-1.1 to create a repository from scratch. cas-workflow-1.1 reported an incompatibility with workflow repository that was created with cas-workflow-0.10 and has been used with cas-workflow-1.0 for a while now. (Also note that, if the repository directory exists but it is empty, workflow manager will crash. So, I had to delete the repository directory itself rather than just delete the contents within it.) The error I received is below. It looks like the change from lucene-core-2.0.0.jar to the lucene-core-6.1.0.jar caused this incompatibility between cas-workflow-1.0 and cas-workflow-1.1. Deleting my pipeline's workflow respository and allowing cas-workflow-1.1 to create a new one was not a problem for me, but I can't speak for everyone. Jul 19, 2017 10:45:02 AM org.apache.oodt.cas.workflow.instrepo.LuceneWorkflowInstanceRepository addWorkflowInstanceToCatalog WARNING: Unable to index workflow instance: [d8dad567-6c90-11e7-9e4e-7574e3a12106]: Message: Format version is not supported (resource BufferedChecksumIndexInput(MMapIndexInput(path="/project/oodt/dev/jedi/data/workflow/slothrop/segments"))): -1 (needs to be between 1071082519 and 1071082519). This version of Lucene only supports indexes created with release 5.0 and later. java.lang.NullPointerException Val -----Original Message----- From: Chris Mattmann [mailto:mattm...@apache.org] Sent: Tuesday, July 18, 2017 6:54 PM To: dev@oodt.apache.org Subject: [VOTE] Apache OODT 1.1 RC #1 Hi Folks, I have posted a 1st release candidate for the Apache OODT 1.1 release. The source code is at: https://dist.apache.org/repos/dist/dev/oodt/ For more detailed information, see the included CHANGES.txt file for details on release contents and latest changes. The release was made using the OODT release process, documented on the Wiki here: https://cwiki.apache.org/confluence/display/OODT/Release+Process The release was made from the OODT 1.1 tag at: https://github.com/apache/oodt/tree/1.1 A staged Maven repository is available at: https://repository.apache.org/content/repositories/orgapacheoodt-1012/ Please vote on releasing these packages as Apache OODT 1.1. The vote is open for at least the next 72 hours. Only votes from OODT PMC are binding, but folks are welcome to check the release candidate and voice their approval or disapproval. The vote passes if at least three binding +1 votes are cast. [ ] +1 Release the packages as Apache OODT 1.1 [ ] -1 Do not release the packages because... Thanks! Chris Mattmann P.S. Here is my +1.
