Re: Review Request 14626: OODT 562 - Exposing parent-child & element addition/deletion functionality for product types using cas-curator
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14626/#review26979 --- Ship it! Ship It! - Chris Mattmann On Oct. 13, 2013, 10:45 p.m., Varun Ratnakar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/14626/ > --- > > (Updated Oct. 13, 2013, 10:45 p.m.) > > > Review request for oodt and Chris Mattmann. > > > Bugs: OODT-562 > https://issues.apache.org/jira/browse/OODT-562 > > > Repository: oodt > > > Description > --- > > This patch is for https://issues.apache.org/jira/browse/OODT-562 > > 1. It adds some functions to the cas-curator's Metadata service > 2. Added functions to the file-manager in XMLValidationLayer for > adding/removing/retrieving parents of product types) > > > Diffs > - > > > trunk/curator/src/main/java/org/apache/oodt/cas/curation/service/MetadataResource.java > 1523273 > > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/validation/XMLValidationLayer.java > 1523273 > > Diff: https://reviews.apache.org/r/14626/diff/ > > > Testing > --- > > > Thanks, > > Varun Ratnakar > >
Re: Question regarding OODT patch 563 (Workflow API additions)
Hey Varun, -Original Message- From: Varun Ratnakar Date: Monday, October 14, 2013 2:11 PM To: jpluser Subject: Re: Question regarding OODT patch 563 (Workflow API additions) >Ah I see Chris. Thanks ! > >So, I'm assuming that I add this functionality in pcs-services ? PCS services would be a good place, or maybe creating like a workflow JAX-RS services package. PCS services typically are services that extend beyond multiple individual CAS components. > i.e. extend the API in there to interface with >PackagedWorkflowRepository in order to be able to add/remove dynamic >tasks. See above, I would work on defining a new API e.g., and create a webapp/wm-services/ App based on Apache CXF/JAX-RS similar to the new FMprod stuff. > > >I had some questions, if you don't mind: >1. What do you mean by dynamic tasks in this context ? Tasks that are added e.g., when a Dynamic workflow is created (see OODT-215 and OODT-491) > >2. After the tasks are added, how will the workflow be run ? The same way as if the workflow originally existed in the repo. >3. Will the workflow manager will have to configured in a certain way to >load tasks from this PackagedWorkflowRepository directory ? Nope, the only issue is how to save dynamic workflows after restart, which I'm working on. Cheers, Chris > >Regards, >Varun Cheers, Chris > > On 14 Oct 2013, at 19:33, "Mattmann, Chris A (398J)" > wrote: > >> Hey Varun, >> >> The point about the PackagedWorkflowRepository is that it adds dynamic >> tasks >> to the workflow repo -- thoughts? >> >> Cheers, >> Chris >> >> ++ >> Chris Mattmann, Ph.D. >> Senior Computer Scientist >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >> Office: 171-266B, Mailstop: 171-246 >> Email: chris.a.mattm...@nasa.gov >> WWW: http://sunset.usc.edu/~mattmann/ >> ++ >> Adjunct Assistant Professor, Computer Science Department >> University of Southern California, Los Angeles, CA 90089 USA >> ++ >> >> >> >> >> >> >> -Original Message- >> From: Varun Ratnakar >> Date: Monday, October 14, 2013 9:39 AM >> To: jpluser >> Subject: Re: Question regarding OODT patch 563 (Workflow API additions) >> >>> Hi Chris, >>> >>> just took a look at PackagedWorkflowRepository & the files in the >>> tar-ball you sent a link to. >>> >>> I noticed that the PackagedWorkflowRepository loads workflow files and >>> allows one to: >>> - Add a task >>> - Add a workflow >>> >>> I also see that the files you sent have code for >>>PoicyAwareWorkflowTasks. >>> >>> I'm afraid I'm unable to connect the two. What did you have in mind ? >>> >>> Kind regards, >>> Varun >>> >>> >>> >>> On 12 Oct 2013, at 20:23, "Mattmann, Chris A (398J)" >>> wrote: >>> >>>> Hey Varun, >>>> >>>> See the PackagedWorkflowRepository (I do it there). >>>> >>>> Also see the work done here: >>>> >>>> https://issues.apache.org/jira/browse/OODT-213 >>>> >>>> >>>> I uploaded a series of the Java files that implement this from >>>> OCO in 2007 (needs to be ALv2'ed, and cleaned up). It's a tar-ball >>>> on that issue. >>>> >>>> Check those out and let's talk. >>>> >>>> Cheers, >>>> Chris >>>> >>>> ++ >>>> Chris Mattmann, Ph.D. >>>> Senior Computer Scientist >>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>> Office: 171-266B, Mailstop: 171-246 >>>> Email: chris.a.mattm...@nasa.gov >>>> WWW: http://sunset.usc.edu/~mattmann/ >>>> ++ >>>> Adjunct Assistant Professor, Computer Science Department >>>> University of Southern California, Los Angeles, CA 90089 USA >>>> ++ >>>> >>>> >>>> >>>> >>>> >>>> >>>> -Original Message- >>>> From: Varun Ratnakar >>>> Date: Friday, October 11, 2013 4:55 PM >>>> To: Chris Mattmann >>>> Subject: Question regarding OODT patch 563 (Workflow API additions) >>>> >>>>> Hi Chris, >>>>> >>>>> You had mentioned making the home for patch 563 in pcs-services. Did >>>>> you >>>>> have any pointers for me to look at regarding this ? >>>>> >>>>> What we basically need is: >>>>> >>>>> a) A way to add tasks with some configuration (inputs, outputs, etc) >>>>> b) A way to invoke a workflow which includes the added tasks >>>>> >>>>> Note that the tasks in a) above doesn't need to be persistent. In >>>>>fact, >>>>> making it persistent might be counter-productive as creating a new >>>>>task >>>>> for each instantiation of a component might result in a large number >>>>>of >>>>> tasks being added in. >>>>> >>>>> Regards, >>>>> Varun >>>>> >>>> >>>> >>> >> >> >
Re: Question regarding OODT patch 563 (Workflow API additions)
Hey Varun, Thanks, yep it is possible. Those parameters are the dynamic "metadata" parameters and should be used to override anything statically set from the workflow config. Cheers, Chris -Original Message- From: Varun Ratnakar Reply-To: Date: Tuesday, October 15, 2013 2:24 AM To: Chris Mattmann Cc: Subject: Re: Question regarding OODT patch 563 (Workflow API additions) >Thanks Chris ! > >I'll get to work on creating the webapp/wm-services JAX-RS module. > >Question: >Is it possible to run a stored workflow and specify some configuration >parameters of it's tasks at run-time (i.e. just before submitting the >workflow for execution) ? > >Regards, >Varun > > >On 15 Oct 2013, at 04:48, Chris Mattmann wrote: > >> Hey Varun, >> >> -Original Message- >> >> From: Varun Ratnakar >> Date: Monday, October 14, 2013 2:11 PM >> To: jpluser >> Subject: Re: Question regarding OODT patch 563 (Workflow API additions) >> >>> Ah I see Chris. Thanks ! >>> >>> So, I'm assuming that I add this functionality in pcs-services ? >> >> PCS services would be a good place, or maybe creating like a workflow >> JAX-RS services package. >> PCS services typically are services that extend beyond multiple >>individual >> CAS components. >> >>> i.e. extend the API in there to interface with >>> PackagedWorkflowRepository in order to be able to add/remove dynamic >>> tasks. >> >> See above, I would work on defining a new API e.g., and create a >> webapp/wm-services/ App based on Apache CXF/JAX-RS similar to >> the new FMprod stuff. >> >>> >>> >>> I had some questions, if you don't mind: >>> 1. What do you mean by dynamic tasks in this context ? >> >> Tasks that are added e.g., when a Dynamic workflow is created (see >> OODT-215 and OODT-491) >> >>> >>> 2. After the tasks are added, how will the workflow be run ? >> >> The same way as if the workflow originally existed in the repo. >> >>> 3. Will the workflow manager will have to configured in a certain way >>>to >>> load tasks from this PackagedWorkflowRepository directory ? >> >> Nope, the only issue is how to save dynamic workflows after restart, >>which >> I'm working on. >> >> Cheers, >> Chris >> >>> >>> Regards, >>> Varun >> >> Cheers, >> Chris >> >>> >>> On 14 Oct 2013, at 19:33, "Mattmann, Chris A (398J)" >>> wrote: >>> >>>> Hey Varun, >>>> >>>> The point about the PackagedWorkflowRepository is that it adds dynamic >>>> tasks >>>> to the workflow repo -- thoughts? >>>> >>>> Cheers, >>>> Chris >>>> >>>> ++ >>>> Chris Mattmann, Ph.D. >>>> Senior Computer Scientist >>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>> Office: 171-266B, Mailstop: 171-246 >>>> Email: chris.a.mattm...@nasa.gov >>>> WWW: http://sunset.usc.edu/~mattmann/ >>>> ++ >>>> Adjunct Assistant Professor, Computer Science Department >>>> University of Southern California, Los Angeles, CA 90089 USA >>>> ++ >>>> >>>> >>>> >>>> >>>> >>>> >>>> -Original Message- >>>> From: Varun Ratnakar >>>> Date: Monday, October 14, 2013 9:39 AM >>>> To: jpluser >>>> Subject: Re: Question regarding OODT patch 563 (Workflow API >>>>additions) >>>> >>>>> Hi Chris, >>>>> >>>>> just took a look at PackagedWorkflowRepository & the files in the >>>>> tar-ball you sent a link to. >>>>> >>>>> I noticed that the PackagedWorkflowRepository loads workflow files >>>>>and >>>>> allows one to: >>>>> - Add a task >>>>> - Add a workflow >>>>> >>>>> I also see that the files you sent have code for >>>>> PoicyAwareWorkflowTasks. >>>>> >>>>> I'm afraid I'm unable to connect the two.
Re: Review Request 14513: OODT-612 & OODT-613 - Content writers and configurations for the CAS Product Server Web Application (JAX-RS part)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14513/#review27055 --- trunk/webapp/fmprod/pom.xml <https://reviews.apache.org/r/14513/#comment52682> may want to upgrade this to 1.4 but not a blocker - Chris Mattmann On Oct. 6, 2013, 10:37 a.m., Ross Laidlaw wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/14513/ > --- > > (Updated Oct. 6, 2013, 10:37 a.m.) > > > Review request for oodt, Chris Mattmann and Rishi Verma. > > > Bugs: OODT-612 and OODT-613 > https://issues.apache.org/jira/browse/OODT-612 > https://issues.apache.org/jira/browse/OODT-613 > > > Repository: oodt > > > Description > --- > > This patch aims to address issues OODT-612 and OODT-613. Please see the JIRA > issues for discussions on the chosen designs/implementations. > > The Java code for this patch is contained in a new package > 'org.apache.oodt.cas.product.jaxrs'. This package is intended to replace the > 'org.apache.oodt.cas.product.service' package that was committed for the GSoC > (Google Summer of Code 2013) midterm, i.e. the cas.product.service package > will be deleted and replaced with this reworked design. > > Documentation for this work has been added to the OODT wiki as detailed on > OODT-614. > > Work is ongoing for this package and many improvements can be made, for > example those mentioned in the 'development ideas' section at the bottom of > the developer guide. > > > Diffs > - > > trunk/webapp/fmprod/pom.xml 1517156 > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/configurations/RdfConfiguration.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/configurations/RssConfiguration.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/configurations/RssNamespace.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/configurations/RssTag.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/configurations/RssTagAttribute.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/configurations/package-info.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/exceptions/BadRequestException.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/exceptions/InternalServerErrorException.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/exceptions/NotFoundException.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/exceptions/package-info.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/filters/BackwardsCompatibleInterceptor.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/filters/package-info.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/package-info.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/resources/DatasetResource.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/resources/MetadataResource.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/resources/ProductResource.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/resources/ReferenceResource.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/resources/TransferResource.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/resources/TransfersResource.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/resources/package-info.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/services/CasProductJaxrsService.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/services/package-info.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/produ
Re: Review Request 14513: OODT-612 & OODT-613 - Content writers and configurations for the CAS Product Server Web Application (JAX-RS part)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14513/#review27056 --- Ship it! Ship It! - Chris Mattmann On Oct. 6, 2013, 10:37 a.m., Ross Laidlaw wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/14513/ > --- > > (Updated Oct. 6, 2013, 10:37 a.m.) > > > Review request for oodt, Chris Mattmann and Rishi Verma. > > > Bugs: OODT-612 and OODT-613 > https://issues.apache.org/jira/browse/OODT-612 > https://issues.apache.org/jira/browse/OODT-613 > > > Repository: oodt > > > Description > --- > > This patch aims to address issues OODT-612 and OODT-613. Please see the JIRA > issues for discussions on the chosen designs/implementations. > > The Java code for this patch is contained in a new package > 'org.apache.oodt.cas.product.jaxrs'. This package is intended to replace the > 'org.apache.oodt.cas.product.service' package that was committed for the GSoC > (Google Summer of Code 2013) midterm, i.e. the cas.product.service package > will be deleted and replaced with this reworked design. > > Documentation for this work has been added to the OODT wiki as detailed on > OODT-614. > > Work is ongoing for this package and many improvements can be made, for > example those mentioned in the 'development ideas' section at the bottom of > the developer guide. > > > Diffs > - > > trunk/webapp/fmprod/pom.xml 1517156 > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/configurations/RdfConfiguration.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/configurations/RssConfiguration.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/configurations/RssNamespace.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/configurations/RssTag.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/configurations/RssTagAttribute.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/configurations/package-info.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/exceptions/BadRequestException.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/exceptions/InternalServerErrorException.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/exceptions/NotFoundException.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/exceptions/package-info.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/filters/BackwardsCompatibleInterceptor.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/filters/package-info.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/package-info.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/resources/DatasetResource.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/resources/MetadataResource.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/resources/ProductResource.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/resources/ReferenceResource.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/resources/TransferResource.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/resources/TransfersResource.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/resources/package-info.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/services/CasProductJaxrsService.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/services/package-info.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/servlets/CasProductJaxrsServlet.java > PRE-CREATION > > trunk/webapp/fmprod/src/main/java/org/apa
Re: Review Request 15030: New RestfulProductQueryServlet
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15030/#review28543 --- Ship it! Ship It! - Chris Mattmann On Oct. 29, 2013, 4:28 p.m., Luca Cinquini wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/15030/ > --- > > (Updated Oct. 29, 2013, 4:28 p.m.) > > > Review request for oodt and Chris Mattmann. > > > Repository: oodt > > > Description > --- > > Insert a sub-class of QueryServlet that allows clients to use a URL syntax > easier than the traditional DIS-style. > > > Diffs > - > > > http://svn.apache.org/repos/asf/oodt/trunk/grid/src/main/java/org/apache/oodt/grid/QueryServlet.java > 1533206 > > http://svn.apache.org/repos/asf/oodt/trunk/grid/src/main/java/org/apache/oodt/grid/RestfulProductQueryServlet.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/15030/diff/ > > > Testing > --- > > Tested with the PDS transport-registry service. > Both the old URL syntax and the new URL syntax worked. > > > Thanks, > > Luca Cinquini > >
Re: How to process files in a sorted order
Hey Konstantinos,
The best way to take care of this is, as Cam said, to sub-class the
crawler, and define a new FileFilter (note there are default ones defined
in ProductCrawler). So you could create e.g., a:
SortedFilesCrawler extends ProductCrawler{
// new FileFilter defined here
// override crawl methods
@Override
public void crawl(){ // use your filter}
@Override
public void crawl(File dirRoot) {//use your filter}
}
Hope that helps!
Cheers,
Chris
Chris Mattmann
chris.mattm...@gmail.com
-Original Message-
From: Konstantinos Mavrommatis
Reply-To:
Date: Thursday, November 7, 2013 8:44 PM
To: "dev@oodt.apache.org"
Subject: How to process files in a sorted order
>Hi,
>In my environment I am using cas-crawler to process directories of 1000s
>of files. The metadata for these files are extracted automatically using
>the mimetypes definitions and small wrapper scripts.
>In these directories some of the files are derived from other files and
>metadata from the older files need to be transferred to the newer file.
>In order to achieve this I need to have the files processed by the
>cas-crawler starting from the older file to the newer file or in other
>cases in alphabetical order..
>Any ideas how this can be achieved?
>
>The crawler command I currently use is:
>./crawler_launcher --operation --launchAutoCrawler --productPath
>$FILEPATH --filemgrUrl $FMURL --clientTransferer
>org.apache.oodt.cas.filemgr.datatransfer.InPlaceDataTransferFactory
>--mimeExtr
>actorRepo ../policy/mime-extractor-map.xml
>
>Thanks in advance for your help
>Konstantinos
>
>*
>THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS
>CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED
>INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL
>OR INDIVIDUALS NAMED ABOVE.
>If the reader is not the intended recipient, or the
>employee or agent responsible to deliver it to the
>intended recipient, you are hereby notified that any
>dissemination, distribution or copying of this
>communication is strictly prohibited. If you have
>received this communication in error, please reply to the
>sender to notify us of the error and delete the original
>message. Thank You.
>*
Re: Review Request 15937: OODT 563 - b) New webapp: cas-wm-services (Workflow Manager Services) to allow adding/removing workflows
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15937/#review30735 --- trunk/webapp/wmservices/src/main/java/org/apache/oodt/cas/wmservices/repository/PackagedWorkflowManager.java <https://reviews.apache.org/r/15937/#comment58831> ALv2 header trunk/webapp/wmservices/src/main/java/org/apache/oodt/cas/wmservices/repository/PackagedWorkflowManager.java <https://reviews.apache.org/r/15937/#comment58832> I don't get what the point of this class is -- it seems to replicate functionality from the PackagedWorkflowRepository -- can you elaborate? trunk/webapp/wmservices/src/main/java/org/apache/oodt/cas/wmservices/resources/Resource.java <https://reviews.apache.org/r/15937/#comment58833> ALv2 header trunk/webapp/wmservices/src/main/java/org/apache/oodt/cas/wmservices/resources/Resource.java <https://reviews.apache.org/r/15937/#comment58834> is this a base class for WMresource services? If so I think it should have a more explicit name -- like AbstractWorkflowResource or something like that and then I think the methods should be protected. trunk/webapp/wmservices/src/main/java/org/apache/oodt/cas/wmservices/resources/WorkflowResource.java <https://reviews.apache.org/r/15937/#comment58835> this should call the actual PackagedWorkflowRepository (since it's a read only operation). trunk/webapp/wmservices/src/main/java/org/apache/oodt/cas/wmservices/resources/WorkflowResource.java <https://reviews.apache.org/r/15937/#comment58836> would be great to add this read only method to packaged workflow repo in WM. trunk/webapp/wmservices/src/main/java/org/apache/oodt/cas/wmservices/servlets/WmServicesServlet.java <https://reviews.apache.org/r/15937/#comment58837> seems like at some level, this class shouldn't be needed, and that the config here should be moved to AbstractWorkflowResource? trunk/webapp/wmservices/src/main/webapp/META-INF/context.xml <https://reviews.apache.org/r/15937/#comment58838> how about /wmservices? trunk/webapp/wmservices/src/main/webapp/META-INF/context.xml <https://reviews.apache.org/r/15937/#comment58840> should be [WORKFLOW_URL] and envReplaced. trunk/webapp/wmservices/src/main/webapp/META-INF/context.xml <https://reviews.apache.org/r/15937/#comment58839> should be replaced with [WORKFLOW_HOME]/policy/wengine, and then inside of AbstractWorkflowResource should be PathUtils.envReplaced (all read context params) - Chris Mattmann On Dec. 1, 2013, 7 p.m., Varun Ratnakar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/15937/ > ------- > > (Updated Dec. 1, 2013, 7 p.m.) > > > Review request for oodt and Chris Mattmann. > > > Bugs: OODT-563 > https://issues.apache.org/jira/browse/OODT-563 > > > Repository: oodt > > > Description > --- > > This patch addresses the second part of > https://issues.apache.org/jira/browse/OODT-563. It adds a new webapp under > webapps/wmservices which provides: > 1. A servlet with Jax-RS functionality to Add/Remove workflows (just packaged > repository workflows for now) > > > Diffs > - > > trunk/webapp/wmservices/pom.xml PRE-CREATION > > trunk/webapp/wmservices/src/main/java/org/apache/oodt/cas/wmservices/client/WmServicesClient.java > PRE-CREATION > > trunk/webapp/wmservices/src/main/java/org/apache/oodt/cas/wmservices/repository/PackagedWorkflowManager.java > PRE-CREATION > > trunk/webapp/wmservices/src/main/java/org/apache/oodt/cas/wmservices/resources/Resource.java > PRE-CREATION > > trunk/webapp/wmservices/src/main/java/org/apache/oodt/cas/wmservices/resources/WorkflowResource.java > PRE-CREATION > > trunk/webapp/wmservices/src/main/java/org/apache/oodt/cas/wmservices/servlets/WmServicesServlet.java > PRE-CREATION > trunk/webapp/wmservices/src/main/webapp/META-INF/context.xml PRE-CREATION > trunk/webapp/wmservices/src/main/webapp/WEB-INF/web.xml PRE-CREATION > > Diff: https://reviews.apache.org/r/15937/diff/ > > > Testing > --- > > > Thanks, > > Varun Ratnakar > >
Re: Review Request 15936: OODT 563 - a) Modifying Workflow Manager to allow adding new workflows
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15936/#review30736
---
trunk/workflow/src/main/java/org/apache/oodt/cas/workflow/repository/PackagedWorkflowRepository.java
<https://reviews.apache.org/r/15936/#comment58841>
why wouldn't it be a PC workflow here?
trunk/workflow/src/main/java/org/apache/oodt/cas/workflow/repository/PackagedWorkflowRepository.java
<https://reviews.apache.org/r/15936/#comment58842>
nice catch!
- Chris Mattmann
On Dec. 1, 2013, 6:54 p.m., Varun Ratnakar wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15936/
> ---
>
> (Updated Dec. 1, 2013, 6:54 p.m.)
>
>
> Review request for oodt and Chris Mattmann.
>
>
> Bugs: OODT-563
> https://issues.apache.org/jira/browse/OODT-563
>
>
> Repository: oodt
>
>
> Description
> ---
>
> This patch addresses a part of
> https://issues.apache.org/jira/browse/OODT-563. It provides:
>
> 1. Additions to XmlRpcWorkflowManager(Client) to refreshRepository (i.e.
> reload repository)
> 2. Modifications to PackagedWorkflowRepository's addWorkflow function to
> properly deal with added workflows
>
>
> Diffs
> -
>
>
> trunk/workflow/src/main/java/org/apache/oodt/cas/workflow/repository/PackagedWorkflowRepository.java
> 1546774
>
> trunk/workflow/src/main/java/org/apache/oodt/cas/workflow/system/XmlRpcWorkflowManager.java
> 1546774
>
> trunk/workflow/src/main/java/org/apache/oodt/cas/workflow/system/XmlRpcWorkflowManagerClient.java
> 1546774
>
> Diff: https://reviews.apache.org/r/15936/diff/
>
>
> Testing
> ---
>
>
> Thanks,
>
> Varun Ratnakar
>
>
Re: Review Request 15936: OODT 563 - a) Modifying Workflow Manager to allow adding new workflows
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15936/#review30776 --- Ship it! Ship It! - Chris Mattmann On Dec. 1, 2013, 6:54 p.m., Varun Ratnakar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/15936/ > --- > > (Updated Dec. 1, 2013, 6:54 p.m.) > > > Review request for oodt and Chris Mattmann. > > > Bugs: OODT-563 > https://issues.apache.org/jira/browse/OODT-563 > > > Repository: oodt > > > Description > --- > > This patch addresses a part of > https://issues.apache.org/jira/browse/OODT-563. It provides: > > 1. Additions to XmlRpcWorkflowManager(Client) to refreshRepository (i.e. > reload repository) > 2. Modifications to PackagedWorkflowRepository's addWorkflow function to > properly deal with added workflows > > > Diffs > - > > > trunk/workflow/src/main/java/org/apache/oodt/cas/workflow/repository/PackagedWorkflowRepository.java > 1546774 > > trunk/workflow/src/main/java/org/apache/oodt/cas/workflow/system/XmlRpcWorkflowManager.java > 1546774 > > trunk/workflow/src/main/java/org/apache/oodt/cas/workflow/system/XmlRpcWorkflowManagerClient.java > 1546774 > > Diff: https://reviews.apache.org/r/15936/diff/ > > > Testing > --- > > > Thanks, > > Varun Ratnakar > >
Re: ApacheCon 2014
Hi Tom, I was thinking of submitting one on the use of Apache OODT in the Airborne Snow Observatory (ASO) project at JPL. or on the Wicket-based web interfaces. Do you have any ideas of what to submit? Cheers, Chris Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Tom Barber Reply-To: Date: Tuesday, January 14, 2014 2:34 PM To: "dev@oodt.apache.org" Subject: ApacheCon 2014 >Hi folks, > >Just a quick query. I was wondering if any OODT developers were planning >to submit any talk proposals into ApacheCon? > >Thanks > >Tom > >-- >*Tom Barber* | Technical Director > >meteorite bi >*T:* +44 20 8133 3730 >*W:* www.meteorite.bi | *Skype:* meteorite.consulting >*A:* Surrey Technology Centre, Surrey Research Park, Guildford, GU2 7YG, >UK
Re: ApacheCon 2014
Tom this sounds fantastic and yes please submit! Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Tom Barber Reply-To: Date: Monday, January 27, 2014 11:08 AM To: Subject: Re: ApacheCon 2014 >So I saw the call go out that they were short of applications and it got >me thinking once again. > >My issue with Apache OODT is the fact currently a load of science bods >use it (no offence ;) ) and the stuff you see written about and >presented whilst a) is cool as hell b) lacks day to day "real world" >applications for non scientists because if people do use it for non >science applications, its not publicised very well which is where I >want to try and advocate OODT > >But with my current clients don't have an obvious use case, but there >must be a bunch if I sit down and think about them, so I was thinking >about pitching a talk along the lines of > >"Utilising the power of Apache OODT with mainstream Business >Intelligence tools" > >Where I can discuss a few real world scenarios and I was thinking I >could demonstrate OODT integration with Pentaho Data >Integration(Apache Licensed) using ETL to control ingestion and use >Pentaho Data Integration to query and extract data from the Catalog. > >I could also use the Pentaho BI server to create a Dashboard >framework to display relevant OODT information and as an alternative >ingestion platform to the default File manager. > > > >Clearly non of this is yet written and I guess I'd write it if the talk >got accepted and I got my fellow colleagues to help sponsor the trip, >but i thought the idea was worth putting out there to garner feedback >from the development community. > >Thoughts on a postcard please. > >Thanks > >Tom > >On 17/01/14 00:12, Chris Mattmann wrote: >> Hi Tom, >> >> I was thinking of submitting one on the use of Apache OODT >> in the Airborne Snow Observatory (ASO) project at JPL. or on >> the Wicket-based web interfaces. Do you have any ideas of what >> to submit? >> >> Cheers, >> Chris >> >> >> Chris Mattmann >> chris.mattm...@gmail.com >> >> >> >> >> -Original Message- >> From: Tom Barber >> Reply-To: >> Date: Tuesday, January 14, 2014 2:34 PM >> To: "dev@oodt.apache.org" >> Subject: ApacheCon 2014 >> >>> Hi folks, >>> >>> Just a quick query. I was wondering if any OODT developers were >>>planning >>> to submit any talk proposals into ApacheCon? >>> >>> Thanks >>> >>> Tom >>> >>> -- >>> *Tom Barber* | Technical Director >>> >>> meteorite bi >>> *T:* +44 20 8133 3730 >>> *W:* www.meteorite.bi | *Skype:* meteorite.consulting >>> *A:* Surrey Technology Centre, Surrey Research Park, Guildford, GU2 >>>7YG, >>> UK >> > > >-- >*Tom Barber* | Technical Director > >meteorite bi >*T:* +44 20 8133 3730 >*W:* www.meteorite.bi | *Skype:* meteorite.consulting >*A:* Surrey Technology Centre, Surrey Research Park, Guildford, GU2 7YG, >UK
Re: ApacheCon 2014
Chris, smells like another talk. Please submit :) Seriously, you have it written below :) Submit! :) Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Christopher Warner Reply-To: Date: Wednesday, January 29, 2014 8:45 AM To: Subject: Re: ApacheCon 2014 >One of the more difficult things for me trying to sell OODT was espousing >why it's useful for structured processes whether they be government >initiated protocols for data handling or workflow handling of that data. >In >some cases I'm dealing with large data sets that exist in things like >excel, word, or some other unstructured crazy format. So part of the >problem is social and dealing with researchers outside of the science >paradigm whom aren't readily used to processing or extracting information >from their data in that way. > >I think it's a great idea that would give an immediate visually accessible >way of how it's useful for data sets not-related to science alone. > >From New York.. > > >On Mon, Jan 27, 2014 at 2:08 PM, Tom Barber >wrote: > >> So I saw the call go out that they were short of applications and it got >> me thinking once again. >> >> My issue with Apache OODT is the fact currently a load of science bods >>use >> it (no offence ;) ) and the stuff you see written about and presented >> whilst a) is cool as hell b) lacks day to day "real world" applications >>for >> non scientists because if people do use it for non science applications, >> its not publicised very well which is where I want to try and advocate >> OODT >> >> But with my current clients don't have an obvious use case, but there >>must >> be a bunch if I sit down and think about them, so I was thinking about >> pitching a talk along the lines of >> >>"Utilising the power of Apache OODT with mainstream Business >>Intelligence tools" >> >>Where I can discuss a few real world scenarios and I was thinking I >>could demonstrate OODT integration with Pentaho Data >>Integration(Apache Licensed) using ETL to control ingestion and use >>Pentaho Data Integration to query and extract data from the Catalog. >> >>I could also use the Pentaho BI server to create a Dashboard >>framework to display relevant OODT information and as an alternative >>ingestion platform to the default File manager. >> >> >> >> Clearly non of this is yet written and I guess I'd write it if the talk >> got accepted and I got my fellow colleagues to help sponsor the trip, >>but i >> thought the idea was worth putting out there to garner feedback from the >> development community. >> >> Thoughts on a postcard please. >> >> Thanks >> >> Tom >> >> >> On 17/01/14 00:12, Chris Mattmann wrote: >> >>> Hi Tom, >>> >>> I was thinking of submitting one on the use of Apache OODT >>> in the Airborne Snow Observatory (ASO) project at JPL. or on >>> the Wicket-based web interfaces. Do you have any ideas of what >>> to submit? >>> >>> Cheers, >>> Chris >>> >>> >>> Chris Mattmann >>> chris.mattm...@gmail.com >>> >>> >>> >>> >>> -Original Message- >>> From: Tom Barber >>> Reply-To: >>> Date: Tuesday, January 14, 2014 2:34 PM >>> To: "dev@oodt.apache.org" >>> Subject: ApacheCon 2014 >>> >>> Hi folks, >>>> >>>> Just a quick query. I was wondering if any OODT developers were >>>>planning >>>> to submit any talk proposals into ApacheCon? >>>> >>>> Thanks >>>> >>>> Tom >>>> >>>> -- >>>> *Tom Barber* | Technical Director >>>> >>>> meteorite bi >>>> *T:* +44 20 8133 3730 >>>> *W:* www.meteorite.bi | *Skype:* meteorite.consulting >>>> *A:* Surrey Technology Centre, Surrey Research Park, Guildford, GU2 >>>>7YG, >>>> UK >>>> >>> >>> >> >> -- >> *Tom Barber* | Technical Director >> >> meteorite bi >> *T:* +44 20 8133 3730 >> *W:* www.meteorite.bi | *Skype:* meteorite.consulting >> *A:* Surrey Technology Centre, Surrey Research Park, Guildford, GU2 >>7YG, UK >> > > > >-- >Christopher Warner >http://cwarner.kernelcode.com
OODT's Git mirror at Github is borked
Hey Guys, I just tried to do some Github.com/apache/oodt work and submitted a pull request and saw that the default branch at Github for OODT is the 0.1-incubating branch which in turn seems to have borked my pull request (thinks it's over 800+ changes). Any ideas? Thanks! Cheers, Chris
Re: OODT's Git mirror at Github is borked
Thanks Jake. I guess I was under the (potentially) wrong impression that we could do pull requests against our Github mirrors for ASF projects and they would show up as emails on our dev@oodt.apache.org mailing lists with emails back to the patch that the pull request represented. I've seen this in Tika and in a few other projects. Is that not the case anymore? I could directly commit to OODT as I'm a PMC member for the project, but I wanted to be cool and try Github ;) Cheers, Chris -Original Message- From: Jake Farrell Reply-To: "jfarr...@apache.org" Date: Wednesday, January 29, 2014 9:19 PM To: jpluser Cc: "infrastruct...@apache.org" , "dev@oodt.apache.org" Subject: Re: OODT's Git mirror at Github is borked >Hey Chris >I have switched the default branch on the Github mirror for OODT to be >trunk and closed out the two open pull requests that you had issued. Can >you please submit the patches you have to OODT > and not issue pull requests against the mirror as no one from OODT will >be able to accept/merge them or close the pull request. > > >Thanks >-Jake > > > > > > >On Wed, Jan 29, 2014 at 11:58 PM, Chris Mattmann > wrote: > >Hey Guys, > > >I just tried to do some Github.com/apache/oodt work and submitted >a pull request and saw that the default branch at Github for OODT >is the 0.1-incubating branch which in turn seems to have borked >my pull request (thinks it's over 800+ changes). > >Any ideas? > >Thanks! > >Cheers, >Chris > > > > > > >
Re: OODT's Git mirror at Github is borked
Hey Jake, Thanks. I know what the official source repository is for OODT, and that SVN is the canonical bits, and that the mirror at Github is simply a mirror. I also know that merges aren't automatic and that they can't automatically be integrated. My question is can you configure the pull request-to-email list notification? We should be able to do that without switching to Git, correct? I'm not proposing to move OODT to Git. Thanks. Cheers, Chris -Original Message- From: Jake Farrell Reply-To: "jfarr...@apache.org" Date: Wednesday, January 29, 2014 9:36 PM To: jpluser Cc: "jfarr...@apache.org" , "infrastruct...@apache.org" , "dev@oodt.apache.org" Subject: Re: OODT's Git mirror at Github is borked >Hey Chris >The official source repository for OODT is currently in svn, and Github >is just a mirror of that svn repository. When a pull request is issued >against one of our mirrors on Github it can email the configured mailing >list for that mirror that a pull request > is available, but it can not be merged into that mirror. The Github >notification email that is sent contains a link to the patch and the >description or comment that was given, not the actual patch itself. > > >Tika has 3 open pull requests which are 8 months old to 2 years old, >comments can be made by anyone on them, but only Github Apache org admins >can close them out. > > >If OODT is interested in switching over to git and using git-wip-us.a.o >as the primary repository I would be happy to help migrate them. > > >If you have any other questions let me know >-Jake > > > > > > >On Thu, Jan 30, 2014 at 12:21 AM, Chris Mattmann > wrote: > >Thanks Jake. I guess I was under the (potentially) wrong impression >that we could do pull requests against our Github mirrors for ASF >projects and they would show up as emails on our >dev@oodt.apache.org <mailto:dev@oodt.apache.org> >mailing lists with emails back to the patch that the pull request >represented. I've seen this in Tika and in a few other projects. > >Is that not the case anymore? I could directly commit to OODT as I'm >a PMC member for the project, but I wanted to be cool and try Github ;) > >Cheers, >Chris > > > > >-Original Message- >From: Jake Farrell >Reply-To: "jfarr...@apache.org" >Date: Wednesday, January 29, 2014 9:19 PM >To: jpluser >Cc: "infrastruct...@apache.org" , >"dev@oodt.apache.org" >Subject: Re: OODT's Git mirror at Github is borked > >>Hey Chris >>I have switched the default branch on the Github mirror for OODT to be >>trunk and closed out the two open pull requests that you had issued. Can >>you please submit the patches you have to OODT >> and not issue pull requests against the mirror as no one from OODT will >>be able to accept/merge them or close the pull request. >> >> >>Thanks >>-Jake >> >> >> >> >> >> >>On Wed, Jan 29, 2014 at 11:58 PM, Chris Mattmann >> wrote: >> >>Hey Guys, >> >> >>I just tried to do some Github.com/apache/oodt work and submitted >>a pull request and saw that the default branch at Github for OODT >>is the 0.1-incubating branch which in turn seems to have borked >>my pull request (thinks it's over 800+ changes). >> >>Any ideas? >> >>Thanks! >> >>Cheers, >>Chris >> >> >> >> >> >> >> > > > > > > > > >
Re: OODT's Git mirror at Github is borked
Thanks Jake. I just tested this out by issuing a pull request, see: https://github.com/apache/oodt/pull/3 I didn't see an email show up on dev@oodt and if I look at apmail: [hermes] 5:12am oodt.apache.org > ezmlm-list dev | grep git g...@git.apache.org [hermes] 5:12am oodt.apache.org > Any idea what's up? Is there some delay on the hook script? Cheers, Chris -Original Message- From: Jake Farrell Reply-To: "jfarr...@apache.org" Date: Wednesday, January 29, 2014 9:59 PM To: jpluser Cc: "jfarr...@apache.org" , "infrastruct...@apache.org" , "dev@oodt.apache.org" Subject: Re: OODT's Git mirror at Github is borked >Hey Chris >Hook added > > >-Jake > > > >On Thu, Jan 30, 2014 at 12:47 AM, Chris Mattmann > wrote: > >Hey Jake, > >Thanks. I know what the official source repository is for OODT, >and that SVN is the canonical bits, and that the mirror at Github >is simply a mirror. I also know that merges aren't automatic and >that they can't automatically be integrated. > >My question is can you configure the pull request-to-email list >notification? We should be able to do that without switching to >Git, correct? I'm not proposing to move OODT to Git. Thanks. > >Cheers, >Chris > > > >-Original Message- >From: Jake Farrell >Reply-To: "jfarr...@apache.org" >Date: Wednesday, January 29, 2014 9:36 PM >To: jpluser >Cc: "jfarr...@apache.org" , >"infrastruct...@apache.org" , >"dev@oodt.apache.org" >Subject: Re: OODT's Git mirror at Github is borked > >>Hey Chris >>The official source repository for OODT is currently in svn, and Github >>is just a mirror of that svn repository. When a pull request is issued >>against one of our mirrors on Github it can email the configured mailing >>list for that mirror that a pull request >> is available, but it can not be merged into that mirror. The Github >>notification email that is sent contains a link to the patch and the >>description or comment that was given, not the actual patch itself. >> >> >>Tika has 3 open pull requests which are 8 months old to 2 years old, >>comments can be made by anyone on them, but only Github Apache org admins >>can close them out. >> >> >>If OODT is interested in switching over to git and using git-wip-us.a.o >>as the primary repository I would be happy to help migrate them. >> >> >>If you have any other questions let me know >>-Jake >> >> >> >> >> >> >>On Thu, Jan 30, 2014 at 12:21 AM, Chris Mattmann >> wrote: >> >>Thanks Jake. I guess I was under the (potentially) wrong impression >>that we could do pull requests against our Github mirrors for ASF >>projects and they would show up as emails on our >>dev@oodt.apache.org <mailto:dev@oodt.apache.org> >>mailing lists with emails back to the patch that the pull request >>represented. I've seen this in Tika and in a few other projects. >> >>Is that not the case anymore? I could directly commit to OODT as I'm >>a PMC member for the project, but I wanted to be cool and try Github ;) >> >>Cheers, >>Chris >> >> >> >> >>-Original Message- >>From: Jake Farrell >>Reply-To: "jfarr...@apache.org" >>Date: Wednesday, January 29, 2014 9:19 PM >>To: jpluser >>Cc: "infrastruct...@apache.org" , >>"dev@oodt.apache.org" >>Subject: Re: OODT's Git mirror at Github is borked >> >>>Hey Chris >>>I have switched the default branch on the Github mirror for OODT to be >>>trunk and closed out the two open pull requests that you had issued. Can >>>you please submit the patches you have to OODT >>> and not issue pull requests against the mirror as no one from OODT will >>>be able to accept/merge them or close the pull request. >>> >>> >>>Thanks >>>-Jake >>> >>> >>> >>> >>> >>> >>>On Wed, Jan 29, 2014 at 11:58 PM, Chris Mattmann >>> wrote: >>> >>>Hey Guys, >>> >>> >>>I just tried to do some Github.com/apache/oodt work and submitted >>>a pull request and saw that the default branch at Github for OODT >>>is the 0.1-incubating branch which in turn seems to have borked >>>my pull request (thinks it's over 800+ changes). >>> >>>Any ideas? >>> >>>Thanks! >>> >>>Cheers, >>>Chris >>> >>> >>> >>> >>> >>> >>> >> >> >> >> >> >> >> >> >> > > > > > > >
My Talk for ApacheCon on CAS-PGE
Hey Guys, I submitted a CAS-PGE talk for ApacheCon. See: Title Rapid and Unobtrusive Algorithm Integration with CAS-PGE Event ApacheCon North America Submission Type Presentation Category Developer Biography Chris Mattmann has a wealth of experience in software design, and in the construction of large-scale data-intensive systems. His work has infected a broad set of communities, ranging from helping NASA unlock data from its next generation of earth science system satellites, to assisting graduate students at the University of Southern California (his Alma mater) in the study of software architecture, all the way to helping industry and open source as a member of the Apache Software Foundation. When he's not busy being busy, he's spending time with his lovely wife and son braving the mean streets of Southern California. Abstract Apache OODT is a science data processing framework that is NASA's first project to be released to the ASF. One of OODT's core capabilities is to incorporate science user developed code and automate it and make it run at scale. The algorithm integration facilities of Apache OODT are encapsulated within the CAS-PGE framework. Put simply our answer to you to make your science algorithm run at scale isn't to rewrite it as Map Reduce; or to rewrite it as a web service. Leave it the way it is and let CAS-PGE take care of it for you. CAS-PGE models your algorithms' inputs and outputs, describes its execution framework and catalogs and archives those inputs and outputs using the file management component. This tutorial will walk you through CAS-PGE. We'll set it up. We'll run a Moon image processing example end to end through CAS-PGE and get you crackin' away at algorithm integration! Audience Users of Apache OODT; folks who write code in Matlab, Python R, Scala, and who use science packages like CASApy, IDL/ENVI and other algorithm frameworks. Folks who are looking for something different than write it in Map Reduce. Folks who are interested in YARN/MR2 and Spark/Shark. Experience Level Intermediate Benefits to the Ecosystem We haven't given a detailed presentation on CAS-PGE yet. I've given presentations on CAS, on the OODT ecosystem, on PCS and the OPSUI, on how to apply OODT to use cases, and on how to install OODT and workflows within OODT. A core component of OODT without an ApacheCon talk is CAS-PGE. Technical Requirements Nothing more than a projector, beer (did I say that?), and don't even need an Internet connection (though would be great to have one). Status New
Re: OODT's Git mirror at Github is borked
Thank you Jake! -Original Message- From: Jake Farrell Reply-To: "jfarr...@apache.org" Date: Friday, January 31, 2014 8:05 AM To: jpluser Cc: "jfarr...@apache.org" , "infrastruct...@apache.org" , "dev@oodt.apache.org" Subject: Re: OODT's Git mirror at Github is borked >Hey Chris >Looks like oodt is missing the svn mirror on git-wip, hook depends on it >being on git-wip. its only showing on git.a.o. will get it synced shortly >and see why it isnt in both places > > >-Jake > > > >On Fri, Jan 31, 2014 at 12:13 AM, Chris Mattmann > wrote: > >Thanks Jake. > >I just tested this out by issuing a pull request, see: > >https://github.com/apache/oodt/pull/3 > > >I didn't see an email show up on dev@oodt and if I look at apmail: > >[hermes] 5:12am oodt.apache.org <http://oodt.apache.org> > ezmlm-list dev >| grep git >g...@git.apache.org >[hermes] 5:12am oodt.apache.org <http://oodt.apache.org> > > > >Any idea what's up? Is there some delay on the hook script? > >Cheers, >Chris > > > > > >-Original Message- >From: Jake Farrell >Reply-To: "jfarr...@apache.org" >Date: Wednesday, January 29, 2014 9:59 PM >To: jpluser >Cc: "jfarr...@apache.org" , >"infrastruct...@apache.org" , >"dev@oodt.apache.org" >Subject: Re: OODT's Git mirror at Github is borked > >>Hey Chris >>Hook added >> >> >>-Jake >> >> >> >>On Thu, Jan 30, 2014 at 12:47 AM, Chris Mattmann >> wrote: >> >>Hey Jake, >> >>Thanks. I know what the official source repository is for OODT, >>and that SVN is the canonical bits, and that the mirror at Github >>is simply a mirror. I also know that merges aren't automatic and >>that they can't automatically be integrated. >> >>My question is can you configure the pull request-to-email list >>notification? We should be able to do that without switching to >>Git, correct? I'm not proposing to move OODT to Git. Thanks. >> >>Cheers, >>Chris >> >> >> >>-Original Message- >>From: Jake Farrell >>Reply-To: "jfarr...@apache.org" >>Date: Wednesday, January 29, 2014 9:36 PM >>To: jpluser >>Cc: "jfarr...@apache.org" , >>"infrastruct...@apache.org" , >>"dev@oodt.apache.org" >>Subject: Re: OODT's Git mirror at Github is borked >> >>>Hey Chris >>>The official source repository for OODT is currently in svn, and Github >>>is just a mirror of that svn repository. When a pull request is issued >>>against one of our mirrors on Github it can email the configured mailing >>>list for that mirror that a pull request >>> is available, but it can not be merged into that mirror. The Github >>>notification email that is sent contains a link to the patch and the >>>description or comment that was given, not the actual patch itself. >>> >>> >>>Tika has 3 open pull requests which are 8 months old to 2 years old, >>>comments can be made by anyone on them, but only Github Apache org >>>admins >>>can close them out. >>> >>> >>>If OODT is interested in switching over to git and using git-wip-us.a.o >>>as the primary repository I would be happy to help migrate them. >>> >>> >>>If you have any other questions let me know >>>-Jake >>> >>> >>> >>> >>> >>> >>>On Thu, Jan 30, 2014 at 12:21 AM, Chris Mattmann >>> wrote: >>> >>>Thanks Jake. I guess I was under the (potentially) wrong impression >>>that we could do pull requests against our Github mirrors for ASF >>>projects and they would show up as emails on our >>>dev@oodt.apache.org <mailto:dev@oodt.apache.org> >>>mailing lists with emails back to the patch that the pull request >>>represented. I've seen this in Tika and in a few other projects. >>> >>>Is that not the case anymore? I could directly commit to OODT as I'm >>>a PMC member for the project, but I wanted to be cool and try Github ;) >>> >>>Cheers, >>>Chris >>> >>> >>> >>> >>>-Original Message- >>>From: Jake Farrell >>>Reply-To: "jfarr...@apache.org" >>>Date: Wednesday, January 29, 2014 9:19 PM >>>To: jpluser >>>Cc: "infrastruct...@apache.org" , >>>"dev@oodt.apache.org" >>>Subject: Re: OODT's Git mirror at Github is borked >>> >>>>Hey Chris >>>>I have switched the default branch on the Github mirror for OODT to be >>>>trunk and closed out the two open pull requests that you had issued. >>>>Can >>>>you please submit the patches you have to OODT >>>> and not issue pull requests against the mirror as no one from OODT >>>>will >>>>be able to accept/merge them or close the pull request. >>>> >>>> >>>>Thanks >>>>-Jake >>>> >>>> >>>> >>>> >>>> >>>> >>>>On Wed, Jan 29, 2014 at 11:58 PM, Chris Mattmann >>>> wrote: >>>> >>>>Hey Guys, >>>> >>>> >>>>I just tried to do some Github.com/apache/oodt work and submitted >>>>a pull request and saw that the default branch at Github for OODT >>>>is the 0.1-incubating branch which in turn seems to have borked >>>>my pull request (thinks it's over 800+ changes). >>>> >>>>Any ideas? >>>> >>>>Thanks! >>>> >>>>Cheers, >>>>Chris >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >> >> >> >> >> >> >> > > > > > > >
Re: ApacheCon 2014
Great job Tom! Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Tom Barber Reply-To: Date: Friday, January 31, 2014 1:15 PM To: Subject: Re: ApacheCon 2014 >Submitted it is! > >On 31/01/14 16:29, ke...@apache.org wrote: >> Nothing short of brilliant. >> >> Submit! Submit with extreme prejudice! 🍻 >> >> --k >> >> On 2014-01-31, at 8:16 AM, Tom Barber wrote: >> >>> Alright folks I was planning on pitching this, what do you think? >>> >>> Title: Utilising the power of Apache OODT with open source Business >>>Intelligence tools >>> >>> Event: ApacheCon North America 2014 >>> >>> Submission Type: Presentation >>> >>> Category: Developer >>> >>> Biography: Tom Barber has been a vocal Open Source Business >>>Intelligence proponent for many years. He is well known in the Pentaho >>>community having instigated a number of Pentaho community meetings in >>>the UK and Europe. He has extensive experience of information >>>management, business intelligence, data warehousing, and data >>>integration with a range of technologies. Tom is also the project >>>founder of the Apache Licensed Saiku OLAP Analysis Tool. When not >>>hacking Open Source technologies Tom is the Technical Director of >>>Meteorite Consulting, an open source focussed Business Intelligence >>>consultancy. >>> >>> Abstract: Apache OODT is a science data processing framework that is >>>NASA's >>> >>> first project to be released to the ASF. OODT has many publicised use >>>cases in the science world, but less discussed is the use of Apache >>>OODT in the world of business. We'll run through a number of examples >>>detailing how OODT can be used for non science based applications. We >>>will also demonstrate how Open Source Business Intelligence tools can >>>interact with Apache OODT and help integrate Apache OODT into Business >>>Intelligence ecosystems. Using Pentaho Data Integration we will >>>demonstrate data ingestion and extraction, and how ETL tools can help >>>you manipulate data from within Apache OODT. >>> >>> Audience: User of OODT, and Business Intelligence developers who have >>>an interest in Metadata storage and processing. >>> >>> Experience Level: Beginner >>> >>> Technical Requirements: Projector >>> >>> >>> Cheers >>> >>> Tom >>> >>> >>> On 30/01/14 21:11, Christopher Warner wrote: >>>> Ok, i've got a day to decide.. i'm swamped and I feel really lazy >>>>right >>>> now, let me procrastinate a little while I think of how I can lay >>>>this out >>>> into a compelling talk of some sort. >>>> >>>> >>>> On Wed, Jan 29, 2014 at 11:54 PM, Chris Mattmann >>>> wrote: >>>> >>>>> Chris, smells like another talk. Please submit :) Seriously, you >>>>>have it >>>>> written below :) Submit! :) >>>>> >>>>> >>>>> Chris Mattmann >>>>> chris.mattm...@gmail.com >>>>> >>>>> >>>>> >>>>> >>>>> -Original Message- >>>>> From: Christopher Warner >>>>> Reply-To: >>>>> Date: Wednesday, January 29, 2014 8:45 AM >>>>> To: >>>>> Subject: Re: ApacheCon 2014 >>>>> >>>>>> One of the more difficult things for me trying to sell OODT was >>>>>>espousing >>>>>> why it's useful for structured processes whether they be government >>>>>> initiated protocols for data handling or workflow handling of that >>>>>>data. >>>>>> In >>>>>> some cases I'm dealing with large data sets that exist in things >>>>>>like >>>>>> excel, word, or some other unstructured crazy format. So part of the >>>>>> problem is social and dealing with researchers outside of the >>>>>>science >>>>>> paradigm whom aren't readily used to processing or extracting >>>>>>information >>>>> >from their data in that way. >>>>>> I think it's a great idea that would give an immediate visually >>>>>>accessible >>>
Re: Review Request 18727: Implementation of DataTransferer which stores files in Amazon S3
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18727/#review36630 --- Ship it! Ship It! - Chris Mattmann On March 4, 2014, 4 a.m., brian Foster wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/18727/ > --- > > (Updated March 4, 2014, 4 a.m.) > > > Review request for oodt, Chris Mattmann and Paul Ramirez. > > > Repository: oodt > > > Description > --- > > Implementation of DataTransferer which stores files in Amazon S3 > > > Diffs > - > > trunk/filemgr/pom.xml 1573239 > > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/S3DataTransferer.java > PRE-CREATION > > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/S3DataTransfererFactory.java > PRE-CREATION > trunk/filemgr/src/main/resources/filemgr.properties 1573239 > > trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/datatransfer/TestS3DataTransferer.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/18727/diff/ > > > Testing > --- > > Unit-tests > > > Thanks, > > brian Foster > >
Re: [DISCUSS] 0.7 Roadmap
+1 to stable build. Also we should get the contribution Tom/Chris RE: NFS data movement and the patch from Rajith for Ganglia/Resource Manager in. I'll review it today and get them both committed. RE: sshtools -- have we shipped before with deps on it? For this build we could simply remove the dep, and then disable any code/tests that ref it through exclusions. Would that work? Cheers, Chris Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Lewis John Mcgibbney Reply-To: Date: Thursday, May 1, 2014 10:53 AM To: Subject: Re: [DISCUSS] 0.7 Roadmap >I think a stable build on project Jenkins is desirable... migration of >documentation is not a blocker imho. >The issue with ssh tools is a blocker... licensing is an issue here. >Stable build and ssh tools are two blockers imho folks. >Lewis >On May 1, 2014 10:41 AM, "Mattmann, Chris A (3980)" < >chris.a.mattm...@jpl.nasa.gov> wrote: > >> Lewis, great roadmap. >> >> Thanks for pushing the issues too. >> >> Looking at my own personal roadmap (16 issues) I think it's more >> than I have time for or need to get into this release. >> >> I'd say let's take the 31 fixed issues and release OODT 0.7 with >> them and then address remaining issues in 0.8. >> >> Thoughts? >> >> Cheers, >> Chris >> >> >> ++ >> Chris Mattmann, Ph.D. >> Chief Architect >> Instrument Software and Science Data Systems Section (398) >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >> Office: 168-519, Mailstop: 168-5th floor >> Email: chris.a.mattm...@nasa.gov >> WWW: http://sunset.usc.edu/~mattmann/ >> ++ >> Adjunct Associate Professor, Computer Science Department >> University of Southern California, Los Angeles, CA 90089 USA >> ++ >> >> >> >> >> >> >> -Original Message- >> From: Lewis John Mcgibbney >> Reply-To: "dev@oodt.apache.org" >> Date: Wednesday, April 30, 2014 3:48 PM >> To: "dev@oodt.apache.org" >> Subject: [DISCUSS] 0.7 Roadmap >> >> >Hi Troops, >> >Do we have a roadmap for 0.7? >> >I propose the following: >> >OODT-667, OODT-630 & 632 >> >We should also stabilize Jenkins build. >> >Any other suggestions? >> >Thanks >> >Lewis >> >>
Re: Review Request 20849: Fix copy to failure dir from NFS mount
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20849/#review41929 --- Great work Tom and Chris! Chris, would you be willing to fill out an ICLA here? https://www.apache.org/licenses/icla.txt - Chris Mattmann On April 29, 2014, 3:52 p.m., Thomas Bennett wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/20849/ > --- > > (Updated April 29, 2014, 3:52 p.m.) > > > Review request for oodt and Chris Mattmann. > > > Bugs: OODT-674 > https://issues.apache.org/jira/browse/OODT-674 > > > Repository: oodt > > > Description > --- > > Solution is to use org.apache.commons.io.FileUtils.copyFileToDirectory. > > Note this solution came from my colleague Chris Schollar. > > > Diffs > - > > trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/action/MoveFile.java > 1591001 > > Diff: https://reviews.apache.org/r/20849/diff/ > > > Testing > --- > > Tested on local machine with NFS mounted directories > > > Thanks, > > Thomas Bennett > >
Re: Review Request 18944: [OODT-669] Disabling resource monitor in Assignment monitor feature
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18944/#review42060 --- Rajith, I'm not sure I get this patch. Why do we have to disable the Resource monitor for this to work? We used to have an AssignmentMonitor that didn't need Ganglia up and running -- so we could simply set AssignmentMonitor as the default Monitor and we were good. What's changed? - Chris Mattmann On March 9, 2014, 4:44 a.m., Rajith Siriwardana wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/18944/ > --- > > (Updated March 9, 2014, 4:44 a.m.) > > > Review request for oodt and Chris Mattmann. > > > Bugs: OODT-669 > https://issues.apache.org/jira/browse/OODT-669 > > > Repository: oodt > > > Description > --- > > Patch contains disabling Resource Monitor feature in Assignment Monitor and > also some re-factored code for assignment monitor > > > Diffs > - > > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/AssignmentMonitor.java > 1575649 > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/disabledmonitor/DisabledResourceMonitor.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/disabledmonitor/DisabledResourceMonitorFactory.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/GangliaAdapter.java > 1575649 > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/GangliaResourceMonitor.java > 1575649 > trunk/resource/src/main/resources/resource.properties 1575649 > > trunk/resource/src/test/org/apache/oodt/cas/resource/monitor/TestAssignmentMonWithNoResourceMon.java > PRE-CREATION > trunk/resource/src/testdata/test.disabledmon.resource.properties > PRE-CREATION > > Diff: https://reviews.apache.org/r/18944/diff/ > > > Testing > --- > > Disabled monitor with Assignment monitor was tested > > > Thanks, > > Rajith Siriwardana > >
Re: Review Request 18944: [OODT-669] Disabling resource monitor in Assignment monitor feature
> On May 2, 2014, 9:20 p.m., Chris Mattmann wrote: > > Rajith, I'm not sure I get this patch. Why do we have to disable the > > Resource monitor for this to work? We used to have an AssignmentMonitor > > that didn't need Ganglia up and running -- so we could simply set > > AssignmentMonitor as the default Monitor and we were good. What's changed? > > Rajith Siriwardana wrote: > Chris, earlier it was just generating bogus values in the > AssignmentMonitor. but now it has the ability to use a dynamically asigned > resource monitor (currenlty we have only Ganglia), according to [1] disabling > Ganglia should happend at that level I guess. AssignmentMonitor contains > things like governing the resource assignments like features so I guess it's > a higher level, dynamic assignment of resource monitor is a level below that. > So AssignmentMonitor to work there should be a ResourceMonitor assigned to > it. Correct me if I'm wrong. > > [1] > https://cwiki.apache.org/confluence/display/OODT/Ganglia+plugin+for+CAS+Resource+Manager Thanks Rajith I need to look and read about this. AssignmentMonitor didn't generate bogus values before -- it just generated values that were based on profiling rather than actual data collection. Subtle difference, but important to understand. Anyhoo I will take a look! - Chris --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18944/#review42060 --- On March 9, 2014, 4:44 a.m., Rajith Siriwardana wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/18944/ > --- > > (Updated March 9, 2014, 4:44 a.m.) > > > Review request for oodt and Chris Mattmann. > > > Bugs: OODT-669 > https://issues.apache.org/jira/browse/OODT-669 > > > Repository: oodt > > > Description > --- > > Patch contains disabling Resource Monitor feature in Assignment Monitor and > also some re-factored code for assignment monitor > > > Diffs > - > > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/AssignmentMonitor.java > 1575649 > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/disabledmonitor/DisabledResourceMonitor.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/disabledmonitor/DisabledResourceMonitorFactory.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/GangliaAdapter.java > 1575649 > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/GangliaResourceMonitor.java > 1575649 > trunk/resource/src/main/resources/resource.properties 1575649 > > trunk/resource/src/test/org/apache/oodt/cas/resource/monitor/TestAssignmentMonWithNoResourceMon.java > PRE-CREATION > trunk/resource/src/testdata/test.disabledmon.resource.properties > PRE-CREATION > > Diff: https://reviews.apache.org/r/18944/diff/ > > > Testing > --- > > Disabled monitor with Assignment monitor was tested > > > Thanks, > > Rajith Siriwardana > >
Re: Review Request 21615: Multi-Thread CAS-Crawler
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21615/#review43329 --- Ship it! Ship It! - Chris Mattmann On May 18, 2014, 7:37 p.m., brian Foster wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/21615/ > --- > > (Updated May 18, 2014, 7:37 p.m.) > > > Review request for oodt, Chris Mattmann and Paul Ramirez. > > > Bugs: OODT-684 > https://issues.apache.org/jira/browse/OODT-684 > > > Repository: oodt > > > Description > --- > > Multi-Thread CAS-Crawler > > > Diffs > - > > trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/ProductCrawler.java > 1590893 > > trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/ProductCrawlerWorkerPool.java > PRE-CREATION > > trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/config/ProductCrawlerBean.java > 1590893 > trunk/crawler/src/main/resources/cmd-line-options.xml 1590893 > > Diff: https://reviews.apache.org/r/21615/diff/ > > > Testing > --- > > > Thanks, > > brian Foster > >
Re: Review Request 21615: Multi-Thread CAS-Crawler
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21615/#review43328 --- trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/ProductCrawler.java <https://reviews.apache.org/r/21615/#comment77428> Hey buddy was this not used? Why this removal? - Chris Mattmann On May 18, 2014, 7:37 p.m., brian Foster wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/21615/ > --- > > (Updated May 18, 2014, 7:37 p.m.) > > > Review request for oodt, Chris Mattmann and Paul Ramirez. > > > Bugs: OODT-684 > https://issues.apache.org/jira/browse/OODT-684 > > > Repository: oodt > > > Description > --- > > Multi-Thread CAS-Crawler > > > Diffs > - > > trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/ProductCrawler.java > 1590893 > > trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/ProductCrawlerWorkerPool.java > PRE-CREATION > > trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/config/ProductCrawlerBean.java > 1590893 > trunk/crawler/src/main/resources/cmd-line-options.xml 1590893 > > Diff: https://reviews.apache.org/r/21615/diff/ > > > Testing > --- > > > Thanks, > > brian Foster > >
Re: Review Request 22302: Exception when creating a zipfile of a hierarchical product
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22302/#review44904 --- Ship it! Ship It! - Chris Mattmann On June 6, 2014, 1:16 p.m., Thomas Bennett wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/22302/ > --- > > (Updated June 6, 2014, 1:16 p.m.) > > > Review request for oodt and Chris Mattmann. > > > Bugs: oodt-703 > https://issues.apache.org/jira/browse/oodt-703 > > > Repository: oodt > > > Description > --- > > Skip a data store reference if it is a directory > > Unit test will zip a test hierarchical product > > > Diffs > - > > > trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/data/DataUtils.java > 1600844 > > trunk/webapp/fmprod/src/test/java/org/apache/oodt/cas/product/data/TestDataUtils.java > PRE-CREATION > trunk/webapp/fmprod/src/testdata/data/test-file-1.txt PRE-CREATION > trunk/webapp/fmprod/src/testdata/data/test-file-2.txt PRE-CREATION > trunk/webapp/fmprod/src/testdata/data/test-file-3.txt PRE-CREATION > > Diff: https://reviews.apache.org/r/22302/diff/ > > > Testing > --- > > Unit test passes > > > Thanks, > > Thomas Bennett > >
Re: Keeping CHANGES.txt in sync with actual CHANGES
Just another reason for Lewis and Brian Foster to meet in real life :) Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Lewis John Mcgibbney Reply-To: Date: Friday, June 6, 2014 1:38 PM To: "dev@oodt.apache.org" Subject: Keeping CHANGES.txt in sync with actual CHANGES >Hi Folks, >Can I please ask that we keep CHANGES.txt up-to-date. >I forget this ALL the time... so I know how easy it is. >It is really helpful for quick provenance tracking over time. >Right now I have a local copy of OODT, with a number of files changed >within filemgr component and I just did a svn update... which of course >led >to conflicts. I then went to CHANGES.txt, and I am now having to sniff my >way through through svn log and viewvc to try and piece together what has >been committed to trunk codebase. > >Have a great weekend folks :) >Lewis > >-- >*Lewis*
Re: Review Request 22440: OODT-708: Adding PGE to RADiX
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22440/#review45289 --- Ship it! Ship It! - Chris Mattmann On June 10, 2014, 8:30 p.m., Shakeh Khudikyan wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/22440/ > --- > > (Updated June 10, 2014, 8:30 p.m.) > > > Review request for oodt and Paul Ramirez. > > > Repository: oodt > > > Description > --- > > https://issues.apache.org/jira/browse/OODT-708 > > > Diffs > - > > > ./trunk/mvn/archetypes/radix/src/main/resources/META-INF/maven/archetype-metadata.xml > 1601761 > > ./trunk/mvn/archetypes/radix/src/main/resources/archetype-resources/distribution/pom.xml > 1601761 > > ./trunk/mvn/archetypes/radix/src/main/resources/archetype-resources/distribution/src/main/resources/bin/env.sh > 1601761 > > ./trunk/mvn/archetypes/radix/src/main/resources/archetype-resources/pge/pom.xml > PRE-CREATION > > ./trunk/mvn/archetypes/radix/src/main/resources/archetype-resources/pge/src/main/assembly/assembly.xml > PRE-CREATION > ./trunk/mvn/archetypes/radix/src/main/resources/archetype-resources/pom.xml > 1601761 > > ./trunk/mvn/archetypes/radix/src/main/resources/archetype-resources/resmgr/pom.xml > 1601761 > > ./trunk/mvn/archetypes/radix/src/main/resources/archetype-resources/workflow/pom.xml > 1601761 > > Diff: https://reviews.apache.org/r/22440/diff/ > > > Testing > --- > > > Thanks, > > Shakeh Khudikyan > >
Re: Review Request 22436: See: https://issues.apache.org/jira/browse/OODT-706
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22436/#review45295 --- Ship it! Ship It! - Chris Mattmann On June 10, 2014, 9:12 p.m., Rishi Verma wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/22436/ > --- > > (Updated June 10, 2014, 9:12 p.m.) > > > Review request for oodt, Chris Mattmann and Paul Ramirez. > > > Repository: oodt > > > Description > --- > > See: > https://issues.apache.org/jira/browse/OODT-706 > and > https://issues.apache.org/jira/browse/OODT-707 > > > Diffs > - > > > http://svn.apache.org/repos/asf/oodt/trunk/mvn/archetypes/radix/src/main/resources/META-INF/maven/archetype-metadata.xml > 1601497 > > http://svn.apache.org/repos/asf/oodt/trunk/mvn/archetypes/radix/src/main/resources/archetype-resources/README.txt > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/mvn/archetypes/radix/src/main/resources/archetype-resources/distribution/pom.xml > 1601497 > > http://svn.apache.org/repos/asf/oodt/trunk/mvn/archetypes/radix/src/main/resources/archetype-resources/filemgr/pom.xml > 1601497 > > http://svn.apache.org/repos/asf/oodt/trunk/mvn/archetypes/radix/src/main/resources/archetype-resources/filemgr/src/main/assembly/assembly.fm-solr-catalog.xml > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/mvn/archetypes/radix/src/main/resources/archetype-resources/filemgr/src/main/assembly/assembly.xml > 1601497 > > http://svn.apache.org/repos/asf/oodt/trunk/mvn/archetypes/radix/src/main/resources/archetype-resources/filemgr/src/main/resources/etc/filemgr.fm-solr-catalog.properties > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/mvn/archetypes/radix/src/main/resources/archetype-resources/filemgr/src/main/resources/etc/filemgr.properties > 1601497 > > Diff: https://reviews.apache.org/r/22436/diff/ > > > Testing > --- > > Locally tested via fresh 'mvn install' of RADiX archetype. > > Two scenarios tested: > 1. Default OODT RADiX build: mvn package > 2. Solr OODT RADiX build: mvn package -Pfm-solr-catalog > > Test 1 passed (OPSUI up on localhost:8080/opsui and connected to Lucene FM) > Test 2 passed (OPSUI up on localhost:8080/opsui and connected to Solr FM, > solr webapp accessible http://localhost:8080/solr, solr directory present > under OODT deployment dir) > > > Thanks, > > Rishi Verma > >
Re: Review Request 22575: Simplify OODT RADiX setup: provide pre-configured RADiX via Vagrant
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22575/#review45624 --- Ship it! Ship It! - Chris Mattmann On June 13, 2014, 1:09 p.m., Rishi Verma wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/22575/ > --- > > (Updated June 13, 2014, 1:09 p.m.) > > > Review request for oodt. > > > Repository: oodt > > > Description > --- > > Please see: https://issues.apache.org/jira/browse/OODT-676 > > > Diffs > - > > http://svn.apache.org/repos/asf/oodt/trunk/vagrant/radix/README.txt > PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/vagrant/radix/Vagrantfile > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/vagrant/radix/vagrant/conf/terminal/bashrc > PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/vagrant/radix/vagrant/env.sh > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/vagrant/radix/vagrant/provision.sh > PRE-CREATION > > Diff: https://reviews.apache.org/r/22575/diff/ > > > Testing > --- > > Locally tested from fresh install of tar-ball: > https://issues.apache.org/jira/secure/attachment/12650125/radix-vagrant.tar > Results: OODT RADiX running filemgr, resmgr, workflow mgr, and tomcat - > accessible from OPSUI webapp. > > > Thanks, > > Rishi Verma > >
Re: Review Request 22575: Simplify OODT RADiX setup: provide pre-configured RADiX via Vagrant
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22575/#review45622
---
http://svn.apache.org/repos/asf/oodt/trunk/vagrant/radix/Vagrantfile
<https://reviews.apache.org/r/22575/#comment80494>
How about: "Welcome to Apache OODT Radix! Built by Vagrant, managed by
Puppet."
http://svn.apache.org/repos/asf/oodt/trunk/vagrant/radix/vagrant/env.sh
<https://reviews.apache.org/r/22575/#comment80495>
this is OpenJDK, right?
http://svn.apache.org/repos/asf/oodt/trunk/vagrant/radix/vagrant/provision.sh
<https://reviews.apache.org/r/22575/#comment80496>
maybe call it "oodt-${version}" instead of "oodt-trunk"
- Chris Mattmann
On June 13, 2014, 1:09 p.m., Rishi Verma wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22575/
> ---
>
> (Updated June 13, 2014, 1:09 p.m.)
>
>
> Review request for oodt.
>
>
> Repository: oodt
>
>
> Description
> ---
>
> Please see: https://issues.apache.org/jira/browse/OODT-676
>
>
> Diffs
> -
>
> http://svn.apache.org/repos/asf/oodt/trunk/vagrant/radix/README.txt
> PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/vagrant/radix/Vagrantfile
> PRE-CREATION
>
> http://svn.apache.org/repos/asf/oodt/trunk/vagrant/radix/vagrant/conf/terminal/bashrc
> PRE-CREATION
> http://svn.apache.org/repos/asf/oodt/trunk/vagrant/radix/vagrant/env.sh
> PRE-CREATION
>
> http://svn.apache.org/repos/asf/oodt/trunk/vagrant/radix/vagrant/provision.sh
> PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22575/diff/
>
>
> Testing
> ---
>
> Locally tested from fresh install of tar-ball:
> https://issues.apache.org/jira/secure/attachment/12650125/radix-vagrant.tar
> Results: OODT RADiX running filemgr, resmgr, workflow mgr, and tomcat -
> accessible from OPSUI webapp.
>
>
> Thanks,
>
> Rishi Verma
>
>
Re: Review Request 18944: [OODT-669] Disabling resource monitor in Assignment monitor feature
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18944/#review45711 --- trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/disabledmonitor/DisabledResourceMonitor.java <https://reviews.apache.org/r/18944/#comment80672> Why do we need this class? I don't get it. trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/GangliaAdapter.java <https://reviews.apache.org/r/18944/#comment80673> don't get this change? trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/GangliaAdapter.java <https://reviews.apache.org/r/18944/#comment80674> what's up with this change? trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/GangliaAdapter.java <https://reviews.apache.org/r/18944/#comment80675> Ditto trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/GangliaResourceMonitor.java <https://reviews.apache.org/r/18944/#comment80676> why all these spacing changes? trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/GangliaResourceMonitor.java <https://reviews.apache.org/r/18944/#comment80677> spacing change trunk/resource/src/main/resources/resource.properties <https://reviews.apache.org/r/18944/#comment80678> shouldn't be disabled - we just need the code that was there before for AssignmentMonitor. Then we should have a GangliaAssignmentMonitor that you can enable - but we don't need a disabled version? - Chris Mattmann On March 9, 2014, 4:44 a.m., Rajith Siriwardana wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/18944/ > --- > > (Updated March 9, 2014, 4:44 a.m.) > > > Review request for oodt and Chris Mattmann. > > > Bugs: OODT-669 > https://issues.apache.org/jira/browse/OODT-669 > > > Repository: oodt > > > Description > --- > > Patch contains disabling Resource Monitor feature in Assignment Monitor and > also some re-factored code for assignment monitor > > > Diffs > - > > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/AssignmentMonitor.java > 1575649 > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/disabledmonitor/DisabledResourceMonitor.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/disabledmonitor/DisabledResourceMonitorFactory.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/GangliaAdapter.java > 1575649 > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/GangliaResourceMonitor.java > 1575649 > trunk/resource/src/main/resources/resource.properties 1575649 > > trunk/resource/src/test/org/apache/oodt/cas/resource/monitor/TestAssignmentMonWithNoResourceMon.java > PRE-CREATION > trunk/resource/src/testdata/test.disabledmon.resource.properties > PRE-CREATION > > Diff: https://reviews.apache.org/r/18944/diff/ > > > Testing > --- > > Disabled monitor with Assignment monitor was tested > > > Thanks, > > Rajith Siriwardana > >
Re: Review Request 22575: Simplify OODT RADiX setup: provide pre-configured RADiX via Vagrant
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22575/#review45802 --- Ship it! Ship It! - Chris Mattmann On June 16, 2014, 7:30 p.m., Rishi Verma wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/22575/ > --- > > (Updated June 16, 2014, 7:30 p.m.) > > > Review request for oodt. > > > Repository: oodt > > > Description > --- > > Please see: https://issues.apache.org/jira/browse/OODT-676 > > > Diffs > - > > http://svn.apache.org/repos/asf/oodt/trunk/vagrant/radix/README.txt > PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/vagrant/radix/Vagrantfile > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/vagrant/radix/vagrant/conf/terminal/bashrc > PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/vagrant/radix/vagrant/env.sh > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/vagrant/radix/vagrant/provision.sh > PRE-CREATION > > Diff: https://reviews.apache.org/r/22575/diff/ > > > Testing > --- > > Locally tested from fresh install of tar-ball: > https://issues.apache.org/jira/secure/attachment/12650125/radix-vagrant.tar > Results: OODT RADiX running filemgr, resmgr, workflow mgr, and tomcat - > accessible from OPSUI webapp. > > > Thanks, > > Rishi Verma > >
Re: Review Request 22671: OODT-685 Fix OODT 0.7-SNAPSHOT Jenkins nightly build on Ubuntu (1st pass - e.g. define new test structure and resources)
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22671/#review45888
---
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/AddProductTypeCliAction.java
<https://reviews.apache.org/r/22671/#comment80973>
Seems orthogonal, but definitely needed.
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/REMOVE.log
<https://reviews.apache.org/r/22671/#comment80974>
I assume all these get added back, right? Did you forget to add the new
paths to your patch for review?
- Chris Mattmann
On June 17, 2014, 3:02 a.m., Lewis McGibbney wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22671/
> ---
>
> (Updated June 17, 2014, 3:02 a.m.)
>
>
> Review request for oodt and Chris Mattmann.
>
>
> Repository: oodt
>
>
> Description
> ---
>
> This is a multi-faceted issue and the core issue {OODT-685 Fix OODT
> 0.7-SNAPSHOT Jenkins nightly build on Ubuntu} IS NOT fixed entirely by this
> patch.
> This patch does however:
>
> * Set a new directory structure for filemgr component which reflects newer
> Maven build and /resources directories as explained in [0]
> * Address OODT-688 filemgr jar should not ship with ALL resources, as it
> sets filters within the module Maven pom.xml for what we include within the
> generated filemgr jar file. This excludes a bunch of junk that was previously
> shipping with the jar.
> * Addresses OODT-691 by removing the j-globus repository entry within the
> pushpull module pom.xml.
> * Addresses some javac warnings we were getting by not ensuring type
> checking was in place for local variables in a number of places.
>
> It is my intention to step through the review request and address the failing
> tests incrementally when I can. The first step is actually addressing the
> above... the remainder will come in due course.
>
>
> [0] https://issues.apache.org/jira/browse/OODT-685
> [1] https://issues.apache.org/jira/browse/OODT-688
> [2] https://issues.apache.org/jira/browse/OODT-691
>
>
> Diffs
> -
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/pom.xml 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/DataSourceCatalog.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/AddProductTypeCliAction.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/ingest/CmdLineIngester.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/repository/XMLRepositoryManager.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/exceptions/RepositoryManagerException.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManagerClient.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/GenericFileManagerObjectFactory.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/XmlRpcStructFactory.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/XmlStructFactory.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/REMOVE.log
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/filemgr.properties
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/MockCatalog.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/MockCatalogFactory.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestDataSourceCatalog.java
Re: Review Request 22671: OODT-685 Fix OODT 0.7-SNAPSHOT Jenkins nightly build on Ubuntu (1st pass - e.g. define new test structure and resources)
> On June 17, 2014, 3:09 a.m., Chris Mattmann wrote:
> > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/AddProductTypeCliAction.java,
> > line 43
> > <https://reviews.apache.org/r/22671/diff/1/?file=611086#file611086line43>
> >
> > Seems orthogonal, but definitely needed.
>
> Lewis McGibbney wrote:
> This relates to Rishi's patch on
> https://reviews.apache.org/r/21672/
> I was getting NPE when running tests... we can revert this and deal with
> it in Rishi's issue. wdyt?
+1, SGTM.
> On June 17, 2014, 3:09 a.m., Chris Mattmann wrote:
> > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/REMOVE.log,
> > line 3
> > <https://reviews.apache.org/r/22671/diff/1/?file=611096#file611096line3>
> >
> > I assume all these get added back, right? Did you forget to add the new
> > paths to your patch for review?
>
> Lewis McGibbney wrote:
> Yes Chris, the planned patch essentially does a bulk svn mv of
>
> D filemgr/src/testdata
> > moved to filemgr/src/test/resources/testdata
+1
- Chris
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22671/#review45888
---
On June 17, 2014, 3:02 a.m., Lewis McGibbney wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22671/
> ---
>
> (Updated June 17, 2014, 3:02 a.m.)
>
>
> Review request for oodt and Chris Mattmann.
>
>
> Repository: oodt
>
>
> Description
> ---
>
> This is a multi-faceted issue and the core issue {OODT-685 Fix OODT
> 0.7-SNAPSHOT Jenkins nightly build on Ubuntu} IS NOT fixed entirely by this
> patch.
> This patch does however:
>
> * Set a new directory structure for filemgr component which reflects newer
> Maven build and /resources directories as explained in [0]
> * Address OODT-688 filemgr jar should not ship with ALL resources, as it
> sets filters within the module Maven pom.xml for what we include within the
> generated filemgr jar file. This excludes a bunch of junk that was previously
> shipping with the jar.
> * Addresses OODT-691 by removing the j-globus repository entry within the
> pushpull module pom.xml.
> * Addresses some javac warnings we were getting by not ensuring type
> checking was in place for local variables in a number of places.
>
> It is my intention to step through the review request and address the failing
> tests incrementally when I can. The first step is actually addressing the
> above... the remainder will come in due course.
>
>
> [0] https://issues.apache.org/jira/browse/OODT-685
> [1] https://issues.apache.org/jira/browse/OODT-688
> [2] https://issues.apache.org/jira/browse/OODT-691
>
>
> Diffs
> -
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/pom.xml 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/DataSourceCatalog.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/AddProductTypeCliAction.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/ingest/CmdLineIngester.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/repository/XMLRepositoryManager.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/exceptions/RepositoryManagerException.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManagerClient.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/GenericFileManagerObjectFactory.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/XmlRpcStructFactory.java
> 1603043
>
> http://svn.apache.org/repos/
Re: Review Request 22671: OODT-685 Fix OODT 0.7-SNAPSHOT Jenkins nightly build on Ubuntu (1st pass - e.g. define new test structure and resources)
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22671/#review45892
---
Ship it!
Ship It!
- Chris Mattmann
On June 17, 2014, 3:02 a.m., Lewis McGibbney wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22671/
> ---
>
> (Updated June 17, 2014, 3:02 a.m.)
>
>
> Review request for oodt and Chris Mattmann.
>
>
> Repository: oodt
>
>
> Description
> ---
>
> This is a multi-faceted issue and the core issue {OODT-685 Fix OODT
> 0.7-SNAPSHOT Jenkins nightly build on Ubuntu} IS NOT fixed entirely by this
> patch.
> This patch does however:
>
> * Set a new directory structure for filemgr component which reflects newer
> Maven build and /resources directories as explained in [0]
> * Address OODT-688 filemgr jar should not ship with ALL resources, as it
> sets filters within the module Maven pom.xml for what we include within the
> generated filemgr jar file. This excludes a bunch of junk that was previously
> shipping with the jar.
> * Addresses OODT-691 by removing the j-globus repository entry within the
> pushpull module pom.xml.
> * Addresses some javac warnings we were getting by not ensuring type
> checking was in place for local variables in a number of places.
>
> It is my intention to step through the review request and address the failing
> tests incrementally when I can. The first step is actually addressing the
> above... the remainder will come in due course.
>
>
> [0] https://issues.apache.org/jira/browse/OODT-685
> [1] https://issues.apache.org/jira/browse/OODT-688
> [2] https://issues.apache.org/jira/browse/OODT-691
>
>
> Diffs
> -
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/pom.xml 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/DataSourceCatalog.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/AddProductTypeCliAction.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/ingest/CmdLineIngester.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/repository/XMLRepositoryManager.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/exceptions/RepositoryManagerException.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManagerClient.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/GenericFileManagerObjectFactory.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/XmlRpcStructFactory.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/XmlStructFactory.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/REMOVE.log
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/filemgr.properties
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/MockCatalog.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/MockCatalogFactory.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestDataSourceCatalog.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestLuceneCatalog.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestMappedDataSourceCatalog.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestOrderedDataSourceCatalog.java
> 16030
Re: Review Request 22671: OODT-685 Fix OODT 0.7-SNAPSHOT Jenkins nightly build on Ubuntu (1st pass - e.g. define new test structure and resources)
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22671/#review45895
---
Ship it!
Ship It!
- Chris Mattmann
On June 17, 2014, 3:33 a.m., Lewis McGibbney wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22671/
> ---
>
> (Updated June 17, 2014, 3:33 a.m.)
>
>
> Review request for oodt and Chris Mattmann.
>
>
> Bugs: oodt-685, oodt-688 and oodt-691
> https://issues.apache.org/jira/browse/oodt-685
> https://issues.apache.org/jira/browse/oodt-688
> https://issues.apache.org/jira/browse/oodt-691
>
>
> Repository: oodt
>
>
> Description
> ---
>
> This is a multi-faceted issue and the core issue {OODT-685 Fix OODT
> 0.7-SNAPSHOT Jenkins nightly build on Ubuntu} IS NOT fixed entirely by this
> patch.
> This patch does however:
>
> * Set a new directory structure for filemgr component which reflects newer
> Maven build and /resources directories as explained in [0]
> * Address OODT-688 filemgr jar should not ship with ALL resources, as it
> sets filters within the module Maven pom.xml for what we include within the
> generated filemgr jar file. This excludes a bunch of junk that was previously
> shipping with the jar.
> * Addresses OODT-691 by removing the j-globus repository entry within the
> pushpull module pom.xml.
> * Addresses some javac warnings we were getting by not ensuring type
> checking was in place for local variables in a number of places.
>
> It is my intention to step through the review request and address the failing
> tests incrementally when I can. The first step is actually addressing the
> above... the remainder will come in due course.
>
>
> [0] https://issues.apache.org/jira/browse/OODT-685
> [1] https://issues.apache.org/jira/browse/OODT-688
> [2] https://issues.apache.org/jira/browse/OODT-691
>
>
> Diffs
> -
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/pom.xml 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/DataSourceCatalog.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/AddProductTypeCliAction.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/ingest/CmdLineIngester.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/repository/XMLRepositoryManager.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/exceptions/RepositoryManagerException.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManagerClient.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/GenericFileManagerObjectFactory.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/XmlRpcStructFactory.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/XmlStructFactory.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/REMOVE.log
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/cmd-line-options.xml
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/filemgr.properties
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/MockCatalog.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/MockCatalogFactory.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestDataSourceCatalog.java
> 1603043
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalo
Re: cas-product .war artifact huge
@Ross Laidlaw can answer this - I'm thinking the CXF integration? Note -- this can be adapted as Sergey B in Tika has shown in some thread that we can just dep on the minimal jar for CXF Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Lewis John Mcgibbney Reply-To: Date: Monday, June 30, 2014 8:10 PM To: "dev@oodt.apache.org" Subject: cas-product .war artifact huge >Hi Folks, >Anyone have a clue why the cas-product war artifact is 30,500KB in size? >Lewis > >-- >*Lewis*
FW: [jira] [Created] (LEGAL-206) "greenmail" library purported to be ASL may actually be LGPL; is a dependency in several ASF projects
FYI OODT peeps, per hossman (CC'ed). -Original Message- From: "Hoss Man (JIRA)" Reply-To: "legal-disc...@apache.org" Date: Thursday, July 3, 2014 10:14 AM To: "legal-disc...@apache.org" Subject: [jira] [Created] (LEGAL-206) "greenmail" library purported to be ASL may actually be LGPL; is a dependency in several ASF projects >Hoss Man created LEGAL-206: >-- > > Summary: "greenmail" library purported to be ASL may >actually be LGPL; is a dependency in several ASF projects > Key: LEGAL-206 > URL: https://issues.apache.org/jira/browse/LEGAL-206 > Project: Legal Discuss > Issue Type: Question >Reporter: Hoss Man > > >Be advised that at least 5 Apache projects appear to depend on a third >party library known as "greenmail" which advertises itself in may places >as being licensed under the ASL 2.0, however most of the greemail source >files contain headers claiming LGPL. > >This issue requests that the Legal VP advise the affected projects on >what they need to do (if anything) moving forward to be in compliance >with ASF policy. > >Projects known to be affected (but there may be more)... > >* Apache Syncope: http://svn.apache.org/repos/asf/syncope/trunk/pom.xml >* Apache Geronimo: >https://svn.apache.org/repos/asf/geronimo/javamail/trunk/geronimo-javamail >_1.4/geronimo-javamail_1.4_provider/pom.xml >* Apache OODT: >http://svn.apache.org/repos/asf/oodt/trunk/protocol/imaps/pom.xml >* Apache Oozie: >https://git-wip-us.apache.org/repos/asf?p=oozie.git;a=blob;f=pom.xml;h=bad >1e0fbee619f2e5020733792c4a09256b69dcf;hb=master >* Apache Axis2: >http://svn.apache.org/repos/asf/axis/axis2/java/core/trunk/modules/transpo >rt/mail/pom.xml > > > > > >-- >This message was sent by Atlassian JIRA >(v6.2#6252) > >- >To unsubscribe, e-mail: legal-discuss-unsubscr...@apache.org >For additional commands, e-mail: legal-discuss-h...@apache.org >
Re: [jira] [Created] (LEGAL-206) "greenmail" library purported to be ASL may actually be LGPL; is a dependency in several ASF projects
Yep, https://github.com/chrismattmann/drat/ Cheers, Chris Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Tyler Palsulich Reply-To: Date: Thursday, July 3, 2014 10:50 AM To: Cc: Subject: Re: FW: [jira] [Created] (LEGAL-206) "greenmail" library purported to be ASL may actually be LGPL; is a dependency in several ASF projects >Sounds like a job for DRAT. We've wanted to run it on all of the Apache >svn >for a while. > >Tyler >On Jul 3, 2014 10:17 AM, "Chris Mattmann" wrote: > >> FYI OODT peeps, per hossman (CC'ed). >> >> >> >> -Original Message- >> From: "Hoss Man (JIRA)" >> Reply-To: "legal-disc...@apache.org" >> Date: Thursday, July 3, 2014 10:14 AM >> To: "legal-disc...@apache.org" >> Subject: [jira] [Created] (LEGAL-206) "greenmail" library purported to >>be >> ASL may actually be LGPL; is a dependency in several ASF projects >> >> >Hoss Man created LEGAL-206: >> >-- >> > >> > Summary: "greenmail" library purported to be ASL may >> >actually be LGPL; is a dependency in several ASF projects >> > Key: LEGAL-206 >> > URL: https://issues.apache.org/jira/browse/LEGAL-206 >> > Project: Legal Discuss >> > Issue Type: Question >> >Reporter: Hoss Man >> > >> > >> >Be advised that at least 5 Apache projects appear to depend on a third >> >party library known as "greenmail" which advertises itself in may >>places >> >as being licensed under the ASL 2.0, however most of the greemail >>source >> >files contain headers claiming LGPL. >> > >> >This issue requests that the Legal VP advise the affected projects on >> >what they need to do (if anything) moving forward to be in compliance >> >with ASF policy. >> > >> >Projects known to be affected (but there may be more)... >> > >> >* Apache Syncope: http://svn.apache.org/repos/asf/syncope/trunk/pom.xml >> >* Apache Geronimo: >> > >> >>https://svn.apache.org/repos/asf/geronimo/javamail/trunk/geronimo-javamai >>l >> >_1.4/geronimo-javamail_1.4_provider/pom.xml >> >* Apache OODT: >> >http://svn.apache.org/repos/asf/oodt/trunk/protocol/imaps/pom.xml >> >* Apache Oozie: >> > >> >>https://git-wip-us.apache.org/repos/asf?p=oozie.git;a=blob;f=pom.xml;h=ba >>d >> >1e0fbee619f2e5020733792c4a09256b69dcf;hb=master >> >* Apache Axis2: >> > >> >>http://svn.apache.org/repos/asf/axis/axis2/java/core/trunk/modules/transp >>o >> >rt/mail/pom.xml >> > >> > >> > >> > >> > >> >-- >> >This message was sent by Atlassian JIRA >> >(v6.2#6252) >> > >> >- >> >To unsubscribe, e-mail: legal-discuss-unsubscr...@apache.org >> >For additional commands, e-mail: legal-discuss-h...@apache.org >> > >> >> >>
Re: UPDATE-PROBABLY SOLVED: File manager keeps connections to SOLR in CLOSE_WAIT state for hours
Hey Konstantinos can you sign up for a JIRA account?
Then I can grant you permissions to add/update/edit issues.
Chris Mattmann
chris.mattm...@gmail.com
-Original Message-
From: Konstantinos Mavrommatis
Reply-To:
Date: Thursday, July 10, 2014 7:11 AM
To: "dev@oodt.apache.org"
Subject: RE: UPDATE-PROBABLY SOLVED: File manager keeps connections to
SOLR in CLOSE_WAIT state for hours
>Unless I am missing something I am not able to create an issue in JIRA :(
>Thanks
>K
>
>> -Original Message-
>> From: Ramirez, Paul M (398J) [mailto:paul.m.rami...@jpl.nasa.gov]
>> Sent: Thursday, July 10, 2014 4:06 PM
>> To:
>> Subject: Re: UPDATE-PROBABLY SOLVED: File manager keeps connections to
>> SOLR in CLOSE_WAIT state for hours
>>
>> Konstantinos,
>>
>> Could you open a Jira issue for this and attach a patch? This would be
>> helpful for the project.
>>
>> https://issues.apache.org/jira/browse/OODT/?selectedTab=com.atlassian.j
>> ira.jira-projects-plugin:summary-panel
>>
>>
>> Thanks,
>> Paul Ramirez
>>
>> > On Jul 10, 2014, at 5:23 AM, "Konstantinos Mavrommatis"
>> wrote:
>> >
>> > Hi,
>> >
>> > Following the suggestion at
>> >
>> > http://blogs.nuxeo.com/development/2013/02/using-httpclient-properly-
>> a
>> > void-closewait-tcp-connections/
>> >
>> >
>> >
>> > I modified the code in
>> >
>> >
>> src/main/java/org/apache/oodt/cas/filemgr/catalog/solr/SolrClient.java
>> >
>> >
>> >
>> > and added the line highlighted in red
>> (method.setRequestHeader("Connection","close").
>> >
>> > This seems to have resolved the problem at least until now ingestion
>> of few thousand files have not produced any stale connection.
>> >
>> > Please let me know your thoughts on this solution.
>> >
>> > Best,
>> >
>> > Konstantinos
>> >
>> >
>> >
>> > ==
>> >
>> > private String doHttp(HttpMethod method) throws Exception {
>> >
>> >
>> >
>> >StringBuilder response = new StringBuilder();
>> >
>> >BufferedReader br = null;
>> >
>> >try {
>> >
>> >
>> >
>> >// send request
>> >
>> >HttpClient httpClient = new HttpClient();
>> >
>> >// 10 July 2014. attempting to avoid problems
>> > with CLOSE_WAIT connections Based on
>> >
>> >
>> > //http://blogs.nuxeo.com/development/2013/02/using-httpclient-
>> properly
>> > -avoid-closewait-tcp-connections/
>> >
>> >method.setRequestHeader("Connection",
>> "close");
>> >
>> >int statusCode = httpClient.executeMethod(method);
>> >
>> >
>> >
>> >// read response
>> >
>> >if (statusCode != HttpStatus.SC_OK) {
>> >
>> >
>> >
>> >// still consume the response
>> >
>> >method.getResponseBodyAsString();
>> >
>> > throw new CatalogException("HTTP method failed: " +
>> > method.getStatusLine());
>> >
>> >
>> >
>> >} else {
>> >
>> >
>> >
>> >// read the response body.
>> >
>> >br = new BufferedReader(new
>> > InputStreamReader(method.getResponseBodyAsStream()));
>> >
>> >String readLine;
>> >
>> >while(((readLine = br.readLine()) != null)) {
>> >
>> > response.append(readLine);
>> >
>> >}
>> >
>> >
>> >
>> > ===
>> >
>> >
>> >
>> > Information about the system:
>> >
>> > OS is MacOS Mavericks,
>> >
>> > SOLR version : 4.6.1 , SOLR runs on a single server, no replication
>> at all.
>> >
>> > Tomcat verision: 7.0.50
>> >
>> > OODT version is 0.6
>> >
>> > Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
>> >
>> >
>> >
>>
Re: [ANNOUNCE] Tyler Palsulich is now on the PMC too!
Addressed, a while ago :) http://s.apache.org/uI https://issues.apache.org/jira/browse/OODT-369 -Original Message- From: Reply-To: Date: Wednesday, July 23, 2014 10:20 AM To: Cc: dev Subject: Re: [ANNOUNCE] Tyler Palsulich is now on the PMC too! >Give it a shot with Maven-2.2.1. (I don't think we're ready for Maven 3, >which itself is another issue that needs addressing.) > >--k > >On 2014-07-23, at 10:49 AM, Tyler Palsulich wrote: > >> Thanks all! I'm looking forward to the roadmap call. >> >> Before then, I'm having issues even passing the Commons tests on a >>clean install... Here is the output of `mvn test -X`: >>https://paste.apache.org/75u9. >> >> ? mvn --version >> Apache Maven 3.2.1 (ea8b2b07643dbb1b84b6d16e1f08391b666bc1e9; >>2014-02-14T09:37:52-08:00) >> Maven home: /usr/local/Cellar/maven/3.2.1/libexec >> Java version: 1.7.0_60, vendor: Oracle Corporation >> Java home: >>/Library/Java/JavaVirtualMachines/jdk1.7.0_60.jdk/Contents/Home/jre >> Default locale: en_US, platform encoding: UTF-8 >> OS name: "mac os x", version: "10.9.3", arch: "x86_64", family: "mac" >> >> TL;DR: I don't see any failed tests. -DfailIfNoTests=false doesn't >>help. My surefire directory doesn't have any failures. >> >> Is there any config that needs to be set up before install? Or, am I >>missing something in the output above? >> >> Thank for any help, >> Tyler >> >> >> On Tue, Jul 22, 2014 at 8:43 PM, Rajith Siriwardana >> wrote: >> Welcome Tyler..!!! >> >> >> Cheers, >> --Rajith >> >> >> On Wed, Jul 23, 2014 at 1:06 AM, Michael Starch >>wrote: >> Great! Welcome, Tyler. >> >> As another new PMC I aggree that we should discuss a roadmap. >> >> -Michael >> >> On Jul 22, 2014 11:58 AM, "Michael Joyce" wrote: >> Yay! Welcome Tyler!!! >> >> >> -- Joyce >> >> >> On Tue, Jul 22, 2014 at 11:36 AM, Tom Barber >>wrote: >> Aye I'll +1 that, and move this over to dev@ >> >> We have an ever expanding PMC, so lets do some Project Managing! ;) >> >> I think we need to come up with some roadmap goals, both regarding >>features, and also the existing code base, build systems and delivery. >> >> Personally I think it would be benificial to get a bunch of us on the >>phone and discuss it in real time (I'm GMT based which does make it a >>little tricker) rather than hash it out over a 6 month chain of emails. >>Like I said the other day, I don't really know how other projects manage >>this type of thing so those with more experience may want to chime in >>here. >> >> Regards >> >> Tom >> >> >> >> On 22/07/14 18:33, Lewis John Mcgibbney wrote: >>> Hi Tyler, >>> Please see Tom's thread from before the weekend. >>> I say we have a full on discussion on establishing a roadmap. >>> For us to establish and agree on this will be extremely beneficial for >>>the project. >>> Lewis >>> >>> >>> On Tue, Jul 22, 2014 at 10:29 AM, Tyler Palsulich >>> wrote: >>> Hi All, >>> >>> Thanks for inviting me to join as a committer and PMC member! I'm >>>happy to be part of the project. Any tips on where to dive in? >>> >>> Tyler >> >> >> >> >
Re: Review Request 23878: Successor to rlaidlaw reviewboard #23870 for OODT-685
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23878/#review48676 --- Ship it! Ship It! - Chris Mattmann On July 24, 2014, 5:52 a.m., Lewis McGibbney wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23878/ > --- > > (Updated July 24, 2014, 5:52 a.m.) > > > Review request for oodt, Tom Barber, Chris Mattmann, Sean Kelly, Paul > Ramirez, Rishi Verma, Ross Laidlaw, and Tyler Palsulich. > > > Bugs: OODT-685 > https://issues.apache.org/jira/browse/OODT-685 > > > Repository: oodt > > > Description > --- > > Hi Folks, > OK in the name of staying sane, I am fully behind the decision to just get > filemgr back stable. > This review board is somewhat of a successor to @rlaidlaw's patch attached to > review board request - https://reviews.apache.org/r/23870/ > Bascially, there is now one failing test (as below) which I cannot debug as > it passes for me locally within my IDE!!! > What is more is that from the command line it actually DOES fail!!! > > Results : > > Failed tests: > > testGetElements(org.apache.oodt.cas.filemgr.validation.TestXMLValidationLayer): > There aren't exactly 4 elements in the test samples! expected:<4> but > was:<11> > > Tests run: 190, Failures: 1, Errors: 0, Skipped: 0 > > From within my IDE the failing test is located at > org.apache.oodt.cas.filemgr.validation.TestXMLValidationLayer#testModifyElement, > which DOES NOT fail when I load it from the command line. > > I think we should commit these improvements to the filemgr test suite before > moving on to work on maybe upgrading to JUnit 4.11 as per > https://issues.apache.org/jira/browse/OODT-687 > > > Diffs > - > > > http://svn.apache.org/repos/asf/oodt/trunk/commons/src/main/java/org/apache/oodt/commons/xml/XMLUtils.java > 1612972 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/validation/XMLValidationLayer.java > 1612972 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/catalog/TestDataSourceCatalog.java > 1612972 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/catalog/TestMappedDataSourceCatalog.java > 1612972 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/catalog/TestOrderedDataSourceCatalog.java > 1612972 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/TestFileManagerCli.java > 1612972 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/datatransfer/TestLocalDataTransferer.java > 1612972 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/ingest/TestCachedIngester.java > 1612972 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/ingest/TestLocalCache.java > 1612972 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/ingest/TestRmiCache.java > 1612972 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/ingest/TestStdIngester.java > 1612972 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/repository/TestXMLRepositoryManager.java > 1612972 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/structs/type/TestTypeHandler.java > 1612972 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/system/TestXmlRpcFileManager.java > 1612972 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/system/TestXmlRpcFileManagerClient.java > 1612972 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/tools/TestExpImpCatalog.java > 1612972 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/tools/TestMetadataBasedProductMover.java > 1612972 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/util/TestXmlRpcStructFactory.java > 1612972 > > http://svn.apache.org/repos/as
Re: Multiple Processing Paradigms at Once
Hi Mike, I think you are using the wengine branch of Apache OODT. That is unmaintained. I would sincerely urge you to get this working in trunk, that's where the developers are working right now. Cheers, Chris P.S. Let me think more about the below I have some ideas there. Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Michael Starch Reply-To: Date: Wednesday, August 6, 2014 8:29 AM To: Subject: Multiple Processing Paradigms at Once >All, > >I am working on upgrading OODT to allow it to process streaming data, >alongside traditional non-streaming jobs. This means that some jobs need >to be run by the resource manager, and other jobs need to be submitted to >the stream-processing. Therefore, processing needs to be forked or >multiplexed at some point in the life-cycle. > >There are two places where this can be done: workflow manager runners, and >the resource manager. Currently, I am working on building workflow >runners, and doing the job-multiplexing there because this cuts out one >superfluous step for the streaming jobs (namely going to the resource >manager before being routed). > >Are there any comments on this approach or does this approach make sense? > >-Michael Starch
Re: OODT Officially Stable - Hats off to Ross Laidlaw
Ba-Bewm Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Lewis John Mcgibbney Reply-To: Date: Saturday, August 9, 2014 10:38 AM To: "dev@oodt.apache.org" Subject: Re: OODT Officially Stable - Hats off to Ross Laidlaw >ACK >AutoACK >+1 >Bewm > > >On Sat, Aug 9, 2014 at 8:10 AM, Cameron Goodale >wrote: > >> Great work Ross. I know it hasn't been easy getting those tests fixed >>up, >> so thank you for being persistent and seeing it through. >> >> -Cam >> >> >> On Fri, Aug 8, 2014 at 2:05 PM, Ross Laidlaw >> wrote: >> >> > Thanks all! Glad to contribute! There are some remaining issues to >> > discuss/resolve with file manager tests (e.g. see OODT-736 and >>OODT-741), >> > but we're getting there! >> > >> > There are also some issues with the protocol tests that Chris >>highlighted >> > in OODT-644. So Jenkins builds will probably still fail sometimes, >>but >> > hopefully we can fix these soon. >> > >> > >> > Ross >> > >> > >> > >> > On 8 August 2014 13:47, Verma, Rishi (398J) >> > wrote: >> > >> >> Way to go Ross, way to go everyone! >> >> >> >> Rishi >> >> Sent from my iPhone >> >> >> >> On Aug 8, 2014, at 1:29 PM, "Chris Mattmann" >> >> >> wrote: >> >> >> >> > Fscking awesome!! >> >> > >> >> > Thank you Ross and everyone! >> >> > >> >> > >> >> > Chris Mattmann >> >> > chris.mattm...@gmail.com >> >> > >> >> > >> >> > >> >> > >> >> > -Original Message- >> >> > From: Tom Barber >> >> > Reply-To: >> >> > Date: Friday, August 8, 2014 1:07 PM >> >> > To: >> >> > Subject: Re: OODT Officially Stable - Hats off to Ross Laidlaw >> >> > >> >> >> Wp >> >> >> On 8 Aug 2014 21:05, "Lewis John Mcgibbney" < >> lewis.mcgibb...@gmail.com >> >> > >> >> >> wrote: >> >> >> >> >> >> Folks, >> >> >> >> >> >> The OODT trunk (0.7-SNAPSHOT) codebase is stable [0] now for the >> first >> >> >> time in a while which is extremely good news. >> >> >> >> >> >> I personally (and I'm sure I speak on behalf of many of us) am >> >> extremely >> >> >> appreciative of the work Ross has recently put in to ensuring the >> >> >> codebase is stabalized. >> >> >> >> >> >> Hats off Ross. The drinks are on me... and make sure to remind me >> about >> >> >> it. >> >> >> >> >> >> Have a great weekend folks. >> >> >> >> >> >> Lewis >> >> >> >> >> >> [0] https://builds.apache.org/job/oodt-trunk/1438/ >> >> >> >> >> >> >> >> >> -- >> >> >> Lewis >> >> > >> >> > >> >> >> > >> > >> > > > >-- >*Lewis*
Review Request 24529: CAS-PGE no longer respects writers and file tags from earlier pgeConfig.xml files
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24529/ --- Review request for oodt. Bugs: OODT-667 https://issues.apache.org/jira/browse/OODT-667 Repository: oodt Description --- This patch restores functionality and fixes CAS-PGE in trunk for users expecting 0.3 CAS-PGE and before style configuration in which you didn't need a MIME extractor repo and could configure CAS-PGE entirely from the pge-config.xml files. This patch effectively makes CAS-PGE usable again in trunk for 0.7 and going forward and is fully forward compatible with Brian's changes. If you specify a MIME extractor repo, you get an AutoDetectCrawler - otherwise you get the familiar StdProductCrawler. Diffs - ./trunk/metadata/src/main/java/org/apache/oodt/cas/metadata/filenaming/PathUtilsNamingConvention.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/OutputDir.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfigMetKeys.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/RegExprOutputFiles.java PRE-CREATION ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeMetadata.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskMetKeys.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskStatus.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/util/GenericPgeObjectFactory.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/util/XmlHelper.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/CsvConfigFileWriter.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/DynamicConfigFileWriter.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/ExternExtractorMetWriter.java PRE-CREATION ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/FilenameExtractorWriter.java PRE-CREATION ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/MetadataKeyReplacerTemplateWriter.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/PcsMetFileWriter.java PRE-CREATION ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/SciPgeConfigFileWriter.java PRE-CREATION ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/TextConfigFileWriter.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/VelocityConfigFileWriter.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/XslTransformWriter.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/metlist/MetadataListPcsMetFileWriter.java PRE-CREATION ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/xslt/XslTransformWriter.java PRE-CREATION ./trunk/pge/src/main/resources/examples/PgeConfigFiles/pge-config.xml 1616402 ./trunk/pge/src/test/java/org/apache/oodt/cas/pge/staging/TestFileStager.java 1616402 ./trunk/pge/src/test/java/org/apache/oodt/cas/pge/writers/MockDynamicConfigFileWriter.java 1616402 Diff: https://reviews.apache.org/r/24529/diff/ Testing --- I've tested this on my DARPA XDATA translation ETL pipeline. Full tests are ongoing, but this works up the point of ingestion. There is something weird going on here with InPlaceIngestion, which I'm going to take a look at, and fix, but it's pretty much done. Enjoy! Thanks, Chris Mattmann
Re: Review Request 24529: CAS-PGE no longer respects writers and file tags from earlier pgeConfig.xml files
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24529/ --- (Updated Aug. 9, 2014, 10:01 p.m.) Review request for oodt. Changes --- - updated diff, now fully working and tested on DARPA XDATA. Bugs: OODT-667 https://issues.apache.org/jira/browse/OODT-667 Repository: oodt Description --- This patch restores functionality and fixes CAS-PGE in trunk for users expecting 0.3 CAS-PGE and before style configuration in which you didn't need a MIME extractor repo and could configure CAS-PGE entirely from the pge-config.xml files. This patch effectively makes CAS-PGE usable again in trunk for 0.7 and going forward and is fully forward compatible with Brian's changes. If you specify a MIME extractor repo, you get an AutoDetectCrawler - otherwise you get the familiar StdProductCrawler. Diffs (updated) - ./trunk/metadata/src/main/java/org/apache/oodt/cas/metadata/filenaming/PathUtilsNamingConvention.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/OutputDir.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfigMetKeys.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/RegExprOutputFiles.java PRE-CREATION ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeMetadata.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskMetKeys.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskStatus.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/util/GenericPgeObjectFactory.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/util/XmlHelper.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/CsvConfigFileWriter.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/DynamicConfigFileWriter.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/ExternExtractorMetWriter.java PRE-CREATION ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/FilenameExtractorWriter.java PRE-CREATION ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/MetadataKeyReplacerTemplateWriter.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/PcsMetFileWriter.java PRE-CREATION ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/SciPgeConfigFileWriter.java PRE-CREATION ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/TextConfigFileWriter.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/VelocityConfigFileWriter.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/XslTransformWriter.java 1616402 ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/metlist/MetadataListPcsMetFileWriter.java PRE-CREATION ./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/xslt/XslTransformWriter.java PRE-CREATION ./trunk/pge/src/main/resources/examples/PgeConfigFiles/pge-config.xml 1616402 ./trunk/pge/src/test/java/org/apache/oodt/cas/pge/metadata/TestPgeMetadata.java 1616402 ./trunk/pge/src/test/java/org/apache/oodt/cas/pge/staging/TestFileStager.java 1616402 ./trunk/pge/src/test/java/org/apache/oodt/cas/pge/writers/MockDynamicConfigFileWriter.java 1616402 Diff: https://reviews.apache.org/r/24529/diff/ Testing --- I've tested this on my DARPA XDATA translation ETL pipeline. Full tests are ongoing, but this works up the point of ingestion. There is something weird going on here with InPlaceIngestion, which I'm going to take a look at, and fix, but it's pretty much done. Enjoy! Thanks, Chris Mattmann
Re: OODT Roadmap Telecon: summary of the event
Lewis you now have access thanks Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Lewis John Mcgibbney Reply-To: Date: Monday, August 11, 2014 5:08 PM To: "dev@oodt.apache.org" Subject: Re: OODT Roadmap Telecon: summary of the event >Hi Folks, >Can someone please add these to the Wiki Page Tom created? >I have to request access to the wiki as I don;t have write access. >Can oneone do that for me here or do I need to go to #asfinfra? >Thanks >Lewis > > >On Thu, Aug 7, 2014 at 1:44 PM, wrote: > >> Folks: >> >> Thanks for dialing into the OODT Roadmap Telecon. >> >> The attendees were: >> >> - Tom Barber, UK business intelligence consultant >> - Roger Carter, JPL intern >> - Cameron Goodale, JPL >> - Michael Joyce, JPL >> - Sean Kelly, consultant >> - Ross Laidlaw, JPL intern >> - Lewis John McGibbney, JPL >> - Tyler Palsulich, JPL intern >> - Rishi Verma, JPL >> >> >> Here are the milestones that I recorded from the telecon, presented in >>no >> particular order: >> >> A. Stability to the codebase. Tests must pass before we can reliably >>move >> forward. This is especially troublesome since there are new features >>we do >> want to implement, yet our confidence level of not breaking existing >> function is low since the tests just don't work. >> >> B. Upgrade OODT's outdated components. A number of parts within OODT >>has >> ossified and become brittle (XML parsing, for example). As the (mainly >> Java) ecosystem has matured, these components have not, and they >>continue >> to be "software hangnails". >> >> C. End-to-end story-driven testing. While we'd love to see more and >>more >> unit testing, complete integration testing is also vital. OODT is a >>large >> and complex system, and ensuring all the parts work in a story-driven >> manner is important. >> >> D. Changing 5 to 10 config files to get OODT just to work is terrible. >> And worse, they're largely XML configuration files. OODT needs to >> get-up-and-go out-of-the-box. >> >> E. Documentation and website movement to Apache CMS-based tech. The >> static nature of the OODT website is a barrier to updates. We'd like >>for >> updates to frequent and timely. >> >> F. Make OODT more of a product, less of an architecture. It's difficult >> for new users to approach OODT since it's presented mainly as a software >> architecture that has some software. If we could change it into a >>product >> (by providing sensible defaults, IoC, etc.) it could have a lower >>barrier >> of entry. >> >> G. Remove PHP. OODT requires a Java servlet container for its core >> function, so why bring in yet another technology for the Balance >> components? This increases the exploitable surface of an OODT server as >> well as making adoption of OODT trickier. >> >> H. XML specification and/or schema so you can know what can be where. >>For >> these XML configuration files, the lack of a schema means it's >>difficult to >> tell what elements and attributes go where, which are required, which >>are >> optional, etc. With a schema (and an XML-aware editor) this becomes >>easier >> to doand gives validation-before-run as a bonus. >> >> I. Where and what are the extension points? This needs to be clearly >> documented and highlighted. >> >> J. Tutorials are static and don't allow for community updates. We need >> them on the wiki instead so everyone can help. Leave a "getting >>started" >> tutorial on the main website, but move everything else to the Apache >> Confluence wiki. >> >> K. Put Jenkins build status on the OODT website! >> >> L. Videos (screencasts). But keep them upbeat. Edit them carefully so >> users aren't watching slow typists backspacing over mistakes repeatedly. >> >> M. Regular release cycle. Four times a year. This gives "liveness" to >> the project, but also gives confidence to new users knowing that a >>certain >> bug or new feature will be addressed in an upcoming release. >> >> O. Report list subscription numbers in board reports, not # of messages. >> This is a more interesting metric since it demonstrates the breadth of >> OODT adoption, which is orthogonal to the amount of discussion which >>can be >> dominated by one or two members. >> >> What milestones did I miss? Do these sound correct? Please reply to >>the >> list and let's discuss further. Once we've hammered out this list, we >> should then priortize them. >> >> Thanks again for your participation, >> --k >> >> -- >> Sean Kelly >> Vice President, Apache OODT >> Member, Apache Software Foundation >> >> > > >-- >*Lewis*
Re: [ANNOUNCE] Tyler Palsulich is now on the PMC too!
Glad that you got it working. I have JDK7 on my Mac so I will give it a test.. Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Tyler Palsulich Reply-To: Date: Thursday, August 14, 2014 11:53 AM To: dev Subject: Re: [ANNOUNCE] Tyler Palsulich is now on the PMC too! >Hi All, > >> I'm having issues even passing the Commons tests on a clean install... > >Working with Ross Laidlaw, we figured out that I was having a Java >versioning issue. I was using jdk1.7.0_65 and tests were not running and >failing completely silently. So, I downgraded to 1.6.0_65 and now all >tests >pass. > >Odd... Anyone have any thoughts on why? Can others build with 1.7? > >Tyler > > >On Sat, Jul 26, 2014 at 7:17 PM, Tyler Palsulich >wrote: > >> Hi, >> >> The odd thing is, I can't find a failing test... Which doesn't make >>sense. >> Every commons test has 0 errors, 0 failures, and 0 skipped. >> >> I'd really like to avoid -DskipTests, for sanity's sake. >> >> Tyler >> On Jul 24, 2014 10:40 PM, "Tom Barber" wrote: >> >>> Hey Tyler, >>> >>> I've seen some failures but not this one before, can you dig out the >>> failure message from the surefire test? You could always just munge it >>>and >>> try -DskipTests >>> >>> On 23/07/14 16:49, Tyler Palsulich wrote: >>> >>>> Thanks all! I'm looking forward to the roadmap call. >>>> >>>> Before then, I'm having issues even passing the Commons tests on a >>>>clean >>>> install... Here is the output of `mvn test -X`: >>>> https://paste.apache.org/75u9. >>>> >>>> ? mvn --version >>>> Apache Maven 3.2.1 (ea8b2b07643dbb1b84b6d16e1f08391b666bc1e9; >>>> 2014-02-14T09:37:52-08:00) >>>> Maven home: /usr/local/Cellar/maven/3.2.1/libexec >>>> Java version: 1.7.0_60, vendor: Oracle Corporation >>>> Java home: >>>> /Library/Java/JavaVirtualMachines/jdk1.7.0_60.jdk/Contents/Home/jre >>>> Default locale: en_US, platform encoding: UTF-8 >>>> OS name: "mac os x", version: "10.9.3", arch: "x86_64", family: "mac" >>>> >>>> TL;DR: I don't see any failed tests. -DfailIfNoTests=false doesn't >>>>help. >>>> My >>>> surefire directory doesn't have any failures. >>>> >>>> Is there any config that needs to be set up before install? Or, am I >>>> missing something in the output above? >>>> >>>> Thank for any help, >>>> Tyler >>>> >>>> >>>> On Tue, Jul 22, 2014 at 8:43 PM, Rajith Siriwardana < >>>> rajithsiriward...@gmail.com> wrote: >>>> >>>> Welcome Tyler..!!! >>>>> >>>>> >>>>> Cheers, >>>>> --Rajith >>>>> >>>>> >>>>> On Wed, Jul 23, 2014 at 1:06 AM, Michael Starch >>>>> wrote: >>>>> >>>>> Great! Welcome, Tyler. >>>>>> >>>>>> As another new PMC I aggree that we should discuss a roadmap. >>>>>> >>>>>> -Michael >>>>>> On Jul 22, 2014 11:58 AM, "Michael Joyce" wrote: >>>>>> >>>>>> Yay! Welcome Tyler!!! >>>>>>> >>>>>>> >>>>>>> -- Joyce >>>>>>> >>>>>>> >>>>>>> On Tue, Jul 22, 2014 at 11:36 AM, Tom Barber >>>>>>>>>>>>> > >>>>>>> wrote: >>>>>>> >>>>>>>Aye I'll +1 that, and move this over to dev@ >>>>>>>> >>>>>>>> We have an ever expanding PMC, so lets do some Project Managing! >>>>>>>>;) >>>>>>>> >>>>>>>> I think we need to come up with some roadmap goals, both regarding >>>>>>>> features, and also the existing code base, build systems and >>>>>>>> delivery. >>>>>>>> >>>>>>>> Personally I think it would be benificial to get a bunch of us on >>>>>>>>the >>>>>>>> phone and discuss it in real time (I'm GMT based which does make >>>>>>>>it a >>>>>>>> little tricker) rather than hash it out over a 6 month chain of >>>>>>>> emails. >>>>>>>> Like I said the other day, I don't really know how other projects >>>>>>>> manage >>>>>>>> this type of thing so those with more experience may want to chime >>>>>>>> in here. >>>>>>>> >>>>>>>> Regards >>>>>>>> >>>>>>>> Tom >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 22/07/14 18:33, Lewis John Mcgibbney wrote: >>>>>>>> >>>>>>>>Hi Tyler, >>>>>>>> Please see Tom's thread from before the weekend. >>>>>>>> I say we have a full on discussion on establishing a roadmap. >>>>>>>> For us to establish and agree on this will be extremely >>>>>>>>beneficial >>>>>>>> for >>>>>>>> the project. >>>>>>>> Lewis >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Jul 22, 2014 at 10:29 AM, Tyler Palsulich < >>>>>>>> tpalsul...@gmail.com >>>>>>>> >>>>>>>>> wrote: >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> Thanks for inviting me to join as a committer and PMC member! >>>>>>>>>I'm >>>>>>>>> happy to be part of the project. Any tips on where to dive in? >>>>>>>>> >>>>>>>>> Tyler >>>>>>>>> >>>>>>>>> >>>>>>>> >>> >>> -- >>> *Tom Barber* | Technical Director >>> >>> meteorite bi >>> *T:* +44 20 8133 3730 >>> *W:* www.meteorite.bi | *Skype:* meteorite.consulting >>> *A:* Surrey Technology Centre, Surrey Research Park, Guildford, GU2 >>>7YG, >>> UK >>> >>
Re: Review Request 18944: [OODT-669] Disabling resource monitor in Assignment monitor feature
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18944/#review50140 --- Please see my attached patch. - Chris Mattmann On March 9, 2014, 4:44 a.m., Rajith Siriwardana wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/18944/ > --- > > (Updated March 9, 2014, 4:44 a.m.) > > > Review request for oodt and Chris Mattmann. > > > Bugs: OODT-669 > https://issues.apache.org/jira/browse/OODT-669 > > > Repository: oodt > > > Description > --- > > Patch contains disabling Resource Monitor feature in Assignment Monitor and > also some re-factored code for assignment monitor > > > Diffs > - > > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/AssignmentMonitor.java > 1575649 > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/disabledmonitor/DisabledResourceMonitor.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/disabledmonitor/DisabledResourceMonitorFactory.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/GangliaAdapter.java > 1575649 > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/GangliaResourceMonitor.java > 1575649 > trunk/resource/src/main/resources/resource.properties 1575649 > > trunk/resource/src/test/org/apache/oodt/cas/resource/monitor/TestAssignmentMonWithNoResourceMon.java > PRE-CREATION > trunk/resource/src/testdata/test.disabledmon.resource.properties > PRE-CREATION > > Diff: https://reviews.apache.org/r/18944/diff/ > > > Testing > --- > > Disabled monitor with Assignment monitor was tested > > > Thanks, > > Rajith Siriwardana > >
Review Request 24539: Fix Resource Manager in Trunk.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24539/ --- Review request for oodt. Bugs: OODT-669 https://issues.apache.org/jira/browse/OODT-669 Repository: oodt Description --- Related to OODT-669, get Resource manager working in trunk again. Diffs - ./trunk/resource/src/main/java/org/apache/oodt/cas/resource/cli/action/AddNodeCliAction.java 1617057 ./trunk/resource/src/main/java/org/apache/oodt/cas/resource/cli/action/SetNodeCapacityCliAction.java 1617057 ./trunk/resource/src/main/java/org/apache/oodt/cas/resource/examples/TestResmgr.java 1617057 ./trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/AssignmentMonitor.java 1617057 ./trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/AssignmentMonitorFactory.java 1617057 ./trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/Monitor.java 1617057 ./trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ResourceMonitor.java 1617057 ./trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ResourceMonitorFactory.java 1617057 ./trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/GangliaResourceMonitor.java 1617057 ./trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/GangliaResourceMonitorFactory.java 1617057 ./trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/loadcalc/LoadCalculator.java 1617057 ./trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/ganglia/loadcalc/WeightedAverageLoadCalc.java 1617057 ./trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/LRUScheduler.java 1617057 ./trunk/resource/src/main/java/org/apache/oodt/cas/resource/structs/Job.java 1617057 ./trunk/resource/src/main/java/org/apache/oodt/cas/resource/structs/ResourceNode.java 1617057 ./trunk/resource/src/main/java/org/apache/oodt/cas/resource/system/XmlRpcResourceManager.java 1617057 ./trunk/resource/src/main/java/org/apache/oodt/cas/resource/system/XmlRpcResourceManagerClient.java 1617057 ./trunk/resource/src/main/java/org/apache/oodt/cas/resource/util/GenericResourceManagerObjectFactory.java 1617057 ./trunk/resource/src/main/java/org/apache/oodt/cas/resource/util/XmlRpcStructFactory.java 1617057 ./trunk/resource/src/main/java/org/apache/oodt/cas/resource/util/XmlStructFactory.java 1617057 ./trunk/resource/src/main/resources/cmd-line-options.xml 1617057 ./trunk/resource/src/test/org/apache/oodt/cas/resource/cli/TestResourceCli.java 1617057 ./trunk/resource/src/test/org/apache/oodt/cas/resource/monitor/TestAssignmentMonitor.java 1617057 ./trunk/resource/src/test/org/apache/oodt/cas/resource/monitor/TestGangliaResourceMonitor.java 1617057 ./trunk/resource/src/test/org/apache/oodt/cas/resource/system/MockXmlRpcResourceManagerClient.java 1617057 ./trunk/resource/src/test/org/apache/oodt/cas/resource/system/TestXmlRpcResourceManager.java 1617057 ./trunk/resource/src/testdata/policy/nodes.xml 1617057 ./trunk/resource/src/testdata/resourcemon/nodes.xml 1617057 Diff: https://reviews.apache.org/r/24539/diff/ Testing --- testing this patch in XDATA. So far, resource manager starts up correctly, but it doesn't seem to respond to the resource manager client in OPSUI. I think this is a back compat issue. I went ahead and updated it to support double-based load values and capacity. I think that's the problem. Thanks, Chris Mattmann
Re: Review Request 24539: Fix Resource Manager in Trunk.
Hi BW, what do you mean, do you mean is there a types package in the resource manager? Cheers, Chris -Original Message- From: BW Reply-To: Date: Monday, August 18, 2014 5:48 PM To: , Chris Mattmann Subject: Re: Review Request 24539: Fix Resource Manager in Trunk. >Is there not a types package? > >-BW > > >On Sun, Aug 10, 2014 at 9:36 PM, Chris Mattmann >wrote: > >> >> --- >> This is an automatically generated e-mail. To reply, visit: >> https://reviews.apache.org/r/24539/ >> --- >> >> Review request for oodt. >> >> >> Bugs: OODT-669 >> https://issues.apache.org/jira/browse/OODT-669 >> >> >> Repository: oodt >> >> >> Description >> --- >> >> Related to OODT-669, get Resource manager working in trunk again. >> >> >> Diffs >> - >> >> >> >>./trunk/resource/src/main/java/org/apache/oodt/cas/resource/cli/action/Ad >>dNodeCliAction.java >> 1617057 >> >> >>./trunk/resource/src/main/java/org/apache/oodt/cas/resource/cli/action/Se >>tNodeCapacityCliAction.java >> 1617057 >> >> >>./trunk/resource/src/main/java/org/apache/oodt/cas/resource/examples/Test >>Resmgr.java >> 1617057 >> >> >>./trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/Assig >>nmentMonitor.java >> 1617057 >> >> >>./trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/Assig >>nmentMonitorFactory.java >> 1617057 >> >> >>./trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/Monit >>or.java >> 1617057 >> >> >>./trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/Resou >>rceMonitor.java >> 1617057 >> >> >>./trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/Resou >>rceMonitorFactory.java >> 1617057 >> >> >>./trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/gangl >>ia/GangliaResourceMonitor.java >> 1617057 >> >> >>./trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/gangl >>ia/GangliaResourceMonitorFactory.java >> 1617057 >> >> >>./trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/gangl >>ia/loadcalc/LoadCalculator.java >> 1617057 >> >> >>./trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/gangl >>ia/loadcalc/WeightedAverageLoadCalc.java >> 1617057 >> >> >>./trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/LRU >>Scheduler.java >> 1617057 >> >> >>./trunk/resource/src/main/java/org/apache/oodt/cas/resource/structs/Job.j >>ava >> 1617057 >> >> >>./trunk/resource/src/main/java/org/apache/oodt/cas/resource/structs/Resou >>rceNode.java >> 1617057 >> >> >>./trunk/resource/src/main/java/org/apache/oodt/cas/resource/system/XmlRpc >>ResourceManager.java >> 1617057 >> >> >>./trunk/resource/src/main/java/org/apache/oodt/cas/resource/system/XmlRpc >>ResourceManagerClient.java >> 1617057 >> >> >>./trunk/resource/src/main/java/org/apache/oodt/cas/resource/util/GenericR >>esourceManagerObjectFactory.java >> 1617057 >> >> >>./trunk/resource/src/main/java/org/apache/oodt/cas/resource/util/XmlRpcSt >>ructFactory.java >> 1617057 >> >> >>./trunk/resource/src/main/java/org/apache/oodt/cas/resource/util/XmlStruc >>tFactory.java >> 1617057 >> ./trunk/resource/src/main/resources/cmd-line-options.xml 1617057 >> >> >>./trunk/resource/src/test/org/apache/oodt/cas/resource/cli/TestResourceCl >>i.java >> 1617057 >> >> >>./trunk/resource/src/test/org/apache/oodt/cas/resource/monitor/TestAssign >>mentMonitor.java >> 1617057 >> >> >>./trunk/resource/src/test/org/apache/oodt/cas/resource/monitor/TestGangli >>aResourceMonitor.java >> 1617057 >> >> >>./trunk/resource/src/test/org/apache/oodt/cas/resource/system/MockXmlRpcR >>esourceManagerClient.java >> 1617057 >> >> >>./trunk/resource/src/test/org/apache/oodt/cas/resource/system/TestXmlRpcR >>esourceManager.java >> 1617057 >> ./trunk/resource/src/testdata/policy/nodes.xml 1617057 >> ./trunk/resource/src/testdata/resourcemon/nodes.xml 1617057 >> >> Diff: https://reviews.apache.org/r/24539/diff/ >> >> >> Testing >> --- >> >> testing this patch in XDATA. So far, resource manager starts up >>correctly, >> but it doesn't seem to respond to the resource manager client in OPSUI. >>I >> think this is a back compat issue. I went ahead and updated it to >>support >> double-based load values and capacity. I think that's the problem. >> >> >> Thanks, >> >> Chris Mattmann >> >>
Re: Review Request 22791: Streaming OODT Changes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22791/#review52324 --- http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh <https://reviews.apache.org/r/22791/#comment91085> Mike, instead of directly including cluster-tools in oodt, can you simply make: # A Maven AntRun script (build.xml) or something that downloads the cluster-tools as part of the Resource Manager build? We shouldn't have to maintain these scripts in OODT. http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml <https://reviews.apache.org/r/22791/#comment91086> Custom Maven repos are difficult since the Central repository is phasing them out: http://blog.sonatype.com/2010/03/why-external-repos-are-being-phased-out-of-central/ If we absolutely need to ref this repo, can you make it a Maven profile, not enabled by default, so that users won't have the issue for this when downloading and building OODT? http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java <https://reviews.apache.org/r/22791/#comment91087> how about instead of NA, for the filename, we call it a "stream-UUID" where we generate a unique stream UUID as the FILENAME field. http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java <https://reviews.apache.org/r/22791/#comment91088> how about a similarly derived STREAM loc here, perhaps the same as the FILENAME suggestion using UUID? http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java <https://reviews.apache.org/r/22791/#comment91089> rather than silently do something, maybe take this out until there is something to do? http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java <https://reviews.apache.org/r/22791/#comment91090> not sure what this change is? http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java <https://reviews.apache.org/r/22791/#comment91091> All of these files need ALv2 license headers. http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java <https://reviews.apache.org/r/22791/#comment91092> We shoudln't be checking in generated files - can this simply be generated during the Resmgr build? http://svn.apache.org/repos/asf/oodt/trunk/streamer/pom.xml <https://reviews.apache.org/r/22791/#comment91093> if we don't need this, remove it. http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStream.java.bak <https://reviews.apache.org/r/22791/#comment91094> ALv2 headers here and everywhere http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/streamer.properties <https://reviews.apache.org/r/22791/#comment91095> remove JPL specific URLs. - Chris Mattmann On Sept. 4, 2014, 5:23 p.m., Michael Starch wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/22791/ > ------- > > (Updated Sept. 4, 2014, 5:23 p.m.) > > > Review request for oodt, Lewis McGibbney and Chris Mattmann. > > > Repository: oodt > > > Description > --- > > This patch contains all the changes needed to add in "streaming oodt" into > the oodt svn repository. > > There are four main portions: >-Mesos Framework for Resource Manager (Prototype working) >-Spark Runner for Workflow Manager (Prototype working) >-Filemanager "streaming" type (In development) >-Deployment and cluster management scripts (In development) > > Where can this stuff be put so that it is available to use, even while it is > in development? > > > Diffs > - > > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash > PRE-CREATION >
Re: Review Request 22791: Streaming OODT Changes
> On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote: > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh, > > line 1 > > <https://reviews.apache.org/r/22791/diff/2/?file=659631#file659631line1> > > > > Mike, instead of directly including cluster-tools in oodt, can you > > simply make: > > > > # A Maven AntRun script (build.xml) or something that downloads the > > cluster-tools as part of the Resource Manager build? We shouldn't have to > > maintain these scripts in OODT. > > Michael Starch wrote: > Download from where? Do we have a place to upload unmaintained OPS > scripts? THanks Mike. Ask @paulramirez about how to use Maven AntRun or assembly.xml - you could theoretically reference Mesos if it has a distribution for its scripts in the Central repo - you could ref the dist dependency as a dependency and then unpack them dynamically into a directory. If it doesn't have a Maven dist assembly for the Mesos scripts, then my recommendation would simply be to: 1. Create a build.xml that downloads (e.g., from Mesos trunk or a tag in Apache git) those scripts into whatever OODT build directory you want (inside of Resource Manager probably makes the most sense) 2. Use the Maven Ant-Run plugin to call that build.xml in resource/pom.xml 3. Consider using a Maven profile (e.g., mvn -Pstreaming) to insulate your streaming profile cluster-tools download - Chris --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22791/#review52324 --- On Sept. 4, 2014, 5:23 p.m., Michael Starch wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/22791/ > ------- > > (Updated Sept. 4, 2014, 5:23 p.m.) > > > Review request for oodt, Lewis McGibbney and Chris Mattmann. > > > Repository: oodt > > > Description > --- > > This patch contains all the changes needed to add in "streaming oodt" into > the oodt svn repository. > > There are four main portions: >-Mesos Framework for Resource Manager (Prototype working) >-Spark Runner for Workflow Manager (Prototype working) >-Filemanager "streaming" type (In development) >-Deployment and cluster management scripts (In development) > > Where can this stuff be put so that it is available to use, even while it is > in development? > > > Diffs > - > > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl > PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts > PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt > PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java > 1617800 > > http://sv
Re: filemgr ingestion
Hi Etienne, Thanks for your query. The default protocol used by the RemoteDataTransfer is XML-RPC. It's a call-back mechanism that I baked up myself with a configurable chunk size parameter. My suggestion is rather than use that DataTransfer, I would leverage: (1) a distributed filesystem like HadoopFS, Tachyon (from BDAS), GlusterFS, or even NFS that logically mounts a set of local shared nothing disks as a virtual global mount (2) use the LocalDataTransfer in the File Manager, with #1 in place The main reason for this is that it allows the people who work on distributed and reliable filesystems that solve those problems in a great way, it allows us in the OODT community to take advantage of that work without having to write all the complex functionality ourselves. That way we just have light-weight insulated pieces/components in OODT that rely on the workhorses that do things right. HTH! Cheers, Chris Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Etienne Koen Date: Wednesday, September 10, 2014 1:57 AM To: Shakeh Khudikyan Cc: Chris Mattmann Subject: filemgr ingestion >Hi Shakeh, >What is the default protocol that filemgr uses for RemoteDataTransfer >ingestion? > >Also, is the a way to specify the protocol? > >Cheers >Etienne >
[VOTE] Apache OODT 0.7 release candidate #1
Hi Folks, I have posted a 1st release candidate for the Apache OODT 0.7 release. The source code is at: https://dist.apache.org/repos/dist/dev/oodt/ For more detailed information, see the included CHANGES.txt file for details on release contents and latest changes. The release was made using the OODT release process, documented on the Wiki here: https://cwiki.apache.org/confluence/display/OODT/Release+Process The release was made from the OODT 0.7 tag (r1624941) at: http://svn.apache.org/repos/asf/oodt/tags/0.7/ A staged Maven repository is available at: https://repository.apache.org/content/repositories/orgapacheoodt-1000/ Please vote on releasing these packages as Apache OODT . The vote is open for the next 72 hours. Only votes from OODT PMC are binding, but folks are welcome to check the release candidate and voice their approval or disapproval. The vote passes if at least three binding +1 votes are cast. [ ] +1 Release the packages as Apache OODT [ ] -1 Do not release the packages because... Thanks! Chris P.S. Here is my +1.
Re: [VOTE] Apache OODT 0.7 release candidate #1
Hey Sean, Weird on the test - can you try with Java7? I ran and built with Java7 and am not sure the tests pass with Java8? Cheers, Chris -Original Message- From: Reply-To: Date: Monday, September 15, 2014 8:00 AM To: Cc: Subject: Re: [VOTE] Apache OODT 0.7 release candidate #1 >Apache OODT 0.7 RC 1: > >Signature: confirmed >SHA1: matches >MD5: matches >Python tests: pass >Java tests: intermittent failures >Also, let's change oodt/agility/setup.py's description to "Apache OODT". > >The failure seems to be an ordering issue: > >-- >- >Test set: org.apache.oodt.cas.resource.queuerepo.TestXmlQueueRepository >-- >- >Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.249 sec ><<< FAILURE! >testMapping(org.apache.oodt.cas.resource.queuerepo.TestXmlQueueRepository) > Time elapsed: 0.224 sec <<< FAILURE! >junit.framework.AssertionFailedError: expected:<[test-queue-1, quick]> >but was:<[quick, test-queue-1]> >at junit.framework.Assert.fail(Assert.java:47) >at junit.framework.Assert.failNotEquals(Assert.java:280) >at junit.framework.Assert.assertEquals(Assert.java:64) >at junit.framework.Assert.assertEquals(Assert.java:71) >at >org.apache.oodt.cas.resource.queuerepo.TestXmlQueueRepository.testMapping( >TestXmlQueueRepository.java:64) > >Probably a quick fix. This is with Java 1.8.0_05. > > > > > >On 2014-09-15, at 8:50 AM, Chris Mattmann wrote: > >> Hi Folks, >> >> I have posted a 1st release candidate for the Apache OODT 0.7 release. >>The >> source code is at: >> >> https://dist.apache.org/repos/dist/dev/oodt/ >> >> For more detailed information, see the included CHANGES.txt file for >> details on >> release contents and latest changes. The release was made using the OODT >> release process, documented on the Wiki here: >> >> https://cwiki.apache.org/confluence/display/OODT/Release+Process >> >> The release was made from the OODT 0.7 tag (r1624941) at: >> >> http://svn.apache.org/repos/asf/oodt/tags/0.7/ >> >> >> A staged Maven repository is available at: >> >> https://repository.apache.org/content/repositories/orgapacheoodt-1000/ >> >> >> Please vote on releasing these packages as Apache OODT . The >>vote >> is >> open for the next 72 hours. >> >> Only votes from OODT PMC are binding, but folks are welcome to check the >> release candidate and voice their approval or disapproval. The vote >>passes >> if at least three binding +1 votes are cast. >> >> [ ] +1 Release the packages as Apache OODT >> >> [ ] -1 Do not release the packages because... >> >> Thanks! >> >> Chris >> >> P.S. Here is my +1. >> >> >> >> >
Re: [VOTE] Apache OODT 0.7 release candidate #1
Awesome, I've captured your test error and filed a JIRA issue for 0.8, thanks Sean! I've also captured your issue about updating the description of Agility: https://issues.apache.org/jira/browse/OODT-747 https://issues.apache.org/jira/browse/OODT-748 Cheers, Chris -Original Message- From: Reply-To: Date: Monday, September 15, 2014 8:09 AM To: Cc: Subject: Re: [VOTE] Apache OODT 0.7 release candidate #1 >Java 7 is A-O-K. No problems there. > >If we release with that caveat, then I'm +1. > >--k > > >On 2014-09-15, at 1:01 PM, Chris Mattmann wrote: > >> Hey Sean, >> >> Weird on the test - can you try with Java7? I ran and built >> with Java7 and am not sure the tests pass with Java8? >> >> Cheers, >> Chris >> >> >> -Original Message- >> From: >> Reply-To: >> Date: Monday, September 15, 2014 8:00 AM >> To: >> Cc: >> Subject: Re: [VOTE] Apache OODT 0.7 release candidate #1 >> >>> Apache OODT 0.7 RC 1: >>> >>> Signature: confirmed >>> SHA1: matches >>> MD5: matches >>> Python tests: pass >>> Java tests: intermittent failures >>> Also, let's change oodt/agility/setup.py's description to "Apache >>>OODT". >>> >>> The failure seems to be an ordering issue: >>> >>> >>> >>>-- >>> - >>> Test set: org.apache.oodt.cas.resource.queuerepo.TestXmlQueueRepository >>> >>> >>>-- >>> - >>> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.249 >>>sec >>> <<< FAILURE! >>> >>>testMapping(org.apache.oodt.cas.resource.queuerepo.TestXmlQueueRepositor >>>y) >>> Time elapsed: 0.224 sec <<< FAILURE! >>> junit.framework.AssertionFailedError: expected:<[test-queue-1, quick]> >>> but was:<[quick, test-queue-1]> >>> at junit.framework.Assert.fail(Assert.java:47) >>> at junit.framework.Assert.failNotEquals(Assert.java:280) >>> at junit.framework.Assert.assertEquals(Assert.java:64) >>> at junit.framework.Assert.assertEquals(Assert.java:71) >>> at >>> >>>org.apache.oodt.cas.resource.queuerepo.TestXmlQueueRepository.testMappin >>>g( >>> TestXmlQueueRepository.java:64) >>> >>> Probably a quick fix. This is with Java 1.8.0_05. >>> >>> >>> >>> >>> >>> On 2014-09-15, at 8:50 AM, Chris Mattmann wrote: >>> >>>> Hi Folks, >>>> >>>> I have posted a 1st release candidate for the Apache OODT 0.7 release. >>>> The >>>> source code is at: >>>> >>>> https://dist.apache.org/repos/dist/dev/oodt/ >>>> >>>> For more detailed information, see the included CHANGES.txt file for >>>> details on >>>> release contents and latest changes. The release was made using the >>>>OODT >>>> release process, documented on the Wiki here: >>>> >>>> https://cwiki.apache.org/confluence/display/OODT/Release+Process >>>> >>>> The release was made from the OODT 0.7 tag (r1624941) at: >>>> >>>> http://svn.apache.org/repos/asf/oodt/tags/0.7/ >>>> >>>> >>>> A staged Maven repository is available at: >>>> >>>> https://repository.apache.org/content/repositories/orgapacheoodt-1000/ >>>> >>>> >>>> Please vote on releasing these packages as Apache OODT . The >>>> vote >>>> is >>>> open for the next 72 hours. >>>> >>>> Only votes from OODT PMC are binding, but folks are welcome to check >>>>the >>>> release candidate and voice their approval or disapproval. The vote >>>> passes >>>> if at least three binding +1 votes are cast. >>>> >>>> [ ] +1 Release the packages as Apache OODT >>>> >>>> [ ] -1 Do not release the packages because... >>>> >>>> Thanks! >>>> >>>> Chris >>>> >>>> P.S. Here is my +1. >>>> >>>> >>>> >>>> >>> >> >> >
Re: [VOTE] Apache OODT 0.7 release candidate #1
Okey dok - this VOTE has passed with the following tallies: +1 Chris Mattmann Michael Joyce Tom Barber Sean Kelly Tyler Palsulich Paul Ramirez Lewis John McGibbney I will go ahead and push the release out to the mirrors and push the Central sync and PyPI push. Gav - can you help us with the PHP 0.7 OODT API update? Website update will come last. Thanks all! Thank you! Cheers, Chris -Original Message- From: Chris Mattmann Date: Monday, September 15, 2014 6:50 AM To: Cc: Subject: [VOTE] Apache OODT 0.7 release candidate #1 >Hi Folks, > >I have posted a 1st release candidate for the Apache OODT 0.7 release. The >source code is at: > >https://dist.apache.org/repos/dist/dev/oodt/ > >For more detailed information, see the included CHANGES.txt file for >details on >release contents and latest changes. The release was made using the OODT >release process, documented on the Wiki here: > >https://cwiki.apache.org/confluence/display/OODT/Release+Process > >The release was made from the OODT 0.7 tag (r1624941) at: > >http://svn.apache.org/repos/asf/oodt/tags/0.7/ > > >A staged Maven repository is available at: > >https://repository.apache.org/content/repositories/orgapacheoodt-1000/ > > >Please vote on releasing these packages as Apache OODT . The >vote >is >open for the next 72 hours. > >Only votes from OODT PMC are binding, but folks are welcome to check the >release candidate and voice their approval or disapproval. The vote passes >if at least three binding +1 votes are cast. > >[ ] +1 Release the packages as Apache OODT > >[ ] -1 Do not release the packages because... > >Thanks! > >Chris > >P.S. Here is my +1. > >
[ANNOUNCE] Apache OODT 0.7 Released
The Apache OODT community is proud to announce the release of Apache OODT 0.7. Apache OODT is a software framework, and an architectural style for the rapid construction of scientific data systems. It provides components for data capture, curation, metadata extraction, workflow management, resource management and data processing. The 0.7 release addresses 79 issues from our JIRA management system: http://issues.apache.org/jira/browse/OODT This release is a core stability release with numerous bug fixes, and a solid baseline for upgrading. It fixes 79 JIRA issues and provides a backwards compatible CAS-PGE upgrade allowing the wrapper to be used with existing config files. 0.7 fixes some bugs in Ganglia support and allows the Resource Manager to work without Ganglia being present. This version also makes significant fixes to File Manager tests, provides a Vagrant version of OODT RADIX and upgrades RADIX to include CAS-PGE support. This version also includes improved support for a new JAX-RS CAS product service, and this version eliminates several bugs in the workflow manager in its support for dynamic workflow instances. The release is available from: http://www.apache.org/dyn/closer.cgi/oodt/ In the first 48 hours, the release will be propagating to the mirrors, so allow time for it to show up in a mirror near you. The release is also made available via Maven Central: http://repo1.maven.org/maven2/org/apache/oodt/ And from PyPi: https://pypi.python.org/pypi/oodt And from PEAR: http://pear.apache.org/oodt/ The Apache OODT website will be updated shortly to reflect the releases. As always if you find anything that you'd like to report (including praise!) please do so on our u...@oodt.apache.org (use of OODT) or dev@oodt.apache.org (architecture/design of OODT) mailing lists. Also please visit our website: http://oodt.apache.org And our wiki: https://cwiki.apache.org/confluence/display/OODT/Home Thanks! Chris Mattmann (on behalf of the Apache OODT PMC)
Re: HttpClient NoClassDefFoundError For the url-downloader Script of the Apache OODT Crawler
s/Melanie/Angela :) Chris Mattmann chris.mattm...@gmail.com -Original Message- From: "Ramirez, Paul M (398J)" Reply-To: Date: Saturday, September 20, 2014 6:44 AM To: "" Subject: Re: HttpClient NoClassDefFoundError For the url-downloader Script of the Apache OODT Crawler >Melanie, > >Your answer for right now then is to run that script from its directory. >It would great if you could open a Jira issue for improvement [1]. If you >happen to update the script to make it more robust that would be a great >contribution back to the community. To do that you would just create a >patch file and attach it to the issue. > >Thanks, >Paul > >[1] https://issues.apache.org/jira/browse/OODT > > > >Sent from my iPhone > >> On Sep 19, 2014, at 8:34 PM, "Lewis John Mcgibbney" >> wrote: >> >> You are correct Paul R. >> The http client dependency is present in crawler Pom.xml >> >> commons-httpclient >> commons-httpclient >> 3.0 >> >> The crawler launcher script had the following hard pathS which we should >> improve >> >> Java.ext.dirs=../lib >> >> If this I not invoked from crawler module path then dependencies a are >>not >> on classpath >> >> On Friday, September 19, 2014, Ramirez, Paul M (398J) < >> paul.m.rami...@jpl.nasa.gov> wrote: >> >>> Could also just be the way the script references the jars. Maybe they >>> aren't on the classpath. Not near a computer at the moment or I would >>>look >>> at the script. >>> >>> --Paul >>> >>> Sent from my iPhone >>> >>>>> On Sep 19, 2014, at 7:25 PM, "Lewis John Mcgibbney" < >>>> lewis.mcgibb...@gmail.com > wrote: >>>> >>>> Hi Angela. >>>> If you are using OODT from source, you can simply add the http client >>>> commons library to the Pom.xml and rebuild the module. This does >>>>however >>>> mean working with the XML and maven to build. Which version of oodt do >>> you >>>> have? >>>> >>>>> On Friday, September 19, 2014, Mengying Wang >> > wrote: >>>>> >>>>> Dear Everyone, >>>>> >>>>> I am trying to use the url-downloader script of the Apache OODT >>>>>Crawler. >>>>> However, it raises the following "no class definition" error, I am >>>>>not >>> sure >>>>> what is going on. Could you please give me some hints? Many thanks. >>>>> >>>>> $ ./url-downloader >>> >>>http://pds-imaging.jpl.nasa.gov/data/msl/MSLHAZ_0XXX/CATALOG/CATINFO.TXT >>> . >>>>> Exception in thread "main" java.lang.NoClassDefFoundError: >>>>> org/apache/oodt/cas/pushpull/protocol/http/HttpClient >>>>> Caused by: java.lang.ClassNotFoundException: >>>>> org.apache.oodt.cas.pushpull.protocol.http.HttpClient >>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >>>>> >>>>> Best, >>>>> Angela Wang >>>> >>>> >>>> -- >>>> *Lewis* >> >> >> -- >> *Lewis*
Re: No data files are downloaded using Apache OODT Pushpull
Hi Angela, I didn't get your screenshot, it was likely stripped. Can you use something like min.us and then paste the link? Cheers, Chris -Original Message- From: Mengying Wang Reply-To: Date: Friday, September 19, 2014 12:22 PM To: Subject: No data files are downloaded using Apache OODT Pushpull >Dear Everyone, > >I am trying to use the Apache OODT Pushpull to download some remote data >files to my local machine. According to the Apache OODT Pushpull user >guide >(https://cwiki.apache.org/confluence/display/OODT/OODT+Push-Pull+User+Guid >e), >I have completed all steps. However, when I run the command ./pushpull, no >data files are downloaded to my staging area after 15 minutes. I have used >FileZilla to login to the remove server and confirmed that there are some >data files. So I am not sure what is wrong with my configurations? Could >you please help me out? Many thanks! > >Following are my running screenshot and configuration details: > >$ ./pushpull > >TRANSFER: >org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory > >^C > >*[CAS_PP_HOME]/etc/push_pull_framework.properties* >line >21 #external configuration files >22 >org.apache.oodt.cas.pushpull.config.external.properties.files=[CAS_PP_HOME >]/etc/default.properties > >35 # ingester filemgr url >36 org.apache.oodt.cas.filemgr.url=http://localhost:9000 > >61 #protocolfactory specification for protocol types >62 >org.apache.oodt.cas.pushpull.config.protocolfactory.info.files=[CAS_PP_HOM >E]/policy/ProtocolFactoryInfo.xml > >69 #parser to retrievalmethod map >70 >org.apache.oodt.cas.pushpull.config.parser.info.files=[CAS_PP_HOME]/policy >/ParserToRetrievalMethodMap.xml >71 >72 #unique metadata element info >73 >org.apache.oodt.cas.pushpull.config.type.detection.file=[CAS_PP_HOME]/poli >cy/mimetypes.xml >74 >75 #directory below which all data file will be downloaded to >76 >org.apache.oodt.cas.pushpull.data.files.base.staging.area=[CAS_PP_HOME]/st >aging > > >*[CAS_PP_HOME]/policy/mimetypes.xml* > > ><_comment> > > > > >*[CAS_PP_HOME]/etc/examples/ExternalSourcesFiles/ExternalSources.xml* > > > >anonymous >u...@host.com > > > > >*[CAS_PP_HOME]/etc/examples/RemoteSpecsFiles/RemoteSpecs.xml* > > >file="[CAS_PP_HOME]/etc/examples/ExternalSources/ExternalSources.xml"/> > > > > >epsilon="20s" runOnReboot="yes"/> >dir="/Users/AngelaWang/Documents/programs/oodt/cas-pushpull/etc/examples/D >irStructXmlParserFiles"> >parser="org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXml >Parser"/> > > > > > > >*[CAS_PP_HOME]/etc/examples/DirStructXmlParserFiles/RTG_SST.xml* > > > > > > > >Thanks again! > >Best, >Angela Wang
Re: [Proposal] Taverna workflow
oooh, grabs popcorn :) Will check it out, thanks Bruce. -Original Message- From: Bruce Barkstrom Reply-To: Date: Tuesday, September 23, 2014 9:56 AM To: Subject: Re: FW: [Proposal] Taverna workflow >You should probably check out Condor >(base site at http://research.cs.wisc.edu/htcondor/} >and (to resolve a suit against the U. Wisc. changed >to HTCondor). This is now >http://research.cs.wisc.edu/htcondor/ > >This system has been through a LOT of experience. >The Globabl Grid Forum (back in the early 2000's >before IBM and Fujitsu "stole" the GGF recommendations >and started cloud computing). > >Bruce B. > >On Tue, Sep 23, 2014 at 12:27 PM, Mattmann, Chris A (3980) < >chris.a.mattm...@jpl.nasa.gov> wrote: > >> Taverna is a hugely famous workflow system! Wow! >> >> ++ >> Chris Mattmann, Ph.D. >> Chief Architect >> Instrument Software and Science Data Systems Section (398) >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >> Office: 168-519, Mailstop: 168-527 >> Email: chris.a.mattm...@nasa.gov >> WWW: http://sunset.usc.edu/~mattmann/ >> ++ >> Adjunct Associate Professor, Computer Science Department >> University of Southern California, Los Angeles, CA 90089 USA >> ++ >> >> >> >> >> >> >> -Original Message- >> From: Stian Soiland-Reyes >> Reply-To: "gene...@incubator.apache.org" >> Date: Tuesday, September 23, 2014 5:43 AM >> To: "gene...@incubator.apache.org" >> Cc: List for general discussion and hacking of the Taverna project >> >> Subject: [Proposal] Taverna workflow >> >> >I hereby present the Apache Incubator proposal for the project Taverna. >> > >> > >> >Also available in rich text in the Taverna wiki (with more >>hyperlinks!): >> > >> > >> >>http://dev.mygrid.org.uk/wiki/display/developer/Taverna+incubator+proposa >>l >> > >> >(Could someone grant me access to edit the Incubator wiki pages? My >> >wiki username is soilandreyes) >> > >> > >> > >> > >> ># Abstract >> > >> >Taverna is an open source and domain-independent suite of tools used >> >to design and execute data-driven workflows. >> > >> > >> ># Proposal >> > >> >The Taverna suite includes: >> > >> >* Taverna Workbench, a Java-based desktop application for graphically >> >composing, editing and executing workflows of distributed web services >> >and local tools >> >* Taverna Commandline Tool which allows repeated execution of >> >parameterized workflow definitions >> >* Taverna Server provides a REST and SOAP API for executing workflows >> >* Taverna Player is a Ruby-based web interface towards the Server, >> >providing a high-level view of workflow executions and their results, >> >and allows further integrations with Ruby on Rails applications. >> > >> >Taverna can browse and combine different service types, allowing >> >workflows to integrate steps of arbitrary REST and SOAP web services >> >with command line tools (local and SSH), scripts (Beanshell, R, >> >Jython) and finally visualize the results. >> > >> >The goal of the Taverna suite is to help researchers to access >> >distributed datasets and processing capabilities by the construction >> >of pipelines, and also to simplify the execution of these pipelines >> >in various environments. >> > >> >The Taverna suite of products is already successful and in wide-use >> >across different domains. The software is currently licensed as LGPL >> >2.1, with copyright owned by University of Manchester. External >> >contributors have all signed Apache-like CLAs. >> > >> > >> ># Background >> > >> >Taverna workflows coordinate inputs and outputs between computational >> >processes and Web Services. The workflow is designed in a graphical >> >interface which shows the workflow as a series of boxes and arrows; >> >representing processes and their data connections. The different >> >processes in a workflow can be command line tools, REST and WSDL Web >> >Services; which are used for combining steps such as data acquisition, >> >filtering, cleaning, integrating, analysis and visual
Re: Some Further Thoughts on Configuration Management of Complex Workflows
These are definitely important considerations Bruce and I am going to check out the papers referenced herein. Thanks, Chris -Original Message- From: Bruce Barkstrom Reply-To: Date: Tuesday, September 23, 2014 6:16 AM To: , Subject: Some Further Thoughts on Configuration Management of Complex Workflows >While I won't claim to have done a thorough examination of the proposal >to use the IBM tool for developing workflows, I am concerned about several >items relating to configuration management. There are several articles >in the new Comm. ACM that bear on security and configuration management. >(CACM, Vol. 57, No. 9, 2014). I'd highly recommend getting a copy and >taking a look at the articles in the middle of the issue. > >1. Kern, C., 2014: Securing the Tangled Web, CACM, 57, 38-67 presents >a view of security issues due to script injection vulnerabilities that >makes >JSON and other technologies that use Java Script less secure than one >would like. Kern is an information security engineer for Google. He >discusses >not only the nature of the XSS vulnerabilities, but also work Google has >undertaken >to reduce their risk. These include building in special character >exception handling, >designing and testing automated templates for interface designers to use, >and >project management enforcement of strict disciplines that forbid use of >vulnerable software. Unfortunately, the cures add to the leaning curve >for >using these tools - and increase the maintenance cost of software because >they need to be applied "forever". > >2. Workflows (or, in normal project management nomenclature, Work >Breakdown >Structures) are graphs whose complexity increases markedly as more >activities >and objects get included. If one is aiming for high integrity or fully >replicable and >transparent software systems, one must maintain the ability to retain >configuration. >The old NCAR FORTRAN manuals (ca. 1980) had a cover that embedded the >notion "It ran yesterday. It's been running for years. I only changed >one >card." >This means that software that is updated (by revisions due to concerns >over >security or to make other improvements) could require verification that >the >updates haven't changed numerical values. Based on my personal experience >with Ubuntu Linux (or Windows - whatever), updates occur on at least a >weekly >basis, with the organizations responsible for their software deciding when >to >send out updates. This rate of update makes the Web a pretty volatile >environment. >In most organizations that have system administrators, they bear the >burden >this >turmoil creates. End users may not realize the impact, but it costs time >and attention >to avoid being overwhelmed. > >3. In many of the software packages we use, the organization providing >the >software >manages package updates with a centralized package manager. In Linux, >Debian (and >the derivative Ubuntu family of software) uses one centralized manager to >produce the >.deb packages that contain appropriate provenance metadata for maintaining >configuration. >Red Hat and SuSE Linux use an alternative format for the RPM package with >its metadata >format. These package managers do not operate in the same way. For >example, if >you want to ingest RPM packages into Ubuntu, you have to install a package >called >alien and use that to convert the RPM to .deb formats. The same >pleasantries affect >Java, databases, and Web standards. Because some of these organizations >are real >commercial enterprises making their money from customers outside of the >federal >contracting venue, it seems unlikely that expecting funding agencies will >develop >one common standard for configuration management. While funding agencies >might >think a single standard for configuration would solve their problems, that >would require >an unprecedented degree of cooperation between agencies, data producers, >and data >users. The time scale for reaching agreements on this kind of "social >engineering" >is almost certainly at least a decade, during which the technological >basis >in hardware >and software will have evolved out from under the agencies. > >I suspect that the security issues relating to JSON and such are the >immediate >concern. On a slightly longer time frame, it's important to remember the >complexity >of workflow scaling makes a single tool unlikely. A solution for data >production with >short chains of objects that are relatively isolated (single investigator >conducting a >few investigations per year) is vastly different from production flows >such >as weather >forecasting or some kinds of climate data production (large teams of >software developers and >scientists (100's of people) running 1000's of jobs per day). >Configuration management >for the latter kinds of project requires building group cultures that >recognize the importance >of managing the configuration - and does take up a lot of time - even for >the scientists >i
Re: Bash Security Issue
THanks for the FYI Bruce Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Lewis John Mcgibbney Reply-To: Date: Thursday, September 25, 2014 8:23 AM To: "dev@oodt.apache.org" Subject: Re: Bash Security Issue >Great snakes > >On Thu, Sep 25, 2014 at 11:05 AM, Bruce Barkstrom >wrote: > >> Bash security issue: >> >> >>http://news.msn.com/science-technology/new-bash-software-bug-may-pose-big >>ger-threat-than-heartbleed >> >> Bruce B. >> > > > >-- >*Lewis*
Re: Bash Security Issue
Thank you Bruce RE: the portrait :) Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Bruce Barkstrom Reply-To: Date: Friday, September 26, 2014 at 11:50 AM To: Subject: Re: Bash Security Issue >Given the dispersed nature of the developer/user communities, >there's always a concern that CERT's bulletins may not reach >folks. I noticed that my Ubuntu installation had a new batch of >updates this morning, including one called "bash". Hope this >helped. > >In short, you're welcome. [Also, I hope Chris is pleased with his >new portrait as a member of the Board of Directors. I don't remember >that he looked like that last time I saw him, but keeping order >in a group can have serious ageing effects.] > >Bruce B. > >On Fri, Sep 26, 2014 at 1:43 PM, Chris Mattmann >wrote: > >> THanks for the FYI Bruce >> >> >> Chris Mattmann >> chris.mattm...@gmail.com >> >> >> >> >> -Original Message- >> From: Lewis John Mcgibbney >> Reply-To: >> Date: Thursday, September 25, 2014 8:23 AM >> To: "dev@oodt.apache.org" >> Subject: Re: Bash Security Issue >> >> >Great snakes >> > >> >On Thu, Sep 25, 2014 at 11:05 AM, Bruce Barkstrom >> >> >wrote: >> > >> >> Bash security issue: >> >> >> >> >> >> >> >>http://news.msn.com/science-technology/new-bash-software-bug-may-pose-big >> >>ger-threat-than-heartbleed >> >> >> >> Bruce B. >> >> >> > >> > >> > >> >-- >> >*Lewis* >> >> >>
Re: Review Request 26064: OODT-754 contribute ProdTypePatternMetExtractor
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26064/#review54767 --- Ship it! Ship It! - Chris Mattmann On Sept. 26, 2014, 1:04 a.m., Ricky Nguyen wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/26064/ > --- > > (Updated Sept. 26, 2014, 1:04 a.m.) > > > Review request for oodt, Lewis McGibbney, Chris Mattmann, Paul Ramirez, and > Shakeh Khudikyan. > > > Bugs: OODT-754 > https://issues.apache.org/jira/browse/OODT-754 > > > Repository: oodt > > > Description > --- > > This is a cleaned up version of the ProdTypePatternMetExtractor used at one > point at CHLA. Wrote some unit tests (and fixed a bug!). Made sure to write > some good javadocs and site docs too (they are basically the same as the > wiki). > > > Diffs > - > > trunk/CHANGES.txt 1627442 > trunk/metadata/pom.xml 1627442 > > trunk/metadata/src/main/java/org/apache/oodt/cas/metadata/extractors/AbstractSAXConfigReader.java > PRE-CREATION > > trunk/metadata/src/main/java/org/apache/oodt/cas/metadata/extractors/ProdTypePatternMetExtractor.java > PRE-CREATION > trunk/metadata/src/site/xdoc/user/basic.xml 1627442 > > trunk/metadata/src/test/org/apache/oodt/cas/metadata/extractors/TestAbstractSAXConfigReader.java > PRE-CREATION > > trunk/metadata/src/test/org/apache/oodt/cas/metadata/extractors/TestProdTypePatternMetExtractor.java > PRE-CREATION > trunk/metadata/src/testdata/product-type-patterns-2.xml PRE-CREATION > trunk/metadata/src/testdata/product-type-patterns.xml PRE-CREATION > > Diff: https://reviews.apache.org/r/26064/diff/ > > > Testing > --- > > 2 test classes, one for the SAX reader and one for the met extractor. > > (1) testExtractMetadata: Sanity check, make sure it works as advertised. > (2) testNewConfigFile: If a second config file is parsed, ignore the first > config file. > (3) testElementDeclarationOrder: The order of element declaration is > independent of the order of usage in the product-type template. > > > Thanks, > > Ricky Nguyen > >
Re: Review Request 26064: OODT-754 contribute ProdTypePatternMetExtractor
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26064/#review54768 --- Ship it! Please, ship this! Welcome back RickDN :) - Chris Mattmann On Sept. 26, 2014, 1:04 a.m., Ricky Nguyen wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/26064/ > --- > > (Updated Sept. 26, 2014, 1:04 a.m.) > > > Review request for oodt, Lewis McGibbney, Chris Mattmann, Paul Ramirez, and > Shakeh Khudikyan. > > > Bugs: OODT-754 > https://issues.apache.org/jira/browse/OODT-754 > > > Repository: oodt > > > Description > --- > > This is a cleaned up version of the ProdTypePatternMetExtractor used at one > point at CHLA. Wrote some unit tests (and fixed a bug!). Made sure to write > some good javadocs and site docs too (they are basically the same as the > wiki). > > > Diffs > - > > trunk/CHANGES.txt 1627442 > trunk/metadata/pom.xml 1627442 > > trunk/metadata/src/main/java/org/apache/oodt/cas/metadata/extractors/AbstractSAXConfigReader.java > PRE-CREATION > > trunk/metadata/src/main/java/org/apache/oodt/cas/metadata/extractors/ProdTypePatternMetExtractor.java > PRE-CREATION > trunk/metadata/src/site/xdoc/user/basic.xml 1627442 > > trunk/metadata/src/test/org/apache/oodt/cas/metadata/extractors/TestAbstractSAXConfigReader.java > PRE-CREATION > > trunk/metadata/src/test/org/apache/oodt/cas/metadata/extractors/TestProdTypePatternMetExtractor.java > PRE-CREATION > trunk/metadata/src/testdata/product-type-patterns-2.xml PRE-CREATION > trunk/metadata/src/testdata/product-type-patterns.xml PRE-CREATION > > Diff: https://reviews.apache.org/r/26064/diff/ > > > Testing > --- > > 2 test classes, one for the SAX reader and one for the met extractor. > > (1) testExtractMetadata: Sanity check, make sure it works as advertised. > (2) testNewConfigFile: If a second config file is parsed, ignore the first > config file. > (3) testElementDeclarationOrder: The order of element declaration is > independent of the order of usage in the product-type template. > > > Thanks, > > Ricky Nguyen > >
Re: OODT & Apache
Awesome, Dan! Here is the timeline file: https://svn.apache.org/repos/asf/oodt/docs/site/ https://svn.apache.org/repos/asf/oodt/docs/site/about/history/index.html Once you update the index.html file, you can either: (1) follow this process: https://cwiki.apache.org/confluence/display/OODT/How+to+Publish+the+OODT+We b+Site to build and publish the site (2) hit me up, or someone else that already has a local copy of the website built and I can push the changes. Cheers, Chris -Original Message- From: Dan Crichton Reply-To: Date: Monday, September 29, 2014 at 11:32 AM To: "dev@oodt.apache.org" Subject: Re: OODT & Apache >Hi Chris, > >I'll volunteer to update the timeline. I'd like to hit the 2000-2005 era >and also add in airborne instruments, etc. > >Thanks, >Dan > >On 9/29/14 9:38 AM, "Mattmann, Chris A (3980)" > wrote: > >>Hi All, recently I came up with some answers to questions >>about Apache OODT that would likely be useful to the community >>so here ya go! :) >> >>(with permission from Jen Cruz to share) >> >>++ >>Chris Mattmann, Ph.D. >>Chief Architect >>Instrument Software and Science Data Systems Section (398) >>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>Office: 168-519, Mailstop: 168-527 >>Email: chris.a.mattm...@nasa.gov >>WWW: http://sunset.usc.edu/~mattmann/ >>++ >>Adjunct Associate Professor, Computer Science Department >>University of Southern California, Los Angeles, CA 90089 USA >>++ >> >> >> >> >> >> >>-Original Message- >>From: , Chris Mattmann >>Date: Monday, September 29, 2014 at 9:31 AM >>To: "Cruz, Jennifer W (3980)" >>Cc: "Freeborn, Dana J (398J)" , "Cuddy, >>David T (398B)" , "Ramirez, Paul M (398J)" >> >>Subject: Re: OODT & Apache >> >>>Hi Jennifer, >>> >>>-Original Message- >>> >>>From: , "Jennifer W (3980)" >>>Date: Monday, September 29, 2014 at 9:20 AM >>>To: Chris Mattmann >>>Cc: "Cruz, Jennifer W (3980)" , "Freeborn, >>>Dana J (398J)" , "Cuddy, David T (398B)" >>> >>>Subject: OODT & Apache >>> >>>>Chris, >>>> >>>> >>>>I¹m working on a chart for SMAP feed forward to NISAR & SWOT. I would >>>>like to show a little of OODT evolution and ties into flight projects. >>> >>>Great. >>> >>>Current list of recent flight projects powered by OODT (Apache and >>>otherwise): >>> >>>OCO-2 - JPL >>>NPP Sounder PEATE - JPL >>>SMAP - JPL >>>CARVE - JPL >>>Airborne Snow Observatory - JPL >>>JPSS GRAVITE - GSFC >>> >>>Note that APL is also currently working on infusing OODT into its Deep >>>Horizons pipeline. >>> >>>> >>>> >>>>What was the timeframe when OODT was available open source/Apache? >>> >>>See: http://oodt.apache.org/about/history/ >>> >>>(relevant subset) >>> >>>2005 OODT becomes open source through Open Channel Foundation, but gains >>>little traction. >>>2007 OODT deployed for the Orbiting Carbon Observatory and Seawinds >>>missions. OODT embraces earth science. >>>2008 OODT deployed in for National Polar-Orbiting Environmental >>>Satellite >>>System. OODT embraces atmospheric science. >>>2010 JanuaryApache OODT enters the Apache Incubator. >>>2010 November OODT makes it's first official release. >>> Apache OODT is voted a Top Level Project. >>>2011 February OODT makes it's first Official Release as a Top Level >>>Project bringing v0.2 to the community. >>>2011 June OODT makes the 0.3 release to the community. >>>2012 June OODT makes the 0.4 release to the community. >>>2012 December OODT makes the 0.5 release to the community. >>>2013 July OODT makes the 0.6 release to the community. >>>2014 September OODT makes the 0.7 release to the community. >>> >>> >>> >>>> >>>> >>>>When might a non Java (python??) version be available? >>> >>>This is an extremely loaded question :) Per Ohloh, rewriting >>>OODT¹s core in Java would mean likely replacing 53
Re: OODT & Apache
Attachment stripped, can you resend Chris Mattmann chris.mattm...@gmail.com -Original Message- From: "Hardman, Sean H (398J)" Reply-To: Date: Monday, September 29, 2014 at 11:44 AM To: "dev@oodt.apache.org" Subject: Re: OODT & Apache >Although a tad outdated, the attached poster shows the timeline from the >early years. > >Sean > >On 9/29/14 11:36 AM, "Chris Mattmann" wrote: > >>Awesome, Dan! >> >>Here is the timeline file: >> >>https://svn.apache.org/repos/asf/oodt/docs/site/ >> >>https://svn.apache.org/repos/asf/oodt/docs/site/about/history/index.html >> >> >>Once you update the index.html file, you can either: >> >>(1) follow this process: >>https://cwiki.apache.org/confluence/display/OODT/How+to+Publish+the+OODT+ >>W >>e >>b+Site >> >>to build and publish the site >> >>(2) hit me up, or someone else that already has a local copy of the >>website >>built and I can push the changes. >> >>Cheers, >>Chris >> >> >> >> >>-Original Message- >>From: Dan Crichton >>Reply-To: >>Date: Monday, September 29, 2014 at 11:32 AM >>To: "dev@oodt.apache.org" >>Subject: Re: OODT & Apache >> >>>Hi Chris, >>> >>>I'll volunteer to update the timeline. I'd like to hit the 2000-2005 >>>era >>>and also add in airborne instruments, etc. >>> >>>Thanks, >>>Dan >>> >>>On 9/29/14 9:38 AM, "Mattmann, Chris A (3980)" >>> wrote: >>> >>>>Hi All, recently I came up with some answers to questions >>>>about Apache OODT that would likely be useful to the community >>>>so here ya go! :) >>>> >>>>(with permission from Jen Cruz to share) >>>> >>>>++ >>>>Chris Mattmann, Ph.D. >>>>Chief Architect >>>>Instrument Software and Science Data Systems Section (398) >>>>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>Office: 168-519, Mailstop: 168-527 >>>>Email: chris.a.mattm...@nasa.gov >>>>WWW: http://sunset.usc.edu/~mattmann/ >>>>++ >>>>Adjunct Associate Professor, Computer Science Department >>>>University of Southern California, Los Angeles, CA 90089 USA >>>>++ >>>> >>>> >>>> >>>> >>>> >>>> >>>>-Original Message- >>>>From: , Chris Mattmann >>>>Date: Monday, September 29, 2014 at 9:31 AM >>>>To: "Cruz, Jennifer W (3980)" >>>>Cc: "Freeborn, Dana J (398J)" , "Cuddy, >>>>David T (398B)" , "Ramirez, Paul M (398J)" >>>> >>>>Subject: Re: OODT & Apache >>>> >>>>>Hi Jennifer, >>>>> >>>>>-Original Message- >>>>> >>>>>From: , "Jennifer W (3980)" >>>>>Date: Monday, September 29, 2014 at 9:20 AM >>>>>To: Chris Mattmann >>>>>Cc: "Cruz, Jennifer W (3980)" , >>>>>"Freeborn, >>>>>Dana J (398J)" , "Cuddy, David T (398B)" >>>>> >>>>>Subject: OODT & Apache >>>>> >>>>>>Chris, >>>>>> >>>>>> >>>>>>I¹m working on a chart for SMAP feed forward to NISAR & SWOT. I >>>>>>would >>>>>>like to show a little of OODT evolution and ties into flight >>>>>>projects. >>>>> >>>>>Great. >>>>> >>>>>Current list of recent flight projects powered by OODT (Apache and >>>>>otherwise): >>>>> >>>>>OCO-2 - JPL >>>>>NPP Sounder PEATE - JPL >>>>>SMAP - JPL >>>>>CARVE - JPL >>>>>Airborne Snow Observatory - JPL >>>>>JPSS GRAVITE - GSFC >>>>> >>>>>Note that APL is also currently working on infusing OODT into its Deep >>>>>Horizons pipeline. >>>>> >>>>>> >>>>>> >>>>>>What was the timeframe when OODT was available open source/Apache? >
Re: error ingesting product :(
Thanks for checking on this Val. Did you make sure to restart the File Manager after your policy change? FM only reads those policy files on start up. One thing to note too - that catalog folder is created by the FM on its own. It shouldn¹t be created by you beforehand. Try removing catalog. Stopping FM. Then restarting it (trusting it will be created on ingest). Did that fix it? Cheers, Chris Chris Mattmann chris.mattm...@gmail.com -Original Message- From: "Mallder, Valerie" Reply-To: Date: Thursday, October 2, 2014 at 2:30 PM To: "dev@oodt.apache.org" Subject: RE: error ingesting product :( >Hi John, > >Thanks for writing! I was just getting close to posting an update on my >progress, but you beat me to it. I was able to fix the first part of >this error when I found that I had a typo in my product-types.xml file. I >scanned that file a thousand times before I posted my message, but I >never saw the typo so now I feel pretty stupid. > >I've made a bunch of progress since this afternoon, but I am still stuck >and not able to ingest my files. I have been unable to generate the >correct reference for the final location of the files in the archive >folder. And, I CatalogExceptions because the archive/catalog/segments >file does not exist. I think I need to use the >MetadataBasedFileVersioner, so that I can specify the destination folder >using values in the metadata, but I can't seem to get that to work. So, I >am going to post a new topic to see if I can get some help with that. > >Thanks, >Val > > > >Valerie A. Mallder >New Horizons Deputy Mission System Engineer >Johns Hopkins University/Applied Physics Laboratory > >> -Original Message- >> From: Lewis John Mcgibbney [mailto:lewis.mcgibb...@gmail.com] >> Sent: Thursday, October 02, 2014 2:48 PM >> To: dev@oodt.apache.org >> Subject: Re: error ingesting product :( >> >> Hi Valerie, >> >> It looks like you've run into our good old friend 'null' :) >> >> A basic question before we go further. >> Which version of OODT are you using? >> I ask this because the stack trace is valuable for hopefully stepping >>throgh code >> myself. >> >> On Thu, Oct 2, 2014 at 9:28 AM, Mallder, Valerie >>> > wrote: >> >> > >> > PID file >> > (/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/filemgr/ >> > run/cas.filemgr.pid) found. Is File Manager still running? Start >> > aborted. >> > >> >> I don't really like the look of this one? Have you tried killing all >>processes relating >> to filemgr before starting the filemgr again? The start aborted in >>particular is quite >> worrying. Does anyone know where this message comes from? >> >> >> > org.apache.xmlrpc.XmlRpcException: java.lang.Exception: >> > org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error >> > ingesting product >>[org.apache.oodt.cas.filemgr.structs.Product@fb60123] : >> > null >> > >> >> We really need to sort out these cryptic error messages. I've logged an >>issue to >> address this >> >> https://issues.apache.org/jira/browse/OODT-755
Re: how to use MetadataBasedFileVersioner properly
You¹re almost there Val! Unfortunately Versioners right now aren¹t configurable from product type policy (would be great to capture this in a JIRA issue, here: https://issues.apache.org/jira/browse/OODT). If they were, it would have picked up your https://issues.apache.org/jira/browse/OODT-639 So you can amend your definition below (also note you always have to include /[Filename] at the end to get the filename you want). The default product type for any kind of file. filePathSpec /[YearDir]/[DoyDir]/[Filename] See if that fixes it! Cheers, Chris Chris Mattmann chris.mattm...@gmail.com -Original Message- From: "Mallder, Valerie" Reply-To: Date: Thursday, October 2, 2014 at 2:56 PM To: "dev@oodt.apache.org" Subject: how to use MetadataBasedFileVersioner properly >Hi All, > >I am trying to use the MetadataBasedFileVersioner to store my files in a >subdirectory of the data/archive folder based on the values of two >metadata elements. Has anyone does this before, and if so, can you give >me some hints. > >My files need to be organized in subdirectories by year and day of year. >So, lets say the file name is 'myfile', the year is 2014, and the doy of >year is 002. Then, the end result that I am looking for is to have the >final location of my file be: 'data/archive/ops/eng/2014/002/myfile. > >However, in my log file, the INFO messages indicate that the generated >final location reference is 'data/archive/ops/eng/myfile'. And it >doesn't include the year and day of year at all. And the incorrect >location leads to other errors. So I want to solve this one first. Has >anyone tried to do something like this before? > >Here is what I have done so far: > >Added two new elements to the .met file 'YearDir' and 'DoyDir'. >Added these new elements to the elements.xml file. >Added these new elements to the product in the >product-type-element-map.xml file. >And, I have made changes to the product-types.xml file, but here's where >I am not sure I've done this properly. > >Here's what product type definition looks like: > > >class="org.apache.oodt.cas.filemgr.versioning.MetadataBasedFileVersioner"> > > >The default product type for any kind of >file. > > class="org.apache.oodt.cas.filemgr.metadata.extractors.CoreMetExtractor"> > > > > > value="ProductReceivedTime,ProductName,ProductId" /> > > > > > > >Thanks in advance for any help or ideas you might have! > >Valerie > > > >Valerie A. Mallder > >New Horizons Deputy Mission System Engineer >The Johns Hopkins University/Applied Physics Laboratory >11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723 >240-228-7846 (Office) 410-504-2233 (Blackberry) >
Re: how to pass arguments to workflow task that is external script
Hi Val,
Thanks for the detailed report. My suggestion would be to use CAS-PGE
directly instead of
ExternScriptTaskInstance. That application is not well maintained, doesn¹t
produce a log, etc,
etc, all of the things you¹ve noted.
CAS-PGE on the other hand, will (a) prepare input for your task; (b)
describe how to run your task (even as a script and will generate a
script); and (c) will run met extractors and fork a crawler in your job
directory in the end.
I think it¹s what you¹re looking for and it¹s way more well documented on
the wiki.
Please check it out and let me know what you think.
Cheers,
Chris
Chris Mattmann
chris.mattm...@gmail.com
-Original Message-
From: "Mallder, Valerie"
Reply-To:
Date: Monday, October 6, 2014 at 11:53 PM
To: "dev@oodt.apache.org"
Subject: how to pass arguments to workflow task that is external script
>Hello,
>
>I'm stuck again L This time I'm stuck trying to start my crawler as a
>task using the workflow manager. I am not using a PGE task right now.
>I'm just trying to do something simple with the workflow manager,
>filemgr, and crawler. I have read all of the documentation that is
>available on the workflow manager and have tried to piece together a
>setup based on the examples, but, things seem to be working differently
>now and the documentation hasn't caught up, which is totally
>understandable and not a criticism. Just want you to know that I try to
>do my due diligence before bothering anyone for help.
>
>I am not running the resource manager, and I have commented out setting
>the resource manager url in the workflow.properties file so that workflow
>manager will execute the job locally.
>
>I am sending workflow manager an event (via the command line using
>wmgr-client) called "startJediPipeline". Workflow manager receives the
>event, and retrieves my workflow from the repository and tries to execute
>the first (and only) task, and then it crashes. My task is an external
>script (the crawler_launcher script) and I need to pass several arguments
>to it. I've spent all day trying to figure out how to pass arguments to
>the and ExternScriptTaskInstance, but there are no examples of doing
>this, so I had to wing it. I tried putting the arguments in the task
>configuration properties. That didn't work. So I tried putting the
>arguments in the metadata properties, and that hasn't worked. So, your
>suggestions are welcome! Thanks so much. Here's the error log, And
>contents of my tasks.xml file follow it at the end.
>
>Workflow Manager started PID file
>(/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/workflow/run
>/cas.workflow.pid).
>Starting OODT File Manager [ Successful ]
>Starting OODT Resource Manager [ Failed ]
>Starting OODT Workflow Manager [ Successful ]
>slothrop:{~/project/jedi/users/jedi-pipeline/oodt-deploy/bin} Oct 06,
>2014 5:48:30 PM org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager
>loadProperties
>INFO: Loading Workflow Manager Configuration Properties from:
>[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/workflow/etc
>/workflow.properties]
>Oct 06, 2014 5:48:30 PM
>org.apache.oodt.cas.workflow.engine.ThreadPoolWorkflowEngineFactory
>getResmgrUrl
>INFO: No Resource Manager URL provided or malformed URL: executing jobs
>locally. URL: [null]
>Oct 06, 2014 5:48:30 PM
>org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager
>INFO: Workflow Manager started by malldva1
>Oct 06, 2014 5:48:41 PM
>org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager handleEvent
>INFO: WorkflowManager: Received event: startJediPipeline
>Oct 06, 2014 5:48:41 PM
>org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager handleEvent
>INFO: WorkflowManager: Workflow Jedi Pipeline Workflow retrieved for
>event startJediPipeline
>Oct 06, 2014 5:48:41 PM
>org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
>checkTaskRequiredMetadata
>INFO: Task: [Crawler Task] has no required metadata fields
>Oct 06, 2014 5:48:42 PM
>org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
>executeTaskLocally
>INFO: Executing task: [Crawler Task] locally
>java.lang.NullPointerException
>at
>org.apache.oodt.cas.workflow.examples.ExternScriptTaskInstance.run(ExternS
>criptTaskInstance.java:72)
>at
>org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.execu
>teTaskLocally(IterativeWorkflowProcessorThread.java:574)
>at
>org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.run(I
>terativeWorkflowProcessorThread.java:321)
>at
>EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
Re: how to pass arguments to workflow task that is external script
Thanks Val, I agree, yes, CAS-PGE is complex. Did you see the learn by example wiki page: https://cwiki.apache.org/confluence/display/OODT/CAS-PGE+Learn+by+Example I think it¹s pretty basic and illustrates what CAS-PGE does. Basically the jist of it is: 1. you only need to create a PGEConfig.xml file that specifies: - how to generate input for your integrated algorithm - how to execute your algorithm (e.g., how to generate a script that executes it) - how to generate metadata from the output, and then to crawl the files + met and get the outputs into the file manager 2. you go into workflow tasks.xml, define a new CAS-PGE type task, point at this config file, and provide CAS-PGE task properties (an example is here: http://svn.apache.org/repos/asf/oodt/trunk/pge/src/main/resources/examples/ WorkflowTask/ If you want to see a basic example of CAS-PGE in action, check out DRAT: https://github.com/chrismattmann/drat/ It¹s a RADIX-based deployment with 2 CAS-PGEs (one for the MIME partition; and another for RAT). Check that out, see how DRAT works (and integrates CAS-PGE) and then let me know if you are still confused and I will be glad to help more. Cheers, Chris Chris Mattmann chris.mattm...@gmail.com -Original Message- From: "Mallder, Valerie" Reply-To: Date: Tuesday, October 7, 2014 at 4:56 PM To: "dev@oodt.apache.org" Subject: RE: how to pass arguments to workflow task that is external script >Thanks Chris, > >The CAS-PGE is pretty complex, I've read the documentation and it is >still way over my head. Is there any documentation or examples for how >to integrate the crawler into it? For instance, can I still use the >crawler_launcher script? Will the ExternMetExtractor and a >postIngestSuccess ExternAction script work that I created to work with >the crawler still work "as is" in the CAS-PGE ? Or, should I invoke them >differently? What about the Metadata that I extracted with the crawler? >Do I have to redefine the metadata elements in another configuration file >or policy file? If there is any documentation on doing this please point >me to the right place because I didn't see anything that addressed these >kinds of questions. > >Thanks, >Val > >Do I have to define these any differently in the PGE configuration > > >Valerie A. Mallder >New Horizons Deputy Mission System Engineer >Johns Hopkins University/Applied Physics Laboratory > >> -Original Message- >> From: Chris Mattmann [mailto:chris.mattm...@gmail.com] >> Sent: Tuesday, October 07, 2014 8:16 AM >> To: dev@oodt.apache.org >> Subject: Re: how to pass arguments to workflow task that is external >>script >> >> Hi Val, >> >> Thanks for the detailed report. My suggestion would be to use CAS-PGE >>directly >> instead of ExternScriptTaskInstance. That application is not well >>maintained, >> doesn?t produce a log, etc, etc, all of the things you?ve noted. >> >> CAS-PGE on the other hand, will (a) prepare input for your task; (b) >>describe how >> to run your task (even as a script and will generate a script); and (c) >>will run met >> extractors and fork a crawler in your job directory in the end. >> >> I think it?s what you?re looking for and it?s way more well documented >>on the wiki. >> >> Please check it out and let me know what you think. >> >> Cheers, >> Chris >> >> >> Chris Mattmann >> chris.mattm...@gmail.com >> >> >> >> >> -Original Message- >> From: "Mallder, Valerie" >> Reply-To: >> Date: Monday, October 6, 2014 at 11:53 PM >> To: "dev@oodt.apache.org" >> Subject: how to pass arguments to workflow task that is external script >> >> >Hello, >> > >> >I'm stuck again L This time I'm stuck trying to start my crawler as a >> >task using the workflow manager. I am not using a PGE task right now. >> >I'm just trying to do something simple with the workflow manager, >> >filemgr, and crawler. I have read all of the documentation that is >> >available on the workflow manager and have tried to piece together a >> >setup based on the examples, but, things seem to be working differently >> >now and the documentation hasn't caught up, which is totally >> >understandable and not a criticism. Just want you to know that I try >> >to do my due diligence before bothering anyone for help. >> > >> >I am not running the resource manager, and I have commented out setting >> >the resource manager url in the workflow.propertie
Re: HttpClient NoClassDefFoundError For the url-downloader Script of the Apache OODT Crawler
Yep woot! Committed!
Chris Mattmann
chris.mattm...@gmail.com
-Original Message-
From: Lewis John Mcgibbney
Reply-To:
Date: Thursday, October 9, 2014 at 12:01 AM
To: MengYing Wang
Cc: Chris Mattmann , Paul M Ramirez
, "dev@oodt.apache.org"
Subject: Re: HttpClient NoClassDefFoundError For the url-downloader Script
of the Apache OODT Crawler
>Dynamite Angela.
>Thanks for your persistence.
>
>On Sat, Oct 4, 2014 at 8:22 PM, MengYing Wang
>wrote:
>
>> Dear Lewis,
>>
>> Done. This is the jira url:
>>https://issues.apache.org/jira/browse/OODT-756
>>
>> Best,
>> Mengying Wang
>>
>> On Tue, Sep 30, 2014 at 6:59 PM, Lewis John Mcgibbney <
>> lewis.mcgibb...@gmail.com> wrote:
>>
>>> Ok it sounds like we need to file an issue for this.
>>> Can you possibly of an issue in Jira?
>>> If someone else can reproduce then it is a bug which we need to fix.
>>> Thank you
>>> Lewis
>>>
>>>
>>> On Tuesday, September 30, 2014, MengYing Wang
>>> wrote:
>>>
>>>> Dear Lewis,
>>>>
>>>> By the way, no need to change any code in the script url-downloader.
>>>>
>>>> Thank you.
>>>>
>>>> Best,
>>>> Angela Wang
>>>>
>>>> On Tue, Sep 30, 2014 at 1:29 PM, MengYing Wang
>>>>
>>>> wrote:
>>>>
>>>>> Dear Lewis,
>>>>>
>>>>> The problem has been solved! *~*
>>>>>
>>>>> I just noticed that the class
>>>>> org.apache.oodt.cas.pushpull.protocol.http.HttpClient does exist in
>>>>> cas-pushpull-0.2.jar, but disappeared in cas-pushpull-0.6 (Why?). So
>>>>>I just
>>>>> downloaded the cas-pushpull-0.2.jar file from
>>>>> http://www.java2s.com/Code/Jar/c/Downloadcaspushpull02jar.htm, and
>>>>> threw it into my pushpull lib directory. Then I can start the
>>>>> url-downloader script normally. Thank you!
>>>>>
>>>>> Best,
>>>>> Angela Wang
>>>>>
>>>>> On Tue, Sep 30, 2014 at 8:28 AM, Lewis John Mcgibbney <
>>>>> lewis.mcgibb...@gmail.com> wrote:
>>>>>
>>>>>> Hi Angela,
>>>>>> If this is a change against the code base and you think that is an
>>>>>> improvement can you possibly log an issue in our issue tracking
>>>>>>system?
>>>>>>
>>>>>>
>>>>>>
>>>>>>https://issues.apache.org/jira/browse/OODT/?selectedTab=com.atlassian
>>>>>>.jira.jira-projects-plugin:summary-panel
>>>>>>
>>>>>> The usual process is to create a Jira account, log in, create an
>>>>>>issue.
>>>>>> We would encourage you to be as verbose as possible when logging
>>>>>> issues. It really helps to describe what was wrong, how it can be
>>>>>> reproduced and what your suggested fix is.
>>>>>>
>>>>>> If you are able to attach a patch for fixing the issue then this is
>>>>>> the cherry on the cake.
>>>>>> Thanks
>>>>>> Lewis
>>>>>>
>>>>>> On Monday, September 29, 2014, MengYing Wang
>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>> Dear Lewis,
>>>>>>>
>>>>>>> Thank you for your help. I have modified the url-downloader script
>>>>>>> to add all jar files into the classpath. Following is the new
>>>>>>>content.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>--
>>>>>>>
>>>>>>> set LIB_DEPS
>>>>>>>
>>>>>>> foreach file (`find ../lib/*.jar`)
>>>>>>>
>>>>>>> set LIB_DEPS = "${file}:${LIB_DEPS}"
>>>>>>>
>>>>>>> end
>>>>>>>
>>>>>>>
>>>>>>> set
>>>>>>>
>>>>>>>JAVA_HOME="/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Conten
>>>>>>>ts/Home"
>>>>>>>
>&
Re: How to ingest files when metadata contain non standard characters?
cas-metadata should handle this escaping/unescaping in its SerDe capabilities. Kostsas, can yo provide the exact file that I can test on and upload it to JIRA? Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Lewis John Mcgibbney Reply-To: Date: Thursday, October 9, 2014 at 2:59 AM To: "dev@oodt.apache.org" Subject: Re: How to ingest files when metadata contain non standard characters? >Hi Kos, >Thanks for reply > >On Wed, Oct 8, 2014 at 5:16 PM, Konstantinos Mavrommatis < >kmavromma...@celgene.com> wrote: > >> I escaped the characters using the CGI::escapeHTML function from the CGI >> perl module. >> > >Wow. I am surpised at this one. I wonder if this is a bug which results in >the discrepancy or if this is intential behaviour! > > >> >> The differences between the two versions (mine escaped vs yours escaped) >> is in the encoding of the single quote "'" character, if I am not >>mistaken. >> I want to clarify this because your email come as simple ASCII (not >>HTML) >> > >Yes that is correct. > > >> >> I did try your command and it worked !!! >> > >OK grand. > > >> >> Now the question is how to do this encoding (your version) ☺ >> >> >Is this the question? My thoughts would be that this should be >encapsulated >within OODT somewhere and that it should not be necessary to escape >everything as you/we have been doing. This is extremely time consuming and >painful. > >I escaped everything here >http://www.freeformatter.com/html-escape.html > >and compared the strings here >http://text-compare.com/ > >The latter resource will verify that it is the single quote that is the >offending char here. >Thanks >Lewis
Re: what is batch stub? Is it necessary?
Hi Val, I don¹t think you need to run a CAS-PGE task to call crawler_launcher. If you define blocks in the .. section of the XML file, a crawler will be forked in the job working directory of CAS-PGE and crawl your specified output. I believe that will accomplish the same goal of what you are looking for. No need to have crawling be a separate task from CAS-PGE - CAS-PGE will do the crawling for you! :) Cheers, Chris Chris Mattmann chris.mattm...@gmail.com -Original Message- From: "Verma, Rishi (398J)" Reply-To: Date: Thursday, October 9, 2014 at 2:44 AM To: "dev@oodt.apache.org" Subject: Re: what is batch stub? Is it necessary? >Hi Val, > >Yep - here¹s a link to the tasks.xml file: >https://github.com/riverma/xdata-jpl-netscan/blob/master/oodt-netscan/work >flow/src/main/resources/policy/tasks.xml > >> The problem is that the ExternScriptTaskInstance is unable to recognize >>the command line arguments that I want to pass to the crawler_launcher >>script. > > >Hmm.. could you share your workflow manager log, or better yet, the >batch_stub output? Curious to see what error is thrown. > >Is a script file being generated for your PGE? For example, inside your >[PGE_HOME] directory, and within the particular job directory created for >your execution of a workflow, you will see some files starting with >³sciPgeExeScript_². You¹ll find one for your pgeConfig, and you can >check to see what the PGE commands actually translate into, with respect >to a shell script format. If that file is there, take a look at it, and >validate whether the command works within the script (i.e. copy/paste and >run the crawler command manually). > >Another suggestion is to take a step back, and build up slowly, i.e.: >1. Do an ³echo² command within your PGE first. (e.g. echo ³Hello >APL.² > /tmp/test.txt) >2. If above works, do a crawler_launcher empty command(e.g. >/path/to/oodt/crawler/bin/crawler_launcher) and verify the >batch_stub or Workflow Manager prints some kind of output when you run >the workflow. >3. Build up your crawler_launcher command piece by piece to see where it >is failing > >Thanks, >Rishi > >On Oct 8, 2014, at 4:24 PM, Mallder, Valerie >wrote: > >> Hi Rishi, >> >> Thank you very much for pointing me to your working example. This is >>very helpful. My pgeConfig looks very similar to yours. So, I >>commented out the resource manager like you suggested and tried running >>again without the resource manager. And my problem still exists. The >>problem is that the ExternScriptTaskInstance is unable to recognize the >>command line arguments that I want to pass to the crawler_launcher >>script. Could you send me a link to your tasks.xml file? I'm curious as >>to how you defined your task. My pgeConfig and tasks.xml are below. >> >> Thanks! >> Val >> >> >> >> >> >> >> >>[CRAWLER_HOME]/bin/crawler_launcher --operation >>--launchAutoCrawler \ >>--filemgrUrl [FILEMGR_URL] \ >>--clientTransferer >>org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory \ >>--productPath [JobInputDir] \ >>--mimeExtractorRepo >>[OODT_HOME]/extensions/policy/mime-extractor-map.xml \ >>--actionIds MoveFileToLevel0Dir >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> http://oodt.jpl.nasa.gov/1.0/cas";> >> >> >class="org.apache.oodt.cas.workflow.examples.ExternScriptTaskInstance"> >> >> >> >> >> >value="[CRAWLER_HOME]/bin/crawler_launcher" envReplace="true" /> >> >> >> >value="[OODT_HOME]/extensions/config/crawler-pge-config.xml" >>envReplace="true" /> >> >> >> >> >> >> Valerie A. Mallder >> New Horizons Deputy Mission System Engineer >> Johns Hopkins University/Applied Physics Laboratory >> >> >>> -Original Message- >>> From: Verma, Rishi (398J) [mailto:rishi.ve...@jpl.nasa.gov] >>> Sent: Wednesday, October 08, 2014 6:01 PM >>> To: dev@oodt.apache.org >>> Subject: Re: what is batch stub? Is it necessary? >>> >>> Hi Valerie, >>> >>>>>>> All I am trying to do is run "crawler_launcher" as a workflow task >>>>>>>
Re: what is batch stub? Is it necessary?
ack, git push Chris Mattmann chris.mattm...@gmail.com -Original Message- From: "Ramirez, Paul M (398J)" Reply-To: Date: Thursday, October 9, 2014 at 4:37 AM To: "" Subject: Re: what is batch stub? Is it necessary? >+1 billion > >--Paul > >Sent from my iPhone > >> On Oct 8, 2014, at 5:55 PM, Lewis John Mcgibbney >> wrote: >> >> Folks, >> Is it possible to create a parent issue for defining XSD's for all of >>the >> XML file we need ti OODT? >> I do not know them all, but from this thread alone, it is clear that we >> could do with setting some kind of restrictions on what can be included >> within task and configuration XML within OODT. >> Thoughts? >> Lewis >> >> On Wed, Oct 8, 2014 at 5:44 PM, Verma, Rishi (398J) < >> rishi.ve...@jpl.nasa.gov> wrote: >> >>> Hi Val, >>> >>> Yep - here¹s a link to the tasks.xml file: >>> >>> >>>https://github.com/riverma/xdata-jpl-netscan/blob/master/oodt-netscan/wo >>>rkflow/src/main/resources/policy/tasks.xml >>> >>>> The problem is that the ExternScriptTaskInstance is unable to >>>>recognize >>> the command line arguments that I want to pass to the crawler_launcher >>> script. >>> >>> >>> Hmm.. could you share your workflow manager log, or better yet, the >>> batch_stub output? Curious to see what error is thrown. >>> >>> Is a script file being generated for your PGE? For example, inside your >>> [PGE_HOME] directory, and within the particular job directory created >>>for >>> your execution of a workflow, you will see some files starting with >>> ³sciPgeExeScript_². You¹ll find one for your pgeConfig, and you can >>>check >>> to see what the PGE commands actually translate into, with respect to a >>> shell script format. If that file is there, take a look at it, and >>>validate >>> whether the command works within the script (i.e. copy/paste and run >>>the >>> crawler command manually). >>> >>> Another suggestion is to take a step back, and build up slowly, i.e.: >>> 1. Do an ³echo² command within your PGE first. (e.g. echo ³Hello >>> APL.² > /tmp/test.txt) >>> 2. If above works, do a crawler_launcher empty command(e.g. >>> /path/to/oodt/crawler/bin/crawler_launcher) and verify the >>> batch_stub or Workflow Manager prints some kind of output when you run >>>the >>> workflow. >>> 3. Build up your crawler_launcher command piece by piece to see where >>>it >>> is failing >>> >>> Thanks, >>> Rishi >>> >>> On Oct 8, 2014, at 4:24 PM, Mallder, Valerie >>> >>> wrote: >>> >>>> Hi Rishi, >>>> >>>> Thank you very much for pointing me to your working example. This is >>> very helpful. My pgeConfig looks very similar to yours. So, I >>>commented >>> out the resource manager like you suggested and tried running again >>>without >>> the resource manager. And my problem still exists. The problem is that >>>the >>> ExternScriptTaskInstance is unable to recognize the command line >>>arguments >>> that I want to pass to the crawler_launcher script. Could you send me a >>> link to your tasks.xml file? I'm curious as to how you defined your >>>task. >>> My pgeConfig and tasks.xml are below. >>>> >>>> Thanks! >>>> Val >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> [CRAWLER_HOME]/bin/crawler_launcher --operation >>> --launchAutoCrawler \ >>>> --filemgrUrl [FILEMGR_URL] \ >>>> --clientTransferer >>> org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory \ >>>> --productPath [JobInputDir] \ >>>> --mimeExtractorRepo >>> [OODT_HOME]/extensions/policy/mime-extractor-map.xml \ >>>> --actionIds MoveFileToLevel0Dir >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> http://oodt.jpl.nasa.gov/1.0/cas&quo
Re: How to ingest files when metadata contain non standard characters?
Thanks Kostas. Can you upload somewhere and then point here, the message list strips attachments.. Cheers, Chris Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Konstantinos Mavrommatis Reply-To: Date: Thursday, October 9, 2014 at 5:48 AM To: "dev@oodt.apache.org" Subject: RE: How to ingest files when metadata contain non standard characters? >Thanks Chris, > >attached is an offending file before escape. >For the record perl module HTML::Entities does provide an escapeHTML >alternative that produces acceptable files. > >Thanks >K > > >> -Original Message- >> From: Chris Mattmann [mailto:chris.mattm...@gmail.com] >> Sent: Wednesday, October 08, 2014 11:38 AM >> To: dev@oodt.apache.org >> Subject: Re: How to ingest files when metadata contain non standard >> characters? >> >> cas-metadata should handle this escaping/unescaping in its SerDe >> capabilities. >> >> Kostsas, can yo provide the exact file that I can test on and upload it >> to JIRA? >> >> >> Chris Mattmann >> chris.mattm...@gmail.com >> >> >> >> >> -Original Message- >> From: Lewis John Mcgibbney >> Reply-To: >> Date: Thursday, October 9, 2014 at 2:59 AM >> To: "dev@oodt.apache.org" >> Subject: Re: How to ingest files when metadata contain non standard >> characters? >> >> >Hi Kos, >> >Thanks for reply >> > >> >On Wed, Oct 8, 2014 at 5:16 PM, Konstantinos Mavrommatis < >> >kmavromma...@celgene.com> wrote: >> > >> >> I escaped the characters using the CGI::escapeHTML function from the >> >> CGI perl module. >> >> >> > >> >Wow. I am surpised at this one. I wonder if this is a bug which >> results >> >in the discrepancy or if this is intential behaviour! >> > >> > >> >> >> >> The differences between the two versions (mine escaped vs yours >> >>escaped) is in the encoding of the single quote "'" character, if I >> >>am not mistaken. >> >> I want to clarify this because your email come as simple ASCII (not >> >>HTML) >> >> >> > >> >Yes that is correct. >> > >> > >> >> >> >> I did try your command and it worked !!! >> >> >> > >> >OK grand. >> > >> > >> >> >> >> Now the question is how to do this encoding (your version) ☺ >> >> >> >> >> >Is this the question? My thoughts would be that this should be >> >encapsulated within OODT somewhere and that it should not be necessary >> >to escape everything as you/we have been doing. This is extremely time >> >consuming and painful. >> > >> >I escaped everything here >> >http://www.freeformatter.com/html-escape.html >> > >> >and compared the strings here >> >http://text-compare.com/ >> > >> >The latter resource will verify that it is the single quote that is >> the >> >offending char here. >> >Thanks >> >Lewis >> > >* >THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS >CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED >INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL >OR INDIVIDUALS NAMED ABOVE. >If the reader is not the intended recipient, or the >employee or agent responsible to deliver it to the >intended recipient, you are hereby notified that any >dissemination, distribution or copying of this >communication is strictly prohibited. If you have >received this communication in error, please reply to the >sender to notify us of the error and delete the original >message. Thank You.
Re: how to use MetadataBasedFileVersioner properly
No problem at all, Val. Keep up all the great work. You will be there soon and we will be so happy when you get there! In the meanwhile we¹re here to help. Cheers, Chris Chris Mattmann chris.mattm...@gmail.com -Original Message- From: "Mallder, Valerie" Reply-To: Date: Tuesday, October 14, 2014 at 9:51 AM To: "dev@oodt.apache.org" Subject: RE: how to use MetadataBasedFileVersioner properly >Hi Chris, > >I'm taking a few minutes here to respond to all the folks who have >offered me some help in the past couple weeks, and I don't know if I ever >responded to this one from you. Your help below was golden and it helped >me make more progress at that time. Thanks so much! > >Val > > > >Valerie A. Mallder >New Horizons Deputy Mission System Engineer >Johns Hopkins University/Applied Physics Laboratory > >> -Original Message- >> From: Chris Mattmann [mailto:chris.mattm...@gmail.com] >> Sent: Friday, October 03, 2014 3:03 AM >> To: dev@oodt.apache.org >> Subject: Re: how to use MetadataBasedFileVersioner properly >> >> You?re almost there Val! >> >> Unfortunately Versioners right now aren?t configurable from product >>type policy >> (would be great to capture this in a JIRA issue, here: >> https://issues.apache.org/jira/browse/OODT). If they were, it would >>have picked up >> your > It?s been on my TODO list for a long time. >> >> Instead I created this: >> >> https://issues.apache.org/jira/browse/OODT-639 >> >> >> So you can amend your definition below (also note you always have to >>include >> /[Filename] at the end to get the filename you want). >> >> >> >> > class=?org.apache.oodt.cas.filemgr.versioning.ProductTypeMetVersioner?/> >> The default product type for any kind of >>file. >> >> > >>class="org.apache.oodt.cas.filemgr.metadata.extractors.CoreMetExtractor"> >> >> >> >> >> > value="ProductReceivedTime,ProductName,ProductId" /> >> >> >> >> >> >> filePathSpec >> /[YearDir]/[DoyDir]/[Filename] >> >> >> >> >> >> >> See if that fixes it! >> >> Cheers, >> Chris >> >> >> Chris Mattmann >> chris.mattm...@gmail.com >> >> >> >> >> -Original Message- >> From: "Mallder, Valerie" >> Reply-To: >> Date: Thursday, October 2, 2014 at 2:56 PM >> To: "dev@oodt.apache.org" >> Subject: how to use MetadataBasedFileVersioner properly >> >> >Hi All, >> > >> >I am trying to use the MetadataBasedFileVersioner to store my files in >> >a subdirectory of the data/archive folder based on the values of two >> >metadata elements. Has anyone does this before, and if so, can you give >> >me some hints. >> > >> >My files need to be organized in subdirectories by year and day of >>year. >> >So, lets say the file name is 'myfile', the year is 2014, and the doy >> >of year is 002. Then, the end result that I am looking for is to have >> >the final location of my file be: >>'data/archive/ops/eng/2014/002/myfile. >> > >> >However, in my log file, the INFO messages indicate that the generated >> >final location reference is 'data/archive/ops/eng/myfile'. And it >> >doesn't include the year and day of year at all. And the incorrect >> >location leads to other errors. So I want to solve this one first. Has >> >anyone tried to do something like this before? >> > >> >Here is what I have done so far: >> > >> >Added two new elements to the .met file 'YearDir' and 'DoyDir'. >> >Added these new elements to the elements.xml file. >> >Added these new elements to the product in the >> >product-type-element-map.xml file. >> >And, I have made changes to the product-types.xml file, but here's >> >where I am not sure I've done this properly. >> > >> >Here's what product type definition looks like: >> > >> > >> >> >>>class="org.apache.oodt.cas.filemgr.versioning.MetadataBasedFileVersioner >>>"> >> > >> > >> >The default product type for any kind of >> >file. >> > >> > > >>>class="org.apache.oodt.cas.filemgr.metadata.extractors.CoreMetExtractor" >>>> >> > >> > >> > >> > >> > > >value="ProductReceivedTime,ProductName,ProductId" /> >> > >> > >> > >> > >> > >> > >> >Thanks in advance for any help or ideas you might have! >> > >> >Valerie >> > >> > >> > >> >Valerie A. Mallder >> > >> >New Horizons Deputy Mission System Engineer The Johns Hopkins >> >University/Applied Physics Laboratory >> >11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723 >> >240-228-7846 (Office) 410-504-2233 (Blackberry) >> > >> >
Re: radix install cas-workflow-0.3.jar vs.cas-workflow-0.7.jar
Thanks Val I¹ve added you to the JIRA permissions list so you can create, assign issues, etc. I¹ve assigned OODT-761 to you for now! :) Chris Mattmann chris.mattm...@gmail.com -Original Message- From: "Mallder, Valerie" Reply-To: Date: Tuesday, October 14, 2014 at 11:28 AM To: "dev@oodt.apache.org" Subject: RE: radix install cas-workflow-0.3.jar vs.cas-workflow-0.7.jar >Hi Paul, > >That is the nicest thing anyone has ever said to me! Thank you! > >I submitted a new issue in Jira, but I don't see how I can assign it to >you. Here's the link. >https://issues.apache.org/jira/browse/OODT-761 > >Val > > > >Valerie A. Mallder >New Horizons Deputy Mission System Engineer >Johns Hopkins University/Applied Physics Laboratory > > >> -Original Message- >> From: Ramirez, Paul M (398J) [mailto:paul.m.rami...@jpl.nasa.gov] >> Sent: Friday, October 10, 2014 11:59 PM >> To: >> Subject: Re: radix install cas-workflow-0.3.jar vs.cas-workflow-0.7.jar >> >> Hi Valerie, >> >> The cas-pge at 0.3 was on purpose as there was a bug in OODT 0.6 that >>required >> is to push it back to 0.3. I believe that big is resolved I 0.7 and we >>should put that >> dependency back at 0.7. This is actually set in the main pom file >>before you build >> your RADiX install of OODT. >> >> The cas-workflow being 0.7 is a big and should be changed. Can you >>create a Jira >> issue and assign it to me and I will fix it this weekend. >> >> May I also say that you are awesome for you extensive testing and >> troubleshooting. >> >> Thanks, >> Paul >> >> Sent from my iPhone >> >> On Oct 10, 2014, at 3:31 PM, Mallder, Valerie >> mailto:valerie.mall...@jhuapl.edu>> wrote: >> >> Hi Lewis, >> >> I used the radix installation script to install oodt on my system. I >>modified it to to >> pull version 7 rather than version 6, and my project name is jedi. Here >>are listings >> of all the files in my project directory (jedi-oodt) and my deployment >>(oodt-deploy) >> directory. >> >> Please note that I had to copy oodt-deploy/cas-filemgr-0.7.jar to >>oodt-deploy >> /workflow/lib in order to get past the CoreMetKeys class not found >>exception that >> occurred when I tried to start my pge crawler task via the workflow >>manager (which >> still doesn?t start up, but for different reasons which I will post >>separately) >> >> I noticed that oodt-deploy/resmgr/lib contained, cas-workflow-0.3.jar, >>but oodt- >> deploy/pge/lib, oodt-deploy/workflow/lib, oodt-deploy/crawler/lib, and >>oodt- >> deploy/filemgr/lib contain cas-workflow-0.7.jar. So I copied v0.7 to >>resmgr/lib and I >> am running with that now. >> >> I also noticed that oodt-deploy/pge/lib, oodt-deploy/resmgr/lib, and >>oodt- >> deploy/workflow/lib contain cas-pge-0.3.jar. Is this the correct >>version? Shouldn?t >> it be version 0.7? I am noticing that some of the runtime exceptions >>that I have >> received actually occur on line numbers in pge that do not match up to >>actual lines >> of code. So, I am thinking that I might have the wrong version of cas >>pge installed. >> >> I did pull a copy of the full oodt-0.7 from the repository into a >>different directory >> so that I could have a copy here locally for referencing the code. But >>I hadn?t >> actually built the whole thing yet. Is it still correct to simply >>execute ?mvn >> package?? I hope so because that?s what I did. I just wanted to check >>to see if >> cas-pge-0.7.jar gets created rather than cas-pge-0.3.jar. But the build >>didn?t finish >> because maven is telling me I have an OutOfMemoryError right in the >>middle of >> compiling the cas-pge!. Oh joy. So I had to build it in parts and I >>attached the >> output files. And yes, cas-pge-0.7.jar does get created, so I copied >>that to my >> oodt-deploy directory and I?m running with that now as well. >> >> Thanks! >> Valerie >> >> >> >> Valerie A. Mallder >> >> New Horizons Deputy Mission System Engineer The Johns Hopkins >> University/Applied Physics Laboratory >> 11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723 >> 240-228-7846 (Office) 410-504-2233 (Blackberry) >> >>
Re: Failed to build PgeConfig, exception in PathUtils.doDynamicReplacement
Awesome Val, glad it helped and that you are moving further!
Here¹s a quick example of how to call your perl script as an
extractor in CAS-PGE (barring yes, the duplication of specifying
a regExp to select the type to run the extractor on, but I¹ll get
to that later):
Under :
Your extern-config.xml file should work like this:
http://svn.apache.org/repos/asf/oodt/trunk/metadata/src/main/resources/exam
ples/extern-config.xml
Note that the ExternExtractor is documented here:
http://oodt.apache.org/components/maven/metadata/user/basic.html
That will get you your Perl script integrated as an extractor into the
system.
On to your question about selecting information from the File Manager:
Look up the SQL(.. portions of the DRAT PGE configs to get an idea of how
to do this. In short, yes you can do it, and the way to do it is to use
the special CAS-PGE SQL-like language that you can prefix your met key
with (make sure envReplace is turned on). You can do like SQL([options]){
SELECT FROM WHERE
The results of your above query are stored into the metadata field name
in your CAS-PGE config file.
I¹ll walk you through more of this later, but have to run to a meeting.
Gluck!
Cheers,
Chris
Chris Mattmann
chris.mattm...@gmail.com
-Original Message-
From: "Mallder, Valerie"
Reply-To:
Date: Tuesday, October 14, 2014 at 12:58 PM
To: "dev@oodt.apache.org"
Subject: RE: Failed to build PgeConfig, exception in
PathUtils.doDynamicReplacement
>Hi Chris,
>
>Thanks for your emails from Friday and Saturday. They were extremely
>helpful and I have now worked my way through all of my setup issues so
>far. I was using the new style CASE-PGE keys because Rishi used those in
>his working example and at one point I had been trying to mimic that
>example. I didn't know anything about the legacy mode flag. But, after
>changing the keys to the old style, everything started to come together a
>little faster.
>
>To start my crawler at the beginning of my pipeline, I chose the approach
>that Rishi had taken where the pgeConfig starts the crawler_launcher
>script directly rather than rely on the definition of the output files to
>trigger the crawler to run. This was because the met extractor I am
>using is a perl script, and your DRAT system only shows how to specify a
>class for the metFileWriterClass attribute. I could not find anything to
>show me how to specify a perl script as the met file writer instead.
>Also, having to specify the file types to crawl for by using a regular
>expression in the config file seemed like an unnecessary duplication of
>the definition of the files types to crawl for (since the regular
>expression just specifies a mime-type that is already defined in
>mime-types.xml). I didn't want to specify the mime-type in two different
>places.
>
>So, now I've been through the process of setting up my CAS-PGE to ingest
>the raw science and spacecraft housekeeping telemetry files, and run a
>postIngestSuccess action to copy some of those files to a new directory
>for further processing.
>
>The next step in my pipeline may be a little tricky. I need to query the
>filemgr and pass the results of the query (it could be a long list) as an
>input parameter to the first algorithm in the pipeline which, happens to
>be another perl script. Does anyone know how I might do this in a PGE
>config file?
>
>Thanks!!
>Val
>
>
>
>
>
>Valerie A. Mallder
>New Horizons Deputy Mission System Engineer
>Johns Hopkins University/Applied Physics Laboratory
>
>
>> -Original Message-
>> From: Mattmann, Chris A (3980) [mailto:chris.a.mattm...@jpl.nasa.gov]
>> Sent: Friday, October 10, 2014 7:59 PM
>> To: dev@oodt.apache.org
>> Subject: Re: Failed to build PgeConfig, exception in
>> PathUtils.doDynamicReplacement
>>
>> Thanks Val you are close!
>>
>> Looking at what you show below, I think the issue is one of the
>>following:
>>
>> 1. You are using the new style CAS-PGE Keys. Try using the old ones
>>(e.g., the
>> ones present here:
>>
>>http://svn.apache.org/repos/asf/oodt/trunk/pge/src/main/resources/example
>>s/
>> WorkflowTask/tasks.xml
>>
>> (note the ?_? and not the ?/? used in keys).
>> These keys are used by CAS-PGE if you see something in your wmgr bin
>>script (or
>> your resource manager batch_stub script) stating ?legacyMode=true?).
>> If you
>> are using RADIX, I believe that?s the case:
>>
>> http://s.apache.org/hO
>>
>> 2. Regarding better building environments. The m2e plugin for Eclipse
>>is fantastic
>> nowadays and can literally checkout a multi-module Maven project from
>&g
Re: Failed to build PgeConfig, exception in PathUtils.doDynamicReplacement
LOL
Chris Mattmann
chris.mattm...@gmail.com
-Original Message-
From: "Mallder, Valerie"
Reply-To:
Date: Tuesday, October 14, 2014 at 1:20 PM
To: "dev@oodt.apache.org"
Subject: RE: Failed to build PgeConfig, exception in
PathUtils.doDynamicReplacement
>Ok, Thanks! I will look at this more tomorrow while I'm sitting in a long
>boring meeting.
>
>
>Valerie A. Mallder
>New Horizons Deputy Mission System Engineer
>Johns Hopkins University/Applied Physics Laboratory
>
>
>> -Original Message-
>> From: Chris Mattmann [mailto:chris.mattm...@gmail.com]
>> Sent: Tuesday, October 14, 2014 4:11 PM
>> To: dev@oodt.apache.org
>> Subject: Re: Failed to build PgeConfig, exception in
>> PathUtils.doDynamicReplacement
>>
>> Awesome Val, glad it helped and that you are moving further!
>>
>> Here?s a quick example of how to call your perl script as an extractor
>>in CAS-PGE
>> (barring yes, the duplication of specifying a regExp to select the type
>>to run the
>> extractor on, but I?ll get to that later):
>>
>> Under :
>>
>> >
>>
>>metFileWriterClass="org.apache.oodt.cas.pge.writers.ExternalExtractorMetW
>>ri
>> ter"
>> args=?/path/to/your-extern-config.xml?>
>>
>>
>> Your extern-config.xml file should work like this:
>>
>>
>>http://svn.apache.org/repos/asf/oodt/trunk/metadata/src/main/resources/ex
>>am
>> ples/extern-config.xml
>>
>>
>> Note that the ExternExtractor is documented here:
>>
>> http://oodt.apache.org/components/maven/metadata/user/basic.html
>>
>>
>> That will get you your Perl script integrated as an extractor into the
>>system.
>>
>> On to your question about selecting information from the File Manager:
>>
>> Look up the SQL(.. portions of the DRAT PGE configs to get an idea of
>>how to do
>> this. In short, yes you can do it, and the way to do it is to use the
>>special CAS-
>> PGE SQL-like language that you can prefix your met key with (make sure
>> envReplace is turned on). You can do like SQL([options]){ SELECT FROM
>> WHERE
>>
>> The results of your above query are stored into the metadata field name
>>in your
>> CAS-PGE config file.
>>
>> I?ll walk you through more of this later, but have to run to a meeting.
>>
>> Gluck!
>>
>> Cheers,
>> Chris
>>
>>
>>
>> Chris Mattmann
>> chris.mattm...@gmail.com
>>
>>
>>
>>
>> -Original Message-
>> From: "Mallder, Valerie"
>> Reply-To:
>> Date: Tuesday, October 14, 2014 at 12:58 PM
>> To: "dev@oodt.apache.org"
>> Subject: RE: Failed to build PgeConfig, exception in
>> PathUtils.doDynamicReplacement
>>
>> >Hi Chris,
>> >
>> >Thanks for your emails from Friday and Saturday. They were extremely
>> >helpful and I have now worked my way through all of my setup issues so
>> >far. I was using the new style CASE-PGE keys because Rishi used those
>> >in his working example and at one point I had been trying to mimic that
>> >example. I didn't know anything about the legacy mode flag. But, after
>> >changing the keys to the old style, everything started to come together
>> >a little faster.
>> >
>> >To start my crawler at the beginning of my pipeline, I chose the
>> >approach that Rishi had taken where the pgeConfig starts the
>> >crawler_launcher script directly rather than rely on the definition of
>> >the output files to trigger the crawler to run. This was because the
>> >met extractor I am using is a perl script, and your DRAT system only
>> >shows how to specify a class for the metFileWriterClass attribute. I
>> >could not find anything to show me how to specify a perl script as the
>>met file
>> writer instead.
>> >Also, having to specify the file types to crawl for by using a regular
>> >expression in the config file seemed like an unnecessary duplication of
>> >the definition of the files types to crawl for (since the regular
>> >expression just specifies a mime-type that is already defined in
>> >mime-types.xml). I didn't want to specify the mime-type in two
>> >different places.
>> >
>> >So, now I've been through the process of setting up my CAS-PGE to
>> >ingest the raw science and spacecraft housekeeping telemetry files, and
>> >run a pos
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Have you guys considered using JCC [1] as a means to expose the workflow API currently in Java as a Python API? We are exploring its use in OODT, and we have already created a Tika [2] JCC-based python API. Cheers, Chris [1] http://lucene.apache.org/pylucene/jcc/ [2] http://github.com/chrismattmann/tika-python/ Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Supun Kamburugamuva Reply-To: Date: Thursday, October 16, 2014 at 3:43 PM To: dev Cc: "Alek Jones (Indiana)" , Suresh Marru , "architect...@airavata.apache.org" , "dev@oodt.apache.org" Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata. >Once we had an offline discussion about the Airavata Workflow language >(with Milinda, Saliya and Shameera). In that discussion one thing came out >was why we need to invent a different language when a simple library like >Python will full fill of Airavata requirements. > >There are many benefits in using a Python library as the API for >controlling Airavata workflows. > >1. It is a library, gives the ultimate control over the execution and it >can be simpler than any domain specific language that we can come with >like >XML, JSON etc >2. Most people use python and can learn it easily than any Domain specific >language >3. You can easily create a Python library for Airavata because all the >APIs >are thrift based. >4. If you design the constructs correctly you can plug an XBaya. > >Any thoughts? > >Thanks, >Supun.. > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva >wrote: > >> Hi Shameera, >> >> Why you prefer JavaScript over a language like Python? >> >> Thanks, >> Supun.. >> >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka < >> shameerai...@gmail.com> wrote: >> >>> Hi, >>> >>> First of all thanks everyone for giving valuable inputs. After doing >>>some >>> background search and talking to different people in the University >>>who has >>> used different workflow languages, I myself convinced that introducing >>>an >>> another workflow language is not what actually they need. By changing >>> exiting workflow language to another will not solve problems. What they >>> asking is a easy way to construct the workflows. Indirectly what they >>> asking for a sort of API which they can use to generate the workflows >>>and >>> run it. Correct me if i am wrong here. >>> >>> As most of above replies depict, if we can get a simple API, as an >>> example, for a web based application, JavaScript API would be a good >>> solution, and probably JSON would be a good candidate for language, >>>instead >>> of XML. >>> >>> Airavata community already have started to implement web base GUI. >>>Hence >>> introducing a JSON base JavaScript API would be great help. WDYT? >>> >>> Thanks, >>> Shameera. >>> >>> >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) < >>> alek...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> it is not dataflow instead focused on orchestrating REST services but >>>> you may find it useful datapoint - we created worfklow service that >>>>uses >>>> natively JavaScript and JSON to describe what happens during workflow >>>> execution: >>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002 >>>> >>>> HTH, >>>> >>>> Alek >>>> >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru >>>>wrote: >>>> >>>>> Hi Chris, >>>>> >>>>> Great to hear OODT community will be interested in adopting a JSON >>>>> based workflow language and potentially a web based composer as well. >>>>> Airavata previously had BPEL support initially through a home grown >>>>> implementation [1] by Alek Slominski and later through Apache ODE >>>>>[2]. Also >>>>> a white paper [3] by Alek on this topic is an interesting read. >>>>> >>>>> I am of the same opinion that we should adopt something more modern >>>>>as >>>>> the challenges from scientific workflows seems to be converging with >>>>>the >>>>> data flow patterns in business workflows. >>>>> >>>>> It will be great if we can all compile a list of potential candidates >>>>> and
Re: SQL parsing error in PgeConfig
Not stupid at all, an honest mistake that we need better error logging to inform users of. Thanks for sticking with it, Val. Cheers, Chris Chris Mattmann chris.mattm...@gmail.com -Original Message- From: "Mallder, Valerie" Reply-To: Date: Thursday, October 16, 2014 at 11:10 AM To: "dev@oodt.apache.org" Subject: RE: SQL parsing error in PgeConfig >I was not setting the property "PCS_FileManagerUrl" to the [FILEMGR_URL] >in the tasks.xml file. Instead, I was setting a property name that >doesn't exist called "PGETask_Query_FileManagerUrl" to [FILEMGR_URL] . > >This is what happens when you run into so many different types of >problems. You forget what you changed, and why you change it, and so you >don't go back and review all the changes to verify they are correct. I >thought I knew the correct name of the key, and so I didn't check it >against the code, and it was wrong. Stupid, stupid, stupid. > > >Valerie A. Mallder >New Horizons Deputy Mission System Engineer >Johns Hopkins University/Applied Physics Laboratory > > >> -Original Message- >> From: Tyler Palsulich [mailto:tpalsul...@gmail.com] >> Sent: Thursday, October 16, 2014 1:56 PM >> To: dev >> Subject: Re: SQL parsing error in PgeConfig >> >> > Found it! >> >> Woo! What was wrong? >> >> Tyler
Re: No data files are downloaded using Apache OODT Pushpull
Thank you Angela! Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Mengying Wang Reply-To: Date: Friday, October 17, 2014 at 11:06 AM To: , Chris Mattmann Subject: Re: No data files are downloaded using Apache OODT Pushpull >Dear Mattmann, > >Problem solved. I have made the following 2 changes: > >1: Change the active status of the daemon from ¹no¹ to yes¹. That is, > >[CAS_PP_HOME]/etc/examples/RemoteSpecsFiles/RemoteSpecs.xml > > > period="3m">epsilon="20s" runOnReboot="yes"/> > .. > > > >2: It is mainly caused by the protocol issues of the PushPull ftp plugins. >So I used the other PushPull ftp plugins. For the details please refer to >OODT Push Pull Plugins. > >Then everything is ok. Thank you for your help! > >Best, >Angela Wang
Re: Review Request 27162: Multiplexing Resource Manger Backend
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27162/#review58370 --- Ship it! Ship It! - Chris Mattmann On Oct. 24, 2014, 8:59 p.m., Michael Starch wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/27162/ > --- > > (Updated Oct. 24, 2014, 8:59 p.m.) > > > Review request for oodt and Chris Mattmann. > > > Repository: oodt > > > Description > --- > > Changes to add a Multiplexing Resource Manager Backend. Please see: > https://issues.apache.org/jira/browse/OODT-764 > > > Diffs > - > > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/mux/BackendManager.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/mux/BackendRepository.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/mux/BackendRepositoryFactory.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/mux/QueueMuxBatchManager.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/mux/QueueMuxBatchManagerFactory.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/mux/QueueMuxMonitor.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/mux/QueueMuxScheduler.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/mux/QueueMuxSchedulerFactory.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/mux/StandardBackendManager.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/mux/XmlBackendRepository.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/mux/XmlBackendRepositoryFactory.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/structs/exceptions/RepositoryException.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/util/GenericResourceManagerObjectFactory.java > 1634124 > trunk/resource/src/main/resources/examples/queue-to-backend-mapping.xml > PRE-CREATION > trunk/resource/src/main/resources/resource.properties 1634124 > > trunk/resource/src/test/org/apache/oodt/cas/resource/mux/TestQueueMuxBatchmgr.java > PRE-CREATION > > trunk/resource/src/test/org/apache/oodt/cas/resource/mux/TestQueueMuxMonitor.java > PRE-CREATION > > trunk/resource/src/test/org/apache/oodt/cas/resource/mux/mocks/MockBatchManager.java > PRE-CREATION > > trunk/resource/src/test/org/apache/oodt/cas/resource/mux/mocks/MockMonitor.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/27162/diff/ > > > Testing > --- > > Unit tests written. Basic functionality written. > > > Thanks, > > Michael Starch > >
Re: Review Request 27162: Multiplexing Resource Manger Backend
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27162/#review58371 --- Nice, Mike I like it. Clean, and doesn't stomp existing functionality. Next step - add some docs on the wiki. Thanks! - Chris Mattmann On Oct. 24, 2014, 8:59 p.m., Michael Starch wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/27162/ > --- > > (Updated Oct. 24, 2014, 8:59 p.m.) > > > Review request for oodt and Chris Mattmann. > > > Repository: oodt > > > Description > --- > > Changes to add a Multiplexing Resource Manager Backend. Please see: > https://issues.apache.org/jira/browse/OODT-764 > > > Diffs > - > > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/mux/BackendManager.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/mux/BackendRepository.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/mux/BackendRepositoryFactory.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/mux/QueueMuxBatchManager.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/mux/QueueMuxBatchManagerFactory.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/mux/QueueMuxMonitor.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/mux/QueueMuxScheduler.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/mux/QueueMuxSchedulerFactory.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/mux/StandardBackendManager.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/mux/XmlBackendRepository.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/mux/XmlBackendRepositoryFactory.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/structs/exceptions/RepositoryException.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/util/GenericResourceManagerObjectFactory.java > 1634124 > trunk/resource/src/main/resources/examples/queue-to-backend-mapping.xml > PRE-CREATION > trunk/resource/src/main/resources/resource.properties 1634124 > > trunk/resource/src/test/org/apache/oodt/cas/resource/mux/TestQueueMuxBatchmgr.java > PRE-CREATION > > trunk/resource/src/test/org/apache/oodt/cas/resource/mux/TestQueueMuxMonitor.java > PRE-CREATION > > trunk/resource/src/test/org/apache/oodt/cas/resource/mux/mocks/MockBatchManager.java > PRE-CREATION > > trunk/resource/src/test/org/apache/oodt/cas/resource/mux/mocks/MockMonitor.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/27162/diff/ > > > Testing > --- > > Unit tests written. Basic functionality written. > > > Thanks, > > Michael Starch > >
Re: Review Request 27172: Filemanager Changes for Streaming OODT Changes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27172/#review58499 --- Ship it! Ship It! - Chris Mattmann On Oct. 24, 2014, 10:32 p.m., Michael Starch wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/27172/ > --- > > (Updated Oct. 24, 2014, 10:32 p.m.) > > > Review request for oodt and Chris Mattmann. > > > Repository: oodt > > > Description > --- > > A sub-patch habdling the filemanager components of > https://reviews.apache.org/r/22791/ > > These changes will not affect other users with the small exception that there > is now a "guard clause" in setProductStructure. Thus if a user specifies an > invalid product structure, they will get an error. Anything that is broken > by this is effectively a bug. Three tests have been updated to remove bugs > descovered by this guard-clause. > > All other changes add functionality for the new STREAM structure. All > recommendations from: https://reviews.apache.org/r/22791/ have been fixed. > > These changes are a preparation step for SOODT. > > > Diffs > - > > > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java > 1634141 > > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java > 1634141 > > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java > 1634141 > > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java > 1634141 > > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java > 1634141 > > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java > 1634141 > > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java > 1634141 > > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java > 1634141 > > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java > 1634141 > > trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java > 1634141 > > trunk/webapp/fmprod/src/test/java/org/apache/oodt/cas/product/jaxrs/resources/ProductResourceTest.java > 1634141 > > trunk/webapp/fmprod/src/test/java/org/apache/oodt/cas/product/jaxrs/resources/TransferResourceTest.java > 1634141 > > trunk/webapp/fmprod/src/test/java/org/apache/oodt/cas/product/jaxrs/resources/TransfersResourceTest.java > 1634141 > > Diff: https://reviews.apache.org/r/27172/diff/ > > > Testing > --- > > Build and ran unit tests these pass. Testing was done to check the new > STREAM type, it works. > > > Thanks, > > Michael Starch > >
Re: Review Request 27179: Cluster Tools from Streaming OODT Changes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27179/#review58442 --- Ship it! Ship It! - Chris Mattmann On Oct. 24, 2014, 11:10 p.m., Michael Starch wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/27179/ > --- > > (Updated Oct. 24, 2014, 11:10 p.m.) > > > Review request for oodt and Chris Mattmann. > > > Repository: oodt > > > Description > --- > > Cluster tools from: https://reviews.apache.org/r/22791/ > > Apache headers added and all ready to go. These scripts are added to aid in > running an apache cluster. > > > Diffs > - > > trunk/cluster-tools/scripts/shutdown.sh PRE-CREATION > trunk/cluster-tools/scripts/start-up.sh PRE-CREATION > trunk/cluster-tools/scripts/start-up/mesos-master.bash PRE-CREATION > trunk/cluster-tools/scripts/start-up/mesos-slave.bash PRE-CREATION > trunk/cluster-tools/scripts/start-up/resource.bash PRE-CREATION > trunk/cluster-tools/scripts/utilites.sh PRE-CREATION > trunk/cluster-tools/setup/env-vars.sh.tmpl PRE-CREATION > trunk/cluster-tools/setup/hosts PRE-CREATION > trunk/cluster-tools/setup/install.sh PRE-CREATION > > Diff: https://reviews.apache.org/r/27179/diff/ > > > Testing > --- > > Scripts function, and do not interfere with anything else. > > > Thanks, > > Michael Starch > >
Re: Error occurs when using the query_tool script of the file manager for a new product type
Thanks for reporting this Angela. It would be great if you can file an issue for this in JIRA? Does this happen with the SQL Query only, or does it do so also with Lucene? Cheers, Chris Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Mengying Wang Reply-To: Date: Tuesday, October 28, 2014 at 12:10 AM To: Subject: Error occurs when using the query_tool script of the file manager for a new product type >Hi Everyone, > >According to the tutorial >http://oodt.apache.org/components/maven/curator/user/basic.html, I added a >new JSON product type to my file manager. However, if I run the query_tool >script before any json files ingested, it will raise some errors, as shown >here: > >$ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM >JSON" >Oct 27, 2014 11:52:07 PM org.apache.oodt.cas.filemgr.catalog.LuceneCatalog >paginateQuery >WARNING: Query: [q=] for Product Type: [urn:example:JSON] returned no >results >java.lang.NullPointerException >at >org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.complexQuery(XmlRpcFi >leManager.java:602) >at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) >at >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm >pl.java:43) >at java.lang.reflect.Method.invoke(Method.java:606) >at org.apache.xmlrpc.Invoker.execute(Invoker.java:130) >at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84) >at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146) >at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139) >at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125) >at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761) >at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642) >at java.lang.Thread.run(Thread.java:744) >org.apache.xmlrpc.XmlRpcException: java.lang.Exception: >org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to >perform complex query : null >at >org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClie >ntResponseProcessor.java:104) >at >org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClien >tResponseProcessor.java:71) >at >org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73) >at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194) >at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185) >at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178) >at >org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(Xm >lRpcFileManagerClient.java:990) >at >org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java >:252) >at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:242) >Exception in thread "main" >org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: >java.lang.Exception: >org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to >perform complex query : null >at >org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(Xm >lRpcFileManagerClient.java:996) >at >org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java >:252) >at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:242) > >By the way, if some files with the product type JSON are already ingested >in my file manager, then the query_tool script works very well. > >Thanks. >Mengying (Angela) Wang
Re: OODT delete on server
Thanks Etienne. Apache OODT by itself doesn¹t call any compression directly before sending the file, but we¹ve seen people develop e.g., Chunk servers, and also kick compression down into the file formats (e.g., since HDF and NetCDF have compression, folks will use them sometimes). Hope that helps and hope others can chime in too! Cheers, Chris Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Shakeh Khudikyan Date: Friday, October 31, 2014 at 10:49 AM To: Etienne Koen Cc: Chris Mattmann Subject: Re: OODT delete on server >Hi Etienne, > > >I'm not sure about compressing data. I've never had to do that. OODT >might have it, Chris? > > >-Shakeh >~~ >Shakeh E. Khudikyan >Software Engineer >Data Management Systems and Technologies >Instrument Software and Science Data Systems Section (398) > >NASA Jet Propulsion Laboratory >4800 Oak Grove Drive MS: 158-242 >Pasadena, CA 91109 > >Phone: 818/354-5897 >Email: shakeh.e.khudik...@jpl.nasa.gov > >~~ > > > > >From: Etienne Koen >Date: Wednesday, October 29, 2014 3:02 AM >To: Shakeh Khudikyan >Cc: Chris Mattmann >Subject: Re: OODT delete on server > > > >Hi Shakeh, > > >I can't seem to detect another query about OODT pushpull in my inbox, so >sorry if I am repeating myself... > > >Does OODT have the capability of compressing data before it is sent or >has other parties implemented this functionality successfully themselves? > > >Regards >Etienne > > >On 29 October 2014 09:55, Etienne Koen wrote: > >Hi Shakeh, > > >Do you know whether OODT has the functionality of deleting a product in a >similar way as a client would use pushpull to request a file being >downloaded from a remote server? > > >If so, what component of OODT would I use? > > >Regards >Etienne > > > > > > > >
Re: RADiX OPSUI Webapp Page Expired Error
Guys also note I opened an issue for this a while back: https://issues.apache.org/jira/browse/OODT-751 Chris Mattmann chris.mattm...@gmail.com -Original Message- From: "Verma, Rishi (398M)" Reply-To: Date: Friday, October 31, 2014 at 2:20 PM To: MengYing Wang Cc: "dev@oodt.apache.org" , Chris Mattmann Subject: Re: RADiX OPSUI Webapp Page Expired Error >Hi Mengying, > >The ³fix code² is still there [1]. However, it seems this particular >solution is not working anymore in 0.7. > >Could you try removing the fix and observing the results? This was a >temporary fix, and a more fundamental one may be necessary. Check out the >opsui code [2] to play around with potential solutions. Will look into it >as well. > >Thanks much, >Rishi > >[1] >http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apa >che/oodt/pcs/opsui/OPSUIWebPage.java >[2] >http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apa >che/oodt/pcs/opsui/ > >On Oct 30, 2014, at 3:23 PM, MengYing Wang >mailto:mengyingwa...@gmail.com>> wrote: > >Hey Rishi, > >I see you have fixed this error in the OODT 0.6 >https://issues.apache.org/jira/browse/OODT-596. However, currently I am >using the OODT 0.7 with RADiX, and keep receiving the error. Could you >tell me what should I do to fix it, or is there any patch file that I >could apply? Many thanks! > >-- >Best, >Mengying (Angela) Wang > >--- >Rishi Verma >NASA Jet Propulsion Laboratory >California Institute of Technology >4800 Oak Grove Drive, M/S 158-248 >Pasadena, CA 91109 >Tel: 1-818-393-5826 >
Re: Multiple RemoteSpecs files in pushpull components
Hey Lewis, I am not sure that the PushPull is single-threaded - check out FileRetrievalSystem - which I believe is multi-threaded - maybe that¹s why? CC @bfoster, can you help here? Cheers, Chris Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Lewis John Mcgibbney Reply-To: Date: Friday, October 31, 2014 at 10:16 PM To: "dev@oodt.apache.org" Subject: Multiple RemoteSpecs files in pushpull components >Hi Folks, >I am looking for some details on why we would require multiple RemoteSpecs >files within the pushpull component. >An example remote specs file can be found here [0] where essentially we >define individual pushpull client deamon's which do the heavily lifting >for >us in an automated manner. >The problem I am having here is that we reserve and initiate a LinkedList >data structure [1] for multiple RemoteSpecs.xml files and we then iterate >through this data structure [2] in an attempt to locate and process >configuration defined within each such file. >All of this is held in memory when we run one specific daemon as AFAIK the >pushpull component is a single threaded client. > >My questions are simple (and maybe trivial), do we require such a data >structure? Does anyone ever pass in multiple RemoteSpecs.xml files? > >Thank you >Lewis > >[0] >https://svn.apache.org/repos/asf/oodt/trunk/pushpull/src/main/resources/ex >amples/RemoteSpecsFiles/RemoteSpecs.xml >[1] >https://github.com/apache/oodt/blob/trunk/pushpull/src/main/java/org/apach >e/oodt/cas/pushpull/daemon/DaemonLauncher.java#L63 >[2] >https://github.com/apache/oodt/blob/trunk/pushpull/src/main/java/org/apach >e/oodt/cas/pushpull/daemon/DaemonLauncher.java#L102 > >-- >*Lewis*
Re: Why is pushpull not always working?
Thanks Lewis - in my mind it boils down somewhat to the protocols that are implemented and the way they behave with FTP servers and so forth. Ultimately, we should be using the ones that are in PushPull plugins, but they are virally licensed and I haven¹t found a better alternative. Another @bfoster update here would be great, Brian? Cheers, Chris Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Lewis John Mcgibbney Reply-To: Date: Saturday, November 1, 2014 at 12:56 AM To: "dev@oodt.apache.org" Subject: Why is pushpull not always working? >Hi Folks, >I really couldn't think of a better title for this thread so I apologize >in >advance. >I recently set up pushpull with FTP to crawl remote ftp site [0] which is >publicly available. >I am getting the following > >INFO: Launching Daemons . . . >Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.Daemon >registerRMIServer >INFO: Created Daemon ID = 90121 on RMI registry port 9012 >Nov 01, 2014 12:46:01 AM >org.apache.oodt.cas.pushpull.daemon.DaemonLauncher >launchDaemons >INFO: Creating Daemon with ID = 1 >Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager >setAsRunning >INFO: Daemon with ID = 90121 was given permission to run >Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep >until Sat Nov 01 00:48:00 PDT 2014 >Nov 01, 2014 12:48:00 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager >setAsRunning >INFO: Daemon with ID = 90121 was given permission to run >Nov 01, 2014 12:48:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep >until Sat Nov 01 00:51:00 PDT 2014 >Nov 01, 2014 12:50:59 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager >setAsRunning >INFO: Daemon with ID = 90121 was given permission to run >Nov 01, 2014 12:51:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep >until Sat Nov 01 00:54:00 PDT 2014 >Nov 01, 2014 12:53:59 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager >setAsRunning >INFO: Daemon with ID = 90121 was given permission to run >Nov 01, 2014 12:54:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep >until Sat Nov 01 00:57:00 PDT 2014 > >No errors... no pulling... no nothing! >I also came across this FAQ (thanks Cam/Chris) [1] leading to Chris' >plugins page [2]. >My question is simple... why is pushpull known NOT to work on occasion? I >am going to debug this tomorrow but I thought I would ask this >none-the-less. >Thanks >Lewis > >[0] ftp://n5eil01u.ecs.nsidc.org/SAN/AQUARIUS/AQ3_SSS.004/ >[1] >https://cwiki.apache.org/confluence/display/OODT/OODT+Push-Pull+User+Guide >#OODTPush-PullUserGuide-Nodatafileisdownloadedtomystagingdirectoryafterrun >ningthe./pushpullscript.WhatshouldIdo >? >[2] >https://cwiki.apache.org/confluence/display/OODT/OODT+Push+Pull+Plugins > > >-- >*Lewis*
Re: Extra Compiler Tools
Hi Mike, Great discussion. It would be nice if there was a protoc Maven plugin: http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html Looks like there is. My suggestion: 1. use a Profile, something like -Pwith-mesos and then when activated; 2. call the above plugin if -Pwith-mesos is activated in the resource manager Sound good? Cheers, Chris Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Michael Starch Reply-To: Date: Wednesday, November 5, 2014 at 11:46 AM To: Subject: Extra Compiler Tools >All, > >I am trying to integrate apache-mesos with our resource manager. However, >mesos uses a technology called "protobuff" from Google for >marshaling/unmarshaling data. > >This requires running a tool called "protoc" to generate a source file in >java. What is the best way to integrate this step into our build process? > >Options I can conceive of: > -Check in generated java file > -Require "protoc" installation to build resource manager > -Separate extra resource package into new module > >None of these ideas are very clean. > >Any other ideas? I tried setting up a profile to only compile these >sources when selected, but that turned out not to work. > >-Michael Starch
Re: Extra Compiler Tools
Hi Mike, Thanks for flushing this out. My thoughts on the below: -Original Message- From: Michael Starch Reply-To: Date: Wednesday, November 5, 2014 at 12:12 PM To: Subject: Re: Extra Compiler Tools >I tried this approach. The plugin requires a path to the "protoc" tool and >thus a working installation. This is what prompted the discussion. Ah - no worries, what you could do is: 1. only enable to plugin if -Pwith-mesos is enabled; and > >Running the plugin under a profile works. Yep. > However, not running the plugin >causes compile errors in dependant code. Excluding this code except >within >the profile doesn't seem to work, and is considered by some to be bad form >because there is nothing inside the jar file that notes which profiles >were >used to compile. Got it. Suggestion here would be: 2. create a new module, cas-resource-mesos, and inside of that module, take one of the following approaches, assuming the module is activated when -Pwith-mesos is enabled: 2a. Maven Antrun like so (in this old example): http://stackoverflow.com/questions/1578456/integrate-protocol-buffers-into- maven2-build (pro: more flexibility in case protoc isn¹t there; to fail on error; to only compile if protoc is available 2b. Maven protobuf plugin http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html Here¹s how to enable a module with a profile: http://blog.soebes.de/blog/2013/11/09/why-is-it-bad-to-activate-slash-deact ive-modules-by-profiles-in-maven/ It seems like that is a bad idea though, based on that discussion. So, here¹s another option: 1. Inside of cas-resource (no special new module or anything else) 2. include some custom Ant magic via a build.xml file and the Maven AntRun plugin: 2a. test if protoc is on the system path, and if not, download it, e.g., into the target directory (gets deleted on clean) 2b. call protoc and compile after 2a I would suggest this solution as I think it¹s the most robust and ensures we always have a cas-resource that includes mesos and compiled correctly. Cheers, Chris > >Any ideas on how to continue? > >Michael > On Nov 5, 2014 11:04 AM, "Chris Mattmann" >wrote: > >> Hi Mike, >> >> Great discussion. It would be nice if there was >> a protoc Maven plugin: >> >> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html >> >> >> Looks like there is. My suggestion: >> >> 1. use a Profile, something like -Pwith-mesos and >> then when activated; >> 2. call the above plugin if -Pwith-mesos is activated >> in the resource manager >> >> Sound good? >> >> Cheers, >> Chris >> >> >> Chris Mattmann >> chris.mattm...@gmail.com >> >> >> >> >> -Original Message- >> From: Michael Starch >> Reply-To: >> Date: Wednesday, November 5, 2014 at 11:46 AM >> To: >> Subject: Extra Compiler Tools >> >> >All, >> > >> >I am trying to integrate apache-mesos with our resource manager. >>However, >> >mesos uses a technology called "protobuff" from Google for >> >marshaling/unmarshaling data. >> > >> >This requires running a tool called "protoc" to generate a source file >>in >> >java. What is the best way to integrate this step into our build >>process? >> > >> >Options I can conceive of: >> > -Check in generated java file >> > -Require "protoc" installation to build resource manager >> > -Separate extra resource package into new module >> > >> >None of these ideas are very clean. >> > >> >Any other ideas? I tried setting up a profile to only compile these >> >sources when selected, but that turned out not to work. >> > >> >-Michael Starch >> >> >>
Re: Extra Compiler Tools
Thanks Mike +1 Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Michael Starch Reply-To: Date: Wednesday, November 5, 2014 at 12:31 PM To: Subject: Re: Extra Compiler Tools >Looks like you followed the same reasoning chain that I did. Yes, I came >to the same conclusion that ant-build was best. > >I wasn't sure how to download protoc, but you just answered thatso I >think this is a great solution! > >Thanks, > >Michael > > >On Wed, Nov 5, 2014 at 10:23 AM, Chris Mattmann >wrote: > >> Hi Mike, >> >> Thanks for flushing this out. >> >> My thoughts on the below: >> >> >> -Original Message- >> From: Michael Starch >> Reply-To: >> Date: Wednesday, November 5, 2014 at 12:12 PM >> To: >> Subject: Re: Extra Compiler Tools >> >> >I tried this approach. The plugin requires a path to the "protoc" tool >>and >> >thus a working installation. This is what prompted the discussion. >> >> Ah - no worries, what you could do is: >> >> 1. only enable to plugin if -Pwith-mesos is enabled; and >> >> > >> >Running the plugin under a profile works. >> >> Yep. >> >> > However, not running the plugin >> >causes compile errors in dependant code. Excluding this code except >> >within >> >the profile doesn't seem to work, and is considered by some to be bad >>form >> >because there is nothing inside the jar file that notes which profiles >> >were >> >used to compile. >> >> Got it. Suggestion here would be: >> >> 2. create a new module, cas-resource-mesos, and inside of that module, >> take one of the following approaches, assuming the module is activated >> when -Pwith-mesos is enabled: >> >> 2a. Maven Antrun like so (in this old example): >> >>http://stackoverflow.com/questions/1578456/integrate-protocol-buffers-int >>o- >> maven2-build >> >> (pro: more flexibility in case protoc isn¹t there; to fail on error; to >> only compile if >> protoc is available >> >> 2b. Maven protobuf plugin >> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html >> >> Here¹s how to enable a module with a profile: >> >> >>http://blog.soebes.de/blog/2013/11/09/why-is-it-bad-to-activate-slash-dea >>ct >> ive-modules-by-profiles-in-maven/ >> >> >> It seems like that is a bad idea though, based on that discussion. >> >> So, here¹s another option: >> >> 1. Inside of cas-resource (no special new module or anything else) >> 2. include some custom Ant magic via a build.xml file and the Maven >> AntRun plugin: >> 2a. test if protoc is on the system path, and if not, download it, >>e.g., >> into the target directory (gets deleted on clean) >> 2b. call protoc and compile after 2a >> >> I would suggest this solution as I think it¹s the most robust and >>ensures >> we always have a cas-resource that includes mesos and compiled >>correctly. >> >> Cheers, >> Chris >> >> > >> >Any ideas on how to continue? >> > >> >Michael >> > On Nov 5, 2014 11:04 AM, "Chris Mattmann" >> >wrote: >> > >> >> Hi Mike, >> >> >> >> Great discussion. It would be nice if there was >> >> a protoc Maven plugin: >> >> >> >> http://sergei-ivanov.github.io/maven-protoc-plugin/usage.html >> >> >> >> >> >> Looks like there is. My suggestion: >> >> >> >> 1. use a Profile, something like -Pwith-mesos and >> >> then when activated; >> >> 2. call the above plugin if -Pwith-mesos is activated >> >> in the resource manager >> >> >> >> Sound good? >> >> >> >> Cheers, >> >> Chris >> >> >> >> >> >> Chris Mattmann >> >> chris.mattm...@gmail.com >> >> >> >> >> >> >> >> >> >> -Original Message- >> >> From: Michael Starch >> >> Reply-To: >> >> Date: Wednesday, November 5, 2014 at 11:46 AM >> >> To: >> >> Subject: Extra Compiler Tools >> >> >> >> >All, >> >> > >> >> >I am trying to integrate apache-mesos with our resource manager. >> >>However, >> >> >mesos uses a technology called "protobuff" from Google for >> >> >marshaling/unmarshaling data. >> >> > >> >> >This requires running a tool called "protoc" to generate a source >>file >> >>in >> >> >java. What is the best way to integrate this step into our build >> >>process? >> >> > >> >> >Options I can conceive of: >> >> > -Check in generated java file >> >> > -Require "protoc" installation to build resource manager >> >> > -Separate extra resource package into new module >> >> > >> >> >None of these ideas are very clean. >> >> > >> >> >Any other ideas? I tried setting up a profile to only compile these >> >> >sources when selected, but that turned out not to work. >> >> > >> >> >-Michael Starch >> >> >> >> >> >> >> >> >>
