OODT Devs, I'm experiencing unacceptable slowness when ingesting some what large files(>10MB) using OODT file manager. When I check the OODT server side logs I see multiple attempts to overwrite the same input file again and again and failing. I am using org.apache.oodt.cas.filemgr.ingest.StdIngester.
Any clue on what I am doing wrong here? Thanks On Mon, Oct 12, 2015 at 4:58 PM, Chris Mattmann <chris.mattm...@gmail.com> wrote: > Yep it can do either. > > 1. Use Product.STRUCTURE_HIERARCHICAL for directories. > This feature works, and also with data movement, but it’s preferred > to do option 2. > 2. Manage the hierarchy simply with Product.STRUCTURE_FLAT products, > but with versioning to place and co-locate the files according to the > organized hierarchy, and then link them with metadata keys. > > Make sense? > > Cheers, > Chris > > — > Chris Mattmann > chris.mattm...@gmail.com > > > > > > > -----Original Message----- > From: Suresh Marru <sma...@apache.org> > Reply-To: <dev@oodt.apache.org> > Date: Monday, October 12, 2015 at 1:47 PM > To: <d...@airavata.apache.org>, <dev@oodt.apache.org> > Subject: Re: OODT File Manager - Airavata Integration Document > > >Thanks Chris this is helpful to get a clarification. The only reason we > >want to do at a service level so we can have more fine grain control based > >on users/groups and their roles. We will start by looking at the tomcat > >layer and will ask more questions along the way. > > > >Paraphrasing Supun: Seems like OODT currently supports files organized in > >a > >flat structure. Any suggestions on how to support data products organized > >in hierarchal directories. As an example, say an Experiment has a Workflow > >which runs bunch of simulations all organized as inputs and outputs under > >a > >parent experiment. Can we point the file manager to such a organization or > >is it expected that the hierarchy be managed outside (through metadata key > >value pairs or otherwise)? > > > >Thanks, > >Suresh > > > >On Mon, Oct 12, 2015 at 4:32 PM Chris Mattmann <chris.mattm...@gmail.com> > >wrote: > > > >> Hey Suresh, > >> > >> Sure. Right now I would recommend not trying to do auth/auth > >> at the File Manager Level, and instead using e.g., Tomcat or > >> something to set up like an OAuth realm, and doing it that way. > >> Is there a specific need to lock it down at the service level? > >> At their core they are XML-RPC, and I think it will be more difficult > >> to implement there than trying to do in an app server that sits > >> in front. > >> > >> Cheers, > >> Chris > >> > >> — > >> Chris Mattmann > >> chris.mattm...@gmail.com > >> > >> > >> > >> > >> > >> > >> -----Original Message----- > >> From: Suresh Marru <sma...@apache.org> > >> Reply-To: <dev@oodt.apache.org> > >> Date: Monday, October 12, 2015 at 1:09 PM > >> To: <d...@airavata.apache.org>, <dev@oodt.apache.org> > >> Subject: Re: OODT File Manager - Airavata Integration Document > >> > >> >OODT Dev's, > >> > > >> >Can we plugin custom authentication to CAS File Manager? I see that you > >> >could potentially use the OODT single sign on package and have it talk > >>to > >> >a > >> >LDAP but wondering if we could do OAuth2 like authentication and > >> >authorization. > >> > > >> >Thanks, > >> >Suresh > >> > > >> >On Tue, Sep 1, 2015 at 12:03 AM DImuthu Upeksha > >> ><dimuthu.upeks...@gmail.com> > >> >wrote: > >> > > >> >> Hi Chris, > >> >> > >> >> Thank you for your feedback. This is a POC level implementation done > >>by > >> >>me > >> >> and there is a lot of room for improvement. It would be great if both > >> >> Airavata and OODT teams can go through the design and decide whether > >> >> Airavata can be integrated with OODT File manager. But my idea is, > >>yes > >> >>it > >> >> is possible. > >> >> > >> >> Thanks, > >> >> Dimuthu > >> >> > >> >> On Tue, Sep 1, 2015 at 5:07 AM, Mattmann, Chris A (3980) < > >> >> chris.a.mattm...@jpl.nasa.gov> wrote: > >> >> > >> >>> Great work Dimuthu! > >> >>> > >> >>> So this means now Airavata and OODT’s File Manager are integrated? > >> >>> :=) > >> >>> > >> >>> CC/dev@oodt.a.o > >> >>> > >> >>> Cheers, > >> >>> Chris > >> >>> > >> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >> >>> Chris Mattmann, Ph.D. > >> >>> Chief Architect > >> >>> Instrument Software and Science Data Systems Section (398) > >> >>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA > >> >>> Office: 168-519, Mailstop: 168-527 > >> >>> Email: chris.a.mattm...@nasa.gov > >> >>> WWW: http://sunset.usc.edu/~mattmann/ > >> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >> >>> Adjunct Associate Professor, Computer Science Department > >> >>> University of Southern California, Los Angeles, CA 90089 USA > >> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> -----Original Message----- > >> >>> From: DImuthu Upeksha <dimuthu.upeks...@gmail.com> > >> >>> Reply-To: "d...@airavata.apache.org" <d...@airavata.apache.org> > >> >>> Date: Saturday, August 29, 2015 at 1:27 AM > >> >>> To: "d...@airavata.apache.org" <d...@airavata.apache.org> > >> >>> Subject: OODT File Manager - Airavata Integration Document > >> >>> > >> >>> >Hi all, > >> >>> > > >> >>> > > >> >>> >Please find the document [1] that describes the steps followed in > >>the > >> >>> >attempt of integrating OODT File Manager and AIravata. This setup > >>is > >> >>> >currently working on gw77. > >> >>> > > >> >>> > > >> >>> >[1] > >> >>> > > >> >>> > >> >>> > >> > https://docs.google.com/document/d/1mr3k_OmgEweeVhJtuMTvofL__v9uPwyWbWrP > >> >>>-M > >> >>> >OFUDw/edit?usp=sharing > >> >>> > > >> >>> > > >> >>> >Thanks > >> >>> >Dimuthu > >> >>> > > >> >>> > > >> >>> >-- > >> >>> >Regards > >> >>> >W.Dimuthu Upeksha > >> >>> >Undergraduate > >> >>> >Department of Computer Science And Engineering > >> >>> >University of Moratuwa, Sri Lanka > >> >>> > > >> >>> > > >> >>> > > >> >>> > >> >>> > >> >> > >> >> > >> >> -- > >> >> Regards > >> >> > >> >> W.Dimuthu Upeksha > >> >> Undergraduate > >> >> Department of Computer Science And Engineering > >> >> > >> >> University of Moratuwa, Sri Lanka > >> >> > >> > >> > >> > > > -- Thank you Supun Nakandala Dept. Computer Science and Engineering University of Moratuwa
