Hi Supun, 1) Do you have a test file that you can share and I will try? 2) How long is it taking? 3) Can you make your log available with DEBUG mode turned on so I can see what is going on? Thanks, Lewis
On Mon, Oct 26, 2015 at 11:52 AM, Supun Nakandala <supun.nakand...@gmail.com > wrote: > OODT Devs, > > I'm experiencing unacceptable slowness when ingesting some what large > files(>10MB) using OODT file manager. When I check the OODT server side > logs I see multiple attempts to overwrite the same input file again and > again and failing. I am > using org.apache.oodt.cas.filemgr.ingest.StdIngester. > > Any clue on what I am doing wrong here? > > Thanks > > On Mon, Oct 12, 2015 at 4:58 PM, Chris Mattmann <chris.mattm...@gmail.com> > wrote: > > > Yep it can do either. > > > > 1. Use Product.STRUCTURE_HIERARCHICAL for directories. > > This feature works, and also with data movement, but it’s preferred > > to do option 2. > > 2. Manage the hierarchy simply with Product.STRUCTURE_FLAT products, > > but with versioning to place and co-locate the files according to the > > organized hierarchy, and then link them with metadata keys. > > > > Make sense? > > > > Cheers, > > Chris > > > > — > > Chris Mattmann > > chris.mattm...@gmail.com > > > > > > > > > > > > > > -----Original Message----- > > From: Suresh Marru <sma...@apache.org> > > Reply-To: <dev@oodt.apache.org> > > Date: Monday, October 12, 2015 at 1:47 PM > > To: <d...@airavata.apache.org>, <dev@oodt.apache.org> > > Subject: Re: OODT File Manager - Airavata Integration Document > > > > >Thanks Chris this is helpful to get a clarification. The only reason we > > >want to do at a service level so we can have more fine grain control > based > > >on users/groups and their roles. We will start by looking at the tomcat > > >layer and will ask more questions along the way. > > > > > >Paraphrasing Supun: Seems like OODT currently supports files organized > in > > >a > > >flat structure. Any suggestions on how to support data products > organized > > >in hierarchal directories. As an example, say an Experiment has a > Workflow > > >which runs bunch of simulations all organized as inputs and outputs > under > > >a > > >parent experiment. Can we point the file manager to such a organization > or > > >is it expected that the hierarchy be managed outside (through metadata > key > > >value pairs or otherwise)? > > > > > >Thanks, > > >Suresh > > > > > >On Mon, Oct 12, 2015 at 4:32 PM Chris Mattmann < > chris.mattm...@gmail.com> > > >wrote: > > > > > >> Hey Suresh, > > >> > > >> Sure. Right now I would recommend not trying to do auth/auth > > >> at the File Manager Level, and instead using e.g., Tomcat or > > >> something to set up like an OAuth realm, and doing it that way. > > >> Is there a specific need to lock it down at the service level? > > >> At their core they are XML-RPC, and I think it will be more difficult > > >> to implement there than trying to do in an app server that sits > > >> in front. > > >> > > >> Cheers, > > >> Chris > > >> > > >> — > > >> Chris Mattmann > > >> chris.mattm...@gmail.com > > >> > > >> > > >> > > >> > > >> > > >> > > >> -----Original Message----- > > >> From: Suresh Marru <sma...@apache.org> > > >> Reply-To: <dev@oodt.apache.org> > > >> Date: Monday, October 12, 2015 at 1:09 PM > > >> To: <d...@airavata.apache.org>, <dev@oodt.apache.org> > > >> Subject: Re: OODT File Manager - Airavata Integration Document > > >> > > >> >OODT Dev's, > > >> > > > >> >Can we plugin custom authentication to CAS File Manager? I see that > you > > >> >could potentially use the OODT single sign on package and have it > talk > > >>to > > >> >a > > >> >LDAP but wondering if we could do OAuth2 like authentication and > > >> >authorization. > > >> > > > >> >Thanks, > > >> >Suresh > > >> > > > >> >On Tue, Sep 1, 2015 at 12:03 AM DImuthu Upeksha > > >> ><dimuthu.upeks...@gmail.com> > > >> >wrote: > > >> > > > >> >> Hi Chris, > > >> >> > > >> >> Thank you for your feedback. This is a POC level implementation > done > > >>by > > >> >>me > > >> >> and there is a lot of room for improvement. It would be great if > both > > >> >> Airavata and OODT teams can go through the design and decide > whether > > >> >> Airavata can be integrated with OODT File manager. But my idea is, > > >>yes > > >> >>it > > >> >> is possible. > > >> >> > > >> >> Thanks, > > >> >> Dimuthu > > >> >> > > >> >> On Tue, Sep 1, 2015 at 5:07 AM, Mattmann, Chris A (3980) < > > >> >> chris.a.mattm...@jpl.nasa.gov> wrote: > > >> >> > > >> >>> Great work Dimuthu! > > >> >>> > > >> >>> So this means now Airavata and OODT’s File Manager are integrated? > > >> >>> :=) > > >> >>> > > >> >>> CC/dev@oodt.a.o > > >> >>> > > >> >>> Cheers, > > >> >>> Chris > > >> >>> > > >> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > >> >>> Chris Mattmann, Ph.D. > > >> >>> Chief Architect > > >> >>> Instrument Software and Science Data Systems Section (398) > > >> >>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA > > >> >>> Office: 168-519, Mailstop: 168-527 > > >> >>> Email: chris.a.mattm...@nasa.gov > > >> >>> WWW: http://sunset.usc.edu/~mattmann/ > > >> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > >> >>> Adjunct Associate Professor, Computer Science Department > > >> >>> University of Southern California, Los Angeles, CA 90089 USA > > >> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > >> >>> > > >> >>> > > >> >>> > > >> >>> > > >> >>> > > >> >>> -----Original Message----- > > >> >>> From: DImuthu Upeksha <dimuthu.upeks...@gmail.com> > > >> >>> Reply-To: "d...@airavata.apache.org" <d...@airavata.apache.org> > > >> >>> Date: Saturday, August 29, 2015 at 1:27 AM > > >> >>> To: "d...@airavata.apache.org" <d...@airavata.apache.org> > > >> >>> Subject: OODT File Manager - Airavata Integration Document > > >> >>> > > >> >>> >Hi all, > > >> >>> > > > >> >>> > > > >> >>> >Please find the document [1] that describes the steps followed in > > >>the > > >> >>> >attempt of integrating OODT File Manager and AIravata. This setup > > >>is > > >> >>> >currently working on gw77. > > >> >>> > > > >> >>> > > > >> >>> >[1] > > >> >>> > > > >> >>> > > >> >>> > > >> > > https://docs.google.com/document/d/1mr3k_OmgEweeVhJtuMTvofL__v9uPwyWbWrP > > >> >>>-M > > >> >>> >OFUDw/edit?usp=sharing > > >> >>> > > > >> >>> > > > >> >>> >Thanks > > >> >>> >Dimuthu > > >> >>> > > > >> >>> > > > >> >>> >-- > > >> >>> >Regards > > >> >>> >W.Dimuthu Upeksha > > >> >>> >Undergraduate > > >> >>> >Department of Computer Science And Engineering > > >> >>> >University of Moratuwa, Sri Lanka > > >> >>> > > > >> >>> > > > >> >>> > > > >> >>> > > >> >>> > > >> >> > > >> >> > > >> >> -- > > >> >> Regards > > >> >> > > >> >> W.Dimuthu Upeksha > > >> >> Undergraduate > > >> >> Department of Computer Science And Engineering > > >> >> > > >> >> University of Moratuwa, Sri Lanka > > >> >> > > >> > > >> > > >> > > > > > > > > > -- > Thank you > Supun Nakandala > Dept. Computer Science and Engineering > University of Moratuwa > -- *Lewis*
