Can spell, try again: https://cwiki.apache.org/confluence/display/OODT/File+Manager+Scale+Out+Planning
On Thu, Mar 17, 2016 at 10:56 PM, Tom Barber <[email protected]> wrote: > Hi guys > > I've started doing some planning about support for multiple filemanagers > and scale out stuff, here: > https://cwiki.apache.org/confluence/display/OODT/File+Manage+Scale+Out+Planning > > Work in progress and Chris you understand the underlying infrastructure > far better than me, so let me know if I'm thinking about it the wrong way > or going off piste. > > Val, add in any specific requirements you have to the doc and we'll look > at addressing them. > > Cheers > > Tom > > On Thu, Mar 17, 2016 at 4:01 PM, Tom Barber <[email protected]> > wrote: > >> Okay clearly this is a requirement not just something I made up because I >> clearly don't have enough work and could do with spending more time on >> unpaid stuff! ;) >> >> I'll draft a plan in the wiki to try and formulate the best way for cross >> FM comms and let people critique it before diving in head first. >> >> Tom >> >> On Thu, Mar 17, 2016 at 1:03 AM, Mallder, Valerie < >> [email protected]> wrote: >> >>> Hi Tom, >>> >>> When you figure out how how to do it could you document it, or post a >>> complete example to the mailing list so that the info can be found with a >>> search of the email archive? >>> >>> Val >>> >>> >>> >>> Sent with Good (www.good.com) >>> ________________________________ >>> From: Tom Barber <[email protected]> >>> Sent: Wednesday, March 16, 2016 7:54:51 PM >>> To: [email protected] >>> Subject: Re: Multiple File Managers >>> >>> Yeah I was wondering if there was some catalog sharing going on. Okay, >>> i'll >>> have a play over the next few days, see what ideas I can come up with. >>> >>> Ta! >>> >>> On Wed, Mar 16, 2016 at 11:50 PM, Chris Mattmann < >>> [email protected]> >>> wrote: >>> >>> > yeah not as simple as a config file, but doable via PCS >>> > as a base. I always wanted to use PCS to do multi-horizontal >>> > scale out. Right now people just put up multiple FMs and WMs >>> > and then write a script to coordinate them. Or rely on the fact >>> > that they write to e.g., the same index, or same file system or >>> > etc to deal with that. >>> > >>> > — >>> > Chris Mattmann >>> > [email protected] >>> > >>> > >>> > >>> > >>> > >>> > >>> > -----Original Message----- >>> > From: Tom Barber <[email protected]> >>> > Reply-To: <[email protected]> >>> > Date: Wednesday, March 16, 2016 at 4:42 PM >>> > To: "[email protected]" <[email protected]> >>> > Subject: Re: Multiple File Managers >>> > >>> > >Okay, I'm missing something here, if you have FM on Site A, FM on >>> Site B, >>> > >and PCS to unify them..... >>> > > >>> > >I look at ops ui and fm client configs and only see entries for single >>> > >services, if I want to run filemgr-client on my local file manager, >>> but in >>> > >reality also search filemanager on Site B at the same time as if it >>> were a >>> > >single unit, how would that occur? If they are on separate networks >>> they >>> > >can't broadcast, but I also can't find a way to set that type of >>> stuff up >>> > >in a config file. >>> > > >>> > >Ta >>> > > >>> > >Tom >>> > > >>> > >On Wed, Mar 16, 2016 at 10:43 PM, Tom Barber <[email protected] >>> > >>> > >wrote: >>> > > >>> > >> I knew I was over googling it! >>> > >> >>> > >> Thanks >>> > >> >>> > >> On Wed, Mar 16, 2016 at 10:33 PM, Mattmann, Chris A (3980) < >>> > >> [email protected]> wrote: >>> > >> >>> > >>> multiple file managers is easy ;) >>> > >>> >>> > >>> just create them - the unifying framework is PCS when you do things >>> > >>> like that. >>> > >>> >>> > >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> > >>> Chris Mattmann, Ph.D. >>> > >>> Chief Architect >>> > >>> Instrument Software and Science Data Systems Section (398) >>> > >>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>> > >>> Office: 168-519, Mailstop: 168-527 >>> > >>> Email: [email protected] >>> > >>> WWW: http://sunset.usc.edu/~mattmann/ >>> > >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> > >>> Director, Information Retrieval and Data Science Group (IRDS) >>> > >>> Adjunct Associate Professor, Computer Science Department >>> > >>> University of Southern California, Los Angeles, CA 90089 USA >>> > >>> WWW: http://irds.usc.edu/ >>> > >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> > >>> >>> > >>> >>> > >>> >>> > >>> >>> > >>> >>> > >>> -----Original Message----- >>> > >>> From: Tom Barber <[email protected]> >>> > >>> Reply-To: "[email protected]" <[email protected]> >>> > >>> Date: Wednesday, March 16, 2016 at 2:57 PM >>> > >>> To: "[email protected]" <[email protected]> >>> > >>> Subject: Multiple File Managers >>> > >>> >>> > >>> >Here's something I assume is possible but can't find on the wiki. >>> > >>> > >>> > >>> >I know for workflow stuff you add resources to the pool, >>> configure a >>> > >>> >workflow manager and scale up. >>> > >>> > >>> > >>> >If I want to actually do the "distributed" stuff though, say >>> multiple >>> > >>> file >>> > >>> >managers, is this where the Grid components come in? >>> > >>> > >>> > >>> >What I'm looking for is a setup of multiple file managers in >>> different >>> > >>> >locations but I'd like Joe User to query the whole lot like it was >>> > >>>one. >>> > >>> >But >>> > >>> >coming from single node Radix land there isn't much consumer info >>> on >>> > >>>the >>> > >>> >scaling out. >>> > >>> > >>> > >>> >Cheers >>> > >>> > >>> > >>> >Tom >>> > >>> >>> > >>> >>> > >> >>> > >>> > >>> > >>> >> >> >
