No problem Etienne, and yes that's a fair statement! :) Cheers man.
Chris ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: [email protected] WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -----Original Message----- From: Etienne Koen <[email protected]> Date: Wednesday, September 10, 2014 9:37 AM To: Chris Mattmann <[email protected]> Cc: Shakeh Khudikyan <[email protected]>, "[email protected]" <[email protected]> Subject: Re: filemgr ingestion >Thanks for clearing that out Chris, >I think that the capability of the filemgr should be considered as a test >on its own and does not actually fall into the category which we are >currently benchmarking for the baseline WAN/LAN performance of the tools. > >There should be a separate test section for archiving... >Thanks for the help! >Etienne >On Sep 10, 2014 6:15 PM, "Chris Mattmann" <[email protected]> wrote: > >Hi Etienne, > >Thanks for your query. The default protocol used by the >RemoteDataTransfer is XML-RPC. It's a call-back mechanism >that I baked up myself with a configurable chunk size parameter. > >My suggestion is rather than use that DataTransfer, I would leverage: > >(1) a distributed filesystem like HadoopFS, Tachyon (from BDAS), >GlusterFS, >or even NFS that logically mounts a set of local shared nothing disks as a >virtual global mount > >(2) use the LocalDataTransfer in the File Manager, with #1 in place > >The main reason for this is that it allows the people who work on >distributed >and reliable filesystems that solve those problems in a great way, it >allows >us in the OODT community to take advantage of that work without having to >write >all the complex functionality ourselves. That way we just have >light-weight insulated >pieces/components in OODT that rely on the workhorses that do things >right. > >HTH! > >Cheers, >Chris > >------------------------ >Chris Mattmann >[email protected] > > > > >-----Original Message----- >From: Etienne Koen <[email protected]> >Date: Wednesday, September 10, 2014 1:57 AM >To: Shakeh Khudikyan <[email protected]> >Cc: Chris Mattmann <[email protected]> >Subject: filemgr ingestion > >>Hi Shakeh, >>What is the default protocol that filemgr uses for RemoteDataTransfer >>ingestion? >> >>Also, is the a way to specify the protocol? >> >>Cheers >>Etienne >> > > > > >
