Yeah that had crossed my mind as well, be nice to improve the performance as well though. I'll mull it over a bit.
On Fri, Oct 21, 2016 at 9:04 PM, Chris Mattmann <mattm...@apache.org> wrote: > Another option: > > 1. Move the files to where they should be in controlled access storage > 2. Use InPlaceDataTransfer (no move, copy, anything, just cataloging!) > > > > > On 10/21/16, 1:01 PM, "Tom Barber" <tom.bar...@meteorite.bi> wrote: > > Yeah I figured there was some data safety to it which is why I asked. > Okay, > well it would certainly help the performance with ingesting large > files in > bulk, and we have 1TB to ingest in one go, so I'll give it a stab early > next week. > > Tom > > On Fri, Oct 21, 2016 at 8:57 PM, BW <w...@apache.org> wrote: > > > +1 > > > > On Friday, October 21, 2016, Chris Mattmann <mattm...@apache.org> > wrote: > > > > > The rationale is to not make the staging area a moving target, > whilst > > > building controlled > > > access storage. Creating a copy is always more safe in the area of > > > preservation and provenance, > > > than making everyting a moving target. > > > > > > I would strongly caution against doing a .moveFile() unless you add > > > facilities in LocalDataTransfer to: > > > > > > 1. Make it configurable (by default off) maybe something like > > > org.apache.oodt.cas.filemgr.datatransferer.local.move > > > and settable in filemgr.properties > > > 2. Preserve (in metadata) the original file location > > > 3. Add locking to the addMetadata facilities and addReferences > since they > > > may be called in different control flows > > > > > > I would like to see a design that handles the above and unit tests > before > > > +1’ing such a change. > > > > > > > > > > > > On 10/21/16, 11:35 AM, "Tom Barber" <tom.bar...@meteorite.bi > > > <javascript:;>> wrote: > > > > > > Hello folks > > > > > > I'm asking here just incase someone knows a reason why this is > a bad > > > idea: > > > > > > We have a bunch of large files on a slow NFS mount which we're > > > ingesting in > > > bulk. Its using the LocalDataTransferer to do the ingestion > move and > > in > > > that code all the move calls are really file copies. > > > > > > As you'll all know in doing a copy the drive is actually > writing bits > > > to > > > disk, where as doing a move is just moving the file pointer. > > > > > > Is there a reason why the moveFile method actually uses > > > FileUtils.copyFile() before I try FileUtils.moveFile() ? > > > > > > Because 1 min for a copy vs 0.06 seconds for a move is far more > > > preferable. > > > > > > Thanks > > > > > > Tom > > > > > > > > > > > > > > > > > >
