Yeah I figured there was some data safety to it which is why I asked. Okay, well it would certainly help the performance with ingesting large files in bulk, and we have 1TB to ingest in one go, so I'll give it a stab early next week.
Tom On Fri, Oct 21, 2016 at 8:57 PM, BW <w...@apache.org> wrote: > +1 > > On Friday, October 21, 2016, Chris Mattmann <mattm...@apache.org> wrote: > > > The rationale is to not make the staging area a moving target, whilst > > building controlled > > access storage. Creating a copy is always more safe in the area of > > preservation and provenance, > > than making everyting a moving target. > > > > I would strongly caution against doing a .moveFile() unless you add > > facilities in LocalDataTransfer to: > > > > 1. Make it configurable (by default off) maybe something like > > org.apache.oodt.cas.filemgr.datatransferer.local.move > > and settable in filemgr.properties > > 2. Preserve (in metadata) the original file location > > 3. Add locking to the addMetadata facilities and addReferences since they > > may be called in different control flows > > > > I would like to see a design that handles the above and unit tests before > > +1’ing such a change. > > > > > > > > On 10/21/16, 11:35 AM, "Tom Barber" <tom.bar...@meteorite.bi > > <javascript:;>> wrote: > > > > Hello folks > > > > I'm asking here just incase someone knows a reason why this is a bad > > idea: > > > > We have a bunch of large files on a slow NFS mount which we're > > ingesting in > > bulk. Its using the LocalDataTransferer to do the ingestion move and > in > > that code all the move calls are really file copies. > > > > As you'll all know in doing a copy the drive is actually writing bits > > to > > disk, where as doing a move is just moving the file pointer. > > > > Is there a reason why the moveFile method actually uses > > FileUtils.copyFile() before I try FileUtils.moveFile() ? > > > > Because 1 min for a copy vs 0.06 seconds for a move is far more > > preferable. > > > > Thanks > > > > Tom > > > > > > > > >
