Another option:
1. Move the files to where they should be in controlled access storage
2. Use InPlaceDataTransfer (no move, copy, anything, just cataloging!)
On 10/21/16, 1:01 PM, "Tom Barber" <tom.bar...@meteorite.bi> wrote:
Yeah I figured there was some data safety to it which is why I asked. Okay,
well it would certainly help the performance with ingesting large files in
bulk, and we have 1TB to ingest in one go, so I'll give it a stab early
next week.
Tom
On Fri, Oct 21, 2016 at 8:57 PM, BW <w...@apache.org> wrote:
> +1
>
> On Friday, October 21, 2016, Chris Mattmann <mattm...@apache.org> wrote:
>
> > The rationale is to not make the staging area a moving target, whilst
> > building controlled
> > access storage. Creating a copy is always more safe in the area of
> > preservation and provenance,
> > than making everyting a moving target.
> >
> > I would strongly caution against doing a .moveFile() unless you add
> > facilities in LocalDataTransfer to:
> >
> > 1. Make it configurable (by default off) maybe something like
> > org.apache.oodt.cas.filemgr.datatransferer.local.move
> > and settable in filemgr.properties
> > 2. Preserve (in metadata) the original file location
> > 3. Add locking to the addMetadata facilities and addReferences since
they
> > may be called in different control flows
> >
> > I would like to see a design that handles the above and unit tests
before
> > +1’ing such a change.
> >
> >
> >
> > On 10/21/16, 11:35 AM, "Tom Barber" <tom.bar...@meteorite.bi
> > <javascript:;>> wrote:
> >
> > Hello folks
> >
> > I'm asking here just incase someone knows a reason why this is a bad
> > idea:
> >
> > We have a bunch of large files on a slow NFS mount which we're
> > ingesting in
> > bulk. Its using the LocalDataTransferer to do the ingestion move and
> in
> > that code all the move calls are really file copies.
> >
> > As you'll all know in doing a copy the drive is actually writing
bits
> > to
> > disk, where as doing a move is just moving the file pointer.
> >
> > Is there a reason why the moveFile method actually uses
> > FileUtils.copyFile() before I try FileUtils.moveFile() ?
> >
> > Because 1 min for a copy vs 0.06 seconds for a move is far more
> > preferable.
> >
> > Thanks
> >
> > Tom
> >
> >
> >
> >
>
