Yeah, please file a JIRA. - Cham
On Thu, Jul 26, 2018 at 11:33 AM Jozef Vilcek <[email protected]> wrote: > Yes, rename can be tricky with cross-directory. This is related > https://issues.apache.org/jira/browse/BEAM-4861 > I guess I can file a JIRA for this, right? > > On Thu, Jul 26, 2018 at 7:31 PM Chamikara Jayalath <[email protected]> > wrote: > >> Also, we'll have to use StandardMoveOptions.IGNORE_MISSING_FILES for >> supporting failures of the rename step. I think this is a good change to do >> if the change significantly improves the performance of some of the >> FileSystems (note that some FileSystems, for example GCS, implement rename >> in the form of a copy+delete, so there will be no significant performance >> improvements for such FileSystems). >> >> -Cham >> >> On Thu, Jul 26, 2018 at 10:14 AM Reuven Lax <[email protected]> wrote: >> >>> We might be able to replace this with Filesystem.rename(). One thing to >>> keep in mind - the destination files might be in a different directory, so >>> we would need to make sure that all Filesystems support cross-directory >>> rename. >>> >>> On Thu, Jul 26, 2018 at 9:58 AM Lukasz Cwik <[email protected]> wrote: >>> >>>> +dev >>>> >>>> On Thu, Jul 26, 2018 at 2:40 AM Jozef Vilcek <[email protected]> >>>> wrote: >>>> >>>>> Hello, >>>>> >>>>> just came across FileBasedSink.WriteOperation class which does have >>>>> moveToOutput() method. Implementation does a Filesystem.copy() instead of >>>>> "move". With large files I find it quote no efficient if underlying FS >>>>> supports more efficient ways, so I wonder what is the story behind it? >>>>> Must >>>>> it be a copy? >>>>> >>>>> >>>>> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java#L761 >>>>> >>>>
