[
https://issues.apache.org/jira/browse/BEAM-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16654857#comment-16654857
]
Jozef Vilcek commented on BEAM-5036:
------------------------------------
[~timrobertson100], sorry, I am a bit lost in conversation in the PR. What
exactly is the problem making ti not mergeable? What needs to be addressed or
is there another PR to track?
Also, to make change only in HDFS filesystem implementation, how do you propose
to do it? The WriteOperation.moveToOutput() is now using `FileSystems.copy`,
this would need to change to `rename` anyway, right?
> Optimize FileBasedSink's WriteOperation.moveToOutput()
> ------------------------------------------------------
>
> Key: BEAM-5036
> URL: https://issues.apache.org/jira/browse/BEAM-5036
> Project: Beam
> Issue Type: Improvement
> Components: io-java-files
> Affects Versions: 2.5.0
> Reporter: Jozef Vilcek
> Assignee: Tim Robertson
> Priority: Major
> Time Spent: 11h
> Remaining Estimate: 0h
>
> moveToOutput() methods in FileBasedSink.WriteOperation implements move by
> copy+delete. It would be better to use a rename() which can be much more
> effective for some filesystems.
> Filesystem must support cross-directory rename. BEAM-4861 is related to this
> for the case of HDFS filesystem.
> Feature was discussed here:
> http://mail-archives.apache.org/mod_mbox/beam-dev/201807.mbox/%3CCAF9t7_4Mp54pQ+vRrJrBh9Vx0=uaknupzd_qdh_qdm9vxll...@mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
