[
https://issues.apache.org/jira/browse/HDFS-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16625818#comment-16625818
]
Steve Loughran commented on HDFS-13936:
---------------------------------------
bq. If there anything in the contract of HDFS specifying that it must fail
when concatting empty files? If so, is it a bad idea to allow this behaviour?
w'eve got
{code}
HDFS: All source files except the final one MUST be a complete block:
for s in (sources[0:length(sources)-1] + [p]):
(length(FS, s) mod getBlockSize(FS, p)) == 0
{code}
doesn't cover a 0-byte final block; the error message implies it thinks the
file might still be open.
* spec might need changing to cover the HDFS behaviour, if not as the required
action, as a "HDFS does this" note.
* {{AbstractContractConcatTest}} looks like it's expecting empty concats to
fail.
assuming 1+ FS can't handle empty block concats, the uploader will have to do
something about tracking empty uploads or simply exclude them from the concat
call, and if there are no non-empty uploads to concat, just touch() the empty
file. Nothing too complex.
> multipart upload to HDFS to support 0 byte upload
> -------------------------------------------------
>
> Key: HDFS-13936
> URL: https://issues.apache.org/jira/browse/HDFS-13936
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs, hdfs
> Affects Versions: 3.2.0
> Reporter: Steve Loughran
> Assignee: Ewan Higgs
> Priority: Major
>
> MPUs to HDFS fail as you can't concat an empty block.
> Whatever uploads to HDFS needs to recognise that specific case "0-byte file"
> and rather than try and concat things, just create a 0-byte file there.
> Without this, you can't use MPU as a replacement for distcp or alternative
> commit protocols.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
