[
https://issues.apache.org/jira/browse/HDFS-16716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17623709#comment-17623709
]
ASF GitHub Bot commented on HDFS-16716:
---------------------------------------
tomscut commented on code in PR #4697:
URL: https://github.com/apache/hadoop/pull/4697#discussion_r1003970562
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java:
##########
@@ -1544,6 +1544,39 @@ public FSDataOutputStream append(Path f, int bufferSize)
throws IOException {
public abstract FSDataOutputStream append(Path f, int bufferSize,
Progressable progress) throws IOException;
+ /**
+ * Append to an existing file (optional operation).
+ * @param f the existing file to be appended.
+ * @param appendToNewBlock whether to append data to a new block
+ * instead of the end of the last partial block
+ * @throws IOException IO failure
+ * @throws UnsupportedOperationException if the operation is unsupported
+ * (default).
+ * @return output stream.
+ */
+ public FSDataOutputStream append(Path f, boolean appendToNewBlock) throws
IOException {
+ return append(f, getConf().getInt(IO_FILE_BUFFER_SIZE_KEY,
+ IO_FILE_BUFFER_SIZE_DEFAULT), null, appendToNewBlock);
+ }
+
+ /**
+ * Append to an existing file (optional operation).
+ * This function is used for being overridden by some FileSystem like
DistributedFileSystem
+ * @param f the existing file to be appended.
+ * @param bufferSize the size of the buffer to be used.
+ * @param progress for reporting progress if it is not null.
+ * @param appendToNewBlock whether to append data to a new block
+ * instead of the end of the last partial block
+ * @throws IOException IO failure
+ * @throws UnsupportedOperationException if the operation is unsupported
+ * (default).
+ * @return output stream.
+ */
+ public FSDataOutputStream append(Path f, int bufferSize,
+ Progressable progress, boolean appendToNewBlock) throws IOException {
+ return append(f, bufferSize, progress);
Review Comment:
> TestDFSShell#testAppendToFileWithOptionN didn't contain a partial last
block, because the block size is `1024` and the file size in the UT is `1024 *
1024`, so the last block is not partial block. Maybe you can change the file
size to `1024 * 1024 + 10` to verify it again.
Hi @ZanderXu , there is a new configuration. And block size is not `1024`
but `128 * 1024 * 1024`. So the last block is partial.
> Improve appendToFile command: support appending on file with new block
> ----------------------------------------------------------------------
>
> Key: HDFS-16716
> URL: https://issues.apache.org/jira/browse/HDFS-16716
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: guojunhao
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> HDFS client DistributedFileSystem#append supports appending to a file with
> optional create flags.
> However, appendToFile command only supports the default create flag APPEND so
> that append on EC file without NEW_BLOCK create flag is not supported.
> Thus, it's necessary to improve appendToFile command by adding option n for
> it. Option n represents that use NEW_BLOCK create flag while appending file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
