[
https://issues.apache.org/jira/browse/HADOOP-17271?focusedWorklogId=517145&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-517145
]
ASF GitHub Bot logged work on HADOOP-17271:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 26/Nov/20 18:17
Start Date: 26/Nov/20 18:17
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#2324:
URL: https://github.com/apache/hadoop/pull/2324#discussion_r531179022
##########
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java
##########
@@ -102,21 +104,32 @@ public boolean outputImmediatelyVisible() {
* @param uploadId Upload ID
* @param parts list of parts
* @param bytesWritten bytes written
+ * @param iostatistics nullable IO statistics
* @return false, indicating that the commit must fail.
* @throws IOException any IO problem.
* @throws IllegalArgumentException bad argument
*/
@Override
public boolean aboutToComplete(String uploadId,
List<PartETag> parts,
- long bytesWritten)
+ long bytesWritten,
+ final IOStatistics iostatistics)
throws IOException {
Preconditions.checkArgument(StringUtils.isNotEmpty(uploadId),
"empty/null upload ID: "+ uploadId);
Preconditions.checkArgument(parts != null,
"No uploaded parts list");
Preconditions.checkArgument(!parts.isEmpty(),
"No uploaded parts to save");
+
+ // put a 0-byte file with the name of the original under-magic path
Review comment:
no, it's always done. I just added the docs for people to understand
what is going on. We put the file so that code which expects a file to exist
after close() is satisified. Some code in spark does. Luckily nothing ever
tries to read the data until the job is committed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 517145)
Time Spent: 6h 10m (was: 6h)
> S3A statistics to support IOStatistics
> --------------------------------------
>
> Key: HADOOP-17271
> URL: https://issues.apache.org/jira/browse/HADOOP-17271
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> S3A to rework statistics with
> * API + Implementation split of the interfaces used by subcomponents when
> reporting stats
> * S3A Instrumentation to implement all the interfaces
> * streams, etc to all implement IOStatisticsSources and serve to callers
> * Add some tracking of durations of remote requests
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
