[
https://issues.apache.org/jira/browse/HADOOP-19047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17814185#comment-17814185
]
ASF GitHub Bot commented on HADOOP-19047:
-----------------------------------------
shameersss1 commented on PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#issuecomment-1926348440
>1. static map of path to metadata. This will grow without constraint on a
long live process.
The entries to the Map are removed during commitTask or abortTask operation
to keep memory under control.
---
> 2. Two jobs writing to same path will it corrupt the Map ?
No, The path (complete) is guaranteed to be unique The paths stored here as
part of `private static Map<String, List<Path>> taskAttemptIdToPath = new
ConcurrentHashMap<>();` is the magic path, Eventhough the file name might be
same, The magic path for two different jobs will be different since the jobId
is included in the path.
-----
>3. the static map would be a weak ref to something held strongly by the
actual committer (see WeakReferenceMap). Once the actual task attempt is gc'd,
Since the entries from the HashMap are removed during commitTask or
abortTask operation is WeakHashMap still required?
----
>4. static structures should be per fs instances, so when an fs is cleaned
up
I am not sure why it should be scoped under fs object. For a simiar
behaviour with storing in s3, Shouldn't the static structure be available to
the whole JVM ? I mean shouldn't we able to access static structure
irrespective of the fs object.
----
>5. 'm also worried about how a job could abort a task attempt on a
different process which has failed. Before worrying about that too much, why
don't you look in spark to see how it calls abort. I'm not worried about
MapReduce except for testing -so how do itself calls the committee isn't so
important. For example: we don't care about recovery from a failed attempt as
spark itself cannot do this.
I have covered this as part of the comment
[here](https://github.com/apache/hadoop/pull/6468#issuecomment-1926304528).
---
> Support InMemory Tracking Of S3A Magic Commits
> ----------------------------------------------
>
> Key: HADOOP-19047
> URL: https://issues.apache.org/jira/browse/HADOOP-19047
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Reporter: Syed Shameerur Rahman
> Assignee: Syed Shameerur Rahman
> Priority: Major
> Labels: pull-request-available
>
> The following are the operations which happens within a Task when it uses S3A
> Magic Committer.
> *During closing of stream*
> 1. A 0-byte file with a same name of the original file is uploaded to S3
> using PUT operation. Refer
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L152]
> for more information. This is done so that the downstream application like
> Spark could get the size of the file which is being written.
> 2. MultiPartUpload(MPU) metadata is uploaded to S3. Refer
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L176]
> for more information.
> *During TaskCommit*
> 1. All the MPU metadata which the task wrote to S3 (There will be 'x' number
> of metadata file in S3 if a single task writes to 'x' files) are read and
> rewritten to S3 as a single metadata file. Refer
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicS3GuardCommitter.java#L201]
> for more information
> Since these operations happens with the Task JVM, We could optimize as well
> as save cost by storing these information in memory when Task memory usage is
> not a constraint. Hence the proposal here is to introduce a new MagicCommit
> Tracker called "InMemoryMagicCommitTracker" which will store the
> 1. Metadata of MPU in memory till the Task is committed
> 2. Store the size of the file which can be used by the downstream application
> to get the file size before it is committed/visible to the output path.
> This optimization will save 2 PUT S3 calls, 1 LIST S3 call, and 1 GET S3 call
> given a Task writes only 1 file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
