hadoop-yetus commented on PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#issuecomment-2020947224
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
steveloughran merged PR #6468:
URL: https://github.com/apache/hadoop/pull/6468
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
shameersss1 commented on PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#issuecomment-2020548007
@steveloughran - I have addressed your comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
hadoop-yetus commented on PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#issuecomment-2020230242
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1539051935
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/impl/CommitOperations.java:
##
@@ -584,7 +584,7 @@ public SinglePendingCommit
steveloughran commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1539027620
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/impl/CommitOperations.java:
##
@@ -584,7 +584,7 @@ public SinglePendingCommit
shameersss1 commented on PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#issuecomment-2019942477
@steveloughran - Thanks a lot for the detailed review. I have addressed
your comments.
> note, you will need to a followup in the docs -but we can get this in and
tested while
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1538875809
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/InMemoryMagicCommitTracker.java:
##
@@ -113,15 +122,26 @@ public boolean
steveloughran commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1537910244
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/InMemoryMagicCommitTracker.java:
##
@@ -113,15 +122,26 @@ public boolean
shameersss1 commented on PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#issuecomment-2011393345
@steveloughran - Gentle reminder for review
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
shameersss1 commented on PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#issuecomment-1954651307
@steveloughran - I am glad to assist with the testing. Is there any release
candidate branch for the same? Could you please share the wiki on what tests
needs to be done?
--
This
steveloughran commented on PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#issuecomment-1954463412
@shameersss1 I'm working on assisting getting the 3.4.0 release out right
now. Anything you can do to assist testing would be wonderful, as I'm only
worrying about release blockers.
shameersss1 commented on PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#issuecomment-1951725768
@steveloughran - Gentle reminder to review the changes.
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and
hadoop-yetus commented on PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#issuecomment-1938307370
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
shameersss1 commented on PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#issuecomment-1938151034
@steveloughran - Thanks a lot for review comments, I have addressed the
comments with the new commit `d8db729e5568df5dc920604eff8167a575a5894c`
1. Added test in
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1485758682
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/InMemoryMagicCommitTracker.java:
##
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1485746226
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTrackerUtils.java:
##
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1485745785
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java:
##
@@ -118,76 +103,21 @@ public boolean
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1485744738
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/InMemoryMagicCommitTracker.java:
##
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1485743070
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/InMemoryMagicCommitTracker.java:
##
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1485740943
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/MagicCommitIntegration.java:
##
@@ -20,17 +20,19 @@
import java.util.List;
+import
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1485740785
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/CommitConstants.java:
##
@@ -242,6 +242,13 @@ private CommitConstants() {
*/
public
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1485740055
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:
##
@@ -3906,6 +3908,21 @@ public void access(final Path f, final FsAction
steveloughran commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1484762085
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/CommitConstants.java:
##
@@ -242,6 +242,13 @@ private CommitConstants() {
*/
steveloughran commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1484756242
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicS3GuardCommitter.java:
##
@@ -264,9 +326,14 @@ public void
shameersss1 commented on PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#issuecomment-1934214757
@steveloughran - Gentle reminder for review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
hadoop-yetus commented on PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#issuecomment-1926764689
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
shameersss1 commented on PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#issuecomment-1926512939
@steveloughran - Thanks a lot a detailed review as well as amazing follow
up question. I have addressed your comments, Please let me know your thoughts.
--
This is an automated
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1477845326
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/magic/ITestMagicCommitProtocol.java:
##
@@ -71,6 +79,26 @@ public void setup() throws
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1477808239
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:
##
@@ -52,6 +52,7 @@
import java.util.concurrent.atomic.AtomicBoolean;
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1477804569
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/S3MagicCommitTracker.java:
##
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1477795947
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicS3GuardCommitter.java:
##
@@ -248,6 +236,80 @@ private PendingSet
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1477749474
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicS3GuardCommitter.java:
##
@@ -264,9 +326,14 @@ public void
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1477747332
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:
##
@@ -3906,6 +3908,21 @@ public void access(final Path f, final FsAction
shameersss1 commented on PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#issuecomment-1926348440
>1. static map of path to metadata. This will grow without constraint on a
long live process.
The entries to the Map are removed during commitTask or abortTask operation
to
shameersss1 commented on PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#issuecomment-1926304528
@steveloughran - Thanks a lot a for a detailed review and some amazing
question, The following are my thoughts on the different asks.
> 1. Marker files at the end of each path
steveloughran commented on PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#issuecomment-1925340023
I've thought about this some more. Here are some things which I believe we
need
1. Marker files at the end of each path so that spark status reporting on
different processes
steveloughran commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1476480590
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:
##
@@ -52,6 +52,7 @@
import java.util.concurrent.atomic.AtomicBoolean;
shameersss1 commented on PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#issuecomment-1919397323
@steveloughran - Could you please review the changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
hadoop-yetus commented on PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#issuecomment-1913253726
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
hadoop-yetus commented on PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#issuecomment-1911683899
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
hadoop-yetus commented on PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#issuecomment-1911508281
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
shameersss1 commented on PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#issuecomment-1911341093
@steveloughran - I have converted draft PR to final one. Could you please
review the changes.
Thanks
--
This is an automated message from the Apache Git Service.
To respond to
shameersss1 commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1467185040
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/S3MagicCommitTracker.java:
##
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache
steveloughran commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1461054737
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/S3MagicCommitTracker.java:
##
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache
shameersss1 opened a new pull request, #6468:
URL: https://github.com/apache/hadoop/pull/6468
Draft CR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe,
46 matches
Mail list logo
