steveloughran closed pull request #6494: HADOOP-18679. Add API for bulk/paged
object deletion
URL: https://github.com/apache/hadoop/pull/6494
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
steveloughran closed pull request #5993: HADOOP-18679. Add API for bulk/paged
object deletion
URL: https://github.com/apache/hadoop/pull/5993
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
steveloughran closed pull request #6738: HADOOP-18679. Add API for bulk/paged
object deletion
URL: https://github.com/apache/hadoop/pull/6738
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
steveloughran commented on PR #6726:
URL: https://github.com/apache/hadoop/pull/6726#issuecomment-2108677588
mukund, if you can do those naming changes then I'm +1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
hadoop-yetus commented on PR #6726:
URL: https://github.com/apache/hadoop/pull/6726#issuecomment-2102481437
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks
steveloughran commented on code in PR #6726:
URL: https://github.com/apache/hadoop/pull/6726#discussion_r1592834729
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/wrappedio/WrappedIO.java:
##
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software
mukund-thakur commented on PR #6726:
URL: https://github.com/apache/hadoop/pull/6726#issuecomment-2097047455
> can you do the same here? some style checker will complain but it will
help us to separate the methods in the new class.
I don't understand what to do here.
--
This
steveloughran commented on code in PR #6726:
URL: https://github.com/apache/hadoop/pull/6726#discussion_r1588274603
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java:
##
@@ -4980,4 +4982,17 @@ public MultipartUploaderBuilder
steveloughran commented on code in PR #6726:
URL: https://github.com/apache/hadoop/pull/6726#discussion_r1584787287
##
hadoop-common-project/hadoop-common/src/site/markdown/filesystem/bulkdelete.md:
##
@@ -0,0 +1,136 @@
+
+
+# interface `BulkDelete`
+
+
+
+The `BulkDelete`
mukund-thakur commented on code in PR #6726:
URL: https://github.com/apache/hadoop/pull/6726#discussion_r1583846245
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractBulkDelete.java:
##
@@ -85,6 +88,9 @@ public
hadoop-yetus commented on PR #6726:
URL: https://github.com/apache/hadoop/pull/6726#issuecomment-2083482008
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks
hadoop-yetus commented on PR #6738:
URL: https://github.com/apache/hadoop/pull/6738#issuecomment-2082900564
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks
steveloughran commented on PR #6726:
URL: https://github.com/apache/hadoop/pull/6726#issuecomment-2079798916
iceberg poc pr https://github.com/apache/iceberg/pull/10233
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use
steveloughran commented on PR #6726:
URL: https://github.com/apache/hadoop/pull/6726#issuecomment-2079798082
for the iceberg support to work, all filesystems MUST implement the api or
we have to modify that PoC to handle the case where they don't. I'd rather the
spec says any FS which
steveloughran commented on code in PR #6726:
URL: https://github.com/apache/hadoop/pull/6726#discussion_r1581288128
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DefaultBulkDeleteOperation.java:
##
@@ -17,61 +17,86 @@
*/
package
hadoop-yetus commented on PR #6726:
URL: https://github.com/apache/hadoop/pull/6726#issuecomment-2078874481
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
hadoop-yetus commented on PR #6726:
URL: https://github.com/apache/hadoop/pull/6726#issuecomment-2078773861
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
mukund-thakur commented on code in PR #6726:
URL: https://github.com/apache/hadoop/pull/6726#discussion_r1580173720
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DefalutBulkDeleteSource.java:
##
@@ -0,0 +1,38 @@
+/**
+ * Licensed to the Apache
hadoop-yetus commented on PR #6726:
URL: https://github.com/apache/hadoop/pull/6726#issuecomment-2073964537
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks
hadoop-yetus commented on PR #6738:
URL: https://github.com/apache/hadoop/pull/6738#issuecomment-2072921149
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks
steveloughran commented on code in PR #6726:
URL: https://github.com/apache/hadoop/pull/6726#discussion_r1572223236
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DefaultBulkDeleteOperation.java:
##
@@ -0,0 +1,84 @@
+/**
+ * Licensed to the Apache
steveloughran commented on code in PR #6726:
URL: https://github.com/apache/hadoop/pull/6726#discussion_r1572230146
##
hadoop-common-project/hadoop-common/src/site/markdown/filesystem/bulkdelete.md:
##
@@ -161,124 +105,20 @@ store.hasPathCapability(path,
hadoop-yetus commented on PR #6726:
URL: https://github.com/apache/hadoop/pull/6726#issuecomment-2065627094
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
mukund-thakur commented on code in PR #6726:
URL: https://github.com/apache/hadoop/pull/6726#discussion_r1569566843
##
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/aws_sdk_upgrade.md:
##
@@ -324,6 +324,7 @@ They have also been updated to return V2 SDK classes.
mukund-thakur commented on code in PR #6726:
URL: https://github.com/apache/hadoop/pull/6726#discussion_r1569528772
##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractBulkDeleteTest.java:
##
@@ -0,0 +1,222 @@
+/*
+ * Licensed to
mukund-thakur commented on code in PR #6726:
URL: https://github.com/apache/hadoop/pull/6726#discussion_r1567902648
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractBulkDelete.java:
##
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache
hadoop-yetus commented on PR #6738:
URL: https://github.com/apache/hadoop/pull/6738#issuecomment-2059423509
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
steveloughran commented on PR #6726:
URL: https://github.com/apache/hadoop/pull/6726#issuecomment-2059138849
commented. I've also done a PR #6738 which tunes the API to work with
iceberg, having just written a PoC of the iceberg binding.
My PR
* moved the wrapper methods to a new
steveloughran commented on code in PR #6726:
URL: https://github.com/apache/hadoop/pull/6726#discussion_r1566433464
##
hadoop-common-project/hadoop-common/src/site/markdown/filesystem/bulkdelete.md:
##
@@ -0,0 +1,284 @@
+
+
+# interface `BulkDelete`
Review Comment:
needs
steveloughran commented on PR #6738:
URL: https://github.com/apache/hadoop/pull/6738#issuecomment-2059110117
This is #6726 with another commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
hadoop-yetus commented on PR #6726:
URL: https://github.com/apache/hadoop/pull/6726#issuecomment-2057942133
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
mukund-thakur commented on code in PR #6494:
URL: https://github.com/apache/hadoop/pull/6494#discussion_r1561315131
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/BulkDeleteOperationCallbacksImpl.java:
##
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache
steveloughran commented on code in PR #6494:
URL: https://github.com/apache/hadoop/pull/6494#discussion_r1561126867
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/BulkDeleteOperationCallbacksImpl.java:
##
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache
mukund-thakur commented on code in PR #6494:
URL: https://github.com/apache/hadoop/pull/6494#discussion_r1560055171
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/BulkDeleteOperationCallbacksImpl.java:
##
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache
mukund-thakur commented on code in PR #6494:
URL: https://github.com/apache/hadoop/pull/6494#discussion_r1556505328
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/BulkDelete.java:
##
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation
mukund-thakur commented on code in PR #6494:
URL: https://github.com/apache/hadoop/pull/6494#discussion_r1556490279
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/BulkDelete.java:
##
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation
mukund-thakur commented on code in PR #6494:
URL: https://github.com/apache/hadoop/pull/6494#discussion_r1556489762
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/BulkDeleteOperationCallbacksImpl.java:
##
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache
steveloughran commented on PR #6494:
URL: https://github.com/apache/hadoop/pull/6494#issuecomment-2025696873
In #6686 I'm creating a new utils class for reflection access, nothing else.
And proposing that all tests of the API use reflection to be really confident
it works and that there's
steveloughran commented on PR #6494:
URL: https://github.com/apache/hadoop/pull/6494#issuecomment-2025591287
FYI i want to pull the rate limiter API of #6596 in here too; we'd have a
rate limiter in s3a store which if enabled would limit #of deletes which can be
issued on a bucket. Ideally
ahmarsuhail commented on code in PR #6494:
URL: https://github.com/apache/hadoop/pull/6494#discussion_r1542647008
##
hadoop-common-project/hadoop-common/src/site/markdown/filesystem/bulkdelete.md:
##
@@ -0,0 +1,284 @@
+
+
+# interface `BulkDelete`
+
+
+
+The `BulkDelete`
steveloughran commented on code in PR #6494:
URL: https://github.com/apache/hadoop/pull/6494#discussion_r1535463621
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:
##
@@ -5457,7 +5421,11 @@ public boolean hasPathCapability(final Path path,
steveloughran commented on code in PR #6494:
URL: https://github.com/apache/hadoop/pull/6494#discussion_r1535462548
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/BulkDeleteOperationCallbacksImpl.java:
##
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache
mukund-thakur commented on code in PR #6494:
URL: https://github.com/apache/hadoop/pull/6494#discussion_r1532928858
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:
##
@@ -5457,7 +5421,11 @@ public boolean hasPathCapability(final Path path,
hadoop-yetus commented on PR #6494:
URL: https://github.com/apache/hadoop/pull/6494#issuecomment-1998394225
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
hadoop-yetus commented on PR #6494:
URL: https://github.com/apache/hadoop/pull/6494#issuecomment-1996080816
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
hadoop-yetus commented on PR #6494:
URL: https://github.com/apache/hadoop/pull/6494#issuecomment-1947315201
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
hadoop-yetus commented on PR #6494:
URL: https://github.com/apache/hadoop/pull/6494#issuecomment-1936382465
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
steveloughran commented on PR #6494:
URL: https://github.com/apache/hadoop/pull/6494#issuecomment-1935875347
+add a FileUtils method to assist deletion here, with
`FileUtils.bulkDeletePageSize(path) -> int` and `FileUtils.bulkDelete(path,
List) -> List; each will create a bulk delete
hadoop-yetus commented on PR #6494:
URL: https://github.com/apache/hadoop/pull/6494#issuecomment-1912353079
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
hadoop-yetus commented on PR #6494:
URL: https://github.com/apache/hadoop/pull/6494#issuecomment-1908768612
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
hadoop-yetus commented on PR #5993:
URL: https://github.com/apache/hadoop/pull/5993#issuecomment-1908677204
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
hadoop-yetus commented on PR #5993:
URL: https://github.com/apache/hadoop/pull/5993#issuecomment-1873467248
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: |
steveloughran commented on code in PR #5993:
URL: https://github.com/apache/hadoop/pull/5993#discussion_r1349184302
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/BulkDelete.java:
##
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation
steveloughran commented on PR #5993:
URL: https://github.com/apache/hadoop/pull/5993#issuecomment-1748524668
@ahmarsuhail
- caller provides a remote iterator, such as the ones we do for listing or
another source/transformation (see RemoteIterators)
- build() call returns some
54 matches
Mail list logo
