[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=710762=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-710762
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 18/Jan/22 18:52
Start Date: 18/Jan/22 18:52
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3897:
URL: https://github.com/apache/hadoop/pull/3897#issuecomment-1015724202
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 110 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +0 :ok: | mvndep | 3m 24s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 19s | | branch-3.3 passed |
| +1 :green_heart: | compile | 18m 17s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 2m 53s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 3m 34s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 3m 16s | | branch-3.3 passed |
| +0 :ok: | spotbugs | 0m 30s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 25m 30s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 13s | | the patch passed |
| +1 :green_heart: | compile | 17m 31s | | the patch passed |
| -1 :x: | javac | 17m 31s |
[/results-compile-javac-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/4/artifact/out/results-compile-javac-root.txt)
| root generated 1 new + 1924 unchanged - 50 fixed = 1925 total (was 1974) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 46s | | root: The patch generated
0 new + 72 unchanged - 83 fixed = 72 total (was 155) |
| +1 :green_heart: | mvnsite | 3m 28s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 28s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | javadoc | 1m 34s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | javadoc | 0m 36s | | hadoop-tools_hadoop-aws
generated 0 new + 40 unchanged - 23 fixed = 40 total (was 63) |
| +1 :green_heart: | javadoc | 0m 34s | | hadoop-azure in the patch
passed. |
| +0 :ok: | spotbugs | 0m 28s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 25m 21s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 26s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | unit | 18m 42s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 5s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | unit | 2m 10s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not
generate ASF License warnings. |
| | | 182m 26s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3897 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle
shellcheck shelldocs |
| uname | Linux 2fbbb28e1b23 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29
06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / cdb6aacc10df809bb0589bca5a7e79dfba884b69 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
|
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=710700=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-710700 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 18/Jan/22 18:05 Start Date: 18/Jan/22 18:05 Worklog Time Spent: 10m Work Description: steveloughran merged pull request #3897: URL: https://github.com/apache/hadoop/pull/3897 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 710700) Time Spent: 15h 10m (was: 15h) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 15h 10m > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=710698=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-710698 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 18/Jan/22 18:04 Start Date: 18/Jan/22 18:04 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #3897: URL: https://github.com/apache/hadoop/pull/3897#issuecomment-1015675137 ok, yetus happy. will merge -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 710698) Time Spent: 15h (was: 14h 50m) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 15h > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=710626=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-710626
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 18/Jan/22 16:28
Start Date: 18/Jan/22 16:28
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3897:
URL: https://github.com/apache/hadoop/pull/3897#issuecomment-1015582590
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 10m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 5s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 1s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 110 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +0 :ok: | mvndep | 3m 29s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 30s | | branch-3.3 passed |
| +1 :green_heart: | compile | 17m 33s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 2m 45s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 4m 6s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 3m 58s | | branch-3.3 passed |
| +0 :ok: | spotbugs | 0m 41s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 24m 2s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 17s | | the patch passed |
| +1 :green_heart: | compile | 16m 54s | | the patch passed |
| +1 :green_heart: | javac | 16m 54s | | root generated 0 new + 1921
unchanged - 50 fixed = 1921 total (was 1971) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 36s | | root: The patch generated
0 new + 72 unchanged - 83 fixed = 72 total (was 155) |
| +1 :green_heart: | mvnsite | 4m 3s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 37s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | javadoc | 1m 40s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | javadoc | 0m 46s | | hadoop-tools_hadoop-aws
generated 0 new + 40 unchanged - 23 fixed = 40 total (was 63) |
| +1 :green_heart: | javadoc | 0m 44s | | hadoop-azure in the patch
passed. |
| +0 :ok: | spotbugs | 0m 37s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 24m 12s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 37s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | unit | 19m 35s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 24s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | unit | 2m 24s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not
generate ASF License warnings. |
| | | 190m 40s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3897 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle
shellcheck shelldocs |
| uname | Linux f84d65c15cb0 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17
17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / c4a3883500e48fe594f48dc6ae1d32b7943a3c17 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/3/testReport/ |
| Max. process+thread count | 1265 (vs.
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=710601=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-710601
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 18/Jan/22 15:52
Start Date: 18/Jan/22 15:52
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3897:
URL: https://github.com/apache/hadoop/pull/3897#issuecomment-1015547846
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 10m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 110 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +0 :ok: | mvndep | 3m 24s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 16s | | branch-3.3 passed |
| +1 :green_heart: | compile | 20m 21s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 2m 59s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 3m 49s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 3m 18s | | branch-3.3 passed |
| +0 :ok: | spotbugs | 0m 31s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 30m 8s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 46s | | the patch passed |
| +1 :green_heart: | compile | 23m 14s | | the patch passed |
| +1 :green_heart: | javac | 23m 14s | | root generated 0 new + 1921
unchanged - 50 fixed = 1921 total (was 1971) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 26s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/2/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 72 unchanged - 83 fixed = 73 total (was
155) |
| +1 :green_heart: | mvnsite | 4m 23s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 7s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 29s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | javadoc | 1m 51s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | javadoc | 0m 43s | | hadoop-tools_hadoop-aws
generated 0 new + 40 unchanged - 23 fixed = 40 total (was 63) |
| +1 :green_heart: | javadoc | 0m 38s | | hadoop-azure in the patch
passed. |
| +0 :ok: | spotbugs | 0m 31s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 30m 28s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 29s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | unit | 20m 22s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 20s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | unit | 2m 32s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 1s | | The patch does not
generate ASF License warnings. |
| | | 218m 2s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3897 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle
shellcheck shelldocs |
| uname | Linux e5d846a5058e 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5
11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / 075edbe9bcf36cb36f34b222b4fcf6a301e5c892 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
|
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=710586=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-710586
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 18/Jan/22 15:31
Start Date: 18/Jan/22 15:31
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #3897:
URL: https://github.com/apache/hadoop/pull/3897#issuecomment-1014984991
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 10m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 110 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +0 :ok: | mvndep | 3m 25s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 14s | | branch-3.3 passed |
| +1 :green_heart: | compile | 18m 12s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 2m 52s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 3m 33s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 3m 18s | | branch-3.3 passed |
| +0 :ok: | spotbugs | 0m 31s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 25m 18s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 28s |
[/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/1/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| -1 :x: | compile | 16m 28s |
[/patch-compile-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/1/artifact/out/patch-compile-root.txt)
| root in the patch failed. |
| -1 :x: | javac | 16m 28s |
[/patch-compile-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/1/artifact/out/patch-compile-root.txt)
| root in the patch failed. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 24s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 72 unchanged - 83 fixed = 73 total (was
155) |
| -1 :x: | mvnsite | 0m 47s |
[/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/1/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 26s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | javadoc | 1m 34s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | javadoc | 0m 36s | | hadoop-tools_hadoop-aws
generated 0 new + 40 unchanged - 23 fixed = 40 total (was 63) |
| +1 :green_heart: | javadoc | 0m 34s | | hadoop-azure in the patch
passed. |
| +0 :ok: | spotbugs | 0m 29s | | hadoop-project has no data from
spotbugs |
| -1 :x: | spotbugs | 0m 40s |
[/patch-spotbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/1/artifact/out/patch-spotbugs-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| +1 :green_heart: | shadedclient | 25m 35s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 26s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | unit | 18m 35s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 0m 41s |
[/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/1/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| +1 :green_heart: | unit | 2m 9s | | hadoop-azure in the patch
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=710493=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-710493 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 18/Jan/22 13:16 Start Date: 18/Jan/22 13:16 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #3897: URL: https://github.com/apache/hadoop/pull/3897#issuecomment-1015402176 tested s3 london, `-Dparallel-tests -DtestsThreadCount=6 -Dmarkers=delete -Dscale` all happy -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 710493) Time Spent: 14h 20m (was: 14h 10m) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 14h 20m > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=710201=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-710201
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 18/Jan/22 01:06
Start Date: 18/Jan/22 01:06
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3897:
URL: https://github.com/apache/hadoop/pull/3897#issuecomment-1014984991
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 10m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 110 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +0 :ok: | mvndep | 3m 25s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 14s | | branch-3.3 passed |
| +1 :green_heart: | compile | 18m 12s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 2m 52s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 3m 33s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 3m 18s | | branch-3.3 passed |
| +0 :ok: | spotbugs | 0m 31s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 25m 18s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 28s |
[/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/1/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| -1 :x: | compile | 16m 28s |
[/patch-compile-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/1/artifact/out/patch-compile-root.txt)
| root in the patch failed. |
| -1 :x: | javac | 16m 28s |
[/patch-compile-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/1/artifact/out/patch-compile-root.txt)
| root in the patch failed. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 24s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 72 unchanged - 83 fixed = 73 total (was
155) |
| -1 :x: | mvnsite | 0m 47s |
[/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/1/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 26s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | javadoc | 1m 34s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | javadoc | 0m 36s | | hadoop-tools_hadoop-aws
generated 0 new + 40 unchanged - 23 fixed = 40 total (was 63) |
| +1 :green_heart: | javadoc | 0m 34s | | hadoop-azure in the patch
passed. |
| +0 :ok: | spotbugs | 0m 29s | | hadoop-project has no data from
spotbugs |
| -1 :x: | spotbugs | 0m 40s |
[/patch-spotbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/1/artifact/out/patch-spotbugs-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| +1 :green_heart: | shadedclient | 25m 35s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 26s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | unit | 18m 35s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 0m 41s |
[/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/1/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| +1 :green_heart: | unit | 2m 9s | | hadoop-azure in the patch
passed.
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=710162=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-710162 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 17/Jan/22 21:58 Start Date: 17/Jan/22 21:58 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #3897: URL: https://github.com/apache/hadoop/pull/3897#issuecomment-1014912928 this is #3534 for branch 3.3 doesn't compile yet. I want to cherrypick the lists change of #3072 first, even though now I've dealt with a lot of merge conflict here on deleted files, it'll make rebasing this needlessly hard -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 710162) Time Spent: 14h (was: 13h 50m) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 14h > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=710161=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-710161 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 17/Jan/22 21:55 Start Date: 17/Jan/22 21:55 Worklog Time Spent: 10m Work Description: steveloughran opened a new pull request #3897: URL: https://github.com/apache/hadoop/pull/3897 Completely removes S3Guard support from the S3A codebase. If the connector is configured to use any metastore other than the null and local stores (i.e. DynamoDB is selected) the s3a client will raise an exception and refuse to initialize. This is to ensure that there is no mix of S3Guard enabled and disabled deployments with the same configuration but different hadoop releases -it must be turned off completely. The "hadoop s3guard" command has been retained -but the supported subcommands have been reduced to those which are not purely S3Guard related: "bucket-info" and "uploads". This is major change in terms of the number of files changed; before cherry picking subsequent s3a patches into older releases, this patch will probably need backporting first. Goodbye S3Guard, your work is done. Time to die. Contributed by Steve Loughran. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 710161) Time Spent: 13h 50m (was: 13h 40m) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 13h 50m > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=710087=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-710087 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 17/Jan/22 18:10 Start Date: 17/Jan/22 18:10 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1014789921 done. will do the backporting next -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 710087) Time Spent: 13h 40m (was: 13.5h) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 13h 40m > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=710086=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-710086 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 17/Jan/22 18:09 Start Date: 17/Jan/22 18:09 Worklog Time Spent: 10m Work Description: steveloughran merged pull request #3534: URL: https://github.com/apache/hadoop/pull/3534 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 710086) Time Spent: 13.5h (was: 13h 20m) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 13.5h > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=710077=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-710077
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 17/Jan/22 17:54
Start Date: 17/Jan/22 17:54
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1012537514
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 13s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 111 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 18s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 13s | | trunk passed |
| +1 :green_heart: | compile | 28m 49s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 24m 5s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 15s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 49s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 4m 24s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 44s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 27m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 41s | | the patch passed |
| +1 :green_heart: | compile | 28m 26s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 28m 26s |
[/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/17/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt)
| root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 1 new + 1813 unchanged - 51
fixed = 1814 total (was 1864) |
| +1 :green_heart: | compile | 25m 24s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 25m 24s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/17/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 1688
unchanged - 51 fixed = 1689 total (was 1739) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 23s | | root: The patch generated
0 new + 72 unchanged - 83 fixed = 72 total (was 155) |
| +1 :green_heart: | mvnsite | 4m 55s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 3m 27s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 40s | | hadoop-project in the patch
passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 2m 4s | | hadoop-common in the patch
passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 0m 49s | |
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new +
39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 46s | | hadoop-azure in the
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=710060=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-710060
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 17/Jan/22 17:09
Start Date: 17/Jan/22 17:09
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1014745441
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 110 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 7s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 46s | | trunk passed |
| +1 :green_heart: | compile | 22m 21s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 19m 35s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 39s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 18s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 18s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 57s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 42s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 20m 53s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 21s | | the patch passed |
| +1 :green_heart: | compile | 21m 32s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 21m 32s | |
root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 0 new + 1816 unchanged - 50
fixed = 1816 total (was 1866) |
| +1 :green_heart: | compile | 19m 32s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 19m 32s | |
root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new + 1692 unchanged -
50 fixed = 1692 total (was 1742) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 34s | | root: The patch generated
0 new + 72 unchanged - 83 fixed = 72 total (was 155) |
| +1 :green_heart: | mvnsite | 4m 13s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 3m 17s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 35s | | hadoop-project in the patch
passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 1m 46s | | hadoop-common in the patch
passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 0m 48s | |
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new +
39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 46s | | hadoop-azure in the patch
passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +0 :ok: | spotbugs | 0m 38s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 21m 13s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 35s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | unit | 18m 11s | | hadoop-common in
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=710046=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-710046
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 17/Jan/22 16:30
Start Date: 17/Jan/22 16:30
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1014716071
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 13s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 3s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 110 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 57s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 33s | | trunk passed |
| +1 :green_heart: | compile | 24m 25s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 58s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 50s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 26s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 32s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 23m 52s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 23s | | the patch passed |
| +1 :green_heart: | compile | 23m 43s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 23m 43s |
[/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/19/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt)
| root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 1 new + 1813 unchanged - 51
fixed = 1814 total (was 1864) |
| +1 :green_heart: | compile | 20m 51s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 20m 51s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/19/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 1688
unchanged - 51 fixed = 1689 total (was 1739) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 50s | | root: The patch generated
0 new + 72 unchanged - 83 fixed = 72 total (was 155) |
| +1 :green_heart: | mvnsite | 3m 44s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 2m 45s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 28s | | hadoop-project in the patch
passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 1m 39s | | hadoop-common in the patch
passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 0m 41s | |
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new +
39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 37s | | hadoop-azure in the patch
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=709925=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-709925 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 17/Jan/22 13:17 Start Date: 17/Jan/22 13:17 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1014526054 rebased to trunk to line everything up for a merge and to retest locally first -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 709925) Time Spent: 12h 50m (was: 12h 40m) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 12h 50m > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=709882=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-709882 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 17/Jan/22 11:51 Start Date: 17/Jan/22 11:51 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1014438843 thanks. realised there's one thing left -an entry in troubleshooting about the new error message. building a release, generating the real output and adding. this is doc only change so shouldn't force a new review. time to "knife the baby" , as bill gates said to steve jobs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 709882) Time Spent: 12h 40m (was: 12.5h) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 12h 40m > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=709878=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-709878
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 17/Jan/22 11:48
Start Date: 17/Jan/22 11:48
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1012520863
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 5s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 110 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 17s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 39s | | trunk passed |
| +1 :green_heart: | compile | 24m 30s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 7s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 47s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 17s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 24s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 57s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 43s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 21m 34s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 20s | | the patch passed |
| +1 :green_heart: | compile | 23m 57s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 23m 57s |
[/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/18/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt)
| root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 1 new + 1813 unchanged - 51
fixed = 1814 total (was 1864) |
| +1 :green_heart: | compile | 20m 52s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 20m 52s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/18/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 1688
unchanged - 51 fixed = 1689 total (was 1739) |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 58s | | root: The patch generated
0 new + 72 unchanged - 83 fixed = 72 total (was 155) |
| +1 :green_heart: | mvnsite | 4m 13s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 3m 20s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 38s | | hadoop-project in the patch
passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 1m 42s | | hadoop-common in the patch
passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 0m 48s | |
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new +
39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 46s | | hadoop-azure in the
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=709877=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-709877
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 17/Jan/22 11:48
Start Date: 17/Jan/22 11:48
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1012520838
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 110 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 65m 18s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 5s | | trunk passed |
| +1 :green_heart: | compile | 24m 26s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 42s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 46s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 24s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 33s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 23m 57s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 18s | | the patch passed |
| +1 :green_heart: | compile | 23m 30s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 23m 30s |
[/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/16/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt)
| root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 1 new + 1813 unchanged - 51
fixed = 1814 total (was 1864) |
| +1 :green_heart: | compile | 20m 43s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 20m 43s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/16/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 1688
unchanged - 51 fixed = 1689 total (was 1739) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 45s | | root: The patch generated
0 new + 72 unchanged - 83 fixed = 72 total (was 155) |
| +1 :green_heart: | mvnsite | 3m 39s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 2m 43s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 27s | | hadoop-project in the patch
passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 1m 41s | | hadoop-common in the patch
passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 0m 38s | |
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new +
39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 36s | | hadoop-azure in the
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=708682=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-708682
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 13/Jan/22 21:41
Start Date: 13/Jan/22 21:41
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1012537514
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 13s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 111 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 18s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 13s | | trunk passed |
| +1 :green_heart: | compile | 28m 49s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 24m 5s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 15s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 49s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 4m 24s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 44s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 27m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 41s | | the patch passed |
| +1 :green_heart: | compile | 28m 26s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 28m 26s |
[/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/17/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt)
| root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 1 new + 1813 unchanged - 51
fixed = 1814 total (was 1864) |
| +1 :green_heart: | compile | 25m 24s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 25m 24s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/17/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 1688
unchanged - 51 fixed = 1689 total (was 1739) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 23s | | root: The patch generated
0 new + 72 unchanged - 83 fixed = 72 total (was 155) |
| +1 :green_heart: | mvnsite | 4m 55s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 3m 27s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 40s | | hadoop-project in the patch
passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 2m 4s | | hadoop-common in the patch
passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 0m 49s | |
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new +
39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 46s | | hadoop-azure in the patch
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=708661=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-708661 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 13/Jan/22 21:14 Start Date: 13/Jan/22 21:14 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1012520838 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 708661) Time Spent: 12h (was: 11h 50m) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 12h > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=708503=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-708503 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 13/Jan/22 17:03 Start Date: 13/Jan/22 17:03 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1012330131 mistakenly got the test changes for #3864 in this line; will remove and force the branch up -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 708503) Time Spent: 11h 50m (was: 11h 40m) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 11h 50m > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=708501=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-708501 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 13/Jan/22 17:02 Start Date: 13/Jan/22 17:02 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1011428802 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 708501) Time Spent: 11h 40m (was: 11.5h) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 11h 40m > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=708460=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-708460 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 13/Jan/22 16:26 Start Date: 13/Jan/22 16:26 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1012297514 latest build tested against s3 london, `-Dparallel-tests -DtestsThreadCount=6 -Dmarkers=keep -Dscale` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 708460) Time Spent: 11.5h (was: 11h 20m) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 11.5h > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=708267=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-708267
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 13/Jan/22 12:40
Start Date: 13/Jan/22 12:40
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r783920317
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -3821,23 +3481,14 @@ S3AFileStatus s3GetFileStatus(final Path path,
// by the time of listing so that the response includes some
// which have not.
-int listSize;
-if (tombstones == null) {
- // no tombstones so look for a marker and at least one child.
- listSize = 2;
-} else {
- // build a listing > tombstones. If the caller has many thousands
- // of tombstones this won't work properly, which is why pruning
- // of expired tombstones matters.
- listSize = Math.min(2 + tombstones.size(), Math.max(2, maxKeys));
-}
+final int listSize = 2;
S3ListRequest request = createListObjectsRequest(dirKey, "/",
listSize);
// execute the request
S3ListResult listResult = listObjects(request,
getDurationTrackerFactory());
-if (listResult.hasPrefixesOrObjects(contextAccessors, tombstones)) {
+if (listResult.hasPrefixesOrObjects(contextAccessors, null)) {
Review comment:
well spotted!
i've just gone through looking for all refs to *tombstone*
we can cut all the list reconciliation out, but also various test bits set
up to generate unique paths and so avoid tombstone problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 708267)
Time Spent: 11h 20m (was: 11h 10m)
> Remove S3Guard - no longer needed
> -
>
> Key: HADOOP-17409
> URL: https://issues.apache.org/jira/browse/HADOOP-17409
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 11h 20m
> Remaining Estimate: 0h
>
> With Consistent S3, S3Guard is superfluous.
> stop developing it and wean people off it as soon as they can.
> Then we can worry about what to do in the code. It has gradually insinuated
> its way through the layers, especially things like multi-object delete
> handling (see HADOOP-17244). Things would be a lot simpler without it
> This work is being done in the feature branch HADOOP-17409-remove-s3guard
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=708207=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-708207
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 13/Jan/22 10:45
Start Date: 13/Jan/22 10:45
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r783839500
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RenameOperation.java
##
@@ -724,4 +608,22 @@ private String maybeAddTrailingSlash(String key) {
}
}
+ /**
+ * Convert a passed in exception (expected to be an IOE or AWS exception)
+ * into an IOException.
+ * @param ex exception caught
+ * @return the exception to throw in the failure handler.
+ */
+ protected IOException convertToIOException(final Exception ex) {
+if (ex instanceof IOException) {
+ return (IOException) ex;
Review comment:
so we can return it as an IOE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 708207)
Time Spent: 11h 10m (was: 11h)
> Remove S3Guard - no longer needed
> -
>
> Key: HADOOP-17409
> URL: https://issues.apache.org/jira/browse/HADOOP-17409
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 11h 10m
> Remaining Estimate: 0h
>
> With Consistent S3, S3Guard is superfluous.
> stop developing it and wean people off it as soon as they can.
> Then we can worry about what to do in the code. It has gradually insinuated
> its way through the layers, especially things like multi-object delete
> handling (see HADOOP-17244). Things would be a lot simpler without it
> This work is being done in the feature branch HADOOP-17409-remove-s3guard
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=708206=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-708206
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 13/Jan/22 10:44
Start Date: 13/Jan/22 10:44
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r783839214
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -3821,23 +3481,14 @@ S3AFileStatus s3GetFileStatus(final Path path,
// by the time of listing so that the response includes some
// which have not.
-int listSize;
-if (tombstones == null) {
- // no tombstones so look for a marker and at least one child.
- listSize = 2;
-} else {
- // build a listing > tombstones. If the caller has many thousands
- // of tombstones this won't work properly, which is why pruning
- // of expired tombstones matters.
- listSize = Math.min(2 + tombstones.size(), Math.max(2, maxKeys));
-}
+final int listSize = 2;
S3ListRequest request = createListObjectsRequest(dirKey, "/",
listSize);
// execute the request
S3ListResult listResult = listObjects(request,
getDurationTrackerFactory());
-if (listResult.hasPrefixesOrObjects(contextAccessors, tombstones)) {
+if (listResult.hasPrefixesOrObjects(contextAccessors, null)) {
Review comment:
Probably
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 708206)
Time Spent: 11h (was: 10h 50m)
> Remove S3Guard - no longer needed
> -
>
> Key: HADOOP-17409
> URL: https://issues.apache.org/jira/browse/HADOOP-17409
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 11h
> Remaining Estimate: 0h
>
> With Consistent S3, S3Guard is superfluous.
> stop developing it and wean people off it as soon as they can.
> Then we can worry about what to do in the code. It has gradually insinuated
> its way through the layers, especially things like multi-object delete
> handling (see HADOOP-17244). Things would be a lot simpler without it
> This work is being done in the feature branch HADOOP-17409-remove-s3guard
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=708156=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-708156
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 13/Jan/22 09:58
Start Date: 13/Jan/22 09:58
Worklog Time Spent: 10m
Work Description: mukund-thakur commented on a change in pull request
#3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r783712717
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RenameOperation.java
##
@@ -724,4 +608,22 @@ private String maybeAddTrailingSlash(String key) {
}
}
+ /**
+ * Convert a passed in exception (expected to be an IOE or AWS exception)
+ * into an IOException.
+ * @param ex exception caught
+ * @return the exception to throw in the failure handler.
+ */
+ protected IOException convertToIOException(final Exception ex) {
+if (ex instanceof IOException) {
+ return (IOException) ex;
Review comment:
Why casting again?
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -3821,23 +3481,14 @@ S3AFileStatus s3GetFileStatus(final Path path,
// by the time of listing so that the response includes some
// which have not.
-int listSize;
-if (tombstones == null) {
- // no tombstones so look for a marker and at least one child.
- listSize = 2;
-} else {
- // build a listing > tombstones. If the caller has many thousands
- // of tombstones this won't work properly, which is why pruning
- // of expired tombstones matters.
- listSize = Math.min(2 + tombstones.size(), Math.max(2, maxKeys));
-}
+final int listSize = 2;
S3ListRequest request = createListObjectsRequest(dirKey, "/",
listSize);
// execute the request
S3ListResult listResult = listObjects(request,
getDurationTrackerFactory());
-if (listResult.hasPrefixesOrObjects(contextAccessors, tombstones)) {
+if (listResult.hasPrefixesOrObjects(contextAccessors, null)) {
Review comment:
Can we remove the tombstones field altogether? I suppose this was only
for S3Guard entires.
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -3316,31 +3111,19 @@ protected void maybeCreateFakeParentDirectory(Path path)
}
}
// Here we have a directory which may or may not be empty.
-// So we update the metastore and return.
-return S3Guard.dirListingUnion(
-metadataStore,
-path,
-statusesAssumingNonEmptyDir.getLeft(),
-statusesAssumingNonEmptyDir.getMiddle(),
-allowAuthoritative(path),
-ttlTimeProvider, p ->
-listing.createProvidedFileStatusIterator(
-dirMetaToStatuses(statusesAssumingNonEmptyDir.getMiddle()),
-ACCEPT_ALL,
-Listing.ACCEPT_ALL_BUT_S3N));
+return statusIt;
}
/**
* Is a path to be considered as authoritative?
- * True iff there is an authoritative metastore or if there
- * is a non-auth store with the supplied path under
+ * is a store with the supplied path under
* one of the paths declared as authoritative.
* @param path path
* @return true if the path is auth
*/
public boolean allowAuthoritative(final Path path) {
return S3Guard.allowAuthoritative(path, this,
-allowAuthoritativeMetadataStore, allowAuthoritativePaths);
+allowAuthoritativePaths);
Review comment:
Should this be deprecated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 708156)
Time Spent: 10h 50m (was: 10h 40m)
> Remove S3Guard - no longer needed
> -
>
> Key: HADOOP-17409
> URL: https://issues.apache.org/jira/browse/HADOOP-17409
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10h 50m
> Remaining Estimate: 0h
>
> With Consistent S3, S3Guard is superfluous.
> stop developing it and wean people off it as soon as they can.
> Then we can worry about
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=708152=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-708152 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 13/Jan/22 09:52 Start Date: 13/Jan/22 09:52 Worklog Time Spent: 10m Work Description: mukund-thakur edited a comment on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1011970010 > `[ERROR] Failures: [ERROR] ITestS3AMiscOperations.testEmptyFileChecksums:182->Assert.assertEquals:120->Assert.failNotEquals:835->Assert.fail:89 checksums of empty files expected: but was: [ERROR] ITestS3AMiscOperations.testNonEmptyFileChecksumsUnencrypted:258->Assert.assertEquals:120->Assert.failNotEquals:835->Assert.fail:89 checksums expected: but was: [ERROR] Errors: ` > > Also these failures not sure if they are related. This is failing in trunk as well even after removing the encryption configs from the auth-keys. Although I have enabled the encryption as SSE-KMS in aws console. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 708152) Time Spent: 10h 40m (was: 10.5h) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 10h 40m > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=708149=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-708149 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 13/Jan/22 09:50 Start Date: 13/Jan/22 09:50 Worklog Time Spent: 10m Work Description: mukund-thakur commented on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1011970010 > `[ERROR] Failures: [ERROR] ITestS3AMiscOperations.testEmptyFileChecksums:182->Assert.assertEquals:120->Assert.failNotEquals:835->Assert.fail:89 checksums of empty files expected: but was: [ERROR] ITestS3AMiscOperations.testNonEmptyFileChecksumsUnencrypted:258->Assert.assertEquals:120->Assert.failNotEquals:835->Assert.fail:89 checksums expected: but was: [ERROR] Errors: ` > > Also these failures not sure if they are related. This is failing in trunk as well even after removing the encryption configs from the auth-keys. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 708149) Time Spent: 10.5h (was: 10h 20m) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 10.5h > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=707946=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-707946
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 12/Jan/22 23:09
Start Date: 12/Jan/22 23:09
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1011534353
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 109 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 59s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 59s | | trunk passed |
| +1 :green_heart: | compile | 24m 17s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 54s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 57s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 50s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 26s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 32s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 24m 22s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 22s | | the patch passed |
| +1 :green_heart: | compile | 23m 29s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 23m 29s |
[/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/15/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt)
| root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 5 new + 1813 unchanged - 51
fixed = 1818 total (was 1864) |
| +1 :green_heart: | compile | 20m 46s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 20m 46s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/15/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 5 new + 1688
unchanged - 51 fixed = 1693 total (was 1739) |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/15/artifact/out/blanks-eol.txt)
| The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 3m 49s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/15/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 8 new + 71 unchanged - 84 fixed = 79 total (was
155) |
| +1 :green_heart: | mvnsite | 3m 44s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 2m 46s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 27s | | hadoop-project in the patch
passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 1m 39s | | hadoop-common in the patch
passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=707864=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-707864
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 12/Jan/22 20:34
Start Date: 12/Jan/22 20:34
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1011428802
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 3s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 109 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 55m 46s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 11s | | trunk passed |
| +1 :green_heart: | compile | 24m 16s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 43s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 51s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 21s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 46s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 23m 54s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 19s | | the patch passed |
| +1 :green_heart: | compile | 23m 26s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 23m 26s |
[/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/14/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt)
| root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 7 new + 1813 unchanged - 51
fixed = 1820 total (was 1864) |
| +1 :green_heart: | compile | 20m 42s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 20m 42s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/14/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 7 new + 1688
unchanged - 51 fixed = 1695 total (was 1739) |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/14/artifact/out/blanks-eol.txt)
| The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 3m 47s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/14/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 8 new + 71 unchanged - 84 fixed = 79 total (was
155) |
| +1 :green_heart: | mvnsite | 3m 41s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 2m 40s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 28s | | hadoop-project in the patch
passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 1m 38s | | hadoop-common in the patch
passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=707808=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-707808 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 12/Jan/22 19:03 Start Date: 12/Jan/22 19:03 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1011360617 @mukund-thakur sounds like your encryption setting settings are causing problems. does this happen on other branches? maybe I should rebase this one -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 707808) Time Spent: 10h (was: 9h 50m) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 10h > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=707807=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-707807 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 12/Jan/22 19:02 Start Date: 12/Jan/22 19:02 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1011359446 new pr up; testing s3 london. `-Dparallel-tests -DtestsThreadCount=6 -Dmarkers=delete -Dscale`. functional but slow (!) ``` [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 44:39 min (Wall Clock) [INFO] Finished at: 2022-01-12T16:29:50Z ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 707807) Time Spent: 9h 50m (was: 9h 40m) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 9h 50m > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=707805=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-707805
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 12/Jan/22 18:59
Start Date: 12/Jan/22 18:59
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1011225626
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 8s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 108 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 56s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 42s | | trunk passed |
| +1 :green_heart: | compile | 24m 16s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 34s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 57s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 49s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 25s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 33s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 23m 51s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 19s | | the patch passed |
| +1 :green_heart: | compile | 28m 14s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 28m 14s |
[/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/13/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt)
| root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 4 new + 1813 unchanged - 51
fixed = 1817 total (was 1864) |
| +1 :green_heart: | compile | 22m 36s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 22m 36s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/13/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 4 new + 1688
unchanged - 51 fixed = 1692 total (was 1739) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 54s | | root: The patch generated
0 new + 71 unchanged - 84 fixed = 71 total (was 155) |
| +1 :green_heart: | mvnsite | 3m 45s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 2m 43s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 27s | | hadoop-project in the patch
passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 1m 38s | | hadoop-common in the patch
passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 0m 39s | |
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new +
39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 38s | | hadoop-azure in the
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=707804=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-707804
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 12/Jan/22 18:59
Start Date: 12/Jan/22 18:59
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1010549488
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 108 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 19s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 1s | | trunk passed |
| +1 :green_heart: | compile | 24m 35s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 41s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 56s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 54s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 24s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 33s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 23m 49s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 36s |
[/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| -1 :x: | compile | 22m 10s |
[/patch-compile-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-compile-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt)
| root in the patch failed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04. |
| -1 :x: | javac | 22m 10s |
[/patch-compile-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-compile-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt)
| root in the patch failed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04. |
| -1 :x: | compile | 19m 44s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root in the patch failed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | javac | 19m 44s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root in the patch failed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 16s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 71 unchanged - 84 fixed = 72 total (was
155) |
| -1 :x: | mvnsite | 0m 38s |
[/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 2m
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=707698=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-707698 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 12/Jan/22 17:37 Start Date: 12/Jan/22 17:37 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1011292027 and warnings ``` hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/CommitOperations.java:436:30:[deprecation] METADATASTORE_AUTHORITATIVE in Constants has been deprecated ``` it has been adding some metastore info to the job commit. i will cut it all out -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 707698) Time Spent: 9h 20m (was: 9h 10m) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 9h 20m > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=707692=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-707692 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 12/Jan/22 17:35 Start Date: 12/Jan/22 17:35 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1011290633 hadoop common failure is testIpc -covered in HADOOP-18046 and unrelated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 707692) Time Spent: 9h 10m (was: 9h) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 9h 10m > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=707628=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-707628
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 12/Jan/22 16:26
Start Date: 12/Jan/22 16:26
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1011225626
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 8s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 108 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 56s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 42s | | trunk passed |
| +1 :green_heart: | compile | 24m 16s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 34s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 57s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 49s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 25s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 33s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 23m 51s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 19s | | the patch passed |
| +1 :green_heart: | compile | 28m 14s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 28m 14s |
[/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/13/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt)
| root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 4 new + 1813 unchanged - 51
fixed = 1817 total (was 1864) |
| +1 :green_heart: | compile | 22m 36s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 22m 36s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/13/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 4 new + 1688
unchanged - 51 fixed = 1692 total (was 1739) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 54s | | root: The patch generated
0 new + 71 unchanged - 84 fixed = 71 total (was 155) |
| +1 :green_heart: | mvnsite | 3m 45s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 2m 43s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 27s | | hadoop-project in the patch
passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 1m 38s | | hadoop-common in the patch
passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | javadoc | 0m 39s | |
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new +
39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 38s | | hadoop-azure in the patch
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=707596=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-707596 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 12/Jan/22 15:57 Start Date: 12/Jan/22 15:57 Worklog Time Spent: 10m Work Description: mukund-thakur commented on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1011191687 `[ERROR] Failures: [ERROR] ITestS3AMiscOperations.testEmptyFileChecksums:182->Assert.assertEquals:120->Assert.failNotEquals:835->Assert.fail:89 checksums of empty files expected: but was: [ERROR] ITestS3AMiscOperations.testNonEmptyFileChecksumsUnencrypted:258->Assert.assertEquals:120->Assert.failNotEquals:835->Assert.fail:89 checksums expected: but was: [ERROR] Errors: ` Also these failures not sure if they are related. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 707596) Time Spent: 8h 50m (was: 8h 40m) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 8h 50m > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=707595=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-707595 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 12/Jan/22 15:56 Start Date: 12/Jan/22 15:56 Worklog Time Spent: 10m Work Description: mukund-thakur commented on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1011191145 `[ERROR] Tests run: 15, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 31.898 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardTool [ERROR] testLandsatBucketRequireUnencrypted(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardTool) Time elapsed: 2.139 s <<< ERROR! 46: Bucket s3a://landsat-pds: required encryption is none but actual encryption is CSE-KMS at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.exitException(S3GuardTool.java:914) at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.badState(S3GuardTool.java:880) at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$BucketInfo.run(S3GuardTool.java:509) at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:282) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81) at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:962) at org.apache.hadoop.fs.s3a.s3guard.S3GuardToolTestHelper.runS3GuardCommand(S3GuardToolTestHelper.java:147) at org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.run(AbstractS3GuardToolTestBase.java:124) at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardTool.testLandsatBucketRequireUnencrypted(ITestS3GuardTool.java:73)` Seeing this failure in my setup while raw S3 tests. Shouldn't this test be removed ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 707595) Time Spent: 8h 40m (was: 8.5h) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 8h 40m > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=707429=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-707429 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 12/Jan/22 12:04 Start Date: 12/Jan/22 12:04 Worklog Time Spent: 10m Work Description: mehakmeet commented on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1010976232 +1 pending the yetus checks, tested on non-CSE and CSE test suite, everything went fine and getting the correct "not supported" error when turning s3guard on. One suggestion would be maybe referencing this Jira ticket in `s3guard.md` where we are explaining that it was removed? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 707429) Time Spent: 8.5h (was: 8h 20m) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 8.5h > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=707391=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-707391
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 12/Jan/22 10:36
Start Date: 12/Jan/22 10:36
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1010898339
test run s3 london. `-Dparallel-tests -DtestsThreadCount=6 -Dmarkers=delete
-Dscale`
one failure from DT tests as I'd turned session tokens on there, otherwise
all good. filing a jira to get the failing test to clear more options
```
[ERROR] Failures:
[ERROR]
ITestDelegatedMRJob.testJobSubmissionCollectsTokens:303->AbstractDelegationIT.lookupToken:70->Assert.assertEquals:120->Assert.failNotEquals:835->Assert.fail:89
Kind of token Kind: S3ADelegationToken/Session, Service: s3a://stevel-london,
Ident: (S3ATokenIdentifier{S3ADelegationToken/Session; uri=s3a://stevel-london;
timestamp=1641943367014; renewer=stevel; encryption=SSE-KMS;
df39af66-6a55-4d84-9ce0-7be719f58087; Created on stevel-mbp1376/127.0.0.1 at
time 2022-01-11T23:22:46.805Z.}; session credentials, expiry
2022-01-12T11:22:47Z; (valid)) expected: but
was:
[ERROR]
ITestDelegatedMRJob.testJobSubmissionCollectsTokens:303->AbstractDelegationIT.lookupToken:70->Assert.assertEquals:120->Assert.failNotEquals:835->Assert.fail:89
Kind of token Kind: S3ADelegationToken/Session, Service: s3a://stevel-london,
Ident: (S3ATokenIdentifier{S3ADelegationToken/Session; uri=s3a://stevel-london;
timestamp=1641943383624; renewer=stevel; encryption=SSE-KMS;
fcf63971-7249-4339-92c1-89a508fee7d2; Created on stevel-mbp1376/127.0.0.1 at
time 2022-01-11T23:23:03.247Z.}; session credentials, expiry
2022-01-12T11:23:03Z; (valid)) expected: but
was:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 707391)
Time Spent: 8h 20m (was: 8h 10m)
> Remove S3Guard - no longer needed
> -
>
> Key: HADOOP-17409
> URL: https://issues.apache.org/jira/browse/HADOOP-17409
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 8h 20m
> Remaining Estimate: 0h
>
> With Consistent S3, S3Guard is superfluous.
> stop developing it and wean people off it as soon as they can.
> Then we can worry about what to do in the code. It has gradually insinuated
> its way through the layers, especially things like multi-object delete
> handling (see HADOOP-17244). Things would be a lot simpler without it
> This work is being done in the feature branch HADOOP-17409-remove-s3guard
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=707217=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-707217
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 12/Jan/22 01:51
Start Date: 12/Jan/22 01:51
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1010549488
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 108 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 19s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 1s | | trunk passed |
| +1 :green_heart: | compile | 24m 35s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 41s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 56s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 54s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 24s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 33s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 23m 49s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 36s |
[/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| -1 :x: | compile | 22m 10s |
[/patch-compile-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-compile-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt)
| root in the patch failed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04. |
| -1 :x: | javac | 22m 10s |
[/patch-compile-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-compile-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt)
| root in the patch failed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04. |
| -1 :x: | compile | 19m 44s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root in the patch failed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | javac | 19m 44s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root in the patch failed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 16s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 71 unchanged - 84 fixed = 72 total (was
155) |
| -1 :x: | mvnsite | 0m 38s |
[/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/12/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 2m 42s | |
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=707144=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-707144 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 11/Jan/22 21:44 Start Date: 11/Jan/22 21:44 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-1010386405 hboss is just using the config options to switch to local s3guard in tests. we leave the options alone but: should we treat asking for local as a no op? because they call it to get a consistent s3 store...there's no risk of ddb tables corrupting the view of other processes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 707144) Time Spent: 8h (was: 7h 50m) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 8h > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=707142=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-707142 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 11/Jan/22 21:42 Start Date: 11/Jan/22 21:42 Worklog Time Spent: 10m Work Description: steveloughran commented on a change in pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#discussion_r782546599 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md ## @@ -343,7 +343,7 @@ stores pretend that they are a FileSystem, a FileSystem with the same features and operations as HDFS. This is ultimatelya pretence: they have different characteristics and occasionally the illusion fails. -1. **Consistency**. Object stores are generally *Eventually Consistent*: it +1. **Consistency**. Object mqy be *Eventually Consistent*: it Review comment: fixed. thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 707142) Time Spent: 7h 40m (was: 7.5h) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 7h 40m > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=707143=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-707143
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 11/Jan/22 21:42
Start Date: 11/Jan/22 21:42
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-977215558
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 6s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 3s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 107 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 46s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 9s | | trunk passed |
| +1 :green_heart: | compile | 23m 37s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 20m 8s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 56s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 44s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 19s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 33s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 23m 35s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 16s | | the patch passed |
| +1 :green_heart: | compile | 22m 54s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 22m 54s |
[/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/11/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 1888 unchanged - 5
fixed = 1889 total (was 1893) |
| +1 :green_heart: | compile | 20m 13s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 20m 13s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/11/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new + 1764
unchanged - 5 fixed = 1765 total (was 1769) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 51s | | root: The patch generated
0 new + 71 unchanged - 83 fixed = 71 total (was 154) |
| +1 :green_heart: | mvnsite | 3m 38s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 2m 41s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 27s | | hadoop-project in the patch
passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | javadoc | 1m 37s | | hadoop-common in the patch
passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | javadoc | 0m 38s | |
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 0 new +
39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 36s | | hadoop-azure in the patch
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=707139=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-707139
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 11/Jan/22 21:36
Start Date: 11/Jan/22 21:36
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r782538228
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -4284,31 +3929,8 @@ private CopyResult copyFile(String srcKey, String
dstKey, long size,
String action = "copyFile(" + srcKey + ", " + dstKey + ")";
Invoker readInvoker = readContext.getReadInvoker();
-ObjectMetadata srcom;
-try {
- srcom = once(action, srcKey,
- () ->
- getObjectMetadata(srcKey, changeTracker, readInvoker, "copy"));
-} catch (FileNotFoundException e) {
- // if rename fails at this point it means that the expected file was not
- // found.
- // The cause is believed to always be one of
- // - File was deleted since LIST/S3Guard metastore.list.() knew of it.
- // - S3Guard is asking for a specific version and it's been removed by
- //lifecycle rules.
- // - there's a 404 cached in the S3 load balancers.
- LOG.debug("getObjectMetadata({}) failed to find an expected file",
- srcKey, e);
- // We create an exception, but the text depends on the S3Guard state
- String message = hasMetadataStore()
Review comment:
this is during a rename where the list call found the file, now we call
copy and it's not there.
the causes used to be
1. s3guard inconsistent
2. unguarded and 404 caching making the file not visible
3. something else deleted the file between list and copy
only #3 can happen now, so i was just leaving as it.
but I could revert, as we may want to differentiate the failure.
will revert
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 707139)
Time Spent: 7.5h (was: 7h 20m)
> Remove S3Guard - no longer needed
> -
>
> Key: HADOOP-17409
> URL: https://issues.apache.org/jira/browse/HADOOP-17409
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> With Consistent S3, S3Guard is superfluous.
> stop developing it and wean people off it as soon as they can.
> Then we can worry about what to do in the code. It has gradually insinuated
> its way through the layers, especially things like multi-object delete
> handling (see HADOOP-17244). Things would be a lot simpler without it
> This work is being done in the feature branch HADOOP-17409-remove-s3guard
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=707135=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-707135
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 11/Jan/22 21:28
Start Date: 11/Jan/22 21:28
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r782538228
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -4284,31 +3929,8 @@ private CopyResult copyFile(String srcKey, String
dstKey, long size,
String action = "copyFile(" + srcKey + ", " + dstKey + ")";
Invoker readInvoker = readContext.getReadInvoker();
-ObjectMetadata srcom;
-try {
- srcom = once(action, srcKey,
- () ->
- getObjectMetadata(srcKey, changeTracker, readInvoker, "copy"));
-} catch (FileNotFoundException e) {
- // if rename fails at this point it means that the expected file was not
- // found.
- // The cause is believed to always be one of
- // - File was deleted since LIST/S3Guard metastore.list.() knew of it.
- // - S3Guard is asking for a specific version and it's been removed by
- //lifecycle rules.
- // - there's a 404 cached in the S3 load balancers.
- LOG.debug("getObjectMetadata({}) failed to find an expected file",
- srcKey, e);
- // We create an exception, but the text depends on the S3Guard state
- String message = hasMetadataStore()
Review comment:
we always generated different messages when s3guard was on/off, didn;t
we?
now you only get the unguarded text
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 707135)
Time Spent: 7h 20m (was: 7h 10m)
> Remove S3Guard - no longer needed
> -
>
> Key: HADOOP-17409
> URL: https://issues.apache.org/jira/browse/HADOOP-17409
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
> With Consistent S3, S3Guard is superfluous.
> stop developing it and wean people off it as soon as they can.
> Then we can worry about what to do in the code. It has gradually insinuated
> its way through the layers, especially things like multi-object delete
> handling (see HADOOP-17244). Things would be a lot simpler without it
> This work is being done in the feature branch HADOOP-17409-remove-s3guard
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=706840=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-706840
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 11/Jan/22 13:32
Start Date: 11/Jan/22 13:32
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r782145091
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java
##
@@ -422,7 +315,7 @@ public void checkBasicFileOperations() throws Throwable {
readonlyFS.getFileStatus(emptyDir);
// now look at a file; the outcome depends on the mode.
-accessDeniedIf(!guardedInAuthMode, () ->
+accessDeniedIf(!false, () ->
Review comment:
fixedl and updated the comment above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 706840)
Time Spent: 7h 10m (was: 7h)
> Remove S3Guard - no longer needed
> -
>
> Key: HADOOP-17409
> URL: https://issues.apache.org/jira/browse/HADOOP-17409
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> With Consistent S3, S3Guard is superfluous.
> stop developing it and wean people off it as soon as they can.
> Then we can worry about what to do in the code. It has gradually insinuated
> its way through the layers, especially things like multi-object delete
> handling (see HADOOP-17244). Things would be a lot simpler without it
> This work is being done in the feature branch HADOOP-17409-remove-s3guard
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=693310=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-693310
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 09/Dec/21 15:06
Start Date: 09/Dec/21 15:06
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r765877448
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Listing.java
##
@@ -279,82 +205,19 @@ private ObjectListingIterator createObjectListingIterator(
LOG.debug("Requesting all entries under {} with delimiter '{}'",
key, delimiter);
}
-final RemoteIterator cachedFilesIterator;
-final Set tombstones;
-boolean allowAuthoritative = listingOperationCallbacks
-.allowAuthoritative(path);
-if (recursive) {
- final PathMetadata pm = getStoreContext()
- .getMetadataStore()
- .get(path, true);
- if (pm != null) {
-if (pm.isDeleted()) {
- OffsetDateTime deletedAt = OffsetDateTime
- .ofInstant(Instant.ofEpochMilli(
- pm.getFileStatus().getModificationTime()),
- ZoneOffset.UTC);
- throw new FileNotFoundException("Path " + path + " is recorded as " +
- "deleted by S3Guard at " + deletedAt);
-}
- }
- MetadataStoreListFilesIterator metadataStoreListFilesIterator =
- new MetadataStoreListFilesIterator(
- getStoreContext().getMetadataStore(),
- pm,
- allowAuthoritative);
- tombstones = metadataStoreListFilesIterator.listTombstones();
- // if all of the below is true
- // - authoritative access is allowed for this metadatastore
- // for this directory,
- // - all the directory listings are authoritative on the client
- // - the caller does not force non-authoritative access
- // return the listing without any further s3 access
- if (!forceNonAuthoritativeMS &&
- allowAuthoritative &&
- metadataStoreListFilesIterator.isRecursivelyAuthoritative()) {
-S3AFileStatus[] statuses = S3AUtils.iteratorToStatuses(
-metadataStoreListFilesIterator, tombstones);
-cachedFilesIterator = createProvidedFileStatusIterator(
-statuses, ACCEPT_ALL, acceptor);
-return createLocatedFileStatusIterator(cachedFilesIterator);
- }
- cachedFilesIterator = metadataStoreListFilesIterator;
-} else {
- DirListingMetadata meta =
- S3Guard.listChildrenWithTtl(
- getStoreContext().getMetadataStore(),
- path,
- listingOperationCallbacks.getUpdatedTtlTimeProvider(),
- allowAuthoritative);
- if (meta != null) {
-tombstones = meta.listTombstones();
- } else {
-tombstones = null;
- }
- cachedFilesIterator = createProvidedFileStatusIterator(
- S3Guard.dirMetaToStatuses(meta), ACCEPT_ALL, acceptor);
- if (allowAuthoritative && meta != null && meta.isAuthoritative()) {
-// metadata listing is authoritative, so return it directly
-return createLocatedFileStatusIterator(cachedFilesIterator);
- }
-}
-return createTombstoneReconcilingIterator(
-createLocatedFileStatusIterator(
-createFileStatusListingIterator(path,
-listingOperationCallbacks
-.createListObjectsRequest(key,
-delimiter,
-span),
-ACCEPT_ALL,
-acceptor,
-cachedFilesIterator,
-span)),
-collectTombstones ? tombstones : null);
+return createLocatedFileStatusIterator(
Review comment:

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 693310)
Time Spent: 7h (was: 6h 50m)
> Remove S3Guard - no longer needed
> -
>
> Key: HADOOP-17409
> URL: https://issues.apache.org/jira/browse/HADOOP-17409
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=685766=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685766
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 24/Nov/21 10:18
Start Date: 24/Nov/21 10:18
Worklog Time Spent: 10m
Work Description: mukund-thakur commented on a change in pull request
#3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r754893451
##
File path:
hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md
##
@@ -343,7 +343,7 @@ stores pretend that they are a FileSystem, a FileSystem
with the same
features and operations as HDFS. This is ultimatelya pretence:
they have different characteristics and occasionally the illusion fails.
-1. **Consistency**. Object stores are generally *Eventually Consistent*: it
+1. **Consistency**. Object mqy be *Eventually Consistent*: it
Review comment:
nit : typo mqy -> may
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Listing.java
##
@@ -279,82 +205,19 @@ private ObjectListingIterator createObjectListingIterator(
LOG.debug("Requesting all entries under {} with delimiter '{}'",
key, delimiter);
}
-final RemoteIterator cachedFilesIterator;
-final Set tombstones;
-boolean allowAuthoritative = listingOperationCallbacks
-.allowAuthoritative(path);
-if (recursive) {
- final PathMetadata pm = getStoreContext()
- .getMetadataStore()
- .get(path, true);
- if (pm != null) {
-if (pm.isDeleted()) {
- OffsetDateTime deletedAt = OffsetDateTime
- .ofInstant(Instant.ofEpochMilli(
- pm.getFileStatus().getModificationTime()),
- ZoneOffset.UTC);
- throw new FileNotFoundException("Path " + path + " is recorded as " +
- "deleted by S3Guard at " + deletedAt);
-}
- }
- MetadataStoreListFilesIterator metadataStoreListFilesIterator =
- new MetadataStoreListFilesIterator(
- getStoreContext().getMetadataStore(),
- pm,
- allowAuthoritative);
- tombstones = metadataStoreListFilesIterator.listTombstones();
- // if all of the below is true
- // - authoritative access is allowed for this metadatastore
- // for this directory,
- // - all the directory listings are authoritative on the client
- // - the caller does not force non-authoritative access
- // return the listing without any further s3 access
- if (!forceNonAuthoritativeMS &&
- allowAuthoritative &&
- metadataStoreListFilesIterator.isRecursivelyAuthoritative()) {
-S3AFileStatus[] statuses = S3AUtils.iteratorToStatuses(
-metadataStoreListFilesIterator, tombstones);
-cachedFilesIterator = createProvidedFileStatusIterator(
-statuses, ACCEPT_ALL, acceptor);
-return createLocatedFileStatusIterator(cachedFilesIterator);
- }
- cachedFilesIterator = metadataStoreListFilesIterator;
-} else {
- DirListingMetadata meta =
- S3Guard.listChildrenWithTtl(
- getStoreContext().getMetadataStore(),
- path,
- listingOperationCallbacks.getUpdatedTtlTimeProvider(),
- allowAuthoritative);
- if (meta != null) {
-tombstones = meta.listTombstones();
- } else {
-tombstones = null;
- }
- cachedFilesIterator = createProvidedFileStatusIterator(
- S3Guard.dirMetaToStatuses(meta), ACCEPT_ALL, acceptor);
- if (allowAuthoritative && meta != null && meta.isAuthoritative()) {
-// metadata listing is authoritative, so return it directly
-return createLocatedFileStatusIterator(cachedFilesIterator);
- }
-}
-return createTombstoneReconcilingIterator(
-createLocatedFileStatusIterator(
-createFileStatusListingIterator(path,
-listingOperationCallbacks
-.createListObjectsRequest(key,
-delimiter,
-span),
-ACCEPT_ALL,
-acceptor,
-cachedFilesIterator,
-span)),
-collectTombstones ? tombstones : null);
+return createLocatedFileStatusIterator(
Review comment:
Wow. Listing is super simple now. ï’¯
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -4284,31 +3929,8 @@ private CopyResult
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=685756=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685756 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 24/Nov/21 10:00 Start Date: 24/Nov/21 10:00 Worklog Time Spent: 10m Work Description: harshavardhana commented on a change in pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#discussion_r755876603 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md ## @@ -343,7 +343,7 @@ stores pretend that they are a FileSystem, a FileSystem with the same features and operations as HDFS. This is ultimatelya pretence: they have different characteristics and occasionally the illusion fails. -1. **Consistency**. Object stores are generally *Eventually Consistent*: it +1. **Consistency**. Object mqy be *Eventually Consistent*: it Review comment: ```suggestion 1. **Consistency**. Object may be *Eventually Consistent*: it ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 685756) Time Spent: 6h 40m (was: 6.5h) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 6h 40m > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=685527=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685527
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 23/Nov/21 22:12
Start Date: 23/Nov/21 22:12
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-977215558
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 6s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 3s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 107 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 46s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 9s | | trunk passed |
| +1 :green_heart: | compile | 23m 37s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 20m 8s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 56s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 44s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 19s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 33s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 23m 35s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 16s | | the patch passed |
| +1 :green_heart: | compile | 22m 54s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 22m 54s |
[/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/11/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 1888 unchanged - 5
fixed = 1889 total (was 1893) |
| +1 :green_heart: | compile | 20m 13s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 20m 13s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/11/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new + 1764
unchanged - 5 fixed = 1765 total (was 1769) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 51s | | root: The patch generated
0 new + 71 unchanged - 83 fixed = 71 total (was 154) |
| +1 :green_heart: | mvnsite | 3m 38s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 2m 41s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 27s | | hadoop-project in the patch
passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | javadoc | 1m 37s | | hadoop-common in the patch
passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | javadoc | 0m 38s | |
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 0 new +
39 unchanged - 23 fixed = 39 total (was 62) |
| +1 :green_heart: | javadoc | 0m 36s | | hadoop-azure in the patch
passed
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=685445=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685445
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 23/Nov/21 18:07
Start Date: 23/Nov/21 18:07
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-976828074
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 107 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 44s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 41s | | trunk passed |
| +1 :green_heart: | compile | 28m 56s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 23m 36s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 14s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 20s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 19s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 54s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 39s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 25m 0s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 52s | | the patch passed |
| +1 :green_heart: | compile | 29m 21s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 29m 21s |
[/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/10/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 4 new + 1883 unchanged - 8
fixed = 1887 total (was 1891) |
| +1 :green_heart: | compile | 25m 36s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 25m 36s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/10/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 4 new + 1758
unchanged - 8 fixed = 1762 total (was 1766) |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/10/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 4m 20s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/10/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 3 new + 71 unchanged - 83 fixed = 74 total (was
154) |
| +1 :green_heart: | mvnsite | 4m 31s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 3m 29s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 34s | | hadoop-project in the patch
passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | javadoc | 2m 4s | | hadoop-common in the patch
passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=685444=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685444
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 23/Nov/21 18:07
Start Date: 23/Nov/21 18:07
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-975888432
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 5s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 107 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 56s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 21s | | trunk passed |
| +1 :green_heart: | compile | 25m 18s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 21m 9s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 33s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 4m 8s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 44s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 21m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 35s | | the patch passed |
| +1 :green_heart: | compile | 23m 51s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 23m 51s |
[/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/9/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 4 new + 1883 unchanged - 8
fixed = 1887 total (was 1891) |
| +1 :green_heart: | compile | 21m 21s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 21m 21s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/9/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 4 new + 1758
unchanged - 8 fixed = 1762 total (was 1766) |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/9/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | checkstyle | 3m 49s | | root: The patch generated
0 new + 71 unchanged - 83 fixed = 71 total (was 154) |
| +1 :green_heart: | mvnsite | 4m 29s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 7s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 3m 32s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 40s | | hadoop-project in the patch
passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | javadoc | 1m 49s | | hadoop-common in the patch
passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | javadoc | 0m 52s | |
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
with JDK Private
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=685443=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685443 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 23/Nov/21 18:06 Start Date: 23/Nov/21 18:06 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-974899616 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 685443) Time Spent: 6h (was: 5h 50m) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 6h > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=685413=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685413
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 23/Nov/21 16:40
Start Date: 23/Nov/21 16:40
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-976828074
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 107 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 44s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 41s | | trunk passed |
| +1 :green_heart: | compile | 28m 56s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 23m 36s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 14s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 20s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 19s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 54s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 39s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 25m 0s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 52s | | the patch passed |
| +1 :green_heart: | compile | 29m 21s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 29m 21s |
[/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/10/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 4 new + 1883 unchanged - 8
fixed = 1887 total (was 1891) |
| +1 :green_heart: | compile | 25m 36s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 25m 36s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/10/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 4 new + 1758
unchanged - 8 fixed = 1762 total (was 1766) |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/10/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 4m 20s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/10/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 3 new + 71 unchanged - 83 fixed = 74 total (was
154) |
| +1 :green_heart: | mvnsite | 4m 31s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 3m 29s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 34s | | hadoop-project in the patch
passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | javadoc | 2m 4s | | hadoop-common in the patch
passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=685291=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685291 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 23/Nov/21 14:07 Start Date: 23/Nov/21 14:07 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-976586826 HBoss contains references to local metastore & ability to set it in hadoop-testutils/src/main/java/org/apache/hadoop/hbase/oss/EmbeddedS3.java maybe we should make that a no-op. or at least fix up hboss to never use it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 685291) Time Spent: 5h 40m (was: 5.5h) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 5h 40m > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=684952=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-684952
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 22/Nov/21 20:23
Start Date: 22/Nov/21 20:23
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-975888432
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 5s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 107 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 56s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 21s | | trunk passed |
| +1 :green_heart: | compile | 25m 18s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 21m 9s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 33s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 4m 8s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 44s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 21m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 35s | | the patch passed |
| +1 :green_heart: | compile | 23m 51s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 23m 51s |
[/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/9/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 4 new + 1883 unchanged - 8
fixed = 1887 total (was 1891) |
| +1 :green_heart: | compile | 21m 21s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 21m 21s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/9/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 4 new + 1758
unchanged - 8 fixed = 1762 total (was 1766) |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/9/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | checkstyle | 3m 49s | | root: The patch generated
0 new + 71 unchanged - 83 fixed = 71 total (was 154) |
| +1 :green_heart: | mvnsite | 4m 29s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 7s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 3m 32s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 40s | | hadoop-project in the patch
passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | javadoc | 1m 49s | | hadoop-common in the patch
passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | javadoc | 0m 52s | |
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
with JDK Private
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=684655=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-684655 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 22/Nov/21 12:23 Start Date: 22/Nov/21 12:23 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-975468562 minor spotbugs to fix up -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 684655) Time Spent: 5h 20m (was: 5h 10m) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 5h 20m > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=684473=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-684473
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 21/Nov/21 21:33
Start Date: 21/Nov/21 21:33
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-974899616
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 107 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 32s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 34s | | trunk passed |
| +1 :green_heart: | compile | 23m 38s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 20m 10s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 54s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 37s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 14s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | spotbugs | 2m 27s |
[/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/8/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html)
| hadoop-common-project/hadoop-common in trunk has 1 extant spotbugs
warnings. |
| +1 :green_heart: | shadedclient | 23m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 33s | | the patch passed |
| +1 :green_heart: | compile | 22m 50s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 22m 50s |
[/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/8/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 1888 unchanged - 5
fixed = 1889 total (was 1893) |
| +1 :green_heart: | compile | 20m 8s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 20m 8s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/8/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new + 1764
unchanged - 5 fixed = 1765 total (was 1769) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 52s | | root: The patch generated
0 new + 71 unchanged - 83 fixed = 71 total (was 154) |
| +1 :green_heart: | mvnsite | 2m 23s | | the patch passed |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 1m 34s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 36s | | hadoop-common in the patch
passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | javadoc | 0m 38s | |
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 0 new +
39 unchanged - 23 fixed = 39 total (was 62) |
| -1 :x: | spotbugs | 1m 24s |
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=684465=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-684465
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 21/Nov/21 19:43
Start Date: 21/Nov/21 19:43
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-974846673
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 3s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 107 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 28s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 15s | | trunk passed |
| +1 :green_heart: | compile | 23m 36s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 20m 7s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 52s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 37s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 18s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | spotbugs | 2m 27s |
[/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html)
| hadoop-common-project/hadoop-common in trunk has 1 extant spotbugs
warnings. |
| +1 :green_heart: | shadedclient | 23m 42s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 19s |
[/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| -1 :x: | compile | 21m 28s |
[/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 21m 28s |
[/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 19m 5s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | javac | 19m 5s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 48s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 16 new + 71 unchanged - 83 fixed = 87 total (was
154) |
| -1 :x: | mvnsite | 0m 36s |
[/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=684450=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-684450
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 21/Nov/21 19:42
Start Date: 21/Nov/21 19:42
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r753787918
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java
##
@@ -422,7 +315,7 @@ public void checkBasicFileOperations() throws Throwable {
readonlyFS.getFileStatus(emptyDir);
// now look at a file; the outcome depends on the mode.
-accessDeniedIf(!guardedInAuthMode, () ->
+accessDeniedIf(!false, () ->
Review comment:
came from me fixing the calls to be true/false and then inlining; can
make clause non-conditional
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/InconsistentAmazonS3Client.java
##
@@ -60,12 +55,13 @@
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.Path;
/**
* A wrapper around {@link com.amazonaws.services.s3.AmazonS3} that injects
- * inconsistency and/or errors. Used for testing S3Guard.
- * Currently only delays listing visibility, not affecting GET.
+ * failures.
+ * It used to also inject inconsistency, but this was removed with S3Guard;
+ * what is retained is the ability to throttle AWS operations and for the
+ * input stream to be inconsistent.
*/
@InterfaceAudience.Private
@InterfaceStability.Unstable
Review comment:
keeping failure injection as it can set read failures and throttling,
but cutting paths for delayed listing, the settings/probes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 684450)
Time Spent: 4h 50m (was: 4h 40m)
> Remove S3Guard - no longer needed
> -
>
> Key: HADOOP-17409
> URL: https://issues.apache.org/jira/browse/HADOOP-17409
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> With Consistent S3, S3Guard is superfluous.
> stop developing it and wean people off it as soon as they can.
> Then we can worry about what to do in the code. It has gradually insinuated
> its way through the layers, especially things like multi-object delete
> handling (see HADOOP-17244). Things would be a lot simpler without it
> This work is being done in the feature branch HADOOP-17409-remove-s3guard
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=684439=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-684439 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 21/Nov/21 19:41 Start Date: 21/Nov/21 19:41 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-974862729 spotbugs is #3630; compilation error is from the patch chain not deleting a file. squashing to see what that does checkstyles are all valid; fixed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 684439) Time Spent: 4h 40m (was: 4.5h) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 4h 40m > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=684411=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-684411 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 21/Nov/21 19:37 Start Date: 21/Nov/21 19:37 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-947660904 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 684411) Time Spent: 4.5h (was: 4h 20m) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 4.5h > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=684335=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-684335 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 21/Nov/21 17:45 Start Date: 21/Nov/21 17:45 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-974862729 spotbugs is #3630; compilation error is from the patch chain not deleting a file. squashing to see what that does checkstyles are all valid; fixed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 684335) Time Spent: 4h 20m (was: 4h 10m) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 4h 20m > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=684330=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-684330
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 21/Nov/21 17:26
Start Date: 21/Nov/21 17:26
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-969227284
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 1s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 107 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 49s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 10s | | trunk passed |
| +1 :green_heart: | compile | 28m 7s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 23m 36s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 48s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 50s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 30s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 10s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 23s |
[/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| -1 :x: | compile | 25m 45s |
[/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 25m 45s |
[/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 22m 18s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | javac | 22m 18s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 11s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 15 new + 71 unchanged - 83 fixed = 86 total (was
154) |
| -1 :x: | mvnsite | 0m 40s |
[/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| +1 :green_heart: | xml | 0m 4s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 1m 45s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javadoc | 0m 43s |
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=684326=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-684326
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 21/Nov/21 16:13
Start Date: 21/Nov/21 16:13
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-974846673
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 3s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 107 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 28s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 15s | | trunk passed |
| +1 :green_heart: | compile | 23m 36s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 20m 7s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 52s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 37s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 18s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | spotbugs | 2m 27s |
[/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html)
| hadoop-common-project/hadoop-common in trunk has 1 extant spotbugs
warnings. |
| +1 :green_heart: | shadedclient | 23m 42s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 19s |
[/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| -1 :x: | compile | 21m 28s |
[/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 21m 28s |
[/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 19m 5s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | javac | 19m 5s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 48s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 16 new + 71 unchanged - 83 fixed = 87 total (was
154) |
| -1 :x: | mvnsite | 0m 36s |
[/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/7/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=684307=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-684307
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 21/Nov/21 12:04
Start Date: 21/Nov/21 12:04
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r753789722
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/InconsistentAmazonS3Client.java
##
@@ -60,12 +55,13 @@
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.Path;
/**
* A wrapper around {@link com.amazonaws.services.s3.AmazonS3} that injects
- * inconsistency and/or errors. Used for testing S3Guard.
- * Currently only delays listing visibility, not affecting GET.
+ * failures.
+ * It used to also inject inconsistency, but this was removed with S3Guard;
+ * what is retained is the ability to throttle AWS operations and for the
+ * input stream to be inconsistent.
*/
@InterfaceAudience.Private
@InterfaceStability.Unstable
Review comment:
keeping failure injection as it can set read failures and throttling,
but cutting paths for delayed listing, the settings/probes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 684307)
Time Spent: 3h 50m (was: 3h 40m)
> Remove S3Guard - no longer needed
> -
>
> Key: HADOOP-17409
> URL: https://issues.apache.org/jira/browse/HADOOP-17409
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> With Consistent S3, S3Guard is superfluous.
> stop developing it and wean people off it as soon as they can.
> Then we can worry about what to do in the code. It has gradually insinuated
> its way through the layers, especially things like multi-object delete
> handling (see HADOOP-17244). Things would be a lot simpler without it
> This work is being done in the feature branch HADOOP-17409-remove-s3guard
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=684303=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-684303
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 21/Nov/21 11:49
Start Date: 21/Nov/21 11:49
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r753787918
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java
##
@@ -422,7 +315,7 @@ public void checkBasicFileOperations() throws Throwable {
readonlyFS.getFileStatus(emptyDir);
// now look at a file; the outcome depends on the mode.
-accessDeniedIf(!guardedInAuthMode, () ->
+accessDeniedIf(!false, () ->
Review comment:
came from me fixing the calls to be true/false and then inlining; can
make clause non-conditional
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 684303)
Time Spent: 3h 40m (was: 3.5h)
> Remove S3Guard - no longer needed
> -
>
> Key: HADOOP-17409
> URL: https://issues.apache.org/jira/browse/HADOOP-17409
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> With Consistent S3, S3Guard is superfluous.
> stop developing it and wean people off it as soon as they can.
> Then we can worry about what to do in the code. It has gradually insinuated
> its way through the layers, especially things like multi-object delete
> handling (see HADOOP-17244). Things would be a lot simpler without it
> This work is being done in the feature branch HADOOP-17409-remove-s3guard
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=684301=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-684301
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 21/Nov/21 11:43
Start Date: 21/Nov/21 11:43
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-947660904
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 6s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 3s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 94 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 15s | | trunk passed |
| +1 :green_heart: | compile | 0m 50s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 22s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 17s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 41s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 0m 41s |
[/results-compile-javac-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/artifact/out/results-compile-javac-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 12 new + 16 unchanged - 5
fixed = 28 total (was 21) |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 0m 33s |
[/results-compile-javac-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/artifact/out/results-compile-javac-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
|
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 11 new
+ 16 unchanged - 5 fixed = 27 total (was 21) |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 0m 19s |
[/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt)
| hadoop-tools/hadoop-aws: The patch generated 29 new + 54 unchanged - 81
fixed = 83 total (was 135) |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javadoc | 0m 26s |
[/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
|
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new +
41
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=683940=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-683940 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 19/Nov/21 13:48 Start Date: 19/Nov/21 13:48 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-974088101 in RenameOperation & DeleteOperatoin, move off guava preconditions at the same time; fix imports. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 683940) Time Spent: 3h 20m (was: 3h 10m) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=683354=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-683354
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 18/Nov/21 14:58
Start Date: 18/Nov/21 14:58
Worklog Time Spent: 10m
Work Description: mehakmeet commented on a change in pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r752291332
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/InconsistentAmazonS3Client.java
##
@@ -60,12 +55,13 @@
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.Path;
/**
* A wrapper around {@link com.amazonaws.services.s3.AmazonS3} that injects
- * inconsistency and/or errors. Used for testing S3Guard.
- * Currently only delays listing visibility, not affecting GET.
+ * failures.
+ * It used to also inject inconsistency, but this was removed with S3Guard;
+ * what is retained is the ability to throttle AWS operations and for the
+ * input stream to be inconsistent.
*/
@InterfaceAudience.Private
@InterfaceStability.Unstable
Review comment:
Should we remove some methods like `setFailureInjectionPolicy()` and
`clearInconsistency()` which are explicitly trying to create S3 client for
"s3guard"
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFailureHandling.java
##
@@ -171,14 +143,10 @@ public void testMultiObjectDeleteMissingEntriesSucceeds()
throws Throwable {
List keys = keysToDelete(
Review comment:
Ref of `TestPartialDeleteFailures` and `MultiObjectDeleteSupport` which
are removed now.
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
##
@@ -1014,6 +931,7 @@ private Constants() {
"2s";
Review comment:
Should mark S3GUARD_CONSISTENCY_RETRY_INTERVAL configs as Deprecated as
well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 683354)
Time Spent: 3h 10m (was: 3h)
> Remove S3Guard - no longer needed
> -
>
> Key: HADOOP-17409
> URL: https://issues.apache.org/jira/browse/HADOOP-17409
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> With Consistent S3, S3Guard is superfluous.
> stop developing it and wean people off it as soon as they can.
> Then we can worry about what to do in the code. It has gradually insinuated
> its way through the layers, especially things like multi-object delete
> handling (see HADOOP-17244). Things would be a lot simpler without it
> This work is being done in the feature branch HADOOP-17409-remove-s3guard
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=681669=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681669
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 15/Nov/21 19:08
Start Date: 15/Nov/21 19:08
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-969227284
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 1s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 107 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 49s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 10s | | trunk passed |
| +1 :green_heart: | compile | 28m 7s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 23m 36s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 48s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 50s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 30s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 10s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 23s |
[/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| -1 :x: | compile | 25m 45s |
[/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 25m 45s |
[/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 22m 18s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | javac | 22m 18s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 11s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 15 new + 71 unchanged - 83 fixed = 86 total (was
154) |
| -1 :x: | mvnsite | 0m 40s |
[/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/6/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| +1 :green_heart: | xml | 0m 4s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 1m 45s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javadoc | 0m 43s |
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=681625=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681625 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 15/Nov/21 17:12 Start Date: 15/Nov/21 17:12 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-969126236 testing s3 london, all good. removed ITestPartialRenamesDeletes test which was failing as the output of the processing of the multi object delete exception didn't match expectations. We don't use that feature in production any more, once the need to update the s3guard tables goes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681625) Time Spent: 2h 50m (was: 2h 40m) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=681623=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681623 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 15/Nov/21 17:11 Start Date: 15/Nov/21 17:11 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-962095446 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681623) Time Spent: 2h 40m (was: 2.5h) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=679019=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-679019
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 09/Nov/21 13:02
Start Date: 09/Nov/21 13:02
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-964130727
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 5s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 106 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 48s | | trunk passed |
| +1 :green_heart: | compile | 0m 48s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 49s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 0s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 22s |
[/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/5/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| -1 :x: | compile | 0m 23s |
[/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/5/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-aws in the patch failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 0m 23s |
[/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/5/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-aws in the patch failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 0m 22s |
[/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/5/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-aws in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | javac | 0m 22s |
[/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/5/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-aws in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 20s |
[/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/5/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt)
| hadoop-tools/hadoop-aws: The patch generated 20 new + 71 unchanged - 83
fixed = 91 total (was 154) |
| -1 :x: | mvnsite | 0m 24s |
[/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/5/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| +1 :green_heart: | xml | 0m 3s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javadoc | 0m 26s |
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=677840=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-677840
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 05/Nov/21 20:43
Start Date: 05/Nov/21 20:43
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r743649221
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/InconsistentS3Object.java
##
@@ -173,7 +173,7 @@ private void skipFailpoint(long len) throws IOException {
}
private boolean shouldInjectFailure(String key) {
-if (policy.shouldDelay(key) &&
+if (false &&
readFailureCounter < MAX_READ_FAILURES) {
readFailureCounter++;
return true;
Review comment:
i've reinstated the whole ability to go inconsistent, in case anyone
really wants it.
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -3587,116 +3399,13 @@ S3AFileStatus innerGetFileStatus(final Path f,
String key = pathToKey(path);
LOG.debug("Getting path status for {} ({}); needEmptyDirectory={}",
path, key, needEmptyDirectoryFlag);
+// there was no entry in S3Guard
+// retrieve the data and update the metadata store in the process.
Review comment:
yeah, updated javadocs too. having inner and then s3 file status calls
is surplus now; let's see about merging
##
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
##
@@ -84,705 +89,31 @@ Once you are confident that all applications have been
restarted, _Delete the Dy
This is to avoid paying for a database you no longer need.
This is best done from the AWS GUI.
-## Setting up S3Guard
-
-### S3A to warn or fail if S3Guard is disabled
-A seemingly recurrent problem with S3Guard is that people think S3Guard is
-turned on but it isn't.
-You can set `org.apache.hadoop.fs.s3a.s3guard.disabled.warn.level`
-to avoid this. The property sets what to do when an S3A FS is instantiated
-without S3Guard. The following values are available:
-
-* `SILENT`: Do nothing.
-* `INFORM`: Log at info level that FS is instantiated without S3Guard.
-* `WARN`: Warn that data may be at risk in workflows.
-* `FAIL`: S3AFileSystem instantiation will fail.
-
-The default setting is `SILENT`. The setting is case insensitive.
-The required level can be set in the `core-site.xml`.
-
-The latest configuration parameters are defined in `core-default.xml`. You
-should consult that file for full information, but a summary is provided here.
-
+## Removing S3Guard Configurations
-### 1. Choose the Database
+The `fs.s3a.metadatastore.impl` option must be deleted, set to the empty
string "",
+or to the "Null" Metadata store
`org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore`.
-A core concept of S3Guard is that the directory listing data of the object
-store, *the metadata* is replicated in a higher-performance, consistent,
-database. In S3Guard, this database is called *The Metadata Store*
-
-By default, S3Guard is not enabled.
-
-The Metadata Store to use in production is bonded to Amazon's DynamoDB
-database service. The following setting will enable this Metadata Store:
```xml
fs.s3a.metadatastore.impl
-org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
+
```
-Note that the `NullMetadataStore` store can be explicitly requested if desired.
-This offers no metadata storage, and effectively disables S3Guard.
-
```xml
fs.s3a.metadatastore.impl
org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
```
-### 2. Configure S3Guard Settings
-
-More settings will may be added in the future.
-Currently the only Metadata Store-independent setting, besides the
-implementation class above, are the *allow authoritative* and *fail-on-error*
-flags.
-
- Authoritative S3Guard
-
-Authoritative S3Guard is a complicated configuration which delivers performance
-at the expense of being unsafe for other applications to use the same directory
-tree/bucket unless configured consistently.
-
-It can also be used to support [directory marker
retention](directory_markers.html)
-in higher-performance but non-backwards-compatible modes.
-
-Most deployments do not use this setting -it is ony used in deployments where
-specific parts of a bucket (e.g. Apache Hive managed tables) are known to
-have exclusive access by a single application (Hive) and other
tools/applications
-from exactly the same Hadoop release.
-
-The _authoritative_ expression in S3Guard is present in two different layers,
for
-two different reasons:
-
-* Authoritative S3Guard
-* S3Guard can be set as authoritative, which means that an S3A client will
-
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=677576=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-677576 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 05/Nov/21 20:11 Start Date: 05/Nov/21 20:11 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-962095446 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 677576) Time Spent: 2h 10m (was: 2h) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=677470=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-677470
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 05/Nov/21 19:58
Start Date: 05/Nov/21 19:58
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r743649221
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/InconsistentS3Object.java
##
@@ -173,7 +173,7 @@ private void skipFailpoint(long len) throws IOException {
}
private boolean shouldInjectFailure(String key) {
-if (policy.shouldDelay(key) &&
+if (false &&
readFailureCounter < MAX_READ_FAILURES) {
readFailureCounter++;
return true;
Review comment:
i've reinstated the whole ability to go inconsistent, in case anyone
really wants it.
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -3587,116 +3399,13 @@ S3AFileStatus innerGetFileStatus(final Path f,
String key = pathToKey(path);
LOG.debug("Getting path status for {} ({}); needEmptyDirectory={}",
path, key, needEmptyDirectoryFlag);
+// there was no entry in S3Guard
+// retrieve the data and update the metadata store in the process.
Review comment:
yeah, updated javadocs too. having inner and then s3 file status calls
is surplus now; let's see about merging
##
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
##
@@ -84,705 +89,31 @@ Once you are confident that all applications have been
restarted, _Delete the Dy
This is to avoid paying for a database you no longer need.
This is best done from the AWS GUI.
-## Setting up S3Guard
-
-### S3A to warn or fail if S3Guard is disabled
-A seemingly recurrent problem with S3Guard is that people think S3Guard is
-turned on but it isn't.
-You can set `org.apache.hadoop.fs.s3a.s3guard.disabled.warn.level`
-to avoid this. The property sets what to do when an S3A FS is instantiated
-without S3Guard. The following values are available:
-
-* `SILENT`: Do nothing.
-* `INFORM`: Log at info level that FS is instantiated without S3Guard.
-* `WARN`: Warn that data may be at risk in workflows.
-* `FAIL`: S3AFileSystem instantiation will fail.
-
-The default setting is `SILENT`. The setting is case insensitive.
-The required level can be set in the `core-site.xml`.
-
-The latest configuration parameters are defined in `core-default.xml`. You
-should consult that file for full information, but a summary is provided here.
-
+## Removing S3Guard Configurations
-### 1. Choose the Database
+The `fs.s3a.metadatastore.impl` option must be deleted, set to the empty
string "",
+or to the "Null" Metadata store
`org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore`.
-A core concept of S3Guard is that the directory listing data of the object
-store, *the metadata* is replicated in a higher-performance, consistent,
-database. In S3Guard, this database is called *The Metadata Store*
-
-By default, S3Guard is not enabled.
-
-The Metadata Store to use in production is bonded to Amazon's DynamoDB
-database service. The following setting will enable this Metadata Store:
```xml
fs.s3a.metadatastore.impl
-org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
+
```
-Note that the `NullMetadataStore` store can be explicitly requested if desired.
-This offers no metadata storage, and effectively disables S3Guard.
-
```xml
fs.s3a.metadatastore.impl
org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
```
-### 2. Configure S3Guard Settings
-
-More settings will may be added in the future.
-Currently the only Metadata Store-independent setting, besides the
-implementation class above, are the *allow authoritative* and *fail-on-error*
-flags.
-
- Authoritative S3Guard
-
-Authoritative S3Guard is a complicated configuration which delivers performance
-at the expense of being unsafe for other applications to use the same directory
-tree/bucket unless configured consistently.
-
-It can also be used to support [directory marker
retention](directory_markers.html)
-in higher-performance but non-backwards-compatible modes.
-
-Most deployments do not use this setting -it is ony used in deployments where
-specific parts of a bucket (e.g. Apache Hive managed tables) are known to
-have exclusive access by a single application (Hive) and other
tools/applications
-from exactly the same Hadoop release.
-
-The _authoritative_ expression in S3Guard is present in two different layers,
for
-two different reasons:
-
-* Authoritative S3Guard
-* S3Guard can be set as authoritative, which means that an S3A client will
-
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=677108=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-677108
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 05/Nov/21 17:49
Start Date: 05/Nov/21 17:49
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-962095446
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 3s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 106 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 25s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 23s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 51s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 20s |
[/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/4/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| -1 :x: | compile | 0m 22s |
[/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/4/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-aws in the patch failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 0m 22s |
[/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/4/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-aws in the patch failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 0m 19s |
[/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/4/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-aws in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | javac | 0m 19s |
[/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/4/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-aws in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 18s |
[/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/4/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt)
| hadoop-tools/hadoop-aws: The patch generated 20 new + 71 unchanged - 83
fixed = 91 total (was 154) |
| -1 :x: | mvnsite | 0m 21s |
[/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/4/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| +1 :green_heart: | xml | 0m 3s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 15s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javadoc | 0m 24s |
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=677044=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-677044 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 05/Nov/21 14:04 Start Date: 05/Nov/21 14:04 Worklog Time Spent: 10m Work Description: steveloughran commented on a change in pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#discussion_r743689411 ## File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md ## @@ -84,705 +89,31 @@ Once you are confident that all applications have been restarted, _Delete the Dy This is to avoid paying for a database you no longer need. This is best done from the AWS GUI. -## Setting up S3Guard - -### S3A to warn or fail if S3Guard is disabled -A seemingly recurrent problem with S3Guard is that people think S3Guard is -turned on but it isn't. -You can set `org.apache.hadoop.fs.s3a.s3guard.disabled.warn.level` -to avoid this. The property sets what to do when an S3A FS is instantiated -without S3Guard. The following values are available: - -* `SILENT`: Do nothing. -* `INFORM`: Log at info level that FS is instantiated without S3Guard. -* `WARN`: Warn that data may be at risk in workflows. -* `FAIL`: S3AFileSystem instantiation will fail. - -The default setting is `SILENT`. The setting is case insensitive. -The required level can be set in the `core-site.xml`. - -The latest configuration parameters are defined in `core-default.xml`. You -should consult that file for full information, but a summary is provided here. - +## Removing S3Guard Configurations -### 1. Choose the Database +The `fs.s3a.metadatastore.impl` option must be deleted, set to the empty string "", +or to the "Null" Metadata store `org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore`. -A core concept of S3Guard is that the directory listing data of the object -store, *the metadata* is replicated in a higher-performance, consistent, -database. In S3Guard, this database is called *The Metadata Store* - -By default, S3Guard is not enabled. - -The Metadata Store to use in production is bonded to Amazon's DynamoDB -database service. The following setting will enable this Metadata Store: ```xml fs.s3a.metadatastore.impl -org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore + ``` -Note that the `NullMetadataStore` store can be explicitly requested if desired. -This offers no metadata storage, and effectively disables S3Guard. - ```xml fs.s3a.metadatastore.impl org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore ``` -### 2. Configure S3Guard Settings - -More settings will may be added in the future. -Currently the only Metadata Store-independent setting, besides the -implementation class above, are the *allow authoritative* and *fail-on-error* -flags. - - Authoritative S3Guard - -Authoritative S3Guard is a complicated configuration which delivers performance -at the expense of being unsafe for other applications to use the same directory -tree/bucket unless configured consistently. - -It can also be used to support [directory marker retention](directory_markers.html) -in higher-performance but non-backwards-compatible modes. - -Most deployments do not use this setting -it is ony used in deployments where -specific parts of a bucket (e.g. Apache Hive managed tables) are known to -have exclusive access by a single application (Hive) and other tools/applications -from exactly the same Hadoop release. - -The _authoritative_ expression in S3Guard is present in two different layers, for -two different reasons: - -* Authoritative S3Guard -* S3Guard can be set as authoritative, which means that an S3A client will -avoid round-trips to S3 when **getting file metadata**, and **getting -directory listings** if there is a fully cached version of the directory -stored in metadata store. -* This mode can be set as a configuration property -`fs.s3a.metadatastore.authoritative` -* It can also be set only on specific directories by setting -`fs.s3a.authoritative.path` to one or more prefixes, for example -`s3a://bucket/path` or "/auth1,/auth2". -* All interactions with the S3 bucket(s) must be through S3A clients sharing -the same metadata store. -* This is independent from which metadata store implementation is used. -* In authoritative mode the metadata TTL metadata expiry is not effective. -This means that the metadata entries won't expire on authoritative paths. - -* Authoritative directory listings (isAuthoritative bit) -* Tells if the stored directory listing metadata is complete. -* This is set by the FileSystem client (e.g. s3a) via the `DirListingMetadata` -class (`org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata`). -(The MetadataStore
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=677045=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-677045
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 05/Nov/21 14:08
Start Date: 05/Nov/21 14:08
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r743692552
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java
##
@@ -437,7 +330,7 @@ public void checkBasicFileOperations() throws Throwable {
// This means that permissions on the file do not get checked.
// See: HADOOP-16464.
Optional optIn = accessDeniedIf(
-!guardedInAuthMode, () -> readonlyFS.open(emptyFile));
+!false, () -> readonlyFS.open(emptyFile));
Review comment:
removed the conditional nature entirely
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 677045)
Time Spent: 1h 40m (was: 1.5h)
> Remove S3Guard - no longer needed
> -
>
> Key: HADOOP-17409
> URL: https://issues.apache.org/jira/browse/HADOOP-17409
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> With Consistent S3, S3Guard is superfluous.
> stop developing it and wean people off it as soon as they can.
> Then we can worry about what to do in the code. It has gradually insinuated
> its way through the layers, especially things like multi-object delete
> handling (see HADOOP-17244). Things would be a lot simpler without it
> This work is being done in the feature branch HADOOP-17409-remove-s3guard
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=677039=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-677039 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 05/Nov/21 13:30 Start Date: 05/Nov/21 13:30 Worklog Time Spent: 10m Work Description: steveloughran commented on a change in pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#discussion_r743662102 ## File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md ## @@ -84,705 +89,31 @@ Once you are confident that all applications have been restarted, _Delete the Dy This is to avoid paying for a database you no longer need. This is best done from the AWS GUI. -## Setting up S3Guard - -### S3A to warn or fail if S3Guard is disabled -A seemingly recurrent problem with S3Guard is that people think S3Guard is -turned on but it isn't. -You can set `org.apache.hadoop.fs.s3a.s3guard.disabled.warn.level` -to avoid this. The property sets what to do when an S3A FS is instantiated -without S3Guard. The following values are available: - -* `SILENT`: Do nothing. -* `INFORM`: Log at info level that FS is instantiated without S3Guard. -* `WARN`: Warn that data may be at risk in workflows. -* `FAIL`: S3AFileSystem instantiation will fail. - -The default setting is `SILENT`. The setting is case insensitive. -The required level can be set in the `core-site.xml`. - -The latest configuration parameters are defined in `core-default.xml`. You -should consult that file for full information, but a summary is provided here. - +## Removing S3Guard Configurations -### 1. Choose the Database +The `fs.s3a.metadatastore.impl` option must be deleted, set to the empty string "", +or to the "Null" Metadata store `org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore`. -A core concept of S3Guard is that the directory listing data of the object -store, *the metadata* is replicated in a higher-performance, consistent, -database. In S3Guard, this database is called *The Metadata Store* - -By default, S3Guard is not enabled. - -The Metadata Store to use in production is bonded to Amazon's DynamoDB -database service. The following setting will enable this Metadata Store: ```xml fs.s3a.metadatastore.impl -org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore + ``` -Note that the `NullMetadataStore` store can be explicitly requested if desired. -This offers no metadata storage, and effectively disables S3Guard. - ```xml fs.s3a.metadatastore.impl org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore ``` -### 2. Configure S3Guard Settings - -More settings will may be added in the future. -Currently the only Metadata Store-independent setting, besides the -implementation class above, are the *allow authoritative* and *fail-on-error* -flags. - - Authoritative S3Guard - -Authoritative S3Guard is a complicated configuration which delivers performance -at the expense of being unsafe for other applications to use the same directory -tree/bucket unless configured consistently. - -It can also be used to support [directory marker retention](directory_markers.html) -in higher-performance but non-backwards-compatible modes. - -Most deployments do not use this setting -it is ony used in deployments where -specific parts of a bucket (e.g. Apache Hive managed tables) are known to -have exclusive access by a single application (Hive) and other tools/applications -from exactly the same Hadoop release. - -The _authoritative_ expression in S3Guard is present in two different layers, for -two different reasons: - -* Authoritative S3Guard -* S3Guard can be set as authoritative, which means that an S3A client will -avoid round-trips to S3 when **getting file metadata**, and **getting -directory listings** if there is a fully cached version of the directory -stored in metadata store. -* This mode can be set as a configuration property -`fs.s3a.metadatastore.authoritative` -* It can also be set only on specific directories by setting -`fs.s3a.authoritative.path` to one or more prefixes, for example -`s3a://bucket/path` or "/auth1,/auth2". -* All interactions with the S3 bucket(s) must be through S3A clients sharing -the same metadata store. -* This is independent from which metadata store implementation is used. -* In authoritative mode the metadata TTL metadata expiry is not effective. -This means that the metadata entries won't expire on authoritative paths. - -* Authoritative directory listings (isAuthoritative bit) -* Tells if the stored directory listing metadata is complete. -* This is set by the FileSystem client (e.g. s3a) via the `DirListingMetadata` -class (`org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata`). -(The MetadataStore
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=677036=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-677036
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 05/Nov/21 13:15
Start Date: 05/Nov/21 13:15
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r743650731
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -3587,116 +3399,13 @@ S3AFileStatus innerGetFileStatus(final Path f,
String key = pathToKey(path);
LOG.debug("Getting path status for {} ({}); needEmptyDirectory={}",
path, key, needEmptyDirectoryFlag);
+// there was no entry in S3Guard
+// retrieve the data and update the metadata store in the process.
Review comment:
yeah, updated javadocs too. having inner and then s3 file status calls
is surplus now; let's see about merging
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 677036)
Time Spent: 1h 10m (was: 1h)
> Remove S3Guard - no longer needed
> -
>
> Key: HADOOP-17409
> URL: https://issues.apache.org/jira/browse/HADOOP-17409
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> With Consistent S3, S3Guard is superfluous.
> stop developing it and wean people off it as soon as they can.
> Then we can worry about what to do in the code. It has gradually insinuated
> its way through the layers, especially things like multi-object delete
> handling (see HADOOP-17244). Things would be a lot simpler without it
> This work is being done in the feature branch HADOOP-17409-remove-s3guard
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=677035=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-677035
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 05/Nov/21 13:13
Start Date: 05/Nov/21 13:13
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r743649221
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/InconsistentS3Object.java
##
@@ -173,7 +173,7 @@ private void skipFailpoint(long len) throws IOException {
}
private boolean shouldInjectFailure(String key) {
-if (policy.shouldDelay(key) &&
+if (false &&
readFailureCounter < MAX_READ_FAILURES) {
readFailureCounter++;
return true;
Review comment:
i've reinstated the whole ability to go inconsistent, in case anyone
really wants it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 677035)
Time Spent: 1h (was: 50m)
> Remove S3Guard - no longer needed
> -
>
> Key: HADOOP-17409
> URL: https://issues.apache.org/jira/browse/HADOOP-17409
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> With Consistent S3, S3Guard is superfluous.
> stop developing it and wean people off it as soon as they can.
> Then we can worry about what to do in the code. It has gradually insinuated
> its way through the layers, especially things like multi-object delete
> handling (see HADOOP-17244). Things would be a lot simpler without it
> This work is being done in the feature branch HADOOP-17409-remove-s3guard
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=667779=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-667779
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 20/Oct/21 14:02
Start Date: 20/Oct/21 14:02
Worklog Time Spent: 10m
Work Description: bogthe commented on a change in pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r732739468
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -3587,116 +3399,13 @@ S3AFileStatus innerGetFileStatus(final Path f,
String key = pathToKey(path);
LOG.debug("Getting path status for {} ({}); needEmptyDirectory={}",
path, key, needEmptyDirectoryFlag);
+// there was no entry in S3Guard
+// retrieve the data and update the metadata store in the process.
Review comment:
Does this still update the metadata store? If not, the description of
this method can be updated too since it mentions S3Guard too
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
##
@@ -494,27 +497,25 @@ private Constants() {
public static final String CUSTOM_SIGNERS = "fs.s3a.custom.signers";
/**
- * There's 3 parameters that can be used to specify a non-default signing
+ * Multiple can be used to specify a non-default signing
Review comment:
nit: + `parameters`
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java
##
@@ -554,7 +439,7 @@ public void checkLocatedFileStatusScanFile() throws
Throwable {
true,
TEXT_FILE,
true);
-accessDeniedIf(!guardedInAuthMode,
+accessDeniedIf(!false,
Review comment:
nit: `!false` => `true`
##
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
##
@@ -84,705 +89,31 @@ Once you are confident that all applications have been
restarted, _Delete the Dy
This is to avoid paying for a database you no longer need.
This is best done from the AWS GUI.
-## Setting up S3Guard
-
-### S3A to warn or fail if S3Guard is disabled
-A seemingly recurrent problem with S3Guard is that people think S3Guard is
-turned on but it isn't.
-You can set `org.apache.hadoop.fs.s3a.s3guard.disabled.warn.level`
-to avoid this. The property sets what to do when an S3A FS is instantiated
-without S3Guard. The following values are available:
-
-* `SILENT`: Do nothing.
-* `INFORM`: Log at info level that FS is instantiated without S3Guard.
-* `WARN`: Warn that data may be at risk in workflows.
-* `FAIL`: S3AFileSystem instantiation will fail.
-
-The default setting is `SILENT`. The setting is case insensitive.
-The required level can be set in the `core-site.xml`.
-
-The latest configuration parameters are defined in `core-default.xml`. You
-should consult that file for full information, but a summary is provided here.
-
+## Removing S3Guard Configurations
-### 1. Choose the Database
+The `fs.s3a.metadatastore.impl` option must be deleted, set to the empty
string "",
+or to the "Null" Metadata store
`org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore`.
-A core concept of S3Guard is that the directory listing data of the object
-store, *the metadata* is replicated in a higher-performance, consistent,
-database. In S3Guard, this database is called *The Metadata Store*
-
-By default, S3Guard is not enabled.
-
-The Metadata Store to use in production is bonded to Amazon's DynamoDB
-database service. The following setting will enable this Metadata Store:
```xml
fs.s3a.metadatastore.impl
-org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
+
```
-Note that the `NullMetadataStore` store can be explicitly requested if desired.
-This offers no metadata storage, and effectively disables S3Guard.
-
```xml
fs.s3a.metadatastore.impl
org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
```
-### 2. Configure S3Guard Settings
-
-More settings will may be added in the future.
-Currently the only Metadata Store-independent setting, besides the
-implementation class above, are the *allow authoritative* and *fail-on-error*
-flags.
-
- Authoritative S3Guard
-
-Authoritative S3Guard is a complicated configuration which delivers performance
-at the expense of being unsafe for other applications to use the same directory
-tree/bucket unless configured consistently.
-
-It can also be used to support [directory marker
retention](directory_markers.html)
-in higher-performance but non-backwards-compatible modes.
-
-Most deployments do not use this setting -it is ony used in deployments where
-specific parts of a bucket (e.g. Apache Hive managed tables) are known to
-have exclusive access by a single application
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=667725=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-667725
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 20/Oct/21 13:21
Start Date: 20/Oct/21 13:21
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-947660904
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 6s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 3s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 94 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 15s | | trunk passed |
| +1 :green_heart: | compile | 0m 50s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 22s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 17s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 41s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 0m 41s |
[/results-compile-javac-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/artifact/out/results-compile-javac-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 12 new + 16 unchanged - 5
fixed = 28 total (was 21) |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 0m 33s |
[/results-compile-javac-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/artifact/out/results-compile-javac-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
|
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 11 new
+ 16 unchanged - 5 fixed = 27 total (was 21) |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 0m 19s |
[/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt)
| hadoop-tools/hadoop-aws: The patch generated 29 new + 54 unchanged - 81
fixed = 83 total (was 135) |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javadoc | 0m 26s |
[/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/3/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
|
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new +
41 unchanged -
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=662780=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-662780
]
ASF GitHub Bot logged work on HADOOP-17409:
---
Author: ASF GitHub Bot
Created on: 08/Oct/21 15:16
Start Date: 08/Oct/21 15:16
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#issuecomment-938723368
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 7s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 40 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 17s | | trunk passed |
| +1 :green_heart: | compile | 0m 52s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 5s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 23s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 32s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 34s |
[/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/1/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| -1 :x: | compile | 0m 46s |
[/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/1/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-aws in the patch failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 0m 46s |
[/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/1/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-aws in the patch failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 0m 34s |
[/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/1/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-aws in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | javac | 0m 34s |
[/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/1/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-aws in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 23s |
[/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/1/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt)
| hadoop-tools/hadoop-aws: The patch generated 16 new + 47 unchanged - 34
fixed = 63 total (was 81) |
| -1 :x: | mvnsite | 0m 33s |
[/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3534/1/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 16s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javadoc | 0m 30s |
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=662717=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-662717 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 08/Oct/21 13:28 Start Date: 08/Oct/21 13:28 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #3534: URL: https://github.com/apache/hadoop/pull/3534#issuecomment-938643502 @mehakmeet @mukund-thakur @bogthe @bgaborg going to be a big patch, but it removes everything. Current PR turns s3guard off, but doesn't yet remove the (no always no-op API calls) from the listing/delete/rename code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 662717) Time Spent: 20m (was: 10m) > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=662716=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-662716 ] ASF GitHub Bot logged work on HADOOP-17409: --- Author: ASF GitHub Bot Created on: 08/Oct/21 13:26 Start Date: 08/Oct/21 13:26 Worklog Time Spent: 10m Work Description: steveloughran opened a new pull request #3534: URL: https://github.com/apache/hadoop/pull/3534 ### Description of PR * Remove all S3Guard code from S3A production and test modules * CLI entry point to remove obsolete commands, downgrading to warn and fail * FS to reject instantiation if metastore set to anything other than null store ### How was this patch tested? deleting tests until everything worked again (WiP) ### For code changes: - [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? TODO - [ ] all tests to complete - [ ] Remove all tests related to delayed consistency, + a lot of remote file changed - [ ] remove `whenGuarded()` clauses from cost tests; `withWhenRaw()` to become simply `with()` - [ ] new CLI entry point "s3a"; "s3guard will map to this" - [ ] Docs - [ ] AssumedRole rules & docs - [ ] Remove the inconsistent client? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 662716) Remaining Estimate: 0h Time Spent: 10m > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it > This work is being done in the feature branch HADOOP-17409-remove-s3guard -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
