[jira] [Work logged] (HADOOP-17261) s3a rename() now requires s3:deleteObjectVersion permission
[ https://issues.apache.org/jira/browse/HADOOP-17261?focusedWorklogId=500541=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500541 ] ASF GitHub Bot logged work on HADOOP-17261: --- Author: ASF GitHub Bot Created on: 14/Oct/20 09:00 Start Date: 14/Oct/20 09:00 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #2303: URL: https://github.com/apache/hadoop/pull/2303#issuecomment-693428054 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 30m 50s | trunk passed | | +1 :green_heart: | compile | 0m 40s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 37s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 29s | trunk passed | | +1 :green_heart: | mvnsite | 0m 49s | trunk passed | | +1 :green_heart: | shadedclient | 17m 3s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 19s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 26s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 4s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 2s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 32s | the patch passed | | +1 :green_heart: | compile | 0m 35s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 35s | the patch passed | | +1 :green_heart: | compile | 0m 27s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 27s | the patch passed | | +1 :green_heart: | checkstyle | 0m 18s | the patch passed | | +1 :green_heart: | mvnsite | 0m 32s | the patch passed | | -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | shadedclient | 15m 51s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 15s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 24s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 1m 7s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 31s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. | | | | 80m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2303/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2303 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint | | uname | Linux beb9b125d5c6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / ce861836918 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | whitespace | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2303/3/artifact/out/whitespace-eol.txt | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2303/3/testReport/ | | Max. process+thread count | 337 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output |
[jira] [Work logged] (HADOOP-17261) s3a rename() now requires s3:deleteObjectVersion permission
[ https://issues.apache.org/jira/browse/HADOOP-17261?focusedWorklogId=489279=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-489279 ] ASF GitHub Bot logged work on HADOOP-17261: --- Author: ASF GitHub Bot Created on: 23/Sep/20 04:49 Start Date: 23/Sep/20 04:49 Worklog Time Spent: 10m Work Description: steveloughran merged pull request #2303: URL: https://github.com/apache/hadoop/pull/2303 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 489279) Time Spent: 2h 10m (was: 2h) > s3a rename() now requires s3:deleteObjectVersion permission > --- > > Key: HADOOP-17261 > URL: https://issues.apache.org/jira/browse/HADOOP-17261 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > With the directory marker change (HADOOP-13230) you need the > s3:deleteObjectVersion permission in your role, else the operation will fail > in the bulk delete, *if S3Guard is in use* > Root cause > -if fileStatus has a versionId, we pass that in to the delete KeyVersion pair > -an unguarded listing doesn't get that versionId, so this is not an issue > -but if files in a directory were previously created such that S3Guard has > their versionId in its tables, that is used in the request > -which then fails if the caller doesn't have the permission > Although we say "you need s3:delete*", this is a regression as any IAM role > without the permission will have rename fail during delete -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17261) s3a rename() now requires s3:deleteObjectVersion permission
[ https://issues.apache.org/jira/browse/HADOOP-17261?focusedWorklogId=489162=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-489162 ] ASF GitHub Bot logged work on HADOOP-17261: --- Author: ASF GitHub Bot Created on: 23/Sep/20 04:40 Start Date: 23/Sep/20 04:40 Worklog Time Spent: 10m Work Description: bgaborg commented on pull request #2303: URL: https://github.com/apache/hadoop/pull/2303#issuecomment-696794405 LGTM, +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 489162) Time Spent: 2h (was: 1h 50m) > s3a rename() now requires s3:deleteObjectVersion permission > --- > > Key: HADOOP-17261 > URL: https://issues.apache.org/jira/browse/HADOOP-17261 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > With the directory marker change (HADOOP-13230) you need the > s3:deleteObjectVersion permission in your role, else the operation will fail > in the bulk delete, *if S3Guard is in use* > Root cause > -if fileStatus has a versionId, we pass that in to the delete KeyVersion pair > -an unguarded listing doesn't get that versionId, so this is not an issue > -but if files in a directory were previously created such that S3Guard has > their versionId in its tables, that is used in the request > -which then fails if the caller doesn't have the permission > Although we say "you need s3:delete*", this is a regression as any IAM role > without the permission will have rename fail during delete -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17261) s3a rename() now requires s3:deleteObjectVersion permission
[ https://issues.apache.org/jira/browse/HADOOP-17261?focusedWorklogId=489157=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-489157 ] ASF GitHub Bot logged work on HADOOP-17261: --- Author: ASF GitHub Bot Created on: 23/Sep/20 04:40 Start Date: 23/Sep/20 04:40 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2303: URL: https://github.com/apache/hadoop/pull/2303#issuecomment-696829470 thank you :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 489157) Time Spent: 1h 50m (was: 1h 40m) > s3a rename() now requires s3:deleteObjectVersion permission > --- > > Key: HADOOP-17261 > URL: https://issues.apache.org/jira/browse/HADOOP-17261 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > With the directory marker change (HADOOP-13230) you need the > s3:deleteObjectVersion permission in your role, else the operation will fail > in the bulk delete, *if S3Guard is in use* > Root cause > -if fileStatus has a versionId, we pass that in to the delete KeyVersion pair > -an unguarded listing doesn't get that versionId, so this is not an issue > -but if files in a directory were previously created such that S3Guard has > their versionId in its tables, that is used in the request > -which then fails if the caller doesn't have the permission > Although we say "you need s3:delete*", this is a regression as any IAM role > without the permission will have rename fail during delete -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17261) s3a rename() now requires s3:deleteObjectVersion permission
[ https://issues.apache.org/jira/browse/HADOOP-17261?focusedWorklogId=488392=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-488392 ] ASF GitHub Bot logged work on HADOOP-17261: --- Author: ASF GitHub Bot Created on: 22/Sep/20 16:22 Start Date: 22/Sep/20 16:22 Worklog Time Spent: 10m Work Description: steveloughran merged pull request #2303: URL: https://github.com/apache/hadoop/pull/2303 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 488392) Time Spent: 1h 40m (was: 1.5h) > s3a rename() now requires s3:deleteObjectVersion permission > --- > > Key: HADOOP-17261 > URL: https://issues.apache.org/jira/browse/HADOOP-17261 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > With the directory marker change (HADOOP-13230) you need the > s3:deleteObjectVersion permission in your role, else the operation will fail > in the bulk delete, *if S3Guard is in use* > Root cause > -if fileStatus has a versionId, we pass that in to the delete KeyVersion pair > -an unguarded listing doesn't get that versionId, so this is not an issue > -but if files in a directory were previously created such that S3Guard has > their versionId in its tables, that is used in the request > -which then fails if the caller doesn't have the permission > Although we say "you need s3:delete*", this is a regression as any IAM role > without the permission will have rename fail during delete -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17261) s3a rename() now requires s3:deleteObjectVersion permission
[ https://issues.apache.org/jira/browse/HADOOP-17261?focusedWorklogId=488391=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-488391 ] ASF GitHub Bot logged work on HADOOP-17261: --- Author: ASF GitHub Bot Created on: 22/Sep/20 16:21 Start Date: 22/Sep/20 16:21 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2303: URL: https://github.com/apache/hadoop/pull/2303#issuecomment-696829470 thank you :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 488391) Time Spent: 1.5h (was: 1h 20m) > s3a rename() now requires s3:deleteObjectVersion permission > --- > > Key: HADOOP-17261 > URL: https://issues.apache.org/jira/browse/HADOOP-17261 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > With the directory marker change (HADOOP-13230) you need the > s3:deleteObjectVersion permission in your role, else the operation will fail > in the bulk delete, *if S3Guard is in use* > Root cause > -if fileStatus has a versionId, we pass that in to the delete KeyVersion pair > -an unguarded listing doesn't get that versionId, so this is not an issue > -but if files in a directory were previously created such that S3Guard has > their versionId in its tables, that is used in the request > -which then fails if the caller doesn't have the permission > Although we say "you need s3:delete*", this is a regression as any IAM role > without the permission will have rename fail during delete -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17261) s3a rename() now requires s3:deleteObjectVersion permission
[ https://issues.apache.org/jira/browse/HADOOP-17261?focusedWorklogId=488342=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-488342 ] ASF GitHub Bot logged work on HADOOP-17261: --- Author: ASF GitHub Bot Created on: 22/Sep/20 15:25 Start Date: 22/Sep/20 15:25 Worklog Time Spent: 10m Work Description: bgaborg commented on pull request #2303: URL: https://github.com/apache/hadoop/pull/2303#issuecomment-696794405 LGTM, +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 488342) Time Spent: 1h 20m (was: 1h 10m) > s3a rename() now requires s3:deleteObjectVersion permission > --- > > Key: HADOOP-17261 > URL: https://issues.apache.org/jira/browse/HADOOP-17261 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > With the directory marker change (HADOOP-13230) you need the > s3:deleteObjectVersion permission in your role, else the operation will fail > in the bulk delete, *if S3Guard is in use* > Root cause > -if fileStatus has a versionId, we pass that in to the delete KeyVersion pair > -an unguarded listing doesn't get that versionId, so this is not an issue > -but if files in a directory were previously created such that S3Guard has > their versionId in its tables, that is used in the request > -which then fails if the caller doesn't have the permission > Although we say "you need s3:delete*", this is a regression as any IAM role > without the permission will have rename fail during delete -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17261) s3a rename() now requires s3:deleteObjectVersion permission
[ https://issues.apache.org/jira/browse/HADOOP-17261?focusedWorklogId=486293=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-486293 ] ASF GitHub Bot logged work on HADOOP-17261: --- Author: ASF GitHub Bot Created on: 18/Sep/20 17:23 Start Date: 18/Sep/20 17:23 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2303: URL: https://github.com/apache/hadoop/pull/2303#issuecomment-694989215 @bgaborg - could you look @ this when you have a chance? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 486293) Time Spent: 1h 10m (was: 1h) > s3a rename() now requires s3:deleteObjectVersion permission > --- > > Key: HADOOP-17261 > URL: https://issues.apache.org/jira/browse/HADOOP-17261 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > With the directory marker change (HADOOP-13230) you need the > s3:deleteObjectVersion permission in your role, else the operation will fail > in the bulk delete, *if S3Guard is in use* > Root cause > -if fileStatus has a versionId, we pass that in to the delete KeyVersion pair > -an unguarded listing doesn't get that versionId, so this is not an issue > -but if files in a directory were previously created such that S3Guard has > their versionId in its tables, that is used in the request > -which then fails if the caller doesn't have the permission > Although we say "you need s3:delete*", this is a regression as any IAM role > without the permission will have rename fail during delete -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17261) s3a rename() now requires s3:deleteObjectVersion permission
[ https://issues.apache.org/jira/browse/HADOOP-17261?focusedWorklogId=485164=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-485164 ] ASF GitHub Bot logged work on HADOOP-17261: --- Author: ASF GitHub Bot Created on: 16/Sep/20 14:04 Start Date: 16/Sep/20 14:04 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2303: URL: https://github.com/apache/hadoop/pull/2303#issuecomment-693428054 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 30m 50s | trunk passed | | +1 :green_heart: | compile | 0m 40s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 37s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 29s | trunk passed | | +1 :green_heart: | mvnsite | 0m 49s | trunk passed | | +1 :green_heart: | shadedclient | 17m 3s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 19s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 26s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 4s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 2s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 32s | the patch passed | | +1 :green_heart: | compile | 0m 35s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 35s | the patch passed | | +1 :green_heart: | compile | 0m 27s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 27s | the patch passed | | +1 :green_heart: | checkstyle | 0m 18s | the patch passed | | +1 :green_heart: | mvnsite | 0m 32s | the patch passed | | -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | shadedclient | 15m 51s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 15s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 24s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 1m 7s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 31s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. | | | | 80m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2303/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2303 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint | | uname | Linux beb9b125d5c6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / ce861836918 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | whitespace | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2303/3/artifact/out/whitespace-eol.txt | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2303/3/testReport/ | | Max. process+thread count | 337 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output |
[jira] [Work logged] (HADOOP-17261) s3a rename() now requires s3:deleteObjectVersion permission
[ https://issues.apache.org/jira/browse/HADOOP-17261?focusedWorklogId=485147=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-485147 ] ASF GitHub Bot logged work on HADOOP-17261: --- Author: ASF GitHub Bot Created on: 16/Sep/20 13:30 Start Date: 16/Sep/20 13:30 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2303: URL: https://github.com/apache/hadoop/pull/2303#issuecomment-693406274 Test with - mvit -Dparallel-tests -DtestsThreadCount=6 -Dmarkers=keep -Ds3guard -Ddynamo -Dfs.s3a.directory.marker.audit=true -Dscale https://issues.apache.org/jira/browse/HADOOP-17263 : read() didn't get as much back as expected (unrelated) https://issues.apache.org/jira/browse/HADOOP-17226 Failure of ITestAssumeRole.testRestrictedCommitActions That -Dscale option kicks off some MR jobs which spawn extra processes beyond the maven JUnit runners, the system may be overloading. Surprised how it surfaces though. I can see it amplifies a race condition in a non-atomic ++ call (more chance of pre-emption, longer delay for rescheduling), but read() not receiving enough data This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 485147) Time Spent: 50m (was: 40m) > s3a rename() now requires s3:deleteObjectVersion permission > --- > > Key: HADOOP-17261 > URL: https://issues.apache.org/jira/browse/HADOOP-17261 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > With the directory marker change (HADOOP-13230) you need the > s3:deleteObjectVersion permission in your role, else the operation will fail > in the bulk delete, *if S3Guard is in use* > Root cause > -if fileStatus has a versionId, we pass that in to the delete KeyVersion pair > -an unguarded listing doesn't get that versionId, so this is not an issue > -but if files in a directory were previously created such that S3Guard has > their versionId in its tables, that is used in the request > -which then fails if the caller doesn't have the permission > Although we say "you need s3:delete*", this is a regression as any IAM role > without the permission will have rename fail during delete -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17261) s3a rename() now requires s3:deleteObjectVersion permission
[ https://issues.apache.org/jira/browse/HADOOP-17261?focusedWorklogId=485116=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-485116 ] ASF GitHub Bot logged work on HADOOP-17261: --- Author: ASF GitHub Bot Created on: 16/Sep/20 12:43 Start Date: 16/Sep/20 12:43 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2303: URL: https://github.com/apache/hadoop/pull/2303#issuecomment-693379615 testing in progress: S3 london This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 485116) Time Spent: 40m (was: 0.5h) > s3a rename() now requires s3:deleteObjectVersion permission > --- > > Key: HADOOP-17261 > URL: https://issues.apache.org/jira/browse/HADOOP-17261 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > With the directory marker change (HADOOP-13230) you need the > s3:deleteObjectVersion permission in your role, else the operation will fail > in the bulk delete, *if S3Guard is in use* > Root cause > -if fileStatus has a versionId, we pass that in to the delete KeyVersion pair > -an unguarded listing doesn't get that versionId, so this is not an issue > -but if files in a directory were previously created such that S3Guard has > their versionId in its tables, that is used in the request > -which then fails if the caller doesn't have the permission > Although we say "you need s3:delete*", this is a regression as any IAM role > without the permission will have rename fail during delete -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17261) s3a rename() now requires s3:deleteObjectVersion permission
[ https://issues.apache.org/jira/browse/HADOOP-17261?focusedWorklogId=485115=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-485115 ] ASF GitHub Bot logged work on HADOOP-17261: --- Author: ASF GitHub Bot Created on: 16/Sep/20 12:43 Start Date: 16/Sep/20 12:43 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #2303: URL: https://github.com/apache/hadoop/pull/2303#issuecomment-692830480 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 37m 58s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 35m 13s | trunk passed | | +1 :green_heart: | compile | 0m 45s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 37s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 29s | trunk passed | | +1 :green_heart: | mvnsite | 0m 45s | trunk passed | | +1 :green_heart: | shadedclient | 19m 22s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 25s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 33s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 20s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 17s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 41s | the patch passed | | +1 :green_heart: | compile | 0m 40s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 40s | the patch passed | | +1 :green_heart: | compile | 0m 31s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 31s | the patch passed | | +1 :green_heart: | checkstyle | 0m 19s | the patch passed | | +1 :green_heart: | mvnsite | 0m 35s | the patch passed | | -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | shadedclient | 17m 52s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 18s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 27s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 1m 19s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 37s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 35s | The patch does not generate ASF License warnings. | | | | 124m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2303/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2303 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint | | uname | Linux 4bff30c21dc4 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / f4ed9f3f911 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | whitespace | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2303/2/artifact/out/whitespace-eol.txt | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2303/2/testReport/ | | Max. process+thread count | 414 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output |
[jira] [Work logged] (HADOOP-17261) s3a rename() now requires s3:deleteObjectVersion permission
[ https://issues.apache.org/jira/browse/HADOOP-17261?focusedWorklogId=484610=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-484610 ] ASF GitHub Bot logged work on HADOOP-17261: --- Author: ASF GitHub Bot Created on: 15/Sep/20 16:31 Start Date: 15/Sep/20 16:31 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2303: URL: https://github.com/apache/hadoop/pull/2303#issuecomment-692830480 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 37m 58s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 35m 13s | trunk passed | | +1 :green_heart: | compile | 0m 45s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 37s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 29s | trunk passed | | +1 :green_heart: | mvnsite | 0m 45s | trunk passed | | +1 :green_heart: | shadedclient | 19m 22s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 25s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 33s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 20s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 17s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 41s | the patch passed | | +1 :green_heart: | compile | 0m 40s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 40s | the patch passed | | +1 :green_heart: | compile | 0m 31s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 31s | the patch passed | | +1 :green_heart: | checkstyle | 0m 19s | the patch passed | | +1 :green_heart: | mvnsite | 0m 35s | the patch passed | | -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | shadedclient | 17m 52s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 18s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 27s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 1m 19s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 37s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 35s | The patch does not generate ASF License warnings. | | | | 124m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2303/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2303 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint | | uname | Linux 4bff30c21dc4 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / f4ed9f3f911 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | whitespace | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2303/2/artifact/out/whitespace-eol.txt | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2303/2/testReport/ | | Max. process+thread count | 414 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output |
[jira] [Work logged] (HADOOP-17261) s3a rename() now requires s3:deleteObjectVersion permission
[ https://issues.apache.org/jira/browse/HADOOP-17261?focusedWorklogId=484193=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-484193 ] ASF GitHub Bot logged work on HADOOP-17261: --- Author: ASF GitHub Bot Created on: 14/Sep/20 19:59 Start Date: 14/Sep/20 19:59 Worklog Time Spent: 10m Work Description: steveloughran opened a new pull request #2303: URL: https://github.com/apache/hadoop/pull/2303 Stop using the versionId when building the list of files to delete as the rename progresses Testing: manual. For anyone wishing to replicate the problem on an unpatched release 1. Have a versioned bucket. 1. Latest version of cloudstore now allows you to create a set of session keys from an IAM role and json file, printing them as env vars and hadoop config options 1. You need S3Guard enabled and the files to have been created individually by a client through S3Guard (i.e. the entries not discovered through listing). Then S3Guard caches the version ID in the DB, which is then used when deleting the object. 1. mkdir src/; touchz src/file1 file2 2. mv src dst if that works, try renaming back. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 484193) Remaining Estimate: 0h Time Spent: 10m > s3a rename() now requires s3:deleteObjectVersion permission > --- > > Key: HADOOP-17261 > URL: https://issues.apache.org/jira/browse/HADOOP-17261 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > With the directory marker change (HADOOP-13230) you need the > s3:deleteObjectVersion permission in your role, else the operation will fail > in the bulk delete, *if S3Guard is in use* > Root cause > -if fileStatus has a versionId, we pass that in to the delete KeyVersion pair > -an unguarded listing doesn't get that versionId, so this is not an issue > -but if files in a directory were previously created such that S3Guard has > their versionId in its tables, that is used in the request > -which then fails if the caller doesn't have the permission > Although we say "you need s3:delete*", this is a regression as any IAM role > without the permission will have rename fail during delete -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
