[jira] [Commented] (HADOOP-18167) Add metrics to track delegation token secret manager operations
[
https://issues.apache.org/jira/browse/HADOOP-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17511016#comment-17511016
]
Hadoop QA commented on HADOOP-18167:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 8m
54s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} branch-2.10 Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
8s{color} | {color:green}{color} | {color:green} branch-2.10 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
3s{color} | {color:green}{color} | {color:green} branch-2.10 passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
44s{color} | {color:green}{color} | {color:green} branch-2.10 passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green} branch-2.10 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green}{color} | {color:green} branch-2.10 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green}{color} | {color:green} branch-2.10 passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green}{color} | {color:green} branch-2.10 passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 6m
37s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:red}-1{color} | {color:red} spotbugs {color} | {color:red} 2m
2s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HADOOP-Build/261/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html{color}
| {color:red} hadoop-common-project/hadoop-common in branch-2.10 has 2 extant
spotbugs warnings. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
23s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
23s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
50s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 44s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-HADOOP-Build/261/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt{color}
| {color:orange} hadoop-common-project/hadoop-common: The patch generated 8
new + 79 unchanged - 0 fixed = 87 total (was 79) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
16s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green}{color} |
[jira] [Work logged] (HADOOP-18169) getDelegationTokens in ViewFs should also fetch the token from the fallback FS
[
https://issues.apache.org/jira/browse/HADOOP-18169?focusedWorklogId=746342=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746342
]
ASF GitHub Bot logged work on HADOOP-18169:
---
Author: ASF GitHub Bot
Created on: 23/Mar/22 04:26
Start Date: 23/Mar/22 04:26
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #4094:
URL: https://github.com/apache/hadoop/pull/4094#issuecomment-1075901869
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 23s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 58s | | trunk passed |
| +1 :green_heart: | compile | 24m 44s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 21m 20s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 50s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 17s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 17s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 17s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 8s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 26s | | the patch passed |
| +1 :green_heart: | compile | 23m 54s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 23m 54s | | the patch passed |
| +1 :green_heart: | compile | 21m 17s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 21m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 46s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/3/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 41 unchanged - 0 fixed = 42 total (was
41) |
| +1 :green_heart: | mvnsite | 3m 17s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 14s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 19s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 36s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 35s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 325m 53s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 58s | | The patch does not
generate ASF License warnings. |
| | | 566m 31s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4094 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux f93c1bd7146a 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18
11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5d083619c211094ffc10bef57305c47e5f57643a |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
[GitHub] [hadoop] hadoop-yetus commented on pull request #4094: HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS
hadoop-yetus commented on pull request #4094:
URL: https://github.com/apache/hadoop/pull/4094#issuecomment-1075901869
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 23s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 58s | | trunk passed |
| +1 :green_heart: | compile | 24m 44s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 21m 20s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 50s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 17s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 17s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 17s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 8s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 26s | | the patch passed |
| +1 :green_heart: | compile | 23m 54s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 23m 54s | | the patch passed |
| +1 :green_heart: | compile | 21m 17s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 21m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 46s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/3/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 41 unchanged - 0 fixed = 42 total (was
41) |
| +1 :green_heart: | mvnsite | 3m 17s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 14s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 19s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 36s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 35s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 325m 53s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 58s | | The patch does not
generate ASF License warnings. |
| | | 566m 31s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4094 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux f93c1bd7146a 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18
11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5d083619c211094ffc10bef57305c47e5f57643a |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/3/testReport/ |
| Max. process+thread count | 2091 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT
[jira] [Work logged] (HADOOP-18088) Replace log4j 1.x with reload4j
[ https://issues.apache.org/jira/browse/HADOOP-18088?focusedWorklogId=746335=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746335 ] ASF GitHub Bot logged work on HADOOP-18088: --- Author: ASF GitHub Bot Created on: 23/Mar/22 03:59 Start Date: 23/Mar/22 03:59 Worklog Time Spent: 10m Work Description: iwasakims commented on pull request #4084: URL: https://github.com/apache/hadoop/pull/4084#issuecomment-1075889761 Thanks, @steveloughran. merged this to branch-3.2. If I need to cut 3.2.3 RC2, cherry-picking this to branch-3.2.3 could be an option. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 746335) Time Spent: 2h 50m (was: 2h 40m) > Replace log4j 1.x with reload4j > --- > > Key: HADOOP-18088 > URL: https://issues.apache.org/jira/browse/HADOOP-18088 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > As proposed in the dev mailing list > (https://lists.apache.org/thread/fdzkv80mzkf3w74z9120l0k0rc3v7kqk) let's > replace log4j 1 with reload4j in the maintenance releases (i.e. 3.3.x, 3.2.x > and 2.10.x) -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] iwasakims commented on pull request #4084: HADOOP-18088. Replace log4j 1.x with reload4j.
iwasakims commented on pull request #4084: URL: https://github.com/apache/hadoop/pull/4084#issuecomment-1075889761 Thanks, @steveloughran. merged this to branch-3.2. If I need to cut 3.2.3 RC2, cherry-picking this to branch-3.2.3 could be an option. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18088) Replace log4j 1.x with reload4j
[ https://issues.apache.org/jira/browse/HADOOP-18088?focusedWorklogId=746333=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746333 ] ASF GitHub Bot logged work on HADOOP-18088: --- Author: ASF GitHub Bot Created on: 23/Mar/22 03:57 Start Date: 23/Mar/22 03:57 Worklog Time Spent: 10m Work Description: iwasakims merged pull request #4084: URL: https://github.com/apache/hadoop/pull/4084 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 746333) Time Spent: 2h 40m (was: 2.5h) > Replace log4j 1.x with reload4j > --- > > Key: HADOOP-18088 > URL: https://issues.apache.org/jira/browse/HADOOP-18088 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > As proposed in the dev mailing list > (https://lists.apache.org/thread/fdzkv80mzkf3w74z9120l0k0rc3v7kqk) let's > replace log4j 1 with reload4j in the maintenance releases (i.e. 3.3.x, 3.2.x > and 2.10.x) -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] iwasakims merged pull request #4084: HADOOP-18088. Replace log4j 1.x with reload4j.
iwasakims merged pull request #4084: URL: https://github.com/apache/hadoop/pull/4084 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #2918: HDFS-15987. Improve oiv tool to parse fsimage file in parallel with d…
Hexiaoqiao commented on pull request #2918: URL: https://github.com/apache/hadoop/pull/2918#issuecomment-1075880371 revert and re-check in to correct the git message. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18167) Add metrics to track delegation token secret manager operations
[
https://issues.apache.org/jira/browse/HADOOP-18167?focusedWorklogId=746323=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746323
]
ASF GitHub Bot logged work on HADOOP-18167:
---
Author: ASF GitHub Bot
Created on: 23/Mar/22 03:27
Start Date: 23/Mar/22 03:27
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #4092:
URL: https://github.com/apache/hadoop/pull/4092#issuecomment-1075876286
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 32s | | trunk passed |
| +1 :green_heart: | compile | 24m 39s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 21m 0s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 38s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 15s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 59s | | the patch passed |
| +1 :green_heart: | compile | 23m 56s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 23m 56s | | the patch passed |
| +1 :green_heart: | compile | 21m 5s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 21m 5s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 59s |
[/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4092/2/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt)
| hadoop-common-project/hadoop-common: The patch generated 10 new + 74
unchanged - 0 fixed = 84 total (was 74) |
| +1 :green_heart: | mvnsite | 1m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 6s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 39s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 41s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 40s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not
generate ASF License warnings. |
| | | 213m 49s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4092/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4092 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 742e8c30f765 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5
11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4e2c9d9d7cbd071d99788f91fa90d554ea43378f |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4092/2/testReport/ |
| Max. process+thread count | 3135 (vs. ulimit of 5500) |
| modules | C:
[GitHub] [hadoop] hadoop-yetus commented on pull request #4092: HADOOP-18167. Add metrics to track delegation token secret manager op…
hadoop-yetus commented on pull request #4092:
URL: https://github.com/apache/hadoop/pull/4092#issuecomment-1075876286
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 32s | | trunk passed |
| +1 :green_heart: | compile | 24m 39s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 21m 0s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 38s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 15s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 59s | | the patch passed |
| +1 :green_heart: | compile | 23m 56s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 23m 56s | | the patch passed |
| +1 :green_heart: | compile | 21m 5s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 21m 5s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 59s |
[/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4092/2/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt)
| hadoop-common-project/hadoop-common: The patch generated 10 new + 74
unchanged - 0 fixed = 84 total (was 74) |
| +1 :green_heart: | mvnsite | 1m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 6s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 39s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 41s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 40s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not
generate ASF License warnings. |
| | | 213m 49s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4092/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4092 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 742e8c30f765 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5
11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4e2c9d9d7cbd071d99788f91fa90d554ea43378f |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4092/2/testReport/ |
| Max. process+thread count | 3135 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4092/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to
[jira] [Updated] (HADOOP-18167) Add metrics to track delegation token secret manager operations
[ https://issues.apache.org/jira/browse/HADOOP-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Sandoval Chaverri updated HADOOP-18167: -- Attachment: HADOOP-18167-branch-2.10.patch Status: Patch Available (was: Open) > Add metrics to track delegation token secret manager operations > --- > > Key: HADOOP-18167 > URL: https://issues.apache.org/jira/browse/HADOOP-18167 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-18167-branch-2.10.patch > > Time Spent: 20m > Remaining Estimate: 0h > > New metrics to track operations that store, update and remove delegation > tokens in implementations of AbstractDelegationTokenSecretManager. This will > help evaluate the impact of using different secret managers and add > optimizations. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18169) getDelegationTokens in ViewFs should also fetch the token from the fallback FS
[
https://issues.apache.org/jira/browse/HADOOP-18169?focusedWorklogId=746312=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746312
]
ASF GitHub Bot logged work on HADOOP-18169:
---
Author: ASF GitHub Bot
Created on: 23/Mar/22 02:51
Start Date: 23/Mar/22 02:51
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #4094:
URL: https://github.com/apache/hadoop/pull/4094#issuecomment-1075861113
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 30s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 28s | | trunk passed |
| +1 :green_heart: | compile | 24m 37s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 20m 46s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 19s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 32s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 17s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 27s | | the patch passed |
| +1 :green_heart: | compile | 23m 56s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 23m 56s | | the patch passed |
| +1 :green_heart: | compile | 20m 58s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 58s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 37s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/2/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 41 unchanged - 0 fixed = 42 total (was
41) |
| +1 :green_heart: | mvnsite | 3m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 28s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 22s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 27m 17s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 49s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 237m 45s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 13s | | The patch does not
generate ASF License warnings. |
| | | 479m 26s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4094 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 7ea0c80d39e9 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4996eed1f2b0055784ca1452a5c72fc57aa5f8bc |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
[GitHub] [hadoop] hadoop-yetus commented on pull request #4094: HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS
hadoop-yetus commented on pull request #4094:
URL: https://github.com/apache/hadoop/pull/4094#issuecomment-1075861113
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 30s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 28s | | trunk passed |
| +1 :green_heart: | compile | 24m 37s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 20m 46s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 19s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 32s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 17s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 27s | | the patch passed |
| +1 :green_heart: | compile | 23m 56s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 23m 56s | | the patch passed |
| +1 :green_heart: | compile | 20m 58s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 58s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 37s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/2/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 41 unchanged - 0 fixed = 42 total (was
41) |
| +1 :green_heart: | mvnsite | 3m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 28s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 22s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 27m 17s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 49s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 237m 45s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 13s | | The patch does not
generate ASF License warnings. |
| | | 479m 26s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4094 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 7ea0c80d39e9 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4996eed1f2b0055784ca1452a5c72fc57aa5f8bc |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/2/testReport/ |
| Max. process+thread count | 3227 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT
[jira] [Work logged] (HADOOP-18169) getDelegationTokens in ViewFs should also fetch the token from the fallback FS
[
https://issues.apache.org/jira/browse/HADOOP-18169?focusedWorklogId=746304=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746304
]
ASF GitHub Bot logged work on HADOOP-18169:
---
Author: ASF GitHub Bot
Created on: 23/Mar/22 02:17
Start Date: 23/Mar/22 02:17
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #4094:
URL: https://github.com/apache/hadoop/pull/4094#issuecomment-1075837818
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 22s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 41s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 25s | | trunk passed |
| +1 :green_heart: | compile | 24m 12s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 21m 22s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 22s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 30s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 15s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 56s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 22s | | the patch passed |
| +1 :green_heart: | compile | 23m 42s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 23m 42s | | the patch passed |
| +1 :green_heart: | compile | 22m 0s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 22m 0s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 35s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 41 unchanged - 0 fixed = 42 total (was
41) |
| +1 :green_heart: | mvnsite | 3m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 20s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 23s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 24s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 18s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 242m 39s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 9s | | The patch does not
generate ASF License warnings. |
| | | 492m 7s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4094 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 716989d0f40f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f5adca1a0ed64818f09cce3082dcd887ad03102c |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
[GitHub] [hadoop] hadoop-yetus commented on pull request #4094: HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS
hadoop-yetus commented on pull request #4094:
URL: https://github.com/apache/hadoop/pull/4094#issuecomment-1075837818
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 22s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 41s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 25s | | trunk passed |
| +1 :green_heart: | compile | 24m 12s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 21m 22s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 22s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 30s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 15s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 56s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 22s | | the patch passed |
| +1 :green_heart: | compile | 23m 42s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 23m 42s | | the patch passed |
| +1 :green_heart: | compile | 22m 0s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 22m 0s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 35s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 41 unchanged - 0 fixed = 42 total (was
41) |
| +1 :green_heart: | mvnsite | 3m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 20s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 23s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 24s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 18s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 242m 39s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 9s | | The patch does not
generate ASF License warnings. |
| | | 492m 7s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4094 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 716989d0f40f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f5adca1a0ed64818f09cce3082dcd887ad03102c |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/1/testReport/ |
| Max. process+thread count | 3534 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT
[GitHub] [hadoop] xinglin commented on a change in pull request #4089: HDFS-16515. Improve ec exception message
xinglin commented on a change in pull request #4089:
URL: https://github.com/apache/hadoop/pull/4089#discussion_r832768475
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirErasureCodingOp.java
##
@@ -168,8 +168,8 @@ static FileStatus setErasureCodingPolicy(final FSNamesystem
fsn,
throw new FileNotFoundException("Path not found: " + srcIIP.getPath());
}
if (!inode.isDirectory()) {
- throw new IOException("Attempt to set an erasure coding policy " +
- "for a file " + src);
+ throw new IOException("Cannot set an erasure coding policy " +
+ "for a file " + src + ". Expected to be a directory");
Review comment:
nit: "Expected to be a directory" -> "Erasure coding policy can only be
set for directories."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xinglin commented on a change in pull request #4091: HDFS-16517 Distance metric is wrong for non-DN machines in 2.10.
xinglin commented on a change in pull request #4091:
URL: https://github.com/apache/hadoop/pull/4091#discussion_r832726638
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java
##
@@ -823,7 +823,7 @@ private static int getWeightUsingNetworkLocation(Node
reader, Node node) {
}
currentLevel++;
}
-weight = (readerPathToken.length - currentLevel) +
+weight = 2 + (readerPathToken.length - currentLevel) +
Review comment:
nit: could we move "2 + " to the end of the next line, so that it is
identical to what is in trunk?
```
weight = (readerPathToken.length - currentLevel) +
(nodePathToken.length - currentLevel) + 2;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4091: HDFS-16517 Distance metric is wrong for non-DN machines in 2.10.
hadoop-yetus commented on pull request #4091:
URL: https://github.com/apache/hadoop/pull/4091#issuecomment-1075730771
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-2.10 Compile Tests _ |
| +1 :green_heart: | mvninstall | 14m 11s | | branch-2.10 passed |
| +1 :green_heart: | compile | 12m 56s | | branch-2.10 passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | compile | 10m 44s | | branch-2.10 passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | checkstyle | 0m 42s | | branch-2.10 passed |
| +1 :green_heart: | mvnsite | 1m 19s | | branch-2.10 passed |
| +1 :green_heart: | javadoc | 1m 16s | | branch-2.10 passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javadoc | 1m 4s | | branch-2.10 passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| -1 :x: | spotbugs | 2m 1s |
[/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/3/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html)
| hadoop-common-project/hadoop-common in branch-2.10 has 2 extant spotbugs
warnings. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 44s | | the patch passed |
| +1 :green_heart: | compile | 12m 21s | | the patch passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javac | 12m 21s | | the patch passed |
| +1 :green_heart: | compile | 10m 37s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | javac | 10m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 40s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 15s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 15s | | the patch passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javadoc | 1m 2s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | spotbugs | 2m 10s | | the patch passed |
_ Other Tests _ |
| -1 :x: | unit | 9m 12s |
[/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/3/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not
generate ASF License warnings. |
| | | 89m 53s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.fs.sftp.TestSFTPFileSystem |
| | hadoop.io.compress.snappy.TestSnappyCompressorDecompressor |
| | hadoop.util.TestDiskCheckerWithDiskIo |
| | hadoop.io.compress.TestCompressorDecompressor |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4091 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 6b99d20a8120 4.15.0-161-generic #169-Ubuntu SMP Fri Oct 15
13:41:54 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-2.10 / 1326ef2a4f0d22c365603144812fac73dfa1f962 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/zulu-7-amd64:Azul Systems,
Inc.-1.7.0_262-b10 /usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/3/testReport/ |
| Max. process+thread count | 1393 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/3/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
|
[jira] [Work logged] (HADOOP-18169) getDelegationTokens in ViewFs should also fetch the token from the fallback FS
[
https://issues.apache.org/jira/browse/HADOOP-18169?focusedWorklogId=746204=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746204
]
ASF GitHub Bot logged work on HADOOP-18169:
---
Author: ASF GitHub Bot
Created on: 22/Mar/22 22:08
Start Date: 22/Mar/22 22:08
Worklog Time Spent: 10m
Work Description: abhishekdas99 commented on a change in pull request
#4094:
URL: https://github.com/apache/hadoop/pull/4094#discussion_r832668185
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsLinkFallback.java
##
@@ -182,6 +184,26 @@ public void
testMkdirOfNewDirWithOutMatchingToMountOrFallbackDirTree()
assertTrue(fsTarget.exists(test));
}
+ /**
+ * Test getDelegationToken when fallback is configured
+ */
+ @Test
+ public void testGetDelegationToken() throws IOException {
Review comment:
If we are specifically testing for fallback, can you please change the
name to include fallback?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 746204)
Time Spent: 40m (was: 0.5h)
> getDelegationTokens in ViewFs should also fetch the token from the fallback FS
> --
>
> Key: HADOOP-18169
> URL: https://issues.apache.org/jira/browse/HADOOP-18169
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Xing Lin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> getDelegationTokens in ViewFs does not include the delegationToken from the
> fallback FS, while it should.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] abhishekdas99 commented on a change in pull request #4094: HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS
abhishekdas99 commented on a change in pull request #4094:
URL: https://github.com/apache/hadoop/pull/4094#discussion_r832668185
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsLinkFallback.java
##
@@ -182,6 +184,26 @@ public void
testMkdirOfNewDirWithOutMatchingToMountOrFallbackDirTree()
assertTrue(fsTarget.exists(test));
}
+ /**
+ * Test getDelegationToken when fallback is configured
+ */
+ @Test
+ public void testGetDelegationToken() throws IOException {
Review comment:
If we are specifically testing for fallback, can you please change the
name to include fallback?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18167) Add metrics to track delegation token secret manager operations
[ https://issues.apache.org/jira/browse/HADOOP-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510910#comment-17510910 ] Hector Sandoval Chaverri commented on HADOOP-18167: --- Hi [~ste...@apache.org], I saw that a few classes , such as S3AInstrumentation, implement IOStatisticsSource and use IOStatisticsStore to track different counters. Is this the approach that you think we should follow? Could you also help explain what's the consumer of the IOStatistics? > Add metrics to track delegation token secret manager operations > --- > > Key: HADOOP-18167 > URL: https://issues.apache.org/jira/browse/HADOOP-18167 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > New metrics to track operations that store, update and remove delegation > tokens in implementations of AbstractDelegationTokenSecretManager. This will > help evaluate the impact of using different secret managers and add > optimizations. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18167) Add metrics to track delegation token secret manager operations
[
https://issues.apache.org/jira/browse/HADOOP-18167?focusedWorklogId=746153=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746153
]
ASF GitHub Bot logged work on HADOOP-18167:
---
Author: ASF GitHub Bot
Created on: 22/Mar/22 21:10
Start Date: 22/Mar/22 21:10
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #4092:
URL: https://github.com/apache/hadoop/pull/4092#issuecomment-1075642781
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 22s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 34s | | trunk passed |
| +1 :green_heart: | compile | 24m 39s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 21m 6s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 42s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 10s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 0s | | the patch passed |
| +1 :green_heart: | compile | 23m 59s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 23m 59s | | the patch passed |
| +1 :green_heart: | compile | 21m 3s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 21m 3s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4092/1/artifact/out/blanks-eol.txt)
| The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 1m 0s |
[/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4092/1/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt)
| hadoop-common-project/hadoop-common: The patch generated 8 new + 74
unchanged - 0 fixed = 82 total (was 74) |
| +1 :green_heart: | mvnsite | 1m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 4s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 38s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 43s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 45s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 230m 29s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4092/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4092 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 8a53c3388f5c 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5
11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 373b7db473260e8373a0ec8a12a57ea85e296269 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test
[GitHub] [hadoop] hadoop-yetus commented on pull request #4092: HADOOP-18167. Add metrics to track delegation token secret manager op…
hadoop-yetus commented on pull request #4092:
URL: https://github.com/apache/hadoop/pull/4092#issuecomment-1075642781
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 22s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 34s | | trunk passed |
| +1 :green_heart: | compile | 24m 39s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 21m 6s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 42s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 10s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 0s | | the patch passed |
| +1 :green_heart: | compile | 23m 59s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 23m 59s | | the patch passed |
| +1 :green_heart: | compile | 21m 3s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 21m 3s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4092/1/artifact/out/blanks-eol.txt)
| The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 1m 0s |
[/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4092/1/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt)
| hadoop-common-project/hadoop-common: The patch generated 8 new + 74
unchanged - 0 fixed = 82 total (was 74) |
| +1 :green_heart: | mvnsite | 1m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 4s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 38s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 43s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 45s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 230m 29s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4092/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4092 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 8a53c3388f5c 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5
11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 373b7db473260e8373a0ec8a12a57ea85e296269 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4092/1/testReport/ |
| Max. process+thread count | 1237 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4092/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache
[GitHub] [hadoop] hadoop-yetus commented on pull request #4093: YARN-11093: fix for reading files in timeline server in chronological order in case of fs-support-append to false
hadoop-yetus commented on pull request #4093:
URL: https://github.com/apache/hadoop/pull/4093#issuecomment-1075605670
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 59s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 2s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 45s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 29s | | trunk passed |
| +1 :green_heart: | compile | 12m 1s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 9m 34s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 44s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 18s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 52s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 18s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 1s | | the patch passed |
| +1 :green_heart: | compile | 11m 35s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 11m 35s | | the patch passed |
| +1 :green_heart: | compile | 10m 39s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 10m 39s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4093/1/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 1m 51s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4093/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 170 unchanged
- 0 fixed = 171 total (was 170) |
| +1 :green_heart: | mvnsite | 1m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 24s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 24s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 0s | | hadoop-yarn-api in the patch
passed. |
| -1 :x: | unit | 11m 59s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4093/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt)
| hadoop-yarn-server-timeline-pluginstorage in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 169m 58s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4093/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4093 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux a37fe22eaa11 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5
11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
[jira] [Work logged] (HADOOP-18169) getDelegationTokens in ViewFs should also fetch the token from the fallback FS
[ https://issues.apache.org/jira/browse/HADOOP-18169?focusedWorklogId=746103=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746103 ] ASF GitHub Bot logged work on HADOOP-18169: --- Author: ASF GitHub Bot Created on: 22/Mar/22 20:23 Start Date: 22/Mar/22 20:23 Worklog Time Spent: 10m Work Description: xinglin commented on pull request #4094: URL: https://github.com/apache/hadoop/pull/4094#issuecomment-1075602418 > Could you add a pointer to where/how the corresponding logic lives in the non-abstract filesystem object, just to easily verify the logic is the same The logic is in DelegationTokenIssuer.java for FileSystem.java/ViewFileSystem.java. `collectDelegationTokens()` in DelegationTokenIssuer will collect token for the current issuer and all AdditionalTokenIssuers. In FileSystem.java, we overwrite `getAdditionalTokenIssuers()` to call `getChildFileSystem()` and ViewFileSystem overwrites `getChildFileSystem()`, to include the fallback FS as well. So, ViewFileSystem.java is good here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 746103) Time Spent: 0.5h (was: 20m) > getDelegationTokens in ViewFs should also fetch the token from the fallback FS > -- > > Key: HADOOP-18169 > URL: https://issues.apache.org/jira/browse/HADOOP-18169 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xing Lin >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > getDelegationTokens in ViewFs does not include the delegationToken from the > fallback FS, while it should. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xinglin commented on pull request #4094: HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS
xinglin commented on pull request #4094: URL: https://github.com/apache/hadoop/pull/4094#issuecomment-1075602418 > Could you add a pointer to where/how the corresponding logic lives in the non-abstract filesystem object, just to easily verify the logic is the same The logic is in DelegationTokenIssuer.java for FileSystem.java/ViewFileSystem.java. `collectDelegationTokens()` in DelegationTokenIssuer will collect token for the current issuer and all AdditionalTokenIssuers. In FileSystem.java, we overwrite `getAdditionalTokenIssuers()` to call `getChildFileSystem()` and ViewFileSystem overwrites `getChildFileSystem()`, to include the fallback FS as well. So, ViewFileSystem.java is good here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4089: HDFS-16515. Improve ec exception message
hadoop-yetus commented on pull request #4089:
URL: https://github.com/apache/hadoop/pull/4089#issuecomment-1075580681
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 14s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 6s | | trunk passed |
| +1 :green_heart: | compile | 1m 31s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 19s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 29s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 29s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 29s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 19s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 1m 24s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 1m 16s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 23s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 373m 48s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 485m 28s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4089/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4089 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 02502b1e4a91 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5
11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9073517068c58944279fa81478bb3f3cb6e21543 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4089/2/testReport/ |
| Max. process+thread count | 2005 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4089/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4091: HDFS-16517 Distance metric is wrong for non-DN machines in 2.10.
hadoop-yetus commented on pull request #4091:
URL: https://github.com/apache/hadoop/pull/4091#issuecomment-1075543846
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 10s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-2.10 Compile Tests _ |
| +1 :green_heart: | mvninstall | 15m 55s | | branch-2.10 passed |
| +1 :green_heart: | compile | 14m 15s | | branch-2.10 passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | compile | 13m 9s | | branch-2.10 passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | checkstyle | 0m 36s | | branch-2.10 passed |
| +1 :green_heart: | mvnsite | 1m 25s | | branch-2.10 passed |
| +1 :green_heart: | javadoc | 1m 17s | | branch-2.10 passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javadoc | 1m 0s | | branch-2.10 passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| -1 :x: | spotbugs | 2m 4s |
[/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/2/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html)
| hadoop-common-project/hadoop-common in branch-2.10 has 2 extant spotbugs
warnings. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 50s | | the patch passed |
| +1 :green_heart: | compile | 13m 52s | | the patch passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 |
| -1 :x: | javac | 13m 52s |
[/results-compile-javac-root-jdkAzulSystems,Inc.-1.7.0_262-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/2/artifact/out/results-compile-javac-root-jdkAzulSystems,Inc.-1.7.0_262-b10.txt)
| root-jdkAzulSystems,Inc.-1.7.0_262-b10 with JDK Azul Systems,
Inc.-1.7.0_262-b10 generated 1 new + 1678 unchanged - 1 fixed = 1679 total (was
1679) |
| +1 :green_heart: | compile | 11m 35s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| -1 :x: | javac | 11m 35s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/2/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07.txt)
| root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 generated 1 new + 1578
unchanged - 1 fixed = 1579 total (was 1579) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 35s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 17s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 12s | | the patch passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | spotbugs | 2m 20s | | the patch passed |
_ Other Tests _ |
| -1 :x: | unit | 10m 29s |
[/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not
generate ASF License warnings. |
| | | 110m 7s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.io.compress.TestCompressorDecompressor |
| | hadoop.io.compress.snappy.TestSnappyCompressorDecompressor |
| | hadoop.fs.sftp.TestSFTPFileSystem |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4091 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux a08702ad2c29 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5
11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-2.10 /
[jira] [Work logged] (HADOOP-18160) `org.wildfly.openssl` should not be shaded by Hadoop build
[
https://issues.apache.org/jira/browse/HADOOP-18160?focusedWorklogId=746042=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746042
]
ASF GitHub Bot logged work on HADOOP-18160:
---
Author: ASF GitHub Bot
Created on: 22/Mar/22 19:01
Start Date: 22/Mar/22 19:01
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #4074:
URL: https://github.com/apache/hadoop/pull/4074#issuecomment-1075521131
ok, merged in to branch-3.3 and trunk. thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 746042)
Time Spent: 1.5h (was: 1h 20m)
> `org.wildfly.openssl` should not be shaded by Hadoop build
> --
>
> Key: HADOOP-18160
> URL: https://issues.apache.org/jira/browse/HADOOP-18160
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.3.1
> Environment: hadoop 3.3.1
> spark 3.2.1
> JDK8
>Reporter: André F.
>Assignee: André F.
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.3.3
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> `org.wildfly.openssl` is a runtime library and its references are being
> shaded on Hadoop, breaking the integration with other frameworks like Spark,
> whenever the "fs.s3a.ssl.channel.mode" is set to "openssl". The error
> produced in this situation is:
> {code:java}
> Suppressed: java.lang.NoClassDefFoundError:
> org/apache/hadoop/shaded/org/wildfly/openssl/OpenSSLProvider{code}
> Whenever it tries to be instantiated from the `DelegatingSSLSocketFactory`.
> Spark tries to add it to its classpath without the shade, thus creating this
> issue.
> Dependencies which are not on "compile" scope should probably not be shaded
> to avoid this kind of integration issues.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18160) `org.wildfly.openssl` should not be shaded by Hadoop build
[
https://issues.apache.org/jira/browse/HADOOP-18160?focusedWorklogId=746044=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746044
]
ASF GitHub Bot logged work on HADOOP-18160:
---
Author: ASF GitHub Bot
Created on: 22/Mar/22 19:01
Start Date: 22/Mar/22 19:01
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #4074:
URL: https://github.com/apache/hadoop/pull/4074#issuecomment-1069297089
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 746044)
Time Spent: 1h 40m (was: 1.5h)
> `org.wildfly.openssl` should not be shaded by Hadoop build
> --
>
> Key: HADOOP-18160
> URL: https://issues.apache.org/jira/browse/HADOOP-18160
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.3.1
> Environment: hadoop 3.3.1
> spark 3.2.1
> JDK8
>Reporter: André F.
>Assignee: André F.
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.3.3
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> `org.wildfly.openssl` is a runtime library and its references are being
> shaded on Hadoop, breaking the integration with other frameworks like Spark,
> whenever the "fs.s3a.ssl.channel.mode" is set to "openssl". The error
> produced in this situation is:
> {code:java}
> Suppressed: java.lang.NoClassDefFoundError:
> org/apache/hadoop/shaded/org/wildfly/openssl/OpenSSLProvider{code}
> Whenever it tries to be instantiated from the `DelegatingSSLSocketFactory`.
> Spark tries to add it to its classpath without the shade, thus creating this
> issue.
> Dependencies which are not on "compile" scope should probably not be shaded
> to avoid this kind of integration issues.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #4074: HADOOP-18160 Avoid shading wildfly.openssl runtime dependency
hadoop-yetus removed a comment on pull request #4074: URL: https://github.com/apache/hadoop/pull/4074#issuecomment-1069297089 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4091: HDFS-16517 Distance metric is wrong for non-DN machines in 2.10.
hadoop-yetus commented on pull request #4091:
URL: https://github.com/apache/hadoop/pull/4091#issuecomment-1075521316
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 8m 27s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-2.10 Compile Tests _ |
| +1 :green_heart: | mvninstall | 14m 47s | | branch-2.10 passed |
| +1 :green_heart: | compile | 13m 52s | | branch-2.10 passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | compile | 11m 25s | | branch-2.10 passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | checkstyle | 0m 43s | | branch-2.10 passed |
| +1 :green_heart: | mvnsite | 1m 20s | | branch-2.10 passed |
| +1 :green_heart: | javadoc | 1m 16s | | branch-2.10 passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javadoc | 1m 7s | | branch-2.10 passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| -1 :x: | spotbugs | 2m 6s |
[/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/1/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html)
| hadoop-common-project/hadoop-common in branch-2.10 has 2 extant spotbugs
warnings. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 45s | | the patch passed |
| +1 :green_heart: | compile | 13m 8s | | the patch passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javac | 13m 8s | | the patch passed |
| +1 :green_heart: | compile | 11m 16s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | javac | 11m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 41s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 16s | | the patch passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javadoc | 1m 4s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | spotbugs | 2m 9s | | the patch passed |
_ Other Tests _ |
| -1 :x: | unit | 9m 20s |
[/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not
generate ASF License warnings. |
| | | 101m 36s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.fs.sftp.TestSFTPFileSystem |
| | hadoop.io.compress.snappy.TestSnappyCompressorDecompressor |
| | hadoop.io.compress.TestCompressorDecompressor |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4091 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 6317914f6ae6 4.15.0-161-generic #169-Ubuntu SMP Fri Oct 15
13:41:54 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-2.10 / be14b0cad50fed2d8ecb42d236aed397bebc9609 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/zulu-7-amd64:Azul Systems,
Inc.-1.7.0_262-b10 /usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/1/testReport/ |
| Max. process+thread count | 1397 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4091/1/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT
[GitHub] [hadoop] steveloughran commented on pull request #4074: HADOOP-18160 Avoid shading wildfly.openssl runtime dependency
steveloughran commented on pull request #4074: URL: https://github.com/apache/hadoop/pull/4074#issuecomment-1075521131 ok, merged in to branch-3.3 and trunk. thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3470: YARN-10965. Centralize queue resource calculation based on CapacityVectors
hadoop-yetus commented on pull request #3470:
URL: https://github.com/apache/hadoop/pull/3470#issuecomment-1075518999
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 8 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 40s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 58s | | trunk passed |
| +1 :green_heart: | compile | 9m 45s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 8m 26s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 44s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 18s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 59s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 50s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 13s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 13s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 22m 35s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 31s | | the patch passed |
| +1 :green_heart: | compile | 9m 2s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 9m 2s |
[/results-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3470/14/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 392 unchanged
- 1 fixed = 393 total (was 393) |
| +1 :green_heart: | compile | 8m 18s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 8m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 38s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3470/14/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 92 new + 155 unchanged
- 0 fixed = 247 total (was 155) |
| +1 :green_heart: | mvnsite | 2m 8s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 46s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 41s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | spotbugs | 2m 13s |
[/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3470/14/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | shadedclient | 21m 32s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 7s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 97m 12s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not
generate ASF License warnings. |
| | | 241m 30s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| |
[jira] [Assigned] (HADOOP-18160) `org.wildfly.openssl` should not be shaded by Hadoop build
[
https://issues.apache.org/jira/browse/HADOOP-18160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran reassigned HADOOP-18160:
---
Assignee: André F.
> `org.wildfly.openssl` should not be shaded by Hadoop build
> --
>
> Key: HADOOP-18160
> URL: https://issues.apache.org/jira/browse/HADOOP-18160
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.3.1
> Environment: hadoop 3.3.1
> spark 3.2.1
> JDK8
>Reporter: André F.
>Assignee: André F.
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.3.3
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> `org.wildfly.openssl` is a runtime library and its references are being
> shaded on Hadoop, breaking the integration with other frameworks like Spark,
> whenever the "fs.s3a.ssl.channel.mode" is set to "openssl". The error
> produced in this situation is:
> {code:java}
> Suppressed: java.lang.NoClassDefFoundError:
> org/apache/hadoop/shaded/org/wildfly/openssl/OpenSSLProvider{code}
> Whenever it tries to be instantiated from the `DelegatingSSLSocketFactory`.
> Spark tries to add it to its classpath without the shade, thus creating this
> issue.
> Dependencies which are not on "compile" scope should probably not be shaded
> to avoid this kind of integration issues.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18160) `org.wildfly.openssl` should not be shaded by Hadoop build
[
https://issues.apache.org/jira/browse/HADOOP-18160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-18160.
-
Fix Version/s: 3.3.3
Resolution: Fixed
> `org.wildfly.openssl` should not be shaded by Hadoop build
> --
>
> Key: HADOOP-18160
> URL: https://issues.apache.org/jira/browse/HADOOP-18160
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.3.1
> Environment: hadoop 3.3.1
> spark 3.2.1
> JDK8
>Reporter: André F.
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.3.3
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> `org.wildfly.openssl` is a runtime library and its references are being
> shaded on Hadoop, breaking the integration with other frameworks like Spark,
> whenever the "fs.s3a.ssl.channel.mode" is set to "openssl". The error
> produced in this situation is:
> {code:java}
> Suppressed: java.lang.NoClassDefFoundError:
> org/apache/hadoop/shaded/org/wildfly/openssl/OpenSSLProvider{code}
> Whenever it tries to be instantiated from the `DelegatingSSLSocketFactory`.
> Spark tries to add it to its classpath without the shade, thus creating this
> issue.
> Dependencies which are not on "compile" scope should probably not be shaded
> to avoid this kind of integration issues.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18169) getDelegationTokens in ViewFs should also fetch the token from the fallback FS
[
https://issues.apache.org/jira/browse/HADOOP-18169?focusedWorklogId=746014=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746014
]
ASF GitHub Bot logged work on HADOOP-18169:
---
Author: ASF GitHub Bot
Created on: 22/Mar/22 18:25
Start Date: 22/Mar/22 18:25
Worklog Time Spent: 10m
Work Description: mccormickt12 commented on a change in pull request
#4094:
URL: https://github.com/apache/hadoop/pull/4094#discussion_r832492192
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
##
@@ -746,6 +746,17 @@ public void setVerifyChecksum(final boolean verifyChecksum)
result.addAll(tokens);
}
}
+
+// Add tokens from fallback FS
+if (this.fsState.getRootFallbackLink() != null) {
+ AbstractFileSystem linkedFallbackFs =
Review comment:
nit: these should be linkFallbackFs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 746014)
Time Spent: 20m (was: 10m)
> getDelegationTokens in ViewFs should also fetch the token from the fallback FS
> --
>
> Key: HADOOP-18169
> URL: https://issues.apache.org/jira/browse/HADOOP-18169
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Xing Lin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> getDelegationTokens in ViewFs does not include the delegationToken from the
> fallback FS, while it should.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4060: YARN-11084. Introduce new config to specify AM default node-label whe…
hadoop-yetus commented on pull request #4060:
URL: https://github.com/apache/hadoop/pull/4060#issuecomment-1075471974
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 2m 25s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 39s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 8s | | trunk passed |
| +1 :green_heart: | compile | 11m 8s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 9m 1s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 14s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 58s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 4s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 52s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 27s | |
branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site no spotbugs output file
(spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 24m 4s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 25s | | the patch passed |
| +1 :green_heart: | compile | 10m 34s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 10m 34s | | the patch passed |
| +1 :green_heart: | compile | 9m 0s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 9m 0s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 37s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4060/7/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 229 unchanged
- 0 fixed = 230 total (was 229) |
| +1 :green_heart: | mvnsite | 3m 18s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 2m 46s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 39s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 24s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site has no data from spotbugs |
| +1 :green_heart: | shadedclient | 23m 8s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 2s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 4m 43s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 100m 46s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 0m 24s | | hadoop-yarn-site in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 280m 0s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4060/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4060 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml markdownlint |
| uname | Linux f3f6bee8b187 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18
11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 147c2d492aa61854011dff48ef8017bd5c017cce |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
[GitHub] [hadoop] mccormickt12 commented on a change in pull request #4094: HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS
mccormickt12 commented on a change in pull request #4094:
URL: https://github.com/apache/hadoop/pull/4094#discussion_r832492192
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
##
@@ -746,6 +746,17 @@ public void setVerifyChecksum(final boolean verifyChecksum)
result.addAll(tokens);
}
}
+
+// Add tokens from fallback FS
+if (this.fsState.getRootFallbackLink() != null) {
+ AbstractFileSystem linkedFallbackFs =
Review comment:
nit: these should be linkFallbackFs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18169) getDelegationTokens in ViewFs should also fetch the token from the fallback FS
[ https://issues.apache.org/jira/browse/HADOOP-18169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18169: Labels: pull-request-available (was: ) > getDelegationTokens in ViewFs should also fetch the token from the fallback FS > -- > > Key: HADOOP-18169 > URL: https://issues.apache.org/jira/browse/HADOOP-18169 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xing Lin >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > getDelegationTokens in ViewFs does not include the delegationToken from the > fallback FS, while it should. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18169) getDelegationTokens in ViewFs should also fetch the token from the fallback FS
[ https://issues.apache.org/jira/browse/HADOOP-18169?focusedWorklogId=745998=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745998 ] ASF GitHub Bot logged work on HADOOP-18169: --- Author: ASF GitHub Bot Created on: 22/Mar/22 18:04 Start Date: 22/Mar/22 18:04 Worklog Time Spent: 10m Work Description: xinglin opened a new pull request #4094: URL: https://github.com/apache/hadoop/pull/4094 ### Description of PR ### How was this patch tested? mvn test -Dtest="TestViewFs*" in both hadoop-common-project and hadoop-hdfs-project. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745998) Remaining Estimate: 0h Time Spent: 10m > getDelegationTokens in ViewFs should also fetch the token from the fallback FS > -- > > Key: HADOOP-18169 > URL: https://issues.apache.org/jira/browse/HADOOP-18169 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xing Lin >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > getDelegationTokens in ViewFs does not include the delegationToken from the > fallback FS, while it should. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xinglin opened a new pull request #4094: HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS
xinglin opened a new pull request #4094: URL: https://github.com/apache/hadoop/pull/4094 ### Description of PR ### How was this patch tested? mvn test -Dtest="TestViewFs*" in both hadoop-common-project and hadoop-hdfs-project. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-14661) S3A to support Requester Pays Buckets
[ https://issues.apache.org/jira/browse/HADOOP-14661?focusedWorklogId=745990=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745990 ] ASF GitHub Bot logged work on HADOOP-14661: --- Author: ASF GitHub Bot Created on: 22/Mar/22 17:57 Start Date: 22/Mar/22 17:57 Worklog Time Spent: 10m Work Description: dannycjones commented on pull request #3962: URL: https://github.com/apache/hadoop/pull/3962#issuecomment-1075435015 @steveloughran I caught those after I pushed most of the other fixes from the last feedback and pushed a second commit ae8e812e11148bcf007d51fbf94356229df4fae1. There's two Yetus comments, latest one seems happy. Are we good to merge? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745990) Time Spent: 5h 50m (was: 5h 40m) > S3A to support Requester Pays Buckets > - > > Key: HADOOP-14661 > URL: https://issues.apache.org/jira/browse/HADOOP-14661 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, util >Affects Versions: 3.0.0-alpha3 >Reporter: Mandus Momberg >Assignee: Mandus Momberg >Priority: Minor > Labels: pull-request-available > Attachments: HADOOP-14661.patch > > Original Estimate: 2h > Time Spent: 5h 50m > Remaining Estimate: 0h > > Amazon S3 has the ability to charge the requester for the cost of accessing > S3. This is called Requester Pays Buckets. > In order to access these buckets, each request needs to be signed with a > specific header. > http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] dannycjones commented on pull request #3962: HADOOP-14661. Add S3 requester pays bucket support to S3A
dannycjones commented on pull request #3962: URL: https://github.com/apache/hadoop/pull/3962#issuecomment-1075435015 @steveloughran I caught those after I pushed most of the other fixes from the last feedback and pushed a second commit ae8e812e11148bcf007d51fbf94356229df4fae1. There's two Yetus comments, latest one seems happy. Are we good to merge? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] GauthamBanasandra merged pull request #4086: HDFS-16471. Make HDFS ls tool cross platform
GauthamBanasandra merged pull request #4086: URL: https://github.com/apache/hadoop/pull/4086 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18169) getDelegationTokens in ViewFs should also fetch the token from the fallback FS
[ https://issues.apache.org/jira/browse/HADOOP-18169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xing Lin updated HADOOP-18169: -- Summary: getDelegationTokens in ViewFs should also fetch the token from the fallback FS (was: getDelegationToken in ViewFs should also fetch the token from the fallback FS) > getDelegationTokens in ViewFs should also fetch the token from the fallback FS > -- > > Key: HADOOP-18169 > URL: https://issues.apache.org/jira/browse/HADOOP-18169 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xing Lin >Priority: Major > > getDelegationToken in ViewFs does not include the delegationToken from the > fallback FS, while it should. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18169) getDelegationTokens in ViewFs should also fetch the token from the fallback FS
[ https://issues.apache.org/jira/browse/HADOOP-18169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xing Lin updated HADOOP-18169: -- Description: getDelegationTokens in ViewFs does not include the delegationToken from the fallback FS, while it should. (was: getDelegationToken in ViewFs does not include the delegationToken from the fallback FS, while it should. ) > getDelegationTokens in ViewFs should also fetch the token from the fallback FS > -- > > Key: HADOOP-18169 > URL: https://issues.apache.org/jira/browse/HADOOP-18169 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xing Lin >Priority: Major > > getDelegationTokens in ViewFs does not include the delegationToken from the > fallback FS, while it should. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18169) getDelegationToken in ViewFs should also fetch the token from the fallback FS
Xing Lin created HADOOP-18169: - Summary: getDelegationToken in ViewFs should also fetch the token from the fallback FS Key: HADOOP-18169 URL: https://issues.apache.org/jira/browse/HADOOP-18169 Project: Hadoop Common Issue Type: Bug Reporter: Xing Lin getDelegationToken in ViewFs does not include the delegationToken from the fallback FS, while it should. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] guptashailesh92 opened a new pull request #4093: YARN-11093: fix for reading files in timeline server in chronological order in case of fs-support-append to false
guptashailesh92 opened a new pull request #4093: URL: https://github.com/apache/hadoop/pull/4093 ### Description of PR In our setup, we are using Hive and Tez and using offline tez-ui mode, where we copy the ATS events files to other place and start tez-ui. To keep the event files small, we are using fs-support-append = false, and creating a new file for writing the events. We can see that, in this mode, file name contains [suffix](https://github.com/apache/hadoop/blob/1d5650c4d0acf33f141d593762682b3603523104/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/FileSystemTimelineWriter.java#L382) as timestamp. But at the time of [read](https://github.com/apache/hadoop/blob/1d5650c4d0acf33f141d593762682b3603523104/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java#L844), we are relying on the FS to provide correct file order. If that fails, we are adding event out of order, leading to event being discarded, or incorrect information. Fix could be sorting of the file names, based on suffix if append mode is not used. sample file names: summarylog-appattempt_1647348120288_0001_01_460237 entitylog-timelineEntityGroupId_1647348120288_1_dag_1647348120288_0001_1_673147 ### How was this patch tested? 1. Added conf `yarn.timeline-service.fs-support-append` to false. 2. Ran a job, which created small files for domain, entity and summary log. 3. Ran ATS by copying all the entity files to custom location, and checked tez-ui with and without the change. Was able to see latest status of the task correctly after the change. ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18160) `org.wildfly.openssl` should not be shaded by Hadoop build
[
https://issues.apache.org/jira/browse/HADOOP-18160?focusedWorklogId=745961=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745961
]
ASF GitHub Bot logged work on HADOOP-18160:
---
Author: ASF GitHub Bot
Created on: 22/Mar/22 17:32
Start Date: 22/Mar/22 17:32
Worklog Time Spent: 10m
Work Description: steveloughran merged pull request #4074:
URL: https://github.com/apache/hadoop/pull/4074
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 745961)
Time Spent: 1h 20m (was: 1h 10m)
> `org.wildfly.openssl` should not be shaded by Hadoop build
> --
>
> Key: HADOOP-18160
> URL: https://issues.apache.org/jira/browse/HADOOP-18160
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.3.1
> Environment: hadoop 3.3.1
> spark 3.2.1
> JDK8
>Reporter: André F.
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> `org.wildfly.openssl` is a runtime library and its references are being
> shaded on Hadoop, breaking the integration with other frameworks like Spark,
> whenever the "fs.s3a.ssl.channel.mode" is set to "openssl". The error
> produced in this situation is:
> {code:java}
> Suppressed: java.lang.NoClassDefFoundError:
> org/apache/hadoop/shaded/org/wildfly/openssl/OpenSSLProvider{code}
> Whenever it tries to be instantiated from the `DelegatingSSLSocketFactory`.
> Spark tries to add it to its classpath without the shade, thus creating this
> issue.
> Dependencies which are not on "compile" scope should probably not be shaded
> to avoid this kind of integration issues.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #4074: HADOOP-18160 Avoid shading wildfly.openssl runtime dependency
steveloughran merged pull request #4074: URL: https://github.com/apache/hadoop/pull/4074 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18167) Add metrics to track delegation token secret manager operations
[ https://issues.apache.org/jira/browse/HADOOP-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18167: Labels: pull-request-available (was: ) > Add metrics to track delegation token secret manager operations > --- > > Key: HADOOP-18167 > URL: https://issues.apache.org/jira/browse/HADOOP-18167 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > New metrics to track operations that store, update and remove delegation > tokens in implementations of AbstractDelegationTokenSecretManager. This will > help evaluate the impact of using different secret managers and add > optimizations. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18167) Add metrics to track delegation token secret manager operations
[ https://issues.apache.org/jira/browse/HADOOP-18167?focusedWorklogId=745957=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745957 ] ASF GitHub Bot logged work on HADOOP-18167: --- Author: ASF GitHub Bot Created on: 22/Mar/22 17:27 Start Date: 22/Mar/22 17:27 Worklog Time Spent: 10m Work Description: hchaverri opened a new pull request #4092: URL: https://github.com/apache/hadoop/pull/4092 …erations ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745957) Remaining Estimate: 0h Time Spent: 10m > Add metrics to track delegation token secret manager operations > --- > > Key: HADOOP-18167 > URL: https://issues.apache.org/jira/browse/HADOOP-18167 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > New metrics to track operations that store, update and remove delegation > tokens in implementations of AbstractDelegationTokenSecretManager. This will > help evaluate the impact of using different secret managers and add > optimizations. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hchaverri opened a new pull request #4092: HADOOP-18167. Add metrics to track delegation token secret manager op…
hchaverri opened a new pull request #4092: URL: https://github.com/apache/hadoop/pull/4092 …erations ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18154) S3A Authentication to support WebIdentity
[ https://issues.apache.org/jira/browse/HADOOP-18154?focusedWorklogId=745954=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745954 ] ASF GitHub Bot logged work on HADOOP-18154: --- Author: ASF GitHub Bot Created on: 22/Mar/22 17:22 Start Date: 22/Mar/22 17:22 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #4070: URL: https://github.com/apache/hadoop/pull/4070#issuecomment-1075401037 Looking at the `WebIdentityTokenCredentialsProvider` I see that if it doesn't get the parameters then it will fall back to environment variables. We absolutely do not want to be picking up env vars as it will only create support issues where configurations only work on a certain machines. (actually, we can ignore the session name settings as they are harmless) I'm going to propose we go with @dannycjones's suggestion and support the whole set of values and have the prefix `fs.s3a.webidentity` for all of them. for the arn, we could have a property `fs.s3a.webidentity.role.arn` but, what should we do if it wasn't set? 1. fail to initialize 2. have that null value force the env var lookup. I don't see any way to a completely block the environment variable resolution, which is a pain. I also see in the internal Library classes that sometimes roles are set up with an external ID, but it is not possible here. Is that an issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745954) Time Spent: 2h 50m (was: 2h 40m) > S3A Authentication to support WebIdentity > - > > Key: HADOOP-18154 > URL: https://issues.apache.org/jira/browse/HADOOP-18154 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 2.10.1 >Reporter: Ju Clarysse >Assignee: Ju Clarysse >Priority: Major > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > We are using the latest version of > [delta-sharing|https://github.com/delta-io/delta-sharing] which takes > advantage of > [hadoop-aws|https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html] > (S3A) connector in [Hadoop release version > 2.10.1|https://github.com/apache/hadoop/tree/rel/release-2.10.1] to mount an > AWS S3 File System. In our particular setup, all services are operated in > Amazon Elastic Kubernetes Service (EKS) and need to comply to the AWS > security concept [IAM roles for service > accounts|https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html] > (IRSA). > As [Delta sharing S3 connection|https://github.com/delta-io/delta-sharing#s3] > doesn't offer any corresponding support, we patched hadoop-aws-2.10.1 to > address this need via a new credentials provider class > org.apache.hadoop.fs.s3a.OIDCTokenCredentialsProvider. We also upgraded > dependency aws-java-sdk-bundle to its latest version 1.12.167 as [AWS > WebIdentityTokenCredentialsProvider > class|https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/WebIdentityTokenCredentialsProvider.html%E2%80%A6] > was not yet available in original version 1.11.271. > We believe that other delta-sharing users could benefit from this short-term > contribution. Then sooner or later, delta-sharing owners will have to upgrade > their project to a more recent version of hadoop-aws that is probably more > widely used. The effort to promote this change is probably low. > Additional note: AWS WebIdentityTokenCredentialsProvider class is directly > supported by Spark applications submitted with configuration properties > `spark.hadoop.fs.s3a.aws.credentials.provider`and > `spark.kubernetes.authenticate.submission.oauthToken` > ([doc|https://spark.apache.org/docs/latest/running-on-kubernetes.html#spark-properties]). > So bringing this support to Hadoop will primarily be interesting for > non-Spark users. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #4070: HADOOP-18154. S3A Authentication to support WebIdentity
steveloughran commented on pull request #4070: URL: https://github.com/apache/hadoop/pull/4070#issuecomment-1075401037 Looking at the `WebIdentityTokenCredentialsProvider` I see that if it doesn't get the parameters then it will fall back to environment variables. We absolutely do not want to be picking up env vars as it will only create support issues where configurations only work on a certain machines. (actually, we can ignore the session name settings as they are harmless) I'm going to propose we go with @dannycjones's suggestion and support the whole set of values and have the prefix `fs.s3a.webidentity` for all of them. for the arn, we could have a property `fs.s3a.webidentity.role.arn` but, what should we do if it wasn't set? 1. fail to initialize 2. have that null value force the env var lookup. I don't see any way to a completely block the environment variable resolution, which is a pain. I also see in the internal Library classes that sometimes roles are set up with an external ID, but it is not possible here. Is that an issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] omalley opened a new pull request #4091: HDFS-16517 Distance metric is wrong for non-DN machines in 2.10.
omalley opened a new pull request #4091: URL: https://github.com/apache/hadoop/pull/4091 ### Description of PR The distance metric used for machines in 2.10 that aren't in the NetworkTopology, because they aren't running DataNodes, is wrong. It means that off-rack and on-rack, but off-node, are both given a weight of 2. In normal Hadoop clusters, this isn't a big problem because they don't have clients that are on-rack but without DataNodes. For clusters that are striped (federated HDFS going across racks) or separate compute and storage that share racks are both really bad with this bug. ### How was this patch tested? Unit test added. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18160) `org.wildfly.openssl` should not be shaded by Hadoop build
[
https://issues.apache.org/jira/browse/HADOOP-18160?focusedWorklogId=745949=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745949
]
ASF GitHub Bot logged work on HADOOP-18160:
---
Author: ASF GitHub Bot
Created on: 22/Mar/22 17:12
Start Date: 22/Mar/22 17:12
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #4074:
URL: https://github.com/apache/hadoop/pull/4074#issuecomment-1069468112
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 34s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 9s | | trunk passed |
| +1 :green_heart: | compile | 0m 18s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 18s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | mvnsite | 0m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 19s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 19s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | shadedclient | 61m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 12s | | the patch passed |
| +1 :green_heart: | compile | 0m 12s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 12s | | the patch passed |
| +1 :green_heart: | compile | 0m 12s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 14s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 12s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 12s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | shadedclient | 23m 36s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 15s | | hadoop-client-api in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 109m 5s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4074/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4074 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell xml |
| uname | Linux 629688fc11e8 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5
11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 427a93ab844fa863e25f1893b105af9ee2c3cbfb |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4074/2/testReport/ |
| Max. process+thread count | 519 (vs. ulimit of 5500) |
| modules | C: hadoop-client-modules/hadoop-client-api U:
hadoop-client-modules/hadoop-client-api |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4074/2/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #4074: HADOOP-18160 Avoid shading wildfly.openssl runtime dependency
hadoop-yetus removed a comment on pull request #4074:
URL: https://github.com/apache/hadoop/pull/4074#issuecomment-1069468112
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 34s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 9s | | trunk passed |
| +1 :green_heart: | compile | 0m 18s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 18s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | mvnsite | 0m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 19s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 19s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | shadedclient | 61m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 12s | | the patch passed |
| +1 :green_heart: | compile | 0m 12s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 12s | | the patch passed |
| +1 :green_heart: | compile | 0m 12s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 14s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 12s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 12s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | shadedclient | 23m 36s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 15s | | hadoop-client-api in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 109m 5s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4074/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4074 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell xml |
| uname | Linux 629688fc11e8 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5
11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 427a93ab844fa863e25f1893b105af9ee2c3cbfb |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4074/2/testReport/ |
| Max. process+thread count | 519 (vs. ulimit of 5500) |
| modules | C: hadoop-client-modules/hadoop-client-api U:
hadoop-client-modules/hadoop-client-api |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4074/2/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail:
[jira] [Work logged] (HADOOP-18154) S3A Authentication to support WebIdentity
[ https://issues.apache.org/jira/browse/HADOOP-18154?focusedWorklogId=745945=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745945 ] ASF GitHub Bot logged work on HADOOP-18154: --- Author: ASF GitHub Bot Created on: 22/Mar/22 17:06 Start Date: 22/Mar/22 17:06 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #4070: URL: https://github.com/apache/hadoop/pull/4070#issuecomment-1068327215 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745945) Time Spent: 2h 40m (was: 2.5h) > S3A Authentication to support WebIdentity > - > > Key: HADOOP-18154 > URL: https://issues.apache.org/jira/browse/HADOOP-18154 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 2.10.1 >Reporter: Ju Clarysse >Assignee: Ju Clarysse >Priority: Major > Labels: pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > We are using the latest version of > [delta-sharing|https://github.com/delta-io/delta-sharing] which takes > advantage of > [hadoop-aws|https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html] > (S3A) connector in [Hadoop release version > 2.10.1|https://github.com/apache/hadoop/tree/rel/release-2.10.1] to mount an > AWS S3 File System. In our particular setup, all services are operated in > Amazon Elastic Kubernetes Service (EKS) and need to comply to the AWS > security concept [IAM roles for service > accounts|https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html] > (IRSA). > As [Delta sharing S3 connection|https://github.com/delta-io/delta-sharing#s3] > doesn't offer any corresponding support, we patched hadoop-aws-2.10.1 to > address this need via a new credentials provider class > org.apache.hadoop.fs.s3a.OIDCTokenCredentialsProvider. We also upgraded > dependency aws-java-sdk-bundle to its latest version 1.12.167 as [AWS > WebIdentityTokenCredentialsProvider > class|https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/WebIdentityTokenCredentialsProvider.html%E2%80%A6] > was not yet available in original version 1.11.271. > We believe that other delta-sharing users could benefit from this short-term > contribution. Then sooner or later, delta-sharing owners will have to upgrade > their project to a more recent version of hadoop-aws that is probably more > widely used. The effort to promote this change is probably low. > Additional note: AWS WebIdentityTokenCredentialsProvider class is directly > supported by Spark applications submitted with configuration properties > `spark.hadoop.fs.s3a.aws.credentials.provider`and > `spark.kubernetes.authenticate.submission.oauthToken` > ([doc|https://spark.apache.org/docs/latest/running-on-kubernetes.html#spark-properties]). > So bringing this support to Hadoop will primarily be interesting for > non-Spark users. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #4070: HADOOP-18154. S3A Authentication to support WebIdentity
hadoop-yetus removed a comment on pull request #4070: URL: https://github.com/apache/hadoop/pull/4070#issuecomment-1068327215 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18145) Fileutil's unzip method causes unzipped files to lose their original permissions
[
https://issues.apache.org/jira/browse/HADOOP-18145?focusedWorklogId=745931=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745931
]
ASF GitHub Bot logged work on HADOOP-18145:
---
Author: ASF GitHub Bot
Created on: 22/Mar/22 16:46
Start Date: 22/Mar/22 16:46
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#4036:
URL: https://github.com/apache/hadoop/pull/4036#discussion_r832391092
##
File path:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
##
@@ -706,35 +706,40 @@ public void testCreateLocalTempFile() throws IOException {
public void testUnZip() throws IOException {
// make sa simple zip
final File simpleZip = new File(del, FILE);
-OutputStream os = new FileOutputStream(simpleZip);
-ZipOutputStream tos = new ZipOutputStream(os);
-try {
- ZipEntry ze = new ZipEntry("foo");
- byte[] data = "some-content".getBytes("UTF-8");
- ze.setSize(data.length);
- tos.putNextEntry(ze);
- tos.write(data);
- tos.closeEntry();
- tos.flush();
- tos.finish();
-} finally {
- tos.close();
-}
-
-// successfully unzip it into an existing dir:
-FileUtil.unZip(simpleZip, tmp);
-// check result:
-assertTrue(new File(tmp, "foo").exists());
-assertEquals(12, new File(tmp, "foo").length());
-
-final File regularFile = new File(tmp, "QuickBrownFoxJumpsOverTheLazyDog");
-regularFile.createNewFile();
-assertTrue(regularFile.exists());
-try {
- FileUtil.unZip(simpleZip, regularFile);
- assertTrue("An IOException expected.", false);
-} catch (IOException ioe) {
- // okay
+try (OutputStream os = new FileOutputStream(simpleZip);
+ ZipArchiveOutputStream tos = new ZipArchiveOutputStream(os)) {
+ try {
+ZipArchiveEntry ze = new ZipArchiveEntry("foo");
+ze.setUnixMode(0555);
+byte[] data = "some-content".getBytes("UTF-8");
+ze.setSize(data.length);
+tos.putArchiveEntry(ze);
+tos.write(data);
+tos.closeArchiveEntry();
+tos.flush();
+tos.finish();
+ } finally {
+tos.close();
Review comment:
this close will be automatic. if you make the reference on L710
```
ZipArchiveOutputStream tos = new ZipArchiveOutputStream(new
FileOutputStream(simpleZip))
```
then everything will be lined up for automatic close
##
File path:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
##
@@ -706,35 +706,40 @@ public void testCreateLocalTempFile() throws IOException {
public void testUnZip() throws IOException {
// make sa simple zip
final File simpleZip = new File(del, FILE);
-OutputStream os = new FileOutputStream(simpleZip);
-ZipOutputStream tos = new ZipOutputStream(os);
-try {
- ZipEntry ze = new ZipEntry("foo");
- byte[] data = "some-content".getBytes("UTF-8");
- ze.setSize(data.length);
- tos.putNextEntry(ze);
- tos.write(data);
- tos.closeEntry();
- tos.flush();
- tos.finish();
-} finally {
- tos.close();
-}
-
-// successfully unzip it into an existing dir:
-FileUtil.unZip(simpleZip, tmp);
-// check result:
-assertTrue(new File(tmp, "foo").exists());
-assertEquals(12, new File(tmp, "foo").length());
-
-final File regularFile = new File(tmp, "QuickBrownFoxJumpsOverTheLazyDog");
-regularFile.createNewFile();
-assertTrue(regularFile.exists());
-try {
- FileUtil.unZip(simpleZip, regularFile);
- assertTrue("An IOException expected.", false);
-} catch (IOException ioe) {
- // okay
+try (OutputStream os = new FileOutputStream(simpleZip);
+ ZipArchiveOutputStream tos = new ZipArchiveOutputStream(os)) {
+ try {
+ZipArchiveEntry ze = new ZipArchiveEntry("foo");
+ze.setUnixMode(0555);
+byte[] data = "some-content".getBytes("UTF-8");
+ze.setSize(data.length);
+tos.putArchiveEntry(ze);
+tos.write(data);
+tos.closeArchiveEntry();
+tos.flush();
+tos.finish();
+ } finally {
+tos.close();
+ }
+
+ // successfully unzip it into an existing dir:
+ FileUtil.unZip(simpleZip, tmp);
+ // check result:
+ assertTrue(new File(tmp, "foo").exists());
+ assertEquals(12, new File(tmp, "foo").length());
+ assertTrue("file lacks execute permissions", new File(tmp,
"foo").canExecute());
+ assertFalse("file has write permissions", new File(tmp,
"foo").canWrite());
+ assertTrue("file lacks read permissions", new File(tmp,
"foo").canRead());
+
+ final File regularFile = new File(tmp,
"QuickBrownFoxJumpsOverTheLazyDog");
+
[GitHub] [hadoop] steveloughran commented on a change in pull request #4036: HADOOP-18145.Decompress the ZIP file and retain the original file per…
steveloughran commented on a change in pull request #4036:
URL: https://github.com/apache/hadoop/pull/4036#discussion_r832391092
##
File path:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
##
@@ -706,35 +706,40 @@ public void testCreateLocalTempFile() throws IOException {
public void testUnZip() throws IOException {
// make sa simple zip
final File simpleZip = new File(del, FILE);
-OutputStream os = new FileOutputStream(simpleZip);
-ZipOutputStream tos = new ZipOutputStream(os);
-try {
- ZipEntry ze = new ZipEntry("foo");
- byte[] data = "some-content".getBytes("UTF-8");
- ze.setSize(data.length);
- tos.putNextEntry(ze);
- tos.write(data);
- tos.closeEntry();
- tos.flush();
- tos.finish();
-} finally {
- tos.close();
-}
-
-// successfully unzip it into an existing dir:
-FileUtil.unZip(simpleZip, tmp);
-// check result:
-assertTrue(new File(tmp, "foo").exists());
-assertEquals(12, new File(tmp, "foo").length());
-
-final File regularFile = new File(tmp, "QuickBrownFoxJumpsOverTheLazyDog");
-regularFile.createNewFile();
-assertTrue(regularFile.exists());
-try {
- FileUtil.unZip(simpleZip, regularFile);
- assertTrue("An IOException expected.", false);
-} catch (IOException ioe) {
- // okay
+try (OutputStream os = new FileOutputStream(simpleZip);
+ ZipArchiveOutputStream tos = new ZipArchiveOutputStream(os)) {
+ try {
+ZipArchiveEntry ze = new ZipArchiveEntry("foo");
+ze.setUnixMode(0555);
+byte[] data = "some-content".getBytes("UTF-8");
+ze.setSize(data.length);
+tos.putArchiveEntry(ze);
+tos.write(data);
+tos.closeArchiveEntry();
+tos.flush();
+tos.finish();
+ } finally {
+tos.close();
Review comment:
this close will be automatic. if you make the reference on L710
```
ZipArchiveOutputStream tos = new ZipArchiveOutputStream(new
FileOutputStream(simpleZip))
```
then everything will be lined up for automatic close
##
File path:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
##
@@ -706,35 +706,40 @@ public void testCreateLocalTempFile() throws IOException {
public void testUnZip() throws IOException {
// make sa simple zip
final File simpleZip = new File(del, FILE);
-OutputStream os = new FileOutputStream(simpleZip);
-ZipOutputStream tos = new ZipOutputStream(os);
-try {
- ZipEntry ze = new ZipEntry("foo");
- byte[] data = "some-content".getBytes("UTF-8");
- ze.setSize(data.length);
- tos.putNextEntry(ze);
- tos.write(data);
- tos.closeEntry();
- tos.flush();
- tos.finish();
-} finally {
- tos.close();
-}
-
-// successfully unzip it into an existing dir:
-FileUtil.unZip(simpleZip, tmp);
-// check result:
-assertTrue(new File(tmp, "foo").exists());
-assertEquals(12, new File(tmp, "foo").length());
-
-final File regularFile = new File(tmp, "QuickBrownFoxJumpsOverTheLazyDog");
-regularFile.createNewFile();
-assertTrue(regularFile.exists());
-try {
- FileUtil.unZip(simpleZip, regularFile);
- assertTrue("An IOException expected.", false);
-} catch (IOException ioe) {
- // okay
+try (OutputStream os = new FileOutputStream(simpleZip);
+ ZipArchiveOutputStream tos = new ZipArchiveOutputStream(os)) {
+ try {
+ZipArchiveEntry ze = new ZipArchiveEntry("foo");
+ze.setUnixMode(0555);
+byte[] data = "some-content".getBytes("UTF-8");
+ze.setSize(data.length);
+tos.putArchiveEntry(ze);
+tos.write(data);
+tos.closeArchiveEntry();
+tos.flush();
+tos.finish();
+ } finally {
+tos.close();
+ }
+
+ // successfully unzip it into an existing dir:
+ FileUtil.unZip(simpleZip, tmp);
+ // check result:
+ assertTrue(new File(tmp, "foo").exists());
+ assertEquals(12, new File(tmp, "foo").length());
+ assertTrue("file lacks execute permissions", new File(tmp,
"foo").canExecute());
+ assertFalse("file has write permissions", new File(tmp,
"foo").canWrite());
+ assertTrue("file lacks read permissions", new File(tmp,
"foo").canRead());
+
+ final File regularFile = new File(tmp,
"QuickBrownFoxJumpsOverTheLazyDog");
+ regularFile.createNewFile();
+ assertTrue(regularFile.exists());
+ try {
Review comment:
use LambdaTestUtils.intercept() here. e.g.
```java
LambdaTestUtils.intercept(IOException.class, () ->
FileUtil.unZip(simpleZip, regularFile));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
[jira] [Work logged] (HADOOP-18145) Fileutil's unzip method causes unzipped files to lose their original permissions
[
https://issues.apache.org/jira/browse/HADOOP-18145?focusedWorklogId=745930=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745930
]
ASF GitHub Bot logged work on HADOOP-18145:
---
Author: ASF GitHub Bot
Created on: 22/Mar/22 16:43
Start Date: 22/Mar/22 16:43
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #4036:
URL: https://github.com/apache/hadoop/pull/4036#issuecomment-1055349079
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 12s | | trunk passed |
| +1 :green_heart: | compile | 24m 17s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 43s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 59s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 38s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 39s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 6s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 25m 27s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 59s | | the patch passed |
| +1 :green_heart: | compile | 23m 38s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 23m 38s | | the patch passed |
| +1 :green_heart: | compile | 20m 47s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 47s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 0s |
[/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4036/8/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt)
| hadoop-common-project/hadoop-common: The patch generated 3 new + 75
unchanged - 0 fixed = 78 total (was 75) |
| +1 :green_heart: | mvnsite | 1m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 3s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 38s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 42s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 31s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 43s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not
generate ASF License warnings. |
| | | 211m 3s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4036/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4036 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux dc3cb049a2b0 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18
11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7732357fbfae4404d331d08b9b790d4d752e4b86 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
|
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #4036: HADOOP-18145.Decompress the ZIP file and retain the original file per…
hadoop-yetus removed a comment on pull request #4036:
URL: https://github.com/apache/hadoop/pull/4036#issuecomment-1055349079
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 12s | | trunk passed |
| +1 :green_heart: | compile | 24m 17s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 43s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 59s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 38s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 39s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 6s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 25m 27s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 59s | | the patch passed |
| +1 :green_heart: | compile | 23m 38s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 23m 38s | | the patch passed |
| +1 :green_heart: | compile | 20m 47s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 47s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 0s |
[/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4036/8/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt)
| hadoop-common-project/hadoop-common: The patch generated 3 new + 75
unchanged - 0 fixed = 78 total (was 75) |
| +1 :green_heart: | mvnsite | 1m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 3s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 38s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 42s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 31s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 43s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not
generate ASF License warnings. |
| | | 211m 3s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4036/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4036 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux dc3cb049a2b0 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18
11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7732357fbfae4404d331d08b9b790d4d752e4b86 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4036/8/testReport/ |
| Max. process+thread count | 1252 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4036/8/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by |
[jira] [Comment Edited] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing -bucket purged?
[
https://issues.apache.org/jira/browse/HADOOP-18168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510511#comment-17510511
]
Daniel Carl Jones edited comment on HADOOP-18168 at 3/22/22, 4:42 PM:
--
Current content of the landsat-pds bucket (2022-03-22 14:00).
{noformat}
❯ aws s3 ls s3://landsat-pds/
2016-08-19 18:12:04 105 robots.txt
2018-08-29 01:45:15 45603307 scene_list.gz{noformat}
Looks like _scene_list.gz_ is retained. However, ITestMarkerTool is relying on
assumptions about the bucket's contents (large number of objects).
was (Author: JIRAUSER284792):
Current content of the landsat-pds bucket (2022-03-22 14:00).
{noformat}
❯ aws s3 ls s3://landsat-pds/
2016-08-19 18:12:04 105 robots.txt
2018-08-29 01:45:15 45603307 scene_list.gz{noformat}
> ITestMarkerTool.testRunLimitedLandsatAudit failing -bucket purged?
> --
>
> Key: HADOOP-18168
> URL: https://issues.apache.org/jira/browse/HADOOP-18168
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.3
>Reporter: Steve Loughran
>Priority: Minor
>
> {{ITestMarkerTool.testRunLimitedLandsatAudit}} is failing -a scan which was
> meant to stop after the first page of results is finishing because there
> aren't so many objects there.
>
> first visible sign of the landsat-pds cleanup
> now we have requester pays, we could do this against another store with
> stability promises, e.g common crawl.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-15566) Support OpenTelemetry
[ https://issues.apache.org/jira/browse/HADOOP-15566?focusedWorklogId=745914=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745914 ] ASF GitHub Bot logged work on HADOOP-15566: --- Author: ASF GitHub Bot Created on: 22/Mar/22 16:26 Start Date: 22/Mar/22 16:26 Worklog Time Spent: 10m Work Description: kiran-maturi commented on a change in pull request #3445: URL: https://github.com/apache/hadoop/pull/3445#discussion_r832373680 ## File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/tracing/TestTracer.java ## @@ -0,0 +1,33 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.tracing; + +import org.junit.Test; + +import static org.junit.Assert.*; Review comment: @steveloughran sure will start adding more tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745914) Time Spent: 10h 40m (was: 10.5h) > Support OpenTelemetry > - > > Key: HADOOP-15566 > URL: https://issues.apache.org/jira/browse/HADOOP-15566 > Project: Hadoop Common > Issue Type: New Feature > Components: metrics, tracing >Affects Versions: 3.1.0 >Reporter: Todd Lipcon >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available, security > Attachments: HADOOP-15566-WIP.1.patch, HADOOP-15566.000.WIP.patch, > OpenTelemetry Support Scope Doc v2.pdf, OpenTracing Support Scope Doc.pdf, > Screen Shot 2018-06-29 at 11.59.16 AM.png, ss-trace-s3a.png > > Time Spent: 10h 40m > Remaining Estimate: 0h > > The HTrace incubator project has voted to retire itself and won't be making > further releases. The Hadoop project currently has various hooks with HTrace. > It seems in some cases (eg HDFS-13702) these hooks have had measurable > performance overhead. Given these two factors, I think we should consider > removing the HTrace integration. If there is someone willing to do the work, > replacing it with OpenTracing might be a better choice since there is an > active community. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] kiran-maturi commented on a change in pull request #3445: HADOOP-15566 Opentelemetry changes using java agent
kiran-maturi commented on a change in pull request #3445: URL: https://github.com/apache/hadoop/pull/3445#discussion_r832373680 ## File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/tracing/TestTracer.java ## @@ -0,0 +1,33 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.tracing; + +import org.junit.Test; + +import static org.junit.Assert.*; Review comment: @steveloughran sure will start adding more tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing -bucket purged?
[
https://issues.apache.org/jira/browse/HADOOP-18168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510511#comment-17510511
]
Daniel Carl Jones edited comment on HADOOP-18168 at 3/22/22, 4:17 PM:
--
Current content of the landsat-pds bucket (2022-03-22 14:00).
{noformat}
❯ aws s3 ls s3://landsat-pds/
2016-08-19 18:12:04 105 robots.txt
2018-08-29 01:45:15 45603307 scene_list.gz{noformat}
was (Author: JIRAUSER284792):
Bucket definitely looks empty right now (2022-03-22 14:00).
{noformat}
❯ aws s3 ls s3://landsat-pds/
2016-08-19 18:12:04 105 robots.txt
2018-08-29 01:45:15 45603307 scene_list.gz{noformat}
> ITestMarkerTool.testRunLimitedLandsatAudit failing -bucket purged?
> --
>
> Key: HADOOP-18168
> URL: https://issues.apache.org/jira/browse/HADOOP-18168
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.3
>Reporter: Steve Loughran
>Priority: Minor
>
> {{ITestMarkerTool.testRunLimitedLandsatAudit}} is failing -a scan which was
> meant to stop after the first page of results is finishing because there
> aren't so many objects there.
>
> first visible sign of the landsat-pds cleanup
> now we have requester pays, we could do this against another store with
> stability promises, e.g common crawl.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] rguillome commented on pull request #426: correct configuration tag in mapred-site.xml
rguillome commented on pull request #426: URL: https://github.com/apache/hadoop/pull/426#issuecomment-1075277247 Corrected in 3.1.2 (https://issues.apache.org/jira/browse/MAPREDUCE-7165) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] rguillome closed pull request #426: correct configuration tag in mapred-site.xml
rguillome closed pull request #426: URL: https://github.com/apache/hadoop/pull/426 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao closed pull request #2918: HDFS-15987. Improve oiv tool to parse fsimage file in parallel with d…
Hexiaoqiao closed pull request #2918: URL: https://github.com/apache/hadoop/pull/2918 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #2918: HDFS-15987. Improve oiv tool to parse fsimage file in parallel with d…
Hexiaoqiao commented on pull request #2918: URL: https://github.com/apache/hadoop/pull/2918#issuecomment-1075257224 Committed to trunk. Thanks @whbing for your contributions! Thanks all to reviews! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on a change in pull request #4085: HDFS-16511. Change some frequent method lock type in ReplicaMap.
Hexiaoqiao commented on a change in pull request #4085:
URL: https://github.com/apache/hadoop/pull/4085#discussion_r832238320
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
##
@@ -602,6 +605,54 @@ public void run() {}
+ "volumeMap.", 0, totalNumReplicas);
}
+ @Test(timeout = 3)
+ public void testCurrentWriteAndDeleteBlock() throws Exception {
Review comment:
testCurrentXXX -> testConcurrentXXX ?
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
##
@@ -602,6 +605,54 @@ public void run() {}
+ "volumeMap.", 0, totalNumReplicas);
}
+ @Test(timeout = 3)
+ public void testCurrentWriteAndDeleteBlock() throws Exception {
+// Feed FsDataset with block metadata.
+final int numBlocks = 1000;
+final int threadCount = 10;
+// Generate data blocks.
+ExecutorService pool = Executors.newFixedThreadPool(threadCount);
+List> futureList = new ArrayList<>();
+for (int i = 0; i < threadCount; i++) {
+ Thread thread = new Thread() {
+@Override
+public void run() {
+ try {
+for (int i = 0; i < numBlocks; i++) {
+ String bpid = BLOCK_POOL_IDS[numBlocks % BLOCK_POOL_IDS.length];
Review comment:
The `numBlocks` and `BLOCK_POOL_IDS.length` are both static, so bpid is
one certain value. Do you mean to random it?
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
##
@@ -602,6 +605,54 @@ public void run() {}
+ "volumeMap.", 0, totalNumReplicas);
}
+ @Test(timeout = 3)
+ public void testCurrentWriteAndDeleteBlock() throws Exception {
+// Feed FsDataset with block metadata.
+final int numBlocks = 1000;
+final int threadCount = 10;
+// Generate data blocks.
+ExecutorService pool = Executors.newFixedThreadPool(threadCount);
+List> futureList = new ArrayList<>();
+for (int i = 0; i < threadCount; i++) {
+ Thread thread = new Thread() {
+@Override
+public void run() {
+ try {
+for (int i = 0; i < numBlocks; i++) {
+ String bpid = BLOCK_POOL_IDS[numBlocks % BLOCK_POOL_IDS.length];
+ ExtendedBlock eb = new ExtendedBlock(bpid, i);
+ ReplicaHandler replica = null;
+ try {
+replica = dataset.createRbw(StorageType.DEFAULT, null, eb,
+false);
+if (i % 2 > 0) {
+ dataset.invalidate(bpid, new Block[]{eb.getLocalBlock()});
+}
+ } finally {
+if (replica != null) {
+ replica.close();
+}
+ }
+}
+ } catch (Exception e) {
+e.printStackTrace();
+ }
+}
+ };
+ thread.setName("AddBlock" + i);
+ futureList.add(pool.submit(thread));
+}
+// Wait for data generation
+for (Future f : futureList) {
+ f.get();
+}
+int totalNumReplicas = 0;
Review comment:
Suggest to verify different blockpool's blocks num separately here. Sum
them will loss some key information such as above comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18145) Fileutil's unzip method causes unzipped files to lose their original permissions
[
https://issues.apache.org/jira/browse/HADOOP-18145?focusedWorklogId=745847=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745847
]
ASF GitHub Bot logged work on HADOOP-18145:
---
Author: ASF GitHub Bot
Created on: 22/Mar/22 14:12
Start Date: 22/Mar/22 14:12
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#4036:
URL: https://github.com/apache/hadoop/pull/4036#discussion_r832236126
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
##
@@ -667,6 +670,9 @@ public static void unZip(InputStream inputStream, File
toDir)
if (!file.setLastModified(entry.getTime())) {
numOfFailedLastModifiedSet++;
}
+ if (entry.getPlatform() == ZipArchiveEntry.PLATFORM_UNIX) {
+Files.setPosixFilePermissions(file.toPath(),
permissionsFromMode(entry.getUnixMode()));
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 745847)
Time Spent: 5.5h (was: 5h 20m)
> Fileutil's unzip method causes unzipped files to lose their original
> permissions
>
>
> Key: HADOOP-18145
> URL: https://issues.apache.org/jira/browse/HADOOP-18145
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.7.2
>Reporter: jingxiong zhong
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> When Spark decompresses the zip file, if the original file has the executable
> permission, but the unzip method of FileUtil is invoked, the decompressed
> file loses the executable permission, we should save the original permission
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #4036: HADOOP-18145.Decompress the ZIP file and retain the original file per…
steveloughran commented on a change in pull request #4036:
URL: https://github.com/apache/hadoop/pull/4036#discussion_r832236126
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
##
@@ -667,6 +670,9 @@ public static void unZip(InputStream inputStream, File
toDir)
if (!file.setLastModified(entry.getTime())) {
numOfFailedLastModifiedSet++;
}
+ if (entry.getPlatform() == ZipArchiveEntry.PLATFORM_UNIX) {
+Files.setPosixFilePermissions(file.toPath(),
permissionsFromMode(entry.getUnixMode()));
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing -bucket purged?
[
https://issues.apache.org/jira/browse/HADOOP-18168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510511#comment-17510511
]
Daniel Carl Jones commented on HADOOP-18168:
Bucket definitely looks empty right now (2022-03-22 14:00).
{noformat}
❯ aws s3 ls s3://landsat-pds/
2016-08-19 18:12:04 105 robots.txt
2018-08-29 01:45:15 45603307 scene_list.gz{noformat}
> ITestMarkerTool.testRunLimitedLandsatAudit failing -bucket purged?
> --
>
> Key: HADOOP-18168
> URL: https://issues.apache.org/jira/browse/HADOOP-18168
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.3
>Reporter: Steve Loughran
>Priority: Minor
>
> {{ITestMarkerTool.testRunLimitedLandsatAudit}} is failing -a scan which was
> meant to stop after the first page of results is finishing because there
> aren't so many objects there.
>
> first visible sign of the landsat-pds cleanup
> now we have requester pays, we could do this against another store with
> stability promises, e.g common crawl.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14837) Handle S3A "glacier" data
[
https://issues.apache.org/jira/browse/HADOOP-14837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510508#comment-17510508
]
Steve Loughran commented on HADOOP-14837:
-
good questions, -I have no idea what the right answers are
bq. For reporting better, do we want to add in a new statistic, something like
`objects_in_glacier` which will have the count of objects currently in glacier?
why not?
bq. In listings, we can add in a new option to filter out glacier files by
doing something like `!summary.getStorageClass().equals("GLACIER")` in the
acceptor here? After we do this and call `getContentSummary()` it won't return
glacier files in the fileCount.
I'm not worried about that. is the storage type returned in the list call.
allowing it to be filtered there? i wouldn't want to do any HEAD requests here
bq. getBlockLocations()
there's special handling in spark for that location, which says "run your work
anywnere". we doin't want to break that.
I think the best tactic here is to work out what people I want to do here and
provide the bare minimum. Looking at some of the JIRAs there's no consensus as
to what people want. Do they want glaciated files to be skipped in queries? or
for recovery to be triggered (somehow). Returning the storage type ARCHIVE
would be enough for anyone who wants to identify these files (distcp?) and at
least then know there's a cost in accessing them.
> Handle S3A "glacier" data
> -
>
> Key: HADOOP-14837
> URL: https://issues.apache.org/jira/browse/HADOOP-14837
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Priority: Minor
>
> SPARK-21797 covers how if you have AWS S3 set to copy some files to glacier,
> they appear in the listing but GETs fail, and so does everything else
> We should think about how best to handle this.
> # report better
> # if listings can identify files which are glaciated then maybe we could have
> an option to filter them out
> # test & see what happens
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13704) S3A getContentSummary() to move to listFiles(recursive) to count children; instrument use
[
https://issues.apache.org/jira/browse/HADOOP-13704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-13704.
-
Fix Version/s: 3.3.3
Resolution: Fixed
fixed in 3.3.3. thanks!
> S3A getContentSummary() to move to listFiles(recursive) to count children;
> instrument use
> -
>
> Key: HADOOP-13704
> URL: https://issues.apache.org/jira/browse/HADOOP-13704
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Ahmar Suhail
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.3.3
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Hive and a bit of Spark use {{getContentSummary()}} to get some summary stats
> of a filesystem. This is very expensive on S3A (and any other object store),
> especially as the base implementation does the recursive tree walk.
> Because of HADOOP-13208, we have a full enumeration of files under a path
> without directory costs...S3A can/should switch to this to speed up those
> places where the operation is called.
> Also
> * API call needs FS spec and contract tests
> * S3A could instrument invocation, so as to enable real-world popularity to
> be measured
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13704) S3A getContentSummary() to move to listFiles(recursive) to count children; instrument use
[
https://issues.apache.org/jira/browse/HADOOP-13704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran reassigned HADOOP-13704:
---
Assignee: Ahmar Suhail
> S3A getContentSummary() to move to listFiles(recursive) to count children;
> instrument use
> -
>
> Key: HADOOP-13704
> URL: https://issues.apache.org/jira/browse/HADOOP-13704
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Ahmar Suhail
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Hive and a bit of Spark use {{getContentSummary()}} to get some summary stats
> of a filesystem. This is very expensive on S3A (and any other object store),
> especially as the base implementation does the recursive tree walk.
> Because of HADOOP-13208, we have a full enumeration of files under a path
> without directory costs...S3A can/should switch to this to speed up those
> places where the operation is called.
> Also
> * API call needs FS spec and contract tests
> * S3A could instrument invocation, so as to enable real-world popularity to
> be measured
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13704) S3A getContentSummary() to move to listFiles(recursive) to count children; instrument use
[
https://issues.apache.org/jira/browse/HADOOP-13704?focusedWorklogId=745833=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745833
]
ASF GitHub Bot logged work on HADOOP-13704:
---
Author: ASF GitHub Bot
Created on: 22/Mar/22 13:47
Start Date: 22/Mar/22 13:47
Worklog Time Spent: 10m
Work Description: steveloughran closed pull request #3978:
URL: https://github.com/apache/hadoop/pull/3978
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 745833)
Time Spent: 2h 50m (was: 2h 40m)
> S3A getContentSummary() to move to listFiles(recursive) to count children;
> instrument use
> -
>
> Key: HADOOP-13704
> URL: https://issues.apache.org/jira/browse/HADOOP-13704
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Hive and a bit of Spark use {{getContentSummary()}} to get some summary stats
> of a filesystem. This is very expensive on S3A (and any other object store),
> especially as the base implementation does the recursive tree walk.
> Because of HADOOP-13208, we have a full enumeration of files under a path
> without directory costs...S3A can/should switch to this to speed up those
> places where the operation is called.
> Also
> * API call needs FS spec and contract tests
> * S3A could instrument invocation, so as to enable real-world popularity to
> be measured
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13704) S3A getContentSummary() to move to listFiles(recursive) to count children; instrument use
[
https://issues.apache.org/jira/browse/HADOOP-13704?focusedWorklogId=745834=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745834
]
ASF GitHub Bot logged work on HADOOP-13704:
---
Author: ASF GitHub Bot
Created on: 22/Mar/22 13:47
Start Date: 22/Mar/22 13:47
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #3978:
URL: https://github.com/apache/hadoop/pull/3978#issuecomment-1046907425
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 11s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 55s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 43s | | trunk passed |
| +1 :green_heart: | compile | 22m 28s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 19m 46s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 43s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 57s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 33s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 10s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 35s | | the patch passed |
| +1 :green_heart: | compile | 21m 53s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 21m 53s | | the patch passed |
| +1 :green_heart: | compile | 19m 51s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 19m 51s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 32s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 40s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 51s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 34s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 22s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 32m 7s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 46s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not
generate ASF License warnings. |
| | | 233m 23s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3978 |
| Optional Tests | dupname asflicense mvnsite codespell markdownlint compile
javac javadoc mvninstall unit shadedclient spotbugs checkstyle |
| uname | Linux a16af9ec9b1c 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 945a9d9bfbe07a11e49cb325daaeb3bf05eb88ef |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/5/testReport/ |
| Max. process+thread count | 1664
[jira] [Work logged] (HADOOP-13704) S3A getContentSummary() to move to listFiles(recursive) to count children; instrument use
[
https://issues.apache.org/jira/browse/HADOOP-13704?focusedWorklogId=745832=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745832
]
ASF GitHub Bot logged work on HADOOP-13704:
---
Author: ASF GitHub Bot
Created on: 22/Mar/22 13:47
Start Date: 22/Mar/22 13:47
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #3978:
URL: https://github.com/apache/hadoop/pull/3978#issuecomment-1075201852
patch is in, though i realise i forgot to add the pr# to the header, which
is automatic when done through the UI. never mind, at least the JIRA Is in.
closing this PR as done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 745832)
Time Spent: 2h 40m (was: 2.5h)
> S3A getContentSummary() to move to listFiles(recursive) to count children;
> instrument use
> -
>
> Key: HADOOP-13704
> URL: https://issues.apache.org/jira/browse/HADOOP-13704
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Hive and a bit of Spark use {{getContentSummary()}} to get some summary stats
> of a filesystem. This is very expensive on S3A (and any other object store),
> especially as the base implementation does the recursive tree walk.
> Because of HADOOP-13208, we have a full enumeration of files under a path
> without directory costs...S3A can/should switch to this to speed up those
> places where the operation is called.
> Also
> * API call needs FS spec and contract tests
> * S3A could instrument invocation, so as to enable real-world popularity to
> be measured
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3978: HADOOP-13704. Optimised getContentSummary()
hadoop-yetus removed a comment on pull request #3978:
URL: https://github.com/apache/hadoop/pull/3978#issuecomment-1046907425
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 11s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 55s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 43s | | trunk passed |
| +1 :green_heart: | compile | 22m 28s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 19m 46s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 43s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 57s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 33s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 10s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 35s | | the patch passed |
| +1 :green_heart: | compile | 21m 53s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 21m 53s | | the patch passed |
| +1 :green_heart: | compile | 19m 51s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 19m 51s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 32s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 40s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 51s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 34s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 22s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 32m 7s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 46s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not
generate ASF License warnings. |
| | | 233m 23s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3978 |
| Optional Tests | dupname asflicense mvnsite codespell markdownlint compile
javac javadoc mvninstall unit shadedclient spotbugs checkstyle |
| uname | Linux a16af9ec9b1c 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 945a9d9bfbe07a11e49cb325daaeb3bf05eb88ef |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/5/testReport/ |
| Max. process+thread count | 1664 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws
U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated
[GitHub] [hadoop] steveloughran closed pull request #3978: HADOOP-13704. Optimised getContentSummary()
steveloughran closed pull request #3978: URL: https://github.com/apache/hadoop/pull/3978 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3978: HADOOP-13704. Optimised getContentSummary()
steveloughran commented on pull request #3978: URL: https://github.com/apache/hadoop/pull/3978#issuecomment-1075201852 patch is in, though i realise i forgot to add the pr# to the header, which is automatic when done through the UI. never mind, at least the JIRA Is in. closing this PR as done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing -bucket purged?
[
https://issues.apache.org/jira/browse/HADOOP-18168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510496#comment-17510496
]
Steve Loughran commented on HADOOP-18168:
-
{code}
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AConfiguration
[ERROR] Tests run: 18, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 50.261
s <<< FAILURE! - in org.apache.hadoop.fs.s3a.tools.ITestMarkerTool
[ERROR]
testRunLimitedLandsatAudit(org.apache.hadoop.fs.s3a.tools.ITestMarkerTool)
Time elapsed: 2.9 s <<< FAILURE!
java.lang.AssertionError: Expected an exception of type class
org.apache.hadoop.util.ExitUtil$ExitException
at
org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:409)
at
org.apache.hadoop.fs.s3a.s3guard.S3GuardToolTestHelper.runS3GuardCommandToFailure(S3GuardToolTestHelper.java:163)
at
org.apache.hadoop.fs.s3a.tools.AbstractMarkerToolTest.runToFailure(AbstractMarkerToolTest.java:271)
at
org.apache.hadoop.fs.s3a.tools.ITestMarkerTool.testRunLimitedLandsatAudit(ITestMarkerTool.java:320)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
{code}
> ITestMarkerTool.testRunLimitedLandsatAudit failing -bucket purged?
> --
>
> Key: HADOOP-18168
> URL: https://issues.apache.org/jira/browse/HADOOP-18168
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.3
>Reporter: Steve Loughran
>Priority: Minor
>
> {{ITestMarkerTool.testRunLimitedLandsatAudit}} is failing -a scan which was
> meant to stop after the first page of results is finishing because there
> aren't so many objects there.
>
> first visible sign of the landsat-pds cleanup
> now we have requester pays, we could do this against another store with
> stability promises, e.g common crawl.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18168) ITestMarkerTool.testRunLimitedLandsatAudit failing -bucket purged?
Steve Loughran created HADOOP-18168:
---
Summary: ITestMarkerTool.testRunLimitedLandsatAudit failing
-bucket purged?
Key: HADOOP-18168
URL: https://issues.apache.org/jira/browse/HADOOP-18168
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3, test
Affects Versions: 3.3.3
Reporter: Steve Loughran
{{ITestMarkerTool.testRunLimitedLandsatAudit}} is failing -a scan which was
meant to stop after the first page of results is finishing because there aren't
so many objects there.
first visible sign of the landsat-pds cleanup
now we have requester pays, we could do this against another store with
stability promises, e.g common crawl.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] singer-bin commented on pull request #4069: HDFS-16457.Make fs.getspaceused.classname reconfigurable
singer-bin commented on pull request #4069: URL: https://github.com/apache/hadoop/pull/4069#issuecomment-1075198028 Can you review my code, thanks a lot. @jojochuang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] singer-bin commented on pull request #4069: HDFS-16457.Make fs.getspaceused.classname reconfigurable
singer-bin commented on pull request #4069: URL: https://github.com/apache/hadoop/pull/4069#issuecomment-1075184346 The failed check has nothing to do with my changes, the last check passed. @jojochuang ,As can be seen from the generated log, my ut passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-13704) S3A getContentSummary() to move to listFiles(recursive) to count children; instrument use
[
https://issues.apache.org/jira/browse/HADOOP-13704?focusedWorklogId=745807=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745807
]
ASF GitHub Bot logged work on HADOOP-13704:
---
Author: ASF GitHub Bot
Created on: 22/Mar/22 12:55
Start Date: 22/Mar/22 12:55
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #3978:
URL: https://github.com/apache/hadoop/pull/3978#issuecomment-1044978246
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 745807)
Time Spent: 2.5h (was: 2h 20m)
> S3A getContentSummary() to move to listFiles(recursive) to count children;
> instrument use
> -
>
> Key: HADOOP-13704
> URL: https://issues.apache.org/jira/browse/HADOOP-13704
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Hive and a bit of Spark use {{getContentSummary()}} to get some summary stats
> of a filesystem. This is very expensive on S3A (and any other object store),
> especially as the base implementation does the recursive tree walk.
> Because of HADOOP-13208, we have a full enumeration of files under a path
> without directory costs...S3A can/should switch to this to speed up those
> places where the operation is called.
> Also
> * API call needs FS spec and contract tests
> * S3A could instrument invocation, so as to enable real-world popularity to
> be measured
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3978: HADOOP-13704. Optimised getContentSummary()
hadoop-yetus removed a comment on pull request #3978: URL: https://github.com/apache/hadoop/pull/3978#issuecomment-1044978246 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally
[
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=745805=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745805
]
ASF GitHub Bot logged work on HADOOP-16202:
---
Author: ASF GitHub Bot
Created on: 22/Mar/22 12:53
Start Date: 22/Mar/22 12:53
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-1075138032
checkstyle
```
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:120:import
org.apache.hadoop.fs.s3a.select.InternalSelectConstants;:8: Unused import -
org.apache.hadoop.fs.s3a.select.InternalSelectConstants. [UnusedImports]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:194:import
static
org.apache.hadoop.fs.impl.AbstractFSBuilderImpl.rejectUnknownMandatoryKeys;:15:
Unused import -
org.apache.hadoop.fs.impl.AbstractFSBuilderImpl.rejectUnknownMandatoryKeys.
[UnusedImports]'
```
javac warnings
```
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:4099:14:[deprecation]
getDefaultBlockSize() in FileSystem has been deprecated
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CopyFromLocalOperation.java:235:16:[unchecked]
unchecked method invocation: method sort in interface List is applied to given
types
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 745805)
Time Spent: 15h (was: 14h 50m)
> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 15h
> Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows
> the length of the file to be declared. If set, *no check for the existence of
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than
> only S3AFileStatus -and not check that the path matches the path being
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2584: HADOOP-16202. Enhance openFile() for better read performance against object stores
steveloughran commented on pull request #2584: URL: https://github.com/apache/hadoop/pull/2584#issuecomment-1075138032 checkstyle ``` ./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:120:import org.apache.hadoop.fs.s3a.select.InternalSelectConstants;:8: Unused import - org.apache.hadoop.fs.s3a.select.InternalSelectConstants. [UnusedImports] ./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:194:import static org.apache.hadoop.fs.impl.AbstractFSBuilderImpl.rejectUnknownMandatoryKeys;:15: Unused import - org.apache.hadoop.fs.impl.AbstractFSBuilderImpl.rejectUnknownMandatoryKeys. [UnusedImports]' ``` javac warnings ``` hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:4099:14:[deprecation] getDefaultBlockSize() in FileSystem has been deprecated hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CopyFromLocalOperation.java:235:16:[unchecked] unchecked method invocation: method sort in interface List is applied to given types ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-15566) Support OpenTelemetry
[ https://issues.apache.org/jira/browse/HADOOP-15566?focusedWorklogId=745801=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745801 ] ASF GitHub Bot logged work on HADOOP-15566: --- Author: ASF GitHub Bot Created on: 22/Mar/22 12:52 Start Date: 22/Mar/22 12:52 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #3445: URL: https://github.com/apache/hadoop/pull/3445#issuecomment-1075137411 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 24s | | https://github.com/apache/hadoop/pull/3445 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/3445 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3445/8/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745801) Time Spent: 10.5h (was: 10h 20m) > Support OpenTelemetry > - > > Key: HADOOP-15566 > URL: https://issues.apache.org/jira/browse/HADOOP-15566 > Project: Hadoop Common > Issue Type: New Feature > Components: metrics, tracing >Affects Versions: 3.1.0 >Reporter: Todd Lipcon >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available, security > Attachments: HADOOP-15566-WIP.1.patch, HADOOP-15566.000.WIP.patch, > OpenTelemetry Support Scope Doc v2.pdf, OpenTracing Support Scope Doc.pdf, > Screen Shot 2018-06-29 at 11.59.16 AM.png, ss-trace-s3a.png > > Time Spent: 10.5h > Remaining Estimate: 0h > > The HTrace incubator project has voted to retire itself and won't be making > further releases. The Hadoop project currently has various hooks with HTrace. > It seems in some cases (eg HDFS-13702) these hooks have had measurable > performance overhead. Given these two factors, I think we should consider > removing the HTrace integration. If there is someone willing to do the work, > replacing it with OpenTracing might be a better choice since there is an > active community. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally
[
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=745799=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745799
]
ASF GitHub Bot logged work on HADOOP-16202:
---
Author: ASF GitHub Bot
Created on: 22/Mar/22 12:52
Start Date: 22/Mar/22 12:52
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-1025800148
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 745799)
Time Spent: 14h 50m (was: 14h 40m)
> Stabilize openFile() and adopt internally
> -
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 14h 50m
> Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows
> the length of the file to be declared. If set, *no check for the existence of
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than
> only S3AFileStatus -and not check that the path matches the path being
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3445: HADOOP-15566 Opentelemetry changes using java agent
hadoop-yetus commented on pull request #3445: URL: https://github.com/apache/hadoop/pull/3445#issuecomment-1075137411 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 24s | | https://github.com/apache/hadoop/pull/3445 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/3445 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3445/8/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2584: HADOOP-16202. Enhance openFile() for better read performance against object stores
hadoop-yetus removed a comment on pull request #2584: URL: https://github.com/apache/hadoop/pull/2584#issuecomment-1025800148 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] 9uapaw commented on pull request #4066: YARN-11087. Introduce the config to control the refresh interval in R…
9uapaw commented on pull request #4066: URL: https://github.com/apache/hadoop/pull/4066#issuecomment-1075102697 Thanks for the PR @zuston. Committed to trunk. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] 9uapaw closed pull request #4066: YARN-11087. Introduce the config to control the refresh interval in R…
9uapaw closed pull request #4066: URL: https://github.com/apache/hadoop/pull/4066 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
