[GitHub] [hadoop] PrabhuJoseph commented on pull request #4507: MAPREDUCE-7201.Make Job History File Permissions configurable

[GitBox]

PrabhuJoseph commented on PR #4507:
URL: https://github.com/apache/hadoop/pull/4507#issuecomment-1168200530

   Thanks @ashutoshcipher for the patch. Could you please include a testcase in 
TestJobHistoryUtils.java.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] PrabhuJoseph merged pull request #4500: YARN-11197.Backport YARN-9608 - DecommissioningNodesWatcher should get lists of running applications on node from RMNode.

[GitBox]

PrabhuJoseph merged PR #4500:
URL: https://github.com/apache/hadoop/pull/4500


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] PrabhuJoseph merged pull request #4492: YARN-9822.TimelineCollectorWebService#putEntities blocked when ATSV2 HBase is down

[GitBox]

PrabhuJoseph merged PR #4492:
URL: https://github.com/apache/hadoop/pull/4492


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] iwasakims commented on a diff in pull request #4506: YARN-11199. Replace htrace-core with hbase-noop-htrace

[GitBox]

iwasakims commented on code in PR #4506:
URL: https://github.com/apache/hadoop/pull/4506#discussion_r907956267


##
hadoop-project/pom.xml:
##
@@ -1687,8 +1699,19 @@
 jdk.tools
 jdk.tools
   
+  
+  
+org.apache.htrace
+htrace-core
+  
 
   
+  
+  
+org.apache.hbase.thirdparty
+hbase-noop-htrace
+4.1.1

Review Comment:
   This looks based on HTrace 4 which is completely incompatible with HTrace 3. 
(like org.apache.htrace vs. org.apache.htrace.core). Did you see no issue on 
running timelineservice with this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] hadoop-yetus commented on pull request #4507: MAPREDUCE-7201.Make Job History File Permissions configurable

[GitBox]

hadoop-yetus commented on PR #4507:
URL: https://github.com/apache/hadoop/pull/4507#issuecomment-1168071884

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 43s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include 
any new or modified tests. Please justify why no new tests are needed for this 
patch. Also please list what manual steps were performed to verify this patch.  
|
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  15m  3s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  25m 22s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   2m 50s |  |  trunk passed with JDK 
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1  |
   | +1 :green_heart: |  compile  |   2m 31s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   1m 26s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 53s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   1m 48s |  |  trunk passed with JDK 
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1  |
   | +1 :green_heart: |  javadoc  |   1m 35s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   2m 50s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  23m 26s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 31s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   1m 12s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   2m 33s |  |  the patch passed with JDK 
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1  |
   | +1 :green_heart: |  javac  |   2m 33s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   2m 12s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  javac  |   2m 13s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | +1 :green_heart: |  checkstyle  |   1m  2s |  |  
hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 0 new + 
310 unchanged - 1 fixed = 310 total (was 311)  |
   | +1 :green_heart: |  mvnsite  |   1m 17s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   1m  5s |  |  the patch passed with JDK 
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1  |
   | +1 :green_heart: |  javadoc  |   1m  4s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   2m 26s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  20m 12s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |   1m 28s |  |  hadoop-mapreduce-client-common 
in the patch passed.  |
   | +1 :green_heart: |  unit  |   8m 52s |  |  hadoop-mapreduce-client-app in 
the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 52s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 126m 24s |  |  |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4507/1/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/4507 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
   | uname | Linux 9f5bab14dc92 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 223c3040048e24a828cb9b2fd50b6e291cdc7d93 |
   | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private 
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 
/usr/lib/jvm/java-8-openjdk-amd64:Private 
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
   |  Test Results | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4507/1/testReport/ |
   | Max. process+thread count | 707 (vs. ulimit of 5500) |
   | modules | C: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common 

[GitHub] [hadoop] umamaheswararao merged pull request #4474: HADOOP-18306: Warnings should not be shown on cli console when linux user not present on client

[GitBox]

umamaheswararao merged PR #4474:
URL: https://github.com/apache/hadoop/pull/4474


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] umamaheswararao commented on pull request #4474: HADOOP-18306: Warnings should not be shown on cli console when linux user not present on client

[GitBox]

umamaheswararao commented on PR #4474:
URL: https://github.com/apache/hadoop/pull/4474#issuecomment-1168069961

   The -1 is due to no tests added. 
   
-1 overall
   
   [2022-06-23T22:22:07.359Z] 
   
   [2022-06-23T22:22:07.359Z] | Vote |Subsystem |  Runtime   | Comment
   
   [2022-06-23T22:22:07.359Z] 

   
   [2022-06-23T22:22:07.360Z] |   0  |  reexec  |   0m 55s   | Docker 
mode activated. 
   
   [2022-06-23T22:22:07.360Z] 
+---
   
   [2022-06-23T22:22:07.360Z] |  |  || 
Prechecks 
   
   [2022-06-23T22:22:07.360Z] 
+---
   
   [2022-06-23T22:22:07.360Z] |  +1  | dupname  |   0m  0s   | No case 
conflicting files found. 
   
   [2022-06-23T22:22:07.360Z] |   0  |   codespell  |   0m  0s   | 
codespell was not available. 
   
   [2022-06-23T22:22:07.360Z] |   0  |  detsecrets  |   0m  0s   | 
detect-secrets was not available. 
   
   [2022-06-23T22:22:07.360Z] |  +1  | @author  |   0m  0s   | The 
patch does not contain any @author 
   
   [2022-06-23T22:22:07.360Z] |  |  || tags.
   
   [2022-06-23T22:22:07.360Z] |  -1  |  test4tests  |   0m  0s   | The 
patch doesn't appear to include 
   
   [2022-06-23T22:22:07.360Z] |  |  || any new 
or modified tests. Please
   
   [2022-06-23T22:22:07.360Z] |  |  || justify 
why no new tests are needed for
   
   [2022-06-23T22:22:07.360Z] |  |  || this 
patch. Also please list what
   
   [2022-06-23T22:22:07.360Z] |  |  || manual 
steps were performed to verify
   
   [2022-06-23T22:22:07.360Z] |  |  || this 
patch.
   
   [2022-06-23T22:22:07.360Z] 
+---
   
   [2022-06-23T22:22:07.360Z] |  |  || trunk 
Compile Tests 
   
   [2022-06-23T22:22:07.360Z] 
+---
   
   [2022-06-23T22:22:07.360Z] |  +1  |  mvninstall  |  40m 24s   | trunk 
passed 
   
   [2022-06-23T22:22:07.360Z] |  +1  | compile  |  25m 10s   | trunk 
passed with JDK Private 
   
   [2022-06-23T22:22:07.360Z] |  |  || 
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
   
   [2022-06-23T22:22:07.360Z] |  |  || 
   
   [2022-06-23T22:22:07.360Z] |  +1  | compile  |  21m 37s   | trunk 
passed with JDK Private 
   
   [2022-06-23T22:22:07.360Z] |  |  || 
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04
   
   [2022-06-23T22:22:07.360Z] |  |  || -b07
   
   [2022-06-23T22:22:07.360Z] |  +1  |  checkstyle  |   1m 31s   | trunk 
passed 
   
   [2022-06-23T22:22:07.360Z] |  +1  | mvnsite  |   1m 58s   | trunk 
passed 
   
   [2022-06-23T22:22:07.360Z] |  +1  | javadoc  |   1m 31s   | trunk 
passed with JDK Private 
   
   [2022-06-23T22:22:07.360Z] |  |  || 
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
   
   [2022-06-23T22:22:07.360Z] |  |  || 
   
   [2022-06-23T22:22:07.360Z] |  +1  | javadoc  |   1m  4s   | trunk 
passed with JDK Private 
   
   [2022-06-23T22:22:07.360Z] |  |  || 
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04
   
   [2022-06-23T22:22:07.360Z] |  |  || -b07
   
   [2022-06-23T22:22:07.360Z] |  +1  |spotbugs  |   3m  6s   | trunk 
passed 
   
   [2022-06-23T22:22:07.360Z] |  +1  |shadedclient  |  25m 52s   | branch 
has no errors when building and 
   
   [2022-06-23T22:22:07.360Z] |  |  || testing 
our client artifacts.
   
   [2022-06-23T22:22:07.360Z] 
+---
   
   [2022-06-23T22:22:07.360Z] |  |  || Patch 
Compile Tests 
   
   [2022-06-23T22:22:07.360Z] 
+---
   
   [2022-06-23T22:22:07.360Z] |  +1  |  mvninstall  |   1m  6s   | the 
patch passed 
   
   [2022-06-23T22:22:07.360Z] |  +1  | compile  |  24m 13s   | the 
patch passed with JDK Private 
   
   [2022-06-23T22:22:07.360Z] |  |  || 
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
   
   [2022-06-23T22:22:07.360Z] |  |  || 
   
   [2022-06-23T22:22:07.360Z] |  +1  |   javac  |  24m 13s   | the 
patch passed 
   
   [2022-06-23T22:22:07.360Z] |  +1  | compile  |  21m 45s   | the 
patch 

[GitHub] [hadoop] snmvaughan closed pull request #4490: HADOOP-18311. Upgrade dependencies to address several CVEs

[GitBox]

snmvaughan closed pull request #4490: HADOOP-18311. Upgrade dependencies to 
address several CVEs
URL: https://github.com/apache/hadoop/pull/4490


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] snmvaughan commented on pull request #4490: HADOOP-18311. Upgrade dependencies to address several CVEs

[GitBox]

snmvaughan commented on PR #4490:
URL: https://github.com/apache/hadoop/pull/4490#issuecomment-1168069856

   I'm closing this pull request, and replacing it with smaller targeted ones.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] umamaheswararao commented on a diff in pull request #4474: HADOOP-18306: Warnings should not be shown on cli console when linux user not present on client

[GitBox]

umamaheswararao commented on code in PR #4474:
URL: https://github.com/apache/hadoop/pull/4474#discussion_r907909648


##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ShellBasedUnixGroupsMapping.java:
##
@@ -215,7 +215,13 @@ private Set getUnixGroups(String user) throws 
IOException {
   groups = resolvePartialGroupNames(user, e.getMessage(),
   executor.getOutput());
 } catch (PartialGroupNameException pge) {
-  LOG.warn("unable to return groups for user {}", user, pge);
+  if(LOG.isDebugEnabled()) {

Review Comment:
   I see this is stale version. Thanks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] umamaheswararao commented on a diff in pull request #4474: HADOOP-18306: Warnings should not be shown on cli console when linux user not present on client

[GitBox]

umamaheswararao commented on code in PR #4474:
URL: https://github.com/apache/hadoop/pull/4474#discussion_r907908052


##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ShellBasedUnixGroupsMapping.java:
##
@@ -215,7 +215,13 @@ private Set getUnixGroups(String user) throws 
IOException {
   groups = resolvePartialGroupNames(user, e.getMessage(),
   executor.getOutput());
 } catch (PartialGroupNameException pge) {
-  LOG.warn("unable to return groups for user {}", user, pge);
+  if(LOG.isDebugEnabled()) {

Review Comment:
   @swamirishi could you please fix the @smengcl comments above? you need to 
format line number 218



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] snmvaughan commented on pull request #4491: HADOOP-18311. Upgrade dependencies to address several CVEs

[GitBox]

snmvaughan commented on PR #4491:
URL: https://github.com/apache/hadoop/pull/4491#issuecomment-1168059725

   I'm closing this pull request, and replacing it with smaller targeted ones.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] snmvaughan closed pull request #4491: HADOOP-18311. Upgrade dependencies to address several CVEs

[GitBox]

snmvaughan closed pull request #4491: HADOOP-18311. Upgrade dependencies to 
address several CVEs
URL: https://github.com/apache/hadoop/pull/4491


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] hadoop-yetus commented on pull request #4505: HADOOP-18314.Add some description for PowerShellFencer.

[GitBox]

hadoop-yetus commented on PR #4505:
URL: https://github.com/apache/hadoop/pull/4505#issuecomment-1168007380

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 56s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  
|
   | +0 :ok: |  xmllint  |   0m  0s |  |  xmllint was not available.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include 
any new or modified tests. Please justify why no new tests are needed for this 
patch. Also please list what manual steps were performed to verify this patch.  
|
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  14m 55s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  25m 12s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  23m  2s |  |  trunk passed with JDK 
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1  |
   | +1 :green_heart: |  compile  |  20m 37s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   4m 25s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   4m 32s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   3m 34s |  |  trunk passed with JDK 
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1  |
   | +1 :green_heart: |  javadoc  |   3m 39s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   7m 13s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  24m 37s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 34s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   2m 33s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  22m 10s |  |  the patch passed with JDK 
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1  |
   | +1 :green_heart: |  javac  |  22m 10s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  20m 38s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  javac  |  20m 38s |  |  the patch passed  |
   | -1 :x: |  blanks  |   0m  0s | 
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4505/1/artifact/out/blanks-eol.txt)
 |  The patch has 5 line(s) that end in blanks. Use git apply --whitespace=fix 
<>. Refer https://git-scm.com/docs/git-apply  |
   | +1 :green_heart: |  checkstyle  |   4m 12s |  |  the patch passed  |
   | +1 :green_heart: |  mvnsite  |   4m 25s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   3m 26s |  |  the patch passed with JDK 
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1  |
   | +1 :green_heart: |  javadoc  |   3m 37s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   7m 21s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  24m 44s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |  18m 40s |  |  hadoop-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  | 384m 41s |  |  hadoop-hdfs in the patch 
passed.  |
   | +1 :green_heart: |  asflicense  |   2m  2s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 633m  5s |  |  |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4505/1/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/4505 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint 
markdownlint |
   | uname | Linux 88e009d83195 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / f8623a51b2def3497341dc6880c4d5056eb1373a |
   | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private 
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 
/usr/lib/jvm/java-8-openjdk-amd64:Private 
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
   |  Test Results | 

[GitHub] [hadoop] ashutoshcipher opened a new pull request, #4507: MAPREDUCE-7201.Make Job History File Permissions configurable

[GitBox]

ashutoshcipher opened a new pull request, #4507:
URL: https://github.com/apache/hadoop/pull/4507

   ### Description of PR
   
   Make Job History File Permissions configurable.
   
   JIRA: MAPREDUCE-7201
   
   
   ### For code changes:
   
   - [X] Does the title or this PR starts with the corresponding JIRA issue id 
(e.g. 'HADOOP-17799. Your PR title ...')?
   - [ ] Object storage: have the integration tests been executed and the 
endpoint declared according to the connector-specific documentation?
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, 
`NOTICE-binary` files?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] ashutoshcipher commented on a diff in pull request #4446: HADOOP-18294.Ensure build folder exists before writing checksum file.…

[GitBox]

ashutoshcipher commented on code in PR #4446:
URL: https://github.com/apache/hadoop/pull/4446#discussion_r907836527


##
hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/protoc/ProtocRunner.java:
##
@@ -172,6 +172,10 @@ private long computeChecksum(File file) throws IOException 
{
 
 public void writeChecksums() throws IOException {
   ObjectMapper mapper = new ObjectMapper();
+  boolean folderCreated = checksumFile.getParentFile().mkdirs();
+  if (!folderCreated) {
+throw new IOException("Unable to create path");
+  }

Review Comment:
   @aajisaka - Please share your advice. Thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] snmvaughan commented on pull request #4491: HADOOP-18311. Upgrade dependencies to address several CVEs

[GitBox]

snmvaughan commented on PR #4491:
URL: https://github.com/apache/hadoop/pull/4491#issuecomment-1167837080

   HTrace is a transitive dependency that is pulled in from HBase.  You can 
find htrace-core in our current 3.3.3 distribution, and I checked that it is 
also included in a build from trunk.  I've submitted YARN-11199 against trunk 
to specifically target replacement of htrace-core with hbase-noop-htrace.
   
   From: Steve Loughran ***@***.***>
   Sent: Monday, June 27, 2022 2:30 PM
   To: apache/hadoop ***@***.***>
   Cc: Steve Vaughan ***@***.***>; Author ***@***.***>
   Subject: Re: [apache/hadoop] HADOOP-18311. Upgrade dependencies to address 
several CVEs (PR #4491)
   
   
 1.  I've cherrypicked the aws sdk update we'd had in branch-3.3.
 2.  the jquery stuff should all be good now.
 3.  if htrace is still visible then that's a problem across the branches. 
is it getting in to the distribution, or is it a transitive dependency of 
something (test jar?) we don't distribute but do publish on maven.
 4.  jetty will need to be reviewed carefully.
   
   I am away for a week and want to make a release next week which is up to 
date security wise but not going to cause regressions. please can people sort 
htrace/jetty stuff out this week so everything is ready then. thanks
   
   —
   Reply to this email directly, view it on 
GitHub, or 
unsubscribe.
   You are receiving this because you authored the thread.Message ID: 
***@***.***>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] snmvaughan opened a new pull request, #4506: YARN-11199. Replace htrace-core with hbase-noop-htrace

[GitBox]

snmvaughan opened a new pull request, #4506:
URL: https://github.com/apache/hadoop/pull/4506

   Addresses the following CVEs:
   - CVE-2018-7489
   - CVE-2020-35491
   - CVE-2020-35490
   - CVE-2020-36518
   
   ### Description of PR
   
   Distributions of Hadoop still contain `htrace-core`, which is associated 
with 4 CVEs concerning FasterXML `jackson-databind`.  This can be addressed by 
replacing `htrace-core` with `hbase-noop-htrace` in Hadoop builds.
   
   ### How was this patch tested?
   
   The existence of `htrace-core` was verified in published distributions from 
the Apache Hadoop site, as well as by running a build against `trunk`.  Once 
the patch was applied `htrace-core` was replaced with `hbase-noop-htrace` in 
subsequent distribution builds, and all related tests passed as expected.
   
   ### For code changes:
   
   - [] Does the title or this PR starts with the corresponding JIRA issue id 
(e.g. 'HADOOP-17799. Your PR title ...')?
   - [ ] Object storage: have the integration tests been executed and the 
endpoint declared according to the connector-specific documentation?
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [X] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, 
`NOTICE-binary` files?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] slfan1989 commented on pull request #4464: YARN-11169. Support moveApplicationAcrossQueues, getQueueInfo API's for Federation.

[GitBox]

slfan1989 commented on PR #4464:
URL: https://github.com/apache/hadoop/pull/4464#issuecomment-1167799789

   @goiri Please help me to review the code, Thank you very much!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-17461) Add thread-level IOStatistics Context

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-17461?focusedWorklogId=785213=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-785213
 ]

ASF GitHub Bot logged work on HADOOP-17461:
---

Author: ASF GitHub Bot
Created on: 27/Jun/22 19:04
Start Date: 27/Jun/22 19:04
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on code in PR #4352:
URL: https://github.com/apache/hadoop/pull/4352#discussion_r907678976


##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/impl/IOStatisticsContext.java:
##
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+
+package org.apache.hadoop.fs.statistics.impl;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.impl.WeakReferenceThreadMap;
+import org.apache.hadoop.fs.statistics.IOStatistics;
+import org.apache.hadoop.fs.statistics.IOStatisticsAggregator;
+import org.apache.hadoop.fs.statistics.IOStatisticsSnapshot;
+
+import static 
org.apache.hadoop.fs.CommonConfigurationKeys.THREAD_LEVEL_IOSTATISTICS_ENABLED;
+import static 
org.apache.hadoop.fs.CommonConfigurationKeys.THREAD_LEVEL_IOSTATISTICS_ENABLED_DEFAULT;
+
+/**
+ * Implementing the IOStatisticsContext interface.

Review Comment:
   javadocs here don't match with latest code.



##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/impl/IOStatisticsContext.java:
##
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+
+package org.apache.hadoop.fs.statistics.impl;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.impl.WeakReferenceThreadMap;
+import org.apache.hadoop.fs.statistics.IOStatistics;
+import org.apache.hadoop.fs.statistics.IOStatisticsAggregator;
+import org.apache.hadoop.fs.statistics.IOStatisticsSnapshot;
+
+import static 
org.apache.hadoop.fs.CommonConfigurationKeys.THREAD_LEVEL_IOSTATISTICS_ENABLED;
+import static 
org.apache.hadoop.fs.CommonConfigurationKeys.THREAD_LEVEL_IOSTATISTICS_ENABLED_DEFAULT;
+
+/**
+ * Implementing the IOStatisticsContext interface.
+ *
+ * A Context defined for IOStatistics collection per thread which captures
+ * each worker thread's work in FS streams and stores it in the form of
+ * IOStatisticsAggregator which could be either IOStatisticsSnapshot or
+ * EmptyIOStatisticsStore if thread level aggregation is enabled or not for
+ * the FS. An active instance of the IOStatisticsContext can be used to
+ * collect the statistics.
+ *
+ * For the current thread the IOStatisticsSnapshot can be used as a way to
+ * move the IOStatistics data between applications using the Serializable
+ * nature of the class.
+ */
+public class IOStatisticsContext {
+  private static final Logger LOG =
+  LoggerFactory.getLogger(IOStatisticsContext.class);
+  private static final boolean IS_THREAD_IOSTATS_ENABLED;
+
+  private static final WeakReferenceThreadMap
+  ACTIVE_IOSTATS_CONTEXT = new WeakReferenceThreadMap<>(
+  IOStatisticsContext::createNewInstance,

Review Comment:
   nit, indent 

[GitHub] [hadoop] steveloughran commented on a diff in pull request #4352: HADOOP-17461. Thread-level IOStatistics in S3A

[GitBox]

steveloughran commented on code in PR #4352:
URL: https://github.com/apache/hadoop/pull/4352#discussion_r907678976


##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/impl/IOStatisticsContext.java:
##
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+
+package org.apache.hadoop.fs.statistics.impl;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.impl.WeakReferenceThreadMap;
+import org.apache.hadoop.fs.statistics.IOStatistics;
+import org.apache.hadoop.fs.statistics.IOStatisticsAggregator;
+import org.apache.hadoop.fs.statistics.IOStatisticsSnapshot;
+
+import static 
org.apache.hadoop.fs.CommonConfigurationKeys.THREAD_LEVEL_IOSTATISTICS_ENABLED;
+import static 
org.apache.hadoop.fs.CommonConfigurationKeys.THREAD_LEVEL_IOSTATISTICS_ENABLED_DEFAULT;
+
+/**
+ * Implementing the IOStatisticsContext interface.

Review Comment:
   javadocs here don't match with latest code.



##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/impl/IOStatisticsContext.java:
##
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+
+package org.apache.hadoop.fs.statistics.impl;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.impl.WeakReferenceThreadMap;
+import org.apache.hadoop.fs.statistics.IOStatistics;
+import org.apache.hadoop.fs.statistics.IOStatisticsAggregator;
+import org.apache.hadoop.fs.statistics.IOStatisticsSnapshot;
+
+import static 
org.apache.hadoop.fs.CommonConfigurationKeys.THREAD_LEVEL_IOSTATISTICS_ENABLED;
+import static 
org.apache.hadoop.fs.CommonConfigurationKeys.THREAD_LEVEL_IOSTATISTICS_ENABLED_DEFAULT;
+
+/**
+ * Implementing the IOStatisticsContext interface.
+ *
+ * A Context defined for IOStatistics collection per thread which captures
+ * each worker thread's work in FS streams and stores it in the form of
+ * IOStatisticsAggregator which could be either IOStatisticsSnapshot or
+ * EmptyIOStatisticsStore if thread level aggregation is enabled or not for
+ * the FS. An active instance of the IOStatisticsContext can be used to
+ * collect the statistics.
+ *
+ * For the current thread the IOStatisticsSnapshot can be used as a way to
+ * move the IOStatistics data between applications using the Serializable
+ * nature of the class.
+ */
+public class IOStatisticsContext {
+  private static final Logger LOG =
+  LoggerFactory.getLogger(IOStatisticsContext.class);
+  private static final boolean IS_THREAD_IOSTATS_ENABLED;
+
+  private static final WeakReferenceThreadMap
+  ACTIVE_IOSTATS_CONTEXT = new WeakReferenceThreadMap<>(
+  IOStatisticsContext::createNewInstance,

Review Comment:
   nit, indent this and L56



##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/impl/IOStatisticsContext.java:
##
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  

[jira] [Work logged] (HADOOP-18311) Upgrade dependencies to address several CVEs

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18311?focusedWorklogId=785201=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-785201
 ]

ASF GitHub Bot logged work on HADOOP-18311:
---

Author: ASF GitHub Bot
Created on: 27/Jun/22 18:26
Start Date: 27/Jun/22 18:26
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on PR #4491:
URL: https://github.com/apache/hadoop/pull/4491#issuecomment-1167721883

   1. I've cherrypicked the aws sdk update we'd had in branch-3.3.
   2. the jquery stuff should all be good now.
   3. if htrace is still visible then that's a problem across the branches. is 
it getting in to the distribution, or is it a transitive dependency of 
something (test jar?) we don't distribute but do publish on maven.
   4. jetty will need to be reviewed carefully.
   
   I am away for a week and want to make a release next week which is up to 
date security wise but not going to cause regressions. please can people sort 
htrace/jetty stuff out this week so everything is ready then. thanks




Issue Time Tracking
---

Worklog Id: (was: 785201)
Time Spent: 2h 10m  (was: 2h)

> Upgrade dependencies to address several CVEs
> 
>
> Key: HADOOP-18311
> URL: https://issues.apache.org/jira/browse/HADOOP-18311
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: common
>Affects Versions: 3.3.3, 3.3.4
>Reporter: Steve Vaughan
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.3.4
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> The following CVEs can be addressed by upgrading dependencies within the 
> build.  This includes a replacement of HTrace with a noop implementation.
>  * CVE-2018-7489
>  * CVE-2020-10663
>  * CVE-2020-28491
>  * CVE-2020-35490
>  * CVE-2020-35491
>  * CVE-2020-36518
>  * PRISMA-2021-0182
> This addresses all of the CVEs from 3.3.3 except for ones that would require 
> upgrading Netty to 4.x.  I'll be submitting a pull request for 3.3.4.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on pull request #4491: HADOOP-18311. Upgrade dependencies to address several CVEs

[GitBox]

steveloughran commented on PR #4491:
URL: https://github.com/apache/hadoop/pull/4491#issuecomment-1167721883

   1. I've cherrypicked the aws sdk update we'd had in branch-3.3.
   2. the jquery stuff should all be good now.
   3. if htrace is still visible then that's a problem across the branches. is 
it getting in to the distribution, or is it a transitive dependency of 
something (test jar?) we don't distribute but do publish on maven.
   4. jetty will need to be reviewed carefully.
   
   I am away for a week and want to make a release next week which is up to 
date security wise but not going to cause regressions. please can people sort 
htrace/jetty stuff out this week so everything is ready then. thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-18311) Upgrade dependencies to address several CVEs

[Steve Vaughan (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-18311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17559320#comment-17559320
 ] 

Steve Vaughan commented on HADOOP-18311:


I've started extracting these individual dependency upgrades, and will be 
submitting patches for trunk, branch-3.3, and branch-3.3.4.

> Upgrade dependencies to address several CVEs
> 
>
> Key: HADOOP-18311
> URL: https://issues.apache.org/jira/browse/HADOOP-18311
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: common
>Affects Versions: 3.3.3, 3.3.4
>Reporter: Steve Vaughan
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.3.4
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> The following CVEs can be addressed by upgrading dependencies within the 
> build.  This includes a replacement of HTrace with a noop implementation.
>  * CVE-2018-7489
>  * CVE-2020-10663
>  * CVE-2020-28491
>  * CVE-2020-35490
>  * CVE-2020-35491
>  * CVE-2020-36518
>  * PRISMA-2021-0182
> This addresses all of the CVEs from 3.3.3 except for ones that would require 
> upgrading Netty to 4.x.  I'll be submitting a pull request for 3.3.4.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-18242) ABFS Rename Failure when tracking metadata is in incomplete state

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18242?focusedWorklogId=785198=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-785198
 ]

ASF GitHub Bot logged work on HADOOP-18242:
---

Author: ASF GitHub Bot
Created on: 27/Jun/22 18:07
Start Date: 27/Jun/22 18:07
Worklog Time Spent: 10m 
  Work Description: steveloughran merged PR #4331:
URL: https://github.com/apache/hadoop/pull/4331




Issue Time Tracking
---

Worklog Id: (was: 785198)
Time Spent: 5h 10m  (was: 5h)

> ABFS Rename Failure when tracking metadata is in incomplete state
> -
>
> Key: HADOOP-18242
> URL: https://issues.apache.org/jira/browse/HADOOP-18242
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/azure
>Reporter: Mehakmeet Singh
>Assignee: Mehakmeet Singh
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>
> If a node in the datacenter crashes while processing an operation, 
> occasionally it can leave the Storage-internal blob tracking metadata in an 
> incomplete state.  We expect this to happen occasionally, and so all API’s 
> are designed in such a way that if this incomplete state is observed on a 
> blob, the situation is resolved before the current operation proceeds.  
> However, this incident has exposed a bug specifically with the Rename API, 
> where the incomplete state fails to resolve, leading to this incorrect 
> failure.  As a temporary mitigation, if any other operation is performed on 
> this blob – GetBlobProperties, GetBlob, GetFileProperties, SetFileProperties, 
> etc – it should resolve the incomplete state, and rename will no longer hit 
> this issue.
> StackTrace:
> {code:java}
> 2022-03-22 17:52:19,789 DEBUG [regionserver/euwukwlss-hg50:16020.logRoller] 
> services.AbfsClient: HttpRequest: 
> 404,RenameDestinationParentPathNotFound,cid=ef5cbf0f-5d4a-4630-8a59-3d559077fc24,rid=35fef164-101f-000b-1b15-3ed81800,sent=0,recv=212,PUT,https://euwqdaotdfdls03.dfs.core.windows.net/eykbssc/apps/hbase/data/oldWALs/euwukwlss-hg50.tdf.qa%252C16020%252C1647949929877.1647967939315?timeout=90
>{code}



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran merged pull request #4331: HADOOP-18242. ABFS Rename Failure when tracking metadata is in an incomplete state

[GitBox]

steveloughran merged PR #4331:
URL: https://github.com/apache/hadoop/pull/4331


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-18242) ABFS Rename Failure when tracking metadata is in incomplete state

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18242?focusedWorklogId=785197=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-785197
 ]

ASF GitHub Bot logged work on HADOOP-18242:
---

Author: ASF GitHub Bot
Created on: 27/Jun/22 18:05
Start Date: 27/Jun/22 18:05
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on PR #4331:
URL: https://github.com/apache/hadoop/pull/4331#issuecomment-1167695242

   +1, merging. 
   
   tested myself against azure cardiff; transient failure of 
TestAbfsClientThrottlingAnalyzer in the parallel run; went away on a standalone 
one. that test is too brittle.




Issue Time Tracking
---

Worklog Id: (was: 785197)
Time Spent: 5h  (was: 4h 50m)

> ABFS Rename Failure when tracking metadata is in incomplete state
> -
>
> Key: HADOOP-18242
> URL: https://issues.apache.org/jira/browse/HADOOP-18242
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/azure
>Reporter: Mehakmeet Singh
>Assignee: Mehakmeet Singh
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 5h
>  Remaining Estimate: 0h
>
> If a node in the datacenter crashes while processing an operation, 
> occasionally it can leave the Storage-internal blob tracking metadata in an 
> incomplete state.  We expect this to happen occasionally, and so all API’s 
> are designed in such a way that if this incomplete state is observed on a 
> blob, the situation is resolved before the current operation proceeds.  
> However, this incident has exposed a bug specifically with the Rename API, 
> where the incomplete state fails to resolve, leading to this incorrect 
> failure.  As a temporary mitigation, if any other operation is performed on 
> this blob – GetBlobProperties, GetBlob, GetFileProperties, SetFileProperties, 
> etc – it should resolve the incomplete state, and rename will no longer hit 
> this issue.
> StackTrace:
> {code:java}
> 2022-03-22 17:52:19,789 DEBUG [regionserver/euwukwlss-hg50:16020.logRoller] 
> services.AbfsClient: HttpRequest: 
> 404,RenameDestinationParentPathNotFound,cid=ef5cbf0f-5d4a-4630-8a59-3d559077fc24,rid=35fef164-101f-000b-1b15-3ed81800,sent=0,recv=212,PUT,https://euwqdaotdfdls03.dfs.core.windows.net/eykbssc/apps/hbase/data/oldWALs/euwukwlss-hg50.tdf.qa%252C16020%252C1647949929877.1647967939315?timeout=90
>{code}



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on pull request #4331: HADOOP-18242. ABFS Rename Failure when tracking metadata is in an incomplete state

[GitBox]

steveloughran commented on PR #4331:
URL: https://github.com/apache/hadoop/pull/4331#issuecomment-1167695242

   +1, merging. 
   
   tested myself against azure cardiff; transient failure of 
TestAbfsClientThrottlingAnalyzer in the parallel run; went away on a standalone 
one. that test is too brittle.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-18103) High performance vectored read API in Hadoop

[Mukund Thakur (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-18103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17559290#comment-17559290
 ] 

Mukund Thakur commented on HADOOP-18103:


merged to branch-3.3.

> High performance vectored read API in Hadoop
> 
>
> Key: HADOOP-18103
> URL: https://issues.apache.org/jira/browse/HADOOP-18103
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: common, fs, fs/adl, fs/s3
>Reporter: Mukund Thakur
>Assignee: Mukund Thakur
>Priority: Major
>  Labels: perfomance, pull-request-available
> Attachments: Vectored Read API for Hadoop FS_INCOMPLETE.pdf
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Add support for multiple ranged vectored read api in PositionedReadable. The 
> default iterates through the ranges to read each synchronously, but the 
> intent is that FSDataInputStream subclasses can make more efficient readers 
> especially object stores implementation.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-18103) High performance vectored read API in Hadoop

[Mukund Thakur (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mukund Thakur updated HADOOP-18103:
---
Fix Version/s: 3.4.0

> High performance vectored read API in Hadoop
> 
>
> Key: HADOOP-18103
> URL: https://issues.apache.org/jira/browse/HADOOP-18103
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: common, fs, fs/adl, fs/s3
>Reporter: Mukund Thakur
>Assignee: Mukund Thakur
>Priority: Major
>  Labels: perfomance, pull-request-available
> Fix For: 3.4.0
>
> Attachments: Vectored Read API for Hadoop FS_INCOMPLETE.pdf
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Add support for multiple ranged vectored read api in PositionedReadable. The 
> default iterates through the ranges to read each synchronously, but the 
> intent is that FSDataInputStream subclasses can make more efficient readers 
> especially object stores implementation.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-18103) High performance vectored read API in Hadoop

[Mukund Thakur (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mukund Thakur updated HADOOP-18103:
---
Fix Version/s: 3.4.0
   (was: 3.3.5)

> High performance vectored read API in Hadoop
> 
>
> Key: HADOOP-18103
> URL: https://issues.apache.org/jira/browse/HADOOP-18103
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: common, fs, fs/adl, fs/s3
>Reporter: Mukund Thakur
>Assignee: Mukund Thakur
>Priority: Major
>  Labels: perfomance, pull-request-available
> Fix For: 3.4.0
>
> Attachments: Vectored Read API for Hadoop FS_INCOMPLETE.pdf
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Add support for multiple ranged vectored read api in PositionedReadable. The 
> default iterates through the ranges to read each synchronously, but the 
> intent is that FSDataInputStream subclasses can make more efficient readers 
> especially object stores implementation.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-18103) High performance vectored read API in Hadoop

[Mukund Thakur (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mukund Thakur updated HADOOP-18103:
---
Fix Version/s: 3.3.5
   (was: 3.4.0)

> High performance vectored read API in Hadoop
> 
>
> Key: HADOOP-18103
> URL: https://issues.apache.org/jira/browse/HADOOP-18103
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: common, fs, fs/adl, fs/s3
>Reporter: Mukund Thakur
>Assignee: Mukund Thakur
>Priority: Major
>  Labels: perfomance, pull-request-available
> Fix For: 3.3.5
>
> Attachments: Vectored Read API for Hadoop FS_INCOMPLETE.pdf
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Add support for multiple ranged vectored read api in PositionedReadable. The 
> default iterates through the ranges to read each synchronously, but the 
> intent is that FSDataInputStream subclasses can make more efficient readers 
> especially object stores implementation.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-18311) Upgrade dependencies to address several CVEs

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18311?focusedWorklogId=785184=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-785184
 ]

ASF GitHub Bot logged work on HADOOP-18311:
---

Author: ASF GitHub Bot
Created on: 27/Jun/22 17:08
Start Date: 27/Jun/22 17:08
Worklog Time Spent: 10m 
  Work Description: jojochuang commented on PR #4491:
URL: https://github.com/apache/hadoop/pull/4491#issuecomment-1167625773

   Maintenance release update of Jetty is usually fine as long as the unit 
tests pass and that no additional dependencies got introduced.




Issue Time Tracking
---

Worklog Id: (was: 785184)
Time Spent: 2h  (was: 1h 50m)

> Upgrade dependencies to address several CVEs
> 
>
> Key: HADOOP-18311
> URL: https://issues.apache.org/jira/browse/HADOOP-18311
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: common
>Affects Versions: 3.3.3, 3.3.4
>Reporter: Steve Vaughan
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.3.4
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> The following CVEs can be addressed by upgrading dependencies within the 
> build.  This includes a replacement of HTrace with a noop implementation.
>  * CVE-2018-7489
>  * CVE-2020-10663
>  * CVE-2020-28491
>  * CVE-2020-35490
>  * CVE-2020-35491
>  * CVE-2020-36518
>  * PRISMA-2021-0182
> This addresses all of the CVEs from 3.3.3 except for ones that would require 
> upgrading Netty to 4.x.  I'll be submitting a pull request for 3.3.4.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] jojochuang commented on pull request #4491: HADOOP-18311. Upgrade dependencies to address several CVEs

[GitBox]

jojochuang commented on PR #4491:
URL: https://github.com/apache/hadoop/pull/4491#issuecomment-1167625773

   Maintenance release update of Jetty is usually fine as long as the unit 
tests pass and that no additional dependencies got introduced.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on pull request #4331: HADOOP-18242. ABFS Rename Failure when tracking metadata is in an incomplete state

[GitBox]

steveloughran commented on PR #4331:
URL: https://github.com/apache/hadoop/pull/4331#issuecomment-1167506215

   due diligence question: what did you test against?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-18242) ABFS Rename Failure when tracking metadata is in incomplete state

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18242?focusedWorklogId=785152=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-785152
 ]

ASF GitHub Bot logged work on HADOOP-18242:
---

Author: ASF GitHub Bot
Created on: 27/Jun/22 15:35
Start Date: 27/Jun/22 15:35
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on PR #4331:
URL: https://github.com/apache/hadoop/pull/4331#issuecomment-1167506215

   due diligence question: what did you test against?




Issue Time Tracking
---

Worklog Id: (was: 785152)
Time Spent: 4h 50m  (was: 4h 40m)

> ABFS Rename Failure when tracking metadata is in incomplete state
> -
>
> Key: HADOOP-18242
> URL: https://issues.apache.org/jira/browse/HADOOP-18242
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/azure
>Reporter: Mehakmeet Singh
>Assignee: Mehakmeet Singh
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> If a node in the datacenter crashes while processing an operation, 
> occasionally it can leave the Storage-internal blob tracking metadata in an 
> incomplete state.  We expect this to happen occasionally, and so all API’s 
> are designed in such a way that if this incomplete state is observed on a 
> blob, the situation is resolved before the current operation proceeds.  
> However, this incident has exposed a bug specifically with the Rename API, 
> where the incomplete state fails to resolve, leading to this incorrect 
> failure.  As a temporary mitigation, if any other operation is performed on 
> this blob – GetBlobProperties, GetBlob, GetFileProperties, SetFileProperties, 
> etc – it should resolve the incomplete state, and rename will no longer hit 
> this issue.
> StackTrace:
> {code:java}
> 2022-03-22 17:52:19,789 DEBUG [regionserver/euwukwlss-hg50:16020.logRoller] 
> services.AbfsClient: HttpRequest: 
> 404,RenameDestinationParentPathNotFound,cid=ef5cbf0f-5d4a-4630-8a59-3d559077fc24,rid=35fef164-101f-000b-1b15-3ed81800,sent=0,recv=212,PUT,https://euwqdaotdfdls03.dfs.core.windows.net/eykbssc/apps/hbase/data/oldWALs/euwukwlss-hg50.tdf.qa%252C16020%252C1647949929877.1647967939315?timeout=90
>{code}



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-18311) Upgrade dependencies to address several CVEs

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18311?focusedWorklogId=785118=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-785118
 ]

ASF GitHub Bot logged work on HADOOP-18311:
---

Author: ASF GitHub Bot
Created on: 27/Jun/22 14:11
Start Date: 27/Jun/22 14:11
Worklog Time Spent: 10m 
  Work Description: snmvaughan commented on PR #4491:
URL: https://github.com/apache/hadoop/pull/4491#issuecomment-1167405730

   I understand the desire to evaluate and manage the individual changes, so 
I'll resubmit the individual dependency updates.  I was already planning on 
submitting these for branch-3.3 when 3.3.4 switch came to my attention.
   
   The HTrace library is showing up in the distribution as part of a transitive 
dependency from HBase.  Given the goal to remove the dependency, the CVE, and 
ongoing work to move to OpenTelemetry, I would suggest we re-consider the 
hbase-noop-htrace swap.
   
   I was already planning on looking into the Curator 5 related ZooKeeper pull 
request, but felt the elimination of CVEs in the short-term was important.




Issue Time Tracking
---

Worklog Id: (was: 785118)
Time Spent: 1h 50m  (was: 1h 40m)

> Upgrade dependencies to address several CVEs
> 
>
> Key: HADOOP-18311
> URL: https://issues.apache.org/jira/browse/HADOOP-18311
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: common
>Affects Versions: 3.3.3, 3.3.4
>Reporter: Steve Vaughan
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.3.4
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> The following CVEs can be addressed by upgrading dependencies within the 
> build.  This includes a replacement of HTrace with a noop implementation.
>  * CVE-2018-7489
>  * CVE-2020-10663
>  * CVE-2020-28491
>  * CVE-2020-35490
>  * CVE-2020-35491
>  * CVE-2020-36518
>  * PRISMA-2021-0182
> This addresses all of the CVEs from 3.3.3 except for ones that would require 
> upgrading Netty to 4.x.  I'll be submitting a pull request for 3.3.4.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] snmvaughan commented on pull request #4491: HADOOP-18311. Upgrade dependencies to address several CVEs

[GitBox]

snmvaughan commented on PR #4491:
URL: https://github.com/apache/hadoop/pull/4491#issuecomment-1167405730

   I understand the desire to evaluate and manage the individual changes, so 
I'll resubmit the individual dependency updates.  I was already planning on 
submitting these for branch-3.3 when 3.3.4 switch came to my attention.
   
   The HTrace library is showing up in the distribution as part of a transitive 
dependency from HBase.  Given the goal to remove the dependency, the CVE, and 
ongoing work to move to OpenTelemetry, I would suggest we re-consider the 
hbase-noop-htrace swap.
   
   I was already planning on looking into the Curator 5 related ZooKeeper pull 
request, but felt the elimination of CVEs in the short-term was important.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-18311) Upgrade dependencies to address several CVEs

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18311?focusedWorklogId=785103=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-785103
 ]

ASF GitHub Bot logged work on HADOOP-18311:
---

Author: ASF GitHub Bot
Created on: 27/Jun/22 13:56
Start Date: 27/Jun/22 13:56
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on PR #4491:
URL: https://github.com/apache/hadoop/pull/4491#issuecomment-1167385330

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m  0s |  |  Docker mode activated.  |
   | -1 :x: |  patch  |   0m 20s |  |  
https://github.com/apache/hadoop/pull/4491 does not apply to branch-3.3.4. 
Rebase required? Wrong Branch? See 
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.  
|
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | GITHUB PR | https://github.com/apache/hadoop/pull/4491 |
   | Console output | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4491/2/console |
   | versions | git=2.17.1 |
   | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   




Issue Time Tracking
---

Worklog Id: (was: 785103)
Time Spent: 1h 40m  (was: 1.5h)

> Upgrade dependencies to address several CVEs
> 
>
> Key: HADOOP-18311
> URL: https://issues.apache.org/jira/browse/HADOOP-18311
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: common
>Affects Versions: 3.3.3, 3.3.4
>Reporter: Steve Vaughan
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.3.4
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> The following CVEs can be addressed by upgrading dependencies within the 
> build.  This includes a replacement of HTrace with a noop implementation.
>  * CVE-2018-7489
>  * CVE-2020-10663
>  * CVE-2020-28491
>  * CVE-2020-35490
>  * CVE-2020-35491
>  * CVE-2020-36518
>  * PRISMA-2021-0182
> This addresses all of the CVEs from 3.3.3 except for ones that would require 
> upgrading Netty to 4.x.  I'll be submitting a pull request for 3.3.4.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] hadoop-yetus commented on pull request #4491: HADOOP-18311. Upgrade dependencies to address several CVEs

[GitBox]

hadoop-yetus commented on PR #4491:
URL: https://github.com/apache/hadoop/pull/4491#issuecomment-1167385330

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m  0s |  |  Docker mode activated.  |
   | -1 :x: |  patch  |   0m 20s |  |  
https://github.com/apache/hadoop/pull/4491 does not apply to branch-3.3.4. 
Rebase required? Wrong Branch? See 
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.  
|
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | GITHUB PR | https://github.com/apache/hadoop/pull/4491 |
   | Console output | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4491/2/console |
   | versions | git=2.17.1 |
   | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-18273) Support S3 access point alias

[Daniel Carl Jones (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-18273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17559172#comment-17559172
 ] 

Daniel Carl Jones edited comment on HADOOP-18273 at 6/27/22 1:42 PM:
-

I ran the integration tests using an S3 Access Point alias in place of the S3 
Bucket name. This bucket was based in eu-west-1 and I was using the 
s3.amazonaws.com endpoint.

All passing except for 
{_}testCustomSignerAndInitializer(org.apache.hadoop.fs.s3a.auth.ITestCustomSigner){_}.
 It errors with "org.apache.hadoop.fs.s3a.AWSRedirectException: Received 
permanent redirect response to endpoint 
test-aliases-npjsxa34bnh9w6nrynbsnxckpbtzeeuw1a-s3alias.s3-eu-west-1.amazonaws.com".


was (Author: JIRAUSER284792):
I ran the integration tests using an S3 Access Point alias in place of the S3 
Bucket name. This bucket was based in eu-west-1 and I was using the 
s3.amazonaws.com endpoint.

All passing except for 
{_}testCustomSignerAndInitializer(org.apache.hadoop.fs.s3a.auth.ITestCustomSigner){_}.
 It errors with "org.apache.hadoop.fs.s3a.AWSRedirectException: Received 
permanent redirect response" for the alias' endpoint.

> Support S3 access point alias
> -
>
> Key: HADOOP-18273
> URL: https://issues.apache.org/jira/browse/HADOOP-18273
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.3.9
>Reporter: Shen Cong
>Priority: Major
>
> Adding support for using access point alias to access s3 bucket.
>  
> When you create an access point, Amazon S3 automatically generates an alias 
> that you can use instead of an Amazon S3 bucket name for data access. You can 
> use this access point alias instead of an Amazon Resource Name (ARN) for any 
> access point data plane operation.
> https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-points-alias.html



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-18273) Support S3 access point alias

[Daniel Carl Jones (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-18273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17559172#comment-17559172
 ] 

Daniel Carl Jones commented on HADOOP-18273:


I ran the integration tests using an S3 Access Point alias in place of the S3 
Bucket name. This bucket was based in eu-west-1 and I was using the 
s3.amazonaws.com endpoint.

All passing except for 
{_}testCustomSignerAndInitializer(org.apache.hadoop.fs.s3a.auth.ITestCustomSigner){_}.
 It errors with "org.apache.hadoop.fs.s3a.AWSRedirectException: Received 
permanent redirect response" for the alias' endpoint.

> Support S3 access point alias
> -
>
> Key: HADOOP-18273
> URL: https://issues.apache.org/jira/browse/HADOOP-18273
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.3.9
>Reporter: Shen Cong
>Priority: Major
>
> Adding support for using access point alias to access s3 bucket.
>  
> When you create an access point, Amazon S3 automatically generates an alias 
> that you can use instead of an Amazon S3 bucket name for data access. You can 
> use this access point alias instead of an Amazon Resource Name (ARN) for any 
> access point data plane operation.
> https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-points-alias.html



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] hadoop-yetus commented on pull request #4450: YARN-11183. Federation: Remove outdated ApplicationHomeSubCluster in …

[GitBox]

hadoop-yetus commented on PR #4450:
URL: https://github.com/apache/hadoop/pull/4450#issuecomment-1167297126

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 44s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  
|
   | +0 :ok: |  buf  |   0m  1s |  |  buf was not available.  |
   | +0 :ok: |  buf  |   0m  1s |  |  buf was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 5 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  15m  5s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  27m 45s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  10m 55s |  |  trunk passed with JDK 
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1  |
   | +1 :green_heart: |  compile  |   9m 29s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   2m 17s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   9m 19s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   6m 20s |  |  trunk passed with JDK 
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1  |
   | +1 :green_heart: |  javadoc  |   5m 51s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |  20m  0s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  21m 28s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 29s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   8m 19s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  10m 10s |  |  the patch passed with JDK 
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1  |
   | +1 :green_heart: |  cc  |  10m 10s |  |  the patch passed  |
   | +1 :green_heart: |  javac  |  10m 10s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   9m 19s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  cc  |   9m 19s |  |  the patch passed  |
   | +1 :green_heart: |  javac  |   9m 19s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | -0 :warning: |  checkstyle  |   1m 54s | 
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4450/5/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
 |  hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 87 unchanged - 
0 fixed = 88 total (was 87)  |
   | +1 :green_heart: |  mvnsite  |   8m 26s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   5m 46s |  |  the patch passed with JDK 
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1  |
   | +1 :green_heart: |  javadoc  |   5m 21s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |  19m 46s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  20m 58s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  | 235m 51s |  |  hadoop-yarn in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   5m 44s |  |  hadoop-yarn-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   3m 33s |  |  hadoop-yarn-server-common in 
the patch passed.  |
   | +1 :green_heart: |  unit  |  99m  5s |  |  
hadoop-yarn-server-resourcemanager in the patch passed.  |
   | +1 :green_heart: |  unit  |   3m 53s |  |  hadoop-yarn-server-router in 
the patch passed.  |
   | +1 :green_heart: |  asflicense  |   1m 40s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 573m 13s |  |  |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4450/5/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/4450 |
   | Optional Tests | dupname asflicense codespell detsecrets compile javac 
javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle cc buflint 
bufcompat |
   | uname | Linux 79e13e6f23c7 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision 

[jira] [Comment Edited] (HADOOP-14309) Add PowerShell NodeFencer

[JiangHua Zhu (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17559072#comment-17559072
 ] 

JiangHua Zhu edited comment on HADOOP-14309 at 6/27/22 12:23 PM:
-

Nice to talk to you, [~elgoiri].
I found that some documentation here has not been updated.
I want to try to improve them, because good documentation helps make the system 
more robust.
This is my original intention.
Hope to continue to communicate with you.


was (Author: jianghuazhu):
Nice to talk to you, [~elgoiri].
I found that HDFSHighAvailabilityWithQJM.md is not updated.
I will create a new jira if needed.

> Add PowerShell NodeFencer
> -
>
> Key: HADOOP-14309
> URL: https://issues.apache.org/jira/browse/HADOOP-14309
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: ha
>Affects Versions: 2.8.0
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Minor
> Fix For: 3.0.0-alpha4
>
> Attachments: HADOOP-14309-000.patch, HADOOP-14309-001.patch
>
>
> Fencer implementation that uses PowerShell to fence a remote component.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work started] (HADOOP-18314) Add some description for PowerShellFencer

[JiangHua Zhu (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HADOOP-18314 started by JiangHua Zhu.
-
> Add some description for PowerShellFencer
> -
>
> Key: HADOOP-18314
> URL: https://issues.apache.org/jira/browse/HADOOP-18314
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: ha
>Affects Versions: 3.3.3
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Some descriptions related to PowerShellFencer are missing here, such as: 
> core-default.xml, HDFSHighAvailabilityWithQJM.md, 
> HDFSHighAvailabilityWithNFS.md, NodeFencer.java.
> We should have as much documentation as possible.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-18314) Add some description for PowerShellFencer

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HADOOP-18314:

Labels: pull-request-available  (was: )

> Add some description for PowerShellFencer
> -
>
> Key: HADOOP-18314
> URL: https://issues.apache.org/jira/browse/HADOOP-18314
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: ha
>Affects Versions: 3.3.3
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Some descriptions related to PowerShellFencer are missing here, such as: 
> core-default.xml, HDFSHighAvailabilityWithQJM.md, 
> HDFSHighAvailabilityWithNFS.md, NodeFencer.java.
> We should have as much documentation as possible.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-18314) Add some description for PowerShellFencer

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18314?focusedWorklogId=785055=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-785055
 ]

ASF GitHub Bot logged work on HADOOP-18314:
---

Author: ASF GitHub Bot
Created on: 27/Jun/22 12:16
Start Date: 27/Jun/22 12:16
Worklog Time Spent: 10m 
  Work Description: jianghuazhu opened a new pull request, #4505:
URL: https://github.com/apache/hadoop/pull/4505

   
   ### Description of PR
   Some descriptions related to PowerShellFencer are missing here, such as: 
core-default.xml, HDFSHighAvailabilityWithQJM.md, 
HDFSHighAvailabilityWithNFS.md, NodeFencer.java.
   The purpose of this PR is to perfect them.
   Details: HADOOP-18314
   
   ### How was this patch tested?
   Here is mainly to update some documents, for testing, there is not much 
pressure.
   
   




Issue Time Tracking
---

Worklog Id: (was: 785055)
Remaining Estimate: 0h
Time Spent: 10m

> Add some description for PowerShellFencer
> -
>
> Key: HADOOP-18314
> URL: https://issues.apache.org/jira/browse/HADOOP-18314
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: ha
>Affects Versions: 3.3.3
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Some descriptions related to PowerShellFencer are missing here, such as: 
> core-default.xml, HDFSHighAvailabilityWithQJM.md, 
> HDFSHighAvailabilityWithNFS.md, NodeFencer.java.
> We should have as much documentation as possible.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] jianghuazhu opened a new pull request, #4505: HADOOP-18314.Add some description for PowerShellFencer.

[GitBox]

jianghuazhu opened a new pull request, #4505:
URL: https://github.com/apache/hadoop/pull/4505

   
   ### Description of PR
   Some descriptions related to PowerShellFencer are missing here, such as: 
core-default.xml, HDFSHighAvailabilityWithQJM.md, 
HDFSHighAvailabilityWithNFS.md, NodeFencer.java.
   The purpose of this PR is to perfect them.
   Details: HADOOP-18314
   
   ### How was this patch tested?
   Here is mainly to update some documents, for testing, there is not much 
pressure.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] hadoop-yetus commented on pull request #4450: YARN-11183. Federation: Remove outdated ApplicationHomeSubCluster in …

[GitBox]

hadoop-yetus commented on PR #4450:
URL: https://github.com/apache/hadoop/pull/4450#issuecomment-1167268391

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 43s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  1s |  |  detect-secrets was not available.  
|
   | +0 :ok: |  buf  |   0m  0s |  |  buf was not available.  |
   | +0 :ok: |  buf  |   0m  0s |  |  buf was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 4 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  15m 44s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  25m 37s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  10m 18s |  |  trunk passed with JDK 
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1  |
   | +1 :green_heart: |  compile  |   9m 29s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   2m 10s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   8m 34s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   6m  3s |  |  trunk passed with JDK 
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1  |
   | +1 :green_heart: |  javadoc  |   5m 33s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |  18m 43s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  23m 18s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 29s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   7m 33s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  10m 10s |  |  the patch passed with JDK 
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1  |
   | +1 :green_heart: |  cc  |  10m 10s |  |  the patch passed  |
   | +1 :green_heart: |  javac  |  10m 10s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   8m 56s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  cc  |   8m 56s |  |  the patch passed  |
   | +1 :green_heart: |  javac  |   8m 56s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | -0 :warning: |  checkstyle  |   1m 47s | 
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4450/4/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
 |  hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 87 unchanged - 
0 fixed = 88 total (was 87)  |
   | +1 :green_heart: |  mvnsite  |   8m  2s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   5m 46s |  |  the patch passed with JDK 
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1  |
   | +1 :green_heart: |  javadoc  |   5m  6s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |  18m 53s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  23m 10s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | -1 :x: |  unit  | 253m 26s | 
[/patch-unit-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4450/4/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn.txt)
 |  hadoop-yarn in the patch passed.  |
   | +1 :green_heart: |  unit  |   5m 41s |  |  hadoop-yarn-common in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   3m 40s |  |  hadoop-yarn-server-common in 
the patch passed.  |
   | -1 :x: |  unit  |  99m 16s | 
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4450/4/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
 |  hadoop-yarn-server-resourcemanager in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   1m 25s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 580m 43s |  |  |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | 
hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy |
   |   | 
hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | 

[jira] [Created] (HADOOP-18314) Add some description for PowerShellFencer

[JiangHua Zhu (Jira)]

JiangHua Zhu created HADOOP-18314:
-

 Summary: Add some description for PowerShellFencer
 Key: HADOOP-18314
 URL: https://issues.apache.org/jira/browse/HADOOP-18314
 Project: Hadoop Common
  Issue Type: Improvement
  Components: ha
Affects Versions: 3.3.3
Reporter: JiangHua Zhu


Some descriptions related to PowerShellFencer are missing here, such as: 
core-default.xml, HDFSHighAvailabilityWithQJM.md, 
HDFSHighAvailabilityWithNFS.md, NodeFencer.java.
We should have as much documentation as possible.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Assigned] (HADOOP-18314) Add some description for PowerShellFencer

[JiangHua Zhu (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

JiangHua Zhu reassigned HADOOP-18314:
-

Assignee: JiangHua Zhu

> Add some description for PowerShellFencer
> -
>
> Key: HADOOP-18314
> URL: https://issues.apache.org/jira/browse/HADOOP-18314
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: ha
>Affects Versions: 3.3.3
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
>
> Some descriptions related to PowerShellFencer are missing here, such as: 
> core-default.xml, HDFSHighAvailabilityWithQJM.md, 
> HDFSHighAvailabilityWithNFS.md, NodeFencer.java.
> We should have as much documentation as possible.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-18242) ABFS Rename Failure when tracking metadata is in incomplete state

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18242?focusedWorklogId=785048=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-785048
 ]

ASF GitHub Bot logged work on HADOOP-18242:
---

Author: ASF GitHub Bot
Created on: 27/Jun/22 11:43
Start Date: 27/Jun/22 11:43
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on PR #4331:
URL: https://github.com/apache/hadoop/pull/4331#issuecomment-1167249726

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 49s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 4 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  40m 18s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   0m 53s |  |  trunk passed with JDK 
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1  |
   | +1 :green_heart: |  compile  |   0m 46s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   0m 41s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   0m 50s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   0m 48s |  |  trunk passed with JDK 
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1  |
   | +1 :green_heart: |  javadoc  |   0m 40s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   1m 26s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  23m 53s |  |  branch has no errors 
when building and testing our client artifacts.  |
   | -0 :warning: |  patch  |  24m 16s |  |  Used diff version of patch file. 
Binary files and potentially other changes not applied. Please rebase and 
squash commits if necessary.  |
    _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 35s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 38s |  |  the patch passed with JDK 
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1  |
   | +1 :green_heart: |  javac  |   0m 38s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 33s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  javac  |   0m 33s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | +1 :green_heart: |  checkstyle  |   0m 23s |  |  the patch passed  |
   | +1 :green_heart: |  mvnsite  |   0m 37s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 28s |  |  the patch passed with JDK 
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1  |
   | +1 :green_heart: |  javadoc  |   0m 27s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   1m 11s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  23m  1s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |   2m  2s |  |  hadoop-azure in the patch 
passed.  |
   | +1 :green_heart: |  asflicense  |   0m 42s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 102m 34s |  |  |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4331/7/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/4331 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
   | uname | Linux c50fd83515c3 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / fc45b7a86230404bc6ae34a8b93968886cc1750d |
   | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private 
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 
/usr/lib/jvm/java-8-openjdk-amd64:Private 
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
   |  Test Results | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4331/7/testReport/ |
   | Max. process+thread count | 533 (vs. ulimit of 5500) |
   | 

[GitHub] [hadoop] hadoop-yetus commented on pull request #4331: HADOOP-18242. ABFS Rename Failure when tracking metadata is in an incomplete state

[GitBox]

hadoop-yetus commented on PR #4331:
URL: https://github.com/apache/hadoop/pull/4331#issuecomment-1167249726

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 49s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 4 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  40m 18s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   0m 53s |  |  trunk passed with JDK 
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1  |
   | +1 :green_heart: |  compile  |   0m 46s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   0m 41s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   0m 50s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   0m 48s |  |  trunk passed with JDK 
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1  |
   | +1 :green_heart: |  javadoc  |   0m 40s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   1m 26s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  23m 53s |  |  branch has no errors 
when building and testing our client artifacts.  |
   | -0 :warning: |  patch  |  24m 16s |  |  Used diff version of patch file. 
Binary files and potentially other changes not applied. Please rebase and 
squash commits if necessary.  |
    _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 35s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 38s |  |  the patch passed with JDK 
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1  |
   | +1 :green_heart: |  javac  |   0m 38s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 33s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  javac  |   0m 33s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | +1 :green_heart: |  checkstyle  |   0m 23s |  |  the patch passed  |
   | +1 :green_heart: |  mvnsite  |   0m 37s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 28s |  |  the patch passed with JDK 
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1  |
   | +1 :green_heart: |  javadoc  |   0m 27s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   1m 11s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  23m  1s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |   2m  2s |  |  hadoop-azure in the patch 
passed.  |
   | +1 :green_heart: |  asflicense  |   0m 42s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 102m 34s |  |  |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4331/7/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/4331 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
   | uname | Linux c50fd83515c3 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / fc45b7a86230404bc6ae34a8b93968886cc1750d |
   | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private 
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 
/usr/lib/jvm/java-8-openjdk-amd64:Private 
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
   |  Test Results | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4331/7/testReport/ |
   | Max. process+thread count | 533 (vs. ulimit of 5500) |
   | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
   | Console output | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4331/7/console |
   | versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
   | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the 

[jira] [Work logged] (HADOOP-18307) remove hadoop-cos as a dependency of hadoop-cloud-storage

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18307?focusedWorklogId=785027=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-785027
 ]

ASF GitHub Bot logged work on HADOOP-18307:
---

Author: ASF GitHub Bot
Created on: 27/Jun/22 10:48
Start Date: 27/Jun/22 10:48
Worklog Time Spent: 10m 
  Work Description: steveloughran closed pull request #4481: HADOOP-18307. 
Remove hadoop-cos as a dependency of hadoop-cloud-storage.
URL: https://github.com/apache/hadoop/pull/4481




Issue Time Tracking
---

Worklog Id: (was: 785027)
Time Spent: 50m  (was: 40m)

> remove hadoop-cos as a dependency of hadoop-cloud-storage
> -
>
> Key: HADOOP-18307
> URL: https://issues.apache.org/jira/browse/HADOOP-18307
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: bulid, fs
>Affects Versions: 3.3.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.3.4
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> to deal without HADOOP-18159 without qualifying an updated cos library, 
> remove it as an explicit dependency of hadoop cloud storage
> it will still be built and published, 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran closed pull request #4481: HADOOP-18307. Remove hadoop-cos as a dependency of hadoop-cloud-storage.

[GitBox]

steveloughran closed pull request #4481: HADOOP-18307. Remove hadoop-cos as a 
dependency of hadoop-cloud-storage.
URL: https://github.com/apache/hadoop/pull/4481


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-18308) Update to Apache LDAP API 2.0.x

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18308?focusedWorklogId=785019=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-785019
 ]

ASF GitHub Bot logged work on HADOOP-18308:
---

Author: ASF GitHub Bot
Created on: 27/Jun/22 10:15
Start Date: 27/Jun/22 10:15
Worklog Time Spent: 10m 
  Work Description: steveloughran merged PR #4477:
URL: https://github.com/apache/hadoop/pull/4477




Issue Time Tracking
---

Worklog Id: (was: 785019)
Time Spent: 1h  (was: 50m)

> Update to Apache LDAP API 2.0.x
> ---
>
> Key: HADOOP-18308
> URL: https://issues.apache.org/jira/browse/HADOOP-18308
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: build
>Affects Versions: 3.3.3
>Reporter: Colm O hEigeartaigh
>Assignee: Colm O hEigeartaigh
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Update from Apache LDAP API 1.x to 2.0.x.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran merged pull request #4477: HADOOP-18308 - Update to Apache LDAP API 2.0.x

[GitBox]

steveloughran merged PR #4477:
URL: https://github.com/apache/hadoop/pull/4477


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work logged] (HADOOP-18074) Partial/Incomplete groups list can be returned in LDAP groups lookup

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18074?focusedWorklogId=785017=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-785017
 ]

ASF GitHub Bot logged work on HADOOP-18074:
---

Author: ASF GitHub Bot
Created on: 27/Jun/22 10:02
Start Date: 27/Jun/22 10:02
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on code in PR #4503:
URL: https://github.com/apache/hadoop/pull/4503#discussion_r907199594


##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestLdapGroupsMappingWithOneQuery.java:
##
@@ -81,16 +86,72 @@ private void doTestGetGroups(List expectedGroups)
 // enable single-query lookup
 conf.set(LdapGroupsMapping.MEMBEROF_ATTR_KEY, "memberOf");
 
-LdapGroupsMapping groupsMapping = getGroupsMapping();
+TestLdapGroupsMapping groupsMapping = new TestLdapGroupsMapping();
 groupsMapping.setConf(conf);
 // Username is arbitrary, since the spy is mocked to respond the same,
 // regardless of input
 List groups = groupsMapping.getGroups("some_user");
 
 Assert.assertEquals(expectedGroups, groups);
+Assert.assertFalse(groupsMapping.isSecondaryQueryCalled());

Review Comment:
   add an error message for the assertion failure



##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestLdapGroupsMappingWithOneQuery.java:
##
@@ -81,16 +86,72 @@ private void doTestGetGroups(List expectedGroups)
 // enable single-query lookup
 conf.set(LdapGroupsMapping.MEMBEROF_ATTR_KEY, "memberOf");
 
-LdapGroupsMapping groupsMapping = getGroupsMapping();
+TestLdapGroupsMapping groupsMapping = new TestLdapGroupsMapping();
 groupsMapping.setConf(conf);
 // Username is arbitrary, since the spy is mocked to respond the same,
 // regardless of input
 List groups = groupsMapping.getGroups("some_user");
 
 Assert.assertEquals(expectedGroups, groups);
+Assert.assertFalse(groupsMapping.isSecondaryQueryCalled());
 
 // We should have only made one query because single-query lookup is 
enabled
 verify(getContext(), times(1)).search(anyString(), anyString(),
 any(Object[].class), any(SearchControls.class));
   }
-}
\ No newline at end of file
+
+  private void doTestGetGroupsWithFallback()
+  throws NamingException {
+Attribute groupDN = mock(Attribute.class);
+
+NamingEnumeration groupNames = getGroupNames();
+doReturn(groupNames).when(groupDN).getAll();
+String groupName1 = "CN=abc,DC=foo,DC=bar,DC=com";
+String groupName2 = "CN=xyz,DC=foo,DC=bar,DC=com";
+String groupName3 = "ipaUniqueID=e4a9a634-bb24-11ec-aec1-06ede52b5fe1," +
+"CN=sudo,DC=foo,DC=bar,DC=com";
+doReturn(groupName1).doReturn(groupName2).doReturn(groupName3).
+when(groupNames).next();
+when(groupNames.hasMore()).thenReturn(true).thenReturn(true).
+thenReturn(true).thenReturn(false);
+
+when(getAttributes().get(eq("memberOf"))).thenReturn(groupDN);
+
+String ldapUrl = "ldap://test;;
+Configuration conf = getBaseConf(ldapUrl);
+// enable single-query lookup
+conf.set(LdapGroupsMapping.MEMBEROF_ATTR_KEY, "memberOf");
+conf.set(LdapGroupsMapping.LDAP_NUM_ATTEMPTS_KEY, "1");
+
+TestLdapGroupsMapping groupsMapping = new TestLdapGroupsMapping();
+groupsMapping.setConf(conf);
+// Username is arbitrary, since the spy is mocked to respond the same,
+// regardless of input
+List groups = groupsMapping.getGroups("some_user");
+
+// expected to be empty due to invalid memberOf
+Assert.assertEquals(0, groups.size());
+
+// expect secondary query to be called: getGroups()
+Assert.assertTrue(groupsMapping.isSecondaryQueryCalled());

Review Comment:
   add an error message



##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/LdapGroupsMapping.java:
##
@@ -428,7 +428,7 @@ private NamingEnumeration 
lookupPosixGroup(SearchResult result,
* @return a list of strings representing group names of the user.
* @throws NamingException if unable to find group names
*/
-  private Set lookupGroup(SearchResult result, DirContext c,
+  Set lookupGroup(SearchResult result, DirContext c,

Review Comment:
   this has been made private so that you can subclass for testing, right?
   
   tag it as @VisibleForTesting



##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestLdapGroupsMappingWithOneQuery.java:
##
@@ -81,16 +86,72 @@ private void doTestGetGroups(List expectedGroups)
 // enable single-query lookup
 conf.set(LdapGroupsMapping.MEMBEROF_ATTR_KEY, "memberOf");
 
-LdapGroupsMapping groupsMapping = getGroupsMapping();
+TestLdapGroupsMapping groupsMapping = new TestLdapGroupsMapping();
 groupsMapping.setConf(conf);
 // 

[GitHub] [hadoop] steveloughran commented on a diff in pull request #4503: HADOOP-18074 - Partial/Incomplete groups list can be returned in LDAP…

[GitBox]

steveloughran commented on code in PR #4503:
URL: https://github.com/apache/hadoop/pull/4503#discussion_r907199594


##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestLdapGroupsMappingWithOneQuery.java:
##
@@ -81,16 +86,72 @@ private void doTestGetGroups(List expectedGroups)
 // enable single-query lookup
 conf.set(LdapGroupsMapping.MEMBEROF_ATTR_KEY, "memberOf");
 
-LdapGroupsMapping groupsMapping = getGroupsMapping();
+TestLdapGroupsMapping groupsMapping = new TestLdapGroupsMapping();
 groupsMapping.setConf(conf);
 // Username is arbitrary, since the spy is mocked to respond the same,
 // regardless of input
 List groups = groupsMapping.getGroups("some_user");
 
 Assert.assertEquals(expectedGroups, groups);
+Assert.assertFalse(groupsMapping.isSecondaryQueryCalled());

Review Comment:
   add an error message for the assertion failure



##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestLdapGroupsMappingWithOneQuery.java:
##
@@ -81,16 +86,72 @@ private void doTestGetGroups(List expectedGroups)
 // enable single-query lookup
 conf.set(LdapGroupsMapping.MEMBEROF_ATTR_KEY, "memberOf");
 
-LdapGroupsMapping groupsMapping = getGroupsMapping();
+TestLdapGroupsMapping groupsMapping = new TestLdapGroupsMapping();
 groupsMapping.setConf(conf);
 // Username is arbitrary, since the spy is mocked to respond the same,
 // regardless of input
 List groups = groupsMapping.getGroups("some_user");
 
 Assert.assertEquals(expectedGroups, groups);
+Assert.assertFalse(groupsMapping.isSecondaryQueryCalled());
 
 // We should have only made one query because single-query lookup is 
enabled
 verify(getContext(), times(1)).search(anyString(), anyString(),
 any(Object[].class), any(SearchControls.class));
   }
-}
\ No newline at end of file
+
+  private void doTestGetGroupsWithFallback()
+  throws NamingException {
+Attribute groupDN = mock(Attribute.class);
+
+NamingEnumeration groupNames = getGroupNames();
+doReturn(groupNames).when(groupDN).getAll();
+String groupName1 = "CN=abc,DC=foo,DC=bar,DC=com";
+String groupName2 = "CN=xyz,DC=foo,DC=bar,DC=com";
+String groupName3 = "ipaUniqueID=e4a9a634-bb24-11ec-aec1-06ede52b5fe1," +
+"CN=sudo,DC=foo,DC=bar,DC=com";
+doReturn(groupName1).doReturn(groupName2).doReturn(groupName3).
+when(groupNames).next();
+when(groupNames.hasMore()).thenReturn(true).thenReturn(true).
+thenReturn(true).thenReturn(false);
+
+when(getAttributes().get(eq("memberOf"))).thenReturn(groupDN);
+
+String ldapUrl = "ldap://test;;
+Configuration conf = getBaseConf(ldapUrl);
+// enable single-query lookup
+conf.set(LdapGroupsMapping.MEMBEROF_ATTR_KEY, "memberOf");
+conf.set(LdapGroupsMapping.LDAP_NUM_ATTEMPTS_KEY, "1");
+
+TestLdapGroupsMapping groupsMapping = new TestLdapGroupsMapping();
+groupsMapping.setConf(conf);
+// Username is arbitrary, since the spy is mocked to respond the same,
+// regardless of input
+List groups = groupsMapping.getGroups("some_user");
+
+// expected to be empty due to invalid memberOf
+Assert.assertEquals(0, groups.size());
+
+// expect secondary query to be called: getGroups()
+Assert.assertTrue(groupsMapping.isSecondaryQueryCalled());

Review Comment:
   add an error message



##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/LdapGroupsMapping.java:
##
@@ -428,7 +428,7 @@ private NamingEnumeration 
lookupPosixGroup(SearchResult result,
* @return a list of strings representing group names of the user.
* @throws NamingException if unable to find group names
*/
-  private Set lookupGroup(SearchResult result, DirContext c,
+  Set lookupGroup(SearchResult result, DirContext c,

Review Comment:
   this has been made private so that you can subclass for testing, right?
   
   tag it as @VisibleForTesting



##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestLdapGroupsMappingWithOneQuery.java:
##
@@ -81,16 +86,72 @@ private void doTestGetGroups(List expectedGroups)
 // enable single-query lookup
 conf.set(LdapGroupsMapping.MEMBEROF_ATTR_KEY, "memberOf");
 
-LdapGroupsMapping groupsMapping = getGroupsMapping();
+TestLdapGroupsMapping groupsMapping = new TestLdapGroupsMapping();
 groupsMapping.setConf(conf);
 // Username is arbitrary, since the spy is mocked to respond the same,
 // regardless of input
 List groups = groupsMapping.getGroups("some_user");
 
 Assert.assertEquals(expectedGroups, groups);
+Assert.assertFalse(groupsMapping.isSecondaryQueryCalled());
 
 // We should have only made one query because single-query lookup is 
enabled
 verify(getContext(), times(1)).search(anyString(), anyString(),
 

[jira] [Work logged] (HADOOP-18074) Partial/Incomplete groups list can be returned in LDAP groups lookup

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18074?focusedWorklogId=785016=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-785016
 ]

ASF GitHub Bot logged work on HADOOP-18074:
---

Author: ASF GitHub Bot
Created on: 27/Jun/22 10:01
Start Date: 27/Jun/22 10:01
Worklog Time Spent: 10m 
  Work Description: steveloughran commented on PR #4503:
URL: https://github.com/apache/hadoop/pull/4503#issuecomment-1167148712

   production code looks good, commented on the tests with my main concern 
being "mockito test maintenance is painful, so we should help whoever has to do 
it as much as we can"
   
   +1 pending those changes for merging into trunk/branch-3.3, to see if any 
surprises happen this week. i will merge it in to the 3.3.4 release next week




Issue Time Tracking
---

Worklog Id: (was: 785016)
Time Spent: 40m  (was: 0.5h)

> Partial/Incomplete groups list can be returned in LDAP groups lookup
> 
>
> Key: HADOOP-18074
> URL: https://issues.apache.org/jira/browse/HADOOP-18074
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Reporter: Philippe Lanoe
>Assignee: Larry McCay
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Hello,
> The  
> {code:java}
> Set doGetGroups(String user, int goUpHierarchy) {code}
> method in
> [https://github.com/apache/hadoop/blob/b27732c69b114f24358992a5a4d170bc94e2ceaf/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/LdapGroupsMapping.java#L476]
> Looks like having an issue if in the middle of the loop a *NamingException* 
> is caught:
> The groups variable is not reset in the catch clause and therefore the 
> fallback lookup cannot be executed (when goUpHierarchy==0 at least):
> ||
> {code:java}
> if (groups.isEmpty() || goUpHierarchy > 0) {
> groups = lookupGroup(result, c, goUpHierarchy);
> }
> {code}
>  
> Consequence is that only a partial list of groups is returned, which is not 
> correct.
> Following options could be used as solution:
>  * Reset the group to an empty list in the catch clause, to trigger the 
> fallback query.
>  * Add an option flag to enable ignoring groups with Naming Exception (since 
> they are not groups most probably)
> Independently, would any issue also occur (and therefore full list cannot be 
> returned) in the first lookup as well as in the fallback query, the method 
> should/could(with option flag) throw an Exception, because in some scenario 
> accuracy is important.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on pull request #4503: HADOOP-18074 - Partial/Incomplete groups list can be returned in LDAP…

[GitBox]

steveloughran commented on PR #4503:
URL: https://github.com/apache/hadoop/pull/4503#issuecomment-1167148712

   production code looks good, commented on the tests with my main concern 
being "mockito test maintenance is painful, so we should help whoever has to do 
it as much as we can"
   
   +1 pending those changes for merging into trunk/branch-3.3, to see if any 
surprises happen this week. i will merge it in to the 3.3.4 release next week


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on a diff in pull request #4354: YARN-6539. Create SecureLogin inside Router.

[GitBox]

steveloughran commented on code in PR #4354:
URL: https://github.com/apache/hadoop/pull/4354#discussion_r907191547


##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/security/TestSecureLogin.java:
##
@@ -0,0 +1,96 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership.  The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations 
under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.router.security;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeysPublic;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.UserGroupInformation.AuthenticationMethod;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.server.router.Router;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+
+import org.apache.hadoop.minikdc.MiniKdc;
+import org.junit.Test;
+import org.apache.hadoop.security.authentication.KerberosTestUtils;
+
+import java.io.File;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+public class TestSecureLogin {
+
+  private static final File TEST_ROOT_DIR = new File("target",
+  TestSecureLogin.class.getName() + "-root");
+  private static File routerKeytabFile = new File(
+  KerberosTestUtils.getKeytabFile());
+
+  private static MiniKdc testMiniKDC;
+
+  @BeforeClass
+  public static void setUp() {
+try {
+  testMiniKDC = new MiniKdc(MiniKdc.createConf(), TEST_ROOT_DIR);
+  testMiniKDC.start();
+  testMiniKDC.createPrincipal(routerKeytabFile, "yarn/localhost");
+} catch (Exception e) {
+  fail("Couldn't setup MiniKDC");
+}
+  }
+
+  @Test
+  public void testRouterSecureLogin() {
+Router router = null;
+try {
+  Configuration conf = new YarnConfiguration();
+  conf.set(YarnConfiguration.ROUTER_BIND_HOST, "0.0.0.0");
+  conf.set(YarnConfiguration.ROUTER_CLIENTRM_INTERCEPTOR_CLASS_PIPELINE,
+  
"org.apache.hadoop.yarn.server.router.clientrm.FederationClientInterceptor");
+  conf.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION,
+  "kerberos");
+  conf.set("yarn.router.principal", "yarn/localh...@example.com");
+  conf.set("yarn.router.keytab", routerKeytabFile.getAbsolutePath());
+  assertEquals(AuthenticationMethod.SIMPLE,
+  UserGroupInformation.getLoginUser().getAuthenticationMethod());
+  UserGroupInformation.setConfiguration(conf);
+  router = new Router();
+  router.init(conf);
+  router.start();
+  assertEquals(AuthenticationMethod.KERBEROS,
+  UserGroupInformation.getLoginUser().getAuthenticationMethod());
+  assertEquals("yarn/localh...@example.com",
+  UserGroupInformation.getLoginUser().getUserName());
+} catch (Throwable t) {
+  fail("Can't start router!");

Review Comment:
   this will also lose all error messages from any raised assertions. remove 
the try/catch and just declare the test case as throwing a Throwable



##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/security/TestSecureLogin.java:
##
@@ -0,0 +1,96 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership.  The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations 
under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.router.security;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeysPublic;
+import 

[GitHub] [hadoop] steveloughran commented on pull request #4354: YARN-6539. Create SecureLogin inside Router.

[GitBox]

steveloughran commented on PR #4354:
URL: https://github.com/apache/hadoop/pull/4354#issuecomment-1167132717

   > @goiri @steveloughran How about this PR merge to trunk?
   
   i don't go near YARN. better that way for all.
   
   asking me to review things there is counterproductive, because at best all 
that will happen is that I will request changes of the test code, so adding 
extra homework. once you've done that you will still need a YARN expert to 
review the production code


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] cxzl25 commented on pull request #4480: HDFS-16638. Add isDebugEnabled check for debug blockLogs in BlockManager

[GitBox]

cxzl25 commented on PR #4480:
URL: https://github.com/apache/hadoop/pull/4480#issuecomment-1167042941

   Thank you for your help with the stress test. @ZanderXu 
   It is helpful to improve performance for string concat where the `toString` 
cost of some objects is relatively high.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14309) Add PowerShell NodeFencer

[JiangHua Zhu (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17559072#comment-17559072
 ] 

JiangHua Zhu commented on HADOOP-14309:
---

Nice to talk to you, [~elgoiri].
I found that HDFSHighAvailabilityWithQJM.md is not updated.
I will create a new jira if needed.

> Add PowerShell NodeFencer
> -
>
> Key: HADOOP-14309
> URL: https://issues.apache.org/jira/browse/HADOOP-14309
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: ha
>Affects Versions: 2.8.0
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Minor
> Fix For: 3.0.0-alpha4
>
> Attachments: HADOOP-14309-000.patch, HADOOP-14309-001.patch
>
>
> Fencer implementation that uses PowerShell to fence a remote component.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-18313) AliyunOSS: AliyunOSSBlockOutputStream should not mark the temporary file for deletion

[wujinhu (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-18313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17559032#comment-17559032
 ] 

wujinhu commented on HADOOP-18313:
--

Thanks [~ste...@apache.org]  It's a good suggestion for our OSS users. Flink 
users depend on hadoop-aliyun module to write to OSS. Their workers run OOM 
because workers do not exit. This fix can resolve the issue.

> AliyunOSS: AliyunOSSBlockOutputStream should not mark the temporary file for 
> deletion
> -
>
> Key: HADOOP-18313
> URL: https://issues.apache.org/jira/browse/HADOOP-18313
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/oss
>Affects Versions: 2.10.2, 3.2.3, 3.3.3
>Reporter: wujinhu
>Assignee: wujinhu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> AliyunOSSBlockOutputStream buffers data in local directory before uploading 
> to OSS. It uses LocalDirAllocator.createTmpFileForWrite which will create a 
> temp file on this directory and invoke DeleteOnExitHook.add to add the path 
> to a set and will be deleted on VM exit through a shutdown hook. However, the 
> size of set will be growing if the VM does not exit.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] zhengchenyu commented on pull request #4354: YARN-6539. Create SecureLogin inside Router.

[GitBox]

zhengchenyu commented on PR #4354:
URL: https://github.com/apache/hadoop/pull/4354#issuecomment-1166939502

   @goiri @steveloughran How about this PR merge to trunk?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] ZanderXu commented on pull request #4480: HDFS-16638. Add isDebugEnabled check for debug blockLogs in BlockManager

[GitBox]

ZanderXu commented on PR #4480:
URL: https://github.com/apache/hadoop/pull/4480#issuecomment-1166936626

   @cxzl25 @virajjasani @ayushtkn Share the results of my stress test here.
   
   Test code:
   ```
   org.slf4j.Logger LOG = LoggerFactory.getLogger("Test");
   Block b = new Block();
   DatanodeInfo dn = new DatanodeInfo(EMPTY_DATANODE_ID);
   List staleNodes = new ArrayList<>();
   staleNodes.add(dn);
   long beginTime = Time.monotonicNowNanos();
   for (int i = 0; i < 1; i++) {
 LOG.debug("BLOCK* invalidateBlocks: postponing " +
 "invalidation of {} on {} because {} replica(s) are located on 
" +
 "nodes with potentially out-of-date block reports",
 b, dn, staleNodes);
   }
   long endTime = Time.monotonicNowNanos();
   LOG.info("Time1: {}", endTime - beginTime);
   
   long beginTime1 = Time.monotonicNowNanos();
   for (int i = 0; i < 1; i++) {
 if (LOG.isDebugEnabled()) {
   LOG.debug("BLOCK* invalidateBlocks: postponing " +
   "invalidation of {} on {} because {} replica(s) are located 
on " +
   "nodes with potentially out-of-date block reports",
   b, dn, staleNodes);
 }
   }
   long endTime2 = Time.monotonicNowNanos();
   LOG.info("Time2: {}", endTime2 - beginTime1);
   
   long beginTime2 = Time.monotonicNowNanos();
   for (int i = 0; i < 1; i++) {
 LOG.debug("BLOCK* invalidateBlocks: postponing invalidation of {}", b);
   }
   long endTime3 = Time.monotonicNowNanos();
   LOG.info("Time3: {}", endTime3 - beginTime2);
   ```
   
   And the result is:
   ```
   Time1: 353681417
   Time2: 287066916
   Time3: 270287253
   ```
   
   So this PR is help with much performance improvement.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org