[GitHub] [hadoop] hadoop-yetus commented on pull request #3414: YARN-10938. Support reservation scheduling enabled switch for capacity scheduler
hadoop-yetus commented on pull request #3414:
URL: https://github.com/apache/hadoop/pull/3414#issuecomment-916642135
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 42s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 19s | | trunk passed |
| +1 :green_heart: | compile | 1m 2s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 54s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 52s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 0s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 43s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 54s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 10s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 51s | | the patch passed |
| +1 :green_heart: | compile | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 53s | | the patch passed |
| +1 :green_heart: | compile | 0m 47s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 47s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 42s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3414/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 580 unchanged - 0 fixed = 581 total (was 580) |
| +1 :green_heart: | mvnsite | 0m 50s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 37s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 44s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 95m 46s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 173m 21s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3414/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3414 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 0d406cffdb3c 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28
05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bd8ae06c7ea073ccf5a77e9c4e05ac130fce4178 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3414/2/testReport/ |
| Max. process+thread count | 1001 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
[GitHub] [hadoop] tomscut edited a comment on pull request #3366: HDFS-16203. Discover datanodes with unbalanced block pool usage by th…
tomscut edited a comment on pull request #3366:
URL: https://github.com/apache/hadoop/pull/3366#issuecomment-916639192
@aajisaka @ayushtkn @ferhui @Hexiaoqiao Could you please review the code?
Thanks a lot.
The standard deviation is calculated by referring to
FSNamesystem#getNodeUsage. The key change is
FSNamesystem#getBlockPoolUsedPercentStdDev, :
```
public float getBlockPoolUsedPercentStdDev(StorageReport[] storageReports)
{
ArrayList usagePercentList = new ArrayList<>();
float totalUsagePercent = 0.0f;
float dev = 0.0f;
if (storageReports.length == 0) {
return dev;
}
for (StorageReport s : storageReports) {
usagePercentList.add(s.getBlockPoolUsagePercent());
totalUsagePercent += s.getBlockPoolUsagePercent();
}
totalUsagePercent /= storageReports.length;
Collections.sort(usagePercentList);
for (Float usagePercent : usagePercentList) {
dev += (usagePercent - totalUsagePercent)
* (usagePercent - totalUsagePercent);
}
dev = (float) Math.sqrt(dev / usagePercentList.size());
return dev;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on pull request #3366: HDFS-16203. Discover datanodes with unbalanced block pool usage by th…
tomscut commented on pull request #3366:
URL: https://github.com/apache/hadoop/pull/3366#issuecomment-916639192
@ayushtkn @ferhui @Hexiaoqiao Could you please review the code? Thanks a lot.
The standard deviation is calculated by referring to
FSNamesystem#getNodeUsage. The key change is
FSNamesystem#getBlockPoolUsedPercentStdDev, :
```
public float getBlockPoolUsedPercentStdDev(StorageReport[] storageReports)
{
ArrayList usagePercentList = new ArrayList<>();
float totalUsagePercent = 0.0f;
float dev = 0.0f;
if (storageReports.length == 0) {
return dev;
}
for (StorageReport s : storageReports) {
usagePercentList.add(s.getBlockPoolUsagePercent());
totalUsagePercent += s.getBlockPoolUsagePercent();
}
totalUsagePercent /= storageReports.length;
Collections.sort(usagePercentList);
for (Float usagePercent : usagePercentList) {
dev += (usagePercent - totalUsagePercent)
* (usagePercent - totalUsagePercent);
}
dev = (float) Math.sqrt(dev / usagePercentList.size());
return dev;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bshashikant commented on pull request #3340: HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing
bshashikant commented on pull request #3340: URL: https://github.com/apache/hadoop/pull/3340#issuecomment-916626429 > > @bshashikant , thanks! In that case, let's please just add the comments I requested to `INodeDirectory` and `INodeDirectoryAttributes`. Then I'll approve and merge. > > Thanks @cnauroth . As the problem is very specific to snapshot root, "metedataEquals()" function is now overriden in Snapshot.Root() and added the comments as per your suggestion @szetszwo , can you have another look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bshashikant commented on pull request #3340: HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing
bshashikant commented on pull request #3340: URL: https://github.com/apache/hadoop/pull/3340#issuecomment-916626334 > @bshashikant , thanks! In that case, let's please just add the comments I requested to `INodeDirectory` and `INodeDirectoryAttributes`. Then I'll approve and merge. Thanks @cnauroth . As the problem is very specific to snapshot root, "metedataEquals()" function is now overriden in Snapshot.Root() and added the comments as per your suggestion -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17902) Fix Hadoop build on Debian 10
[
https://issues.apache.org/jira/browse/HADOOP-17902?focusedWorklogId=649024=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-649024
]
ASF GitHub Bot logged work on HADOOP-17902:
---
Author: ASF GitHub Bot
Created on: 10/Sep/21 04:36
Start Date: 10/Sep/21 04:36
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3408:
URL: https://github.com/apache/hadoop/pull/3408#issuecomment-916624453
(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/3/console in
case of problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 649024)
Time Spent: 2.5h (was: 2h 20m)
> Fix Hadoop build on Debian 10
> -
>
> Key: HADOOP-17902
> URL: https://issues.apache.org/jira/browse/HADOOP-17902
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> We're using *Debian testing* as one of the package sources to get the latest
> packages. It seems to be broken at the moment. The CI fails to create the
> build environment for the Debian 10 platform -
> {code}
> [2021-09-08T00:21:11.596Z] #13 [ 8/14] RUN apt-get -q update && apt-get
> -q install -y --no-install-recommends python3 && apt-get -q install -y
> --no-install-recommends $(pkg-resolver/resolve.py debian:10) && apt-get
> clean && rm -rf /var/lib/apt/lists/*
> ...
> [2021-09-08T00:21:22.744Z] #13 11.28 Preparing to unpack
> .../libc6_2.31-17_amd64.deb ...
> [2021-09-08T00:21:23.260Z] #13 11.46 Checking for services that may need to
> be restarted...
> [2021-09-08T00:21:23.260Z] #13 11.48 Checking init scripts...
> [2021-09-08T00:21:23.260Z] #13 11.50 Unpacking libc6:amd64 (2.31-17) over
> (2.28-10) ...
> [2021-09-08T00:21:26.290Z] #13 14.38 Setting up libc6:amd64 (2.31-17) ...
> [2021-09-08T00:21:26.290Z] #13 14.42 /usr/bin/perl: error while loading
> shared libraries: libcrypt.so.1: cannot open shared object file: No such file
> or directory
> [2021-09-08T00:21:26.290Z] #13 14.42 dpkg: error processing package
> libc6:amd64 (--configure):
> [2021-09-08T00:21:26.290Z] #13 14.42 installed libc6:amd64 package
> post-installation script subprocess returned error exit status 127
> [2021-09-08T00:21:26.291Z] #13 14.43 Errors were encountered while processing:
> [2021-09-08T00:21:26.291Z] #13 14.43 libc6:amd64
> [2021-09-08T00:21:26.291Z] #13 14.46 E: Sub-process /usr/bin/dpkg returned an
> error code (1)
> [2021-09-08T00:21:27.867Z] #13 ERROR: executor failed running [/bin/bash -o
> pipefail -c apt-get -q update && apt-get -q install -y
> --no-install-recommends python3 && apt-get -q install -y
> --no-install-recommends $(pkg-resolver/resolve.py debian:10) && apt-get
> clean && rm -rf /var/lib/apt/lists/*]: exit code: 100
> [2021-09-08T00:21:27.867Z] --
> [2021-09-08T00:21:27.867Z] > [ 8/14] RUN apt-get -q update && apt-get -q
> install -y --no-install-recommends python3 && apt-get -q install -y
> --no-install-recommends $(pkg-resolver/resolve.py debian:10) && apt-get
> clean && rm -rf /var/lib/apt/lists/*:
> [2021-09-08T00:21:27.867Z] --
> [2021-09-08T00:21:27.867Z] executor failed running [/bin/bash -o pipefail -c
> apt-get -q update && apt-get -q install -y --no-install-recommends
> python3 && apt-get -q install -y --no-install-recommends
> $(pkg-resolver/resolve.py debian:10) && apt-get clean && rm -rf
> /var/lib/apt/lists/*]: exit code: 100
> [2021-09-08T00:21:27.867Z] ERROR: Docker failed to build
> yetus/hadoop:ef5dbc7283a.
> [2021-09-08T00:21:27.867Z]
> {code}
> The above log lines are copied from -
> https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3388/3/pipeline
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3408: HADOOP-17902. Fix Hadoop build on Debian 10
hadoop-yetus commented on pull request #3408: URL: https://github.com/apache/hadoop/pull/3408#issuecomment-916624453 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/3/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17902) Fix Hadoop build on Debian 10
[
https://issues.apache.org/jira/browse/HADOOP-17902?focusedWorklogId=649023=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-649023
]
ASF GitHub Bot logged work on HADOOP-17902:
---
Author: ASF GitHub Bot
Created on: 10/Sep/21 04:33
Start Date: 10/Sep/21 04:33
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3408:
URL: https://github.com/apache/hadoop/pull/3408#issuecomment-916623612
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 22m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 53s | | trunk passed |
| +1 :green_heart: | compile | 3m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 27s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 27s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 3m 10s | | the patch passed |
| +1 :green_heart: | cc | 3m 10s | | the patch passed |
| +1 :green_heart: | golang | 3m 10s | | the patch passed |
| +1 :green_heart: | javac | 3m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | hadolint | 0m 3s | | No new issues. |
| +1 :green_heart: | mvnsite | 0m 23s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | shadedclient | 25m 18s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 85m 45s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 207m 43s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3408 |
| Optional Tests | dupname asflicense codespell hadolint shellcheck
shelldocs compile cc mvnsite javac unit golang |
| uname | Linux 748484b048f5 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18
19:21:19 UTC 2021 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7611377859519eb33b4e7e2b3832479b715b666e |
| Default Java | Debian-11.0.12+7-post-Debian-2deb10u1 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/3/testReport/ |
| Max. process+thread count | 610 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/3/console |
| versions | git=2.20.1 maven=3.6.0 shellcheck=0.5.0
hadolint=1.11.1-0-g0e692dd |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 649023)
Time Spent: 2h 20m (was: 2h 10m)
> Fix Hadoop build on Debian 10
> -
>
> Key: HADOOP-17902
> URL: https://issues.apache.org/jira/browse/HADOOP-17902
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> We're using
[GitHub] [hadoop] hadoop-yetus commented on pull request #3408: HADOOP-17902. Fix Hadoop build on Debian 10
hadoop-yetus commented on pull request #3408:
URL: https://github.com/apache/hadoop/pull/3408#issuecomment-916623612
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 22m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 53s | | trunk passed |
| +1 :green_heart: | compile | 3m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 27s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 27s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 3m 10s | | the patch passed |
| +1 :green_heart: | cc | 3m 10s | | the patch passed |
| +1 :green_heart: | golang | 3m 10s | | the patch passed |
| +1 :green_heart: | javac | 3m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | hadolint | 0m 3s | | No new issues. |
| +1 :green_heart: | mvnsite | 0m 23s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | shadedclient | 25m 18s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 85m 45s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 207m 43s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3408 |
| Optional Tests | dupname asflicense codespell hadolint shellcheck
shelldocs compile cc mvnsite javac unit golang |
| uname | Linux 748484b048f5 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18
19:21:19 UTC 2021 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7611377859519eb33b4e7e2b3832479b715b666e |
| Default Java | Debian-11.0.12+7-post-Debian-2deb10u1 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/3/testReport/ |
| Max. process+thread count | 610 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/3/console |
| versions | git=2.20.1 maven=3.6.0 shellcheck=0.5.0
hadolint=1.11.1-0-g0e692dd |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3415: HDFS-16216. RBF: Wrong path when get mount point status
hadoop-yetus commented on pull request #3415:
URL: https://github.com/apache/hadoop/pull/3415#issuecomment-916613934
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 43s | | trunk passed |
| +1 :green_heart: | compile | 0m 48s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 42s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 1s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 13s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 19s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3415/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 2
unchanged - 0 fixed = 3 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 57s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 21m 29s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 104m 31s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3415/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3415 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 06a972f6a1f4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 697004f46d57eb90cfb0d32f1ca35ef9e64e26fa |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3415/1/testReport/ |
| Max. process+thread count | 2734 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3415/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the
[GitHub] [hadoop] cnauroth commented on pull request #3340: HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing
cnauroth commented on pull request #3340: URL: https://github.com/apache/hadoop/pull/3340#issuecomment-916609296 @bshashikant , thanks! In that case, let's please just add the comments I requested to `INodeDirectory` and `INodeDirectoryAttributes`. Then I'll approve and merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bshashikant edited a comment on pull request #3340: HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing
bshashikant edited a comment on pull request #3340:
URL: https://github.com/apache/hadoop/pull/3340#issuecomment-916599655
Thanks @cnauroth . The problem here seems to be very specific to
snapshotRoot.
`
@Test
public void testXattrOnFileWithSnapshotAndNNRestart() throws Exception {
FileSystem.mkdirs(hdfs, path, FsPermission.createImmutable((short)
0700));
Path file = new Path(path, "file1");
FileSystem.create(hdfs, file, FsPermission.createImmutable((short)
0));
hdfs.setXAttr(file, name1, value1);
hdfs.allowSnapshot(path);
hdfs.createSnapshot(path, snapshotName);
SnapshotDiffReport report =
hdfs.getSnapshotDiffReport(path, snapshotName, "");
System.out.println(report);
Assert.assertEquals(0, report.getDiffList().size());
report =
hdfs.getSnapshotDiffReport(path, snapshotName, "");
System.out.println(report);
Assert.assertEquals(0, report.getDiffList().size());
hdfs.setSafeMode(HdfsConstants.SafeModeAction.SAFEMODE_ENTER);
hdfs.saveNamespace();
hdfs.setSafeMode(HdfsConstants.SafeModeAction.SAFEMODE_LEAVE);
cluster.restartNameNode(true);
report =
hdfs.getSnapshotDiffReport(path, snapshotName, "");
System.out.println(report);
Assert.assertEquals(0, report.getDiffList().size());
}`
The same test with xattrs set on file inside the snapshotRoot/SubDirecory
will just work with/without the patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bshashikant edited a comment on pull request #3340: HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing
bshashikant edited a comment on pull request #3340:
URL: https://github.com/apache/hadoop/pull/3340#issuecomment-916599655
Thanks @cnauroth . The problem here is very specific to snapshotRoot.
`@Test
public void testXattrOnFileWithSnapshotAndNNRestart() throws Exception {
// Init
FileSystem.mkdirs(hdfs, path, FsPermission.createImmutable((short)
0700));
Path file = new Path(path, "file1");
FileSystem.create(hdfs, file, FsPermission.createImmutable((short)
0));
hdfs.setXAttr(file, name1, value1);
hdfs.allowSnapshot(path);
hdfs.createSnapshot(path, snapshotName);
SnapshotDiffReport report =
hdfs.getSnapshotDiffReport(path, snapshotName, "");
System.out.println(report);
Assert.assertEquals(0, report.getDiffList().size());
report =
hdfs.getSnapshotDiffReport(path, snapshotName, "");
System.out.println(report);
Assert.assertEquals(0, report.getDiffList().size());
hdfs.setSafeMode(HdfsConstants.SafeModeAction.SAFEMODE_ENTER);
hdfs.saveNamespace();
hdfs.setSafeMode(HdfsConstants.SafeModeAction.SAFEMODE_LEAVE);
cluster.restartNameNode(true);
report =
hdfs.getSnapshotDiffReport(path, snapshotName, "");
System.out.println(report);
Assert.assertEquals(0, report.getDiffList().size());
}`
The same test with xattrs set on file inside the snapshotRoot/SubDirecory
will just work with/without the patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bshashikant commented on pull request #3340: HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing
bshashikant commented on pull request #3340:
URL: https://github.com/apache/hadoop/pull/3340#issuecomment-916599655
Thanks @cnauroth . The problem here is very specific to snapshotRoot.
`@Test
public void testXattrOnFileWithSnapshotAndNNRestart() throws Exception {
// Init
FileSystem.mkdirs(hdfs, path, FsPermission.createImmutable((short)
0700));
Path file = new Path(path, "file1");
FileSystem.create(hdfs, file, FsPermission.createImmutable((short)
0));
hdfs.setXAttr(file, name1, value1);
hdfs.allowSnapshot(path);
hdfs.createSnapshot(path, snapshotName);
SnapshotDiffReport report =
hdfs.getSnapshotDiffReport(path, snapshotName, "");
System.out.println(report);
Assert.assertEquals(0, report.getDiffList().size());
report =
hdfs.getSnapshotDiffReport(path, snapshotName, "");
System.out.println(report);
Assert.assertEquals(0, report.getDiffList().size());
hdfs.setSafeMode(HdfsConstants.SafeModeAction.SAFEMODE_ENTER);
hdfs.saveNamespace();
hdfs.setSafeMode(HdfsConstants.SafeModeAction.SAFEMODE_LEAVE);
cluster.restartNameNode(true);
report =
hdfs.getSnapshotDiffReport(path, snapshotName, "");
System.out.println(report);
Assert.assertEquals(0, report.getDiffList().size());
}`
The same test with xattrs set on file inside the snapshotRoot/SubDirecory
will just work with/without the patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Hexiaoqiao commented on pull request #3200: URL: https://github.com/apache/hadoop/pull/3200#issuecomment-916596838 Will commit to branch-3.2 for a short while if no more comments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jianghuazhu commented on pull request #3357: HDFS-16197.Simplify getting NNStorage in FSNamesystem.
jianghuazhu commented on pull request #3357: URL: https://github.com/apache/hadoop/pull/3357#issuecomment-916594491 @ferhui @virajjasani , can you help me check the code? If there is something to be corrected, I will continue to work. Thank you very much. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17428) ABFS: Implementation for getContentSummary
[
https://issues.apache.org/jira/browse/HADOOP-17428?focusedWorklogId=648988=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648988
]
ASF GitHub Bot logged work on HADOOP-17428:
---
Author: ASF GitHub Bot
Created on: 10/Sep/21 02:25
Start Date: 10/Sep/21 02:25
Worklog Time Spent: 10m
Work Description: sumangala-patki commented on a change in pull request
#2549:
URL: https://github.com/apache/hadoop/pull/2549#discussion_r705849850
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ContentSummaryProcessor.java
##
@@ -0,0 +1,147 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.util.concurrent.CompletionService;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.SynchronousQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.ContentSummary;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+
+public class ContentSummaryProcessor {
+ private static final int CORE_POOL_SIZE = 1;
+ private static final int MAX_THREAD_COUNT = 16;
+ private static final int KEEP_ALIVE_TIME = 5;
+ private static final int POLL_TIMEOUT = 100;
+ private static final Logger LOG =
LoggerFactory.getLogger(ContentSummaryProcessor.class);
+ private final AtomicLong fileCount = new AtomicLong(0L);
+ private final AtomicLong directoryCount = new AtomicLong(0L);
+ private final AtomicLong totalBytes = new AtomicLong(0L);
+ private final AtomicInteger numTasks = new AtomicInteger(0);
+ private final ListingSupport abfsStore;
+ private final ExecutorService executorService = new ThreadPoolExecutor(
+ CORE_POOL_SIZE, MAX_THREAD_COUNT, KEEP_ALIVE_TIME, TimeUnit.SECONDS,
+ new SynchronousQueue<>());
+ private final CompletionService completionService =
+ new ExecutorCompletionService<>(executorService);
+ private final LinkedBlockingQueue queue = new
LinkedBlockingQueue<>();
+
+ /**
+ * Processes a given path for count of subdirectories, files and total number
+ * of bytes
+ * @param abfsStore Instance of AzureBlobFileSystemStore, used to make
+ * listStatus calls to server
+ */
+ public ContentSummaryProcessor(ListingSupport abfsStore) {
+this.abfsStore = abfsStore;
+ }
+
+ public ContentSummary getContentSummary(Path path, TracingContext
tracingContext)
+ throws IOException, ExecutionException, InterruptedException {
+try {
+ processDirectoryTree(path, tracingContext);
+ while (!queue.isEmpty() || numTasks.get() > 0) {
+try {
+ completionService.take().get();
+} finally {
+ numTasks.decrementAndGet();
+ LOG.debug("FileStatus queue size = {}, number of submitted
unfinished tasks = {}, active thread count = {}",
+ queue.size(), numTasks, ((ThreadPoolExecutor)
executorService).getActiveCount());
+}
+ }
+} finally {
+ executorService.shutdownNow();
+ LOG.debug("Executor shutdown");
+}
+LOG.debug("Processed content summary of subtree under given path");
+ContentSummary.Builder builder = new ContentSummary.Builder()
+.directoryCount(directoryCount.get()).fileCount(fileCount.get())
+.length(totalBytes.get()).spaceConsumed(totalBytes.get());
+return builder.build();
+ }
+
+ /**
+ * Calls listStatus on given path and populated fileStatus queue with
+ * subdirectories. Is called by new tasks to process the complete subtree
+ * under a given path
+ * @param path: Path to a file or directory
+ * @throws IOException: listStatus
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2549: HADOOP-17428. ABFS: Implementation for getContentSummary
sumangala-patki commented on a change in pull request #2549:
URL: https://github.com/apache/hadoop/pull/2549#discussion_r705849850
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ContentSummaryProcessor.java
##
@@ -0,0 +1,147 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.util.concurrent.CompletionService;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.SynchronousQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.ContentSummary;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+
+public class ContentSummaryProcessor {
+ private static final int CORE_POOL_SIZE = 1;
+ private static final int MAX_THREAD_COUNT = 16;
+ private static final int KEEP_ALIVE_TIME = 5;
+ private static final int POLL_TIMEOUT = 100;
+ private static final Logger LOG =
LoggerFactory.getLogger(ContentSummaryProcessor.class);
+ private final AtomicLong fileCount = new AtomicLong(0L);
+ private final AtomicLong directoryCount = new AtomicLong(0L);
+ private final AtomicLong totalBytes = new AtomicLong(0L);
+ private final AtomicInteger numTasks = new AtomicInteger(0);
+ private final ListingSupport abfsStore;
+ private final ExecutorService executorService = new ThreadPoolExecutor(
+ CORE_POOL_SIZE, MAX_THREAD_COUNT, KEEP_ALIVE_TIME, TimeUnit.SECONDS,
+ new SynchronousQueue<>());
+ private final CompletionService completionService =
+ new ExecutorCompletionService<>(executorService);
+ private final LinkedBlockingQueue queue = new
LinkedBlockingQueue<>();
+
+ /**
+ * Processes a given path for count of subdirectories, files and total number
+ * of bytes
+ * @param abfsStore Instance of AzureBlobFileSystemStore, used to make
+ * listStatus calls to server
+ */
+ public ContentSummaryProcessor(ListingSupport abfsStore) {
+this.abfsStore = abfsStore;
+ }
+
+ public ContentSummary getContentSummary(Path path, TracingContext
tracingContext)
+ throws IOException, ExecutionException, InterruptedException {
+try {
+ processDirectoryTree(path, tracingContext);
+ while (!queue.isEmpty() || numTasks.get() > 0) {
+try {
+ completionService.take().get();
+} finally {
+ numTasks.decrementAndGet();
+ LOG.debug("FileStatus queue size = {}, number of submitted
unfinished tasks = {}, active thread count = {}",
+ queue.size(), numTasks, ((ThreadPoolExecutor)
executorService).getActiveCount());
+}
+ }
+} finally {
+ executorService.shutdownNow();
+ LOG.debug("Executor shutdown");
+}
+LOG.debug("Processed content summary of subtree under given path");
+ContentSummary.Builder builder = new ContentSummary.Builder()
+.directoryCount(directoryCount.get()).fileCount(fileCount.get())
+.length(totalBytes.get()).spaceConsumed(totalBytes.get());
+return builder.build();
+ }
+
+ /**
+ * Calls listStatus on given path and populated fileStatus queue with
+ * subdirectories. Is called by new tasks to process the complete subtree
+ * under a given path
+ * @param path: Path to a file or directory
+ * @throws IOException: listStatus error
+ * @throws InterruptedException: error while inserting into queue
+ */
+ private void processDirectoryTree(Path path, TracingContext tracingContext)
+ throws IOException, InterruptedException {
+FileStatus[] fileStatuses = abfsStore.listStatus(path, tracingContext);
+
+for (FileStatus fileStatus : fileStatuses) {
Review comment:
Trying to confirm the advantage of processing page-wise listStatus
[GitHub] [hadoop] zhengzhuobinzzb opened a new pull request #3415: HDFS-16216. RBF: Wrong path when get mount point status
zhengzhuobinzzb opened a new pull request #3415: URL: https://github.com/apache/hadoop/pull/3415 ### Description of PR https://issues.apache.org/jira/browse/HDFS-16216 ### How was this patch tested? UT. TestRouterMountTable.testGetMountPointFileStatus ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648966=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648966
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 10/Sep/21 01:36
Start Date: 10/Sep/21 01:36
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-916364503
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648966)
Time Spent: 12h 10m (was: 12h)
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 12h 10m
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
hadoop-yetus removed a comment on pull request #3385: URL: https://github.com/apache/hadoop/pull/3385#issuecomment-916364503 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17902) Fix Hadoop build on Debian 10
[
https://issues.apache.org/jira/browse/HADOOP-17902?focusedWorklogId=648965=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648965
]
ASF GitHub Bot logged work on HADOOP-17902:
---
Author: ASF GitHub Bot
Created on: 10/Sep/21 01:07
Start Date: 10/Sep/21 01:07
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3408:
URL: https://github.com/apache/hadoop/pull/3408#issuecomment-916553282
(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/3/console in
case of problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648965)
Time Spent: 2h 10m (was: 2h)
> Fix Hadoop build on Debian 10
> -
>
> Key: HADOOP-17902
> URL: https://issues.apache.org/jira/browse/HADOOP-17902
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> We're using *Debian testing* as one of the package sources to get the latest
> packages. It seems to be broken at the moment. The CI fails to create the
> build environment for the Debian 10 platform -
> {code}
> [2021-09-08T00:21:11.596Z] #13 [ 8/14] RUN apt-get -q update && apt-get
> -q install -y --no-install-recommends python3 && apt-get -q install -y
> --no-install-recommends $(pkg-resolver/resolve.py debian:10) && apt-get
> clean && rm -rf /var/lib/apt/lists/*
> ...
> [2021-09-08T00:21:22.744Z] #13 11.28 Preparing to unpack
> .../libc6_2.31-17_amd64.deb ...
> [2021-09-08T00:21:23.260Z] #13 11.46 Checking for services that may need to
> be restarted...
> [2021-09-08T00:21:23.260Z] #13 11.48 Checking init scripts...
> [2021-09-08T00:21:23.260Z] #13 11.50 Unpacking libc6:amd64 (2.31-17) over
> (2.28-10) ...
> [2021-09-08T00:21:26.290Z] #13 14.38 Setting up libc6:amd64 (2.31-17) ...
> [2021-09-08T00:21:26.290Z] #13 14.42 /usr/bin/perl: error while loading
> shared libraries: libcrypt.so.1: cannot open shared object file: No such file
> or directory
> [2021-09-08T00:21:26.290Z] #13 14.42 dpkg: error processing package
> libc6:amd64 (--configure):
> [2021-09-08T00:21:26.290Z] #13 14.42 installed libc6:amd64 package
> post-installation script subprocess returned error exit status 127
> [2021-09-08T00:21:26.291Z] #13 14.43 Errors were encountered while processing:
> [2021-09-08T00:21:26.291Z] #13 14.43 libc6:amd64
> [2021-09-08T00:21:26.291Z] #13 14.46 E: Sub-process /usr/bin/dpkg returned an
> error code (1)
> [2021-09-08T00:21:27.867Z] #13 ERROR: executor failed running [/bin/bash -o
> pipefail -c apt-get -q update && apt-get -q install -y
> --no-install-recommends python3 && apt-get -q install -y
> --no-install-recommends $(pkg-resolver/resolve.py debian:10) && apt-get
> clean && rm -rf /var/lib/apt/lists/*]: exit code: 100
> [2021-09-08T00:21:27.867Z] --
> [2021-09-08T00:21:27.867Z] > [ 8/14] RUN apt-get -q update && apt-get -q
> install -y --no-install-recommends python3 && apt-get -q install -y
> --no-install-recommends $(pkg-resolver/resolve.py debian:10) && apt-get
> clean && rm -rf /var/lib/apt/lists/*:
> [2021-09-08T00:21:27.867Z] --
> [2021-09-08T00:21:27.867Z] executor failed running [/bin/bash -o pipefail -c
> apt-get -q update && apt-get -q install -y --no-install-recommends
> python3 && apt-get -q install -y --no-install-recommends
> $(pkg-resolver/resolve.py debian:10) && apt-get clean && rm -rf
> /var/lib/apt/lists/*]: exit code: 100
> [2021-09-08T00:21:27.867Z] ERROR: Docker failed to build
> yetus/hadoop:ef5dbc7283a.
> [2021-09-08T00:21:27.867Z]
> {code}
> The above log lines are copied from -
> https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3388/3/pipeline
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3408: HADOOP-17902. Fix Hadoop build on Debian 10
hadoop-yetus commented on pull request #3408: URL: https://github.com/apache/hadoop/pull/3408#issuecomment-916553282 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/3/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3408: HADOOP-17902. Fix Hadoop build on Debian 10
hadoop-yetus commented on pull request #3408:
URL: https://github.com/apache/hadoop/pull/3408#issuecomment-916552548
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 32m 11s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | hadolint | 0m 0s | | hadolint was not available. |
| +0 :ok: | shellcheck | 0m 0s | | Shellcheck was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 28m 10s | | trunk passed |
| +1 :green_heart: | compile | 2m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 33s | | trunk passed |
| +1 :green_heart: | shadedclient | 47m 58s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 2m 46s | | the patch passed |
| +1 :green_heart: | cc | 2m 46s | | the patch passed |
| +1 :green_heart: | golang | 2m 46s | | the patch passed |
| +1 :green_heart: | javac | 2m 46s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 24s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 10s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 96m 38s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 200m 35s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3408 |
| Optional Tests | dupname asflicense codespell hadolint shellcheck
shelldocs compile cc mvnsite javac unit golang |
| uname | Linux 0041fead20c4 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7611377859519eb33b4e7e2b3832479b715b666e |
| Default Java | Red Hat, Inc.-1.8.0_302-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/3/testReport/ |
| Max. process+thread count | 579 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/3/console |
| versions | git=2.27.0 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17902) Fix Hadoop build on Debian 10
[
https://issues.apache.org/jira/browse/HADOOP-17902?focusedWorklogId=648963=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648963
]
ASF GitHub Bot logged work on HADOOP-17902:
---
Author: ASF GitHub Bot
Created on: 10/Sep/21 01:05
Start Date: 10/Sep/21 01:05
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3408:
URL: https://github.com/apache/hadoop/pull/3408#issuecomment-916552548
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 32m 11s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | hadolint | 0m 0s | | hadolint was not available. |
| +0 :ok: | shellcheck | 0m 0s | | Shellcheck was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 28m 10s | | trunk passed |
| +1 :green_heart: | compile | 2m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 33s | | trunk passed |
| +1 :green_heart: | shadedclient | 47m 58s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 2m 46s | | the patch passed |
| +1 :green_heart: | cc | 2m 46s | | the patch passed |
| +1 :green_heart: | golang | 2m 46s | | the patch passed |
| +1 :green_heart: | javac | 2m 46s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 24s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 10s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 96m 38s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 200m 35s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3408 |
| Optional Tests | dupname asflicense codespell hadolint shellcheck
shelldocs compile cc mvnsite javac unit golang |
| uname | Linux 0041fead20c4 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7611377859519eb33b4e7e2b3832479b715b666e |
| Default Java | Red Hat, Inc.-1.8.0_302-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/3/testReport/ |
| Max. process+thread count | 579 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/3/console |
| versions | git=2.27.0 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648963)
Time Spent: 2h (was: 1h 50m)
> Fix Hadoop build on Debian 10
> -
>
> Key: HADOOP-17902
> URL: https://issues.apache.org/jira/browse/HADOOP-17902
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> We're using *Debian testing* as one of the package
[GitHub] [hadoop] hadoop-yetus commented on pull request #2971: MAPREDUCE-7341. Intermediate Manifest Committer
hadoop-yetus commented on pull request #2971:
URL: https://github.com/apache/hadoop/pull/2971#issuecomment-916548752
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 10s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 25 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 37s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 0s | | trunk passed |
| +1 :green_heart: | compile | 22m 49s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 26s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 54s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 39s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 6s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 31s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 17m 13s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 17m 34s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 14s | | the patch passed |
| +1 :green_heart: | compile | 22m 8s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 22m 8s |
[/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2971/35/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 2 new + 1913 unchanged - 0
fixed = 1915 total (was 1913) |
| +1 :green_heart: | compile | 19m 33s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 19m 33s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2971/35/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 2 new + 1790
unchanged - 0 fixed = 1792 total (was 1790) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 53s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2971/35/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 30 new + 8 unchanged - 0 fixed = 38 total (was 8)
|
| +1 :green_heart: | mvnsite | 3m 34s | | the patch passed |
| +1 :green_heart: | xml | 0m 10s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 2m 52s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 13s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 27s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 17m 33s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 27s | | hadoop-project in the patch
passed. |
| -1 :x: | unit | 16m 55s |
[/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2971/35/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 7m 34s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | unit | 2m 7s | | hadoop-azure in the patch
[jira] [Work logged] (HADOOP-17901) Performance degradation in Text.append() after HADOOP-16951
[
https://issues.apache.org/jira/browse/HADOOP-17901?focusedWorklogId=648916=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648916
]
ASF GitHub Bot logged work on HADOOP-17901:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 23:17
Start Date: 09/Sep/21 23:17
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3411:
URL: https://github.com/apache/hadoop/pull/3411#issuecomment-916504373
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 5s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 5s | | trunk passed |
| +1 :green_heart: | compile | 25m 30s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 21s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 1s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 32s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 23s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 17s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 56s | | the patch passed |
| +1 :green_heart: | compile | 22m 7s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 22m 7s | | the patch passed |
| +1 :green_heart: | compile | 19m 19s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 19m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 59s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 35s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 17s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 17s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not
generate ASF License warnings. |
| | | 197m 8s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3411/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3411 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux dfd8ade5483b 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14
16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 306adffd0b5fe4821594da5618913dd501131fdf |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3411/3/testReport/ |
| Max. process+thread count | 1260 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3411/3/console |
|
[GitHub] [hadoop] hadoop-yetus commented on pull request #3411: HADOOP-17901. Performance degradation in Text.append() after HADOOP-1…
hadoop-yetus commented on pull request #3411:
URL: https://github.com/apache/hadoop/pull/3411#issuecomment-916504373
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 5s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 5s | | trunk passed |
| +1 :green_heart: | compile | 25m 30s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 21s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 1s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 32s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 23s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 17s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 56s | | the patch passed |
| +1 :green_heart: | compile | 22m 7s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 22m 7s | | the patch passed |
| +1 :green_heart: | compile | 19m 19s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 19m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 59s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 35s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 17s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 17s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not
generate ASF License warnings. |
| | | 197m 8s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3411/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3411 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux dfd8ade5483b 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14
16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 306adffd0b5fe4821594da5618913dd501131fdf |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3411/3/testReport/ |
| Max. process+thread count | 1260 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3411/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For
[jira] [Work logged] (HADOOP-17895) RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3
[
https://issues.apache.org/jira/browse/HADOOP-17895?focusedWorklogId=648911=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648911
]
ASF GitHub Bot logged work on HADOOP-17895:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 22:57
Start Date: 09/Sep/21 22:57
Worklog Time Spent: 10m
Work Description: cnauroth commented on pull request #3391:
URL: https://github.com/apache/hadoop/pull/3391#issuecomment-916495654
```
Path dir = path("testMkDirRmDirUnicode/dir-\\U0001f63b");
```
Something looks off with the test. That looks like `\\U`, so the
double-slash escapes to just one slash, followed by the rest of the string
literal. A Unicode character escape needs `\u` (single slash followed by
lower-case u).
Still, I tried changing the test locally to use a Unicode escape, and I
still couldn't reproduce a failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648911)
Time Spent: 50m (was: 40m)
> RawLocalFileSystem mkdirs with unicode filename doesn't work in Hadoop 3
>
>
> Key: HADOOP-17895
> URL: https://issues.apache.org/jira/browse/HADOOP-17895
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, fs
>Affects Versions: 3.3.1
>Reporter: Zamil Majdy
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> *Bug description:*
> `fs.mkdirs` command for `RawLocalFileSystem` doesn't work in Hadoop 3 with
> NativeIO enabled.
> The failure was happening when doing the native `chmod` command to the file
> (the `mkdir` command itself is working).
> Stacktrace:
> {{ENOENT: No such file or directory ENOENT: No such file or directory at
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method) at
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:382) at
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:974)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:660)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:700)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:672)}}
>
> *To reproduce:*
> * Add `fs.mkdirs` in RawLocalFileSystem with NativeIO enabled.
> * Sample: [https://github.com/apache/hadoop/pull/3391]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] cnauroth commented on pull request #3391: HADOOP-17895. Add the test to reproduce the failure of `RawLocalFileSystem.mkdir` with unicode filename
cnauroth commented on pull request #3391:
URL: https://github.com/apache/hadoop/pull/3391#issuecomment-916495654
```
Path dir = path("testMkDirRmDirUnicode/dir-\\U0001f63b");
```
Something looks off with the test. That looks like `\\U`, so the
double-slash escapes to just one slash, followed by the rest of the string
literal. A Unicode character escape needs `\u` (single slash followed by
lower-case u).
Still, I tried changing the test locally to use a Unicode escape, and I
still couldn't reproduce a failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17902) Fix Hadoop build on Debian 10
[
https://issues.apache.org/jira/browse/HADOOP-17902?focusedWorklogId=648894=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648894
]
ASF GitHub Bot logged work on HADOOP-17902:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 21:45
Start Date: 09/Sep/21 21:45
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3408:
URL: https://github.com/apache/hadoop/pull/3408#issuecomment-916459219
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | hadolint | 0m 0s | | hadolint was not available. |
| +0 :ok: | shellcheck | 0m 0s | | Shellcheck was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 13s | | trunk passed |
| +1 :green_heart: | compile | 3m 18s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 60m 43s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 16s | | the patch passed |
| +1 :green_heart: | compile | 3m 16s | | the patch passed |
| +1 :green_heart: | cc | 3m 16s | | the patch passed |
| +1 :green_heart: | golang | 3m 16s | | the patch passed |
| +1 :green_heart: | javac | 3m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 0s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 100m 5s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 188m 57s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3408 |
| Optional Tests | dupname asflicense codespell hadolint shellcheck
shelldocs compile cc mvnsite javac unit golang |
| uname | Linux 20b4218f8421 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7611377859519eb33b4e7e2b3832479b715b666e |
| Default Java | Red Hat, Inc.-1.8.0_302-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/3/testReport/ |
| Max. process+thread count | 602 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/3/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648894)
Time Spent: 1h 50m (was: 1h 40m)
> Fix Hadoop build on Debian 10
> -
>
> Key: HADOOP-17902
> URL: https://issues.apache.org/jira/browse/HADOOP-17902
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> We're using *Debian testing* as one of the
[GitHub] [hadoop] hadoop-yetus commented on pull request #3408: HADOOP-17902. Fix Hadoop build on Debian 10
hadoop-yetus commented on pull request #3408:
URL: https://github.com/apache/hadoop/pull/3408#issuecomment-916459219
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | hadolint | 0m 0s | | hadolint was not available. |
| +0 :ok: | shellcheck | 0m 0s | | Shellcheck was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 13s | | trunk passed |
| +1 :green_heart: | compile | 3m 18s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 60m 43s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 16s | | the patch passed |
| +1 :green_heart: | compile | 3m 16s | | the patch passed |
| +1 :green_heart: | cc | 3m 16s | | the patch passed |
| +1 :green_heart: | golang | 3m 16s | | the patch passed |
| +1 :green_heart: | javac | 3m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 0s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 100m 5s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 188m 57s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3408 |
| Optional Tests | dupname asflicense codespell hadolint shellcheck
shelldocs compile cc mvnsite javac unit golang |
| uname | Linux 20b4218f8421 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7611377859519eb33b4e7e2b3832479b715b666e |
| Default Java | Red Hat, Inc.-1.8.0_302-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/3/testReport/ |
| Max. process+thread count | 602 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/3/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648893=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648893
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 21:35
Start Date: 09/Sep/21 21:35
Worklog Time Spent: 10m
Work Description: viirya commented on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-916454212
Looks working now. @sunchao Does the change look good to you? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648893)
Time Spent: 12h (was: 11h 50m)
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 12h
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] viirya commented on pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
viirya commented on pull request #3385: URL: https://github.com/apache/hadoop/pull/3385#issuecomment-916454212 Looks working now. @sunchao Does the change look good to you? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648892=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648892
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 21:30
Start Date: 09/Sep/21 21:30
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-916450790
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 59s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 49s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 0s | | trunk passed |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 22s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 21s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 28s | |
branch/hadoop-client-modules/hadoop-client-api no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-runtime no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 23s | |
branch/hadoop-client-modules/hadoop-client-check-invariants no spotbugs output
file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 25s | |
branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-check-test-invariants no spotbugs
output file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-integration-tests no spotbugs output
file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 26m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 10m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 15s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 15s | | the patch passed |
| +1 :green_heart: | compile | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 38s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | xml | 0m 8s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 1m 30s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 30s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 15s | |
hadoop-client-modules/hadoop-client-api has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-runtime has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-check-invariants has no data from spotbugs
|
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-minicluster has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 15s | |
hadoop-client-modules/hadoop-client-check-test-invariants has no data from
spotbugs |
| +0 :ok: | spotbugs | 0m 15s | |
[GitHub] [hadoop] hadoop-yetus commented on pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
hadoop-yetus commented on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-916450790
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 59s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 49s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 0s | | trunk passed |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 22s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 21s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 28s | |
branch/hadoop-client-modules/hadoop-client-api no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-runtime no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 23s | |
branch/hadoop-client-modules/hadoop-client-check-invariants no spotbugs output
file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 25s | |
branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-check-test-invariants no spotbugs
output file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-integration-tests no spotbugs output
file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 26m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 10m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 15s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 15s | | the patch passed |
| +1 :green_heart: | compile | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 38s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | xml | 0m 8s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 1m 30s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 30s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 15s | |
hadoop-client-modules/hadoop-client-api has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-runtime has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-check-invariants has no data from spotbugs
|
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-minicluster has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 15s | |
hadoop-client-modules/hadoop-client-check-test-invariants has no data from
spotbugs |
| +0 :ok: | spotbugs | 0m 15s | |
hadoop-client-modules/hadoop-client-integration-tests has no data from spotbugs
|
| +1 :green_heart: | shadedclient | 27m 26s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 20s | | hadoop-client-api in the patch
passed. |
| +1 :green_heart: | unit | 0m 18s | | hadoop-client-runtime in the
patch passed. |
| +1 :green_heart: | unit |
[jira] [Work logged] (HADOOP-17873) ABFS: Fix transient failures in ITestAbfsStreamStatistics and ITestAbfsRestOperationException
[
https://issues.apache.org/jira/browse/HADOOP-17873?focusedWorklogId=648881=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648881
]
ASF GitHub Bot logged work on HADOOP-17873:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 21:02
Start Date: 09/Sep/21 21:02
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#3341:
URL: https://github.com/apache/hadoop/pull/3341#discussion_r705695532
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/oauth2/RetryTestTokenProvider.java
##
@@ -30,30 +30,34 @@
*/
public class RetryTestTokenProvider implements CustomTokenProviderAdaptee {
- // Need to track first token fetch otherwise will get counted as a retry too.
- private static boolean isThisFirstTokenFetch = true;
- public static int reTryCount = 0;
+ private static final Logger LOG = LoggerFactory.getLogger(
+ RetryTestTokenProvider.class);
- private static final Logger LOG = LoggerFactory
- .getLogger(RetryTestTokenProvider.class);
+ // Need to track first token fetch otherwise will get counted as a retry too.
+ private boolean isThisFirstTokenFetch = true;
+ private int retryCount = 0;
@Override
public void initialize(Configuration configuration, String accountName)
throws IOException {
}
- public static void ResetStatusToFirstTokenFetch() {
+ /**
+ * Clear earlier retry details and reset RetryTestTokenProvider instance to
+ * state of first access token fetch call
Review comment:
nit: add a .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648881)
Time Spent: 4h 20m (was: 4h 10m)
> ABFS: Fix transient failures in ITestAbfsStreamStatistics and
> ITestAbfsRestOperationException
> -
>
> Key: HADOOP-17873
> URL: https://issues.apache.org/jira/browse/HADOOP-17873
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.1
>Reporter: Sumangala Patki
>Assignee: Sumangala Patki
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> To address transient failures in the following test classes:
> * ITestAbfsStreamStatistics: Uses a filesystem level instance to record
> read/write statistics, which also tracks these operations in other tests.
> running parallelly. To be marked for sequential run only to avoid transient
> failure
> * ITestAbfsRestOperationException: The use of a static member to track retry
> count causes transient failures when two tests of this class happen to run
> together. Switch to non-static variable for assertions on retry count
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #3341: HADOOP-17873. ABFS: Fix transient failures in ITestAbfsStreamStatistics and ITestAbfsRestOperationException
steveloughran commented on a change in pull request #3341:
URL: https://github.com/apache/hadoop/pull/3341#discussion_r705695532
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/oauth2/RetryTestTokenProvider.java
##
@@ -30,30 +30,34 @@
*/
public class RetryTestTokenProvider implements CustomTokenProviderAdaptee {
- // Need to track first token fetch otherwise will get counted as a retry too.
- private static boolean isThisFirstTokenFetch = true;
- public static int reTryCount = 0;
+ private static final Logger LOG = LoggerFactory.getLogger(
+ RetryTestTokenProvider.class);
- private static final Logger LOG = LoggerFactory
- .getLogger(RetryTestTokenProvider.class);
+ // Need to track first token fetch otherwise will get counted as a retry too.
+ private boolean isThisFirstTokenFetch = true;
+ private int retryCount = 0;
@Override
public void initialize(Configuration configuration, String accountName)
throws IOException {
}
- public static void ResetStatusToFirstTokenFetch() {
+ /**
+ * Clear earlier retry details and reset RetryTestTokenProvider instance to
+ * state of first access token fetch call
Review comment:
nit: add a .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17873) ABFS: Fix transient failures in ITestAbfsStreamStatistics and ITestAbfsRestOperationException
[ https://issues.apache.org/jira/browse/HADOOP-17873?focusedWorklogId=648878=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648878 ] ASF GitHub Bot logged work on HADOOP-17873: --- Author: ASF GitHub Bot Created on: 09/Sep/21 21:00 Start Date: 09/Sep/21 21:00 Worklog Time Spent: 10m Work Description: steveloughran commented on a change in pull request #3341: URL: https://github.com/apache/hadoop/pull/3341#discussion_r705695212 ## File path: hadoop-tools/hadoop-azure/pom.xml ## @@ -506,6 +506,9 @@ **/azurebfs/Test*.java **/azurebfs/**/Test*.java + Review comment: I am mistaken; as this runs methods in the same process in parallel, you will need to move it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 648878) Time Spent: 4h 10m (was: 4h) > ABFS: Fix transient failures in ITestAbfsStreamStatistics and > ITestAbfsRestOperationException > - > > Key: HADOOP-17873 > URL: https://issues.apache.org/jira/browse/HADOOP-17873 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 4h 10m > Remaining Estimate: 0h > > To address transient failures in the following test classes: > * ITestAbfsStreamStatistics: Uses a filesystem level instance to record > read/write statistics, which also tracks these operations in other tests. > running parallelly. To be marked for sequential run only to avoid transient > failure > * ITestAbfsRestOperationException: The use of a static member to track retry > count causes transient failures when two tests of this class happen to run > together. Switch to non-static variable for assertions on retry count -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2971: MAPREDUCE-7341. Intermediate Manifest Committer
hadoop-yetus commented on pull request #2971:
URL: https://github.com/apache/hadoop/pull/2971#issuecomment-916432798
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 2s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 2s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 25 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 30s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 7s | | trunk passed |
| +1 :green_heart: | compile | 23m 2s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 16s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 36s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 8s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 32s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 17m 11s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 17m 32s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 12s | | the patch passed |
| +1 :green_heart: | compile | 22m 8s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 22m 8s |
[/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2971/34/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 2 new + 1913 unchanged - 0
fixed = 1915 total (was 1913) |
| +1 :green_heart: | compile | 19m 17s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 19m 17s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2971/34/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 2 new + 1790
unchanged - 0 fixed = 1792 total (was 1790) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 49s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2971/34/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 30 new + 8 unchanged - 0 fixed = 38 total (was 8)
|
| +1 :green_heart: | mvnsite | 3m 32s | | the patch passed |
| +1 :green_heart: | xml | 0m 10s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 2m 42s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 9s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 28s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 17m 30s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 26s | | hadoop-project in the patch
passed. |
| -1 :x: | unit | 17m 0s |
[/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2971/34/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 7m 35s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | unit | 2m 6s | | hadoop-azure in the patch
[GitHub] [hadoop] steveloughran commented on a change in pull request #3341: HADOOP-17873. ABFS: Fix transient failures in ITestAbfsStreamStatistics and ITestAbfsRestOperationException
steveloughran commented on a change in pull request #3341: URL: https://github.com/apache/hadoop/pull/3341#discussion_r705695212 ## File path: hadoop-tools/hadoop-azure/pom.xml ## @@ -506,6 +506,9 @@ **/azurebfs/Test*.java **/azurebfs/**/Test*.java + Review comment: I am mistaken; as this runs methods in the same process in parallel, you will need to move it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14693) Upgrade JUnit from 4 to 5
[ https://issues.apache.org/jira/browse/HADOOP-14693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412838#comment-17412838 ] Steve Loughran commented on HADOOP-14693: - (it's not junit5; its hadoop-azure surefire set to run different test methods in parallel *and out of order*) > Upgrade JUnit from 4 to 5 > - > > Key: HADOOP-14693 > URL: https://issues.apache.org/jira/browse/HADOOP-14693 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > JUnit 4 does not support Java 9. We need to upgrade this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17873) ABFS: Fix transient failures in ITestAbfsStreamStatistics and ITestAbfsRestOperationException
[ https://issues.apache.org/jira/browse/HADOOP-17873?focusedWorklogId=648876=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648876 ] ASF GitHub Bot logged work on HADOOP-17873: --- Author: ASF GitHub Bot Created on: 09/Sep/21 20:51 Start Date: 09/Sep/21 20:51 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #3341: URL: https://github.com/apache/hadoop/pull/3341#issuecomment-916427326 OK, I have now discovered something while trying to track down why test running of the abfs integration tests in MAPREDUCE-7341 fail. the hadoop-azure pom *does* request parallel method execution ``` both ``` I think this means that different processes do run different methods in the parallel phase. This actually breaks anything which expects static state to be shared across different threads in the same test case, and also means that tests where `@FixMethodOrder(MethodSorters.NAME_ASCENDING)` is used to fix the method order Turning this feature off gets those tests to work. try commenting it out and see what difference it makes for you. I would like to change it to the same that other parallel runs have: all methods in a TestCase is run in the same process, and after that test case has finished, the process ends. I'll make it a separate change from my PR, but will include it my PR as well, for completeness. (thought it was some junit5 thing; not so) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 648876) Time Spent: 4h (was: 3h 50m) > ABFS: Fix transient failures in ITestAbfsStreamStatistics and > ITestAbfsRestOperationException > - > > Key: HADOOP-17873 > URL: https://issues.apache.org/jira/browse/HADOOP-17873 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 4h > Remaining Estimate: 0h > > To address transient failures in the following test classes: > * ITestAbfsStreamStatistics: Uses a filesystem level instance to record > read/write statistics, which also tracks these operations in other tests. > running parallelly. To be marked for sequential run only to avoid transient > failure > * ITestAbfsRestOperationException: The use of a static member to track retry > count causes transient failures when two tests of this class happen to run > together. Switch to non-static variable for assertions on retry count -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3341: HADOOP-17873. ABFS: Fix transient failures in ITestAbfsStreamStatistics and ITestAbfsRestOperationException
steveloughran commented on pull request #3341: URL: https://github.com/apache/hadoop/pull/3341#issuecomment-916427326 OK, I have now discovered something while trying to track down why test running of the abfs integration tests in MAPREDUCE-7341 fail. the hadoop-azure pom *does* request parallel method execution ``` both ``` I think this means that different processes do run different methods in the parallel phase. This actually breaks anything which expects static state to be shared across different threads in the same test case, and also means that tests where `@FixMethodOrder(MethodSorters.NAME_ASCENDING)` is used to fix the method order Turning this feature off gets those tests to work. try commenting it out and see what difference it makes for you. I would like to change it to the same that other parallel runs have: all methods in a TestCase is run in the same process, and after that test case has finished, the process ends. I'll make it a separate change from my PR, but will include it my PR as well, for completeness. (thought it was some junit5 thing; not so) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14693) Upgrade JUnit from 4 to 5
[ https://issues.apache.org/jira/browse/HADOOP-14693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412820#comment-17412820 ] Steve Loughran commented on HADOOP-14693: - I'm having fun with my tests in MAPREDUCE-7341 & parallel surefire running. In the hadoop-azure parallel run, the junit4 @FixMethodOrder(MethodSorters.NAME_ASCENDING) attribute is not working; methods are running out of order. Looks like the mechanism has changed https://junit.org/junit5/docs/current/user-guide/#writing-tests-test-execution-order and, presumably, it means somehow sequential vs parallel runs work differently. Will investigate further. If JUnit5 have gone and changed something else I shall be deeply disappointed. > Upgrade JUnit from 4 to 5 > - > > Key: HADOOP-14693 > URL: https://issues.apache.org/jira/browse/HADOOP-14693 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > JUnit 4 does not support Java 9. We need to upgrade this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3396: YARN-10872. Replace getPropsWithPrefix calls in AutoCreatedQueueTemplate
hadoop-yetus commented on pull request #3396:
URL: https://github.com/apache/hadoop/pull/3396#issuecomment-916384637
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 15s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 23s | | trunk passed |
| +1 :green_heart: | compile | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 56s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 2s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 57s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 7s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 52s | | the patch passed |
| +1 :green_heart: | compile | 1m 1s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 1s | | the patch passed |
| +1 :green_heart: | compile | 0m 49s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 49s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 41s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 36 unchanged - 6 fixed = 36 total (was 42) |
| +1 :green_heart: | mvnsite | 0m 52s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 39s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 0s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 99m 37s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not
generate ASF License warnings. |
| | | 186m 56s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3396/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3396 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux c39569dd7fac 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f61d531390e11365492cd8aa394465734cfb03ce |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3396/4/testReport/ |
| Max. process+thread count | 912 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3396/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on
[jira] [Work logged] (HADOOP-17904) Test Result Not Working In Jenkins Result
[
https://issues.apache.org/jira/browse/HADOOP-17904?focusedWorklogId=648835=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648835
]
ASF GitHub Bot logged work on HADOOP-17904:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 19:34
Start Date: 09/Sep/21 19:34
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3413:
URL: https://github.com/apache/hadoop/pull/3413#issuecomment-916379798
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 43s | | Maven dependency ordering for branch |
| +1 :green_heart: | shadedclient | 13m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | shadedclient | 13m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 44m 59s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3413/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3413 |
| Optional Tests | dupname asflicense codespell shellcheck shelldocs |
| uname | Linux 947f61faaec8 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7c9d751a905da695e30d83a99db6d03539607ebe |
| Max. process+thread count | 542 (vs. ulimit of 5500) |
| modules | C: U: |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3413/2/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648835)
Time Spent: 1h 40m (was: 1.5h)
> Test Result Not Working In Jenkins Result
> -
>
> Key: HADOOP-17904
> URL: https://issues.apache.org/jira/browse/HADOOP-17904
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.4.0
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The jenkins used to show the Test Results, now the list is broken and No PR
> has that working.
> eg:
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3404/1/testReport/]
>
> For Daily Build No Test Result as well:
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/622/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3413: HADOOP-17904. Test Result Not Working In Jenkins Result.
hadoop-yetus commented on pull request #3413:
URL: https://github.com/apache/hadoop/pull/3413#issuecomment-916379798
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 43s | | Maven dependency ordering for branch |
| +1 :green_heart: | shadedclient | 13m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | shadedclient | 13m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 44m 59s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3413/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3413 |
| Optional Tests | dupname asflicense codespell shellcheck shelldocs |
| uname | Linux 947f61faaec8 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7c9d751a905da695e30d83a99db6d03539607ebe |
| Max. process+thread count | 542 (vs. ulimit of 5500) |
| modules | C: U: |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3413/2/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648831=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648831
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 19:28
Start Date: 09/Sep/21 19:28
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-916376206
(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3385/16/console in
case of problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648831)
Time Spent: 11h 40m (was: 11.5h)
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 11h 40m
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
hadoop-yetus commented on pull request #3385: URL: https://github.com/apache/hadoop/pull/3385#issuecomment-916376206 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3385/16/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17902) Fix Hadoop build on Debian 10
[
https://issues.apache.org/jira/browse/HADOOP-17902?focusedWorklogId=648828=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648828
]
ASF GitHub Bot logged work on HADOOP-17902:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 19:12
Start Date: 09/Sep/21 19:12
Worklog Time Spent: 10m
Work Description: goiri commented on a change in pull request #3408:
URL: https://github.com/apache/hadoop/pull/3408#discussion_r705624420
##
File path: dev-support/docker/Dockerfile_debian_10
##
@@ -54,6 +49,20 @@ RUN apt-get -q update \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
+#
+# For installing the latest packages
+#
+RUN echo 'deb http://deb.debian.org/debian testing main' >>
/etc/apt/sources.list
Review comment:
If you set this up, why does the packages.json approach do not work?
I would prefer to install using the json approach than going back to coding
for each.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648828)
Time Spent: 1h 40m (was: 1.5h)
> Fix Hadoop build on Debian 10
> -
>
> Key: HADOOP-17902
> URL: https://issues.apache.org/jira/browse/HADOOP-17902
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> We're using *Debian testing* as one of the package sources to get the latest
> packages. It seems to be broken at the moment. The CI fails to create the
> build environment for the Debian 10 platform -
> {code}
> [2021-09-08T00:21:11.596Z] #13 [ 8/14] RUN apt-get -q update && apt-get
> -q install -y --no-install-recommends python3 && apt-get -q install -y
> --no-install-recommends $(pkg-resolver/resolve.py debian:10) && apt-get
> clean && rm -rf /var/lib/apt/lists/*
> ...
> [2021-09-08T00:21:22.744Z] #13 11.28 Preparing to unpack
> .../libc6_2.31-17_amd64.deb ...
> [2021-09-08T00:21:23.260Z] #13 11.46 Checking for services that may need to
> be restarted...
> [2021-09-08T00:21:23.260Z] #13 11.48 Checking init scripts...
> [2021-09-08T00:21:23.260Z] #13 11.50 Unpacking libc6:amd64 (2.31-17) over
> (2.28-10) ...
> [2021-09-08T00:21:26.290Z] #13 14.38 Setting up libc6:amd64 (2.31-17) ...
> [2021-09-08T00:21:26.290Z] #13 14.42 /usr/bin/perl: error while loading
> shared libraries: libcrypt.so.1: cannot open shared object file: No such file
> or directory
> [2021-09-08T00:21:26.290Z] #13 14.42 dpkg: error processing package
> libc6:amd64 (--configure):
> [2021-09-08T00:21:26.290Z] #13 14.42 installed libc6:amd64 package
> post-installation script subprocess returned error exit status 127
> [2021-09-08T00:21:26.291Z] #13 14.43 Errors were encountered while processing:
> [2021-09-08T00:21:26.291Z] #13 14.43 libc6:amd64
> [2021-09-08T00:21:26.291Z] #13 14.46 E: Sub-process /usr/bin/dpkg returned an
> error code (1)
> [2021-09-08T00:21:27.867Z] #13 ERROR: executor failed running [/bin/bash -o
> pipefail -c apt-get -q update && apt-get -q install -y
> --no-install-recommends python3 && apt-get -q install -y
> --no-install-recommends $(pkg-resolver/resolve.py debian:10) && apt-get
> clean && rm -rf /var/lib/apt/lists/*]: exit code: 100
> [2021-09-08T00:21:27.867Z] --
> [2021-09-08T00:21:27.867Z] > [ 8/14] RUN apt-get -q update && apt-get -q
> install -y --no-install-recommends python3 && apt-get -q install -y
> --no-install-recommends $(pkg-resolver/resolve.py debian:10) && apt-get
> clean && rm -rf /var/lib/apt/lists/*:
> [2021-09-08T00:21:27.867Z] --
> [2021-09-08T00:21:27.867Z] executor failed running [/bin/bash -o pipefail -c
> apt-get -q update && apt-get -q install -y --no-install-recommends
> python3 && apt-get -q install -y --no-install-recommends
> $(pkg-resolver/resolve.py debian:10) && apt-get clean && rm -rf
> /var/lib/apt/lists/*]: exit code: 100
> [2021-09-08T00:21:27.867Z] ERROR: Docker failed to build
> yetus/hadoop:ef5dbc7283a.
> [2021-09-08T00:21:27.867Z]
> {code}
> The above log lines are copied from -
> https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3388/3/pipeline
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hadoop] goiri commented on a change in pull request #3408: HADOOP-17902. Fix Hadoop build on Debian 10
goiri commented on a change in pull request #3408: URL: https://github.com/apache/hadoop/pull/3408#discussion_r705624420 ## File path: dev-support/docker/Dockerfile_debian_10 ## @@ -54,6 +49,20 @@ RUN apt-get -q update \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* +# +# For installing the latest packages +# +RUN echo 'deb http://deb.debian.org/debian testing main' >> /etc/apt/sources.list Review comment: If you set this up, why does the packages.json approach do not work? I would prefer to install using the json approach than going back to coding for each. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648826=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648826
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 19:10
Start Date: 09/Sep/21 19:10
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-916364503
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 46s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 12s | | trunk passed |
| +1 :green_heart: | compile | 0m 26s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 22s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 20s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 28s | |
branch/hadoop-client-modules/hadoop-client-api no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 25s | |
branch/hadoop-client-modules/hadoop-client-runtime no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-check-invariants no spotbugs output
file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 25s | |
branch/hadoop-client-modules/hadoop-client-check-test-invariants no spotbugs
output file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-integration-tests no spotbugs output
file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 26m 44s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 14s |
[/patch-mvninstall-hadoop-client-modules_hadoop-client-check-test-invariants.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3385/15/artifact/out/patch-mvninstall-hadoop-client-modules_hadoop-client-check-test-invariants.txt)
| hadoop-client-check-test-invariants in the patch failed. |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 14s | | the patch passed |
| -1 :x: | mvnsite | 0m 15s |
[/patch-mvnsite-hadoop-client-modules_hadoop-client-check-test-invariants.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3385/15/artifact/out/patch-mvnsite-hadoop-client-modules_hadoop-client-check-test-invariants.txt)
| hadoop-client-check-test-invariants in the patch failed. |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 8s | | The patch has no ill-formed XML
file. |
| -1 :x: | javadoc | 0m 15s |
[/patch-javadoc-hadoop-client-modules_hadoop-client-check-test-invariants-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3385/15/artifact/out/patch-javadoc-hadoop-client-modules_hadoop-client-check-test-invariants-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-client-check-test-invariants in
[GitHub] [hadoop] hadoop-yetus commented on pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
hadoop-yetus commented on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-916364503
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 46s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 12s | | trunk passed |
| +1 :green_heart: | compile | 0m 26s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 22s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 20s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 28s | |
branch/hadoop-client-modules/hadoop-client-api no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 25s | |
branch/hadoop-client-modules/hadoop-client-runtime no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-check-invariants no spotbugs output
file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 25s | |
branch/hadoop-client-modules/hadoop-client-check-test-invariants no spotbugs
output file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-integration-tests no spotbugs output
file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 26m 44s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 14s |
[/patch-mvninstall-hadoop-client-modules_hadoop-client-check-test-invariants.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3385/15/artifact/out/patch-mvninstall-hadoop-client-modules_hadoop-client-check-test-invariants.txt)
| hadoop-client-check-test-invariants in the patch failed. |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 14s | | the patch passed |
| -1 :x: | mvnsite | 0m 15s |
[/patch-mvnsite-hadoop-client-modules_hadoop-client-check-test-invariants.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3385/15/artifact/out/patch-mvnsite-hadoop-client-modules_hadoop-client-check-test-invariants.txt)
| hadoop-client-check-test-invariants in the patch failed. |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 8s | | The patch has no ill-formed XML
file. |
| -1 :x: | javadoc | 0m 15s |
[/patch-javadoc-hadoop-client-modules_hadoop-client-check-test-invariants-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3385/15/artifact/out/patch-javadoc-hadoop-client-modules_hadoop-client-check-test-invariants-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-client-check-test-invariants in the patch failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javadoc | 0m 14s |
[jira] [Work logged] (HADOOP-17904) Test Result Not Working In Jenkins Result
[
https://issues.apache.org/jira/browse/HADOOP-17904?focusedWorklogId=648815=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648815
]
ASF GitHub Bot logged work on HADOOP-17904:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 18:46
Start Date: 09/Sep/21 18:46
Worklog Time Spent: 10m
Work Description: ayushtkn commented on a change in pull request #3413:
URL: https://github.com/apache/hadoop/pull/3413#discussion_r705607236
##
File path: dev-support/Jenkinsfile
##
@@ -30,8 +30,9 @@ def publishJUnitResults() {
boolean surefireReportsExist = findCmdExitCode == 0
if (surefireReportsExist) {
echo "XML files found under surefire-reports, running junit"
+SRC =
"${SOURCEDIR}/**/target/surefire-reports/*.xml".replace("$WORKSPACE/","")
Review comment:
Now:
https://github.com/apache/hadoop/blob/trunk/dev-support/Jenkinsfile#L259
```
SOURCEDIR = "${WORKSPACE}/ubuntu-focal/src"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648815)
Time Spent: 1.5h (was: 1h 20m)
> Test Result Not Working In Jenkins Result
> -
>
> Key: HADOOP-17904
> URL: https://issues.apache.org/jira/browse/HADOOP-17904
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.4.0
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> The jenkins used to show the Test Results, now the list is broken and No PR
> has that working.
> eg:
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3404/1/testReport/]
>
> For Daily Build No Test Result as well:
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/622/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #3413: HADOOP-17904. Test Result Not Working In Jenkins Result.
ayushtkn commented on a change in pull request #3413:
URL: https://github.com/apache/hadoop/pull/3413#discussion_r705607236
##
File path: dev-support/Jenkinsfile
##
@@ -30,8 +30,9 @@ def publishJUnitResults() {
boolean surefireReportsExist = findCmdExitCode == 0
if (surefireReportsExist) {
echo "XML files found under surefire-reports, running junit"
+SRC =
"${SOURCEDIR}/**/target/surefire-reports/*.xml".replace("$WORKSPACE/","")
Review comment:
Now:
https://github.com/apache/hadoop/blob/trunk/dev-support/Jenkinsfile#L259
```
SOURCEDIR = "${WORKSPACE}/ubuntu-focal/src"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17904) Test Result Not Working In Jenkins Result
[
https://issues.apache.org/jira/browse/HADOOP-17904?focusedWorklogId=648813=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648813
]
ASF GitHub Bot logged work on HADOOP-17904:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 18:45
Start Date: 09/Sep/21 18:45
Worklog Time Spent: 10m
Work Description: ayushtkn commented on a change in pull request #3413:
URL: https://github.com/apache/hadoop/pull/3413#discussion_r705606249
##
File path: dev-support/Jenkinsfile
##
@@ -30,8 +30,9 @@ def publishJUnitResults() {
boolean surefireReportsExist = findCmdExitCode == 0
if (surefireReportsExist) {
echo "XML files found under surefire-reports, running junit"
+SRC =
"${SOURCEDIR}/**/target/surefire-reports/*.xml".replace("$WORKSPACE/","")
Review comment:
I don't think so:
https://github.com/apache/hadoop/blob/ef5dbc7283a9283bc446511c3a49304ad3923037/dev-support/Jenkinsfile#L32
And Now:
https://github.com/apache/hadoop/blob/trunk/dev-support/Jenkinsfile#L259
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648813)
Time Spent: 1h 10m (was: 1h)
> Test Result Not Working In Jenkins Result
> -
>
> Key: HADOOP-17904
> URL: https://issues.apache.org/jira/browse/HADOOP-17904
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.4.0
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The jenkins used to show the Test Results, now the list is broken and No PR
> has that working.
> eg:
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3404/1/testReport/]
>
> For Daily Build No Test Result as well:
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/622/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17904) Test Result Not Working In Jenkins Result
[
https://issues.apache.org/jira/browse/HADOOP-17904?focusedWorklogId=648814=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648814
]
ASF GitHub Bot logged work on HADOOP-17904:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 18:45
Start Date: 09/Sep/21 18:45
Worklog Time Spent: 10m
Work Description: ayushtkn commented on a change in pull request #3413:
URL: https://github.com/apache/hadoop/pull/3413#discussion_r705607236
##
File path: dev-support/Jenkinsfile
##
@@ -30,8 +30,9 @@ def publishJUnitResults() {
boolean surefireReportsExist = findCmdExitCode == 0
if (surefireReportsExist) {
echo "XML files found under surefire-reports, running junit"
+SRC =
"${SOURCEDIR}/**/target/surefire-reports/*.xml".replace("$WORKSPACE/","")
Review comment:
Now:
https://github.com/apache/hadoop/blob/trunk/dev-support/Jenkinsfile#L259
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648814)
Time Spent: 1h 20m (was: 1h 10m)
> Test Result Not Working In Jenkins Result
> -
>
> Key: HADOOP-17904
> URL: https://issues.apache.org/jira/browse/HADOOP-17904
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.4.0
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The jenkins used to show the Test Results, now the list is broken and No PR
> has that working.
> eg:
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3404/1/testReport/]
>
> For Daily Build No Test Result as well:
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/622/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #3413: HADOOP-17904. Test Result Not Working In Jenkins Result.
ayushtkn commented on a change in pull request #3413:
URL: https://github.com/apache/hadoop/pull/3413#discussion_r705607236
##
File path: dev-support/Jenkinsfile
##
@@ -30,8 +30,9 @@ def publishJUnitResults() {
boolean surefireReportsExist = findCmdExitCode == 0
if (surefireReportsExist) {
echo "XML files found under surefire-reports, running junit"
+SRC =
"${SOURCEDIR}/**/target/surefire-reports/*.xml".replace("$WORKSPACE/","")
Review comment:
Now:
https://github.com/apache/hadoop/blob/trunk/dev-support/Jenkinsfile#L259
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #3413: HADOOP-17904. Test Result Not Working In Jenkins Result.
ayushtkn commented on a change in pull request #3413:
URL: https://github.com/apache/hadoop/pull/3413#discussion_r705606249
##
File path: dev-support/Jenkinsfile
##
@@ -30,8 +30,9 @@ def publishJUnitResults() {
boolean surefireReportsExist = findCmdExitCode == 0
if (surefireReportsExist) {
echo "XML files found under surefire-reports, running junit"
+SRC =
"${SOURCEDIR}/**/target/surefire-reports/*.xml".replace("$WORKSPACE/","")
Review comment:
I don't think so:
https://github.com/apache/hadoop/blob/ef5dbc7283a9283bc446511c3a49304ad3923037/dev-support/Jenkinsfile#L32
And Now:
https://github.com/apache/hadoop/blob/trunk/dev-support/Jenkinsfile#L259
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17904) Test Result Not Working In Jenkins Result
[
https://issues.apache.org/jira/browse/HADOOP-17904?focusedWorklogId=648812=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648812
]
ASF GitHub Bot logged work on HADOOP-17904:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 18:44
Start Date: 09/Sep/21 18:44
Worklog Time Spent: 10m
Work Description: ayushtkn commented on a change in pull request #3413:
URL: https://github.com/apache/hadoop/pull/3413#discussion_r705606249
##
File path: dev-support/Jenkinsfile
##
@@ -30,8 +30,9 @@ def publishJUnitResults() {
boolean surefireReportsExist = findCmdExitCode == 0
if (surefireReportsExist) {
echo "XML files found under surefire-reports, running junit"
+SRC =
"${SOURCEDIR}/**/target/surefire-reports/*.xml".replace("$WORKSPACE/","")
Review comment:
I don't think so:
https://github.com/apache/hadoop/blob/ef5dbc7283a9283bc446511c3a49304ad3923037/dev-support/Jenkinsfile#L32
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648812)
Time Spent: 1h (was: 50m)
> Test Result Not Working In Jenkins Result
> -
>
> Key: HADOOP-17904
> URL: https://issues.apache.org/jira/browse/HADOOP-17904
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.4.0
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The jenkins used to show the Test Results, now the list is broken and No PR
> has that working.
> eg:
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3404/1/testReport/]
>
> For Daily Build No Test Result as well:
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/622/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #3413: HADOOP-17904. Test Result Not Working In Jenkins Result.
ayushtkn commented on a change in pull request #3413:
URL: https://github.com/apache/hadoop/pull/3413#discussion_r705606249
##
File path: dev-support/Jenkinsfile
##
@@ -30,8 +30,9 @@ def publishJUnitResults() {
boolean surefireReportsExist = findCmdExitCode == 0
if (surefireReportsExist) {
echo "XML files found under surefire-reports, running junit"
+SRC =
"${SOURCEDIR}/**/target/surefire-reports/*.xml".replace("$WORKSPACE/","")
Review comment:
I don't think so:
https://github.com/apache/hadoop/blob/ef5dbc7283a9283bc446511c3a49304ad3923037/dev-support/Jenkinsfile#L32
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17904) Test Result Not Working In Jenkins Result
[
https://issues.apache.org/jira/browse/HADOOP-17904?focusedWorklogId=648809=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648809
]
ASF GitHub Bot logged work on HADOOP-17904:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 18:41
Start Date: 09/Sep/21 18:41
Worklog Time Spent: 10m
Work Description: GauthamBanasandra commented on a change in pull request
#3413:
URL: https://github.com/apache/hadoop/pull/3413#discussion_r705604176
##
File path: dev-support/Jenkinsfile
##
@@ -30,8 +30,9 @@ def publishJUnitResults() {
boolean surefireReportsExist = findCmdExitCode == 0
if (surefireReportsExist) {
echo "XML files found under surefire-reports, running junit"
+SRC =
"${SOURCEDIR}/**/target/surefire-reports/*.xml".replace("$WORKSPACE/","")
Review comment:
This path used to have `env.SOURCEDIR` prefix before I made this change
-
https://github.com/apache/hadoop/blob/ef5dbc7283a9283bc446511c3a49304ad3923037/dev-support/Jenkinsfile#L145.
Wasn't `env.SOURCEDIR` resolving to the absolute path back then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648809)
Time Spent: 50m (was: 40m)
> Test Result Not Working In Jenkins Result
> -
>
> Key: HADOOP-17904
> URL: https://issues.apache.org/jira/browse/HADOOP-17904
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.4.0
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The jenkins used to show the Test Results, now the list is broken and No PR
> has that working.
> eg:
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3404/1/testReport/]
>
> For Daily Build No Test Result as well:
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/622/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] GauthamBanasandra commented on a change in pull request #3413: HADOOP-17904. Test Result Not Working In Jenkins Result.
GauthamBanasandra commented on a change in pull request #3413:
URL: https://github.com/apache/hadoop/pull/3413#discussion_r705604176
##
File path: dev-support/Jenkinsfile
##
@@ -30,8 +30,9 @@ def publishJUnitResults() {
boolean surefireReportsExist = findCmdExitCode == 0
if (surefireReportsExist) {
echo "XML files found under surefire-reports, running junit"
+SRC =
"${SOURCEDIR}/**/target/surefire-reports/*.xml".replace("$WORKSPACE/","")
Review comment:
This path used to have `env.SOURCEDIR` prefix before I made this change
-
https://github.com/apache/hadoop/blob/ef5dbc7283a9283bc446511c3a49304ad3923037/dev-support/Jenkinsfile#L145.
Wasn't `env.SOURCEDIR` resolving to the absolute path back then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17904) Test Result Not Working In Jenkins Result
[
https://issues.apache.org/jira/browse/HADOOP-17904?focusedWorklogId=648806=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648806
]
ASF GitHub Bot logged work on HADOOP-17904:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 18:37
Start Date: 09/Sep/21 18:37
Worklog Time Spent: 10m
Work Description: GauthamBanasandra commented on a change in pull request
#3413:
URL: https://github.com/apache/hadoop/pull/3413#discussion_r705601451
##
File path: dev-support/Jenkinsfile
##
@@ -30,8 +30,9 @@ def publishJUnitResults() {
boolean surefireReportsExist = findCmdExitCode == 0
if (surefireReportsExist) {
echo "XML files found under surefire-reports, running junit"
+SRC =
"${SOURCEDIR}/**/target/surefire-reports/*.xml".replace("$WORKSPACE/","")
Review comment:
Could you please add a comment here mentioning that the path must be
relative to the checkout dir?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648806)
Time Spent: 40m (was: 0.5h)
> Test Result Not Working In Jenkins Result
> -
>
> Key: HADOOP-17904
> URL: https://issues.apache.org/jira/browse/HADOOP-17904
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.4.0
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The jenkins used to show the Test Results, now the list is broken and No PR
> has that working.
> eg:
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3404/1/testReport/]
>
> For Daily Build No Test Result as well:
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/622/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] GauthamBanasandra commented on a change in pull request #3413: HADOOP-17904. Test Result Not Working In Jenkins Result.
GauthamBanasandra commented on a change in pull request #3413:
URL: https://github.com/apache/hadoop/pull/3413#discussion_r705601451
##
File path: dev-support/Jenkinsfile
##
@@ -30,8 +30,9 @@ def publishJUnitResults() {
boolean surefireReportsExist = findCmdExitCode == 0
if (surefireReportsExist) {
echo "XML files found under surefire-reports, running junit"
+SRC =
"${SOURCEDIR}/**/target/surefire-reports/*.xml".replace("$WORKSPACE/","")
Review comment:
Could you please add a comment here mentioning that the path must be
relative to the checkout dir?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17902) Fix Hadoop build on Debian 10
[
https://issues.apache.org/jira/browse/HADOOP-17902?focusedWorklogId=648805=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648805
]
ASF GitHub Bot logged work on HADOOP-17902:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 18:35
Start Date: 09/Sep/21 18:35
Worklog Time Spent: 10m
Work Description: GauthamBanasandra commented on a change in pull request
#3408:
URL: https://github.com/apache/hadoop/pull/3408#discussion_r705600241
##
File path: dev-support/docker/Dockerfile_debian_10
##
@@ -54,6 +49,20 @@ RUN apt-get -q update \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
+#
+# For installing the latest packages
+#
+RUN echo 'deb http://deb.debian.org/debian testing main' >>
/etc/apt/sources.list
+
+#
+# Install GCC 9.3.0
Review comment:
My bad, I just checked, it's GCC 10.3.0. I've rectified it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648805)
Time Spent: 1.5h (was: 1h 20m)
> Fix Hadoop build on Debian 10
> -
>
> Key: HADOOP-17902
> URL: https://issues.apache.org/jira/browse/HADOOP-17902
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> We're using *Debian testing* as one of the package sources to get the latest
> packages. It seems to be broken at the moment. The CI fails to create the
> build environment for the Debian 10 platform -
> {code}
> [2021-09-08T00:21:11.596Z] #13 [ 8/14] RUN apt-get -q update && apt-get
> -q install -y --no-install-recommends python3 && apt-get -q install -y
> --no-install-recommends $(pkg-resolver/resolve.py debian:10) && apt-get
> clean && rm -rf /var/lib/apt/lists/*
> ...
> [2021-09-08T00:21:22.744Z] #13 11.28 Preparing to unpack
> .../libc6_2.31-17_amd64.deb ...
> [2021-09-08T00:21:23.260Z] #13 11.46 Checking for services that may need to
> be restarted...
> [2021-09-08T00:21:23.260Z] #13 11.48 Checking init scripts...
> [2021-09-08T00:21:23.260Z] #13 11.50 Unpacking libc6:amd64 (2.31-17) over
> (2.28-10) ...
> [2021-09-08T00:21:26.290Z] #13 14.38 Setting up libc6:amd64 (2.31-17) ...
> [2021-09-08T00:21:26.290Z] #13 14.42 /usr/bin/perl: error while loading
> shared libraries: libcrypt.so.1: cannot open shared object file: No such file
> or directory
> [2021-09-08T00:21:26.290Z] #13 14.42 dpkg: error processing package
> libc6:amd64 (--configure):
> [2021-09-08T00:21:26.290Z] #13 14.42 installed libc6:amd64 package
> post-installation script subprocess returned error exit status 127
> [2021-09-08T00:21:26.291Z] #13 14.43 Errors were encountered while processing:
> [2021-09-08T00:21:26.291Z] #13 14.43 libc6:amd64
> [2021-09-08T00:21:26.291Z] #13 14.46 E: Sub-process /usr/bin/dpkg returned an
> error code (1)
> [2021-09-08T00:21:27.867Z] #13 ERROR: executor failed running [/bin/bash -o
> pipefail -c apt-get -q update && apt-get -q install -y
> --no-install-recommends python3 && apt-get -q install -y
> --no-install-recommends $(pkg-resolver/resolve.py debian:10) && apt-get
> clean && rm -rf /var/lib/apt/lists/*]: exit code: 100
> [2021-09-08T00:21:27.867Z] --
> [2021-09-08T00:21:27.867Z] > [ 8/14] RUN apt-get -q update && apt-get -q
> install -y --no-install-recommends python3 && apt-get -q install -y
> --no-install-recommends $(pkg-resolver/resolve.py debian:10) && apt-get
> clean && rm -rf /var/lib/apt/lists/*:
> [2021-09-08T00:21:27.867Z] --
> [2021-09-08T00:21:27.867Z] executor failed running [/bin/bash -o pipefail -c
> apt-get -q update && apt-get -q install -y --no-install-recommends
> python3 && apt-get -q install -y --no-install-recommends
> $(pkg-resolver/resolve.py debian:10) && apt-get clean && rm -rf
> /var/lib/apt/lists/*]: exit code: 100
> [2021-09-08T00:21:27.867Z] ERROR: Docker failed to build
> yetus/hadoop:ef5dbc7283a.
> [2021-09-08T00:21:27.867Z]
> {code}
> The above log lines are copied from -
> https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3388/3/pipeline
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail:
[GitHub] [hadoop] GauthamBanasandra commented on a change in pull request #3408: HADOOP-17902. Fix Hadoop build on Debian 10
GauthamBanasandra commented on a change in pull request #3408: URL: https://github.com/apache/hadoop/pull/3408#discussion_r705600241 ## File path: dev-support/docker/Dockerfile_debian_10 ## @@ -54,6 +49,20 @@ RUN apt-get -q update \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* +# +# For installing the latest packages +# +RUN echo 'deb http://deb.debian.org/debian testing main' >> /etc/apt/sources.list + +# +# Install GCC 9.3.0 Review comment: My bad, I just checked, it's GCC 10.3.0. I've rectified it now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17902) Fix Hadoop build on Debian 10
[
https://issues.apache.org/jira/browse/HADOOP-17902?focusedWorklogId=648791=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648791
]
ASF GitHub Bot logged work on HADOOP-17902:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 18:22
Start Date: 09/Sep/21 18:22
Worklog Time Spent: 10m
Work Description: goiri commented on a change in pull request #3408:
URL: https://github.com/apache/hadoop/pull/3408#discussion_r705592136
##
File path: dev-support/docker/Dockerfile_debian_10
##
@@ -54,6 +49,20 @@ RUN apt-get -q update \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
+#
+# For installing the latest packages
+#
+RUN echo 'deb http://deb.debian.org/debian testing main' >>
/etc/apt/sources.list
+
+#
+# Install GCC 9.3.0
Review comment:
It should be mentioned in the comment if so.
##
File path: dev-support/docker/Dockerfile_debian_10
##
@@ -54,6 +49,20 @@ RUN apt-get -q update \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
+#
+# For installing the latest packages
+#
+RUN echo 'deb http://deb.debian.org/debian testing main' >>
/etc/apt/sources.list
+
+#
+# Install GCC 9.3.0
Review comment:
There's no 10 in debian testing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648791)
Time Spent: 1h 20m (was: 1h 10m)
> Fix Hadoop build on Debian 10
> -
>
> Key: HADOOP-17902
> URL: https://issues.apache.org/jira/browse/HADOOP-17902
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> We're using *Debian testing* as one of the package sources to get the latest
> packages. It seems to be broken at the moment. The CI fails to create the
> build environment for the Debian 10 platform -
> {code}
> [2021-09-08T00:21:11.596Z] #13 [ 8/14] RUN apt-get -q update && apt-get
> -q install -y --no-install-recommends python3 && apt-get -q install -y
> --no-install-recommends $(pkg-resolver/resolve.py debian:10) && apt-get
> clean && rm -rf /var/lib/apt/lists/*
> ...
> [2021-09-08T00:21:22.744Z] #13 11.28 Preparing to unpack
> .../libc6_2.31-17_amd64.deb ...
> [2021-09-08T00:21:23.260Z] #13 11.46 Checking for services that may need to
> be restarted...
> [2021-09-08T00:21:23.260Z] #13 11.48 Checking init scripts...
> [2021-09-08T00:21:23.260Z] #13 11.50 Unpacking libc6:amd64 (2.31-17) over
> (2.28-10) ...
> [2021-09-08T00:21:26.290Z] #13 14.38 Setting up libc6:amd64 (2.31-17) ...
> [2021-09-08T00:21:26.290Z] #13 14.42 /usr/bin/perl: error while loading
> shared libraries: libcrypt.so.1: cannot open shared object file: No such file
> or directory
> [2021-09-08T00:21:26.290Z] #13 14.42 dpkg: error processing package
> libc6:amd64 (--configure):
> [2021-09-08T00:21:26.290Z] #13 14.42 installed libc6:amd64 package
> post-installation script subprocess returned error exit status 127
> [2021-09-08T00:21:26.291Z] #13 14.43 Errors were encountered while processing:
> [2021-09-08T00:21:26.291Z] #13 14.43 libc6:amd64
> [2021-09-08T00:21:26.291Z] #13 14.46 E: Sub-process /usr/bin/dpkg returned an
> error code (1)
> [2021-09-08T00:21:27.867Z] #13 ERROR: executor failed running [/bin/bash -o
> pipefail -c apt-get -q update && apt-get -q install -y
> --no-install-recommends python3 && apt-get -q install -y
> --no-install-recommends $(pkg-resolver/resolve.py debian:10) && apt-get
> clean && rm -rf /var/lib/apt/lists/*]: exit code: 100
> [2021-09-08T00:21:27.867Z] --
> [2021-09-08T00:21:27.867Z] > [ 8/14] RUN apt-get -q update && apt-get -q
> install -y --no-install-recommends python3 && apt-get -q install -y
> --no-install-recommends $(pkg-resolver/resolve.py debian:10) && apt-get
> clean && rm -rf /var/lib/apt/lists/*:
> [2021-09-08T00:21:27.867Z] --
> [2021-09-08T00:21:27.867Z] executor failed running [/bin/bash -o pipefail -c
> apt-get -q update && apt-get -q install -y --no-install-recommends
> python3 && apt-get -q install -y --no-install-recommends
> $(pkg-resolver/resolve.py debian:10) && apt-get clean && rm -rf
> /var/lib/apt/lists/*]: exit code: 100
> [2021-09-08T00:21:27.867Z] ERROR: Docker failed to build
>
[GitHub] [hadoop] goiri commented on a change in pull request #3408: HADOOP-17902. Fix Hadoop build on Debian 10
goiri commented on a change in pull request #3408: URL: https://github.com/apache/hadoop/pull/3408#discussion_r705592136 ## File path: dev-support/docker/Dockerfile_debian_10 ## @@ -54,6 +49,20 @@ RUN apt-get -q update \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* +# +# For installing the latest packages +# +RUN echo 'deb http://deb.debian.org/debian testing main' >> /etc/apt/sources.list + +# +# Install GCC 9.3.0 Review comment: It should be mentioned in the comment if so. ## File path: dev-support/docker/Dockerfile_debian_10 ## @@ -54,6 +49,20 @@ RUN apt-get -q update \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* +# +# For installing the latest packages +# +RUN echo 'deb http://deb.debian.org/debian testing main' >> /etc/apt/sources.list + +# +# Install GCC 9.3.0 Review comment: There's no 10 in debian testing? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648786=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648786
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 18:16
Start Date: 09/Sep/21 18:16
Worklog Time Spent: 10m
Work Description: sunchao commented on a change in pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#discussion_r705587930
##
File path: hadoop-client-modules/hadoop-client-integration-tests/pom.xml
##
@@ -52,6 +52,11 @@
junit
test
+
+ org.lz4
+ lz4-java
+ test
Review comment:
we can remove this now since we have the dependency in
`hadoop-client-api`, is that right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648786)
Time Spent: 11h 20m (was: 11h 10m)
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 11h 20m
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on a change in pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
sunchao commented on a change in pull request #3385: URL: https://github.com/apache/hadoop/pull/3385#discussion_r705587930 ## File path: hadoop-client-modules/hadoop-client-integration-tests/pom.xml ## @@ -52,6 +52,11 @@ junit test + + org.lz4 + lz4-java + test Review comment: we can remove this now since we have the dependency in `hadoop-client-api`, is that right? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3396: YARN-10872. Replace getPropsWithPrefix calls in AutoCreatedQueueTemplate
hadoop-yetus commented on pull request #3396:
URL: https://github.com/apache/hadoop/pull/3396#issuecomment-916308445
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 1s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 31s | | trunk passed |
| +1 :green_heart: | compile | 1m 3s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 53s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 59s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 54s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 41s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 51s | | the patch passed |
| +1 :green_heart: | compile | 0m 56s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 56s | | the patch passed |
| +1 :green_heart: | compile | 0m 46s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 46s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 39s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 36 unchanged - 6 fixed = 36 total (was 42) |
| +1 :green_heart: | mvnsite | 0m 50s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 57s | | the patch passed |
| +1 :green_heart: | shadedclient | 17m 40s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 101m 45s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not
generate ASF License warnings. |
| | | 185m 46s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3396/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3396 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 63ffaa34a1db 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c8e0dde0b4ec213faa1d3e0bc2b4f32f4290bb0c |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3396/3/testReport/ |
| Max. process+thread count | 919 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3396/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was
[GitHub] [hadoop] hadoop-yetus commented on pull request #3407: YARN-10910. AbstractCSQueue#setupQueueConfigs: Separate validation lo…
hadoop-yetus commented on pull request #3407:
URL: https://github.com/apache/hadoop/pull/3407#issuecomment-916303330
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 35s | | trunk passed |
| +1 :green_heart: | compile | 1m 2s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 53s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 58s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 55s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 53s | | the patch passed |
| +1 :green_heart: | compile | 0m 56s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 56s | | the patch passed |
| +1 :green_heart: | compile | 0m 46s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 46s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3407/3/artifact/out/blanks-eol.txt)
| The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | checkstyle | 0m 38s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 50s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 37s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 35s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 0s | | the patch passed |
| +1 :green_heart: | shadedclient | 17m 19s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 101m 41s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 184m 52s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3407/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3407 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 37bf46b6110b 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14
16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6c7c4a97a5eab784a344bf1b84b29eb6395d6edf |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3407/3/testReport/ |
| Max. process+thread count | 939 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3407/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT
[GitHub] [hadoop] hadoop-yetus commented on pull request #3414: YARN-10938. Support reservation scheduling enabled switch for capacity scheduler
hadoop-yetus commented on pull request #3414:
URL: https://github.com/apache/hadoop/pull/3414#issuecomment-916303263
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 17s | | trunk passed |
| +1 :green_heart: | compile | 1m 2s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 56s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 52s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 1s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 49s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 52s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 12s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 50s | | the patch passed |
| +1 :green_heart: | compile | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 53s | | the patch passed |
| +1 :green_heart: | compile | 0m 47s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 47s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 43s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3414/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 13 new + 580 unchanged - 0 fixed = 593 total (was 580) |
| +1 :green_heart: | mvnsite | 0m 50s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 37s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 52s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 43s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 95m 43s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 172m 10s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3414/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3414 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 38b05c592fc1 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28
05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f16865bcbb049dbe1cc42f989f9700bac638fe6f |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3414/1/testReport/ |
| Max. process+thread count | 974 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
[jira] [Work logged] (HADOOP-17901) Performance degradation in Text.append() after HADOOP-16951
[
https://issues.apache.org/jira/browse/HADOOP-17901?focusedWorklogId=648761=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648761
]
ASF GitHub Bot logged work on HADOOP-17901:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 17:29
Start Date: 09/Sep/21 17:29
Worklog Time Spent: 10m
Work Description: goiri commented on a change in pull request #3411:
URL: https://github.com/apache/hadoop/pull/3411#discussion_r705553685
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
##
@@ -34,7 +34,6 @@
import java.util.Arrays;
import org.apache.avro.reflect.Stringable;
-
Review comment:
Avoid this change.
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
##
@@ -302,7 +300,10 @@ public void clear() {
*/
private boolean ensureCapacity(final int capacity) {
if (bytes.length < capacity) {
- bytes = new byte[capacity];
+ // Try to expand the backing array by the factor of 1.5x
+ // (by taking the current size + diving it by half)
Review comment:
"Try to expand the backing array by a factor of 1.5x
(by taking the current size + dividing it by half)"
Fix the warning too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648761)
Time Spent: 40m (was: 0.5h)
> Performance degradation in Text.append() after HADOOP-16951
> ---
>
> Key: HADOOP-17901
> URL: https://issues.apache.org/jira/browse/HADOOP-17901
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Labels: pull-request-available
> Attachments: HADOOP-17901-001.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> We discovered a serious performance degradation in {{Text.append()}}.
> The problem is that the logic which intends to increase the size of the
> backing array does not work as intended.
> It's very difficult to spot, so I added extra logs to see what happens.
> Let's add 4096 bytes of textual data in a loop:
> {noformat}
> public static void main(String[] args) {
> Text text = new Text();
> String toAppend = RandomStringUtils.randomAscii(4096);
> for(int i = 0; i < 100; i++) {
> text.append(toAppend.getBytes(), 0, 4096);
> }
> }
> {noformat}
> With some debug printouts, we can observe:
> {noformat}
> 2021-09-08 13:35:29,528 INFO [main] io.Text (Text.java:append(251)) -
> length: 24576, len: 4096, utf8ArraySize: 4096, bytes.length: 30720
> 2021-09-08 13:35:29,528 INFO [main] io.Text (Text.java:append(253)) - length
> + (length >> 1): 36864
> 2021-09-08 13:35:29,528 INFO [main] io.Text (Text.java:append(254)) - length
> + len: 28672
> 2021-09-08 13:35:29,528 INFO [main] io.Text (Text.java:ensureCapacity(287))
> - >>> enhancing capacity from 30720 to 36864
> 2021-09-08 13:35:29,528 INFO [main] io.Text (Text.java:append(251)) -
> length: 28672, len: 4096, utf8ArraySize: 4096, bytes.length: 36864
> 2021-09-08 13:35:29,528 INFO [main] io.Text (Text.java:append(253)) - length
> + (length >> 1): 43008
> 2021-09-08 13:35:29,529 INFO [main] io.Text (Text.java:append(254)) - length
> + len: 32768
> 2021-09-08 13:35:29,529 INFO [main] io.Text (Text.java:ensureCapacity(287))
> - >>> enhancing capacity from 36864 to 43008
> 2021-09-08 13:35:29,529 INFO [main] io.Text (Text.java:append(251)) -
> length: 32768, len: 4096, utf8ArraySize: 4096, bytes.length: 43008
> 2021-09-08 13:35:29,529 INFO [main] io.Text (Text.java:append(253)) - length
> + (length >> 1): 49152
> 2021-09-08 13:35:29,529 INFO [main] io.Text (Text.java:append(254)) - length
> + len: 36864
> 2021-09-08 13:35:29,529 INFO [main] io.Text (Text.java:ensureCapacity(287))
> - >>> enhancing capacity from 43008 to 49152
> ...
> {noformat}
> After a certain number of {{append()}} calls, subsequent capacity increments
> are small.
> It's because the difference between two {{length + (length >> 1)}} values is
> always 6144 bytes. Because the size of the backing array is trailing behind
> the calculated value, the increment will also be 6144 bytes. This means that
> new arrays are constantly created.
> Suggested solution: don't calculate the capacity in advance based on length.
> Instead, pass the required minimum to {{ensureCapacity()}}. Then the
>
[GitHub] [hadoop] goiri commented on a change in pull request #3411: HADOOP-17901. Performance degradation in Text.append() after HADOOP-1…
goiri commented on a change in pull request #3411:
URL: https://github.com/apache/hadoop/pull/3411#discussion_r705553685
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
##
@@ -34,7 +34,6 @@
import java.util.Arrays;
import org.apache.avro.reflect.Stringable;
-
Review comment:
Avoid this change.
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
##
@@ -302,7 +300,10 @@ public void clear() {
*/
private boolean ensureCapacity(final int capacity) {
if (bytes.length < capacity) {
- bytes = new byte[capacity];
+ // Try to expand the backing array by the factor of 1.5x
+ // (by taking the current size + diving it by half)
Review comment:
"Try to expand the backing array by a factor of 1.5x
(by taking the current size + dividing it by half)"
Fix the warning too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17904) Test Result Not Working In Jenkins Result
[ https://issues.apache.org/jira/browse/HADOOP-17904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412731#comment-17412731 ] Íñigo Goiri commented on HADOOP-17904: -- [~gautham] please take a look too. > Test Result Not Working In Jenkins Result > - > > Key: HADOOP-17904 > URL: https://issues.apache.org/jira/browse/HADOOP-17904 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.4.0 >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > The jenkins used to show the Test Results, now the list is broken and No PR > has that working. > eg: > [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3404/1/testReport/] > > For Daily Build No Test Result as well: > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/622/ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17904) Test Result Not Working In Jenkins Result
[ https://issues.apache.org/jira/browse/HADOOP-17904?focusedWorklogId=648759=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648759 ] ASF GitHub Bot logged work on HADOOP-17904: --- Author: ASF GitHub Bot Created on: 09/Sep/21 17:24 Start Date: 09/Sep/21 17:24 Worklog Time Spent: 10m Work Description: goiri commented on pull request #3413: URL: https://github.com/apache/hadoop/pull/3413#issuecomment-916291944 This looks good but I don't have a lot of experience here. @GauthamBanasandra you have been working around here, can you confirm? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 648759) Time Spent: 0.5h (was: 20m) > Test Result Not Working In Jenkins Result > - > > Key: HADOOP-17904 > URL: https://issues.apache.org/jira/browse/HADOOP-17904 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.4.0 >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > The jenkins used to show the Test Results, now the list is broken and No PR > has that working. > eg: > [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3404/1/testReport/] > > For Daily Build No Test Result as well: > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/622/ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on pull request #3413: HADOOP-17904. Test Result Not Working In Jenkins Result.
goiri commented on pull request #3413: URL: https://github.com/apache/hadoop/pull/3413#issuecomment-916291944 This looks good but I don't have a lot of experience here. @GauthamBanasandra you have been working around here, can you confirm? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3352: YARN-10903. Fix the headroom check in ParentQueue and RegularContainerAllocator for DRF
hadoop-yetus commented on pull request #3352:
URL: https://github.com/apache/hadoop/pull/3352#issuecomment-916288511
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 3s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 52s | | trunk passed |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 5s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 12s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 36s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 3s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 1s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 1s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 55s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 2s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 42s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 31s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 3s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 111m 28s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 206m 25s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3352/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3352 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux dbe91e62fbba 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fc5730393f1ecfcfd0443e86391aba77b05dea18 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3352/3/testReport/ |
| Max. process+thread count | 926 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3352/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648747=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648747
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 17:09
Start Date: 09/Sep/21 17:09
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-916281799
(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3385/15/console in
case of problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648747)
Time Spent: 11h 10m (was: 11h)
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 11h 10m
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
hadoop-yetus commented on pull request #3385: URL: https://github.com/apache/hadoop/pull/3385#issuecomment-916281799 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3385/15/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3352: YARN-10903. Fix the headroom check in ParentQueue and RegularContainerAllocator for DRF
hadoop-yetus commented on pull request #3352:
URL: https://github.com/apache/hadoop/pull/3352#issuecomment-916269624
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 18s | | trunk passed |
| +1 :green_heart: | compile | 1m 3s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 54s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 48s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 57s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 38s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 52s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 36s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 50s | | the patch passed |
| +1 :green_heart: | compile | 0m 56s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 56s | | the patch passed |
| +1 :green_heart: | compile | 0m 46s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 46s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 41s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 50s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 56s | | the patch passed |
| +1 :green_heart: | shadedclient | 17m 26s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 99m 37s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 183m 40s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3352/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3352 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 913c90a4649c 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14
16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fc5730393f1ecfcfd0443e86391aba77b05dea18 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3352/2/testReport/ |
| Max. process+thread count | 931 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3352/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For
[GitHub] [hadoop] brumi1024 commented on pull request #3396: YARN-10872. Replace getPropsWithPrefix calls in AutoCreatedQueueTemplate
brumi1024 commented on pull request #3396: URL: https://github.com/apache/hadoop/pull/3396#issuecomment-916258213 Thanks @9uapaw for the review, edited the code a bit and extended a test case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2971: MAPREDUCE-7341. Intermediate Manifest Committer
steveloughran commented on pull request #2971: URL: https://github.com/apache/hadoop/pull/2971#issuecomment-916256532 here's a dump of the aggregate IOStats of all _SUCCESS files created across the entire terasort jobs. ``` 2021-09-09 17:06:23,935 INFO [JUnit-test_140_teracomplete]: statistics.IOStatisticsLogging (IOStatisticsLogging.java:logIOStatisticsAtLevel(269)) - IOStatistics: counters=((committer_bytes_committed=200021) (committer_commit_job=3) (committer_files_committed=4) (committer_task_directory_depth=7) (committer_task_file_count=4) (committer_task_file_size=200021) (op_create_directories=3) (op_delete=7) (op_directory_scan=3) (op_get_file_status=17) (op_get_file_status.failures=10) (op_job_stage_create_target_dirs=3) (op_job_stage_load_manifests=3) (op_job_stage_rename_files=3) (op_job_stage_setup=3) (op_list_status=10) (op_load_all_manifests=3) (op_load_manifest=7) (op_mkdirs=10) (op_rename=4) (op_task_stage_commit=7) (op_task_stage_scan_directory=7) (op_task_stage_setup=7)); gauges=(); minimums=((committer_task_directory_count=0) (committer_task_directory_depth=1) (committer_task_file_count=0) (committer_task_file_size=0) (op_create_directories.min=0) (op_delete.min=30) (op_directory_scan.min=73) (op_get_file_status.failures.min=25) (op_get_file_status.min=24) (op_job_stage_create_target_dirs.min=0) (op_job_stage_load_manifests.min=230) (op_job_stage_rename_files.min=111) (op_job_stage_setup.min=196) (op_list_status.min=28) (op_load_all_manifests.min=151) (op_load_manifest.min=137) (op_mkdirs.min=28) (op_rename.min=54) (op_task_stage_commit.min=224) (op_task_stage_scan_directory.min=224) (op_task_stage_setup.min=81)); maximums=((committer_task_directory_count=0) (committer_task_directory_depth=1) (committer_task_file_count=1) (committer_task_file_size=10) (op_create_directories.max=1) (op_delete.max=43) (op_directory_scan.max=113) (op_get_file_status.failures.max=309) (op_get_file_status.max=30) (op_job_stage_create_target_dirs.max=1) (op_job_stage_load_manifests.max=324) (op_job_stage_rename_files.max=120) (op_job_stage_setup.max=449) (op_list_status.max=37) (op_load_all_manifests.max=218) (op_load_manifest.max=199) (op_mkdirs.max=152) (op_rename.max=59) (op_task_stage_commit.max=806) (op_task_stage_scan_directory.max=806) (op_task_stage_setup.max=277)); means=((committer_task_directory_count=(samples=7, sum=0, mean=0.)) (committer_task_directory_depth=(samples=7, sum=7, mean=1.)) (committer_task_file_count=(samples=7, sum=4, mean=0.5714)) (committer_task_file_size=(samples=7, sum=200021, mean=28574.4286)) (op_create_directories.mean=(samples=3, sum=1, mean=0.)) (op_delete.mean=(samples=7, sum=240, mean=34.2857)) (op_directory_scan.mean=(samples=3, sum=283, mean=94.)) (op_get_file_status.failures.mean=(samples=10, sum=775, mean=77.5000)) (op_get_file_status.mean=(samples=7, sum=181, mean=25.8571)) (op_job_stage_create_target_dirs.mean=(samples=3, sum=2, mean=0.6667)) (op_job_stage_load_manifests.mean=(samples=3, sum=868, mean=289.)) (op_job_stage_rename_files.mean=(samples=3, sum=345, mean=115.)) (op_job_stage_setup.mean=(samples=3, sum=940, mean=313.)) (op_list_status.mean=(samples=10, sum=320, mean=32.)) (op_load_all_manifests.mean=(samples=3, sum=564, mean=188.)) (op_load_manifest.mean=(samples=7, sum=1171, mean=167.2857)) (op_mkdirs.mean=(samples=10, sum=717, mean=71.7000)) (op_rename.mean=(samples=4, sum=230, mean=57.5000)) (op_task_stage_commit.mean=(samples=7, sum=3186, mean=455.1429)) (op_task_stage_scan_directory.mean=(samples=7, sum=3186, mean=455.1429)) (op_task_stage_setup.mean=(samples=7, sum=881, mean=125.8571))); ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2971: MAPREDUCE-7341. Intermediate Manifest Committer
hadoop-yetus removed a comment on pull request #2971:
URL: https://github.com/apache/hadoop/pull/2971#issuecomment-915655232
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 59s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 2s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 23 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 53s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 13s | | trunk passed |
| +1 :green_heart: | compile | 28m 31s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 24m 18s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 16s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 9s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 36s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 36s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 18m 11s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 43s | | the patch passed |
| +1 :green_heart: | compile | 28m 21s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 28m 21s |
[/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2971/33/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 2 new + 1913 unchanged - 0
fixed = 1915 total (was 1913) |
| +1 :green_heart: | compile | 24m 19s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 24m 19s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2971/33/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 2 new + 1790
unchanged - 0 fixed = 1792 total (was 1790) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 21s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2971/33/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 29 new + 6 unchanged - 0 fixed = 35 total (was 6)
|
| +1 :green_heart: | mvnsite | 4m 20s | | the patch passed |
| +1 :green_heart: | xml | 0m 11s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 3m 8s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 37s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 34s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 18m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 31s | | hadoop-project in the patch
passed. |
| -1 :x: | unit | 19m 27s |
[/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2971/33/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 9m 13s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | unit | 2m 31s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 56s | | The patch does not
generate ASF License warnings. |
| | | 267m 26s | | |
| Reason | Tests |
[GitHub] [hadoop] goiri merged pull request #3325: YARN-10893. Adding metrics for getClusterMetrics and getApplications APIs in FederationClientInterceptor
goiri merged pull request #3325: URL: https://github.com/apache/hadoop/pull/3325 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17871) S3A CSE: minor tuning
[
https://issues.apache.org/jira/browse/HADOOP-17871?focusedWorklogId=648688=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648688
]
ASF GitHub Bot logged work on HADOOP-17871:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 15:37
Start Date: 09/Sep/21 15:37
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3412:
URL: https://github.com/apache/hadoop/pull/3412#issuecomment-916212259
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 59s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 19 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 27s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 7s | | trunk passed |
| +1 :green_heart: | compile | 23m 6s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 37s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 51s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 32s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 11s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 7s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 31s | | the patch passed |
| +1 :green_heart: | compile | 22m 17s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 22m 17s | | the patch passed |
| +1 :green_heart: | compile | 19m 38s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 19m 38s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3412/1/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 3m 49s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3412/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 6 new + 170 unchanged - 5 fixed = 176 total (was
175) |
| +1 :green_heart: | mvnsite | 2m 23s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 1m 31s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 14s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 1s | | the patch passed |
| +1 :green_heart: | shadedclient | 17m 21s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 16m 51s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 33s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not
generate ASF License warnings. |
| | | 208m 15s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3412/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3412 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml markdownlint |
| uname | Linux c7e848c98062 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
|
[GitHub] [hadoop] hadoop-yetus commented on pull request #3412: HADOOP-17871. S3A CSE: minor tuning
hadoop-yetus commented on pull request #3412:
URL: https://github.com/apache/hadoop/pull/3412#issuecomment-916212259
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 59s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 19 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 27s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 7s | | trunk passed |
| +1 :green_heart: | compile | 23m 6s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 37s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 51s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 32s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 11s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 7s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 31s | | the patch passed |
| +1 :green_heart: | compile | 22m 17s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 22m 17s | | the patch passed |
| +1 :green_heart: | compile | 19m 38s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 19m 38s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3412/1/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 3m 49s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3412/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 6 new + 170 unchanged - 5 fixed = 176 total (was
175) |
| +1 :green_heart: | mvnsite | 2m 23s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 1m 31s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 14s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 1s | | the patch passed |
| +1 :green_heart: | shadedclient | 17m 21s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 16m 51s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 33s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not
generate ASF License warnings. |
| | | 208m 15s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3412/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3412 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml markdownlint |
| uname | Linux c7e848c98062 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1516d40394ea9f088ae673cb7293d4b25af846e6 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
[GitHub] [hadoop] 9uapaw commented on a change in pull request #3403: YARN-10917. Investigate and simplify CapacitySchedulerConfigValidator…
9uapaw commented on a change in pull request #3403:
URL: https://github.com/apache/hadoop/pull/3403#discussion_r705458694
##
File path:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfigValidator.java
##
@@ -125,76 +133,107 @@ public static void validateQueueHierarchy(
CapacitySchedulerConfiguration newConf) throws IOException {
// check that all static queues are included in the newQueues list
for (CSQueue oldQueue : queues.getQueues()) {
- if (!(AbstractAutoCreatedLeafQueue.class.isAssignableFrom(
- oldQueue.getClass( {
-String queuePath = oldQueue.getQueuePath();
-CSQueue newQueue = newQueues.get(queuePath);
-String configPrefix = newConf.getQueuePrefix(
-oldQueue.getQueuePath());
-String state = newConf.get(configPrefix + "state");
-QueueState newQueueState = null;
-if (state != null) {
- try {
-newQueueState = QueueState.valueOf(state);
- } catch (Exception ex) {
-LOG.warn("Not a valid queue state for queue "
-+ oldQueue.getQueuePath());
+ if
(AbstractAutoCreatedLeafQueue.class.isAssignableFrom(oldQueue.getClass())) {
+continue;
+ }
+
+ final String queuePath = oldQueue.getQueuePath();
+ final String configPrefix =
CapacitySchedulerConfiguration.getQueuePrefix(
+ oldQueue.getQueuePath());
+ final QueueState newQueueState = getQueueState(oldQueue,
newConf.get(configPrefix + "state"));
+ final CSQueue newQueue = newQueues.get(queuePath);
+
+ if (null == newQueue) {
+// old queue doesn't exist in the new XML
+if (oldQueue.getState() == QueueState.STOPPED || newQueueState ==
QueueState.STOPPED) {
+ LOG.info("Deleting Queue {}, as it is not present in the modified
capacity " +
+ "configuration xml", queuePath);
+} else {
+ if (!isDynamicQueue(oldQueue)) {
+throw new IOException(oldQueue.getQueuePath() + " cannot be"
++ " deleted from the capacity scheduler configuration, as the"
++ " queue is not yet in stopped state. Current State : "
++ oldQueue.getState());
}
}
-if (null == newQueue) {
- // old queue doesn't exist in the new XML
- if (oldQueue.getState() == QueueState.STOPPED ||
- newQueueState == QueueState.STOPPED) {
-LOG.info("Deleting Queue " + queuePath + ", as it is not"
-+ " present in the modified capacity configuration xml");
- } else {
-if (!isDynamicQueue(oldQueue)) {
- throw new IOException(oldQueue.getQueuePath() + " cannot be"
- + " deleted from the capacity scheduler configuration, as
the"
- + " queue is not yet in stopped state. Current State : "
- + oldQueue.getState());
-}
- }
-} else if (!oldQueue.getQueuePath().equals(newQueue.getQueuePath())) {
- //Queue's cannot be moved from one hierarchy to other
- throw new IOException(
-queuePath + " is moved from:" + oldQueue.getQueuePath() + " to:"
- + newQueue.getQueuePath()
- + " after refresh, which is not allowed.");
-} else if (oldQueue instanceof ParentQueue
-&& !(oldQueue instanceof ManagedParentQueue)
-&& newQueue instanceof ManagedParentQueue) {
+ } else {
+for (QueueChangeValidator validator : QUEUE_CHANGE_VALIDATORS) {
+ validator.validate(oldQueue, newQueue);
+}
+ }
+}
+ }
+
+ private static QueueState getQueueState(CSQueue queue, String state) {
Review comment:
This method is a bit misleading. It is creating a new state from a
string. Also it does not need the full queue as its argument, only the path.
##
File path:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfigValidator.java
##
@@ -125,76 +133,107 @@ public static void validateQueueHierarchy(
CapacitySchedulerConfiguration newConf) throws IOException {
// check that all static queues are included in the newQueues list
for (CSQueue oldQueue : queues.getQueues()) {
- if (!(AbstractAutoCreatedLeafQueue.class.isAssignableFrom(
- oldQueue.getClass( {
-String queuePath = oldQueue.getQueuePath();
-CSQueue newQueue = newQueues.get(queuePath);
-String configPrefix = newConf.getQueuePrefix(
-oldQueue.getQueuePath());
-String state = newConf.get(configPrefix + "state");
-QueueState newQueueState = null;
-if (state != null) {
-
[jira] [Work logged] (HADOOP-17892) Add Hadoop code formatter in dev-support
[ https://issues.apache.org/jira/browse/HADOOP-17892?focusedWorklogId=648655=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648655 ] ASF GitHub Bot logged work on HADOOP-17892: --- Author: ASF GitHub Bot Created on: 09/Sep/21 15:05 Start Date: 09/Sep/21 15:05 Worklog Time Spent: 10m Work Description: virajjasani edited a comment on pull request #3387: URL: https://github.com/apache/hadoop/pull/3387#issuecomment-916183454 > What is the source of these xmls, you have written them? Not exactly, these are generated from IDE only, so we are good. I have also reimported them to re-verify the behaviour :) And yes, import related checkstyle warnings are also taken care of with this formatter. However, if we want to change import order for thirdparty shaded guava, I believe we can first do it on checkstyle.xml and then update formatter accordingly. WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 648655) Time Spent: 2.5h (was: 2h 20m) > Add Hadoop code formatter in dev-support > > > Key: HADOOP-17892 > URL: https://issues.apache.org/jira/browse/HADOOP-17892 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > We should add Hadoop code formatter xml to dev-support specifically for new > developers to refer to. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani edited a comment on pull request #3387: HADOOP-17892. Add Hadoop code formatter in dev-support
virajjasani edited a comment on pull request #3387: URL: https://github.com/apache/hadoop/pull/3387#issuecomment-916183454 > What is the source of these xmls, you have written them? Not exactly, these are generated from IDE only, so we are good. I have also reimported them to re-verify the behaviour :) And yes, import related checkstyle warnings are also taken care of with this formatter. However, if we want to change import order for thirdparty shaded guava, I believe we can first do it on checkstyle.xml and then update formatter accordingly. WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17892) Add Hadoop code formatter in dev-support
[ https://issues.apache.org/jira/browse/HADOOP-17892?focusedWorklogId=648654=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648654 ] ASF GitHub Bot logged work on HADOOP-17892: --- Author: ASF GitHub Bot Created on: 09/Sep/21 15:04 Start Date: 09/Sep/21 15:04 Worklog Time Spent: 10m Work Description: virajjasani commented on pull request #3387: URL: https://github.com/apache/hadoop/pull/3387#issuecomment-916183454 > What is the source of these xmls, you have written them? Not exactly, these are generated from IDE only, so we are good :) And yes, import related checkstyle warnings are also taken care of with this formatter. However, if we want to change import order for thirdparty shaded guava, I believe we can first do it on checkstyle.xml and then update formatter accordingly. WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 648654) Time Spent: 2h 20m (was: 2h 10m) > Add Hadoop code formatter in dev-support > > > Key: HADOOP-17892 > URL: https://issues.apache.org/jira/browse/HADOOP-17892 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > We should add Hadoop code formatter xml to dev-support specifically for new > developers to refer to. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #3387: HADOOP-17892. Add Hadoop code formatter in dev-support
virajjasani commented on pull request #3387: URL: https://github.com/apache/hadoop/pull/3387#issuecomment-916183454 > What is the source of these xmls, you have written them? Not exactly, these are generated from IDE only, so we are good :) And yes, import related checkstyle warnings are also taken care of with this formatter. However, if we want to change import order for thirdparty shaded guava, I believe we can first do it on checkstyle.xml and then update formatter accordingly. WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] 9uapaw commented on a change in pull request #3396: YARN-10872. Replace getPropsWithPrefix calls in AutoCreatedQueueTemplate
9uapaw commented on a change in pull request #3396:
URL: https://github.com/apache/hadoop/pull/3396#discussion_r705425384
##
File path:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ConfigurationProperties.java
##
@@ -132,7 +132,8 @@ private void copyProperties(
String prefixToTrim = trimPrefix;
if (!trimPrefix.isEmpty()) {
-if (!key.equals(trimPrefix)) {
+if (!key.equals(trimPrefix) &&
+!trimPrefix.endsWith(CapacitySchedulerConfiguration.DOT)) {
Review comment:
Could you make a test case for this scenario?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on pull request #3387: HADOOP-17892. Add Hadoop code formatter in dev-support
ayushtkn commented on pull request #3387: URL: https://github.com/apache/hadoop/pull/3387#issuecomment-916167736 I am quite convinced with having a formatter checked in to the code. Mostly, if it can take care of import ordering. With a exception that we keep the shaded guava imports at the same place as we would have kept, if it was a normal guava import to keep backporting easy(I was told to do so in one PR) I think I missed the thread on common-dev, If others are ok, then it should be fine. What is the source of these xmls, you have written them? If yes, we need someone who can decode this as well, Which I am certainly not :-) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17892) Add Hadoop code formatter in dev-support
[ https://issues.apache.org/jira/browse/HADOOP-17892?focusedWorklogId=648640=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648640 ] ASF GitHub Bot logged work on HADOOP-17892: --- Author: ASF GitHub Bot Created on: 09/Sep/21 14:45 Start Date: 09/Sep/21 14:45 Worklog Time Spent: 10m Work Description: ayushtkn commented on pull request #3387: URL: https://github.com/apache/hadoop/pull/3387#issuecomment-916167736 I am quite convinced with having a formatter checked in to the code. Mostly, if it can take care of import ordering. With a exception that we keep the shaded guava imports at the same place as we would have kept, if it was a normal guava import to keep backporting easy(I was told to do so in one PR) I think I missed the thread on common-dev, If others are ok, then it should be fine. What is the source of these xmls, you have written them? If yes, we need someone who can decode this as well, Which I am certainly not :-) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 648640) Time Spent: 2h 10m (was: 2h) > Add Hadoop code formatter in dev-support > > > Key: HADOOP-17892 > URL: https://issues.apache.org/jira/browse/HADOOP-17892 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > We should add Hadoop code formatter xml to dev-support specifically for new > developers to refer to. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] FrankinRUC opened a new pull request #3414: YARN-10938. Support reservation scheduling enabled switch for capacity scheduler
FrankinRUC opened a new pull request #3414: URL: https://github.com/apache/hadoop/pull/3414 ### Description of PR Currently the CapacityScheduler uses reservations in order to handle requests for large containers and the fact there might not currently be enough space available on a single host. But this algorithm is not suitable for small cluster which only have very limited resources. So we can add a switch property in capacity scheduler's configuration to avoid reservation scheduling in these use cases. CHANGES: Add "yarn.scheduler.capacity.reservation.enabled" in capacity scheduler's configuration. ### How was this patch tested? Add a unit test for switch in resourcemanager.scheduler.capacity.TestReservations ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomicooler commented on a change in pull request #3407: YARN-10910. AbstractCSQueue#setupQueueConfigs: Separate validation lo…
tomicooler commented on a change in pull request #3407:
URL: https://github.com/apache/hadoop/pull/3407#discussion_r705361455
##
File path:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
##
@@ -1012,17 +1019,23 @@ private long getInheritedDefaultAppLifetime(CSQueue q,
// lifetime. Otherwise, use current queue's max lifetime value for its
// default lifetime.
if (defaultAppLifetimeWasSpecifiedInConfig) {
-if (parentsDefaultAppLifetime <= myMaxAppLifetime) {
- defaultAppLifetime = parentsDefaultAppLifetime;
-} else {
- defaultAppLifetime = myMaxAppLifetime;
-}
+defaultAppLifetime =
+Math.min(parentsDefaultAppLifetime, myMaxAppLifetime);
} else {
// Default app lifetime value was not set anywhere in this queue's
// hierarchy. Use current queue's max lifetime as its default.
defaultAppLifetime = myMaxAppLifetime;
}
} // else if >= 0, default lifetime was set at this level. Just use it.
+
+if (myMaxAppLifetime > 0 &&
+defaultAppLifetime > myMaxAppLifetime) {
+ throw new YarnRuntimeException(
+ "Default lifetime " + defaultAppLifetime
+ + " can't exceed maximum lifetime " + myMaxAppLifetime);
+} else if (defaultAppLifetime <= 0) {
Review comment:
The previous if statement sets the value to
min(**parentsDefaultAppLifetime**, **myMaxAppLifetime**) or
**myMaxAppLifetime** already. Maybe there is a case when the
**parentsDefaultAppLifetime** is <= 0, that's the only reason that this was
done later:
`defaultApplicationLifetime = defaultApplicationLifetime > 0 ?
defaultApplicationLifetime : maxApplicationLifetime;`
NIT: Anyway I think this should not be an **else if**, just a standalone
**if**. That's easier to comprehend.
```
// else if...
!(myMaxAppLifetime > 0 && defaultAppLifetime > myMaxAppLifetime) &&
(defaultAppLifetime <= 0)
(myMaxAppLifetime <= 0 || defaultAppLifetime <= myMaxAppLifetime) &&
(defaultAppLifetime <= 0)
// Let's say defaultAppLifetime=0
myMaxAppLifteTime <= 0 || 0 <= myMaxAppLifetime
// So this has nothing to do with the myMaxAppLifetime
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
