[GitHub] [hadoop] hadoop-yetus commented on pull request #4026: MAPREDUCE-7372 MapReduce set permission too late in copyJar method
hadoop-yetus commented on PR #4026:
URL: https://github.com/apache/hadoop/pull/4026#issuecomment-1186789561
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 44s | | trunk passed |
| +1 :green_heart: | compile | 0m 56s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 49s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 52s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 56s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 2s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 31s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 42s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 32s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 10s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 7m 0s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 108m 0s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4026/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4026 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 0c135e37946d 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b46f6becc990f2c477758f0949fab4f41e166d35 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4026/4/testReport/ |
| Max. process+thread count | 1034 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4026/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an
[GitHub] [hadoop] Hexiaoqiao commented on a diff in pull request #4541: HDFS-16655. OIV: print out erasure coding policy name in oiv Delimited output
Hexiaoqiao commented on code in PR #4541:
URL: https://github.com/apache/hadoop/pull/4541#discussion_r922981175
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageTextWriter.java:
##
@@ -17,16 +17,7 @@
*/
package org.apache.hadoop.hdfs.tools.offlineImageViewer;
-import java.io.BufferedInputStream;
-import java.io.Closeable;
-import java.io.File;
-import java.io.FileInputStream;
-import java.io.FileOutputStream;
-import java.io.IOException;
-import java.io.InputStream;
-import java.io.PrintStream;
-import java.io.RandomAccessFile;
-import java.io.UnsupportedEncodingException;
+import java.io.*;
Review Comment:
Please expand the imports rather than use wildcard.
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageViewerPB.java:
##
@@ -80,6 +80,7 @@ public class OfflineImageViewerPB {
+ "delimiter. The default delimiter is \\t, though this may be\n"
+ "changed via the -delimiter argument.\n"
+ "-sp print storage policy, used by delimiter only.\n"
+ + "-ec print erasure coding policy policy, used by delimiter only.\n"
Review Comment:
duplicate word `policy` here.
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageDelimitedTextWriter.java:
##
@@ -146,14 +168,21 @@ public String build() {
PBImageDelimitedTextWriter(PrintStream out, String delimiter,
String tempPath, boolean printStoragePolicy)
throws IOException {
-this(out, delimiter, tempPath, printStoragePolicy, 1, "-");
+this(out, delimiter, tempPath, printStoragePolicy, 1, "-", null, false);
}
PBImageDelimitedTextWriter(PrintStream out, String delimiter,
- String tempPath, boolean printStoragePolicy, int threads,
- String parallelOut) throws IOException {
+ String tempPath, boolean printStoragePolicy,
+ int threads, String parallelOut,
+ Configuration conf, boolean printECPolicy)
Review Comment:
I think it may be better to define this method as the following.
```
PBImageDelimitedTextWriter(PrintStream out, String delimiter,
String tempPath, boolean printStoragePolicy,
boolean printECPolicy, int threads, String
parallelOut,
Configuration conf)
```
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewerForErasureCodingPolicy.java:
##
@@ -0,0 +1,181 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.tools.offlineImageViewer;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.DFSTestUtil;
+import org.apache.hadoop.hdfs.DistributedFileSystem;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.hdfs.protocol.HdfsConstants;
+import org.apache.hadoop.hdfs.server.namenode.FSImageTestUtil;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.*;
Review Comment:
Please expand this wildcard import.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ferhui commented on a diff in pull request #4530: HDFS-15079. RBF: Namenode needs to use the actual client Id and callId when going through RBF proxy.
ferhui commented on code in PR #4530:
URL: https://github.com/apache/hadoop/pull/4530#discussion_r922982928
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:
##
@@ -495,6 +498,94 @@ public static NameNodeMetrics getNameNodeMetrics() {
return metrics;
}
+ /**
+ * Try to obtain the actual client info according to the current user.
+ * @param ipProxyUsers Users who can override client infos
+ */
+ private static String clientInfoFromContext(
+ final String[] ipProxyUsers) {
+if (ipProxyUsers != null) {
+ UserGroupInformation user =
+ UserGroupInformation.getRealUserOrSelf(Server.getRemoteUser());
+ if (user != null &&
+ ArrayUtils.contains(ipProxyUsers, user.getShortUserName())) {
+CallerContext context = CallerContext.getCurrent();
+if (context != null && context.isContextValid()) {
+ return context.getContext();
+}
+ }
+}
+return null;
+ }
+
+ /**
+ * Try to obtain the value corresponding to the key by parsing the content.
+ * @param content the full content to be parsed.
+ * @param key trying to obtain the value of the key.
+ * @return the value corresponding to the key.
+ */
+ @VisibleForTesting
+ public static String parseSpecialValue(String content, String key) {
+int posn = content.indexOf(key);
+if (posn != -1) {
+ posn += key.length();
+ int end = content.indexOf(",", posn);
+ return end == -1 ? content.substring(posn)
+ : content.substring(posn, end);
+}
+return null;
+ }
+
+ /**
+ * Try to obtain the actual client's machine according to the current user.
+ * @param ipProxyUsers Users who can override client infos.
+ * @return The actual client's machine.
+ */
+ public static String getClientMachine(final String[] ipProxyUsers) {
Review Comment:
one question here. Where do we use this method? Didn't find it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] GauthamBanasandra opened a new pull request, #4576: HDFS-16667. Use malloc for buffer allocation in uriparser2
GauthamBanasandra opened a new pull request, #4576: URL: https://github.com/apache/hadoop/pull/4576 ### Description of PR Currently, a variable is used to specify the array size in uriparser2 - https://github.com/apache/hadoop/blob/34e548cb62ed21c5bba7a82f5f1489ca6bdfb8c4/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/uriparser2/uriparser2/uriparser2.c#L71-L79 This results in the following error on Windows - ``` H:\hadoop-hdfs-project\hadoop-hdfs-native-client\src\out\build\x64-Debug\cl : command line warning D9025: overriding '/W4' with '/w' uriparser2.c H:\hadoop-hdfs-project\hadoop-hdfs-native-client\src\main\native\libhdfspp\third_party\uriparser2\uriparser2\uriparser2.c(74,23): error C2057: expected constant expression H:\hadoop-hdfs-project\hadoop-hdfs-native-client\src\main\native\libhdfspp\third_party\uriparser2\uriparser2\uriparser2.c(74,23): error C2466: cannot allocate an array of constant size 0 H:\hadoop-hdfs-project\hadoop-hdfs-native-client\src\main\native\libhdfspp\third_party\uriparser2\uriparser2\uriparser2.c(74,24): error C2133: 'buffer': unknown size ``` I've used malloc to fix this. ### How was this patch tested? 1. Tested this locally by building on my Windows system. 2. Hadoop Jenkins CI validation. ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17461) Add thread-level IOStatistics Context
[ https://issues.apache.org/jira/browse/HADOOP-17461?focusedWorklogId=791846=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-791846 ] ASF GitHub Bot logged work on HADOOP-17461: --- Author: ASF GitHub Bot Created on: 18/Jul/22 05:07 Start Date: 18/Jul/22 05:07 Worklog Time Spent: 10m Work Description: mehakmeet commented on PR #4352: URL: https://github.com/apache/hadoop/pull/4352#issuecomment-1186772913 Cherry-picked S3A committers integration and lining up with spark build. Some javadocs would need modification, adding that in a commit. Need to see how to test this on a spark build locally, since spark might not be using the correct hadoop version iirc, then we could get this in Issue Time Tracking --- Worklog Id: (was: 791846) Time Spent: 5h 10m (was: 5h) > Add thread-level IOStatistics Context > - > > Key: HADOOP-17461 > URL: https://issues.apache.org/jira/browse/HADOOP-17461 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/azure, fs/s3 >Affects Versions: 3.3.1 >Reporter: Steve Loughran >Assignee: Mehakmeet Singh >Priority: Major > Labels: pull-request-available > Time Spent: 5h 10m > Remaining Estimate: 0h > > For effective reporting of the iostatistics of individual worker threads, we > need a thread-level context which IO components update. > * this contact needs to be passed in two background thread forming work on > behalf of a task. > * IO Components (streams, iterators, filesystems) need to update this context > statistics as they perform work > * Without double counting anything. > I imagine a ThreadLocal IOStatisticContext which will be updated in the > FileSystem API Calls. This context MUST be passed into the background threads > used by a task, so that IO is correctly aggregated. > I don't want streams, listIterators to do the updating as there is more > risk of double counting. However, we need to see their statistics if we want > to know things like "bytes discarded in backwards seeks". And I don't want to > be updating a shared context object on every read() call. > If all we want is store IO (HEAD, GET, DELETE, list performance etc) then the > FS is sufficient. > If we do want the stream-specific detail, then I propose > * caching the context in the constructor > * updating it only in close() or unbuffer() (as we do from S3AInputStream to > S3AInstrumenation) > * excluding those we know the FS already collects. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on pull request #4352: HADOOP-17461. Thread-level IOStatistics in S3A
mehakmeet commented on PR #4352: URL: https://github.com/apache/hadoop/pull/4352#issuecomment-1186772913 Cherry-picked S3A committers integration and lining up with spark build. Some javadocs would need modification, adding that in a commit. Need to see how to test this on a spark build locally, since spark might not be using the correct hadoop version iirc, then we could get this in -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17461) Add thread-level IOStatistics Context
[ https://issues.apache.org/jira/browse/HADOOP-17461?focusedWorklogId=791834=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-791834 ] ASF GitHub Bot logged work on HADOOP-17461: --- Author: ASF GitHub Bot Created on: 18/Jul/22 04:32 Start Date: 18/Jul/22 04:32 Worklog Time Spent: 10m Work Description: mehakmeet commented on PR #4566: URL: https://github.com/apache/hadoop/pull/4566#issuecomment-1186757937 > should these committers reset the context before task/job commit. In spark we would be resetting before the execution begins I assume, so having it reset in the committers could mess things up? Was thinking about the case like a disctp job, how do we reset the stats in that case? Issue Time Tracking --- Worklog Id: (was: 791834) Time Spent: 5h (was: 4h 50m) > Add thread-level IOStatistics Context > - > > Key: HADOOP-17461 > URL: https://issues.apache.org/jira/browse/HADOOP-17461 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/azure, fs/s3 >Affects Versions: 3.3.1 >Reporter: Steve Loughran >Assignee: Mehakmeet Singh >Priority: Major > Labels: pull-request-available > Time Spent: 5h > Remaining Estimate: 0h > > For effective reporting of the iostatistics of individual worker threads, we > need a thread-level context which IO components update. > * this contact needs to be passed in two background thread forming work on > behalf of a task. > * IO Components (streams, iterators, filesystems) need to update this context > statistics as they perform work > * Without double counting anything. > I imagine a ThreadLocal IOStatisticContext which will be updated in the > FileSystem API Calls. This context MUST be passed into the background threads > used by a task, so that IO is correctly aggregated. > I don't want streams, listIterators to do the updating as there is more > risk of double counting. However, we need to see their statistics if we want > to know things like "bytes discarded in backwards seeks". And I don't want to > be updating a shared context object on every read() call. > If all we want is store IO (HEAD, GET, DELETE, list performance etc) then the > FS is sufficient. > If we do want the stream-specific detail, then I propose > * caching the context in the constructor > * updating it only in close() or unbuffer() (as we do from S3AInputStream to > S3AInstrumenation) > * excluding those we know the FS already collects. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on pull request #4566: HADOOP-17461. IOStatisticsContext + committer integration
mehakmeet commented on PR #4566: URL: https://github.com/apache/hadoop/pull/4566#issuecomment-1186757937 > should these committers reset the context before task/job commit. In spark we would be resetting before the execution begins I assume, so having it reset in the committers could mess things up? Was thinking about the case like a disctp job, how do we reset the stats in that case? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] skysiders opened a new pull request, #4575: Mapreduce 7396 Job History File Permissions should not have exec perm.
skysiders opened a new pull request, #4575: URL: https://github.com/apache/hadoop/pull/4575 ### Description of PR Job History File Permissions now set with 77X permission and can config by user.But we are creating a file instead of a path, as a file does not need executable permissions ### How was this patch tested? ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4573: HDFS-16665. Fix duplicate sources for HDFS test
hadoop-yetus commented on PR #4573:
URL: https://github.com/apache/hadoop/pull/4573#issuecomment-1186725089
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 25m 0s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 24m 58s | | trunk passed |
| +1 :green_heart: | compile | 3m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 55m 7s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 22s | | the patch passed |
| +1 :green_heart: | compile | 3m 14s | | the patch passed |
| +1 :green_heart: | cc | 3m 14s | | the patch passed |
| +1 :green_heart: | javac | 3m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 58s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 78m 21s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not
generate ASF License warnings. |
| | | 191m 56s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4573/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4573 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell detsecrets |
| uname | Linux 2b3ab75d5860 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17
17:06:04 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / cd9094d30f4fac727dbf556ff317270578ae4a1a |
| Default Java | Debian-11.0.15+10-post-Debian-1deb10u1 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4573/1/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4573/1/console |
| versions | git=2.20.1 maven=3.6.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on a diff in pull request #4565: HDFS-16661. Improve Code With Lambda in AsyncLoggerSet class
ZanderXu commented on code in PR #4565:
URL: https://github.com/apache/hadoop/pull/4565#discussion_r922941580
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/AsyncLoggerSet.java:
##
@@ -206,197 +194,114 @@ List getLoggersForTests() {
///
public QuorumCall
getJournalState() {
-Map> calls =
-Maps.newHashMap();
-for (AsyncLogger logger : loggers) {
- calls.put(logger, logger.getJournalState());
-}
+Map> calls =
Maps.newHashMap();
+loggers.forEach(l -> calls.put(l, l.getJournalState()));
return QuorumCall.create(calls);
}
public QuorumCall isFormatted() {
Map> calls = Maps.newHashMap();
-for (AsyncLogger logger : loggers) {
- calls.put(logger, logger.isFormatted());
-}
+loggers.forEach(l -> calls.put(l, l.isFormatted()));
return QuorumCall.create(calls);
}
- public QuorumCall newEpoch(
- NamespaceInfo nsInfo,
Review Comment:
Thanks @goiri for your review.
`nsInfo` is never used. HDFS-12553 started using `nsInfo` in
`IPCLoggerChannel.class` as follows:
```
@Override
public ListenableFuture newEpoch(
final long epoch) {
return singleThreadExecutor.submit(new Callable()
{
@Override
public NewEpochResponseProto call() throws IOException {
return getProxy().newEpoch(journalId, nameServiceId, nsInfo, epoch);
}
});
}
```
So we can get rid of the useless `nsInfo` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4540: YARN-11160. Support getResourceProfiles, getResourceProfile API's for Federation
hadoop-yetus commented on PR #4540:
URL: https://github.com/apache/hadoop/pull/4540#issuecomment-1186691041
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 42s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | jsonlint | 0m 0s | | jsonlint was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 37s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 44s | | trunk passed |
| +1 :green_heart: | compile | 9m 42s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 8m 18s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 54s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 43s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 30s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 14s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 14s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 39s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 38s | | the patch passed |
| +1 :green_heart: | compile | 8m 57s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 8m 57s | | the patch passed |
| +1 :green_heart: | compile | 8m 17s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 8m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 35s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4540/9/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 4 unchanged -
4 fixed = 5 total (was 8) |
| +1 :green_heart: | mvnsite | 2m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 22s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 15s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 50s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 8s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 4m 50s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 13s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 161m 41s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4540/9/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4540 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets jsonlint
xmllint |
| uname | Linux 054623ac46ec 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3
10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 01c2415a4576e77d8cb8461bda73d79e6b1ddba4 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
[GitHub] [hadoop] hadoop-yetus commented on pull request #4543: YARN-8900. [Router] Federation: routing getContainers REST invocations transparently to multiple RMs
hadoop-yetus commented on PR #4543:
URL: https://github.com/apache/hadoop/pull/4543#issuecomment-1186670133
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 21s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 6s | | trunk passed |
| +1 :green_heart: | compile | 4m 16s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 27s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 30s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 51s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 42s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 2s | | the patch passed |
| +1 :green_heart: | compile | 4m 4s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 4m 4s | | the patch passed |
| +1 :green_heart: | compile | 3m 14s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 12s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 9s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 2s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 25s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 3s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 48s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 3m 6s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 130m 8s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4543/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4543 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 0818502eac0d 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 881022092fe6b5d0559b970ad77e87a07be0cb91 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4543/8/testReport/ |
| Max. process+thread count | 1345 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4543/8/console |
| versions |
[GitHub] [hadoop] hadoop-yetus commented on pull request #4573: HDFS-16665. Fix duplicate sources for HDFS test
hadoop-yetus commented on PR #4573:
URL: https://github.com/apache/hadoop/pull/4573#issuecomment-1186638829
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 24m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 26s | | trunk passed |
| +1 :green_heart: | compile | 4m 16s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 3s | | trunk passed |
| +1 :green_heart: | shadedclient | 47m 17s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 3m 37s | | the patch passed |
| +1 :green_heart: | cc | 3m 37s | | the patch passed |
| +1 :green_heart: | javac | 3m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 24s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 101m 54s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 2s | | The patch does not
generate ASF License warnings. |
| | | 201m 35s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4573/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4573 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell detsecrets |
| uname | Linux f836d92bb685 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17
17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / cd9094d30f4fac727dbf556ff317270578ae4a1a |
| Default Java | Red Hat, Inc.-1.8.0_312-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4573/1/testReport/ |
| Max. process+thread count | 701 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4573/1/console |
| versions | git=2.27.0 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4543: YARN-8900. [Router] Federation: routing getContainers REST invocations transparently to multiple RMs
slfan1989 commented on code in PR #4543:
URL: https://github.com/apache/hadoop/pull/4543#discussion_r922908056
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/webapp/MockDefaultRequestInterceptorREST.java:
##
@@ -231,4 +240,42 @@ public boolean isRunning() {
public void setRunning(boolean runningMode) {
this.isRunning = runningMode;
}
+
+ @Override
+ public ContainersInfo getContainers(HttpServletRequest req,
HttpServletResponse res,
+ String appId, String appAttemptId) {
+if (!isRunning) {
+ throw new RuntimeException("RM is stopped");
+}
+
+// We avoid to check if the Application exists in the system because we
need
+// to validate that each subCluster returns 1 container.
+ContainersInfo containers = new ContainersInfo();
+
+int subClusterId = Integer.valueOf(getSubClusterId().getId());
+
+ContainerId containerId = ContainerId.newContainerId(
+ApplicationAttemptId.fromString(appAttemptId), subClusterId);
+Resource allocatedResource =
+Resource.newInstance(subClusterId, subClusterId);
+
+NodeId assignedNode = NodeId.newInstance("Node", subClusterId);
+Priority priority = Priority.newInstance(subClusterId);
+long creationTime = subClusterId;
+long finishTime = subClusterId;
+String diagnosticInfo = "Diagnostic " + subClusterId;
+String logUrl = "Log " + subClusterId;
+int containerExitStatus = subClusterId;
+ContainerState containerState = ContainerState.COMPLETE;
+String nodeHttpAddress = "HttpAddress " + subClusterId;
+
+ContainerReport containerReport = ContainerReport.newInstance(containerId,
allocatedResource,
+assignedNode, priority, creationTime, finishTime, diagnosticInfo,
logUrl,
+containerExitStatus, containerState, nodeHttpAddress);
Review Comment:
as follows
```
ContainerReport containerReport = ContainerReport.newInstance(
containerId, allocatedResource, assignedNode, priority,
creationTime, finishTime, diagnosticInfo, logUrl,
containerExitStatus, containerState, nodeHttpAddress);
```
Is this possible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4543: YARN-8900. [Router] Federation: routing getContainers REST invocations transparently to multiple RMs
slfan1989 commented on code in PR #4543:
URL: https://github.com/apache/hadoop/pull/4543#discussion_r922908056
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/webapp/MockDefaultRequestInterceptorREST.java:
##
@@ -231,4 +240,42 @@ public boolean isRunning() {
public void setRunning(boolean runningMode) {
this.isRunning = runningMode;
}
+
+ @Override
+ public ContainersInfo getContainers(HttpServletRequest req,
HttpServletResponse res,
+ String appId, String appAttemptId) {
+if (!isRunning) {
+ throw new RuntimeException("RM is stopped");
+}
+
+// We avoid to check if the Application exists in the system because we
need
+// to validate that each subCluster returns 1 container.
+ContainersInfo containers = new ContainersInfo();
+
+int subClusterId = Integer.valueOf(getSubClusterId().getId());
+
+ContainerId containerId = ContainerId.newContainerId(
+ApplicationAttemptId.fromString(appAttemptId), subClusterId);
+Resource allocatedResource =
+Resource.newInstance(subClusterId, subClusterId);
+
+NodeId assignedNode = NodeId.newInstance("Node", subClusterId);
+Priority priority = Priority.newInstance(subClusterId);
+long creationTime = subClusterId;
+long finishTime = subClusterId;
+String diagnosticInfo = "Diagnostic " + subClusterId;
+String logUrl = "Log " + subClusterId;
+int containerExitStatus = subClusterId;
+ContainerState containerState = ContainerState.COMPLETE;
+String nodeHttpAddress = "HttpAddress " + subClusterId;
+
+ContainerReport containerReport = ContainerReport.newInstance(containerId,
allocatedResource,
+assignedNode, priority, creationTime, finishTime, diagnosticInfo,
logUrl,
+containerExitStatus, containerState, nodeHttpAddress);
Review Comment:
as follows
```
ContainerReport containerReport = ContainerReport.newInstance(
containerId, allocatedResource, assignedNode, priority,
creationTime, finishTime, diagnosticInfo, logUrl,
containerExitStatus, containerState, nodeHttpAddress);
```
Is this possible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4540: YARN-11160. Support getResourceProfiles, getResourceProfile API's for Federation
slfan1989 commented on code in PR #4540:
URL: https://github.com/apache/hadoop/pull/4540#discussion_r922910718
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/TestFederationClientInterceptor.java:
##
@@ -1169,4 +1173,64 @@ public void testGetQueueInfo() throws Exception {
Assert.assertEquals(queueInfo.getChildQueues().size(), 12, 0);
Assert.assertEquals(queueInfo.getAccessibleNodeLabels().size(), 1);
}
+
+ @Test
+ public void testGetResourceProfiles() throws Exception {
+LOG.info("Test FederationClientInterceptor : Get Resource Profiles
request.");
+
+// null request
+LambdaTestUtils.intercept(YarnException.class, "Missing
getResourceProfiles request.",
+() -> interceptor.getResourceProfiles(null));
+
+// normal request
+GetAllResourceProfilesRequest request =
GetAllResourceProfilesRequest.newInstance();
+GetAllResourceProfilesResponse response =
interceptor.getResourceProfiles(request);
+
+Assert.assertNotNull(response);
+Map resProfiles = response.getResourceProfiles();
+
+Resource maxResProfiles = resProfiles.get("maximum");
+Assert.assertEquals(maxResProfiles.getMemorySize(), 32768);
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4540: YARN-11160. Support getResourceProfiles, getResourceProfile API's for Federation
slfan1989 commented on code in PR #4540:
URL: https://github.com/apache/hadoop/pull/4540#discussion_r922910660
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java:
##
@@ -1448,13 +1448,52 @@ public UpdateApplicationTimeoutsResponse
updateApplicationTimeouts(
@Override
public GetAllResourceProfilesResponse getResourceProfiles(
GetAllResourceProfilesRequest request) throws YarnException, IOException
{
-throw new NotImplementedException("Code is not implemented");
+if (request == null) {
+ routerMetrics.incrGetResourceProfilesFailedRetrieved();
+ RouterServerUtil.logAndThrowException("Missing getResourceProfiles
request.", null);
+}
+long startTime = clock.getTime();
+ClientMethod remoteMethod = new ClientMethod("getResourceProfiles",
+new Class[] {GetAllResourceProfilesRequest.class}, new Object[]
{request});
+Collection resourceProfiles;
+try {
+ resourceProfiles = invokeAppClientProtocolMethod(true, remoteMethod,
+ GetAllResourceProfilesResponse.class);
+} catch (Exception ex) {
+ routerMetrics.incrGetResourceProfilesFailedRetrieved();
+ LOG.error("Unable to get resource profiles due to exception.", ex);
+ throw ex;
+}
+long stopTime = clock.getTime();
Review Comment:
code show as below:
```
long startTime = clock.getTime();
ClientMethod remoteMethod = new ClientMethod("getResourceProfile",
new Class[] {GetResourceProfileRequest.class}, new Object[]
{request});
Collection resourceProfile = null;
try {
resourceProfile = invokeAppClientProtocolMethod(true, remoteMethod,
GetResourceProfileResponse.class);
} catch (Exception ex) {
routerMetrics.incrGetResourceProfileFailedRetrieved();
RouterServerUtil.logAndThrowException("Unable to get resource profile
due to exception.",
ex);
}
long stopTime = clock.getTime();
```
startTime - stopTime represents the execution time of the code fragment
I think the current way should be a little better, can this part stay as it
is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4540: YARN-11160. Support getResourceProfiles, getResourceProfile API's for Federation
slfan1989 commented on code in PR #4540:
URL: https://github.com/apache/hadoop/pull/4540#discussion_r922908908
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterYarnClientUtils.java:
##
@@ -417,5 +421,51 @@ public static GetQueueInfoResponse mergeQueues(
queueResponse.setQueueInfo(queueInfo);
return queueResponse;
}
+
+ /**
+ * Merges a list of GetAllResourceProfilesResponse.
+ *
+ * @param responses a list of GetAllResourceProfilesResponse to merge.
+ * @return the merged GetAllResourceProfilesResponse.
+ */
+ public static GetAllResourceProfilesResponse
mergeClusterResourceProfilesResponse(
+ Collection responses) {
+GetAllResourceProfilesResponse profilesResponse =
+Records.newRecord(GetAllResourceProfilesResponse.class);
+Map profilesMap = new HashMap<>();
+for (GetAllResourceProfilesResponse response : responses) {
+ if (response != null && response.getResourceProfiles() != null) {
+for (Map.Entry entry :
response.getResourceProfiles().entrySet()) {
+ String key = entry.getKey();
+ Resource r1 = profilesMap.getOrDefault(key, null);
+ Resource r2 = entry.getValue();
+ profilesMap.put(key, r1 == null ? r2 : Resources.add(r1, r2));
Review Comment:
This is a good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4540: YARN-11160. Support getResourceProfiles, getResourceProfile API's for Federation
slfan1989 commented on code in PR #4540:
URL: https://github.com/apache/hadoop/pull/4540#discussion_r922908676
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java:
##
@@ -1448,13 +1448,52 @@ public UpdateApplicationTimeoutsResponse
updateApplicationTimeouts(
@Override
public GetAllResourceProfilesResponse getResourceProfiles(
GetAllResourceProfilesRequest request) throws YarnException, IOException
{
-throw new NotImplementedException("Code is not implemented");
+if (request == null) {
+ routerMetrics.incrGetResourceProfilesFailedRetrieved();
+ RouterServerUtil.logAndThrowException("Missing getResourceProfiles
request.", null);
+}
+long startTime = clock.getTime();
+ClientMethod remoteMethod = new ClientMethod("getResourceProfiles",
+new Class[] {GetAllResourceProfilesRequest.class}, new Object[]
{request});
+Collection resourceProfiles;
+try {
+ resourceProfiles = invokeAppClientProtocolMethod(true, remoteMethod,
+ GetAllResourceProfilesResponse.class);
+} catch (Exception ex) {
+ routerMetrics.incrGetResourceProfilesFailedRetrieved();
+ LOG.error("Unable to get resource profiles due to exception.", ex);
+ throw ex;
+}
+long stopTime = clock.getTime();
+routerMetrics.succeededGetResourceProfilesRetrieved(stopTime - startTime);
+// Merge the GetAllResourceProfilesResponse
Review Comment:
I will remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4543: YARN-8900. [Router] Federation: routing getContainers REST invocations transparently to multiple RMs
slfan1989 commented on code in PR #4543:
URL: https://github.com/apache/hadoop/pull/4543#discussion_r922908101
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/FederationInterceptorREST.java:
##
@@ -1366,4 +1393,42 @@ public void shutdown() {
threadpool.shutdown();
}
}
+
+ private Map
invokeConcurrent(Collection clusterIds,
+ ClientMethod request, Class clazz) {
+
+Map results = new HashMap<>();
+
+// Send the requests in parallel
+CompletionService compSvc = new
ExecutorCompletionService<>(this.threadpool);
+
+for (final SubClusterInfo info : clusterIds) {
+ compSvc.submit(() -> {
+DefaultRequestInterceptorREST interceptor =
getOrCreateInterceptorForSubCluster(
+info.getSubClusterId(), info.getRMWebServiceAddress());
+try {
+ Method method = DefaultRequestInterceptorREST.class.
+ getMethod(request.getMethodName(), request.getTypes());
+ return (R) clazz.cast(method.invoke(interceptor,
request.getParams()));
Review Comment:
i will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4543: YARN-8900. [Router] Federation: routing getContainers REST invocations transparently to multiple RMs
slfan1989 commented on code in PR #4543:
URL: https://github.com/apache/hadoop/pull/4543#discussion_r922908056
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/webapp/MockDefaultRequestInterceptorREST.java:
##
@@ -231,4 +240,42 @@ public boolean isRunning() {
public void setRunning(boolean runningMode) {
this.isRunning = runningMode;
}
+
+ @Override
+ public ContainersInfo getContainers(HttpServletRequest req,
HttpServletResponse res,
+ String appId, String appAttemptId) {
+if (!isRunning) {
+ throw new RuntimeException("RM is stopped");
+}
+
+// We avoid to check if the Application exists in the system because we
need
+// to validate that each subCluster returns 1 container.
+ContainersInfo containers = new ContainersInfo();
+
+int subClusterId = Integer.valueOf(getSubClusterId().getId());
+
+ContainerId containerId = ContainerId.newContainerId(
+ApplicationAttemptId.fromString(appAttemptId), subClusterId);
+Resource allocatedResource =
+Resource.newInstance(subClusterId, subClusterId);
+
+NodeId assignedNode = NodeId.newInstance("Node", subClusterId);
+Priority priority = Priority.newInstance(subClusterId);
+long creationTime = subClusterId;
+long finishTime = subClusterId;
+String diagnosticInfo = "Diagnostic " + subClusterId;
+String logUrl = "Log " + subClusterId;
+int containerExitStatus = subClusterId;
+ContainerState containerState = ContainerState.COMPLETE;
+String nodeHttpAddress = "HttpAddress " + subClusterId;
+
+ContainerReport containerReport = ContainerReport.newInstance(containerId,
allocatedResource,
+assignedNode, priority, creationTime, finishTime, diagnosticInfo,
logUrl,
+containerExitStatus, containerState, nodeHttpAddress);
Review Comment:
as follows
```
ContainerReport containerReport = ContainerReport.newInstance(
containerId, allocatedResource, assignedNode, priority,
creationTime, finishTime, diagnosticInfo, logUrl,
containerExitStatus, containerState, nodeHttpAddress);
```
Is this possible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4573: HDFS-16665. Fix duplicate sources for HDFS test
hadoop-yetus commented on PR #4573:
URL: https://github.com/apache/hadoop/pull/4573#issuecomment-1186605339
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 40m 36s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 17s | | trunk passed |
| +1 :green_heart: | compile | 4m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 62m 9s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 25s | | the patch passed |
| +1 :green_heart: | compile | 3m 44s | | the patch passed |
| +1 :green_heart: | cc | 3m 44s | | the patch passed |
| +1 :green_heart: | javac | 3m 44s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 36s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 112m 27s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 54s | | The patch does not
generate ASF License warnings. |
| | | 242m 56s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4573/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4573 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell detsecrets |
| uname | Linux 845690b8b35a 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17
17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / cd9094d30f4fac727dbf556ff317270578ae4a1a |
| Default Java | Red Hat, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4573/1/testReport/ |
| Max. process+thread count | 555 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4573/1/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4574: HDFS-16666. Pass CMake args for Windows in pom.xml
hadoop-yetus commented on PR #4574:
URL: https://github.com/apache/hadoop/pull/4574#issuecomment-1186605216
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 56s | | trunk passed |
| +1 :green_heart: | compile | 4m 19s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 4m 15s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | mvnsite | 0m 38s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | shadedclient | 72m 49s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 4m 6s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | cc | 4m 6s | | the patch passed |
| +1 :green_heart: | golang | 4m 6s | | the patch passed |
| +1 :green_heart: | javac | 4m 6s | | the patch passed |
| +1 :green_heart: | compile | 4m 7s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 4m 7s | | the patch passed |
| +1 :green_heart: | golang | 4m 7s | | the patch passed |
| +1 :green_heart: | javac | 4m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | shadedclient | 24m 16s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 114m 36s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not
generate ASF License warnings. |
| | | 224m 47s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4574/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4574 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint cc golang |
| uname | Linux 1aee3d046585 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 08e77dbc99bf6ff40399cec2ffba8ad194fff5e6 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4574/1/testReport/ |
| Max. process+thread count | 540 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4574/1/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To
[GitHub] [hadoop] hadoop-yetus commented on pull request #4568: HDFS-16664. Use correct GenerationStamp when invalidating corrupt block replicas
hadoop-yetus commented on PR #4568:
URL: https://github.com/apache/hadoop/pull/4568#issuecomment-1186601022
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 9s | | trunk passed |
| +1 :green_heart: | compile | 1m 40s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 1m 30s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 19s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 19s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 40s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 0s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 23s | | the patch passed |
| +1 :green_heart: | compile | 1m 28s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 1m 28s | | the patch passed |
| +1 :green_heart: | compile | 1m 19s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 1s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4568/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 88 unchanged -
1 fixed = 91 total (was 89) |
| +1 :green_heart: | mvnsite | 1m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 33s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 51s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 362m 11s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 0s | | The patch does not
generate ASF License warnings. |
| | | 479m 0s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4568/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4568 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 233b4644866c 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8
19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9884cf088d26412ee438b22dd977469f9377c63b |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4568/2/testReport/ |
| Max. process+thread count | 1862 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4568/2/console |
| versions | git=2.25.1
[GitHub] [hadoop] goiri commented on a diff in pull request #4543: YARN-8900. [Router] Federation: routing getContainers REST invocations transparently to multiple RMs
goiri commented on code in PR #4543:
URL: https://github.com/apache/hadoop/pull/4543#discussion_r922880359
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/FederationInterceptorREST.java:
##
@@ -1366,4 +1393,42 @@ public void shutdown() {
threadpool.shutdown();
}
}
+
+ private Map
invokeConcurrent(Collection clusterIds,
+ ClientMethod request, Class clazz) {
+
+Map results = new HashMap<>();
+
+// Send the requests in parallel
+CompletionService compSvc = new
ExecutorCompletionService<>(this.threadpool);
+
+for (final SubClusterInfo info : clusterIds) {
+ compSvc.submit(() -> {
+DefaultRequestInterceptorREST interceptor =
getOrCreateInterceptorForSubCluster(
+info.getSubClusterId(), info.getRMWebServiceAddress());
+try {
+ Method method = DefaultRequestInterceptorREST.class.
+ getMethod(request.getMethodName(), request.getTypes());
+ return (R) clazz.cast(method.invoke(interceptor,
request.getParams()));
Review Comment:
I was referring to something like:
```
Object retObj = method.invoke(interceptor, request.getParams());
R ret = clazz.cast(retObj);
return ret;
```
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/webapp/MockDefaultRequestInterceptorREST.java:
##
@@ -231,4 +240,42 @@ public boolean isRunning() {
public void setRunning(boolean runningMode) {
this.isRunning = runningMode;
}
+
+ @Override
+ public ContainersInfo getContainers(HttpServletRequest req,
HttpServletResponse res,
+ String appId, String appAttemptId) {
+if (!isRunning) {
+ throw new RuntimeException("RM is stopped");
+}
+
+// We avoid to check if the Application exists in the system because we
need
+// to validate that each subCluster returns 1 container.
+ContainersInfo containers = new ContainersInfo();
+
+int subClusterId = Integer.valueOf(getSubClusterId().getId());
+
+ContainerId containerId = ContainerId.newContainerId(
+ApplicationAttemptId.fromString(appAttemptId), subClusterId);
+Resource allocatedResource =
+Resource.newInstance(subClusterId, subClusterId);
+
+NodeId assignedNode = NodeId.newInstance("Node", subClusterId);
+Priority priority = Priority.newInstance(subClusterId);
+long creationTime = subClusterId;
+long finishTime = subClusterId;
+String diagnosticInfo = "Diagnostic " + subClusterId;
+String logUrl = "Log " + subClusterId;
+int containerExitStatus = subClusterId;
+ContainerState containerState = ContainerState.COMPLETE;
+String nodeHttpAddress = "HttpAddress " + subClusterId;
+
+ContainerReport containerReport = ContainerReport.newInstance(containerId,
allocatedResource,
+assignedNode, priority, creationTime, finishTime, diagnosticInfo,
logUrl,
+containerExitStatus, containerState, nodeHttpAddress);
Review Comment:
Bad indentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4540: YARN-11160. Support getResourceProfiles, getResourceProfile API's for Federation
goiri commented on code in PR #4540:
URL: https://github.com/apache/hadoop/pull/4540#discussion_r922879676
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java:
##
@@ -1448,13 +1448,52 @@ public UpdateApplicationTimeoutsResponse
updateApplicationTimeouts(
@Override
public GetAllResourceProfilesResponse getResourceProfiles(
GetAllResourceProfilesRequest request) throws YarnException, IOException
{
-throw new NotImplementedException("Code is not implemented");
+if (request == null) {
+ routerMetrics.incrGetResourceProfilesFailedRetrieved();
+ RouterServerUtil.logAndThrowException("Missing getResourceProfiles
request.", null);
+}
+long startTime = clock.getTime();
+ClientMethod remoteMethod = new ClientMethod("getResourceProfiles",
+new Class[] {GetAllResourceProfilesRequest.class}, new Object[]
{request});
+Collection resourceProfiles;
+try {
+ resourceProfiles = invokeAppClientProtocolMethod(true, remoteMethod,
+ GetAllResourceProfilesResponse.class);
+} catch (Exception ex) {
+ routerMetrics.incrGetResourceProfilesFailedRetrieved();
+ LOG.error("Unable to get resource profiles due to exception.", ex);
+ throw ex;
+}
+long stopTime = clock.getTime();
Review Comment:
Couldn't we move this and the setting inside of the try?
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterYarnClientUtils.java:
##
@@ -417,5 +421,51 @@ public static GetQueueInfoResponse mergeQueues(
queueResponse.setQueueInfo(queueInfo);
return queueResponse;
}
+
+ /**
+ * Merges a list of GetAllResourceProfilesResponse.
+ *
+ * @param responses a list of GetAllResourceProfilesResponse to merge.
+ * @return the merged GetAllResourceProfilesResponse.
+ */
+ public static GetAllResourceProfilesResponse
mergeClusterResourceProfilesResponse(
+ Collection responses) {
+GetAllResourceProfilesResponse profilesResponse =
+Records.newRecord(GetAllResourceProfilesResponse.class);
+Map profilesMap = new HashMap<>();
+for (GetAllResourceProfilesResponse response : responses) {
+ if (response != null && response.getResourceProfiles() != null) {
+for (Map.Entry entry :
response.getResourceProfiles().entrySet()) {
+ String key = entry.getKey();
+ Resource r1 = profilesMap.getOrDefault(key, null);
+ Resource r2 = entry.getValue();
+ profilesMap.put(key, r1 == null ? r2 : Resources.add(r1, r2));
Review Comment:
I would extract this as:
```
rAdd = r1 == null ? r2 : Resources.add(r1, r2);
```
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java:
##
@@ -1448,13 +1448,52 @@ public UpdateApplicationTimeoutsResponse
updateApplicationTimeouts(
@Override
public GetAllResourceProfilesResponse getResourceProfiles(
GetAllResourceProfilesRequest request) throws YarnException, IOException
{
-throw new NotImplementedException("Code is not implemented");
+if (request == null) {
+ routerMetrics.incrGetResourceProfilesFailedRetrieved();
+ RouterServerUtil.logAndThrowException("Missing getResourceProfiles
request.", null);
+}
+long startTime = clock.getTime();
+ClientMethod remoteMethod = new ClientMethod("getResourceProfiles",
+new Class[] {GetAllResourceProfilesRequest.class}, new Object[]
{request});
+Collection resourceProfiles;
+try {
+ resourceProfiles = invokeAppClientProtocolMethod(true, remoteMethod,
+ GetAllResourceProfilesResponse.class);
+} catch (Exception ex) {
+ routerMetrics.incrGetResourceProfilesFailedRetrieved();
+ LOG.error("Unable to get resource profiles due to exception.", ex);
+ throw ex;
+}
+long stopTime = clock.getTime();
+routerMetrics.succeededGetResourceProfilesRetrieved(stopTime - startTime);
+// Merge the GetAllResourceProfilesResponse
Review Comment:
This comment is not very useful
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/TestFederationClientInterceptor.java:
##
@@ -1169,4 +1173,64 @@ public void testGetQueueInfo() throws Exception {
Assert.assertEquals(queueInfo.getChildQueues().size(), 12, 0);
Assert.assertEquals(queueInfo.getAccessibleNodeLabels().size(), 1);
}
+
+ @Test
+ public void testGetResourceProfiles() throws Exception {
+LOG.info("Test FederationClientInterceptor : Get Resource Profiles
request.");
+
+// null request
+LambdaTestUtils.intercept(YarnException.class,
[GitHub] [hadoop] GauthamBanasandra opened a new pull request, #4574: HDFS-16666. Pass CMake args for Windows in pom.xml
GauthamBanasandra opened a new pull request, #4574: URL: https://github.com/apache/hadoop/pull/4574 ### Description of PR Need to pass the required HDFS CMake related argument for building on Windows. Currently, these are passed - https://github.com/apache/hadoop/blob/34e548cb62ed21c5bba7a82f5f1489ca6bdfb8c4/hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml#L150 We need to pass these arguments as well for Windows - https://github.com/apache/hadoop/blob/34e548cb62ed21c5bba7a82f5f1489ca6bdfb8c4/hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml#L219-L223 ### How was this patch tested? 1. Tested this locally by building on my Windows system. 2. Hadoop Jenkins CI validation. ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4026: MAPREDUCE-7372 MapReduce set permission too late in copyJar method
hadoop-yetus commented on PR #4026:
URL: https://github.com/apache/hadoop/pull/4026#issuecomment-1186571531
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 11s | | trunk passed |
| +1 :green_heart: | compile | 0m 55s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 48s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 52s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 57s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 5s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 42s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 31s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 41s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 31s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 39s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 7m 2s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 108m 55s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4026/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4026 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 0bd75eb29538 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1f5becb3258cc305867e204166bdc179c8f8468a |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4026/3/testReport/ |
| Max. process+thread count | 1152 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4026/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an
[GitHub] [hadoop] GauthamBanasandra opened a new pull request, #4573: HDFS-16665. Fix duplicate sources for HDFS test
GauthamBanasandra opened a new pull request, #4573:
URL: https://github.com/apache/hadoop/pull/4573
### Description of PR
The library target `hdfspp_test_shim_static` is built using the following
sources, which causes duplicate symbols to be defined -
1. hdfs_shim.c
2. ${LIBHDFSPP_BINDING_C}/hdfs.cc
https://github.com/apache/hadoop/blob/8774f178686487007dcf8c418c989b785a529000/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/CMakeLists.txt#L153
And fails the compilation -
```
H:\hadoop-hdfs-project\hadoop-hdfs-native-client\src\out\build\x64-Debug\hdfspp_test_shim_static.lib(hdfs_shim.obj)
: error LNK2005: hdfsAllowSnapshot already defined in
hdfspp_test_shim_static.lib(hdfs.obj)
H:\hadoop-hdfs-project\hadoop-hdfs-native-client\src\out\build\x64-Debug\hdfspp_test_shim_static.lib(hdfs_shim.obj)
: error LNK2005: hdfsAvailable already defined in
hdfspp_test_shim_static.lib(hdfs.obj)
H:\hadoop-hdfs-project\hadoop-hdfs-native-client\src\out\build\x64-Debug\hdfspp_test_shim_static.lib(hdfs_shim.obj)
: error LNK2005: hdfsBuilderConfSetStr already defined in
hdfspp_test_shim_static.lib(hdfs.obj)
H:\hadoop-hdfs-project\hadoop-hdfs-native-client\src\out\build\x64-Debug\hdfspp_test_shim_static.lib(hdfs_shim.obj)
: error LNK2005: hdfsBuilderConnect already defined in
hdfspp_test_shim_static.lib(hdfs.obj)
Duplicate symbols defined by hdfs_shim.c -
https://github.com/apache/hadoop/blob/440f4c2b28515d2007b81ac00b549bbf14fa9f64/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_shim.c#L583-L585.
```
Adding the source `${LIBHDFSPP_BINDING_C}/hdfs.cc` is redundant here since
this file is transitively included in `hdfs_shim.c` through
`libhdfspp_wrapper.h` -
https://github.com/apache/hadoop/blob/440f4c2b28515d2007b81ac00b549bbf14fa9f64/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_shim.c#L20
Thus, we need to exclude `${LIBHDFSPP_BINDING_C}/hdfs.cc` to fix this issue.
### How was this patch tested?
1. Tested this locally by building on my Windows system.
2. Hadoop Jenkins CI validation.
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #4462: MAPREDUCE-7390 Remove WhiteBox in mapreduce module.
steveloughran commented on PR #4462: URL: https://github.com/apache/hadoop/pull/4462#issuecomment-1186562492 afraid i'm taking a break from all non critical PR review; keeping my typing to a minimum. sorry -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18178) Upgrade jackson to 2.13.2 and jackson-databind to 2.13.2.2
[ https://issues.apache.org/jira/browse/HADOOP-18178?focusedWorklogId=791770=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-791770 ] ASF GitHub Bot logged work on HADOOP-18178: --- Author: ASF GitHub Bot Created on: 17/Jul/22 16:12 Start Date: 17/Jul/22 16:12 Worklog Time Spent: 10m Work Description: ayushtkn closed pull request #4544: Rev: HADOOP-18178 & HADOOP-18033 to Fix Class conflicts in downstreams URL: https://github.com/apache/hadoop/pull/4544 Issue Time Tracking --- Worklog Id: (was: 791770) Time Spent: 4h 40m (was: 4.5h) > Upgrade jackson to 2.13.2 and jackson-databind to 2.13.2.2 > -- > > Key: HADOOP-18178 > URL: https://issues.apache.org/jira/browse/HADOOP-18178 > Project: Hadoop Common > Issue Type: Bug >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.3 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > https://github.com/FasterXML/jackson-databind/issues/2816 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn closed pull request #4544: Rev: HADOOP-18178 & HADOOP-18033 to Fix Class conflicts in downstreams
ayushtkn closed pull request #4544: Rev: HADOOP-18178 & HADOOP-18033 to Fix Class conflicts in downstreams URL: https://github.com/apache/hadoop/pull/4544 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18332) remove rs-api dependency by downgrading jackson to 2.12.7
[ https://issues.apache.org/jira/browse/HADOOP-18332?focusedWorklogId=791768=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-791768 ] ASF GitHub Bot logged work on HADOOP-18332: --- Author: ASF GitHub Bot Created on: 17/Jul/22 16:07 Start Date: 17/Jul/22 16:07 Worklog Time Spent: 10m Work Description: ayushtkn merged PR #4547: URL: https://github.com/apache/hadoop/pull/4547 Issue Time Tracking --- Worklog Id: (was: 791768) Time Spent: 3h 40m (was: 3.5h) > remove rs-api dependency by downgrading jackson to 2.12.7 > - > > Key: HADOOP-18332 > URL: https://issues.apache.org/jira/browse/HADOOP-18332 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.3.4 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > This jsr311-api jar seems to conflict with newly added rs-api jar dependency > - they have many of the same classes (but conflicting copies) - jersey-core > 1.19 needs jsr311-api to work properly (and fails if rs-api used instead) > * https://mvnrepository.com/artifact/javax.ws.rs/jsr311-api > * https://mvnrepository.com/artifact/javax.ws.rs/javax.ws.rs-api > Seems we will need to downgrade jackson to 2.12.7 because of jax-rs > compatibility issues in jackson 2.13 (see > https://github.com/FasterXML/jackson-jaxrs-providers/issues/134) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18074) Partial/Incomplete groups list can be returned in LDAP groups lookup
[
https://issues.apache.org/jira/browse/HADOOP-18074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-18074:
Fix Version/s: 3.3.4
(was: 3.3.5)
> Partial/Incomplete groups list can be returned in LDAP groups lookup
>
>
> Key: HADOOP-18074
> URL: https://issues.apache.org/jira/browse/HADOOP-18074
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Philippe Lanoe
>Assignee: Larry McCay
>Priority: Major
> Labels: Reviewed, pull-request-available
> Fix For: 3.4.0, 3.3.4
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Hello,
> The
> {code:java}
> Set doGetGroups(String user, int goUpHierarchy) {code}
> method in
> [https://github.com/apache/hadoop/blob/b27732c69b114f24358992a5a4d170bc94e2ceaf/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/LdapGroupsMapping.java#L476]
> Looks like having an issue if in the middle of the loop a *NamingException*
> is caught:
> The groups variable is not reset in the catch clause and therefore the
> fallback lookup cannot be executed (when goUpHierarchy==0 at least):
> ||
> {code:java}
> if (groups.isEmpty() || goUpHierarchy > 0) {
> groups = lookupGroup(result, c, goUpHierarchy);
> }
> {code}
>
> Consequence is that only a partial list of groups is returned, which is not
> correct.
> Following options could be used as solution:
> * Reset the group to an empty list in the catch clause, to trigger the
> fallback query.
> * Add an option flag to enable ignoring groups with Naming Exception (since
> they are not groups most probably)
> Independently, would any issue also occur (and therefore full list cannot be
> returned) in the first lookup as well as in the fallback query, the method
> should/could(with option flag) throw an Exception, because in some scenario
> accuracy is important.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18074) Partial/Incomplete groups list can be returned in LDAP groups lookup
[
https://issues.apache.org/jira/browse/HADOOP-18074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17567681#comment-17567681
]
Steve Loughran commented on HADOOP-18074:
-
ok, all is good. merged to 3.3. and 3.3.4; kicking off the 3.3.4 buiild now
> Partial/Incomplete groups list can be returned in LDAP groups lookup
>
>
> Key: HADOOP-18074
> URL: https://issues.apache.org/jira/browse/HADOOP-18074
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Philippe Lanoe
>Assignee: Larry McCay
>Priority: Major
> Labels: Reviewed, pull-request-available
> Fix For: 3.4.0, 3.3.4
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Hello,
> The
> {code:java}
> Set doGetGroups(String user, int goUpHierarchy) {code}
> method in
> [https://github.com/apache/hadoop/blob/b27732c69b114f24358992a5a4d170bc94e2ceaf/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/LdapGroupsMapping.java#L476]
> Looks like having an issue if in the middle of the loop a *NamingException*
> is caught:
> The groups variable is not reset in the catch clause and therefore the
> fallback lookup cannot be executed (when goUpHierarchy==0 at least):
> ||
> {code:java}
> if (groups.isEmpty() || goUpHierarchy > 0) {
> groups = lookupGroup(result, c, goUpHierarchy);
> }
> {code}
>
> Consequence is that only a partial list of groups is returned, which is not
> correct.
> Following options could be used as solution:
> * Reset the group to an empty list in the catch clause, to trigger the
> fallback query.
> * Add an option flag to enable ignoring groups with Naming Exception (since
> they are not groups most probably)
> Independently, would any issue also occur (and therefore full list cannot be
> returned) in the first lookup as well as in the fallback query, the method
> should/could(with option flag) throw an Exception, because in some scenario
> accuracy is important.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn merged pull request #4547: HADOOP-18332: remove rs-api dependency as it conflicts with jsr311-api
ayushtkn merged PR #4547: URL: https://github.com/apache/hadoop/pull/4547 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18074) Partial/Incomplete groups list can be returned in LDAP groups lookup
[
https://issues.apache.org/jira/browse/HADOOP-18074?focusedWorklogId=791767=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-791767
]
ASF GitHub Bot logged work on HADOOP-18074:
---
Author: ASF GitHub Bot
Created on: 17/Jul/22 16:06
Start Date: 17/Jul/22 16:06
Worklog Time Spent: 10m
Work Description: steveloughran merged PR #4570:
URL: https://github.com/apache/hadoop/pull/4570
Issue Time Tracking
---
Worklog Id: (was: 791767)
Time Spent: 4h (was: 3h 50m)
> Partial/Incomplete groups list can be returned in LDAP groups lookup
>
>
> Key: HADOOP-18074
> URL: https://issues.apache.org/jira/browse/HADOOP-18074
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Philippe Lanoe
>Assignee: Larry McCay
>Priority: Major
> Labels: Reviewed, pull-request-available
> Fix For: 3.4.0, 3.3.5
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Hello,
> The
> {code:java}
> Set doGetGroups(String user, int goUpHierarchy) {code}
> method in
> [https://github.com/apache/hadoop/blob/b27732c69b114f24358992a5a4d170bc94e2ceaf/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/LdapGroupsMapping.java#L476]
> Looks like having an issue if in the middle of the loop a *NamingException*
> is caught:
> The groups variable is not reset in the catch clause and therefore the
> fallback lookup cannot be executed (when goUpHierarchy==0 at least):
> ||
> {code:java}
> if (groups.isEmpty() || goUpHierarchy > 0) {
> groups = lookupGroup(result, c, goUpHierarchy);
> }
> {code}
>
> Consequence is that only a partial list of groups is returned, which is not
> correct.
> Following options could be used as solution:
> * Reset the group to an empty list in the catch clause, to trigger the
> fallback query.
> * Add an option flag to enable ignoring groups with Naming Exception (since
> they are not groups most probably)
> Independently, would any issue also occur (and therefore full list cannot be
> returned) in the first lookup as well as in the fallback query, the method
> should/could(with option flag) throw an Exception, because in some scenario
> accuracy is important.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #4570: HADOOP-18074 - Partial/Incomplete groups list can be returned in LDAP. (#4503)
steveloughran merged PR #4570: URL: https://github.com/apache/hadoop/pull/4570 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18332) remove rs-api dependency by downgrading jackson to 2.12.7
[ https://issues.apache.org/jira/browse/HADOOP-18332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18332. - Fix Version/s: 3.3.4 Release Note: Downgrades Jackson from 2.13.2 to 2.12.7 to fix class conflicts in downstream projects. This version of jackson does contain the fix for CVE-2020-36518. (was: Downgrades Jackson from 2.13.2 to 2.12.7 to fix Class conflicts in downstream projects) Resolution: Fixed > remove rs-api dependency by downgrading jackson to 2.12.7 > - > > Key: HADOOP-18332 > URL: https://issues.apache.org/jira/browse/HADOOP-18332 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.3.4 > > Time Spent: 3.5h > Remaining Estimate: 0h > > This jsr311-api jar seems to conflict with newly added rs-api jar dependency > - they have many of the same classes (but conflicting copies) - jersey-core > 1.19 needs jsr311-api to work properly (and fails if rs-api used instead) > * https://mvnrepository.com/artifact/javax.ws.rs/jsr311-api > * https://mvnrepository.com/artifact/javax.ws.rs/javax.ws.rs-api > Seems we will need to downgrade jackson to 2.12.7 because of jax-rs > compatibility issues in jackson 2.13 (see > https://github.com/FasterXML/jackson-jaxrs-providers/issues/134) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18332) remove rs-api dependency by downgrading jackson to 2.12.7
[ https://issues.apache.org/jira/browse/HADOOP-18332?focusedWorklogId=791766=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-791766 ] ASF GitHub Bot logged work on HADOOP-18332: --- Author: ASF GitHub Bot Created on: 17/Jul/22 16:03 Start Date: 17/Jul/22 16:03 Worklog Time Spent: 10m Work Description: steveloughran merged PR #4569: URL: https://github.com/apache/hadoop/pull/4569 Issue Time Tracking --- Worklog Id: (was: 791766) Time Spent: 3.5h (was: 3h 20m) > remove rs-api dependency by downgrading jackson to 2.12.7 > - > > Key: HADOOP-18332 > URL: https://issues.apache.org/jira/browse/HADOOP-18332 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Time Spent: 3.5h > Remaining Estimate: 0h > > This jsr311-api jar seems to conflict with newly added rs-api jar dependency > - they have many of the same classes (but conflicting copies) - jersey-core > 1.19 needs jsr311-api to work properly (and fails if rs-api used instead) > * https://mvnrepository.com/artifact/javax.ws.rs/jsr311-api > * https://mvnrepository.com/artifact/javax.ws.rs/javax.ws.rs-api > Seems we will need to downgrade jackson to 2.12.7 because of jax-rs > compatibility issues in jackson 2.13 (see > https://github.com/FasterXML/jackson-jaxrs-providers/issues/134) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #4569: HADOOP-18332. Remove rs-api dependency by downgrading jackson to 2.12.7. (#4552)
steveloughran merged PR #4569: URL: https://github.com/apache/hadoop/pull/4569 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4463: YARN-11187. Remove WhiteBox in yarn module.
hadoop-yetus commented on PR #4463:
URL: https://github.com/apache/hadoop/pull/4463#issuecomment-1186550127
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 5s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 25s | | trunk passed |
| +1 :green_heart: | compile | 1m 43s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 1m 37s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 49s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 57s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 52s | | trunk passed |
| +1 :green_heart: | shadedclient | 27m 11s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 51s | | the patch passed |
| +1 :green_heart: | compile | 1m 37s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 1m 37s | |
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 0 new +
105 unchanged - 9 fixed = 105 total (was 114) |
| +1 :green_heart: | compile | 1m 32s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 32s | |
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new
+ 78 unchanged - 9 fixed = 78 total (was 87) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 35s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4463/9/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 49s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 38s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 45s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 23m 25s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 133m 38s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4463/9/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4463 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 4c8a9afa26b0 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b89d8e7523d5e9662337a4eabd8005f2b7181538 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
[GitHub] [hadoop] hadoop-yetus commented on pull request #4540: YARN-11160. Support getResourceProfiles, getResourceProfile API's for Federation
hadoop-yetus commented on PR #4540:
URL: https://github.com/apache/hadoop/pull/4540#issuecomment-1186549362
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 42s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | jsonlint | 0m 1s | | jsonlint was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 24s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 37s | | trunk passed |
| +1 :green_heart: | compile | 10m 4s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 8m 37s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 47s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 43s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 23s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 38s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 42s | | the patch passed |
| +1 :green_heart: | compile | 9m 47s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 9m 47s | | the patch passed |
| +1 :green_heart: | compile | 8m 39s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 8m 39s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 39s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4540/8/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 4 unchanged -
4 fixed = 5 total (was 8) |
| +1 :green_heart: | mvnsite | 2m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 21s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 11s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 54s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 9s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 1s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 15s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 53s | | The patch does not
generate ASF License warnings. |
| | | 164m 3s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4540/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4540 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets jsonlint
xmllint |
| uname | Linux dbbfc6cc90c2 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3
10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5198950f04a9b0be4b389154f4115c365caf9cfe |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
[GitHub] [hadoop] GauthamBanasandra merged pull request #4538: HDFS-16654. Link OpenSSL lib for CMake deps check
GauthamBanasandra merged PR #4538: URL: https://github.com/apache/hadoop/pull/4538 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4543: YARN-8900. [Router] Federation: routing getContainers REST invocations transparently to multiple RMs
hadoop-yetus commented on PR #4543:
URL: https://github.com/apache/hadoop/pull/4543#issuecomment-1186544184
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 13s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 19s | | trunk passed |
| +1 :green_heart: | compile | 4m 14s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 24s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 30s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 17s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 36s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 50s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 2s | | the patch passed |
| +1 :green_heart: | compile | 3m 56s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 3m 56s | | the patch passed |
| +1 :green_heart: | compile | 3m 18s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 14s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 9s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 2s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 0s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 47s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 3m 10s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 129m 27s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4543/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4543 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 48b1695935db 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a8cf91dff73978b1a1409eb44ab6f251fbc07341 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4543/7/testReport/ |
| Max. process+thread count | 1371 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4543/7/console |
| versions |
[GitHub] [hadoop] hadoop-yetus commented on pull request #4571: HDFS-16464. Create only libhdfspp static libraries for Windows
hadoop-yetus commented on PR #4571:
URL: https://github.com/apache/hadoop/pull/4571#issuecomment-1186534014
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 21m 34s | | trunk passed |
| +1 :green_heart: | compile | 3m 58s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 4m 4s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | mvnsite | 0m 44s | | trunk passed |
| +1 :green_heart: | shadedclient | 50m 32s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 24s | | the patch passed |
| +1 :green_heart: | compile | 3m 43s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | cc | 3m 43s | | the patch passed |
| +1 :green_heart: | golang | 3m 43s | | the patch passed |
| +1 :green_heart: | javac | 3m 43s | | the patch passed |
| +1 :green_heart: | compile | 3m 43s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 3m 43s | | the patch passed |
| +1 :green_heart: | golang | 3m 43s | | the patch passed |
| +1 :green_heart: | javac | 3m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 51s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 33m 9s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 115m 43s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4571 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell detsecrets golang |
| uname | Linux 06912fdbae28 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9dc5cd0d060ed87bdbc9b20e7bf6d4a90578bf68 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/testReport/ |
| Max. process+thread count | 683 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4540: YARN-11160. Support getResourceProfiles, getResourceProfile API's for Federation
slfan1989 commented on code in PR #4540:
URL: https://github.com/apache/hadoop/pull/4540#discussion_r922826810
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/util/resource/ResourceUtils.java:
##
@@ -917,4 +919,17 @@ private static void validateResourceTypes(
}
return res;
}
+
+ public static Resource mergeResources(Resource r1, Resource r2) {
Review Comment:
I will use the method `Resource add(Resource lhs, Resource rhs)`, this
method looks fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4540: YARN-11160. Support getResourceProfiles, getResourceProfile API's for Federation
slfan1989 commented on code in PR #4540:
URL: https://github.com/apache/hadoop/pull/4540#discussion_r922826623
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/util/resource/ResourceUtils.java:
##
@@ -115,8 +115,8 @@ private static void checkSpecialResources(
* Supporting 'memory', 'memory-mb', 'vcores' also as invalid resource
* names, in addition to 'MEMORY' for historical reasons
*/
-String[] keys = { "memory", ResourceInformation.MEMORY_URI,
-ResourceInformation.VCORES_URI };
+String[] keys = {"memory", ResourceInformation.MEMORY_URI,
Review Comment:
i will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4543: YARN-8900. [Router] Federation: routing getContainers REST invocations transparently to multiple RMs
slfan1989 commented on code in PR #4543:
URL: https://github.com/apache/hadoop/pull/4543#discussion_r922826523
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/webapp/MockDefaultRequestInterceptorREST.java:
##
@@ -231,4 +240,43 @@ public boolean isRunning() {
public void setRunning(boolean runningMode) {
this.isRunning = runningMode;
}
+
+ @Override
+ public ContainersInfo getContainers(HttpServletRequest req,
HttpServletResponse res,
+ String appId, String appAttemptId) {
+if (!isRunning) {
+ throw new RuntimeException("RM is stopped");
+}
+
+// We avoid to check if the Application exists in the system because we
need
+// to validate that each subCluster returns 1 container.
+ContainersInfo containers = new ContainersInfo();
+
+int subClusterId = Integer.valueOf(getSubClusterId().getId());
+
+ContainerId containerId = ContainerId.newContainerId(
+ApplicationAttemptId.fromString(appAttemptId), subClusterId);
+Resource allocatedResource =
+Resource.newInstance(subClusterId, subClusterId);
+
+NodeId assignedNode = NodeId.newInstance("Node", subClusterId);
+Priority priority = Priority.newInstance(subClusterId);
+long creationTime = subClusterId;
+long finishTime = subClusterId;
+String diagnosticInfo = "Diagnostic " + subClusterId;
+String logUrl = "Log " + subClusterId;
+int containerExitStatus = subClusterId;
+ContainerState containerState = ContainerState.COMPLETE;
+String nodeHttpAddress = "HttpAddress " + subClusterId;
+
+ContainerReport containerReport =
Review Comment:
Thanks for your suggestion, I will modify the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18302) Remove WhiteBox in hadoop-common module.
[ https://issues.apache.org/jira/browse/HADOOP-18302?focusedWorklogId=791739=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-791739 ] ASF GitHub Bot logged work on HADOOP-18302: --- Author: ASF GitHub Bot Created on: 17/Jul/22 12:30 Start Date: 17/Jul/22 12:30 Worklog Time Spent: 10m Work Description: slfan1989 commented on PR #4457: URL: https://github.com/apache/hadoop/pull/4457#issuecomment-1186502926 @aajisaka Please help me to review the code again, thank you very much! Issue Time Tracking --- Worklog Id: (was: 791739) Time Spent: 3h 10m (was: 3h) > Remove WhiteBox in hadoop-common module. > > > Key: HADOOP-18302 > URL: https://issues.apache.org/jira/browse/HADOOP-18302 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.4.0, 3.3.9 >Reporter: fanshilun >Assignee: fanshilun >Priority: Minor > Labels: pull-request-available > Time Spent: 3h 10m > Remaining Estimate: 0h > > WhiteBox is deprecated, try to remove this method in hadoop-common. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4571: HDFS-16464. Create only libhdfspp static libraries for Windows
hadoop-yetus commented on PR #4571:
URL: https://github.com/apache/hadoop/pull/4571#issuecomment-1186503041
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 21m 2s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 24m 53s | | trunk passed |
| +1 :green_heart: | compile | 3m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 44s | | trunk passed |
| +1 :green_heart: | shadedclient | 56m 15s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 25s | | the patch passed |
| +1 :green_heart: | compile | 3m 17s | | the patch passed |
| +1 :green_heart: | cc | 3m 17s | | the patch passed |
| +1 :green_heart: | golang | 3m 17s | | the patch passed |
| +1 :green_heart: | javac | 3m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 18s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 32m 22s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 143m 19s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4571 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell detsecrets golang |
| uname | Linux 1237e12f1dd5 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9dc5cd0d060ed87bdbc9b20e7bf6d4a90578bf68 |
| Default Java | Debian-11.0.15+10-post-Debian-1deb10u1 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/console |
| versions | git=2.20.1 maven=3.6.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4457: HADOOP-18302. Remove WhiteBox in hadoop-common module.
slfan1989 commented on PR #4457: URL: https://github.com/apache/hadoop/pull/4457#issuecomment-1186502926 @aajisaka Please help me to review the code again, thank you very much! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] KevinWikant commented on pull request #4568: HDFS-16664. Use correct GenerationStamp when invalidating corrupt block replicas
KevinWikant commented on PR #4568: URL: https://github.com/apache/hadoop/pull/4568#issuecomment-1186492545 ``` [ERROR] Tests run: 6803, Failures: 0, Errors: 2, Skipped: 24, Flakes: 8 [INFO] [ERROR] There are test failures ``` ``` [ERROR] Errors: [ERROR] org.apache.hadoop.hdfs.server.namenode.TestFsck.testFsckCorruptWhenOneReplicaIsCorrupt(org.apache.hadoop.hdfs.server.namenode.TestFsck) [ERROR] Run 1: TestFsck.testFsckCorruptWhenOneReplicaIsCorrupt » Remote java.lang.NullPointer... [ERROR] Run 2: TestFsck.testFsckCorruptWhenOneReplicaIsCorrupt » Remote java.lang.NullPointer... [ERROR] Run 3: TestFsck.testFsckCorruptWhenOneReplicaIsCorrupt » Remote java.lang.NullPointer... [INFO] [ERROR] org.apache.hadoop.hdfs.server.namenode.ha.TestObserverNode.testMkdirsRaceWithObserverRead(org.apache.hadoop.hdfs.server.namenode.ha.TestObserverNode) [ERROR] Run 1: TestObserverNode.testMkdirsRaceWithObserverRead:557 Client #3 lastSeenStateId=-9223372036854775808 activStateId=37 null [ERROR] Run 2: TestObserverNode.testMkdirsRaceWithObserverRead:511 » Connect Call From 653bab... [ERROR] Run 3: TestObserverNode.cleanUp:111 » Connect Call From 653bab42d979/172.17.0.2 to lo... [ERROR] Run 4: TestObserverNode.testMkdirsRaceWithObserverRead:511 » Connect Call From 653bab... [ERROR] Run 5: TestObserverNode.cleanUp:111 » Connect Call From 653bab42d979/172.17.0.2 to lo... ``` - TestObserverNode#testMkdirsRaceWithObserverRead is a flaky test: https://issues.apache.org/jira/browse/HDFS-15646 The other test failure is relevant TestFsck#testFsckCorruptWhenOneReplicaIsCorrupt: ``` [ERROR] Tests run: 35, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 281.876 s <<< FAILURE! - in org.apache.hadoop.hdfs.server.namenode.TestFsck [ERROR] testFsckCorruptWhenOneReplicaIsCorrupt(org.apache.hadoop.hdfs.server.namenode.TestFsck) Time elapsed: 7.446 s <<< ERROR! org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException at org.apache.hadoop.hdfs.server.blockmanagement.CorruptReplicasMap.removeFromCorruptReplicasMap(CorruptReplicasMap.java:145) at org.apache.hadoop.hdfs.server.blockmanagement.CorruptReplicasMap.removeFromCorruptReplicasMap(CorruptReplicasMap.java:134) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.removeStoredBlock(BlockManager.java:4255) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.removeStaleReplicas(BlockManager.java:4262) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.updateLastBlock(BlockManager.java:4772) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updatePipelineInternal(FSNamesystem.java:6004) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updatePipeline(FSNamesystem.java:5966) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updatePipeline(NameNodeRpcServer.java:1009) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updatePipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:1195) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:620) at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:588) at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:572) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1192) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1193) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1116) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1919) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3131) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4571: HDFS-16464. Create only libhdfspp static libraries for Windows
hadoop-yetus commented on PR #4571:
URL: https://github.com/apache/hadoop/pull/4571#issuecomment-1186466964
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 21m 27s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 36s | | trunk passed |
| +1 :green_heart: | compile | 4m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 47m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 3m 43s | | the patch passed |
| +1 :green_heart: | cc | 3m 43s | | the patch passed |
| +1 :green_heart: | golang | 3m 43s | | the patch passed |
| +1 :green_heart: | javac | 3m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 10s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 33m 5s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 2s | | The patch does not
generate ASF License warnings. |
| | | 129m 44s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4571 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell detsecrets golang |
| uname | Linux 5dd21a11e1e5 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9dc5cd0d060ed87bdbc9b20e7bf6d4a90578bf68 |
| Default Java | Red Hat, Inc.-1.8.0_312-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/testReport/ |
| Max. process+thread count | 680 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/console |
| versions | git=2.27.0 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18332) remove rs-api dependency by downgrading jackson to 2.12.7
[
https://issues.apache.org/jira/browse/HADOOP-18332?focusedWorklogId=791732=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-791732
]
ASF GitHub Bot logged work on HADOOP-18332:
---
Author: ASF GitHub Bot
Created on: 17/Jul/22 08:47
Start Date: 17/Jul/22 08:47
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4569:
URL: https://github.com/apache/hadoop/pull/4569#issuecomment-1186448527
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ branch-3.3.4 Compile Tests _ |
| +0 :ok: | mvndep | 5m 1s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 39m 17s | | branch-3.3.4 passed |
| +1 :green_heart: | compile | 17m 58s | | branch-3.3.4 passed |
| +1 :green_heart: | mvnsite | 25m 17s | | branch-3.3.4 passed |
| +1 :green_heart: | javadoc | 8m 15s | | branch-3.3.4 passed |
| +1 :green_heart: | shadedclient | 31m 24s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 48s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 41m 1s | | the patch passed |
| +1 :green_heart: | compile | 17m 28s | | the patch passed |
| +1 :green_heart: | javac | 17m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 20m 49s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 24s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 7m 57s | | the patch passed |
| +1 :green_heart: | shadedclient | 32m 52s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 682m 16s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4569/1/artifact/out/patch-unit-root.txt)
| root in the patch passed. |
| +1 :green_heart: | asflicense | 2m 37s | | The patch does not
generate ASF License warnings. |
| | | 920m 27s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.applications.distributedshell.TestDistributedShell |
| | hadoop.yarn.csi.client.TestCsiClient |
| |
hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4569/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4569 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell xml shellcheck shelldocs |
| uname | Linux 4f9771ad5505 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3.4 / 67ad712fec566a495d389db2bae49efc8f9b748a |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4569/1/testReport/ |
| Max. process+thread count | 3165 (vs. ulimit of 5500) |
| modules | C: hadoop-project
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
[GitHub] [hadoop] hadoop-yetus commented on pull request #4569: HADOOP-18332. Remove rs-api dependency by downgrading jackson to 2.12.7. (#4552)
hadoop-yetus commented on PR #4569:
URL: https://github.com/apache/hadoop/pull/4569#issuecomment-1186448527
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ branch-3.3.4 Compile Tests _ |
| +0 :ok: | mvndep | 5m 1s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 39m 17s | | branch-3.3.4 passed |
| +1 :green_heart: | compile | 17m 58s | | branch-3.3.4 passed |
| +1 :green_heart: | mvnsite | 25m 17s | | branch-3.3.4 passed |
| +1 :green_heart: | javadoc | 8m 15s | | branch-3.3.4 passed |
| +1 :green_heart: | shadedclient | 31m 24s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 48s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 41m 1s | | the patch passed |
| +1 :green_heart: | compile | 17m 28s | | the patch passed |
| +1 :green_heart: | javac | 17m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 20m 49s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 24s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 7m 57s | | the patch passed |
| +1 :green_heart: | shadedclient | 32m 52s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 682m 16s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4569/1/artifact/out/patch-unit-root.txt)
| root in the patch passed. |
| +1 :green_heart: | asflicense | 2m 37s | | The patch does not
generate ASF License warnings. |
| | | 920m 27s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.applications.distributedshell.TestDistributedShell |
| | hadoop.yarn.csi.client.TestCsiClient |
| |
hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4569/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4569 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell xml shellcheck shelldocs |
| uname | Linux 4f9771ad5505 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3.4 / 67ad712fec566a495d389db2bae49efc8f9b748a |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4569/1/testReport/ |
| Max. process+thread count | 3165 (vs. ulimit of 5500) |
| modules | C: hadoop-project
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router
hadoop-yarn-project/hadoop-yarn/
hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
[GitHub] [hadoop] hadoop-yetus commented on pull request #4571: HDFS-16464. Create only libhdfspp static libraries for Windows
hadoop-yetus commented on PR #4571:
URL: https://github.com/apache/hadoop/pull/4571#issuecomment-1186435539
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 3s | | trunk passed |
| +1 :green_heart: | compile | 4m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 65m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 22s | | the patch passed |
| +1 :green_heart: | compile | 3m 31s | | the patch passed |
| +1 :green_heart: | cc | 3m 31s | | the patch passed |
| +1 :green_heart: | golang | 3m 31s | | the patch passed |
| +1 :green_heart: | javac | 3m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 36m 19s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 53s | | The patch does not
generate ASF License warnings. |
| | | 129m 21s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4571 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell detsecrets golang |
| uname | Linux e4ee9e2b0bc6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9dc5cd0d060ed87bdbc9b20e7bf6d4a90578bf68 |
| Default Java | Red Hat, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/testReport/ |
| Max. process+thread count | 704 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18258) Merging of S3A Audit Logs
[
https://issues.apache.org/jira/browse/HADOOP-18258?focusedWorklogId=791718=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-791718
]
ASF GitHub Bot logged work on HADOOP-18258:
---
Author: ASF GitHub Bot
Created on: 17/Jul/22 06:01
Start Date: 17/Jul/22 06:01
Worklog Time Spent: 10m
Work Description: sravanigadey commented on code in PR #4383:
URL: https://github.com/apache/hadoop/pull/4383#discussion_r922774936
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/audit/S3AAuditLogMergerAndParser.java:
##
@@ -0,0 +1,546 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.audit;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.StringReader;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.avro.file.DataFileWriter;
+import org.apache.avro.io.DatumWriter;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.commons.io.IOUtils;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
+
+/**
+ * Merge all the audit logs present in a directory of
+ * multiple audit log files into a single audit log file.
+ */
+public class S3AAuditLogMergerAndParser {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(S3AAuditLogMergerAndParser.class);
+
+ /**
+ * Simple entry: anything up to a space.
+ * {@value}.
+ */
+ private static final String SIMPLE = "[^ ]*";
+
+ /**
+ * Date/Time. Everything within square braces.
+ * {@value}.
+ */
+ private static final String DATETIME = "\\[(.*?)\\]";
+
+ /**
+ * A natural number or "-".
+ * {@value}.
+ */
+ private static final String NUMBER = "(-|[0-9]*)";
+
+ /**
+ * A Quoted field or "-".
+ * {@value}.
+ */
+ private static final String QUOTED = "(-|\"[^\"]*\")";
+
+ /**
+ * An entry in the regexp.
+ *
+ * @param namename of the group
+ * @param pattern pattern to use in the regexp
+ * @return the pattern for the regexp
+ */
+ private static String e(String name, String pattern) {
+return String.format("(?<%s>%s) ", name, pattern);
+ }
+
+ /**
+ * An entry in the regexp.
+ *
+ * @param namename of the group
+ * @param pattern pattern to use in the regexp
+ * @return the pattern for the regexp
+ */
+ private static String eNoTrailing(String name, String pattern) {
+return String.format("(?<%s>%s)", name, pattern);
+ }
+
+ /**
+ * Simple entry using the {@link #SIMPLE} pattern.
+ *
+ * @param name name of the element (for code clarity only)
+ * @return the pattern for the regexp
+ */
+ private static String e(String name) {
+return e(name, SIMPLE);
+ }
+
+ /**
+ * Quoted entry using the {@link #QUOTED} pattern.
+ *
+ * @param name name of the element (for code clarity only)
+ * @return the pattern for the regexp
+ */
+ private static String q(String name) {
+return e(name, QUOTED);
+ }
+
+ /**
+ * Log group {@value}.
+ */
+ public static final String OWNER_GROUP = "owner";
+
+ /**
+ * Log group {@value}.
+ */
+ public static final String BUCKET_GROUP = "bucket";
+
+ /**
+ * Log group {@value}.
+ */
+ public static final String TIMESTAMP_GROUP = "timestamp";
+
+ /**
+ * Log group {@value}.
+ */
+ public static final String REMOTEIP_GROUP = "remoteip";
+
+ /**
+ * Log group {@value}.
+ */
+ public static final String REQUESTER_GROUP = "requester";
+
+ /**
+ * Log group {@value}.
+ */
+ public static final String REQUESTID_GROUP = "requestid";
+
+ /**
+ * Log group {@value}.
+ */
+ public static final
[GitHub] [hadoop] sravanigadey commented on a diff in pull request #4383: HADOOP-18258. Merging of S3A Audit Logs
sravanigadey commented on code in PR #4383:
URL: https://github.com/apache/hadoop/pull/4383#discussion_r922774936
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/audit/S3AAuditLogMergerAndParser.java:
##
@@ -0,0 +1,546 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.audit;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.StringReader;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.avro.file.DataFileWriter;
+import org.apache.avro.io.DatumWriter;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.commons.io.IOUtils;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
+
+/**
+ * Merge all the audit logs present in a directory of
+ * multiple audit log files into a single audit log file.
+ */
+public class S3AAuditLogMergerAndParser {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(S3AAuditLogMergerAndParser.class);
+
+ /**
+ * Simple entry: anything up to a space.
+ * {@value}.
+ */
+ private static final String SIMPLE = "[^ ]*";
+
+ /**
+ * Date/Time. Everything within square braces.
+ * {@value}.
+ */
+ private static final String DATETIME = "\\[(.*?)\\]";
+
+ /**
+ * A natural number or "-".
+ * {@value}.
+ */
+ private static final String NUMBER = "(-|[0-9]*)";
+
+ /**
+ * A Quoted field or "-".
+ * {@value}.
+ */
+ private static final String QUOTED = "(-|\"[^\"]*\")";
+
+ /**
+ * An entry in the regexp.
+ *
+ * @param namename of the group
+ * @param pattern pattern to use in the regexp
+ * @return the pattern for the regexp
+ */
+ private static String e(String name, String pattern) {
+return String.format("(?<%s>%s) ", name, pattern);
+ }
+
+ /**
+ * An entry in the regexp.
+ *
+ * @param namename of the group
+ * @param pattern pattern to use in the regexp
+ * @return the pattern for the regexp
+ */
+ private static String eNoTrailing(String name, String pattern) {
+return String.format("(?<%s>%s)", name, pattern);
+ }
+
+ /**
+ * Simple entry using the {@link #SIMPLE} pattern.
+ *
+ * @param name name of the element (for code clarity only)
+ * @return the pattern for the regexp
+ */
+ private static String e(String name) {
+return e(name, SIMPLE);
+ }
+
+ /**
+ * Quoted entry using the {@link #QUOTED} pattern.
+ *
+ * @param name name of the element (for code clarity only)
+ * @return the pattern for the regexp
+ */
+ private static String q(String name) {
+return e(name, QUOTED);
+ }
+
+ /**
+ * Log group {@value}.
+ */
+ public static final String OWNER_GROUP = "owner";
+
+ /**
+ * Log group {@value}.
+ */
+ public static final String BUCKET_GROUP = "bucket";
+
+ /**
+ * Log group {@value}.
+ */
+ public static final String TIMESTAMP_GROUP = "timestamp";
+
+ /**
+ * Log group {@value}.
+ */
+ public static final String REMOTEIP_GROUP = "remoteip";
+
+ /**
+ * Log group {@value}.
+ */
+ public static final String REQUESTER_GROUP = "requester";
+
+ /**
+ * Log group {@value}.
+ */
+ public static final String REQUESTID_GROUP = "requestid";
+
+ /**
+ * Log group {@value}.
+ */
+ public static final String VERB_GROUP = "verb";
+
+ /**
+ * Log group {@value}.
+ */
+ public static final String KEY_GROUP = "key";
+
+ /**
+ * Log group {@value}.
+ */
+ public static final String REQUESTURI_GROUP = "requesturi";
+
+ /**
+ * Log group {@value}.
+ */
+ public static final String HTTP_GROUP = "http";
+
+ /**
+ * Log group {@value}.
+ */
+ public static final String AWSERRORCODE_GROUP = "awserrorcode";
+
+
