[jira] [Work logged] (HADOOP-17464) Create hadoop-compression module
[ https://issues.apache.org/jira/browse/HADOOP-17464?focusedWorklogId=533356=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533356 ] ASF GitHub Bot logged work on HADOOP-17464: --- Author: ASF GitHub Bot Created on: 09/Jan/21 06:34 Start Date: 09/Jan/21 06:34 Worklog Time Spent: 10m Work Description: viirya commented on pull request #2611: URL: https://github.com/apache/hadoop/pull/2611#issuecomment-757105626 cc @dbtsai @sunchao @steveloughran This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 533356) Time Spent: 20m (was: 10m) > Create hadoop-compression module > > > Key: HADOOP-17464 > URL: https://issues.apache.org/jira/browse/HADOOP-17464 > Project: Hadoop Common > Issue Type: Improvement >Reporter: L. C. Hsieh >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > We added lz4-java, snappy-java dependencies to replace native libs. As per > the suggestion from the review comments, we better add a hadoop module to > have these extra dependencies, to avoid messing up the dependencies of user > application. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] viirya commented on pull request #2611: HADOOP-17464. Create hadoop-compression module
viirya commented on pull request #2611: URL: https://github.com/apache/hadoop/pull/2611#issuecomment-757105626 cc @dbtsai @sunchao @steveloughran This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17464) Create hadoop-compression module
[ https://issues.apache.org/jira/browse/HADOOP-17464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-17464: Labels: pull-request-available (was: ) > Create hadoop-compression module > > > Key: HADOOP-17464 > URL: https://issues.apache.org/jira/browse/HADOOP-17464 > Project: Hadoop Common > Issue Type: Improvement >Reporter: L. C. Hsieh >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > We added lz4-java, snappy-java dependencies to replace native libs. As per > the suggestion from the review comments, we better add a hadoop module to > have these extra dependencies, to avoid messing up the dependencies of user > application. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17464) Create hadoop-compression module
[ https://issues.apache.org/jira/browse/HADOOP-17464?focusedWorklogId=533355=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533355 ] ASF GitHub Bot logged work on HADOOP-17464: --- Author: ASF GitHub Bot Created on: 09/Jan/21 06:33 Start Date: 09/Jan/21 06:33 Worklog Time Spent: 10m Work Description: viirya opened a new pull request #2611: URL: https://github.com/apache/hadoop/pull/2611 See https://issues.apache.org/jira/browse/HADOOP-17464 for details. We added lz4-java, snappy-java dependencies to replace native libs ([HADOOP-17125](https://issues.apache.org/jira/browse/HADOOP-17125), [HADOOP-17292](https://issues.apache.org/jira/browse/HADOOP-17292)). As per the suggestion from the review comments, we better add a hadoop module to include these extra dependencies, to avoid messing up the dependencies of user application. Moved Lz4, Snappy Codec to hadoop-compression module. Codec test code are moved to hadoop-compression module too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 533355) Remaining Estimate: 0h Time Spent: 10m > Create hadoop-compression module > > > Key: HADOOP-17464 > URL: https://issues.apache.org/jira/browse/HADOOP-17464 > Project: Hadoop Common > Issue Type: Improvement >Reporter: L. C. Hsieh >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > We added lz4-java, snappy-java dependencies to replace native libs. As per > the suggestion from the review comments, we better add a hadoop module to > have these extra dependencies, to avoid messing up the dependencies of user > application. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] viirya opened a new pull request #2611: HADOOP-17464. Create hadoop-compression module
viirya opened a new pull request #2611: URL: https://github.com/apache/hadoop/pull/2611 See https://issues.apache.org/jira/browse/HADOOP-17464 for details. We added lz4-java, snappy-java dependencies to replace native libs ([HADOOP-17125](https://issues.apache.org/jira/browse/HADOOP-17125), [HADOOP-17292](https://issues.apache.org/jira/browse/HADOOP-17292)). As per the suggestion from the review comments, we better add a hadoop module to include these extra dependencies, to avoid messing up the dependencies of user application. Moved Lz4, Snappy Codec to hadoop-compression module. Codec test code are moved to hadoop-compression module too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17464) Create hadoop-compression module
L. C. Hsieh created HADOOP-17464: Summary: Create hadoop-compression module Key: HADOOP-17464 URL: https://issues.apache.org/jira/browse/HADOOP-17464 Project: Hadoop Common Issue Type: Improvement Reporter: L. C. Hsieh We added lz4-java, snappy-java dependencies to replace native libs. As per the suggestion from the review comments, we better add a hadoop module to have these extra dependencies, to avoid messing up the dependencies of user application. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2549: Hadoop 17428. ABFS: Implementation for getContentSummary
hadoop-yetus commented on pull request #2549:
URL: https://github.com/apache/hadoop/pull/2549#issuecomment-757104733
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 7s | | trunk passed |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 25s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 59s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 58s | | trunk passed |
| -0 :warning: | patch | 1m 18s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| -0 :warning: | checkstyle | 0m 17s |
[/diff-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2549/14/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 2 new + 2 unchanged - 0
fixed = 4 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 1s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 59s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 22s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 77m 46s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2549/14/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2549 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 07224f808847 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 890f2da6244 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2549/14/testReport/ |
| Max. process+thread count | 548 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2549/14/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT
[jira] [Resolved] (HADOOP-17462) Hadoop Client getRpcResponse May Return Wrong Result
[
https://issues.apache.org/jira/browse/HADOOP-17462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor resolved HADOOP-17462.
-
Resolution: Not A Problem
Thanks [~sjlee0] for pointing outing that the {{call}} object is synchronized
on.
It is implemented a bit differently in the reference docs. This functionality
is implemented within the same class itself instead of synchronizing and
accessing the variable from an external class, but I'm thinking it should be
the same.
https://docs.oracle.com/javase/tutorial/essential/concurrency/guardmeth.html
I'll test more in depth and if I can prove it definitively, I'll re-open this
ticket.
> Hadoop Client getRpcResponse May Return Wrong Result
>
>
> Key: HADOOP-17462
> URL: https://issues.apache.org/jira/browse/HADOOP-17462
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {code:java|Title=Client.java}
> /** @return the rpc response or, in case of timeout, null. */
> private Writable getRpcResponse(final Call call, final Connection
> connection,
> final long timeout, final TimeUnit unit) throws IOException {
> synchronized (call) {
> while (!call.done) {
> try {
> AsyncGet.Util.wait(call, timeout, unit);
> if (timeout >= 0 && !call.done) {
> return null;
> }
> } catch (InterruptedException ie) {
> Thread.currentThread().interrupt();
> throw new InterruptedIOException("Call interrupted");
> }
> }
> */
> static class Call {
> final int id; // call id
> final int retry; // retry count
> ...
> boolean done; // true when call is done
> ...
> }
> {code}
> The {{done}} variable is not marked as {{volatile}} so the thread which is
> checking its status is free to cache the value and never reload it even
> though it is expected to change by a different thread. The while loop may be
> stuck waiting for the change, but is always looking at a cached value. If
> that happens, timeout will occur and then return 'null'.
> In previous versions of Hadoop, there was no time-out at this level, so it
> would cause endless loop. Really tough error to track down if it happens.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17462) Hadoop Client getRpcResponse May Return Wrong Result
[
https://issues.apache.org/jira/browse/HADOOP-17462?focusedWorklogId=53=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-53
]
ASF GitHub Bot logged work on HADOOP-17462:
---
Author: ASF GitHub Bot
Created on: 09/Jan/21 03:56
Start Date: 09/Jan/21 03:56
Worklog Time Spent: 10m
Work Description: belugabehr closed pull request #2610:
URL: https://github.com/apache/hadoop/pull/2610
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 53)
Time Spent: 0.5h (was: 20m)
> Hadoop Client getRpcResponse May Return Wrong Result
>
>
> Key: HADOOP-17462
> URL: https://issues.apache.org/jira/browse/HADOOP-17462
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {code:java|Title=Client.java}
> /** @return the rpc response or, in case of timeout, null. */
> private Writable getRpcResponse(final Call call, final Connection
> connection,
> final long timeout, final TimeUnit unit) throws IOException {
> synchronized (call) {
> while (!call.done) {
> try {
> AsyncGet.Util.wait(call, timeout, unit);
> if (timeout >= 0 && !call.done) {
> return null;
> }
> } catch (InterruptedException ie) {
> Thread.currentThread().interrupt();
> throw new InterruptedIOException("Call interrupted");
> }
> }
> */
> static class Call {
> final int id; // call id
> final int retry; // retry count
> ...
> boolean done; // true when call is done
> ...
> }
> {code}
> The {{done}} variable is not marked as {{volatile}} so the thread which is
> checking its status is free to cache the value and never reload it even
> though it is expected to change by a different thread. The while loop may be
> stuck waiting for the change, but is always looking at a cached value. If
> that happens, timeout will occur and then return 'null'.
> In previous versions of Hadoop, there was no time-out at this level, so it
> would cause endless loop. Really tough error to track down if it happens.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] belugabehr closed pull request #2610: HADOOP-17462: Hadoop Client getRpcResponse May Return Wrong Result
belugabehr closed pull request #2610: URL: https://github.com/apache/hadoop/pull/2610 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17462) Hadoop Client getRpcResponse May Return Wrong Result
[
https://issues.apache.org/jira/browse/HADOOP-17462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261741#comment-17261741
]
David Mollitor commented on HADOOP-17462:
-
Let me review [~sjlee0]'s comments before pushing into the project.
> Hadoop Client getRpcResponse May Return Wrong Result
>
>
> Key: HADOOP-17462
> URL: https://issues.apache.org/jira/browse/HADOOP-17462
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {code:java|Title=Client.java}
> /** @return the rpc response or, in case of timeout, null. */
> private Writable getRpcResponse(final Call call, final Connection
> connection,
> final long timeout, final TimeUnit unit) throws IOException {
> synchronized (call) {
> while (!call.done) {
> try {
> AsyncGet.Util.wait(call, timeout, unit);
> if (timeout >= 0 && !call.done) {
> return null;
> }
> } catch (InterruptedException ie) {
> Thread.currentThread().interrupt();
> throw new InterruptedIOException("Call interrupted");
> }
> }
> */
> static class Call {
> final int id; // call id
> final int retry; // retry count
> ...
> boolean done; // true when call is done
> ...
> }
> {code}
> The {{done}} variable is not marked as {{volatile}} so the thread which is
> checking its status is free to cache the value and never reload it even
> though it is expected to change by a different thread. The while loop may be
> stuck waiting for the change, but is always looking at a cached value. If
> that happens, timeout will occur and then return 'null'.
> In previous versions of Hadoop, there was no time-out at this level, so it
> would cause endless loop. Really tough error to track down if it happens.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17414) Magic committer files don't have the count of bytes written collected by spark
[
https://issues.apache.org/jira/browse/HADOOP-17414?focusedWorklogId=533314=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533314
]
ASF GitHub Bot logged work on HADOOP-17414:
---
Author: ASF GitHub Bot
Created on: 09/Jan/21 01:41
Start Date: 09/Jan/21 01:41
Worklog Time Spent: 10m
Work Description: sunchao commented on a change in pull request #2530:
URL: https://github.com/apache/hadoop/pull/2530#discussion_r554269175
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/HeaderProcessing.java
##
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.TreeMap;
+
+import com.amazonaws.services.s3.Headers;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.Path;
+
+import static org.apache.hadoop.fs.s3a.Constants.HEADER_PREFIX;
+import static
org.apache.hadoop.fs.s3a.commit.CommitConstants.X_HEADER_MAGIC_MARKER;
+
+/**
+ * Part of the S3A FS where object headers are
+ * processed.
+ * Implements all the various XAttr read operations.
+ * Those APIs all expect byte arrays back.
+ * Metadata cloning is also implemented here, so as
+ * to stay in sync with custom header logic.
+ */
+public class HeaderProcessing extends AbstractStoreOperation {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ HeaderProcessing.class);
+
+ private static final byte[] EMPTY = new byte[0];
+
+ /**
+ * Length XAttr.
+ */
+ public static final String XA_CONTENT_LENGTH =
+ HEADER_PREFIX + Headers.CONTENT_LENGTH;
+
+ /**
+ * last modified XAttr.
+ */
+ public static final String XA_LAST_MODIFIED =
+ HEADER_PREFIX + Headers.LAST_MODIFIED;
+
+ public static final String XA_CONTENT_DISPOSITION =
+ HEADER_PREFIX + Headers.CONTENT_DISPOSITION;
+
+ public static final String XA_CONTENT_ENCODING =
+ HEADER_PREFIX + Headers.CONTENT_ENCODING;
+
+ public static final String XA_CONTENT_LANGUAGE =
+ HEADER_PREFIX + Headers.CONTENT_LANGUAGE;
+
+ public static final String XA_CONTENT_MD5 =
+ HEADER_PREFIX + Headers.CONTENT_MD5;
+
+ public static final String XA_CONTENT_RANGE =
+ HEADER_PREFIX + Headers.CONTENT_RANGE;
+
+ public static final String XA_CONTENT_TYPE =
+ HEADER_PREFIX + Headers.CONTENT_TYPE;
+
+ public static final String XA_ETAG = HEADER_PREFIX + Headers.ETAG;
+
+ public HeaderProcessing(final StoreContext storeContext) {
+super(storeContext);
+ }
+
+ /**
+ * Query the store, get all the headers into a map. Each Header
+ * has the "header." prefix.
+ * Caller must have read access.
+ * The value of each header is the string value of the object
+ * UTF-8 encoded.
+ * @param path path of object.
+ * @return the headers
+ * @throws IOException failure, including file not found.
+ */
+ private Map retrieveHeaders(Path path) throws IOException {
Review comment:
nit: I wonder if some kind of caching will be useful here. We are
calling `getObjectMetadata` for every `getXAttr` call.
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -330,6 +331,11 @@
*/
private DirectoryPolicy directoryPolicy;
+ /**
+ * Header processing for XAttr. Created on demand.
Review comment:
nit: is "Created on demand" accurate? it is created in `initialize`.
##
File path:
hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
##
@@ -1873,11 +1873,9 @@
fs.s3a.committer.magic.enabled
- false
+ true
Review comment:
I also wonder if this (along with the documentation change) is required
for this PR.
##
File path:
[GitHub] [hadoop] sunchao commented on a change in pull request #2530: HADOOP-17414. Magic committer files don't have the count of bytes written collected by spark
sunchao commented on a change in pull request #2530:
URL: https://github.com/apache/hadoop/pull/2530#discussion_r554269175
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/HeaderProcessing.java
##
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.TreeMap;
+
+import com.amazonaws.services.s3.Headers;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.Path;
+
+import static org.apache.hadoop.fs.s3a.Constants.HEADER_PREFIX;
+import static
org.apache.hadoop.fs.s3a.commit.CommitConstants.X_HEADER_MAGIC_MARKER;
+
+/**
+ * Part of the S3A FS where object headers are
+ * processed.
+ * Implements all the various XAttr read operations.
+ * Those APIs all expect byte arrays back.
+ * Metadata cloning is also implemented here, so as
+ * to stay in sync with custom header logic.
+ */
+public class HeaderProcessing extends AbstractStoreOperation {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ HeaderProcessing.class);
+
+ private static final byte[] EMPTY = new byte[0];
+
+ /**
+ * Length XAttr.
+ */
+ public static final String XA_CONTENT_LENGTH =
+ HEADER_PREFIX + Headers.CONTENT_LENGTH;
+
+ /**
+ * last modified XAttr.
+ */
+ public static final String XA_LAST_MODIFIED =
+ HEADER_PREFIX + Headers.LAST_MODIFIED;
+
+ public static final String XA_CONTENT_DISPOSITION =
+ HEADER_PREFIX + Headers.CONTENT_DISPOSITION;
+
+ public static final String XA_CONTENT_ENCODING =
+ HEADER_PREFIX + Headers.CONTENT_ENCODING;
+
+ public static final String XA_CONTENT_LANGUAGE =
+ HEADER_PREFIX + Headers.CONTENT_LANGUAGE;
+
+ public static final String XA_CONTENT_MD5 =
+ HEADER_PREFIX + Headers.CONTENT_MD5;
+
+ public static final String XA_CONTENT_RANGE =
+ HEADER_PREFIX + Headers.CONTENT_RANGE;
+
+ public static final String XA_CONTENT_TYPE =
+ HEADER_PREFIX + Headers.CONTENT_TYPE;
+
+ public static final String XA_ETAG = HEADER_PREFIX + Headers.ETAG;
+
+ public HeaderProcessing(final StoreContext storeContext) {
+super(storeContext);
+ }
+
+ /**
+ * Query the store, get all the headers into a map. Each Header
+ * has the "header." prefix.
+ * Caller must have read access.
+ * The value of each header is the string value of the object
+ * UTF-8 encoded.
+ * @param path path of object.
+ * @return the headers
+ * @throws IOException failure, including file not found.
+ */
+ private Map retrieveHeaders(Path path) throws IOException {
Review comment:
nit: I wonder if some kind of caching will be useful here. We are
calling `getObjectMetadata` for every `getXAttr` call.
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -330,6 +331,11 @@
*/
private DirectoryPolicy directoryPolicy;
+ /**
+ * Header processing for XAttr. Created on demand.
Review comment:
nit: is "Created on demand" accurate? it is created in `initialize`.
##
File path:
hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
##
@@ -1873,11 +1873,9 @@
fs.s3a.committer.magic.enabled
- false
+ true
Review comment:
I also wonder if this (along with the documentation change) is required
for this PR.
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/HeaderProcessing.java
##
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the
[GitHub] [hadoop] iwasakims merged pull request #2581: YARN-10553. Refactor TestDistributedShell
iwasakims merged pull request #2581: URL: https://github.com/apache/hadoop/pull/2581 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16524) Automatic keystore reloading for HttpServer2
[
https://issues.apache.org/jira/browse/HADOOP-16524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack updated HADOOP-16524:
---
Fix Version/s: 3.4.0
3.3.1
Hadoop Flags: Reviewed
Release Note:
Adds auto-reload of keystore.
Adds below new config (default 10 seconds):
ssl.{0}.stores.reload.interval
The refresh interval used to check if either of the truststore or keystore
certificate file has changed.
Assignee: Borislav Iordanov (was: Kihwal Lee)
Resolution: Fixed
Status: Resolved (was: Patch Available)
Merged to trunk and branch-3.3. Thanks for the patch [~borislav.iordanov] (I
added you as contributor and assigned you this issue).
> Automatic keystore reloading for HttpServer2
>
>
> Key: HADOOP-16524
> URL: https://issues.apache.org/jira/browse/HADOOP-16524
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Kihwal Lee
>Assignee: Borislav Iordanov
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1, 3.4.0
>
> Attachments: HADOOP-16524.patch
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> Jetty 9 simplified reloading of keystore. This allows hadoop daemon's SSL
> cert to be updated in place without having to restart the service.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2581: YARN-10553. Refactor TestDistributedShell
hadoop-yetus commented on pull request #2581:
URL: https://github.com/apache/hadoop/pull/2581#issuecomment-757016102
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 7 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 41s | | trunk passed |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 20s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 47s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 44s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | | the patch passed |
| +1 :green_heart: | compile | 0m 21s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 21s | | the patch passed |
| +1 :green_heart: | compile | 0m 19s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 19s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 17s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell:
The patch generated 0 new + 138 unchanged - 18 fixed = 138 total (was 156) |
| +1 :green_heart: | mvnsite | 0m 21s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 54s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 44s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 22m 21s | |
hadoop-yarn-applications-distributedshell in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 96m 36s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2581/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2581 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 2d824317a2ec 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 77435a025e5 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2581/6/testReport/ |
| Max. process+thread count | 617 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2581/6/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
[jira] [Work logged] (HADOOP-17462) Hadoop Client getRpcResponse May Return Wrong Result
[
https://issues.apache.org/jira/browse/HADOOP-17462?focusedWorklogId=533252=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533252
]
ASF GitHub Bot logged work on HADOOP-17462:
---
Author: ASF GitHub Bot
Created on: 08/Jan/21 21:54
Start Date: 08/Jan/21 21:54
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2610:
URL: https://github.com/apache/hadoop/pull/2610#issuecomment-757015565
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 4s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 15s | | trunk passed |
| +1 :green_heart: | compile | 21m 32s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 18m 8s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 48s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 44s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 2m 21s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 2m 18s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 53s | | the patch passed |
| +1 :green_heart: | compile | 20m 50s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 20m 50s | | the patch passed |
| +1 :green_heart: | compile | 18m 4s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 18m 4s | | the patch passed |
| -0 :warning: | checkstyle | 0m 46s |
[/diff-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2610/1/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt)
| hadoop-common-project/hadoop-common: The patch generated 1 new + 98
unchanged - 1 fixed = 99 total (was 99) |
| +1 :green_heart: | mvnsite | 1m 24s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 16m 58s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 1m 28s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 2m 22s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 9m 55s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 179m 25s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2610/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2610 |
| JIRA Issue | HADOOP-17462 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 916223cd519b 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23
18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e306f594218 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
[GitHub] [hadoop] hadoop-yetus commented on pull request #2610: HADOOP-17462: Hadoop Client getRpcResponse May Return Wrong Result
hadoop-yetus commented on pull request #2610:
URL: https://github.com/apache/hadoop/pull/2610#issuecomment-757015565
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 4s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 15s | | trunk passed |
| +1 :green_heart: | compile | 21m 32s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 18m 8s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 48s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 44s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 2m 21s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 2m 18s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 53s | | the patch passed |
| +1 :green_heart: | compile | 20m 50s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 20m 50s | | the patch passed |
| +1 :green_heart: | compile | 18m 4s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 18m 4s | | the patch passed |
| -0 :warning: | checkstyle | 0m 46s |
[/diff-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2610/1/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt)
| hadoop-common-project/hadoop-common: The patch generated 1 new + 98
unchanged - 1 fixed = 99 total (was 99) |
| +1 :green_heart: | mvnsite | 1m 24s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 16m 58s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 1m 28s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 2m 22s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 9m 55s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 179m 25s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2610/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2610 |
| JIRA Issue | HADOOP-17462 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 916223cd519b 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23
18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e306f594218 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2610/1/testReport/ |
| Max. process+thread count | 2239 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2610/1/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT
[jira] [Commented] (HADOOP-17462) Hadoop Client getRpcResponse May Return Wrong Result
[
https://issues.apache.org/jira/browse/HADOOP-17462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261610#comment-17261610
]
Hadoop QA commented on HADOOP-17462:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
4s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 35m
15s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
32s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
8s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 44s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
21s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
18s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
50s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
4s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 18m
4s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 46s{color} |
{color:orange}https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2610/1/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt{color}
| {color:orange} hadoop-common-project/hadoop-common: The patch generated 1
new + 98 unchanged - 1 fixed = 99 total (was 99) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 58s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our
[jira] [Work logged] (HADOOP-16524) Automatic keystore reloading for HttpServer2
[ https://issues.apache.org/jira/browse/HADOOP-16524?focusedWorklogId=533245=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533245 ] ASF GitHub Bot logged work on HADOOP-16524: --- Author: ASF GitHub Bot Created on: 08/Jan/21 21:45 Start Date: 08/Jan/21 21:45 Worklog Time Spent: 10m Work Description: saintstack merged pull request #2609: URL: https://github.com/apache/hadoop/pull/2609 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 533245) Time Spent: 4h 40m (was: 4.5h) > Automatic keystore reloading for HttpServer2 > > > Key: HADOOP-16524 > URL: https://issues.apache.org/jira/browse/HADOOP-16524 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-16524.patch > > Time Spent: 4h 40m > Remaining Estimate: 0h > > Jetty 9 simplified reloading of keystore. This allows hadoop daemon's SSL > cert to be updated in place without having to restart the service. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] saintstack merged pull request #2609: HADOOP-16524. Reloading SSL keystore for both DataNode and NameNode (…
saintstack merged pull request #2609: URL: https://github.com/apache/hadoop/pull/2609 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17463) Replace currentTimeMillis with monotonicNow in elapsed time
[
https://issues.apache.org/jira/browse/HADOOP-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261599#comment-17261599
]
Ahmed Hussein commented on HADOOP-17463:
[~kihwal], [~vinayakumarb], [~weichiu]. There was HDFS-6841 to address that
before.
It is obvious to change all busy waiting loops. Do you have any suggestions, or
thoughts of the elements that need actualTime, and how to prevent this pitfall
from happening in the future?
> Replace currentTimeMillis with monotonicNow in elapsed time
> ---
>
> Key: HADOOP-17463
> URL: https://issues.apache.org/jira/browse/HADOOP-17463
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
>
> I noticed that there is a widespread incorrect usage of
> {{System.currentTimeMillis()}} throughout the hadoop code.
> For example:
> {code:java}
> // Some comments here
> long start = System.currentTimeMillis();
> while (System.currentTimeMillis() - start < timeout) {
> // Do something
> }
> {code}
> Elapsed time should be measured using `monotonicNow()`.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16202) Stabilize openFile() and adopt internally
[
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=533238=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533238
]
ASF GitHub Bot logged work on HADOOP-16202:
---
Author: ASF GitHub Bot
Created on: 08/Jan/21 21:30
Start Date: 08/Jan/21 21:30
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-757007173
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 15 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 49s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 44s | | trunk passed |
| +1 :green_heart: | compile | 25m 51s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 22m 39s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 3m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 31m 37s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 5m 52s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 6m 32s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 1m 31s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 12m 13s | | trunk passed |
| -0 :warning: | patch | 1m 56s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 5m 17s | | the patch passed |
| +1 :green_heart: | compile | 19m 33s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 19m 33s | | the patch passed |
| +1 :green_heart: | compile | 17m 23s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 17m 23s | | the patch passed |
| -0 :warning: | checkstyle | 3m 1s |
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/2/artifact/out/diff-checkstyle-root.txt)
| root: The patch generated 6 new + 822 unchanged - 1 fixed = 828 total (was
823) |
| +1 :green_heart: | mvnsite | 7m 25s | | the patch passed |
| -1 :x: | whitespace | 0m 0s |
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/2/artifact/out/whitespace-eol.txt)
| The patch has 14 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | shadedclient | 15m 28s | | patch has no errors
when building and testing our client artifacts. |
| -1 :x: | javadoc | 1m 3s |
[/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/2/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| hadoop-common in the patch failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. |
| +1 :green_heart: | javadoc | 6m 36s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 12m 1s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 9m 41s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 4m 22s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 7m 7s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | unit | 8m 30s | | hadoop-mapreduce-client-app in
the patch passed. |
| +1 :green_heart: | unit | 12m 17s | | hadoop-distcp in the patch
passed. |
| +1 :green_heart: |
[GitHub] [hadoop] hadoop-yetus commented on pull request #2584: HADOOP-16202. Enhance openFile()
hadoop-yetus commented on pull request #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-757007173
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 15 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 49s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 44s | | trunk passed |
| +1 :green_heart: | compile | 25m 51s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 22m 39s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 3m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 31m 37s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 5m 52s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 6m 32s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 1m 31s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 12m 13s | | trunk passed |
| -0 :warning: | patch | 1m 56s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 5m 17s | | the patch passed |
| +1 :green_heart: | compile | 19m 33s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 19m 33s | | the patch passed |
| +1 :green_heart: | compile | 17m 23s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 17m 23s | | the patch passed |
| -0 :warning: | checkstyle | 3m 1s |
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/2/artifact/out/diff-checkstyle-root.txt)
| root: The patch generated 6 new + 822 unchanged - 1 fixed = 828 total (was
823) |
| +1 :green_heart: | mvnsite | 7m 25s | | the patch passed |
| -1 :x: | whitespace | 0m 0s |
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/2/artifact/out/whitespace-eol.txt)
| The patch has 14 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | shadedclient | 15m 28s | | patch has no errors
when building and testing our client artifacts. |
| -1 :x: | javadoc | 1m 3s |
[/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2584/2/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| hadoop-common in the patch failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. |
| +1 :green_heart: | javadoc | 6m 36s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 12m 1s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 9m 41s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 4m 22s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 7m 7s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | unit | 8m 30s | | hadoop-mapreduce-client-app in
the patch passed. |
| +1 :green_heart: | unit | 12m 17s | | hadoop-distcp in the patch
passed. |
| +1 :green_heart: | unit | 0m 59s | | hadoop-mapreduce-examples in
the patch passed. |
| +1 :green_heart: | unit | 6m 38s | | hadoop-streaming in the patch
passed. |
| +1 :green_heart: | unit | 1m 45s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 56s | | The patch does not
generate ASF License warnings. |
| | | 298m 38s | | |
| Subsystem | Report/Notes |
[GitHub] [hadoop] jbrennan333 merged pull request #2601: HADOOP-17408. Optimize NetworkTopology sorting block locations.
jbrennan333 merged pull request #2601: URL: https://github.com/apache/hadoop/pull/2601 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] belugabehr opened a new pull request #2610: HADOOP-17462: Hadoop Client getRpcResponse May Return Wrong Result
belugabehr opened a new pull request #2610: URL: https://github.com/apache/hadoop/pull/2610 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17463) Replace currentTimeMillis with monotonicNow in elapsed time
Ahmed Hussein created HADOOP-17463:
--
Summary: Replace currentTimeMillis with monotonicNow in elapsed
time
Key: HADOOP-17463
URL: https://issues.apache.org/jira/browse/HADOOP-17463
Project: Hadoop Common
Issue Type: Bug
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein
I noticed that there is a widespread incorrect usage of
{{System.currentTimeMillis()}} throughout the hadoop code.
For example:
{code:java}
// Some comments here
long start = System.currentTimeMillis();
while (System.currentTimeMillis() - start < timeout) {
// Do something
}
{code}
Elapsed time should be measured using `monotonicNow()`.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] amahussein commented on a change in pull request #2581: YARN-10553. Refactor TestDistributedShell
amahussein commented on a change in pull request #2581:
URL: https://github.com/apache/hadoop/pull/2581#discussion_r554177697
##
File path:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/DistributedShellBaseTest.java
##
@@ -0,0 +1,557 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.applications.distributedshell;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.URL;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Supplier;
+
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.rules.TemporaryFolder;
+import org.junit.rules.TestName;
+import org.junit.rules.Timeout;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileContext;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.net.ServerSocketUtil;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.util.JarFinder;
+import org.apache.hadoop.util.Shell;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptId;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptReport;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.ContainerReport;
+import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.api.records.timeline.TimelineDomain;
+import org.apache.hadoop.yarn.api.records.timeline.TimelineEntities;
+import org.apache.hadoop.yarn.api.records.timeline.TimelineEntity;
+import org.apache.hadoop.yarn.client.api.YarnClient;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.server.MiniYARNCluster;
+import org.apache.hadoop.yarn.server.nodemanager.NodeManager;
+import org.apache.hadoop.yarn.server.resourcemanager.RMContext;
+import org.apache.hadoop.yarn.server.resourcemanager.ResourceManager;
+import org.apache.hadoop.yarn.server.timeline.NameValuePair;
+import org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin;
+import org.apache.hadoop.yarn.util.ProcfsBasedProcessTree;
+
+/**
+ * Base class for testing DistributedShell features.
+ */
+public abstract class DistributedShellBaseTest {
+ protected static final int MIN_ALLOCATION_MB = 128;
+ protected static final int NUM_DATA_NODES = 1;
+ protected static final int TEST_TIME_OUT = 16;
+ // set the timeout of the yarnClient to be 95% of the globalTimeout.
+ protected static final int TEST_TIME_WINDOW_EXPIRE =
+ (TEST_TIME_OUT * 90) / 100;
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DistributedShellBaseTest.class);
+ private static final String APP_MASTER_JAR =
+ JarFinder.getJar(ApplicationMaster.class);
+ private static final int NUM_NMS = 1;
+ // set the timeout of the yarnClient to be 95% of the globalTimeout.
+ private static final String YARN_CLIENT_TIMEOUT =
+ String.valueOf(TEST_TIME_WINDOW_EXPIRE);
+ private static final String[] COMMON_ARGS = {
+ "--jar",
+ APP_MASTER_JAR,
+ "--timeout",
+ YARN_CLIENT_TIMEOUT,
+ "--appname",
+ ""
+ };
+ @Rule
+ public Timeout globalTimeout = new Timeout(TEST_TIME_OUT,
+ TimeUnit.MILLISECONDS);
+ @Rule
+ public TemporaryFolder tmpFolder = new TemporaryFolder();
+ @Rule
+ public TestName name = new TestName();
+ private Client dsClient;
+ private MiniYARNCluster yarnCluster;
+ private YarnConfiguration conf = null;
+ // location of the filesystem timeline writer for timeline service v.2
+ private String timelineV2StorageDir = null;
+
+
[jira] [Commented] (HADOOP-17462) Hadoop Client getRpcResponse May Return Wrong Result
[
https://issues.apache.org/jira/browse/HADOOP-17462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261562#comment-17261562
]
Sangjin Lee commented on HADOOP-17462:
--
I might be coming in cold here, not having looked at this code in a long while,
but if the {{done}} variable is read or written always with synchronization,
that should make it safe without making it volatile. A cursory look seems to
indicate that {{done}} is accessed only with the instance lock held. So making
it volatile is not going to change things.
Perhaps there is a deadlock or other situation that's causing the pile-up?
> Hadoop Client getRpcResponse May Return Wrong Result
>
>
> Key: HADOOP-17462
> URL: https://issues.apache.org/jira/browse/HADOOP-17462
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java|Title=Client.java}
> /** @return the rpc response or, in case of timeout, null. */
> private Writable getRpcResponse(final Call call, final Connection
> connection,
> final long timeout, final TimeUnit unit) throws IOException {
> synchronized (call) {
> while (!call.done) {
> try {
> AsyncGet.Util.wait(call, timeout, unit);
> if (timeout >= 0 && !call.done) {
> return null;
> }
> } catch (InterruptedException ie) {
> Thread.currentThread().interrupt();
> throw new InterruptedIOException("Call interrupted");
> }
> }
> */
> static class Call {
> final int id; // call id
> final int retry; // retry count
> ...
> boolean done; // true when call is done
> ...
> }
> {code}
> The {{done}} variable is not marked as {{volatile}} so the thread which is
> checking its status is free to cache the value and never reload it even
> though it is expected to change by a different thread. The while loop may be
> stuck waiting for the change, but is always looking at a cached value. If
> that happens, timeout will occur and then return 'null'.
> In previous versions of Hadoop, there was no time-out at this level, so it
> would cause endless loop. Really tough error to track down if it happens.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17408) Optimize NetworkTopology while sorting of block locations
[
https://issues.apache.org/jira/browse/HADOOP-17408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Brennan resolved HADOOP-17408.
--
Fix Version/s: 3.4.0
3.3.1
Resolution: Fixed
Thanks for the contribution [~ahussein] and [~daryn]! I have committed this to
trunk and branch-3.3.
The patch does not apply cleanly to branch-3.2 or earlier. Please provide a
patch for 3.2 if desired.
> Optimize NetworkTopology while sorting of block locations
> -
>
> Key: HADOOP-17408
> URL: https://issues.apache.org/jira/browse/HADOOP-17408
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, net
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1, 3.4.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> In {{NetworkTopology}}, I noticed that there are some hanging fruits to
> improve the performance.
> Inside {{sortByDistance}}, collections.shuffle is performed on the list
> before calling {{secondarySort}}.
> {code:java}
> Collections.shuffle(list, r);
> if (secondarySort != null) {
> secondarySort.accept(list);
> }
> {code}
> However, in different call sites, {{collections.shuffle}} is passed as the
> secondarySort to {{sortByDistance}}. This means that the shuffle is executed
> twice on each list.
> Also, logic wise, it is useless to shuffle before applying a tie breaker
> which might make the shuffle work obsolete.
> In addition, [~daryn] reported that:
> * topology is unnecessarily locking/unlocking to calculate the distance for
> every node
> * shuffling uses a seeded Random, instead of ThreadLocalRandom, which is
> heavily synchronized
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16524) Automatic keystore reloading for HttpServer2
[ https://issues.apache.org/jira/browse/HADOOP-16524?focusedWorklogId=533195=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533195 ] ASF GitHub Bot logged work on HADOOP-16524: --- Author: ASF GitHub Bot Created on: 08/Jan/21 19:47 Start Date: 08/Jan/21 19:47 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2609: URL: https://github.com/apache/hadoop/pull/2609#issuecomment-756959446 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 24m 44s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ branch-3.3 Compile Tests _ | | +1 :green_heart: | mvninstall | 32m 14s | branch-3.3 passed | | +1 :green_heart: | compile | 15m 54s | branch-3.3 passed | | +1 :green_heart: | checkstyle | 0m 55s | branch-3.3 passed | | +1 :green_heart: | mvnsite | 1m 29s | branch-3.3 passed | | +1 :green_heart: | shadedclient | 17m 50s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 4s | branch-3.3 passed | | +0 :ok: | spotbugs | 2m 14s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 2m 11s | branch-3.3 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 53s | the patch passed | | +1 :green_heart: | compile | 15m 12s | the patch passed | | +1 :green_heart: | javac | 15m 12s | the patch passed | | -0 :warning: | checkstyle | 0m 55s | hadoop-common-project/hadoop-common: The patch generated 166 new + 68 unchanged - 11 fixed = 234 total (was 79) | | +1 :green_heart: | mvnsite | 1m 25s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 15m 23s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 3s | the patch passed | | +1 :green_heart: | findbugs | 2m 17s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 9m 28s | hadoop-common in the patch passed. | | +1 :green_heart: | asflicense | 0m 57s | The patch does not generate ASF License warnings. | | | | 145m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2609/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2609 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux c1ffdcedb677 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | branch-3.3 / f6b9f82 | | Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~16.04-b01 | | checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2609/1/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2609/1/testReport/ | | Max. process+thread count | 1691 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2609/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 533195) Time Spent: 4.5h (was: 4h 20m) > Automatic keystore reloading for HttpServer2 > > > Key: HADOOP-16524 > URL: https://issues.apache.org/jira/browse/HADOOP-16524 > Project: Hadoop Common > Issue Type: Improvement >Reporter:
[GitHub] [hadoop] hadoop-yetus commented on pull request #2609: HADOOP-16524. Reloading SSL keystore for both DataNode and NameNode (…
hadoop-yetus commented on pull request #2609: URL: https://github.com/apache/hadoop/pull/2609#issuecomment-756959446 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 24m 44s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ branch-3.3 Compile Tests _ | | +1 :green_heart: | mvninstall | 32m 14s | branch-3.3 passed | | +1 :green_heart: | compile | 15m 54s | branch-3.3 passed | | +1 :green_heart: | checkstyle | 0m 55s | branch-3.3 passed | | +1 :green_heart: | mvnsite | 1m 29s | branch-3.3 passed | | +1 :green_heart: | shadedclient | 17m 50s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 4s | branch-3.3 passed | | +0 :ok: | spotbugs | 2m 14s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 2m 11s | branch-3.3 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 53s | the patch passed | | +1 :green_heart: | compile | 15m 12s | the patch passed | | +1 :green_heart: | javac | 15m 12s | the patch passed | | -0 :warning: | checkstyle | 0m 55s | hadoop-common-project/hadoop-common: The patch generated 166 new + 68 unchanged - 11 fixed = 234 total (was 79) | | +1 :green_heart: | mvnsite | 1m 25s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 15m 23s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 3s | the patch passed | | +1 :green_heart: | findbugs | 2m 17s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 9m 28s | hadoop-common in the patch passed. | | +1 :green_heart: | asflicense | 0m 57s | The patch does not generate ASF License warnings. | | | | 145m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2609/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2609 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux c1ffdcedb677 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | branch-3.3 / f6b9f82 | | Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~16.04-b01 | | checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2609/1/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2609/1/testReport/ | | Max. process+thread count | 1691 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2609/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-17462) Hadoop Client getRpcResponse May Return Wrong Result
[
https://issues.apache.org/jira/browse/HADOOP-17462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261538#comment-17261538
]
David Mollitor edited comment on HADOOP-17462 at 1/8/21, 7:22 PM:
--
[~elgoiri] Hello old friend :) Happy new year. It's been a while.
I am looking at a Hive scenario where the server jstack revealed many hundreds
of threads were stuck on this code. I don't know 100% that the code was stuck
in an endless loop, it could possibly be that HDFS services are slow to respond
and therefore requests are backing up for Hive, but it is my understanding that
the end-user is testing HDFS with other tooling at the same time the see this
issue in Hive and the other tools do not seem to be stuck in the same way as
Hive.
I saw lots of threads blocking here, I noticed this classic issue, just
speculating at this point.
was (Author: belugabehr):
[~elgoiri] Hello old friend :) Happy new year. It's been a while.
I am looking at a Hive scenario where the server jstack revealed many hundreds
of threads were stuck on this code. I don't know 100% that the code was stuck
in an endless loop, it could possibly be that HDFS services are slow to respond
and therefore requests are backing up for Hive, but it is my understanding that
the end-user is testing HDFS with other tooling at the same time the see this
issue in Hive and the other tools do not seem to be stuck in the same way as
Hive.
I saw lots to threads blocking here, I noticed this classic issue, just
speculating at this point.
> Hadoop Client getRpcResponse May Return Wrong Result
>
>
> Key: HADOOP-17462
> URL: https://issues.apache.org/jira/browse/HADOOP-17462
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java|Title=Client.java}
> /** @return the rpc response or, in case of timeout, null. */
> private Writable getRpcResponse(final Call call, final Connection
> connection,
> final long timeout, final TimeUnit unit) throws IOException {
> synchronized (call) {
> while (!call.done) {
> try {
> AsyncGet.Util.wait(call, timeout, unit);
> if (timeout >= 0 && !call.done) {
> return null;
> }
> } catch (InterruptedException ie) {
> Thread.currentThread().interrupt();
> throw new InterruptedIOException("Call interrupted");
> }
> }
> */
> static class Call {
> final int id; // call id
> final int retry; // retry count
> ...
> boolean done; // true when call is done
> ...
> }
> {code}
> The {{done}} variable is not marked as {{volatile}} so the thread which is
> checking its status is free to cache the value and never reload it even
> though it is expected to change by a different thread. The while loop may be
> stuck waiting for the change, but is always looking at a cached value. If
> that happens, timeout will occur and then return 'null'.
> In previous versions of Hadoop, there was no time-out at this level, so it
> would cause endless loop. Really tough error to track down if it happens.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17462) Hadoop Client getRpcResponse May Return Wrong Result
[
https://issues.apache.org/jira/browse/HADOOP-17462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261538#comment-17261538
]
David Mollitor commented on HADOOP-17462:
-
[~elgoiri] Hello old friend :) Happy new year. It's been a while.
I am looking at a Hive scenario where the server jstack revealed many hundreds
of threads were stuck on this code. I don't know 100% that the code was stuck
in an endless loop, it could possibly be that HDFS services are slow to respond
and therefore requests are backing up for Hive, but it is my understanding that
the end-user is testing HDFS with other tooling at the same time the see this
issue in Hive and the other tools do not seem to be stuck in the same way as
Hive.
I saw lots to threads blocking here, I noticed this classic issue, just
speculating at this point.
> Hadoop Client getRpcResponse May Return Wrong Result
>
>
> Key: HADOOP-17462
> URL: https://issues.apache.org/jira/browse/HADOOP-17462
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java|Title=Client.java}
> /** @return the rpc response or, in case of timeout, null. */
> private Writable getRpcResponse(final Call call, final Connection
> connection,
> final long timeout, final TimeUnit unit) throws IOException {
> synchronized (call) {
> while (!call.done) {
> try {
> AsyncGet.Util.wait(call, timeout, unit);
> if (timeout >= 0 && !call.done) {
> return null;
> }
> } catch (InterruptedException ie) {
> Thread.currentThread().interrupt();
> throw new InterruptedIOException("Call interrupted");
> }
> }
> */
> static class Call {
> final int id; // call id
> final int retry; // retry count
> ...
> boolean done; // true when call is done
> ...
> }
> {code}
> The {{done}} variable is not marked as {{volatile}} so the thread which is
> checking its status is free to cache the value and never reload it even
> though it is expected to change by a different thread. The while loop may be
> stuck waiting for the change, but is always looking at a cached value. If
> that happens, timeout will occur and then return 'null'.
> In previous versions of Hadoop, there was no time-out at this level, so it
> would cause endless loop. Really tough error to track down if it happens.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17451) intermittent failure of S3A tests which make assertions on statistics/IOStatistics
[
https://issues.apache.org/jira/browse/HADOOP-17451?focusedWorklogId=533177=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533177
]
ASF GitHub Bot logged work on HADOOP-17451:
---
Author: ASF GitHub Bot
Created on: 08/Jan/21 19:15
Start Date: 08/Jan/21 19:15
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2594:
URL: https://github.com/apache/hadoop/pull/2594#issuecomment-756945504
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 42s | | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 26m 14s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/4/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 0m 28s |
[/branch-compile-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/4/artifact/out/branch-compile-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| root in trunk failed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. |
| -1 :x: | compile | 0m 28s |
[/branch-compile-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/4/artifact/out/branch-compile-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| root in trunk failed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. |
| -0 :warning: | checkstyle | 0m 26s |
[/buildtool-branch-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/4/artifact/out/buildtool-branch-checkstyle-root.txt)
| The patch fails to run checkstyle in root |
| -1 :x: | mvnsite | 0m 27s |
[/branch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/4/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in trunk failed. |
| -1 :x: | mvnsite | 0m 28s |
[/branch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/4/artifact/out/branch-mvnsite-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in trunk failed. |
| +1 :green_heart: | shadedclient | 2m 2s | | branch has no errors
when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 30s |
[/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/4/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| hadoop-common in trunk failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. |
| -1 :x: | javadoc | 0m 31s |
[/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/4/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| hadoop-aws in trunk failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. |
| -1 :x: | javadoc | 0m 30s |
[/branch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/4/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| hadoop-common in trunk failed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. |
| -1 :x: | javadoc | 0m 25s |
[/branch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/4/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| hadoop-aws in trunk failed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. |
| +0 :ok: | spotbugs | 4m 58s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| -1 :x: | findbugs | 0m 28s |
[GitHub] [hadoop] hadoop-yetus commented on pull request #2594: HADOOP-17451. IOStatistics test failures in S3A code.
hadoop-yetus commented on pull request #2594:
URL: https://github.com/apache/hadoop/pull/2594#issuecomment-756945504
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 42s | | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 26m 14s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/4/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 0m 28s |
[/branch-compile-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/4/artifact/out/branch-compile-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| root in trunk failed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. |
| -1 :x: | compile | 0m 28s |
[/branch-compile-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/4/artifact/out/branch-compile-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| root in trunk failed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. |
| -0 :warning: | checkstyle | 0m 26s |
[/buildtool-branch-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/4/artifact/out/buildtool-branch-checkstyle-root.txt)
| The patch fails to run checkstyle in root |
| -1 :x: | mvnsite | 0m 27s |
[/branch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/4/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in trunk failed. |
| -1 :x: | mvnsite | 0m 28s |
[/branch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/4/artifact/out/branch-mvnsite-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in trunk failed. |
| +1 :green_heart: | shadedclient | 2m 2s | | branch has no errors
when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 30s |
[/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/4/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| hadoop-common in trunk failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. |
| -1 :x: | javadoc | 0m 31s |
[/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/4/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| hadoop-aws in trunk failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. |
| -1 :x: | javadoc | 0m 30s |
[/branch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/4/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| hadoop-common in trunk failed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. |
| -1 :x: | javadoc | 0m 25s |
[/branch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/4/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| hadoop-aws in trunk failed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. |
| +0 :ok: | spotbugs | 4m 58s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| -1 :x: | findbugs | 0m 28s |
[/branch-findbugs-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/4/artifact/out/branch-findbugs-hadoop-common-project_hadoop-common.txt)
| hadoop-common in trunk failed. |
| -1 :x: | findbugs | 0m 28s |
[/branch-findbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/4/artifact/out/branch-findbugs-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in trunk failed. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | | Maven dependency ordering for patch
[jira] [Work logged] (HADOOP-17408) Optimize NetworkTopology while sorting of block locations
[
https://issues.apache.org/jira/browse/HADOOP-17408?focusedWorklogId=533175=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533175
]
ASF GitHub Bot logged work on HADOOP-17408:
---
Author: ASF GitHub Bot
Created on: 08/Jan/21 19:10
Start Date: 08/Jan/21 19:10
Worklog Time Spent: 10m
Work Description: jbrennan333 merged pull request #2601:
URL: https://github.com/apache/hadoop/pull/2601
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 533175)
Time Spent: 1h 50m (was: 1h 40m)
> Optimize NetworkTopology while sorting of block locations
> -
>
> Key: HADOOP-17408
> URL: https://issues.apache.org/jira/browse/HADOOP-17408
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, net
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> In {{NetworkTopology}}, I noticed that there are some hanging fruits to
> improve the performance.
> Inside {{sortByDistance}}, collections.shuffle is performed on the list
> before calling {{secondarySort}}.
> {code:java}
> Collections.shuffle(list, r);
> if (secondarySort != null) {
> secondarySort.accept(list);
> }
> {code}
> However, in different call sites, {{collections.shuffle}} is passed as the
> secondarySort to {{sortByDistance}}. This means that the shuffle is executed
> twice on each list.
> Also, logic wise, it is useless to shuffle before applying a tie breaker
> which might make the shuffle work obsolete.
> In addition, [~daryn] reported that:
> * topology is unnecessarily locking/unlocking to calculate the distance for
> every node
> * shuffling uses a seeded Random, instead of ThreadLocalRandom, which is
> heavily synchronized
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17462) Hadoop Client getRpcResponse May Return Wrong Result
[
https://issues.apache.org/jira/browse/HADOOP-17462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261524#comment-17261524
]
Íñigo Goiri commented on HADOOP-17462:
--
Interesting finding.
Can you share in what scenario you found this?
> Hadoop Client getRpcResponse May Return Wrong Result
>
>
> Key: HADOOP-17462
> URL: https://issues.apache.org/jira/browse/HADOOP-17462
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java|Title=Client.java}
> /** @return the rpc response or, in case of timeout, null. */
> private Writable getRpcResponse(final Call call, final Connection
> connection,
> final long timeout, final TimeUnit unit) throws IOException {
> synchronized (call) {
> while (!call.done) {
> try {
> AsyncGet.Util.wait(call, timeout, unit);
> if (timeout >= 0 && !call.done) {
> return null;
> }
> } catch (InterruptedException ie) {
> Thread.currentThread().interrupt();
> throw new InterruptedIOException("Call interrupted");
> }
> }
> */
> static class Call {
> final int id; // call id
> final int retry; // retry count
> ...
> boolean done; // true when call is done
> ...
> }
> {code}
> The {{done}} variable is not marked as {{volatile}} so the thread which is
> checking its status is free to cache the value and never reload it even
> though it is expected to change by a different thread. The while loop may be
> stuck waiting for the change, but is always looking at a cached value. If
> that happens, timeout will occur and then return 'null'.
> In previous versions of Hadoop, there was no time-out at this level, so it
> would cause endless loop. Really tough error to track down if it happens.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17462) Hadoop Client getRpcResponse May Return Wrong Result
[
https://issues.apache.org/jira/browse/HADOOP-17462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HADOOP-17462:
Labels: pull-request-available (was: )
> Hadoop Client getRpcResponse May Return Wrong Result
>
>
> Key: HADOOP-17462
> URL: https://issues.apache.org/jira/browse/HADOOP-17462
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java|Title=Client.java}
> /** @return the rpc response or, in case of timeout, null. */
> private Writable getRpcResponse(final Call call, final Connection
> connection,
> final long timeout, final TimeUnit unit) throws IOException {
> synchronized (call) {
> while (!call.done) {
> try {
> AsyncGet.Util.wait(call, timeout, unit);
> if (timeout >= 0 && !call.done) {
> return null;
> }
> } catch (InterruptedException ie) {
> Thread.currentThread().interrupt();
> throw new InterruptedIOException("Call interrupted");
> }
> }
> */
> static class Call {
> final int id; // call id
> final int retry; // retry count
> ...
> boolean done; // true when call is done
> ...
> }
> {code}
> The {{done}} variable is not marked as {{volatile}} so the thread which is
> checking its status is free to cache the value and never reload it even
> though it is expected to change by a different thread. The while loop may be
> stuck waiting for the change, but is always looking at a cached value. If
> that happens, timeout will occur and then return 'null'.
> In previous versions of Hadoop, there was no time-out at this level, so it
> would cause endless loop. Really tough error to track down if it happens.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17462) Hadoop Client getRpcResponse May Return Wrong Result
[
https://issues.apache.org/jira/browse/HADOOP-17462?focusedWorklogId=533172=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533172
]
ASF GitHub Bot logged work on HADOOP-17462:
---
Author: ASF GitHub Bot
Created on: 08/Jan/21 18:53
Start Date: 08/Jan/21 18:53
Worklog Time Spent: 10m
Work Description: belugabehr opened a new pull request #2610:
URL: https://github.com/apache/hadoop/pull/2610
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 533172)
Remaining Estimate: 0h
Time Spent: 10m
> Hadoop Client getRpcResponse May Return Wrong Result
>
>
> Key: HADOOP-17462
> URL: https://issues.apache.org/jira/browse/HADOOP-17462
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java|Title=Client.java}
> /** @return the rpc response or, in case of timeout, null. */
> private Writable getRpcResponse(final Call call, final Connection
> connection,
> final long timeout, final TimeUnit unit) throws IOException {
> synchronized (call) {
> while (!call.done) {
> try {
> AsyncGet.Util.wait(call, timeout, unit);
> if (timeout >= 0 && !call.done) {
> return null;
> }
> } catch (InterruptedException ie) {
> Thread.currentThread().interrupt();
> throw new InterruptedIOException("Call interrupted");
> }
> }
> */
> static class Call {
> final int id; // call id
> final int retry; // retry count
> ...
> boolean done; // true when call is done
> ...
> }
> {code}
> The {{done}} variable is not marked as {{volatile}} so the thread which is
> checking its status is free to cache the value and never reload it even
> though it is expected to change by a different thread. The while loop may be
> stuck waiting for the change, but is always looking at a cached value. If
> that happens, timeout will occur and then return 'null'.
> In previous versions of Hadoop, there was no time-out at this level, so it
> would cause endless loop. Really tough error to track down if it happens.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17462) Hadoop Client getRpcResponse May Return Wrong Result
[
https://issues.apache.org/jira/browse/HADOOP-17462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HADOOP-17462:
Description:
{code:java|Title=Client.java}
/** @return the rpc response or, in case of timeout, null. */
private Writable getRpcResponse(final Call call, final Connection connection,
final long timeout, final TimeUnit unit) throws IOException {
synchronized (call) {
while (!call.done) {
try {
AsyncGet.Util.wait(call, timeout, unit);
if (timeout >= 0 && !call.done) {
return null;
}
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
throw new InterruptedIOException("Call interrupted");
}
}
*/
static class Call {
final int id; // call id
final int retry; // retry count
...
boolean done; // true when call is done
...
}
{code}
The {{done}} variable is not marked as {{volatile}} so the thread which is
checking its status is free to cache the value and never reload it even though
it is expected to change by a different thread. The while loop may be stuck
waiting for the change, but is always looking at a cached value. If that
happens, timeout will occur and then return 'null'.
In previous versions of Hadoop, there was no time-out at this level, so it
would cause endless loop. Really tough error to track down if it happens.
was:
{code:java|Title=Client.java}
/** @return the rpc response or, in case of timeout, null. */
private Writable getRpcResponse(final Call call, final Connection connection,
final long timeout, final TimeUnit unit) throws IOException {
synchronized (call) {
while (!call.done) {
try {
AsyncGet.Util.wait(call, timeout, unit);
if (timeout >= 0 && !call.done) {
return null;
}
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
throw new InterruptedIOException("Call interrupted");
}
}
*/
static class Call {
final int id; // call id
final int retry; // retry count
...
boolean done; // true when call is done
...
}
{code}
The {{done}} variable is not marked as {{volatile}} so the thread which is
checking its status is free to cache the value and never reload it even though
it is expected to change by a different thread. The while loop may be stuck
waiting for the change, but is always looking at a cached value.
In previous versions of Hadoop, there was no time-out at this level, so it
would cause endless loop. Really tough error to track down if it happens.
> Hadoop Client getRpcResponse May Return Wrong Result
>
>
> Key: HADOOP-17462
> URL: https://issues.apache.org/jira/browse/HADOOP-17462
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
>
> {code:java|Title=Client.java}
> /** @return the rpc response or, in case of timeout, null. */
> private Writable getRpcResponse(final Call call, final Connection
> connection,
> final long timeout, final TimeUnit unit) throws IOException {
> synchronized (call) {
> while (!call.done) {
> try {
> AsyncGet.Util.wait(call, timeout, unit);
> if (timeout >= 0 && !call.done) {
> return null;
> }
> } catch (InterruptedException ie) {
> Thread.currentThread().interrupt();
> throw new InterruptedIOException("Call interrupted");
> }
> }
> */
> static class Call {
> final int id; // call id
> final int retry; // retry count
> ...
> boolean done; // true when call is done
> ...
> }
> {code}
> The {{done}} variable is not marked as {{volatile}} so the thread which is
> checking its status is free to cache the value and never reload it even
> though it is expected to change by a different thread. The while loop may be
> stuck waiting for the change, but is always looking at a cached value. If
> that happens, timeout will occur and then return 'null'.
> In previous versions of Hadoop, there was no time-out at this level, so it
> would cause endless loop. Really tough error to track down if it happens.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17462) Hadoop Client getRpcResponse May Return Wrong Result
David Mollitor created HADOOP-17462:
---
Summary: Hadoop Client getRpcResponse May Return Wrong Result
Key: HADOOP-17462
URL: https://issues.apache.org/jira/browse/HADOOP-17462
Project: Hadoop Common
Issue Type: Improvement
Components: common
Reporter: David Mollitor
Assignee: David Mollitor
{code:java|Title=Client.java}
/** @return the rpc response or, in case of timeout, null. */
private Writable getRpcResponse(final Call call, final Connection connection,
final long timeout, final TimeUnit unit) throws IOException {
synchronized (call) {
while (!call.done) {
try {
AsyncGet.Util.wait(call, timeout, unit);
if (timeout >= 0 && !call.done) {
return null;
}
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
throw new InterruptedIOException("Call interrupted");
}
}
*/
static class Call {
final int id; // call id
final int retry; // retry count
...
boolean done; // true when call is done
...
}
{code}
The {{done}} variable is not marked as {{volatile}} so the thread which is
checking its status is free to cache the value and never reload it even though
it is expected to change by a different thread. The while loop may be stuck
waiting for the change, but is always looking at a cached value.
In previous versions of Hadoop, there was no time-out at this level, so it
would cause endless loop. Really tough error to track down if it happens.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17451) intermittent failure of S3A tests which make assertions on statistics/IOStatistics
[ https://issues.apache.org/jira/browse/HADOOP-17451?focusedWorklogId=533158=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533158 ] ASF GitHub Bot logged work on HADOOP-17451: --- Author: ASF GitHub Bot Created on: 08/Jan/21 18:18 Start Date: 08/Jan/21 18:18 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2594: URL: https://github.com/apache/hadoop/pull/2594#issuecomment-756918476 -checkstyle of unused import; fixing. Ready for review: @bgaborg @mukund-thakur @mehakmeet @sunchao This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 533158) Time Spent: 1h 10m (was: 1h) > intermittent failure of S3A tests which make assertions on > statistics/IOStatistics > -- > > Key: HADOOP-17451 > URL: https://issues.apache.org/jira/browse/HADOOP-17451 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > Intermittent failure of ITestHuge* upload tests, when doing parallel test > runs. > The count of bytes uploaded through StorageStatistics isn't updated. Maybe > the expected counter isn't updated, and somehow in a parallel run with > recycled FS instances/set up directory structure this surfaces the way it > doesn't in a single test run. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2594: HADOOP-17451. IOStatistics test failures in S3A code.
steveloughran commented on pull request #2594: URL: https://github.com/apache/hadoop/pull/2594#issuecomment-756918476 -checkstyle of unused import; fixing. Ready for review: @bgaborg @mukund-thakur @mehakmeet @sunchao This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17201) Spark job with s3acommitter stuck at the last stage
[
https://issues.apache.org/jira/browse/HADOOP-17201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261496#comment-17261496
]
Brandon commented on HADOOP-17201:
--
Adding another observation of this issue. My setup is Spark 2.4.4 with Hadoop
3.2.1. Have tried both the staging committer and magic committer and have seen
this with both. It's a Spark job that writes several large tables to Parquet in
S3. Normally it takes ~2 hours, but nondeterministically it will hang
indefinitely and never finish. When it's hung, the executors have a small
handful of tasks that are stuck in a sleep. The tasks have thread dumps like
these. This example was using the magic committer.
{noformat}
"Executor task launch worker for task 1270082" #897 daemon prio=5 os_prio=0
tid=0x7f6ba4013000 nid=0x5a42 waiting on condition [0x7f6b345ef000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:349)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:285)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.deleteObjects(S3AFileSystem.java:1457)
at org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:1717)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.deleteUnnecessaryFakeDirectories(S3AFileSystem.java:2785)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.finishedWrite(S3AFileSystem.java:2751)
at org.apache.hadoop.fs.s3a.S3AFileSystem.executePut(S3AFileSystem.java:2484)
at
org.apache.hadoop.fs.s3a.WriteOperationHelper.lambda$uploadObject$6(WriteOperationHelper.java:445)
at
org.apache.hadoop.fs.s3a.WriteOperationHelper$$Lambda$1357/1424336210.execute(Unknown
Source)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:265)
at org.apache.hadoop.fs.s3a.Invoker$$Lambda$378/1750315651.execute(Unknown
Source)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:261)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:236)
at
org.apache.hadoop.fs.s3a.WriteOperationHelper.retry(WriteOperationHelper.java:123)
at
org.apache.hadoop.fs.s3a.WriteOperationHelper.uploadObject(WriteOperationHelper.java:443)
at
org.apache.hadoop.fs.s3a.commit.magic.MagicCommitTracker.aboutToComplete(MagicCommitTracker.java:139)
at
org.apache.hadoop.fs.s3a.S3ABlockOutputStream.close(S3ABlockOutputStream.java:382)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
at
org.apache.parquet.hadoop.util.HadoopPositionOutputStream.close(HadoopPositionOutputStream.java:64)
at org.apache.parquet.hadoop.ParquetFileWriter.end(ParquetFileWriter.java:685)
at
org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:122)
at
org.apache.parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:165)
at
org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.close(ParquetOutputWriter.scala:42)
at
org.apache.spark.sql.execution.datasources.FileFormatDataWriter.releaseResources(FileFormatDataWriter.scala:57)
at
org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:74)
{noformat}
> Spark job with s3acommitter stuck at the last stage
> ---
>
> Key: HADOOP-17201
> URL: https://issues.apache.org/jira/browse/HADOOP-17201
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.2.1
> Environment: we are on spark 2.4.5/hadoop 3.2.1 with s3a committer.
> spark.hadoop.fs.s3a.committer.magic.enabled: 'true'
> spark.hadoop.fs.s3a.committer.name: magic
>Reporter: Dyno
>Priority: Major
> Labels: pull-request-available
> Attachments: exec-120.log, exec-125.log, exec-25.log, exec-31.log,
> exec-36.log, exec-44.log, exec-5.log, exec-64.log, exec-7.log
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> usually our spark job took 1 hour or 2 to finish, occasionally it runs for
> more than 3 hour and then we know it's stuck and usually the executor has
> stack like this
> {{
> "Executor task launch worker for task 78620" #265 daemon prio=5 os_prio=0
> tid=0x7f73e0005000 nid=0x12d waiting on condition [0x7f74cb291000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:349)
> at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:285)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.deleteObjects(S3AFileSystem.java:1457)
> at
>
[jira] [Created] (HADOOP-17461) Add thread-level IOStatistics Context
Steve Loughran created HADOOP-17461: --- Summary: Add thread-level IOStatistics Context Key: HADOOP-17461 URL: https://issues.apache.org/jira/browse/HADOOP-17461 Project: Hadoop Common Issue Type: Sub-task Components: fs, fs/azure, fs/s3 Affects Versions: 3.3.1 Reporter: Steve Loughran For effective reporting of the iostatistics of individual worker threads, we need a thread-level context which IO components update. * this contact needs to be passed in two background thread forming work on behalf of a task. * IO Components (streams, iterators, filesystems) need to update this context statistics as they perform work * Without double counting anything. I imagine a ThreadLocal IOStatisticContext which will be updated in the FileSystem API Calls. This context MUST be passed into the background threads used by a task, so that IO is correctly aggregated. I don't want streams, listIterators to do the updating as there is more risk of double counting. However, we need to see their statistics if we want to know things like "bytes discarded in backwards seeks". And I don't want to be updating a shared context object on every read() call. If all we want is store IO (HEAD, GET, DELETE, list performance etc) then the FS is sufficient. If we do want the stream-specific detail, then I propose * caching the context in the constructor * updating it only in close() or unbuffer() (as we do from S3AInputStream to S3AInstrumenation) * excluding those we know the FS already collects. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] touchida commented on a change in pull request #2598: HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStre…
touchida commented on a change in pull request #2598:
URL: https://github.com/apache/hadoop/pull/2598#discussion_r554059473
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMultipleNNPortQOP.java
##
@@ -251,62 +252,48 @@ public void testMultipleNNPortOverwriteDownStream()
throws Exception {
clientConf.set(HADOOP_RPC_PROTECTION, "privacy");
FileSystem fsPrivacy = FileSystem.get(uriPrivacyPort, clientConf);
doTest(fsPrivacy, PATH1);
- for (int i = 0; i < 2; i++) {
-DataNode dn = dataNodes.get(i);
-SaslDataTransferClient saslClient = dn.getSaslClient();
-String qop = null;
-// It may take some time for the qop to populate
-// to all DNs, check in a loop.
-for (int trial = 0; trial < 10; trial++) {
- qop = saslClient.getTargetQOP();
- if (qop != null) {
-break;
- }
- Thread.sleep(100);
-}
-assertEquals("auth", qop);
- }
+ long count = dataNodes.stream()
+ .map(dn -> dn.getSaslClient().getTargetQOP())
+ .filter(Objects::nonNull).filter(qop -> qop.equals("auth")).count();
+ // For each data pipeline, targetQOPs of sasl clients in the first two
+ // datanodes become equal to auth.
+ // Note that it is not necessarily the case for all datanodes,
+ // since a datanode may be always at the last position in pipelines.
+ assertTrue("At least two qops should be auth", count >= 2);
clientConf.set(HADOOP_RPC_PROTECTION, "integrity");
FileSystem fsIntegrity = FileSystem.get(uriIntegrityPort, clientConf);
+ // Reset targetQOPs to null to clear the last state.
+ resetTargetQOP(dataNodes);
doTest(fsIntegrity, PATH2);
- for (int i = 0; i < 2; i++) {
-DataNode dn = dataNodes.get(i);
-SaslDataTransferClient saslClient = dn.getSaslClient();
-String qop = null;
-for (int trial = 0; trial < 10; trial++) {
- qop = saslClient.getTargetQOP();
- if (qop != null) {
-break;
- }
- Thread.sleep(100);
-}
-assertEquals("auth", qop);
- }
+ count = dataNodes.stream().map(dn -> dn.getSaslClient().getTargetQOP())
+ .filter(Objects::nonNull).filter(qop -> qop.equals("auth")).count();
+ assertTrue("At least two qops should be auth", count >= 2);
clientConf.set(HADOOP_RPC_PROTECTION, "authentication");
FileSystem fsAuth = FileSystem.get(uriAuthPort, clientConf);
+ // Reset negotiatedQOPs to null to clear the last state.
+ resetNegotiatedQOP(dataNodes);
doTest(fsAuth, PATH3);
- for (int i = 0; i < 3; i++) {
-DataNode dn = dataNodes.get(i);
-SaslDataTransferServer saslServer = dn.getSaslServer();
-String qop = null;
-for (int trial = 0; trial < 10; trial++) {
- qop = saslServer.getNegotiatedQOP();
- if (qop != null) {
-break;
- }
- Thread.sleep(100);
-}
-assertEquals("auth", qop);
- }
+ count = dataNodes.stream()
+ .map(dn -> dn.getSaslServer().getNegotiatedQOP())
+ .filter(Objects::nonNull).filter(qop -> qop.equals("auth")).count();
+ assertEquals("All qops should be auth", 3, count);
} finally {
if (cluster != null) {
cluster.shutdown();
}
}
}
+ private static void resetTargetQOP(List dns) {
Review comment:
The reason why the reset steps get added is that targetQOP and
negotiatedQOP are already being populated to auth by previous
`doTest(fsPrivacy, PATH1)`.
Without the steps, therefore, the second and third assertions for qop will
become true, regardless of whether the overwriting by `doTest(fsIntegrity,
PATH2)` and `doTest(fsAuth, PATH3)` works or not.
Not-null check in the original code is just to confirm if a qop has been
updated, but I think it's not necessary as answered in
https://github.com/apache/hadoop/pull/2598/files#r554038392.
> Is there a better way to be more specific for each step?
Sorry, but I do not understand your question well :bow:
If you're saying `count >= 2` is ambiguous, then I agree.
But unfortunately, I don't know how to get the order of a datanode pipeline
for now.
If it's possible, we can write an assertion for qops of the first two
datanodes in the pipeline.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail:
[jira] [Updated] (HADOOP-13551) Collect AwsSdkMetrics in S3A FileSystem IOStatistics
[ https://issues.apache.org/jira/browse/HADOOP-13551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13551: Summary: Collect AwsSdkMetrics in S3A FileSystem IOStatistics (was: hook up AwsSdkMetrics to IOStatistics) > Collect AwsSdkMetrics in S3A FileSystem IOStatistics > > > Key: HADOOP-13551 > URL: https://issues.apache.org/jira/browse/HADOOP-13551 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > The S3A Connector has the ability to pass statistics collected by the AWS SDK > into the IOStatistics store of the (stream, FS) > But > * wiring up doesn't (yet) work > * its best if there was thread-context level collection, though FS-level > statistics would be a good start. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13551) hook up AwsSdkMetrics to IOStatistics
[
https://issues.apache.org/jira/browse/HADOOP-13551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13551:
Description:
The S3A Connector has the ability to pass statistics collected by the AWS SDK
into the IOStatistics store of the (stream, FS)
But
* wiring up doesn't (yet) work
* its best if there was thread-context level collection, though FS-level
statistics would be a good start.
was:There's an API in {{com.amazonaws.metrics.AwsSdkMetrics}} to give access
to the internal metrics of the AWS libraries. We might want to get at those
> hook up AwsSdkMetrics to IOStatistics
> -
>
> Key: HADOOP-13551
> URL: https://issues.apache.org/jira/browse/HADOOP-13551
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> The S3A Connector has the ability to pass statistics collected by the AWS SDK
> into the IOStatistics store of the (stream, FS)
> But
> * wiring up doesn't (yet) work
> * its best if there was thread-context level collection, though FS-level
> statistics would be a good start.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16524) Automatic keystore reloading for HttpServer2
[ https://issues.apache.org/jira/browse/HADOOP-16524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261467#comment-17261467 ] Michael Stack commented on HADOOP-16524: Merged to trunk. Put up #2609 backport to branch-3.3. > Automatic keystore reloading for HttpServer2 > > > Key: HADOOP-16524 > URL: https://issues.apache.org/jira/browse/HADOOP-16524 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-16524.patch > > Time Spent: 4h 20m > Remaining Estimate: 0h > > Jetty 9 simplified reloading of keystore. This allows hadoop daemon's SSL > cert to be updated in place without having to restart the service. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13551) hook up AwsSdkMetrics to IOStatistics
[
https://issues.apache.org/jira/browse/HADOOP-13551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13551:
Summary: hook up AwsSdkMetrics to IOStatistics (was: hook up AwsSdkMetrics
to hadoop metrics)
> hook up AwsSdkMetrics to IOStatistics
> -
>
> Key: HADOOP-13551
> URL: https://issues.apache.org/jira/browse/HADOOP-13551
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> There's an API in {{com.amazonaws.metrics.AwsSdkMetrics}} to give access to
> the internal metrics of the AWS libraries. We might want to get at those
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16524) Automatic keystore reloading for HttpServer2
[ https://issues.apache.org/jira/browse/HADOOP-16524?focusedWorklogId=533137=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533137 ] ASF GitHub Bot logged work on HADOOP-16524: --- Author: ASF GitHub Bot Created on: 08/Jan/21 17:21 Start Date: 08/Jan/21 17:21 Worklog Time Spent: 10m Work Description: saintstack opened a new pull request #2609: URL: https://github.com/apache/hadoop/pull/2609 …#2470) Co-authored-by: Borislav Iordanov Signed-off-by: stack ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 533137) Time Spent: 4h 20m (was: 4h 10m) > Automatic keystore reloading for HttpServer2 > > > Key: HADOOP-16524 > URL: https://issues.apache.org/jira/browse/HADOOP-16524 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-16524.patch > > Time Spent: 4h 20m > Remaining Estimate: 0h > > Jetty 9 simplified reloading of keystore. This allows hadoop daemon's SSL > cert to be updated in place without having to restart the service. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] saintstack opened a new pull request #2609: HADOOP-16524. Reloading SSL keystore for both DataNode and NameNode (…
saintstack opened a new pull request #2609: URL: https://github.com/apache/hadoop/pull/2609 …#2470) Co-authored-by: Borislav Iordanov Signed-off-by: stack ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-17455) [s3a] Intermittent failure of ITestS3ADeleteCost.testDeleteSingleFileInDir
[
https://issues.apache.org/jira/browse/HADOOP-17455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HADOOP-17455 started by Steve Loughran.
---
> [s3a] Intermittent failure of ITestS3ADeleteCost.testDeleteSingleFileInDir
> --
>
> Key: HADOOP-17455
> URL: https://issues.apache.org/jira/browse/HADOOP-17455
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Steve Loughran
>Priority: Major
>
> Test failed against ireland intermittently with the following config:
> {{mvn clean verify -Dparallel-tests -DtestsThreadCount=8}}
> xml based config in auth-keys.xml:
> {code:xml}
>
> fs.s3a.metadatastore.impl
> org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-17456) S3A ITestPartialRenamesDeletes.testPartialDirDelete[bulk-delete=true] failure
[
https://issues.apache.org/jira/browse/HADOOP-17456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HADOOP-17456 started by Steve Loughran.
---
> S3A ITestPartialRenamesDeletes.testPartialDirDelete[bulk-delete=true] failure
> -
>
> Key: HADOOP-17456
> URL: https://issues.apache.org/jira/browse/HADOOP-17456
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
>
> Failure in {{ITestPartialRenamesDeletes.testPartialDirDelete}}; wrong #of
> delete requests.
> build options: -Dparallel-tests -DtestsThreadCount=6 -Dscale -Dmarkers=delete
> -Ds3guard -Ddynamo
> The assert fails on a line changes in HADOOP-17271; assumption being, there
> are some test run states where things happen differently.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-17451) intermittent failure of S3A tests which make assertions on statistics/IOStatistics
[ https://issues.apache.org/jira/browse/HADOOP-17451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-17451 started by Steve Loughran. --- > intermittent failure of S3A tests which make assertions on > statistics/IOStatistics > -- > > Key: HADOOP-17451 > URL: https://issues.apache.org/jira/browse/HADOOP-17451 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Intermittent failure of ITestHuge* upload tests, when doing parallel test > runs. > The count of bytes uploaded through StorageStatistics isn't updated. Maybe > the expected counter isn't updated, and somehow in a parallel run with > recycled FS instances/set up directory structure this surfaces the way it > doesn't in a single test run. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17408) Optimize NetworkTopology while sorting of block locations
[
https://issues.apache.org/jira/browse/HADOOP-17408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261440#comment-17261440
]
Jim Brennan commented on HADOOP-17408:
--
Thanks for your work on this [~ahussein]. I have approved the PR and I will
commit later today.
> Optimize NetworkTopology while sorting of block locations
> -
>
> Key: HADOOP-17408
> URL: https://issues.apache.org/jira/browse/HADOOP-17408
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, net
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> In {{NetworkTopology}}, I noticed that there are some hanging fruits to
> improve the performance.
> Inside {{sortByDistance}}, collections.shuffle is performed on the list
> before calling {{secondarySort}}.
> {code:java}
> Collections.shuffle(list, r);
> if (secondarySort != null) {
> secondarySort.accept(list);
> }
> {code}
> However, in different call sites, {{collections.shuffle}} is passed as the
> secondarySort to {{sortByDistance}}. This means that the shuffle is executed
> twice on each list.
> Also, logic wise, it is useless to shuffle before applying a tie breaker
> which might make the shuffle work obsolete.
> In addition, [~daryn] reported that:
> * topology is unnecessarily locking/unlocking to calculate the distance for
> every node
> * shuffling uses a seeded Random, instead of ThreadLocalRandom, which is
> heavily synchronized
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16524) Automatic keystore reloading for HttpServer2
[ https://issues.apache.org/jira/browse/HADOOP-16524?focusedWorklogId=533129=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533129 ] ASF GitHub Bot logged work on HADOOP-16524: --- Author: ASF GitHub Bot Created on: 08/Jan/21 17:10 Start Date: 08/Jan/21 17:10 Worklog Time Spent: 10m Work Description: saintstack merged pull request #2470: URL: https://github.com/apache/hadoop/pull/2470 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 533129) Time Spent: 4h 10m (was: 4h) > Automatic keystore reloading for HttpServer2 > > > Key: HADOOP-16524 > URL: https://issues.apache.org/jira/browse/HADOOP-16524 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-16524.patch > > Time Spent: 4h 10m > Remaining Estimate: 0h > > Jetty 9 simplified reloading of keystore. This allows hadoop daemon's SSL > cert to be updated in place without having to restart the service. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] saintstack merged pull request #2470: HADOOP-16524. Reloading SSL keystore for both DataNode and NameNode
saintstack merged pull request #2470: URL: https://github.com/apache/hadoop/pull/2470 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14391) s3a: auto-detect region for bucket and use right endpoint
[
https://issues.apache.org/jira/browse/HADOOP-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-14391.
-
Resolution: Cannot Reproduce
Latest AWS SDK works this out if you don't set the endpoint.
it adds the overhead of an extra HEAD / call -that is issued with v4 signature,
if that is rejected from s3 central, the 400 error code includes the actual
region of the bucket.
> s3a: auto-detect region for bucket and use right endpoint
> -
>
> Key: HADOOP-14391
> URL: https://issues.apache.org/jira/browse/HADOOP-14391
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-alpha2
>Reporter: Aaron Fabbri
>Assignee: Aaron Fabbri
>Priority: Major
>
> Specifying the S3A endpoint ({{fs.s3a.endpoint}}) is
> - *required* for regions which only support v4 authentication
> - A good practice for all regions.
> The user experience of having to configure endpoints is not great. Often it
> is neglected and leads to additional cost, reduced performance, or failures
> for v4 auth regions.
> I want to explore an option which, when enabled, auto-detects the region for
> an s3 bucket and uses the proper endpoint. Not sure if this is possible or
> anyone has looked into it yet.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15348) S3A Input Stream bytes read counter isn't getting through to StorageStatistics/instrumentation properly
[
https://issues.apache.org/jira/browse/HADOOP-15348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-15348.
-
Fix Version/s: 3.4.0
Resolution: Fixed
Fixed as part of HADOOP-17271
> S3A Input Stream bytes read counter isn't getting through to
> StorageStatistics/instrumentation properly
> ---
>
> Key: HADOOP-15348
> URL: https://issues.apache.org/jira/browse/HADOOP-15348
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Steve Loughran
>Priority: Minor
> Fix For: 3.4.0
>
>
> TL;DR: we should have common storage statistics for bytes read and bytes
> written, and S3A should use them in its instrumentation and have enum names
> to match.
> # in the S3AInputStream we call
> {{S3AInstrumentation.StreamStatistics.bytesRead(long)}}, which adds the
> amount to {{bytesRead}}, in a read(), readFully, or forward seek() reading in
> data
> # and in {{S3AInstrumentation.mergeInputStreamStatistics}}, that is pulled
> into streamBytesRead.
> # which has a Statistics name of ""stream_bytes_read"
> # but that is served up in the Storage statistics as
> "STREAM_SEEK_BYTES_READ", which is the wrong name.
> # and there isn't a common name for the counter across other filesystems.
> For now: people can use the wrong name in the enum; we may want to think
> about retaining it when adding the correct name. And maybe add a
> @Evolving/@LimitedPrivate scope pair to the enum
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] touchida commented on a change in pull request #2598: HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStre…
touchida commented on a change in pull request #2598:
URL: https://github.com/apache/hadoop/pull/2598#discussion_r554073638
##
File path:
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferClient.java
##
@@ -405,6 +405,11 @@ public String getTargetQOP() {
return targetQOP;
}
+ @VisibleForTesting
+ public void setTargetQOP(String targetQOP) {
Review comment:
I totally agree with you.
Let me find a better way.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17451) intermittent failure of S3A tests which make assertions on statistics/IOStatistics
[
https://issues.apache.org/jira/browse/HADOOP-17451?focusedWorklogId=533122=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533122
]
ASF GitHub Bot logged work on HADOOP-17451:
---
Author: ASF GitHub Bot
Created on: 08/Jan/21 16:50
Start Date: 08/Jan/21 16:50
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2594:
URL: https://github.com/apache/hadoop/pull/2594#issuecomment-756866385
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 40s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 45s | | trunk passed |
| +1 :green_heart: | compile | 20m 18s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 17m 21s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 42s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 17s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 43s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 2m 22s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 1m 18s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 34s | | trunk passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 29s | | the patch passed |
| +1 :green_heart: | compile | 19m 19s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 19m 19s | | the patch passed |
| +1 :green_heart: | compile | 17m 20s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 17m 20s | | the patch passed |
| -0 :warning: | checkstyle | 2m 36s |
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/3/artifact/out/diff-checkstyle-root.txt)
| root: The patch generated 1 new + 11 unchanged - 0 fixed = 12 total (was
11) |
| +1 :green_heart: | mvnsite | 2m 26s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 50s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 41s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 2m 23s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 3m 47s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 9m 50s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 1m 42s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not
generate ASF License warnings. |
| | | 186m 40s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2594 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 28bb3e6865f0 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 41767599093 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
[GitHub] [hadoop] hadoop-yetus commented on pull request #2594: HADOOP-17451. IOStatistics test failures in S3A code.
hadoop-yetus commented on pull request #2594:
URL: https://github.com/apache/hadoop/pull/2594#issuecomment-756866385
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 40s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 45s | | trunk passed |
| +1 :green_heart: | compile | 20m 18s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 17m 21s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 42s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 17s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 43s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 2m 22s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 1m 18s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 34s | | trunk passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 29s | | the patch passed |
| +1 :green_heart: | compile | 19m 19s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 19m 19s | | the patch passed |
| +1 :green_heart: | compile | 17m 20s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 17m 20s | | the patch passed |
| -0 :warning: | checkstyle | 2m 36s |
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/3/artifact/out/diff-checkstyle-root.txt)
| root: The patch generated 1 new + 11 unchanged - 0 fixed = 12 total (was
11) |
| +1 :green_heart: | mvnsite | 2m 26s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 50s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 41s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 2m 23s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 3m 47s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 9m 50s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 1m 42s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not
generate ASF License warnings. |
| | | 186m 40s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2594 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 28bb3e6865f0 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 41767599093 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/3/testReport/ |
| Max. process+thread count | 2247 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws
U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/3/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT
[GitHub] [hadoop] touchida commented on a change in pull request #2598: HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStre…
touchida commented on a change in pull request #2598:
URL: https://github.com/apache/hadoop/pull/2598#discussion_r554059473
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMultipleNNPortQOP.java
##
@@ -251,62 +252,48 @@ public void testMultipleNNPortOverwriteDownStream()
throws Exception {
clientConf.set(HADOOP_RPC_PROTECTION, "privacy");
FileSystem fsPrivacy = FileSystem.get(uriPrivacyPort, clientConf);
doTest(fsPrivacy, PATH1);
- for (int i = 0; i < 2; i++) {
-DataNode dn = dataNodes.get(i);
-SaslDataTransferClient saslClient = dn.getSaslClient();
-String qop = null;
-// It may take some time for the qop to populate
-// to all DNs, check in a loop.
-for (int trial = 0; trial < 10; trial++) {
- qop = saslClient.getTargetQOP();
- if (qop != null) {
-break;
- }
- Thread.sleep(100);
-}
-assertEquals("auth", qop);
- }
+ long count = dataNodes.stream()
+ .map(dn -> dn.getSaslClient().getTargetQOP())
+ .filter(Objects::nonNull).filter(qop -> qop.equals("auth")).count();
+ // For each data pipeline, targetQOPs of sasl clients in the first two
+ // datanodes become equal to auth.
+ // Note that it is not necessarily the case for all datanodes,
+ // since a datanode may be always at the last position in pipelines.
+ assertTrue("At least two qops should be auth", count >= 2);
clientConf.set(HADOOP_RPC_PROTECTION, "integrity");
FileSystem fsIntegrity = FileSystem.get(uriIntegrityPort, clientConf);
+ // Reset targetQOPs to null to clear the last state.
+ resetTargetQOP(dataNodes);
doTest(fsIntegrity, PATH2);
- for (int i = 0; i < 2; i++) {
-DataNode dn = dataNodes.get(i);
-SaslDataTransferClient saslClient = dn.getSaslClient();
-String qop = null;
-for (int trial = 0; trial < 10; trial++) {
- qop = saslClient.getTargetQOP();
- if (qop != null) {
-break;
- }
- Thread.sleep(100);
-}
-assertEquals("auth", qop);
- }
+ count = dataNodes.stream().map(dn -> dn.getSaslClient().getTargetQOP())
+ .filter(Objects::nonNull).filter(qop -> qop.equals("auth")).count();
+ assertTrue("At least two qops should be auth", count >= 2);
clientConf.set(HADOOP_RPC_PROTECTION, "authentication");
FileSystem fsAuth = FileSystem.get(uriAuthPort, clientConf);
+ // Reset negotiatedQOPs to null to clear the last state.
+ resetNegotiatedQOP(dataNodes);
doTest(fsAuth, PATH3);
- for (int i = 0; i < 3; i++) {
-DataNode dn = dataNodes.get(i);
-SaslDataTransferServer saslServer = dn.getSaslServer();
-String qop = null;
-for (int trial = 0; trial < 10; trial++) {
- qop = saslServer.getNegotiatedQOP();
- if (qop != null) {
-break;
- }
- Thread.sleep(100);
-}
-assertEquals("auth", qop);
- }
+ count = dataNodes.stream()
+ .map(dn -> dn.getSaslServer().getNegotiatedQOP())
+ .filter(Objects::nonNull).filter(qop -> qop.equals("auth")).count();
+ assertEquals("All qops should be auth", 3, count);
} finally {
if (cluster != null) {
cluster.shutdown();
}
}
}
+ private static void resetTargetQOP(List dns) {
Review comment:
The reason why the reset steps get added is that targetQOP and
negotiatedQOP are already being populated to auth by previous
`doTest(fsPrivacy, PATH1)`.
Without the step, therefore, the second and third assertions for qop will
become true, regardless of whether the overwriting by `doTest(fsIntegrity,
PATH2)` and `doTest(fsAuth, PATH3)` works or not.
Not-null check in the original code is just to confirm if a qop has been
updated, but I think it's not necessary as answered in
https://github.com/apache/hadoop/pull/2598/files#r554038392.
> Is there a better way to be more specific for each step?
Sorry, but I do not understand your question well :bow:
If you're saying `count >= 2` is ambiguous, then I agree.
But unfortunately, I don't know how to get the order of a datanode pipeline
for now.
If it's possible, we can write an assertion for qops of the first two
datanodes in the pipeline.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail:
[jira] [Updated] (HADOOP-16973) Fix typo in ReconfigurationServlet
[ https://issues.apache.org/jira/browse/HADOOP-16973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HADOOP-16973: - Fix Version/s: (was: 3.2.2) > Fix typo in ReconfigurationServlet > -- > > Key: HADOOP-16973 > URL: https://issues.apache.org/jira/browse/HADOOP-16973 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.2.1 >Reporter: bianqi >Priority: Trivial > Attachments: HADOOP-16973-001.patch > > > Fix typo in ReconfigurationServlet -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16976) Fix code style in JMXJsonServlet
[ https://issues.apache.org/jira/browse/HADOOP-16976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HADOOP-16976: - Fix Version/s: (was: 3.2.2) > Fix code style in JMXJsonServlet > > > Key: HADOOP-16976 > URL: https://issues.apache.org/jira/browse/HADOOP-16976 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.2.1 >Reporter: bianqi >Priority: Trivial > Attachments: HADOOP-16976-001.patch > > > Fix code style in JMXJsonServlet -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16975) Fix typo in JMXJsonServlet.java
[ https://issues.apache.org/jira/browse/HADOOP-16975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HADOOP-16975: - Fix Version/s: (was: 3.2.2) > Fix typo in JMXJsonServlet.java > --- > > Key: HADOOP-16975 > URL: https://issues.apache.org/jira/browse/HADOOP-16975 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.2.1 >Reporter: bianqi >Priority: Minor > Attachments: HADOOP-16975-001.patch > > > Fix typo in JMXJsonServlet.java -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16974) Add error Message when thread join throw exception
[ https://issues.apache.org/jira/browse/HADOOP-16974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HADOOP-16974: - Fix Version/s: (was: 3.2.2) > Add error Message when thread join throw exception > -- > > Key: HADOOP-16974 > URL: https://issues.apache.org/jira/browse/HADOOP-16974 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.2.1 >Reporter: bianqi >Priority: Minor > Attachments: HADOOP-16974-001.patch > > > Add error Message when thread join throw exception. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16978) Fix typo getLogSupressionMessage to getLogSuppressionMessage
[ https://issues.apache.org/jira/browse/HADOOP-16978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HADOOP-16978: - Fix Version/s: (was: 3.2.2) > Fix typo getLogSupressionMessage to getLogSuppressionMessage > > > Key: HADOOP-16978 > URL: https://issues.apache.org/jira/browse/HADOOP-16978 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.2.1 >Reporter: bianqi >Priority: Minor > Attachments: HADOOP-16978-001.patch > > > Fix typo getLogSupressionMessage to getLogSuppressionMessage -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16987) update-method-param for ProtocolSignature
[ https://issues.apache.org/jira/browse/HADOOP-16987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HADOOP-16987: - Fix Version/s: (was: 3.2.2) > update-method-param for ProtocolSignature > - > > Key: HADOOP-16987 > URL: https://issues.apache.org/jira/browse/HADOOP-16987 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Affects Versions: 3.2.1 >Reporter: bianqi >Priority: Major > Attachments: HADOOP-16987-001.patch > > > update-method-param for ProtocolSignature -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] touchida commented on a change in pull request #2598: HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStre…
touchida commented on a change in pull request #2598:
URL: https://github.com/apache/hadoop/pull/2598#discussion_r554038392
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMultipleNNPortQOP.java
##
@@ -251,62 +252,48 @@ public void testMultipleNNPortOverwriteDownStream()
throws Exception {
clientConf.set(HADOOP_RPC_PROTECTION, "privacy");
FileSystem fsPrivacy = FileSystem.get(uriPrivacyPort, clientConf);
doTest(fsPrivacy, PATH1);
- for (int i = 0; i < 2; i++) {
-DataNode dn = dataNodes.get(i);
-SaslDataTransferClient saslClient = dn.getSaslClient();
-String qop = null;
-// It may take some time for the qop to populate
-// to all DNs, check in a loop.
-for (int trial = 0; trial < 10; trial++) {
- qop = saslClient.getTargetQOP();
- if (qop != null) {
-break;
- }
- Thread.sleep(100);
-}
-assertEquals("auth", qop);
- }
+ long count = dataNodes.stream()
+ .map(dn -> dn.getSaslClient().getTargetQOP())
+ .filter(Objects::nonNull).filter(qop -> qop.equals("auth")).count();
+ // For each data pipeline, targetQOPs of sasl clients in the first two
+ // datanodes become equal to auth.
+ // Note that it is not necessarily the case for all datanodes,
+ // since a datanode may be always at the last position in pipelines.
+ assertTrue("At least two qops should be auth", count >= 2);
Review comment:
@amahussein Thanks for your review!
Yes, I did.
But I think we don't have to retry it here, since the test already makes
sure that blocks have been replicated.
https://github.com/apache/hadoop/blob/443a77fbe8dc29e4b36cac7b935405d70bc79158/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMultipleNNPortQOP.java#L305-L307
IIUC, qops are populated at sasl negotiation, which should be done before
datanodes send packets (i.e., before replication is completed).
https://github.com/apache/hadoop/blob/443a77fbe8dc29e4b36cac7b935405d70bc79158/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferClient.java#L414
It seems that the author of the description considered that the flakiness of
this test was caused by insufficient sleep time, but I believe it's not.
https://issues.apache.org/jira/browse/HDFS-15148?focusedCommentId=17026282=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17026282
I also considered to wait for replication inside
TestMultipleNNPortQOP#doTest, since FileSystemTestHelper.createFile's success
will not necessarily imply replication is completed.
https://github.com/apache/hadoop/blob/443a77fbe8dc29e4b36cac7b935405d70bc79158/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMultipleNNPortQOP.java#L301
But I'm not sure if we really need this.
May I ask your opinion?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17408) Optimize NetworkTopology while sorting of block locations
[
https://issues.apache.org/jira/browse/HADOOP-17408?focusedWorklogId=533098=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533098
]
ASF GitHub Bot logged work on HADOOP-17408:
---
Author: ASF GitHub Bot
Created on: 08/Jan/21 16:07
Start Date: 08/Jan/21 16:07
Worklog Time Spent: 10m
Work Description: amahussein commented on pull request #2601:
URL: https://github.com/apache/hadoop/pull/2601#issuecomment-756839310
@jbrennan333 The failing unit test is not related. I will take note and
watch for it to file a jira if it shows up in the qbt-report.
I have intuition that the equal-weighted nodes should be shuffled to
maintain load balancing. In order to avoid the redundancy of shuffling twice, I
changed the default `secondSorter` to null.
I made quick evaluation of the performance using a single threaded. The new
changes show less variance in code performance and slightly lower average.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 533098)
Time Spent: 1h 40m (was: 1.5h)
> Optimize NetworkTopology while sorting of block locations
> -
>
> Key: HADOOP-17408
> URL: https://issues.apache.org/jira/browse/HADOOP-17408
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, net
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> In {{NetworkTopology}}, I noticed that there are some hanging fruits to
> improve the performance.
> Inside {{sortByDistance}}, collections.shuffle is performed on the list
> before calling {{secondarySort}}.
> {code:java}
> Collections.shuffle(list, r);
> if (secondarySort != null) {
> secondarySort.accept(list);
> }
> {code}
> However, in different call sites, {{collections.shuffle}} is passed as the
> secondarySort to {{sortByDistance}}. This means that the shuffle is executed
> twice on each list.
> Also, logic wise, it is useless to shuffle before applying a tie breaker
> which might make the shuffle work obsolete.
> In addition, [~daryn] reported that:
> * topology is unnecessarily locking/unlocking to calculate the distance for
> every node
> * shuffling uses a seeded Random, instead of ThreadLocalRandom, which is
> heavily synchronized
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] amahussein commented on pull request #2601: HADOOP-17408. Optimize NetworkTopology sorting block locations.
amahussein commented on pull request #2601: URL: https://github.com/apache/hadoop/pull/2601#issuecomment-756839310 @jbrennan333 The failing unit test is not related. I will take note and watch for it to file a jira if it shows up in the qbt-report. I have intuition that the equal-weighted nodes should be shuffled to maintain load balancing. In order to avoid the redundancy of shuffling twice, I changed the default `secondSorter` to null. I made quick evaluation of the performance using a single threaded. The new changes show less variance in code performance and slightly lower average. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] amahussein commented on a change in pull request #2581: YARN-10553. Refactor TestDistributedShell
amahussein commented on a change in pull request #2581:
URL: https://github.com/apache/hadoop/pull/2581#discussion_r554026595
##
File path:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/DistributedShellBaseTest.java
##
@@ -0,0 +1,557 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.applications.distributedshell;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.URL;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Supplier;
+
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.rules.TemporaryFolder;
+import org.junit.rules.TestName;
+import org.junit.rules.Timeout;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileContext;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.net.ServerSocketUtil;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.util.JarFinder;
+import org.apache.hadoop.util.Shell;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptId;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptReport;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.ContainerReport;
+import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.api.records.timeline.TimelineDomain;
+import org.apache.hadoop.yarn.api.records.timeline.TimelineEntities;
+import org.apache.hadoop.yarn.api.records.timeline.TimelineEntity;
+import org.apache.hadoop.yarn.client.api.YarnClient;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.server.MiniYARNCluster;
+import org.apache.hadoop.yarn.server.nodemanager.NodeManager;
+import org.apache.hadoop.yarn.server.resourcemanager.RMContext;
+import org.apache.hadoop.yarn.server.resourcemanager.ResourceManager;
+import org.apache.hadoop.yarn.server.timeline.NameValuePair;
+import org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin;
+import org.apache.hadoop.yarn.util.ProcfsBasedProcessTree;
+
+/**
+ * Base class for testing DistributedShell features.
+ */
+public abstract class DistributedShellBaseTest {
+ protected static final int MIN_ALLOCATION_MB = 128;
+ protected static final int NUM_DATA_NODES = 1;
+ protected static final int TEST_TIME_OUT = 16;
+ // set the timeout of the yarnClient to be 95% of the globalTimeout.
+ protected static final int TEST_TIME_WINDOW_EXPIRE =
+ (TEST_TIME_OUT * 90) / 100;
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DistributedShellBaseTest.class);
+ private static final String APP_MASTER_JAR =
+ JarFinder.getJar(ApplicationMaster.class);
+ private static final int NUM_NMS = 1;
+ // set the timeout of the yarnClient to be 95% of the globalTimeout.
+ private static final String YARN_CLIENT_TIMEOUT =
+ String.valueOf(TEST_TIME_WINDOW_EXPIRE);
+ private static final String[] COMMON_ARGS = {
+ "--jar",
+ APP_MASTER_JAR,
+ "--timeout",
+ YARN_CLIENT_TIMEOUT,
+ "--appname",
+ ""
+ };
+ @Rule
+ public Timeout globalTimeout = new Timeout(TEST_TIME_OUT,
+ TimeUnit.MILLISECONDS);
+ @Rule
+ public TemporaryFolder tmpFolder = new TemporaryFolder();
+ @Rule
+ public TestName name = new TestName();
+ private Client dsClient;
+ private MiniYARNCluster yarnCluster;
+ private YarnConfiguration conf = null;
+ // location of the filesystem timeline writer for timeline service v.2
+ private String timelineV2StorageDir = null;
+
+
[GitHub] [hadoop] amahussein commented on a change in pull request #2581: YARN-10553. Refactor TestDistributedShell
amahussein commented on a change in pull request #2581:
URL: https://github.com/apache/hadoop/pull/2581#discussion_r554026595
##
File path:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/DistributedShellBaseTest.java
##
@@ -0,0 +1,557 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.applications.distributedshell;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.URL;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Supplier;
+
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.rules.TemporaryFolder;
+import org.junit.rules.TestName;
+import org.junit.rules.Timeout;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileContext;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.net.ServerSocketUtil;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.util.JarFinder;
+import org.apache.hadoop.util.Shell;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptId;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptReport;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.ContainerReport;
+import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.api.records.timeline.TimelineDomain;
+import org.apache.hadoop.yarn.api.records.timeline.TimelineEntities;
+import org.apache.hadoop.yarn.api.records.timeline.TimelineEntity;
+import org.apache.hadoop.yarn.client.api.YarnClient;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.server.MiniYARNCluster;
+import org.apache.hadoop.yarn.server.nodemanager.NodeManager;
+import org.apache.hadoop.yarn.server.resourcemanager.RMContext;
+import org.apache.hadoop.yarn.server.resourcemanager.ResourceManager;
+import org.apache.hadoop.yarn.server.timeline.NameValuePair;
+import org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin;
+import org.apache.hadoop.yarn.util.ProcfsBasedProcessTree;
+
+/**
+ * Base class for testing DistributedShell features.
+ */
+public abstract class DistributedShellBaseTest {
+ protected static final int MIN_ALLOCATION_MB = 128;
+ protected static final int NUM_DATA_NODES = 1;
+ protected static final int TEST_TIME_OUT = 16;
+ // set the timeout of the yarnClient to be 95% of the globalTimeout.
+ protected static final int TEST_TIME_WINDOW_EXPIRE =
+ (TEST_TIME_OUT * 90) / 100;
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DistributedShellBaseTest.class);
+ private static final String APP_MASTER_JAR =
+ JarFinder.getJar(ApplicationMaster.class);
+ private static final int NUM_NMS = 1;
+ // set the timeout of the yarnClient to be 95% of the globalTimeout.
+ private static final String YARN_CLIENT_TIMEOUT =
+ String.valueOf(TEST_TIME_WINDOW_EXPIRE);
+ private static final String[] COMMON_ARGS = {
+ "--jar",
+ APP_MASTER_JAR,
+ "--timeout",
+ YARN_CLIENT_TIMEOUT,
+ "--appname",
+ ""
+ };
+ @Rule
+ public Timeout globalTimeout = new Timeout(TEST_TIME_OUT,
+ TimeUnit.MILLISECONDS);
+ @Rule
+ public TemporaryFolder tmpFolder = new TemporaryFolder();
+ @Rule
+ public TestName name = new TestName();
+ private Client dsClient;
+ private MiniYARNCluster yarnCluster;
+ private YarnConfiguration conf = null;
+ // location of the filesystem timeline writer for timeline service v.2
+ private String timelineV2StorageDir = null;
+
+
[GitHub] [hadoop] amahussein commented on a change in pull request #2581: YARN-10553. Refactor TestDistributedShell
amahussein commented on a change in pull request #2581:
URL: https://github.com/apache/hadoop/pull/2581#discussion_r554026595
##
File path:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/DistributedShellBaseTest.java
##
@@ -0,0 +1,557 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.applications.distributedshell;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.URL;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Supplier;
+
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.rules.TemporaryFolder;
+import org.junit.rules.TestName;
+import org.junit.rules.Timeout;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileContext;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.net.ServerSocketUtil;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.util.JarFinder;
+import org.apache.hadoop.util.Shell;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptId;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptReport;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.ContainerReport;
+import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.api.records.timeline.TimelineDomain;
+import org.apache.hadoop.yarn.api.records.timeline.TimelineEntities;
+import org.apache.hadoop.yarn.api.records.timeline.TimelineEntity;
+import org.apache.hadoop.yarn.client.api.YarnClient;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.server.MiniYARNCluster;
+import org.apache.hadoop.yarn.server.nodemanager.NodeManager;
+import org.apache.hadoop.yarn.server.resourcemanager.RMContext;
+import org.apache.hadoop.yarn.server.resourcemanager.ResourceManager;
+import org.apache.hadoop.yarn.server.timeline.NameValuePair;
+import org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin;
+import org.apache.hadoop.yarn.util.ProcfsBasedProcessTree;
+
+/**
+ * Base class for testing DistributedShell features.
+ */
+public abstract class DistributedShellBaseTest {
+ protected static final int MIN_ALLOCATION_MB = 128;
+ protected static final int NUM_DATA_NODES = 1;
+ protected static final int TEST_TIME_OUT = 16;
+ // set the timeout of the yarnClient to be 95% of the globalTimeout.
+ protected static final int TEST_TIME_WINDOW_EXPIRE =
+ (TEST_TIME_OUT * 90) / 100;
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DistributedShellBaseTest.class);
+ private static final String APP_MASTER_JAR =
+ JarFinder.getJar(ApplicationMaster.class);
+ private static final int NUM_NMS = 1;
+ // set the timeout of the yarnClient to be 95% of the globalTimeout.
+ private static final String YARN_CLIENT_TIMEOUT =
+ String.valueOf(TEST_TIME_WINDOW_EXPIRE);
+ private static final String[] COMMON_ARGS = {
+ "--jar",
+ APP_MASTER_JAR,
+ "--timeout",
+ YARN_CLIENT_TIMEOUT,
+ "--appname",
+ ""
+ };
+ @Rule
+ public Timeout globalTimeout = new Timeout(TEST_TIME_OUT,
+ TimeUnit.MILLISECONDS);
+ @Rule
+ public TemporaryFolder tmpFolder = new TemporaryFolder();
+ @Rule
+ public TestName name = new TestName();
+ private Client dsClient;
+ private MiniYARNCluster yarnCluster;
+ private YarnConfiguration conf = null;
+ // location of the filesystem timeline writer for timeline service v.2
+ private String timelineV2StorageDir = null;
+
+
[GitHub] [hadoop] hadoop-yetus commented on pull request #2608: YARN-10555. missing access check before getAppAttempts
hadoop-yetus commented on pull request #2608:
URL: https://github.com/apache/hadoop/pull/2608#issuecomment-756790297
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 42s | | trunk passed |
| +1 :green_heart: | compile | 1m 1s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 52s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 0s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 50s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 38s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 1m 49s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 47s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 51s | | the patch passed |
| +1 :green_heart: | compile | 0m 51s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 51s | | the patch passed |
| +1 :green_heart: | compile | 0m 45s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 45s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 31s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 47s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 59s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 39s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 35s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 47s | | the patch passed |
_ Other Tests _ |
| -1 :x: | unit | 89m 18s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2608/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 170m 10s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2608/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2608 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux bade78ba7eb7 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2ba7ec2b48b |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2608/1/testReport/ |
| Max. process+thread count | 881 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
[GitHub] [hadoop] hadoop-yetus commented on pull request #2549: Hadoop 17428. ABFS: Implementation for getContentSummary
hadoop-yetus commented on pull request #2549:
URL: https://github.com/apache/hadoop/pull/2549#issuecomment-756772717
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 31s | | trunk passed |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 38s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 1m 0s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 58s | | trunk passed |
| -0 :warning: | patch | 1m 18s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| -0 :warning: | checkstyle | 0m 17s |
[/diff-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2549/13/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 2 new + 2 unchanged - 0
fixed = 4 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 0s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 58s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 21s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 77m 9s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2549/13/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2549 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 53bc5fea06a5 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 41767599093 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2549/13/testReport/ |
| Max. process+thread count | 689 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2549/13/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT
[GitHub] [hadoop] iwasakims commented on a change in pull request #2581: YARN-10553. Refactor TestDistributedShell
iwasakims commented on a change in pull request #2581:
URL: https://github.com/apache/hadoop/pull/2581#discussion_r553959171
##
File path:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/DistributedShellBaseTest.java
##
@@ -0,0 +1,557 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.applications.distributedshell;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.URL;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Supplier;
+
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.rules.TemporaryFolder;
+import org.junit.rules.TestName;
+import org.junit.rules.Timeout;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileContext;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.net.ServerSocketUtil;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.util.JarFinder;
+import org.apache.hadoop.util.Shell;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptId;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptReport;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.ContainerReport;
+import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.api.records.timeline.TimelineDomain;
+import org.apache.hadoop.yarn.api.records.timeline.TimelineEntities;
+import org.apache.hadoop.yarn.api.records.timeline.TimelineEntity;
+import org.apache.hadoop.yarn.client.api.YarnClient;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.server.MiniYARNCluster;
+import org.apache.hadoop.yarn.server.nodemanager.NodeManager;
+import org.apache.hadoop.yarn.server.resourcemanager.RMContext;
+import org.apache.hadoop.yarn.server.resourcemanager.ResourceManager;
+import org.apache.hadoop.yarn.server.timeline.NameValuePair;
+import org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin;
+import org.apache.hadoop.yarn.util.ProcfsBasedProcessTree;
+
+/**
+ * Base class for testing DistributedShell features.
+ */
+public abstract class DistributedShellBaseTest {
+ protected static final int MIN_ALLOCATION_MB = 128;
+ protected static final int NUM_DATA_NODES = 1;
+ protected static final int TEST_TIME_OUT = 16;
+ // set the timeout of the yarnClient to be 95% of the globalTimeout.
+ protected static final int TEST_TIME_WINDOW_EXPIRE =
+ (TEST_TIME_OUT * 90) / 100;
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DistributedShellBaseTest.class);
+ private static final String APP_MASTER_JAR =
+ JarFinder.getJar(ApplicationMaster.class);
+ private static final int NUM_NMS = 1;
+ // set the timeout of the yarnClient to be 95% of the globalTimeout.
+ private static final String YARN_CLIENT_TIMEOUT =
+ String.valueOf(TEST_TIME_WINDOW_EXPIRE);
+ private static final String[] COMMON_ARGS = {
+ "--jar",
+ APP_MASTER_JAR,
+ "--timeout",
+ YARN_CLIENT_TIMEOUT,
+ "--appname",
+ ""
+ };
+ @Rule
+ public Timeout globalTimeout = new Timeout(TEST_TIME_OUT,
+ TimeUnit.MILLISECONDS);
+ @Rule
+ public TemporaryFolder tmpFolder = new TemporaryFolder();
+ @Rule
+ public TestName name = new TestName();
+ private Client dsClient;
+ private MiniYARNCluster yarnCluster;
+ private YarnConfiguration conf = null;
+ // location of the filesystem timeline writer for timeline service v.2
+ private String timelineV2StorageDir = null;
+
+
[GitHub] [hadoop] hadoop-yetus commented on pull request #2583: HDFS-15549. Improve DISK/ARCHIVE movement if they are on same filesystem
hadoop-yetus commented on pull request #2583:
URL: https://github.com/apache/hadoop/pull/2583#issuecomment-756766090
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 2m 11s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 50s | | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 28m 16s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2583/3/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 28m 45s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 22m 45s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 3m 43s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 54s | | trunk passed |
| +1 :green_heart: | shadedclient | 28m 43s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 25s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 3m 40s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 4m 7s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 7m 4s | | trunk passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 33s | | the patch passed |
| +1 :green_heart: | compile | 27m 28s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 27m 28s | | the patch passed |
| +1 :green_heart: | compile | 19m 0s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 19m 0s | | the patch passed |
| -0 :warning: | checkstyle | 2m 52s |
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2583/3/artifact/out/diff-checkstyle-root.txt)
| root: The patch generated 32 new + 240 unchanged - 0 fixed = 272 total (was
240) |
| +1 :green_heart: | mvnsite | 2m 54s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 17m 18s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 58s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 2m 59s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 5m 54s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 9m 54s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 118m 22s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2583/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not
generate ASF License warnings. |
| | | 355m 34s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2583/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2583 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 4637c317d0cf 4.15.0-101-generic #102-Ubuntu SMP Mon May 11
10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 87bd4d2aca5 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
[jira] [Work logged] (HADOOP-16080) hadoop-aws does not work with hadoop-client-api
[
https://issues.apache.org/jira/browse/HADOOP-16080?focusedWorklogId=533023=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533023
]
ASF GitHub Bot logged work on HADOOP-16080:
---
Author: ASF GitHub Bot
Created on: 08/Jan/21 13:45
Start Date: 08/Jan/21 13:45
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2575:
URL: https://github.com/apache/hadoop/pull/2575#issuecomment-756762795
something went wrong with yetus; try a rebase and forced push
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 533023)
Time Spent: 7h (was: 6h 50m)
> hadoop-aws does not work with hadoop-client-api
> ---
>
> Key: HADOOP-16080
> URL: https://issues.apache.org/jira/browse/HADOOP-16080
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.2.0, 3.1.1, 3.4.0
>Reporter: Keith Turner
>Assignee: Chao Sun
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.2, 3.3.1
>
> Time Spent: 7h
> Remaining Estimate: 0h
>
> I attempted to use Accumulo and S3a with the following jars on the classpath.
> * hadoop-client-api-3.1.1.jar
> * hadoop-client-runtime-3.1.1.jar
> * hadoop-aws-3.1.1.jar
> This failed with the following exception.
> {noformat}
> Exception in thread "init" java.lang.NoSuchMethodError:
> org.apache.hadoop.util.SemaphoredDelegatingExecutor.(Lcom/google/common/util/concurrent/ListeningExecutorService;IZ)V
> at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:769)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1108)
> at org.apache.hadoop.fs.FileSystem.createNewFile(FileSystem.java:1413)
> at
> org.apache.accumulo.server.fs.VolumeManagerImpl.createNewFile(VolumeManagerImpl.java:184)
> at
> org.apache.accumulo.server.init.Initialize.initDirs(Initialize.java:479)
> at
> org.apache.accumulo.server.init.Initialize.initFileSystem(Initialize.java:487)
> at
> org.apache.accumulo.server.init.Initialize.initialize(Initialize.java:370)
> at org.apache.accumulo.server.init.Initialize.doInit(Initialize.java:348)
> at org.apache.accumulo.server.init.Initialize.execute(Initialize.java:967)
> at org.apache.accumulo.start.Main.lambda$execKeyword$0(Main.java:129)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> The problem is that {{S3AFileSystem.create()}} looks for
> {{SemaphoredDelegatingExecutor(com.google.common.util.concurrent.ListeningExecutorService)}}
> which does not exist in hadoop-client-api-3.1.1.jar. What does exist is
> {{SemaphoredDelegatingExecutor(org.apache.hadoop.shaded.com.google.common.util.concurrent.ListeningExecutorService)}}.
> To work around this issue I created a version of hadoop-aws-3.1.1.jar that
> relocated references to Guava.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2575: HADOOP-16080. hadoop-aws does not work with hadoop-client-api
steveloughran commented on pull request #2575: URL: https://github.com/apache/hadoop/pull/2575#issuecomment-756762795 something went wrong with yetus; try a rebase and forced push This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17347) ABFS: Optimise read for small files/tails of files
[ https://issues.apache.org/jira/browse/HADOOP-17347?focusedWorklogId=533015=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533015 ] ASF GitHub Bot logged work on HADOOP-17347: --- Author: ASF GitHub Bot Created on: 08/Jan/21 13:37 Start Date: 08/Jan/21 13:37 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2464: URL: https://github.com/apache/hadoop/pull/2464#issuecomment-756759257 It's moot now that the PR is merged, but can I remind everyone ## Add a meaningful message to assertTrue/assertFalse asserts Imagine that you are trying to debug a test run from an automated build. All you have is that an assert failed on a given line. Does that provide enough information to diagnose the problem? Or would you need extra information? If so: what information should be included? It's OK to use assertJ's assertThat, which is where new tests are going. It's a bit more verbose but its assertions are very informative and easily extensible. If you haven't used the library yet -it's on the classpath, try using its assertions in new test suites. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 533015) Time Spent: 12h 20m (was: 12h 10m) > ABFS: Optimise read for small files/tails of files > -- > > Key: HADOOP-17347 > URL: https://issues.apache.org/jira/browse/HADOOP-17347 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.4.0 >Reporter: Bilahari T H >Assignee: Bilahari T H >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 12h 20m > Remaining Estimate: 0h > > Optimize read performance for the following scenarios > # Read small files completely > Files that are of size smaller than the read buffer size can be considered > as small files. In case of such files it would be better to read the full > file into the AbfsInputStream buffer. > # Read last block if the read is for footer > If the read is for the last 8 bytes, read the full file. > This will optimize reads for parquet files. [Parquet file > format|https://www.ellicium.com/parquet-file-format-structure/] > Both these optimizations will be present under configs as follows > # fs.azure.read.smallfilescompletely > # fs.azure.read.optimizefooterread -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2464: HADOOP-17347. ABFS: Read optimizations
steveloughran commented on pull request #2464: URL: https://github.com/apache/hadoop/pull/2464#issuecomment-756759257 It's moot now that the PR is merged, but can I remind everyone ## Add a meaningful message to assertTrue/assertFalse asserts Imagine that you are trying to debug a test run from an automated build. All you have is that an assert failed on a given line. Does that provide enough information to diagnose the problem? Or would you need extra information? If so: what information should be included? It's OK to use assertJ's assertThat, which is where new tests are going. It's a bit more verbose but its assertions are very informative and easily extensible. If you haven't used the library yet -it's on the classpath, try using its assertions in new test suites. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16080) hadoop-aws does not work with hadoop-client-api
[
https://issues.apache.org/jira/browse/HADOOP-16080?focusedWorklogId=533011=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533011
]
ASF GitHub Bot logged work on HADOOP-16080:
---
Author: ASF GitHub Bot
Created on: 08/Jan/21 13:33
Start Date: 08/Jan/21 13:33
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2575:
URL: https://github.com/apache/hadoop/pull/2575#issuecomment-756757565
Ignoring the s3guard/ddb ones (we're clearly still trying to run some when
those tests aren't enabled), you are going to be seeing the failures covered in
https://issues.apache.org/jira/browse/HADOOP-17451 / #2594 . I'll get that
ready for merging today...mostly it's some of the tests using metrics being
brittle to how they are executed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 533011)
Time Spent: 6h 50m (was: 6h 40m)
> hadoop-aws does not work with hadoop-client-api
> ---
>
> Key: HADOOP-16080
> URL: https://issues.apache.org/jira/browse/HADOOP-16080
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.2.0, 3.1.1, 3.4.0
>Reporter: Keith Turner
>Assignee: Chao Sun
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.2, 3.3.1
>
> Time Spent: 6h 50m
> Remaining Estimate: 0h
>
> I attempted to use Accumulo and S3a with the following jars on the classpath.
> * hadoop-client-api-3.1.1.jar
> * hadoop-client-runtime-3.1.1.jar
> * hadoop-aws-3.1.1.jar
> This failed with the following exception.
> {noformat}
> Exception in thread "init" java.lang.NoSuchMethodError:
> org.apache.hadoop.util.SemaphoredDelegatingExecutor.(Lcom/google/common/util/concurrent/ListeningExecutorService;IZ)V
> at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:769)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1108)
> at org.apache.hadoop.fs.FileSystem.createNewFile(FileSystem.java:1413)
> at
> org.apache.accumulo.server.fs.VolumeManagerImpl.createNewFile(VolumeManagerImpl.java:184)
> at
> org.apache.accumulo.server.init.Initialize.initDirs(Initialize.java:479)
> at
> org.apache.accumulo.server.init.Initialize.initFileSystem(Initialize.java:487)
> at
> org.apache.accumulo.server.init.Initialize.initialize(Initialize.java:370)
> at org.apache.accumulo.server.init.Initialize.doInit(Initialize.java:348)
> at org.apache.accumulo.server.init.Initialize.execute(Initialize.java:967)
> at org.apache.accumulo.start.Main.lambda$execKeyword$0(Main.java:129)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> The problem is that {{S3AFileSystem.create()}} looks for
> {{SemaphoredDelegatingExecutor(com.google.common.util.concurrent.ListeningExecutorService)}}
> which does not exist in hadoop-client-api-3.1.1.jar. What does exist is
> {{SemaphoredDelegatingExecutor(org.apache.hadoop.shaded.com.google.common.util.concurrent.ListeningExecutorService)}}.
> To work around this issue I created a version of hadoop-aws-3.1.1.jar that
> relocated references to Guava.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2575: HADOOP-16080. hadoop-aws does not work with hadoop-client-api
steveloughran commented on pull request #2575: URL: https://github.com/apache/hadoop/pull/2575#issuecomment-756757565 Ignoring the s3guard/ddb ones (we're clearly still trying to run some when those tests aren't enabled), you are going to be seeing the failures covered in https://issues.apache.org/jira/browse/HADOOP-17451 / #2594 . I'll get that ready for merging today...mostly it's some of the tests using metrics being brittle to how they are executed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-15760) Include Apache Commons Collections4
[
https://issues.apache.org/jira/browse/HADOOP-15760?focusedWorklogId=532993=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532993
]
ASF GitHub Bot logged work on HADOOP-15760:
---
Author: ASF GitHub Bot
Created on: 08/Jan/21 12:18
Start Date: 08/Jan/21 12:18
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2606:
URL: https://github.com/apache/hadoop/pull/2606#issuecomment-756726785
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 35m 3s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 4s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 21s | | trunk passed |
| +1 :green_heart: | compile | 20m 13s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 17m 17s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | mvnsite | 4m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 94m 4s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 3m 37s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 3m 51s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 23s | | the patch passed |
| +1 :green_heart: | compile | 20m 0s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 20m 0s | | the patch passed |
| +1 :green_heart: | compile | 17m 12s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 17m 12s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 39s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML
file. |
| -1 :x: | shadedclient | 0m 58s | | patch has errors when building
and testing our client artifacts. |
| +1 :green_heart: | javadoc | 3m 36s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 3m 57s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 33s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | unit | 9m 38s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 52s | |
hadoop-yarn-server-applicationhistoryservice in the patch passed. |
| -1 :x: | unit | 152m 44s |
[/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2606/1/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client.txt)
| hadoop-mapreduce-client in the patch passed. |
| +1 :green_heart: | asflicense | 1m 10s | | The patch does not
generate ASF License warnings. |
| | | 363m 43s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2606/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2606 |
| JIRA Issue | HADOOP-15760 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml |
| uname | Linux 0bba6cf65f74 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1b1791075a8 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
[jira] [Commented] (HADOOP-15760) Include Apache Commons Collections4
[
https://issues.apache.org/jira/browse/HADOOP-15760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261270#comment-17261270
]
Hadoop QA commented on HADOOP-15760:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 35m
3s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 14m
4s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
21s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
13s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
17s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
43s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
94m 4s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
37s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
51s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
23s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
0s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m
0s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
12s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
12s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
39s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
6s{color} | {color:green}{color} | {color:green} The patch has no ill-formed
XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 0m
58s{color} | {color:red}{color} | {color:red} patch has errors when building
and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
36s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
57s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private
[GitHub] [hadoop] hadoop-yetus commented on pull request #2606: HADOOP-15760: updated to commons-collections4
hadoop-yetus commented on pull request #2606:
URL: https://github.com/apache/hadoop/pull/2606#issuecomment-756726785
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 35m 3s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 4s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 21s | | trunk passed |
| +1 :green_heart: | compile | 20m 13s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 17m 17s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | mvnsite | 4m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 94m 4s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 3m 37s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 3m 51s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 23s | | the patch passed |
| +1 :green_heart: | compile | 20m 0s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 20m 0s | | the patch passed |
| +1 :green_heart: | compile | 17m 12s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 17m 12s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 39s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML
file. |
| -1 :x: | shadedclient | 0m 58s | | patch has errors when building
and testing our client artifacts. |
| +1 :green_heart: | javadoc | 3m 36s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 3m 57s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 33s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | unit | 9m 38s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 52s | |
hadoop-yarn-server-applicationhistoryservice in the patch passed. |
| -1 :x: | unit | 152m 44s |
[/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2606/1/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client.txt)
| hadoop-mapreduce-client in the patch passed. |
| +1 :green_heart: | asflicense | 1m 10s | | The patch does not
generate ASF License warnings. |
| | | 363m 43s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2606/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2606 |
| JIRA Issue | HADOOP-15760 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml |
| uname | Linux 0bba6cf65f74 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1b1791075a8 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2606/1/testReport/ |
| Max. process+thread count | 1664 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common
[jira] [Created] (HADOOP-17460) s3guard tool dumpStorageStatistics to move to IOStatistics
Steve Loughran created HADOOP-17460: --- Summary: s3guard tool dumpStorageStatistics to move to IOStatistics Key: HADOOP-17460 URL: https://issues.apache.org/jira/browse/HADOOP-17460 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.4.0 Reporter: Steve Loughran S3GuardTool cli's -verbose option prints storage statistics of the FS. If it moves to IOStatistics it will print latencies as well as op counts -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] lujiefsi opened a new pull request #2608: YARN-10555. missing access check before getAppAttempts
lujiefsi opened a new pull request #2608: URL: https://github.com/apache/hadoop/pull/2608 see https://issues.apache.org/jira/browse/YARN-10555 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17459) Fix the test case failures which are failing after the contract test update in hadoop-common
[
https://issues.apache.org/jira/browse/HADOOP-17459?focusedWorklogId=532960=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532960
]
ASF GitHub Bot logged work on HADOOP-17459:
---
Author: ASF GitHub Bot
Created on: 08/Jan/21 11:20
Start Date: 08/Jan/21 11:20
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2607:
URL: https://github.com/apache/hadoop/pull/2607#issuecomment-756702912
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 27s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 2 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 30m 1s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2607/1/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 22s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 19s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 25s | | trunk passed |
| -1 :x: | shadedclient | 17m 44s | | branch has errors when building
and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 21s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 37s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 34s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 0m 18s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 18s | | the patch passed |
| +1 :green_heart: | compile | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 17s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 11s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 19s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | shadedclient | 16m 49s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 37s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 49s | | hadoop-azure-datalake in the
patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 75m 9s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2607/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2607 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 810affa54ce6 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23
18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 87bd4d2aca5 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2607/1/testReport/ |
| Max. process+thread count | 628 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure-datalake U:
[GitHub] [hadoop] hadoop-yetus commented on pull request #2607: DRAFT PR HADOOP-17459. ADLS Gen1: Fixes for rename contract tests
hadoop-yetus commented on pull request #2607:
URL: https://github.com/apache/hadoop/pull/2607#issuecomment-756702912
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 27s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 2 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 30m 1s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2607/1/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 22s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 19s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 25s | | trunk passed |
| -1 :x: | shadedclient | 17m 44s | | branch has errors when building
and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 21s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 37s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 34s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 0m 18s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 18s | | the patch passed |
| +1 :green_heart: | compile | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 17s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 11s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 19s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | shadedclient | 16m 49s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 37s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 49s | | hadoop-azure-datalake in the
patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 75m 9s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2607/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2607 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 810affa54ce6 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23
18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 87bd4d2aca5 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2607/1/testReport/ |
| Max. process+thread count | 628 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure-datalake U:
hadoop-tools/hadoop-azure-datalake |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2607/1/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
This is an automated message from the Apache Git Service.
[jira] [Work logged] (HADOOP-17414) Magic committer files don't have the count of bytes written collected by spark
[ https://issues.apache.org/jira/browse/HADOOP-17414?focusedWorklogId=532941=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532941 ] ASF GitHub Bot logged work on HADOOP-17414: --- Author: ASF GitHub Bot Created on: 08/Jan/21 10:57 Start Date: 08/Jan/21 10:57 Worklog Time Spent: 10m Work Description: bgaborg commented on a change in pull request #2530: URL: https://github.com/apache/hadoop/pull/2530#discussion_r553870733 ## File path: hadoop-common-project/hadoop-common/src/main/resources/core-default.xml ## @@ -1873,11 +1873,9 @@ fs.s3a.committer.magic.enabled - false + true Review comment: with this change we will enable magic committer for all the components. Maybe point that out in the title of the issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 532941) Time Spent: 4h 20m (was: 4h 10m) > Magic committer files don't have the count of bytes written collected by spark > -- > > Key: HADOOP-17414 > URL: https://issues.apache.org/jira/browse/HADOOP-17414 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 4h 20m > Remaining Estimate: 0h > > The spark statistics tracking doesn't correctly assess the size of the > uploaded files as it only calls getFileStatus on the zero byte objects -not > the yet-to-manifest files. Which, given they don't exist yet, isn't easy to > do. > Solution: > * Add getXAttr and listXAttr API calls to S3AFileSystem > * Return all S3 object headers as XAttr attributes prefixed "header." That's > custom and standard (e.g header.Content-Length). > The setXAttr call isn't implemented, so for correctness the FS doesn't > declare its support for the API in hasPathCapability(). > The magic commit file write sets the custom header > set the length of the data final data in the header > x-hadoop-s3a-magic-data-length in the marker file. > A matching patch in Spark will look for the XAttr > "header.x-hadoop-s3a-magic-data-length" when the file > being probed for output data is zero byte long. > As a result, the job tracking statistics will report the > bytes written but yet to be manifest. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on a change in pull request #2530: HADOOP-17414. Magic committer files don't have the count of bytes written collected by spark
bgaborg commented on a change in pull request #2530: URL: https://github.com/apache/hadoop/pull/2530#discussion_r553870733 ## File path: hadoop-common-project/hadoop-common/src/main/resources/core-default.xml ## @@ -1873,11 +1873,9 @@ fs.s3a.committer.magic.enabled - false + true Review comment: with this change we will enable magic committer for all the components. Maybe point that out in the title of the issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17430) Restore ability to set Text to empty byte array
[
https://issues.apache.org/jira/browse/HADOOP-17430?focusedWorklogId=532936=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532936
]
ASF GitHub Bot logged work on HADOOP-17430:
---
Author: ASF GitHub Bot
Created on: 08/Jan/21 10:51
Start Date: 08/Jan/21 10:51
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2545:
URL: https://github.com/apache/hadoop/pull/2545#issuecomment-756690095
sorry, was my cherrypick which broke it. reverted
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 532936)
Time Spent: 4h (was: 3h 50m)
> Restore ability to set Text to empty byte array
>
>
> Key: HADOOP-17430
> URL: https://issues.apache.org/jira/browse/HADOOP-17430
> Project: Hadoop Common
> Issue Type: Wish
> Components: common
>Reporter: gaozhan ding
>Assignee: gaozhan ding
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.3.1
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> In org.apache.hadoop.io.Text:clear() method, the comments show that we can
> free the bytes by call set(new byte[0]), but it's not going to work now.
> Maybe we can follow this comments.
>
>
> {code:java}
> // org.apache.hadoop.io.Text
> /**
> * Clear the string to empty.
> *
> * Note: For performance reasons, this call does not clear the
> * underlying byte array that is retrievable via {@link #getBytes()}.
> * In order to free the byte-array memory, call {@link #set(byte[])}
> * with an empty byte array (For example, new byte[0]).
> */
> public void clear() {
> length = 0;
> textLength = -1;
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2545: HADOOP-17430. Add clear bytes logic for hadoop Text
steveloughran commented on pull request #2545: URL: https://github.com/apache/hadoop/pull/2545#issuecomment-756690095 sorry, was my cherrypick which broke it. reverted This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17347) ABFS: Optimise read for small files/tails of files
[ https://issues.apache.org/jira/browse/HADOOP-17347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-17347. - Fix Version/s: 3.4.0 Resolution: Fixed IF this is to target branch-3.3 then full test rerun. I'd like some docs too, maybe something on tuning the abfs connector for performance. Too many under-documented config options are coming in now > ABFS: Optimise read for small files/tails of files > -- > > Key: HADOOP-17347 > URL: https://issues.apache.org/jira/browse/HADOOP-17347 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.4.0 >Reporter: Bilahari T H >Assignee: Bilahari T H >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 12h 10m > Remaining Estimate: 0h > > Optimize read performance for the following scenarios > # Read small files completely > Files that are of size smaller than the read buffer size can be considered > as small files. In case of such files it would be better to read the full > file into the AbfsInputStream buffer. > # Read last block if the read is for footer > If the read is for the last 8 bytes, read the full file. > This will optimize reads for parquet files. [Parquet file > format|https://www.ellicium.com/parquet-file-format-structure/] > Both these optimizations will be present under configs as follows > # fs.azure.read.smallfilescompletely > # fs.azure.read.optimizefooterread -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17347) ABFS: Optimise read for small files/tails of files
[ https://issues.apache.org/jira/browse/HADOOP-17347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261211#comment-17261211 ] Steve Loughran commented on HADOOP-17347: - I'm assuming its driven a bit by the read sequence of a parquet file which is * tail -8 file to check for magic and offset of real footer * seek to real footer and read > ABFS: Optimise read for small files/tails of files > -- > > Key: HADOOP-17347 > URL: https://issues.apache.org/jira/browse/HADOOP-17347 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.4.0 >Reporter: Bilahari T H >Assignee: Bilahari T H >Priority: Major > Labels: pull-request-available > Time Spent: 12h 10m > Remaining Estimate: 0h > > Optimize read performance for the following scenarios > # Read small files completely > Files that are of size smaller than the read buffer size can be considered > as small files. In case of such files it would be better to read the full > file into the AbfsInputStream buffer. > # Read last block if the read is for footer > If the read is for the last 8 bytes, read the full file. > This will optimize reads for parquet files. [Parquet file > format|https://www.ellicium.com/parquet-file-format-structure/] > Both these optimizations will be present under configs as follows > # fs.azure.read.smallfilescompletely > # fs.azure.read.optimizefooterread -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
