[jira] [Commented] (HADOOP-17276) Extend CallerContext to make it include many items
[ https://issues.apache.org/jira/browse/HADOOP-17276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692550#comment-17692550 ] Takanobu Asanuma commented on HADOOP-17276: --- I added 3.3.5 to fix versions. > Extend CallerContext to make it include many items > -- > > Key: HADOOP-17276 > URL: https://issues.apache.org/jira/browse/HADOOP-17276 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hui Fei >Assignee: Hui Fei >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.5 > > Time Spent: 9h > Remaining Estimate: 0h > > Now context is string. We need to extend the CallerContext because context > may contains many items. > Items include > * router ip > * MR or CLI > * etc -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17276) Extend CallerContext to make it include many items
[ https://issues.apache.org/jira/browse/HADOOP-17276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HADOOP-17276: -- Fix Version/s: 3.3.5 > Extend CallerContext to make it include many items > -- > > Key: HADOOP-17276 > URL: https://issues.apache.org/jira/browse/HADOOP-17276 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hui Fei >Assignee: Hui Fei >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.5 > > Time Spent: 9h > Remaining Estimate: 0h > > Now context is string. We need to extend the CallerContext because context > may contains many items. > Items include > * router ip > * MR or CLI > * etc -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18345) Enhance client protocol to propagate last seen state IDs for multiple nameservices.
[
https://issues.apache.org/jira/browse/HADOOP-18345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HADOOP-18345:
--
Fix Version/s: (was: 3.3.5)
> Enhance client protocol to propagate last seen state IDs for multiple
> nameservices.
> ---
>
> Key: HADOOP-18345
> URL: https://issues.apache.org/jira/browse/HADOOP-18345
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> The RPCHeader in the client protocol currently contains a single value to
> indicate the last seen state ID for a namenode.
> {noformat}
> optional int64 stateId = 8; // The last seen Global State ID
> {noformat}
> When there are multiple namenodes, such as in router based federation, the
> headers need to carry the state IDs for each of these nameservices that are
> part of the federation.
> This change is a prerequisite for HDFS-13522: RBF: Support observer node from
> Router-Based Federation
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18345) Enhance client protocol to propagate last seen state IDs for multiple nameservices.
[
https://issues.apache.org/jira/browse/HADOOP-18345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692548#comment-17692548
]
Takanobu Asanuma commented on HADOOP-18345:
---
It is only in trunk. I corrected the fixed versions.
> Enhance client protocol to propagate last seen state IDs for multiple
> nameservices.
> ---
>
> Key: HADOOP-18345
> URL: https://issues.apache.org/jira/browse/HADOOP-18345
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> The RPCHeader in the client protocol currently contains a single value to
> indicate the last seen state ID for a namenode.
> {noformat}
> optional int64 stateId = 8; // The last seen Global State ID
> {noformat}
> When there are multiple namenodes, such as in router based federation, the
> headers need to carry the state IDs for each of these nameservices that are
> part of the federation.
> This change is a prerequisite for HDFS-13522: RBF: Support observer node from
> Router-Based Federation
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18632) ABFS: Customize and optimize timeouts made based on each separate request
[
https://issues.apache.org/jira/browse/HADOOP-18632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692542#comment-17692542
]
ASF GitHub Bot commented on HADOOP-18632:
-
sreeb-msft commented on code in PR #5399:
URL: https://github.com/apache/hadoop/pull/5399#discussion_r1115320710
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/TimeoutOptimizer.java:
##
@@ -0,0 +1,227 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys;
+import org.apache.hadoop.fs.azurebfs.constants.HttpQueryParams;
+import org.apache.http.client.utils.URIBuilder;
+
+import java.net.MalformedURLException;
+import java.net.URISyntaxException;
+import java.net.URL;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.DEFAULT_TIMEOUT;
+
+public class TimeoutOptimizer {
+AbfsConfiguration abfsConfiguration;
+private URL url;
+private AbfsRestOperationType opType;
+private ExponentialRetryPolicy retryPolicy;
+private int requestTimeout;
+private int readTimeout = -1;
+private int connTimeout = -1;
+private int maxReqTimeout;
+private int timeoutIncRate;
+private boolean shouldOptimizeTimeout;
+
+public TimeoutOptimizer(URL url, AbfsRestOperationType opType,
ExponentialRetryPolicy retryPolicy, AbfsConfiguration abfsConfiguration) {
+this.url = url;
+this.opType = opType;
+if (opType != null) {
+this.retryPolicy = retryPolicy;
+this.abfsConfiguration = abfsConfiguration;
+if
(abfsConfiguration.get(ConfigurationKeys.AZURE_OPTIMIZE_TIMEOUTS) == null) {
+this.shouldOptimizeTimeout = false;
+}
+else {
+this.shouldOptimizeTimeout =
Boolean.parseBoolean(abfsConfiguration.get(ConfigurationKeys.AZURE_OPTIMIZE_TIMEOUTS));
+}
+if (this.shouldOptimizeTimeout) {
+this.maxReqTimeout =
Integer.parseInt(abfsConfiguration.get(ConfigurationKeys.AZURE_MAX_REQUEST_TIMEOUT));
+this.timeoutIncRate =
Integer.parseInt(abfsConfiguration.get(ConfigurationKeys.AZURE_REQUEST_TIMEOUT_INCREASE_RATE));
+initTimeouts();
+updateUrl();
+}
+
+} else {
+this.shouldOptimizeTimeout = false;
+}
+}
+
+public void updateRetryTimeout(int retryCount) {
+if (!this.shouldOptimizeTimeout) {
+return;
+}
+
+// update all timeout values
+updateTimeouts(retryCount);
+updateUrl();
+}
+
+public URL getUrl() {
+return url;
+}
+public boolean getShouldOptimizeTimeout() { return
this.shouldOptimizeTimeout; }
+
+public int getRequestTimeout() { return requestTimeout; }
+
+public int getReadTimeout() {
+return readTimeout;
+}
+
+public int getReadTimeout(final int defaultTimeout) {
+if (readTimeout != -1 && shouldOptimizeTimeout) {
+return readTimeout;
+}
+return defaultTimeout;
+}
+
+public int getConnTimeout() {
+return connTimeout;
+}
+
+public int getConnTimeout(final int defaultTimeout) {
+if (connTimeout == -1) {
+return defaultTimeout;
+}
+return connTimeout;
+}
+
+private void initTimeouts() {
+if (!shouldOptimizeTimeout) {
+requestTimeout = -1;
+readTimeout = -1;
+connTimeout = -1;
+return;
+}
+
+String query = url.getQuery();
+int timeoutPos = query.indexOf("timeout");
+if (timeoutPos < 0) {
+// no value of timeout exists in the URL
+// no optimization is needed for this particular request as well
+requestTimeout = -1;
+readTimeout = -1;
+connTimeout = -1;
+shouldOptimizeTimeout = false;
+return;
+}
+
+String timeout = "";
+
[GitHub] [hadoop] sreeb-msft commented on a diff in pull request #5399: HADOOP-18632: [ABFS] Customize and optimize timeouts made based on each separate request
sreeb-msft commented on code in PR #5399:
URL: https://github.com/apache/hadoop/pull/5399#discussion_r1115320710
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/TimeoutOptimizer.java:
##
@@ -0,0 +1,227 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys;
+import org.apache.hadoop.fs.azurebfs.constants.HttpQueryParams;
+import org.apache.http.client.utils.URIBuilder;
+
+import java.net.MalformedURLException;
+import java.net.URISyntaxException;
+import java.net.URL;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.DEFAULT_TIMEOUT;
+
+public class TimeoutOptimizer {
+AbfsConfiguration abfsConfiguration;
+private URL url;
+private AbfsRestOperationType opType;
+private ExponentialRetryPolicy retryPolicy;
+private int requestTimeout;
+private int readTimeout = -1;
+private int connTimeout = -1;
+private int maxReqTimeout;
+private int timeoutIncRate;
+private boolean shouldOptimizeTimeout;
+
+public TimeoutOptimizer(URL url, AbfsRestOperationType opType,
ExponentialRetryPolicy retryPolicy, AbfsConfiguration abfsConfiguration) {
+this.url = url;
+this.opType = opType;
+if (opType != null) {
+this.retryPolicy = retryPolicy;
+this.abfsConfiguration = abfsConfiguration;
+if
(abfsConfiguration.get(ConfigurationKeys.AZURE_OPTIMIZE_TIMEOUTS) == null) {
+this.shouldOptimizeTimeout = false;
+}
+else {
+this.shouldOptimizeTimeout =
Boolean.parseBoolean(abfsConfiguration.get(ConfigurationKeys.AZURE_OPTIMIZE_TIMEOUTS));
+}
+if (this.shouldOptimizeTimeout) {
+this.maxReqTimeout =
Integer.parseInt(abfsConfiguration.get(ConfigurationKeys.AZURE_MAX_REQUEST_TIMEOUT));
+this.timeoutIncRate =
Integer.parseInt(abfsConfiguration.get(ConfigurationKeys.AZURE_REQUEST_TIMEOUT_INCREASE_RATE));
+initTimeouts();
+updateUrl();
+}
+
+} else {
+this.shouldOptimizeTimeout = false;
+}
+}
+
+public void updateRetryTimeout(int retryCount) {
+if (!this.shouldOptimizeTimeout) {
+return;
+}
+
+// update all timeout values
+updateTimeouts(retryCount);
+updateUrl();
+}
+
+public URL getUrl() {
+return url;
+}
+public boolean getShouldOptimizeTimeout() { return
this.shouldOptimizeTimeout; }
+
+public int getRequestTimeout() { return requestTimeout; }
+
+public int getReadTimeout() {
+return readTimeout;
+}
+
+public int getReadTimeout(final int defaultTimeout) {
+if (readTimeout != -1 && shouldOptimizeTimeout) {
+return readTimeout;
+}
+return defaultTimeout;
+}
+
+public int getConnTimeout() {
+return connTimeout;
+}
+
+public int getConnTimeout(final int defaultTimeout) {
+if (connTimeout == -1) {
+return defaultTimeout;
+}
+return connTimeout;
+}
+
+private void initTimeouts() {
+if (!shouldOptimizeTimeout) {
+requestTimeout = -1;
+readTimeout = -1;
+connTimeout = -1;
+return;
+}
+
+String query = url.getQuery();
+int timeoutPos = query.indexOf("timeout");
+if (timeoutPos < 0) {
+// no value of timeout exists in the URL
+// no optimization is needed for this particular request as well
+requestTimeout = -1;
+readTimeout = -1;
+connTimeout = -1;
+shouldOptimizeTimeout = false;
+return;
+}
+
+String timeout = "";
+if (opType == AbfsRestOperationType.CreateFileSystem) {
+timeout =
abfsConfiguration.get(ConfigurationKeys.AZURE_CREATE_FS_REQUEST_TIMEOUT);
+}
+else if (opType == AbfsRestOperationType.GetFileSystemProperties)
[jira] [Commented] (HADOOP-18632) ABFS: Customize and optimize timeouts made based on each separate request
[
https://issues.apache.org/jira/browse/HADOOP-18632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692536#comment-17692536
]
ASF GitHub Bot commented on HADOOP-18632:
-
sreeb-msft commented on code in PR #5399:
URL: https://github.com/apache/hadoop/pull/5399#discussion_r1115313395
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/TimeoutOptimizer.java:
##
@@ -0,0 +1,227 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys;
+import org.apache.hadoop.fs.azurebfs.constants.HttpQueryParams;
+import org.apache.http.client.utils.URIBuilder;
+
+import java.net.MalformedURLException;
+import java.net.URISyntaxException;
+import java.net.URL;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.DEFAULT_TIMEOUT;
+
+public class TimeoutOptimizer {
+AbfsConfiguration abfsConfiguration;
+private URL url;
+private AbfsRestOperationType opType;
+private ExponentialRetryPolicy retryPolicy;
+private int requestTimeout;
+private int readTimeout = -1;
+private int connTimeout = -1;
+private int maxReqTimeout;
+private int timeoutIncRate;
+private boolean shouldOptimizeTimeout;
+
+public TimeoutOptimizer(URL url, AbfsRestOperationType opType,
ExponentialRetryPolicy retryPolicy, AbfsConfiguration abfsConfiguration) {
+this.url = url;
+this.opType = opType;
+if (opType != null) {
+this.retryPolicy = retryPolicy;
+this.abfsConfiguration = abfsConfiguration;
+if
(abfsConfiguration.get(ConfigurationKeys.AZURE_OPTIMIZE_TIMEOUTS) == null) {
+this.shouldOptimizeTimeout = false;
+}
+else {
+this.shouldOptimizeTimeout =
Boolean.parseBoolean(abfsConfiguration.get(ConfigurationKeys.AZURE_OPTIMIZE_TIMEOUTS));
+}
+if (this.shouldOptimizeTimeout) {
+this.maxReqTimeout =
Integer.parseInt(abfsConfiguration.get(ConfigurationKeys.AZURE_MAX_REQUEST_TIMEOUT));
+this.timeoutIncRate =
Integer.parseInt(abfsConfiguration.get(ConfigurationKeys.AZURE_REQUEST_TIMEOUT_INCREASE_RATE));
+initTimeouts();
+updateUrl();
+}
+
+} else {
+this.shouldOptimizeTimeout = false;
+}
+}
+
+public void updateRetryTimeout(int retryCount) {
+if (!this.shouldOptimizeTimeout) {
+return;
+}
+
+// update all timeout values
+updateTimeouts(retryCount);
+updateUrl();
+}
+
+public URL getUrl() {
+return url;
+}
+public boolean getShouldOptimizeTimeout() { return
this.shouldOptimizeTimeout; }
+
+public int getRequestTimeout() { return requestTimeout; }
+
+public int getReadTimeout() {
+return readTimeout;
+}
+
+public int getReadTimeout(final int defaultTimeout) {
+if (readTimeout != -1 && shouldOptimizeTimeout) {
+return readTimeout;
+}
+return defaultTimeout;
+}
+
+public int getConnTimeout() {
+return connTimeout;
+}
+
+public int getConnTimeout(final int defaultTimeout) {
+if (connTimeout == -1) {
+return defaultTimeout;
+}
+return connTimeout;
+}
+
+private void initTimeouts() {
+if (!shouldOptimizeTimeout) {
+requestTimeout = -1;
Review Comment:
This value of the request timeout would never reach the server. This is just
used as a flag value. In the case where request timeout / read timeout stays
set as -1, the optimizer can be considered to not be doing any work (ie, when
either the config for whether optimization should happen or not is set to
false, or when the request in consideration does not have a timeout value in
its query parameters, like create Filesystem, or when any of the configs are
incorrectly set).
> ABFS: Customize and optimize timeouts made based
[GitHub] [hadoop] sreeb-msft commented on a diff in pull request #5399: HADOOP-18632: [ABFS] Customize and optimize timeouts made based on each separate request
sreeb-msft commented on code in PR #5399:
URL: https://github.com/apache/hadoop/pull/5399#discussion_r1115313395
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/TimeoutOptimizer.java:
##
@@ -0,0 +1,227 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys;
+import org.apache.hadoop.fs.azurebfs.constants.HttpQueryParams;
+import org.apache.http.client.utils.URIBuilder;
+
+import java.net.MalformedURLException;
+import java.net.URISyntaxException;
+import java.net.URL;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.DEFAULT_TIMEOUT;
+
+public class TimeoutOptimizer {
+AbfsConfiguration abfsConfiguration;
+private URL url;
+private AbfsRestOperationType opType;
+private ExponentialRetryPolicy retryPolicy;
+private int requestTimeout;
+private int readTimeout = -1;
+private int connTimeout = -1;
+private int maxReqTimeout;
+private int timeoutIncRate;
+private boolean shouldOptimizeTimeout;
+
+public TimeoutOptimizer(URL url, AbfsRestOperationType opType,
ExponentialRetryPolicy retryPolicy, AbfsConfiguration abfsConfiguration) {
+this.url = url;
+this.opType = opType;
+if (opType != null) {
+this.retryPolicy = retryPolicy;
+this.abfsConfiguration = abfsConfiguration;
+if
(abfsConfiguration.get(ConfigurationKeys.AZURE_OPTIMIZE_TIMEOUTS) == null) {
+this.shouldOptimizeTimeout = false;
+}
+else {
+this.shouldOptimizeTimeout =
Boolean.parseBoolean(abfsConfiguration.get(ConfigurationKeys.AZURE_OPTIMIZE_TIMEOUTS));
+}
+if (this.shouldOptimizeTimeout) {
+this.maxReqTimeout =
Integer.parseInt(abfsConfiguration.get(ConfigurationKeys.AZURE_MAX_REQUEST_TIMEOUT));
+this.timeoutIncRate =
Integer.parseInt(abfsConfiguration.get(ConfigurationKeys.AZURE_REQUEST_TIMEOUT_INCREASE_RATE));
+initTimeouts();
+updateUrl();
+}
+
+} else {
+this.shouldOptimizeTimeout = false;
+}
+}
+
+public void updateRetryTimeout(int retryCount) {
+if (!this.shouldOptimizeTimeout) {
+return;
+}
+
+// update all timeout values
+updateTimeouts(retryCount);
+updateUrl();
+}
+
+public URL getUrl() {
+return url;
+}
+public boolean getShouldOptimizeTimeout() { return
this.shouldOptimizeTimeout; }
+
+public int getRequestTimeout() { return requestTimeout; }
+
+public int getReadTimeout() {
+return readTimeout;
+}
+
+public int getReadTimeout(final int defaultTimeout) {
+if (readTimeout != -1 && shouldOptimizeTimeout) {
+return readTimeout;
+}
+return defaultTimeout;
+}
+
+public int getConnTimeout() {
+return connTimeout;
+}
+
+public int getConnTimeout(final int defaultTimeout) {
+if (connTimeout == -1) {
+return defaultTimeout;
+}
+return connTimeout;
+}
+
+private void initTimeouts() {
+if (!shouldOptimizeTimeout) {
+requestTimeout = -1;
Review Comment:
This value of the request timeout would never reach the server. This is just
used as a flag value. In the case where request timeout / read timeout stays
set as -1, the optimizer can be considered to not be doing any work (ie, when
either the config for whether optimization should happen or not is set to
false, or when the request in consideration does not have a timeout value in
its query parameters, like create Filesystem, or when any of the configs are
incorrectly set).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact
[GitHub] [hadoop] tasanuma commented on pull request #5371: HDFS-18324. Fix race condition in closing IPC connections.
tasanuma commented on PR #5371: URL: https://github.com/apache/hadoop/pull/5371#issuecomment-1441289106 Seems HDFS-18324 is wrong, we should update the title to HDFS-16853 or HADOOP-18324, shouldn't we? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5397: HDFS-16917 Add transfer rate quantile metrics for DataNode reads
hadoop-yetus commented on PR #5397:
URL: https://github.com/apache/hadoop/pull/5397#issuecomment-1441280753
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 50s | | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 30m 58s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/14/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 13m 22s |
[/branch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/14/artifact/out/branch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| -1 :x: | compile | 12m 0s |
[/branch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/14/artifact/out/branch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
| root in trunk failed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08.
|
| +1 :green_heart: | checkstyle | 3m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 5s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 2s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 2m 16s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 5m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 38s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 36s | | the patch passed |
| -1 :x: | compile | 13m 12s |
[/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/14/artifact/out/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| -1 :x: | javac | 13m 12s |
[/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/14/artifact/out/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| -1 :x: | compile | 11m 53s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/14/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
| root in the patch failed with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08. |
| -1 :x: | javac | 11m 53s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/14/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
| root in the patch failed with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08. |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/14/artifact/out/blanks-eol.txt)
| The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 3m 20s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/14/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 205 unchanged - 0 fixed = 206 total (was
205) |
| +1 :green_heart: | mvnsite | 2m 55s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 49s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 2m 5s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 5m
[jira] [Commented] (HADOOP-18399) SingleFilePerBlockCache to use LocalDirAllocator for file allocation
[
https://issues.apache.org/jira/browse/HADOOP-18399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692516#comment-17692516
]
ASF GitHub Bot commented on HADOOP-18399:
-
hadoop-yetus commented on PR #5054:
URL: https://github.com/apache/hadoop/pull/5054#issuecomment-1441271453
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 1s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 34m 2s | | trunk passed |
| +1 :green_heart: | compile | 25m 20s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 21m 43s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 20s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 28m 51s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 50s | | the patch passed |
| +1 :green_heart: | compile | 24m 42s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 24m 42s | | the patch passed |
| +1 :green_heart: | compile | 21m 36s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 21m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 49s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 29s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 53s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 13s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 42s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 53s | | The patch does not
generate ASF License warnings. |
| | | 254m 22s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5054/19/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5054 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 513ffae26e9e 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a61f573c1e73a983455534b95211036d3799affe |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5054/19/testReport/ |
| Max. process+thread count | 3134 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws
U: . |
| Console output |
[GitHub] [hadoop] hadoop-yetus commented on pull request #5054: HADOOP-18399 Prefetch - SingleFilePerBlockCache to use LocalDirAllocator for file allocation
hadoop-yetus commented on PR #5054:
URL: https://github.com/apache/hadoop/pull/5054#issuecomment-1441271453
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 1s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 34m 2s | | trunk passed |
| +1 :green_heart: | compile | 25m 20s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 21m 43s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 20s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 28m 51s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 50s | | the patch passed |
| +1 :green_heart: | compile | 24m 42s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 24m 42s | | the patch passed |
| +1 :green_heart: | compile | 21m 36s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 21m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 49s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 29s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 53s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 13s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 42s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 53s | | The patch does not
generate ASF License warnings. |
| | | 254m 22s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5054/19/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5054 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 513ffae26e9e 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a61f573c1e73a983455534b95211036d3799affe |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5054/19/testReport/ |
| Max. process+thread count | 3134 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws
U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5054/19/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
[jira] [Commented] (HADOOP-18550) Web UI has no session controller.There is a risk of session hijacking.
[ https://issues.apache.org/jira/browse/HADOOP-18550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692513#comment-17692513 ] rickboker commented on HADOOP-18550: [~brahmareddy] yes I have a plane to reslove it. > Web UI has no session controller.There is a risk of session hijacking. > -- > > Key: HADOOP-18550 > URL: https://issues.apache.org/jira/browse/HADOOP-18550 > Project: Hadoop Common > Issue Type: Improvement > Components: website >Affects Versions: 3.3.1 >Reporter: rickboker >Priority: Major > > When I browse the webui,and keep this page connect. the session which jetty > given me.And session will keep alive as long as page alive. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5380: YARN-11079. Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality
hadoop-yetus commented on PR #5380: URL: https://github.com/apache/hadoop/pull/5380#issuecomment-1441258209 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 1s | | Docker mode activated. | | -1 :x: | patch | 0m 19s | | https://github.com/apache/hadoop/pull/5380 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/5380 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/2/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5397: HDFS-16917 Add transfer rate quantile metrics for DataNode reads
hadoop-yetus commented on PR #5397:
URL: https://github.com/apache/hadoop/pull/5397#issuecomment-1441251613
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 53s | | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 31m 14s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/13/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 13m 24s |
[/branch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/13/artifact/out/branch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| -1 :x: | compile | 11m 59s |
[/branch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/13/artifact/out/branch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
| root in trunk failed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08.
|
| +1 :green_heart: | checkstyle | 4m 19s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 6s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 0s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 2m 13s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 5m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 47s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 30s | | the patch passed |
| -1 :x: | compile | 13m 16s |
[/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/13/artifact/out/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| -1 :x: | javac | 13m 16s |
[/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/13/artifact/out/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| -1 :x: | compile | 11m 51s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/13/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
| root in the patch failed with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08. |
| -1 :x: | javac | 11m 51s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/13/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
| root in the patch failed with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08. |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/13/artifact/out/blanks-eol.txt)
| The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 3m 21s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/13/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 2 new + 205 unchanged - 0 fixed = 207 total (was
205) |
| +1 :green_heart: | mvnsite | 2m 50s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 46s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 2m 4s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 5m
[GitHub] [hadoop] hadoop-yetus commented on pull request #5397: HDFS-16917 Add transfer rate quantile metrics for DataNode reads
hadoop-yetus commented on PR #5397:
URL: https://github.com/apache/hadoop/pull/5397#issuecomment-1441216333
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 40s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 11s | | trunk passed |
| +1 :green_heart: | compile | 23m 5s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 20m 32s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 3m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 27s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 2m 40s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 6m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 31s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 33s | | the patch passed |
| +1 :green_heart: | compile | 22m 27s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 22m 27s | | the patch passed |
| +1 :green_heart: | compile | 20m 33s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 20m 33s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/11/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 4m 57s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/11/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 140 unchanged - 0 fixed = 141 total (was
140) |
| +1 :green_heart: | mvnsite | 3m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 19s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 2m 40s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 6m 22s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 34s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 15s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 210m 48s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/11/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 14s | | The patch does not
generate ASF License warnings. |
| | | 458m 33s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestObserverNode |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/11/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5397 |
| Optional Tests | dupname asflicense mvnsite codespell detsecrets
markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs
checkstyle |
| uname | Linux c1a9574555cb 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4ddbda70bf85d12e8e26bb03f88974615d45f049 |
| Default Java |
[jira] [Resolved] (HADOOP-18582) No need to clean tmp files in distcp direct mode
[
https://issues.apache.org/jira/browse/HADOOP-18582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena resolved HADOOP-18582.
---
Hadoop Flags: Reviewed
Resolution: Fixed
> No need to clean tmp files in distcp direct mode
>
>
> Key: HADOOP-18582
> URL: https://issues.apache.org/jira/browse/HADOOP-18582
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.3.4
>Reporter: 1kang
>Assignee: 1kang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
>
> it not necessary to do `cleanupTempFiles` while ditcp commit job in direct
> mode, because it there is no temp files in direct mode.
> This clean operation will increase the task execution time, because it will
> get the list of files in the target path. When the number of files in the
> target path is very large, this operation will be very slow.
> *note* there are two patches which need to be cherrypicked when picking this
> up; the original patch and a followup, both with HADOOP-18582 in the title
> {code}
> 3b7b79b37ae HADOOP-18582. skip unnecessary cleanup logic in distcp (#5251)
> e8a6b2c2c4e HADOOP-18582. Addendum: Skip unnecessary cleanup logic in DistCp.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5397: HDFS-16917 Add transfer rate quantile metrics for DataNode reads
hadoop-yetus commented on PR #5397:
URL: https://github.com/apache/hadoop/pull/5397#issuecomment-1441210715
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 1s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 3s | | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 30m 43s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/12/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 13m 29s |
[/branch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/12/artifact/out/branch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| -1 :x: | compile | 12m 4s |
[/branch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/12/artifact/out/branch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
| root in trunk failed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08.
|
| +1 :green_heart: | checkstyle | 3m 35s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 2s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 0s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 2m 11s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 5m 42s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 42s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 34s | | the patch passed |
| -1 :x: | compile | 13m 17s |
[/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/12/artifact/out/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| -1 :x: | javac | 13m 17s |
[/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/12/artifact/out/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| -1 :x: | compile | 11m 54s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/12/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
| root in the patch failed with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08. |
| -1 :x: | javac | 11m 54s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/12/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
| root in the patch failed with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08. |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/12/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 4m 46s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5397/12/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 205 unchanged - 0 fixed = 206 total (was
205) |
| +1 :green_heart: | mvnsite | 2m 59s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 50s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 2m 11s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 5m
[jira] [Commented] (HADOOP-18399) SingleFilePerBlockCache to use LocalDirAllocator for file allocation
[ https://issues.apache.org/jira/browse/HADOOP-18399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692484#comment-17692484 ] ASF GitHub Bot commented on HADOOP-18399: - virajjasani commented on PR #5054: URL: https://github.com/apache/hadoop/pull/5054#issuecomment-1441203078 > -1 ❌ mvninstall 33m 29s [/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5054/18/artifact/out/branch-mvninstall-root.txt) This has been fixed by https://github.com/apache/hadoop/commit/8025a60ae79382b8885aa9619fed9fa2cd1b62b8 > SingleFilePerBlockCache to use LocalDirAllocator for file allocation > > > Key: HADOOP-18399 > URL: https://issues.apache.org/jira/browse/HADOOP-18399 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > prefetching stream's SingleFilePerBlockCache uses Files.tempFile() to > allocate a temp file. > it should be using LocalDirAllocator to allocate space from a list of dirs, > taking a config key to use. for s3a we will use the Constants.BUFFER_DIR > option, which on yarn deployments is fixed under the env.LOCAL_DIR path, so > automatically cleaned up on container exit -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5054: HADOOP-18399 Prefetch - SingleFilePerBlockCache to use LocalDirAllocator for file allocation
virajjasani commented on PR #5054: URL: https://github.com/apache/hadoop/pull/5054#issuecomment-1441203078 > -1 ❌ mvninstall 33m 29s [/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5054/18/artifact/out/branch-mvninstall-root.txt) This has been fixed by https://github.com/apache/hadoop/commit/8025a60ae79382b8885aa9619fed9fa2cd1b62b8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18399) SingleFilePerBlockCache to use LocalDirAllocator for file allocation
[
https://issues.apache.org/jira/browse/HADOOP-18399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692434#comment-17692434
]
ASF GitHub Bot commented on HADOOP-18399:
-
hadoop-yetus commented on PR #5054:
URL: https://github.com/apache/hadoop/pull/5054#issuecomment-1441113957
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 40s | | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 33m 29s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5054/18/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 14m 58s |
[/branch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5054/18/artifact/out/branch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| -1 :x: | compile | 12m 54s |
[/branch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5054/18/artifact/out/branch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
| root in trunk failed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08.
|
| +1 :green_heart: | checkstyle | 3m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 11s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 20s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 2s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 33s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 22s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 3s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 53s | | the patch passed |
| -1 :x: | compile | 14m 42s |
[/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5054/18/artifact/out/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| -1 :x: | javac | 14m 42s |
[/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5054/18/artifact/out/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| -1 :x: | compile | 12m 44s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5054/18/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
| root in the patch failed with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08. |
| -1 :x: | javac | 12m 44s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5054/18/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
| root in the patch failed with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 47s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 4s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 6s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 25s | | patch has no errors
when building and testing our client artifacts. |
[GitHub] [hadoop] hadoop-yetus commented on pull request #5054: HADOOP-18399 Prefetch - SingleFilePerBlockCache to use LocalDirAllocator for file allocation
hadoop-yetus commented on PR #5054:
URL: https://github.com/apache/hadoop/pull/5054#issuecomment-1441113957
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 40s | | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 33m 29s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5054/18/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 14m 58s |
[/branch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5054/18/artifact/out/branch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| -1 :x: | compile | 12m 54s |
[/branch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5054/18/artifact/out/branch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
| root in trunk failed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08.
|
| +1 :green_heart: | checkstyle | 3m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 11s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 20s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 2s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 33s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 22s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 3s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 53s | | the patch passed |
| -1 :x: | compile | 14m 42s |
[/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5054/18/artifact/out/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| -1 :x: | javac | 14m 42s |
[/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5054/18/artifact/out/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| -1 :x: | compile | 12m 44s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5054/18/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
| root in the patch failed with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08. |
| -1 :x: | javac | 12m 44s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5054/18/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
| root in the patch failed with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 47s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 4s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 6s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 25s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 3s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 27s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 34s |
[jira] [Commented] (HADOOP-18631) Migrate Async appenders to log4j properties
[
https://issues.apache.org/jira/browse/HADOOP-18631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692427#comment-17692427
]
ASF GitHub Bot commented on HADOOP-18631:
-
hadoop-yetus commented on PR #5418:
URL: https://github.com/apache/hadoop/pull/5418#issuecomment-1441096436
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 8 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 48m 36s | | trunk passed |
| +1 :green_heart: | compile | 1m 32s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 29m 34s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 35s | | the patch passed |
| +1 :green_heart: | compile | 1m 28s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 28s | | the patch passed |
| +1 :green_heart: | compile | 1m 20s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 54s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 486 unchanged - 21
fixed = 486 total (was 507) |
| +1 :green_heart: | mvnsite | 1m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 27s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 32s | | the patch passed |
| +1 :green_heart: | shadedclient | 30m 35s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 235m 30s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5418/10/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 368m 13s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5418/10/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5418 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile
javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux f35955930a54 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6c08b124dfbb7ec7c95e39dabaa66e4795d57eec |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5418/10/testReport/ |
| Max. process+thread count | 2422 (vs. ulimit of 5500) |
|
[GitHub] [hadoop] hadoop-yetus commented on pull request #5418: HADOOP-18631. Migrate Async appenders to log4j properties
hadoop-yetus commented on PR #5418:
URL: https://github.com/apache/hadoop/pull/5418#issuecomment-1441096436
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 8 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 48m 36s | | trunk passed |
| +1 :green_heart: | compile | 1m 32s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 29m 34s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 35s | | the patch passed |
| +1 :green_heart: | compile | 1m 28s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 28s | | the patch passed |
| +1 :green_heart: | compile | 1m 20s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 54s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 486 unchanged - 21
fixed = 486 total (was 507) |
| +1 :green_heart: | mvnsite | 1m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 27s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 32s | | the patch passed |
| +1 :green_heart: | shadedclient | 30m 35s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 235m 30s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5418/10/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 368m 13s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5418/10/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5418 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile
javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux f35955930a54 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6c08b124dfbb7ec7c95e39dabaa66e4795d57eec |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5418/10/testReport/ |
| Max. process+thread count | 2422 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5418/10/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
|
[jira] [Commented] (HADOOP-18631) Migrate Async appenders to log4j properties
[
https://issues.apache.org/jira/browse/HADOOP-18631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692426#comment-17692426
]
ASF GitHub Bot commented on HADOOP-18631:
-
hadoop-yetus commented on PR #5418:
URL: https://github.com/apache/hadoop/pull/5418#issuecomment-1441085608
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 8 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 52s | | trunk passed |
| +1 :green_heart: | compile | 1m 39s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 29m 21s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 31s | | the patch passed |
| +1 :green_heart: | compile | 1m 26s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 26s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 56s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 486 unchanged - 21
fixed = 486 total (was 507) |
| +1 :green_heart: | mvnsite | 1m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 26s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 30m 10s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 239m 13s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not
generate ASF License warnings. |
| | | 370m 29s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5418/9/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5418 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile
javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 02d1ab22128a 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f9799a92963a8a61e22568aa722bb927b1e15a91 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5418/9/testReport/ |
| Max. process+thread count | 2117 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5418/9/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0
[GitHub] [hadoop] hadoop-yetus commented on pull request #5418: HADOOP-18631. Migrate Async appenders to log4j properties
hadoop-yetus commented on PR #5418:
URL: https://github.com/apache/hadoop/pull/5418#issuecomment-1441085608
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 8 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 52s | | trunk passed |
| +1 :green_heart: | compile | 1m 39s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 29m 21s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 31s | | the patch passed |
| +1 :green_heart: | compile | 1m 26s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 26s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 56s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 486 unchanged - 21
fixed = 486 total (was 507) |
| +1 :green_heart: | mvnsite | 1m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 26s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 30m 10s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 239m 13s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not
generate ASF License warnings. |
| | | 370m 29s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5418/9/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5418 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile
javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 02d1ab22128a 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f9799a92963a8a61e22568aa722bb927b1e15a91 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5418/9/testReport/ |
| Max. process+thread count | 2117 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5418/9/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
[GitHub] [hadoop] slfan1989 commented on pull request #5335: YARN-11426. Improve YARN NodeLabel Memory Display.
slfan1989 commented on PR #5335: URL: https://github.com/apache/hadoop/pull/5335#issuecomment-1441019375 @ayushtkn Can you help review this pr? Thank you very much! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #5394: YARN-5604. [Federation] Add versioning for FederationStateStore.
slfan1989 commented on PR #5394: URL: https://github.com/apache/hadoop/pull/5394#issuecomment-1441018320 @goiri Can you help review this pr again? Thank you very much! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #5126: YARN-11370. [Federation] Refactor MemoryFederationStateStore code.
slfan1989 commented on PR #5126: URL: https://github.com/apache/hadoop/pull/5126#issuecomment-1441017546 @goiri Thank you very much for your help in reviewing the code! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] rdingankar commented on a diff in pull request #5397: HDFS-16917 Add transfer rate quantile metrics for DataNode reads
rdingankar commented on code in PR #5397: URL: https://github.com/apache/hadoop/pull/5397#discussion_r1115082559 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java: ## @@ -1122,6 +1124,7 @@ public void copyBlock(final ExtendedBlock block, datanode.metrics.incrBytesRead((int) read); datanode.metrics.incrBlocksRead(); datanode.metrics.incrTotalReadTime(duration); + datanode.metrics.addReadTransferRateMBs(DFSUtil.transferRateMBs(read, duration)); Review Comment: updated to log for cases when we get unexpected data. 1) Bytes read is negative 2) Duration is negative or 0. Since we are logging these cases not emitting the data to metrics. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18399) SingleFilePerBlockCache to use LocalDirAllocator for file allocation
[ https://issues.apache.org/jira/browse/HADOOP-18399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692393#comment-17692393 ] ASF GitHub Bot commented on HADOOP-18399: - virajjasani commented on PR #5054: URL: https://github.com/apache/hadoop/pull/5054#issuecomment-1440881795 Ran the tests with `scale` and `prefetch` profiles against `us-west-2`: ``` $mvn -Dparallel-tests -DtestsThreadCount=8 -Dprefetch -Dscale clean verify ``` Test results look good > SingleFilePerBlockCache to use LocalDirAllocator for file allocation > > > Key: HADOOP-18399 > URL: https://issues.apache.org/jira/browse/HADOOP-18399 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > prefetching stream's SingleFilePerBlockCache uses Files.tempFile() to > allocate a temp file. > it should be using LocalDirAllocator to allocate space from a list of dirs, > taking a config key to use. for s3a we will use the Constants.BUFFER_DIR > option, which on yarn deployments is fixed under the env.LOCAL_DIR path, so > automatically cleaned up on container exit -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5054: HADOOP-18399 Prefetch - SingleFilePerBlockCache to use LocalDirAllocator for file allocation
virajjasani commented on PR #5054: URL: https://github.com/apache/hadoop/pull/5054#issuecomment-1440881795 Ran the tests with `scale` and `prefetch` profiles against `us-west-2`: ``` $mvn -Dparallel-tests -DtestsThreadCount=8 -Dprefetch -Dscale clean verify ``` Test results look good -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18399) SingleFilePerBlockCache to use LocalDirAllocator for file allocation
[
https://issues.apache.org/jira/browse/HADOOP-18399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692390#comment-17692390
]
ASF GitHub Bot commented on HADOOP-18399:
-
virajjasani commented on code in PR #5054:
URL: https://github.com/apache/hadoop/pull/5054#discussion_r1115017372
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/SingleFilePerBlockCache.java:
##
@@ -176,7 +179,8 @@ private Entry getEntry(int blockNumber) {
* @throws IllegalArgumentException if buffer.limit() is zero or negative.
*/
@Override
- public void put(int blockNumber, ByteBuffer buffer) throws IOException {
+ public void put(int blockNumber, ByteBuffer buffer, Configuration conf,
Review Comment:
done
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/SingleFilePerBlockCache.java:
##
@@ -340,15 +347,17 @@ public static boolean isCacheSpaceAvailable(long
fileSize) {
private static final String BINARY_FILE_SUFFIX = ".bin";
// File attributes attached to any intermediate temporary file created
during index creation.
- private static final FileAttribute> TEMP_FILE_ATTRS
=
-
PosixFilePermissions.asFileAttribute(EnumSet.of(PosixFilePermission.OWNER_READ,
- PosixFilePermission.OWNER_WRITE));
-
- private static Path getTempFilePath() throws IOException {
-return Files.createTempFile(
-CACHE_FILE_PREFIX,
-BINARY_FILE_SUFFIX,
-TEMP_FILE_ATTRS
-);
+ private static final Set TEMP_FILE_ATTRS =
+ ImmutableSet.of(PosixFilePermission.OWNER_READ,
PosixFilePermission.OWNER_WRITE);
+
+ private static Path getTempFilePath(final Configuration conf,
Review Comment:
done
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingCacheFiles.java:
##
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.io.File;
+import java.net.URI;
+
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.s3a.performance.AbstractS3ACostTest;
+
+import static org.apache.hadoop.fs.s3a.Constants.PREFETCH_BLOCK_DEFAULT_SIZE;
+import static org.apache.hadoop.fs.s3a.Constants.PREFETCH_BLOCK_SIZE_KEY;
+import static org.apache.hadoop.fs.s3a.Constants.PREFETCH_ENABLED_KEY;
+import static org.apache.hadoop.io.IOUtils.cleanupWithLogger;
+
+/**
+ * Test the cache file behaviour with prefetching input stream.
+ */
+public class ITestS3APrefetchingCacheFiles extends AbstractS3ACostTest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(ITestS3APrefetchingInputStream.class);
+
+ private Path testFile;
+ private FileSystem fs;
+ private int prefetchBlockSize;
+
+ public ITestS3APrefetchingCacheFiles() {
+super(true);
+ }
+
+ @Before
+ public void setUp() throws Exception {
+super.setup();
+Configuration conf = getConfiguration();
+String testFileUri = S3ATestUtils.getCSVTestFile(conf);
+
+testFile = new Path(testFileUri);
+prefetchBlockSize = conf.getInt(PREFETCH_BLOCK_SIZE_KEY,
PREFETCH_BLOCK_DEFAULT_SIZE);
+fs = new S3AFileSystem();
+fs.initialize(new URI(testFileUri), getConfiguration());
+ }
+
+ @Override
+ public Configuration createConfiguration() {
+Configuration conf = super.createConfiguration();
+S3ATestUtils.removeBaseAndBucketOverrides(conf, PREFETCH_ENABLED_KEY);
+conf.setBoolean(PREFETCH_ENABLED_KEY, true);
+return conf;
+ }
+
+ @Override
+ public synchronized void teardown() throws Exception {
+super.teardown();
+File tmpFileDir = new File(new File("target", "build"), "test");
+File[] tmpFiles = tmpFileDir.listFiles();
+if (tmpFiles != null) {
+ for (File filePath : tmpFiles) {
+String path = filePath.getPath();
+if (path.endsWith(".bin") &&
[GitHub] [hadoop] virajjasani commented on a diff in pull request #5054: HADOOP-18399 Prefetch - SingleFilePerBlockCache to use LocalDirAllocator for file allocation
virajjasani commented on code in PR #5054:
URL: https://github.com/apache/hadoop/pull/5054#discussion_r1115017372
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/SingleFilePerBlockCache.java:
##
@@ -176,7 +179,8 @@ private Entry getEntry(int blockNumber) {
* @throws IllegalArgumentException if buffer.limit() is zero or negative.
*/
@Override
- public void put(int blockNumber, ByteBuffer buffer) throws IOException {
+ public void put(int blockNumber, ByteBuffer buffer, Configuration conf,
Review Comment:
done
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/SingleFilePerBlockCache.java:
##
@@ -340,15 +347,17 @@ public static boolean isCacheSpaceAvailable(long
fileSize) {
private static final String BINARY_FILE_SUFFIX = ".bin";
// File attributes attached to any intermediate temporary file created
during index creation.
- private static final FileAttribute> TEMP_FILE_ATTRS
=
-
PosixFilePermissions.asFileAttribute(EnumSet.of(PosixFilePermission.OWNER_READ,
- PosixFilePermission.OWNER_WRITE));
-
- private static Path getTempFilePath() throws IOException {
-return Files.createTempFile(
-CACHE_FILE_PREFIX,
-BINARY_FILE_SUFFIX,
-TEMP_FILE_ATTRS
-);
+ private static final Set TEMP_FILE_ATTRS =
+ ImmutableSet.of(PosixFilePermission.OWNER_READ,
PosixFilePermission.OWNER_WRITE);
+
+ private static Path getTempFilePath(final Configuration conf,
Review Comment:
done
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingCacheFiles.java:
##
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.io.File;
+import java.net.URI;
+
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.s3a.performance.AbstractS3ACostTest;
+
+import static org.apache.hadoop.fs.s3a.Constants.PREFETCH_BLOCK_DEFAULT_SIZE;
+import static org.apache.hadoop.fs.s3a.Constants.PREFETCH_BLOCK_SIZE_KEY;
+import static org.apache.hadoop.fs.s3a.Constants.PREFETCH_ENABLED_KEY;
+import static org.apache.hadoop.io.IOUtils.cleanupWithLogger;
+
+/**
+ * Test the cache file behaviour with prefetching input stream.
+ */
+public class ITestS3APrefetchingCacheFiles extends AbstractS3ACostTest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(ITestS3APrefetchingInputStream.class);
+
+ private Path testFile;
+ private FileSystem fs;
+ private int prefetchBlockSize;
+
+ public ITestS3APrefetchingCacheFiles() {
+super(true);
+ }
+
+ @Before
+ public void setUp() throws Exception {
+super.setup();
+Configuration conf = getConfiguration();
+String testFileUri = S3ATestUtils.getCSVTestFile(conf);
+
+testFile = new Path(testFileUri);
+prefetchBlockSize = conf.getInt(PREFETCH_BLOCK_SIZE_KEY,
PREFETCH_BLOCK_DEFAULT_SIZE);
+fs = new S3AFileSystem();
+fs.initialize(new URI(testFileUri), getConfiguration());
+ }
+
+ @Override
+ public Configuration createConfiguration() {
+Configuration conf = super.createConfiguration();
+S3ATestUtils.removeBaseAndBucketOverrides(conf, PREFETCH_ENABLED_KEY);
+conf.setBoolean(PREFETCH_ENABLED_KEY, true);
+return conf;
+ }
+
+ @Override
+ public synchronized void teardown() throws Exception {
+super.teardown();
+File tmpFileDir = new File(new File("target", "build"), "test");
+File[] tmpFiles = tmpFileDir.listFiles();
+if (tmpFiles != null) {
+ for (File filePath : tmpFiles) {
+String path = filePath.getPath();
+if (path.endsWith(".bin") && path.contains("fs-cache-")) {
+ filePath.delete();
+}
+ }
+}
+cleanupWithLogger(LOG, fs);
+fs = null;
+testFile = null;
+ }
+
+ @Test
+ public void testCacheFileExistence() throws Throwable {
Review Comment:
done
--
[jira] [Commented] (HADOOP-18399) SingleFilePerBlockCache to use LocalDirAllocator for file allocation
[
https://issues.apache.org/jira/browse/HADOOP-18399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692389#comment-17692389
]
ASF GitHub Bot commented on HADOOP-18399:
-
virajjasani commented on code in PR #5054:
URL: https://github.com/apache/hadoop/pull/5054#discussion_r1115016369
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingCacheFiles.java:
##
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.io.File;
+import java.net.URI;
+
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.s3a.performance.AbstractS3ACostTest;
+
+import static org.apache.hadoop.fs.s3a.Constants.PREFETCH_BLOCK_DEFAULT_SIZE;
+import static org.apache.hadoop.fs.s3a.Constants.PREFETCH_BLOCK_SIZE_KEY;
+import static org.apache.hadoop.fs.s3a.Constants.PREFETCH_ENABLED_KEY;
+import static org.apache.hadoop.io.IOUtils.cleanupWithLogger;
+
+/**
+ * Test the cache file behaviour with prefetching input stream.
+ */
+public class ITestS3APrefetchingCacheFiles extends AbstractS3ACostTest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(ITestS3APrefetchingInputStream.class);
+
+ private Path testFile;
+ private FileSystem fs;
+ private int prefetchBlockSize;
+
+ public ITestS3APrefetchingCacheFiles() {
+super(true);
+ }
+
+ @Before
+ public void setUp() throws Exception {
+super.setup();
+Configuration conf = getConfiguration();
+String testFileUri = S3ATestUtils.getCSVTestFile(conf);
+
+testFile = new Path(testFileUri);
+prefetchBlockSize = conf.getInt(PREFETCH_BLOCK_SIZE_KEY,
PREFETCH_BLOCK_DEFAULT_SIZE);
+fs = new S3AFileSystem();
+fs.initialize(new URI(testFileUri), getConfiguration());
+ }
+
+ @Override
+ public Configuration createConfiguration() {
+Configuration conf = super.createConfiguration();
+S3ATestUtils.removeBaseAndBucketOverrides(conf, PREFETCH_ENABLED_KEY);
+conf.setBoolean(PREFETCH_ENABLED_KEY, true);
+return conf;
+ }
+
+ @Override
+ public synchronized void teardown() throws Exception {
+super.teardown();
+File tmpFileDir = new File(new File("target", "build"), "test");
+File[] tmpFiles = tmpFileDir.listFiles();
+if (tmpFiles != null) {
+ for (File filePath : tmpFiles) {
+String path = filePath.getPath();
+if (path.endsWith(".bin") && path.contains("fs-cache-")) {
+ filePath.delete();
+}
+ }
+}
+cleanupWithLogger(LOG, fs);
+fs = null;
+testFile = null;
+ }
+
+ @Test
+ public void testCacheFileExistence() throws Throwable {
+describe("Verify that FS cache files exist on local FS");
+
+try (FSDataInputStream in = fs.open(testFile)) {
+ byte[] buffer = new byte[prefetchBlockSize];
+
+ in.read(buffer, 0, prefetchBlockSize - 10240);
+ in.seek(prefetchBlockSize * 2);
+ in.read(buffer, 0, prefetchBlockSize);
+
+ File tmpFileDir = new File(new File("target", "build"), "test");
+ assertTrue("The dir to keep cache files must exist",
tmpFileDir.exists());
+ boolean isCacheFileForBlockFound = false;
+ File[] tmpFiles = tmpFileDir.listFiles();
+ if (tmpFiles != null) {
+for (File filePath : tmpFiles) {
+ String path = filePath.getPath();
+ if (path.endsWith(".bin") && path.contains("fs-cache-")) {
+isCacheFileForBlockFound = true;
+break;
+ }
+}
+ } else {
+LOG.warn("No cache files found under " + tmpFileDir);
+ }
+ assertTrue("File to cache block data must exist",
isCacheFileForBlockFound);
Review Comment:
Comparing the whole byte[] might not be feasible. Rather I added bunch of
stats comparison. WDYT?
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingCacheFiles.java:
[GitHub] [hadoop] virajjasani commented on a diff in pull request #5054: HADOOP-18399 Prefetch - SingleFilePerBlockCache to use LocalDirAllocator for file allocation
virajjasani commented on code in PR #5054:
URL: https://github.com/apache/hadoop/pull/5054#discussion_r1115016369
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingCacheFiles.java:
##
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.io.File;
+import java.net.URI;
+
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.s3a.performance.AbstractS3ACostTest;
+
+import static org.apache.hadoop.fs.s3a.Constants.PREFETCH_BLOCK_DEFAULT_SIZE;
+import static org.apache.hadoop.fs.s3a.Constants.PREFETCH_BLOCK_SIZE_KEY;
+import static org.apache.hadoop.fs.s3a.Constants.PREFETCH_ENABLED_KEY;
+import static org.apache.hadoop.io.IOUtils.cleanupWithLogger;
+
+/**
+ * Test the cache file behaviour with prefetching input stream.
+ */
+public class ITestS3APrefetchingCacheFiles extends AbstractS3ACostTest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(ITestS3APrefetchingInputStream.class);
+
+ private Path testFile;
+ private FileSystem fs;
+ private int prefetchBlockSize;
+
+ public ITestS3APrefetchingCacheFiles() {
+super(true);
+ }
+
+ @Before
+ public void setUp() throws Exception {
+super.setup();
+Configuration conf = getConfiguration();
+String testFileUri = S3ATestUtils.getCSVTestFile(conf);
+
+testFile = new Path(testFileUri);
+prefetchBlockSize = conf.getInt(PREFETCH_BLOCK_SIZE_KEY,
PREFETCH_BLOCK_DEFAULT_SIZE);
+fs = new S3AFileSystem();
+fs.initialize(new URI(testFileUri), getConfiguration());
+ }
+
+ @Override
+ public Configuration createConfiguration() {
+Configuration conf = super.createConfiguration();
+S3ATestUtils.removeBaseAndBucketOverrides(conf, PREFETCH_ENABLED_KEY);
+conf.setBoolean(PREFETCH_ENABLED_KEY, true);
+return conf;
+ }
+
+ @Override
+ public synchronized void teardown() throws Exception {
+super.teardown();
+File tmpFileDir = new File(new File("target", "build"), "test");
+File[] tmpFiles = tmpFileDir.listFiles();
+if (tmpFiles != null) {
+ for (File filePath : tmpFiles) {
+String path = filePath.getPath();
+if (path.endsWith(".bin") && path.contains("fs-cache-")) {
+ filePath.delete();
+}
+ }
+}
+cleanupWithLogger(LOG, fs);
+fs = null;
+testFile = null;
+ }
+
+ @Test
+ public void testCacheFileExistence() throws Throwable {
+describe("Verify that FS cache files exist on local FS");
+
+try (FSDataInputStream in = fs.open(testFile)) {
+ byte[] buffer = new byte[prefetchBlockSize];
+
+ in.read(buffer, 0, prefetchBlockSize - 10240);
+ in.seek(prefetchBlockSize * 2);
+ in.read(buffer, 0, prefetchBlockSize);
+
+ File tmpFileDir = new File(new File("target", "build"), "test");
+ assertTrue("The dir to keep cache files must exist",
tmpFileDir.exists());
+ boolean isCacheFileForBlockFound = false;
+ File[] tmpFiles = tmpFileDir.listFiles();
+ if (tmpFiles != null) {
+for (File filePath : tmpFiles) {
+ String path = filePath.getPath();
+ if (path.endsWith(".bin") && path.contains("fs-cache-")) {
+isCacheFileForBlockFound = true;
+break;
+ }
+}
+ } else {
+LOG.warn("No cache files found under " + tmpFileDir);
+ }
+ assertTrue("File to cache block data must exist",
isCacheFileForBlockFound);
Review Comment:
Comparing the whole byte[] might not be feasible. Rather I added bunch of
stats comparison. WDYT?
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingCacheFiles.java:
##
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright
[GitHub] [hadoop] rdingankar commented on a diff in pull request #5397: HDFS-16917 Add transfer rate quantile metrics for DataNode reads
rdingankar commented on code in PR #5397:
URL: https://github.com/apache/hadoop/pull/5397#discussion_r1115011334
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java:
##
@@ -1936,4 +1936,20 @@ public static boolean isParentEntry(final String path,
final String parent) {
return path.charAt(parent.length()) == Path.SEPARATOR_CHAR
|| parent.equals(Path.SEPARATOR);
}
+
+ /**
+ * Calculate the transfer rate in megabytes/second. Return -1 for any
negative input.
+ * @param bytes bytes
+ * @param durationMS duration in milliseconds
+ * @return the number of megabytes/second of the transfer rate
+ */
+ public static long transferRateMBs(long bytes, long durationMS) {
Review Comment:
Added unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] rdingankar commented on a diff in pull request #5397: HDFS-16917 Add transfer rate quantile metrics for DataNode reads
rdingankar commented on code in PR #5397: URL: https://github.com/apache/hadoop/pull/5397#discussion_r1114847865 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java: ## @@ -1122,6 +1124,7 @@ public void copyBlock(final ExtendedBlock block, datanode.metrics.incrBytesRead((int) read); datanode.metrics.incrBlocksRead(); datanode.metrics.incrTotalReadTime(duration); + datanode.metrics.addReadTransferRateMBs(DFSUtil.transferRateMBs(read, duration)); Review Comment: I was thinking we can publish -1 for the error scenarios which can be visualized from the graphs as well. For duration of 0, changing it to 1 will give us the number of bytes if transferred. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] omalley merged pull request #5346: HDFS-16901: RBF: Propagates real user's username via the caller context, when a proxy user is being used.
omalley merged PR #5346: URL: https://github.com/apache/hadoop/pull/5346 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5426: MAPREDUCE-7434. Fix ShuffleHandler tests.
hadoop-yetus commented on PR #5426:
URL: https://github.com/apache/hadoop/pull/5426#issuecomment-1440849204
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 5s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 38s | | trunk passed |
| +1 :green_heart: | compile | 0m 26s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 23s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 21s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 51s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 50s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | | the patch passed |
| +1 :green_heart: | compile | 0m 19s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 12s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 16s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 45s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 28s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 36s | | hadoop-mapreduce-client-shuffle
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 113m 21s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5426/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5426 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux be97f2be4055 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2bee18271a72fec5e7d309fb17096cd5f18e23c2 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5426/3/testReport/ |
| Max. process+thread count | 536 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5426/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
[GitHub] [hadoop] goiri merged pull request #5126: YARN-11370. [Federation] Refactor MemoryFederationStateStore code.
goiri merged PR #5126: URL: https://github.com/apache/hadoop/pull/5126 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5427: Bump derby from 10.10.2.0 to 10.14.2.0 in /hadoop-project
hadoop-yetus commented on PR #5427:
URL: https://github.com/apache/hadoop/pull/5427#issuecomment-1440756424
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 53s | | trunk passed |
| +1 :green_heart: | compile | 0m 18s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 17s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | mvnsite | 0m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 19s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 75m 9s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 14s | | the patch passed |
| +1 :green_heart: | compile | 0m 12s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 12s | | the patch passed |
| +1 :green_heart: | compile | 0m 11s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 11s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 12s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 12s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 26m 49s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 14s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 106m 45s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5427/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5427 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint |
| uname | Linux e5b907ad1be0 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b7a69398739b038bd14107ad2eebf533497e4ea8 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5427/1/testReport/ |
| Max. process+thread count | 542 (vs. ulimit of 5500) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5427/1/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional
[GitHub] [hadoop] hadoop-yetus commented on pull request #5424: HDFS-16931. Observer nn delete blocks asynchronously when tail OP_DEL…
hadoop-yetus commented on PR #5424:
URL: https://github.com/apache/hadoop/pull/5424#issuecomment-1440749446
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 2s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 2s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 52m 12s | | trunk passed |
| +1 :green_heart: | compile | 1m 38s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 10s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 12s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 34s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 42s | | trunk passed |
| +1 :green_heart: | shadedclient | 29m 47s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 36s | | the patch passed |
| +1 :green_heart: | compile | 1m 30s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 30s | | the patch passed |
| +1 :green_heart: | compile | 1m 21s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 21s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 55s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5424/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 7 new + 118 unchanged
- 0 fixed = 125 total (was 118) |
| +1 :green_heart: | mvnsite | 1m 26s | | the patch passed |
| -1 :x: | javadoc | 0m 58s |
[/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5424/1/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 1m 28s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 45s | | the patch passed |
| +1 :green_heart: | shadedclient | 30m 17s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 269m 3s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5424/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not
generate ASF License warnings. |
| | | 406m 21s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.namenode.TestNameEditsConfigs |
| | hadoop.fs.TestHDFSFileContextMainOperations |
| | hadoop.hdfs.TestAclsEndToEnd |
| | hadoop.hdfs.server.namenode.snapshot.TestRandomOpsWithSnapshots |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshot |
| | hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5424/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5424 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 75cbd5d6c85e 4.15.0-200-generic #211-Ubuntu
[jira] [Updated] (HADOOP-18582) No need to clean tmp files in distcp direct mode
[
https://issues.apache.org/jira/browse/HADOOP-18582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-18582:
Description:
it not necessary to do `cleanupTempFiles` while ditcp commit job in direct
mode, because it there is no temp files in direct mode.
This clean operation will increase the task execution time, because it will get
the list of files in the target path. When the number of files in the target
path is very large, this operation will be very slow.
*note* there are two patches which need to be cherrypicked when picking this
up; the original patch and a followup, both with HADOOP-18582 in the title
{code}
3b7b79b37ae HADOOP-18582. skip unnecessary cleanup logic in distcp (#5251)
e8a6b2c2c4e HADOOP-18582. Addendum: Skip unnecessary cleanup logic in DistCp.
{code}
was:
it not necessary to do `cleanupTempFiles` while ditcp commit job in direct
mode, because it there is no temp files in direct mode.
This clean operation will increase the task execution time, because it will get
the list of files in the target path. When the number of files in the target
path is very large, this operation will be very slow.
> No need to clean tmp files in distcp direct mode
>
>
> Key: HADOOP-18582
> URL: https://issues.apache.org/jira/browse/HADOOP-18582
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.3.4
>Reporter: 1kang
>Assignee: 1kang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
>
> it not necessary to do `cleanupTempFiles` while ditcp commit job in direct
> mode, because it there is no temp files in direct mode.
> This clean operation will increase the task execution time, because it will
> get the list of files in the target path. When the number of files in the
> target path is very large, this operation will be very slow.
> *note* there are two patches which need to be cherrypicked when picking this
> up; the original patch and a followup, both with HADOOP-18582 in the title
> {code}
> 3b7b79b37ae HADOOP-18582. skip unnecessary cleanup logic in distcp (#5251)
> e8a6b2c2c4e HADOOP-18582. Addendum: Skip unnecessary cleanup logic in DistCp.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18582) No need to clean tmp files in distcp direct mode
[ https://issues.apache.org/jira/browse/HADOOP-18582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692338#comment-17692338 ] ASF GitHub Bot commented on HADOOP-18582: - steveloughran commented on PR #5409: URL: https://github.com/apache/hadoop/pull/5409#issuecomment-1440684847 merged; will note on jira that both patches are needed > No need to clean tmp files in distcp direct mode > > > Key: HADOOP-18582 > URL: https://issues.apache.org/jira/browse/HADOOP-18582 > Project: Hadoop Common > Issue Type: Bug > Components: tools/distcp >Affects Versions: 3.3.4 >Reporter: 1kang >Assignee: 1kang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > it not necessary to do `cleanupTempFiles` while ditcp commit job in direct > mode, because it there is no temp files in direct mode. > This clean operation will increase the task execution time, because it will > get the list of files in the target path. When the number of files in the > target path is very large, this operation will be very slow. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5409: HADOOP-18582. Addendum: Skip unnecessary cleanup logic in DistCp.
steveloughran commented on PR #5409: URL: https://github.com/apache/hadoop/pull/5409#issuecomment-1440684847 merged; will note on jira that both patches are needed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mkuchenbecker commented on a diff in pull request #5322: HDFS-16896 clear ignoredNodes list when we clear deadnode list on ref…
mkuchenbecker commented on code in PR #5322:
URL: https://github.com/apache/hadoop/pull/5322#discussion_r1114855356
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java:
##
@@ -197,6 +197,15 @@ private void clearLocalDeadNodes() {
deadNodes.clear();
}
+ /**
+ * Clear list of ignored nodes used for hedged reads.
+ */
+ private void clearIgnoredNodes(Collection ignoredNodes) {
Review Comment:
Nit. Param documentation.
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java:
##
@@ -1337,8 +1352,12 @@ private void hedgedFetchBlockByteRange(LocatedBlock
block, long start,
} catch (InterruptedException ie) {
// Ignore and retry
}
-if (refetch) {
- refetchLocations(block, ignored);
+// If refetch is true, then all nodes are in deadNodes or ignoredNodes.
+// We should loop through all futures and remove them, so we do not
+// have concurrent requests to the same node.
+// Once all futures are cleared, we can clear the ignoredNodes and
retry.
Review Comment:
Ignored and dead nodes are cleared, correct?
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java:
##
@@ -224,7 +233,7 @@ boolean deadNodesContain(DatanodeInfo nodeInfo) {
}
/**
- * Grab the open-file info from namenode
+ * Grab the open-file info from namenode.
Review Comment:
Is this change needed?
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java:
##
@@ -197,6 +197,15 @@ private void clearLocalDeadNodes() {
deadNodes.clear();
}
+ /**
+ * Clear list of ignored nodes used for hedged reads.
+ */
+ private void clearIgnoredNodes(Collection ignoredNodes) {
Review Comment:
Does it make more sense to clean up ignored / dead in the same function so
we don't need to worry about calling both?
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestPread.java:
##
@@ -603,7 +603,9 @@ public Void answer(InvocationOnMock invocation) throws
Throwable {
input.read(0, buffer, 0, 1024);
Assert.fail("Reading the block should have thrown
BlockMissingException");
} catch (BlockMissingException e) {
- assertEquals(3, input.getHedgedReadOpsLoopNumForTesting());
+ // The result of 9 is due to 2 blocks by 4 iterations plus one because
+ // hedgedReadOpsLoopNumForTesting is incremented at start of the loop.
+ assertEquals(9, input.getHedgedReadOpsLoopNumForTesting());
Review Comment:
We are tripling the IO per hedged request?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18582) No need to clean tmp files in distcp direct mode
[ https://issues.apache.org/jira/browse/HADOOP-18582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692336#comment-17692336 ] ASF GitHub Bot commented on HADOOP-18582: - steveloughran commented on PR #5409: URL: https://github.com/apache/hadoop/pull/5409#issuecomment-1440680482 thanks! > No need to clean tmp files in distcp direct mode > > > Key: HADOOP-18582 > URL: https://issues.apache.org/jira/browse/HADOOP-18582 > Project: Hadoop Common > Issue Type: Bug > Components: tools/distcp >Affects Versions: 3.3.4 >Reporter: 1kang >Assignee: 1kang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > it not necessary to do `cleanupTempFiles` while ditcp commit job in direct > mode, because it there is no temp files in direct mode. > This clean operation will increase the task execution time, because it will > get the list of files in the target path. When the number of files in the > target path is very large, this operation will be very slow. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18582) No need to clean tmp files in distcp direct mode
[ https://issues.apache.org/jira/browse/HADOOP-18582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692335#comment-17692335 ] ASF GitHub Bot commented on HADOOP-18582: - steveloughran merged PR #5409: URL: https://github.com/apache/hadoop/pull/5409 > No need to clean tmp files in distcp direct mode > > > Key: HADOOP-18582 > URL: https://issues.apache.org/jira/browse/HADOOP-18582 > Project: Hadoop Common > Issue Type: Bug > Components: tools/distcp >Affects Versions: 3.3.4 >Reporter: 1kang >Assignee: 1kang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > it not necessary to do `cleanupTempFiles` while ditcp commit job in direct > mode, because it there is no temp files in direct mode. > This clean operation will increase the task execution time, because it will > get the list of files in the target path. When the number of files in the > target path is very large, this operation will be very slow. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5409: HADOOP-18582. Addendum: Skip unnecessary cleanup logic in DistCp.
steveloughran commented on PR #5409: URL: https://github.com/apache/hadoop/pull/5409#issuecomment-1440680482 thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #5409: HADOOP-18582. Addendum: Skip unnecessary cleanup logic in DistCp.
steveloughran merged PR #5409: URL: https://github.com/apache/hadoop/pull/5409 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18535) Implement token storage solution based on MySQL
[ https://issues.apache.org/jira/browse/HADOOP-18535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley resolved HADOOP-18535. Fix Version/s: 3.3.6 3.4.0 Resolution: Fixed Thanks, Hector! > Implement token storage solution based on MySQL > --- > > Key: HADOOP-18535 > URL: https://issues.apache.org/jira/browse/HADOOP-18535 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Assignee: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Fix For: 3.3.6, 3.4.0 > > > Hadoop RBF supports custom implementations of secret managers. At the moment, > the only available implementation is ZKDelegationTokenSecretManagerImpl, > which stores tokens and delegation keys in Zookeeper. > During our investigation, we found that the performance of routers is limited > by the writes to the Zookeeper token store, which impacts requests for token > creation, renewal and cancellation. An alternative secret manager > implementation has been created, based on MySQL, to handle a higher number of > writes. > We measured the throughput of each token operation (create/renew/cancel) on > different setups and obtained the following results: > # Sending requests directly to Namenode (no RBF): > Token creations: 290 reqs per sec > Token renewals: 86 reqs per sec > Token cancellations: 97 reqs per sec > # Sending requests to routers using Zookeeper based secret manager: > Token creations: 31 reqs per sec > Token renewals: 29 reqs per sec > Token cancellations: 40 reqs per sec > # Sending requests to routers using SQL based secret manager: > Token creations: 241 reqs per sec > Token renewals: 103 reqs per sec > Token cancellations: 114 reqs per sec > We noticed a significant improvement when using a SQL secret manager, > comparable to the throughput offered by Namenodes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] rdingankar commented on a diff in pull request #5397: HDFS-16917 Add transfer rate quantile metrics for DataNode reads
rdingankar commented on code in PR #5397:
URL: https://github.com/apache/hadoop/pull/5397#discussion_r1114852429
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java:
##
@@ -1936,4 +1936,20 @@ public static boolean isParentEntry(final String path,
final String parent) {
return path.charAt(parent.length()) == Path.SEPARATOR_CHAR
|| parent.equals(Path.SEPARATOR);
}
+
+ /**
+ * Calculate the transfer rate in megabytes/second. Return -1 for any
negative input.
+ * @param bytes bytes
+ * @param durationMS duration in milliseconds
+ * @return the number of megabytes/second of the transfer rate
+ */
+ public static long transferRateMBs(long bytes, long durationMS) {
+if (bytes < 0 || durationMS < 0) {
+ return -1;
+}
+if (durationMS == 0) {
+ durationMS = 1;
+}
+return bytes / (1024 * 1024) * 1000 / durationMS;
Review Comment:
If we want to have the consumer do the conversion, then we can just keep it
to bytes/milliseconds. bytes and milliseconds are the units used for most of
the metrics in HDFS.
Thoughts @xinglin @mkuchenbecker ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] rdingankar commented on a diff in pull request #5397: HDFS-16917 Add transfer rate quantile metrics for DataNode reads
rdingankar commented on code in PR #5397: URL: https://github.com/apache/hadoop/pull/5397#discussion_r1114847865 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java: ## @@ -1122,6 +1124,7 @@ public void copyBlock(final ExtendedBlock block, datanode.metrics.incrBytesRead((int) read); datanode.metrics.incrBlocksRead(); datanode.metrics.incrTotalReadTime(duration); + datanode.metrics.addReadTransferRateMBs(DFSUtil.transferRateMBs(read, duration)); Review Comment: I was thinking we can publish -1 for the error scenarios which can be visualized from the graphs as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mkuchenbecker commented on a diff in pull request #5397: HDFS-16917 Add transfer rate quantile metrics for DataNode reads
mkuchenbecker commented on code in PR #5397:
URL: https://github.com/apache/hadoop/pull/5397#discussion_r1114838006
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java:
##
@@ -1936,4 +1936,20 @@ public static boolean isParentEntry(final String path,
final String parent) {
return path.charAt(parent.length()) == Path.SEPARATOR_CHAR
|| parent.equals(Path.SEPARATOR);
}
+
+ /**
+ * Calculate the transfer rate in megabytes/second. Return -1 for any
negative input.
+ * @param bytes bytes
+ * @param durationMS duration in milliseconds
+ * @return the number of megabytes/second of the transfer rate
+ */
+ public static long transferRateMBs(long bytes, long durationMS) {
Review Comment:
This function needs unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mkuchenbecker commented on a diff in pull request #5397: HDFS-16917 Add transfer rate quantile metrics for DataNode reads
mkuchenbecker commented on code in PR #5397: URL: https://github.com/apache/hadoop/pull/5397#discussion_r1114836649 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java: ## @@ -1122,6 +1124,7 @@ public void copyBlock(final ExtendedBlock block, datanode.metrics.incrBytesRead((int) read); datanode.metrics.incrBlocksRead(); datanode.metrics.incrTotalReadTime(duration); + datanode.metrics.addReadTransferRateMBs(DFSUtil.transferRateMBs(read, duration)); Review Comment: You will be publishing the metrics, including the -1 here. Assuming duration is 0, I would expect bytes read to be 0 as well (only instance with sub millisecond transfers should be when there's an error prior to the transfer). Bytes Read 0, Duration 0: Return 0. Bytes Read N, duration 0: Exception which we log. Bytes Read N, duration K: Normal -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18639) DockerContainerDeletionTask is not removed from the Nodemanager's statestore when the task is completed.
[
https://issues.apache.org/jira/browse/HADOOP-18639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692321#comment-17692321
]
ASF GitHub Bot commented on HADOOP-18639:
-
hadoop-yetus commented on PR #5425:
URL: https://github.com/apache/hadoop/pull/5425#issuecomment-1440644469
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 32s | | trunk passed |
| +1 :green_heart: | compile | 1m 30s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 35s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 44s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 1m 24s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 23s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 30s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 23m 30s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 139m 13s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5425/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5425 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 4cd6557679cc 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b9662f3160da82f744a48ac298eb0e68d1a57a33 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5425/2/testReport/ |
| Max. process+thread count | 536 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5425/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was
[GitHub] [hadoop] hadoop-yetus commented on pull request #5425: HADOOP-18639 DockerContainerDeletionTask is not removed from the Nodemanager's statestore when the task is completed
hadoop-yetus commented on PR #5425:
URL: https://github.com/apache/hadoop/pull/5425#issuecomment-1440644469
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 32s | | trunk passed |
| +1 :green_heart: | compile | 1m 30s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 35s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 44s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 1m 24s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 23s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 30s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 23m 30s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 139m 13s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5425/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5425 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 4cd6557679cc 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b9662f3160da82f744a48ac298eb0e68d1a57a33 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5425/2/testReport/ |
| Max. process+thread count | 536 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5425/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
[GitHub] [hadoop] mkuchenbecker commented on a diff in pull request #5397: HDFS-16917 Add transfer rate quantile metrics for DataNode reads
mkuchenbecker commented on code in PR #5397:
URL: https://github.com/apache/hadoop/pull/5397#discussion_r1114833478
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java:
##
@@ -1936,4 +1936,20 @@ public static boolean isParentEntry(final String path,
final String parent) {
return path.charAt(parent.length()) == Path.SEPARATOR_CHAR
|| parent.equals(Path.SEPARATOR);
}
+
+ /**
+ * Calculate the transfer rate in megabytes/second. Return -1 for any
negative input.
+ * @param bytes bytes
+ * @param durationMS duration in milliseconds
+ * @return the number of megabytes/second of the transfer rate
+ */
+ public static long transferRateMBs(long bytes, long durationMS) {
+if (bytes < 0 || durationMS < 0) {
+ return -1;
+}
+if (durationMS == 0) {
+ durationMS = 1;
+}
+return bytes / (1024 * 1024) * 1000 / durationMS;
Review Comment:
I would make this bytes per second and let the consumer do the conversion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18535) Implement token storage solution based on MySQL
[ https://issues.apache.org/jira/browse/HADOOP-18535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692316#comment-17692316 ] ASF GitHub Bot commented on HADOOP-18535: - omalley closed pull request #5325: HADOOP-18535. Implement token storage solution based on MySQL URL: https://github.com/apache/hadoop/pull/5325 > Implement token storage solution based on MySQL > --- > > Key: HADOOP-18535 > URL: https://issues.apache.org/jira/browse/HADOOP-18535 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Assignee: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > > Hadoop RBF supports custom implementations of secret managers. At the moment, > the only available implementation is ZKDelegationTokenSecretManagerImpl, > which stores tokens and delegation keys in Zookeeper. > During our investigation, we found that the performance of routers is limited > by the writes to the Zookeeper token store, which impacts requests for token > creation, renewal and cancellation. An alternative secret manager > implementation has been created, based on MySQL, to handle a higher number of > writes. > We measured the throughput of each token operation (create/renew/cancel) on > different setups and obtained the following results: > # Sending requests directly to Namenode (no RBF): > Token creations: 290 reqs per sec > Token renewals: 86 reqs per sec > Token cancellations: 97 reqs per sec > # Sending requests to routers using Zookeeper based secret manager: > Token creations: 31 reqs per sec > Token renewals: 29 reqs per sec > Token cancellations: 40 reqs per sec > # Sending requests to routers using SQL based secret manager: > Token creations: 241 reqs per sec > Token renewals: 103 reqs per sec > Token cancellations: 114 reqs per sec > We noticed a significant improvement when using a SQL secret manager, > comparable to the throughput offered by Namenodes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18535) Implement token storage solution based on MySQL
[ https://issues.apache.org/jira/browse/HADOOP-18535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692315#comment-17692315 ] ASF GitHub Bot commented on HADOOP-18535: - omalley commented on PR #5325: URL: https://github.com/apache/hadoop/pull/5325#issuecomment-1440598019 +1 LGTM (I committed with minor touchups.) > Implement token storage solution based on MySQL > --- > > Key: HADOOP-18535 > URL: https://issues.apache.org/jira/browse/HADOOP-18535 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Assignee: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > > Hadoop RBF supports custom implementations of secret managers. At the moment, > the only available implementation is ZKDelegationTokenSecretManagerImpl, > which stores tokens and delegation keys in Zookeeper. > During our investigation, we found that the performance of routers is limited > by the writes to the Zookeeper token store, which impacts requests for token > creation, renewal and cancellation. An alternative secret manager > implementation has been created, based on MySQL, to handle a higher number of > writes. > We measured the throughput of each token operation (create/renew/cancel) on > different setups and obtained the following results: > # Sending requests directly to Namenode (no RBF): > Token creations: 290 reqs per sec > Token renewals: 86 reqs per sec > Token cancellations: 97 reqs per sec > # Sending requests to routers using Zookeeper based secret manager: > Token creations: 31 reqs per sec > Token renewals: 29 reqs per sec > Token cancellations: 40 reqs per sec > # Sending requests to routers using SQL based secret manager: > Token creations: 241 reqs per sec > Token renewals: 103 reqs per sec > Token cancellations: 114 reqs per sec > We noticed a significant improvement when using a SQL secret manager, > comparable to the throughput offered by Namenodes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] dependabot[bot] opened a new pull request, #5427: Bump derby from 10.10.2.0 to 10.14.2.0 in /hadoop-project
dependabot[bot] opened a new pull request, #5427: URL: https://github.com/apache/hadoop/pull/5427 Bumps derby from 10.10.2.0 to 10.14.2.0. [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/hadoop/network/alerts). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] omalley closed pull request #5325: HADOOP-18535. Implement token storage solution based on MySQL
omalley closed pull request #5325: HADOOP-18535. Implement token storage solution based on MySQL URL: https://github.com/apache/hadoop/pull/5325 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] omalley commented on pull request #5325: HADOOP-18535. Implement token storage solution based on MySQL
omalley commented on PR #5325: URL: https://github.com/apache/hadoop/pull/5325#issuecomment-1440598019 +1 LGTM (I committed with minor touchups.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5426: MAPREDUCE-7434. Fix ShuffleHandler tests.
hadoop-yetus commented on PR #5426:
URL: https://github.com/apache/hadoop/pull/5426#issuecomment-1440554689
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 50m 47s | | trunk passed |
| +1 :green_heart: | compile | 0m 26s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 21s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 51s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 37s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | | the patch passed |
| +1 :green_heart: | compile | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 11s |
[/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-shuffle.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5426/2/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-shuffle.txt)
|
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle:
The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 16s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 43s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 57s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 36s | | hadoop-mapreduce-client-shuffle
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 113m 24s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5426/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5426 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 27c850a1f8c4 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 678f8a0a42cf62a340e287d4214000925ab172b1 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5426/2/testReport/ |
| Max. process+thread count | 620 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
|
| Console output |
[GitHub] [hadoop] hadoop-yetus commented on pull request #5426: MAPREDUCE-7434. Fix ShuffleHandler tests.
hadoop-yetus commented on PR #5426:
URL: https://github.com/apache/hadoop/pull/5426#issuecomment-1440413098
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 9s | | trunk passed |
| +1 :green_heart: | compile | 0m 26s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 23s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 21s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 2s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 11s |
[/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-shuffle.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5426/1/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-shuffle.txt)
|
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle:
The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 16s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 42s | | the patch passed |
| +1 :green_heart: | shadedclient | 27m 21s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 36s | | hadoop-mapreduce-client-shuffle
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 110m 25s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5426/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5426 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 5d4aacfc7497 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5f8d5f2a6f9a20621e4b51a035ce4896e41f5be7 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5426/1/testReport/ |
| Max. process+thread count | 536 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
|
| Console output |
[jira] [Commented] (HADOOP-18639) DockerContainerDeletionTask is not removed from the Nodemanager's statestore when the task is completed.
[
https://issues.apache.org/jira/browse/HADOOP-18639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692263#comment-17692263
]
ASF GitHub Bot commented on HADOOP-18639:
-
hadoop-yetus commented on PR #5425:
URL: https://github.com/apache/hadoop/pull/5425#issuecomment-1440331866
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 19s | | trunk passed |
| +1 :green_heart: | compile | 1m 31s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 34s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 52s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 44s | | the patch passed |
| +1 :green_heart: | compile | 1m 25s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 1m 19s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 23s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5425/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7) |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 35s | | the patch passed |
| +1 :green_heart: | shadedclient | 27m 7s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 23m 31s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 139m 32s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5425/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5425 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 71ed66f96066 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ab08810d2c5e73b2a362d9007e83b1a4ec7e33f8 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5425/1/testReport/ |
| Max. process+thread count | 606 (vs. ulimit of 5500) |
| modules |
[GitHub] [hadoop] hadoop-yetus commented on pull request #5425: HADOOP-18639 DockerContainerDeletionTask is not removed from the Nodemanager's statestore when the task is completed
hadoop-yetus commented on PR #5425:
URL: https://github.com/apache/hadoop/pull/5425#issuecomment-1440331866
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 19s | | trunk passed |
| +1 :green_heart: | compile | 1m 31s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 34s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 52s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 44s | | the patch passed |
| +1 :green_heart: | compile | 1m 25s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 1m 19s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 23s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5425/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7) |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 35s | | the patch passed |
| +1 :green_heart: | shadedclient | 27m 7s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 23m 31s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 139m 32s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5425/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5425 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 71ed66f96066 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ab08810d2c5e73b2a362d9007e83b1a4ec7e33f8 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5425/1/testReport/ |
| Max. process+thread count | 606 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
[jira] [Commented] (HADOOP-18631) Migrate Async appenders to log4j properties
[ https://issues.apache.org/jira/browse/HADOOP-18631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692256#comment-17692256 ] ASF GitHub Bot commented on HADOOP-18631: - virajjasani commented on PR #5418: URL: https://github.com/apache/hadoop/pull/5418#issuecomment-1440310580 Will fix the spotbugs warning > Migrate Async appenders to log4j properties > --- > > Key: HADOOP-18631 > URL: https://issues.apache.org/jira/browse/HADOOP-18631 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > Before we can upgrade to log4j2, we need to migrate async appenders that we > add "dynamically in the code" to the log4j.properties file. Instead of using > core/hdfs site configs, log4j properties or system properties should be used > to determine if the given logger should use async appender. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5418: HADOOP-18631. Migrate Async appenders to log4j properties
virajjasani commented on PR #5418: URL: https://github.com/apache/hadoop/pull/5418#issuecomment-1440310580 Will fix the spotbugs warning -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692255#comment-17692255 ] Viraj Jasani commented on HADOOP-16206: --- For sub-task HADOOP-18631, PR is ready for review. A short summary: * Migrating Async appenders from code to log4j properties for namenode audit logger as well as datanode/namenode metric loggers * Provided sample log4j properties on how we can configure AsyncAppender to wrap RFA for the loggers * Incompatible change as three hdfs-site configs are no longer in use, they are to be replaced with log4j properties. Deprecated them, added javadoc and log to indicate the log4j replacement rather than relying on the configs. The configs: ** dfs.namenode.audit.log.async ** dfs.namenode.audit.log.async.blocking ** dfs.namenode.audit.log.async.buffer.size * namenode audit logger as well as datanode/namenode metric loggers now use SLF4J logger rather than log4j logger directly Requesting the review on [https://github.com/apache/hadoop/pull/5418] > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomicooler opened a new pull request, #5426: MAPREDUCE-7434. Fix ShuffleHandler tests.
tomicooler opened a new pull request, #5426: URL: https://github.com/apache/hadoop/pull/5426 ### Description of PR Fix the user for the ShuffleHandler tests. ### How was this patch tested? I tested manually with the docker environment: `./start-build-env.sh` The 3 [failures](https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-linux-x86_64/1143/testReport/junit/org.apache.hadoop.mapred/TestShuffleHandler/) could be reproduced. ``` mvn clean; mvn install -Pdist -Dmaven.javadoc.skip=true -DskipTests -DskipShade cd hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle mvn test mvn test -Dtest=TestShuffleHandler ``` The tests are green with this change. ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18639) DockerContainerDeletionTask is not removed from the Nodemanager's statestore when the task is completed.
[ https://issues.apache.org/jira/browse/HADOOP-18639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sejin Hwang updated HADOOP-18639: - Description: YARN NodeManager's deletion service has two types of deletion tasks: the FileDeletionTask for deleting log files and the DockerContainerDeletionTask for deleting Docker containers. The FileDeletionTask is removed from the statestore when the task is completed, but the DockerContainerDeletionTask is not. Therefore, the DockerContainerDeletionTask accumulates continuously in the statestore. This causes the NodeManager's deletion service to run the accumulated DockerContainerDeletionTask in the statestore when the NodeManager restarts. As a result, the FileDeletionTask and DockerContainerDeletionTask are delayed unnecessarily while processing accumulated tasks, which can cause disk full issues in environments where a large number of containers are allocated and released. I will attach a patch soon was: YARN NodeManager has two types of deletion tasks: the FileDeletionTask for deleting log files and the DockerContainerDeletionTask for deleting Docker containers. The FileDeletionTask is removed from the statestore when the task is completed, but the DockerContainerDeletionTask is not. Therefore, the DockerContainerDeletionTask accumulates continuously in the statestore. This causes the NodeManager's deletion service to run the accumulated DockerContainerDeletionTask in the statestore when the NodeManager restarts. As a result, the FileDeletionTask and DockerContainerDeletionTask are delayed unnecessarily while processing accumulated tasks, which can cause disk full issues in environments where a large number of containers are allocated and released. I will attach a patch soon > DockerContainerDeletionTask is not removed from the Nodemanager's statestore > when the task is completed. > > > Key: HADOOP-18639 > URL: https://issues.apache.org/jira/browse/HADOOP-18639 > Project: Hadoop Common > Issue Type: Bug >Reporter: Sejin Hwang >Priority: Major > Labels: pull-request-available > > YARN NodeManager's deletion service has two types of deletion tasks: the > FileDeletionTask for deleting log files and the DockerContainerDeletionTask > for deleting Docker containers. > > The FileDeletionTask is removed from the statestore when the task is > completed, but the DockerContainerDeletionTask is not. > Therefore, the DockerContainerDeletionTask accumulates continuously in the > statestore. > > This causes the NodeManager's deletion service to run the accumulated > DockerContainerDeletionTask in the statestore when the NodeManager restarts. > As a result, the FileDeletionTask and DockerContainerDeletionTask are delayed > unnecessarily while processing accumulated tasks, which can cause disk full > issues in environments where a large number of containers are allocated and > released. > I will attach a patch soon -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18639) DockerContainerDeletionTask is not removed from the Nodemanager's statestore when the task is completed.
[ https://issues.apache.org/jira/browse/HADOOP-18639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sejin Hwang updated HADOOP-18639: - Description: YARN NodeManager's deletion service has two types of deletion tasks: the FileDeletionTask for deleting log, usercache, appcache files and the DockerContainerDeletionTask for deleting Docker containers. The FileDeletionTask is removed from the statestore when the task is completed, but the DockerContainerDeletionTask is not. Therefore, the DockerContainerDeletionTask accumulates continuously in the statestore. This causes the NodeManager's deletion service to run the accumulated DockerContainerDeletionTask in the statestore when the NodeManager restarts. As a result, the FileDeletionTask and DockerContainerDeletionTask are delayed unnecessarily while processing accumulated tasks, which can cause disk full issues in environments where a large number of containers are allocated and released. I will attach a patch soon was: YARN NodeManager's deletion service has two types of deletion tasks: the FileDeletionTask for deleting log files and the DockerContainerDeletionTask for deleting Docker containers. The FileDeletionTask is removed from the statestore when the task is completed, but the DockerContainerDeletionTask is not. Therefore, the DockerContainerDeletionTask accumulates continuously in the statestore. This causes the NodeManager's deletion service to run the accumulated DockerContainerDeletionTask in the statestore when the NodeManager restarts. As a result, the FileDeletionTask and DockerContainerDeletionTask are delayed unnecessarily while processing accumulated tasks, which can cause disk full issues in environments where a large number of containers are allocated and released. I will attach a patch soon > DockerContainerDeletionTask is not removed from the Nodemanager's statestore > when the task is completed. > > > Key: HADOOP-18639 > URL: https://issues.apache.org/jira/browse/HADOOP-18639 > Project: Hadoop Common > Issue Type: Bug >Reporter: Sejin Hwang >Priority: Major > Labels: pull-request-available > > YARN NodeManager's deletion service has two types of deletion tasks: the > FileDeletionTask for deleting log, usercache, appcache files and the > DockerContainerDeletionTask for deleting Docker containers. > > The FileDeletionTask is removed from the statestore when the task is > completed, but the DockerContainerDeletionTask is not. > Therefore, the DockerContainerDeletionTask accumulates continuously in the > statestore. > > This causes the NodeManager's deletion service to run the accumulated > DockerContainerDeletionTask in the statestore when the NodeManager restarts. > As a result, the FileDeletionTask and DockerContainerDeletionTask are delayed > unnecessarily while processing accumulated tasks, which can cause disk full > issues in environments where a large number of containers are allocated and > released. > I will attach a patch soon -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18565) AWS SDK V2 - Complete outstanding items
[
https://issues.apache.org/jira/browse/HADOOP-18565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692216#comment-17692216
]
ASF GitHub Bot commented on HADOOP-18565:
-
hadoop-yetus commented on PR #5421:
URL: https://github.com/apache/hadoop/pull/5421#issuecomment-1440116271
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 24 new or modified test files. |
_ feature-HADOOP-18073-s3a-sdk-upgrade Compile Tests _ |
| +0 :ok: | mvndep | 15m 48s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 13s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed |
| +1 :green_heart: | compile | 23m 27s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 21m 27s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 3m 44s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed |
| +1 :green_heart: | mvnsite | 1m 26s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed |
| +1 :green_heart: | javadoc | 1m 5s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 10s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 43s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 1m 19s |
[/branch-spotbugs-hadoop-tools_hadoop-aws-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5421/3/artifact/out/branch-spotbugs-hadoop-tools_hadoop-aws-warnings.html)
| hadoop-tools/hadoop-aws in feature-HADOOP-18073-s3a-sdk-upgrade has 1
extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 25m 3s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 55s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 7s | | the patch passed |
| +1 :green_heart: | compile | 24m 41s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| -1 :x: | javac | 24m 41s |
[/results-compile-javac-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5421/3/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 generated 6 new + 2840 unchanged - 2
fixed = 2846 total (was 2842) |
| +1 :green_heart: | compile | 21m 59s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| -1 :x: | javac | 21m 59s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5421/3/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
| root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08 with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08 generated 6 new + 2640 unchanged - 2 fixed
= 2646 total (was 2642) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 35s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5421/3/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 43 unchanged - 4 fixed = 44 total (was
47) |
| +1 :green_heart: | mvnsite | 1m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 31s | | hadoop-project in the patch
passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08. |
| +1 :green_heart: | javadoc | 0m 42s | |
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08 with JDK
Private
[GitHub] [hadoop] hadoop-yetus commented on pull request #5421: HADOOP-18565. Completes outstanding items for the SDK V2 upgrade.
hadoop-yetus commented on PR #5421:
URL: https://github.com/apache/hadoop/pull/5421#issuecomment-1440116271
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 24 new or modified test files. |
_ feature-HADOOP-18073-s3a-sdk-upgrade Compile Tests _ |
| +0 :ok: | mvndep | 15m 48s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 13s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed |
| +1 :green_heart: | compile | 23m 27s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 21m 27s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 3m 44s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed |
| +1 :green_heart: | mvnsite | 1m 26s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed |
| +1 :green_heart: | javadoc | 1m 5s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 10s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 43s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 1m 19s |
[/branch-spotbugs-hadoop-tools_hadoop-aws-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5421/3/artifact/out/branch-spotbugs-hadoop-tools_hadoop-aws-warnings.html)
| hadoop-tools/hadoop-aws in feature-HADOOP-18073-s3a-sdk-upgrade has 1
extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 25m 3s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 55s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 7s | | the patch passed |
| +1 :green_heart: | compile | 24m 41s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| -1 :x: | javac | 24m 41s |
[/results-compile-javac-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5421/3/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 generated 6 new + 2840 unchanged - 2
fixed = 2846 total (was 2842) |
| +1 :green_heart: | compile | 21m 59s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| -1 :x: | javac | 21m 59s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5421/3/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
| root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08 with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08 generated 6 new + 2640 unchanged - 2 fixed
= 2646 total (was 2642) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 35s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5421/3/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 43 unchanged - 4 fixed = 44 total (was
47) |
| +1 :green_heart: | mvnsite | 1m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 31s | | hadoop-project in the patch
passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08. |
| +1 :green_heart: | javadoc | 0m 42s | |
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08 with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 generated 0 new + 1 unchanged - 3
fixed = 1 total (was 4) |
| +0 :ok: | spotbugs | 0m 28s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | spotbugs | 1m 31s | | hadoop-tools/hadoop-aws
[jira] [Commented] (HADOOP-18631) Migrate Async appenders to log4j properties
[
https://issues.apache.org/jira/browse/HADOOP-18631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692209#comment-17692209
]
ASF GitHub Bot commented on HADOOP-18631:
-
hadoop-yetus commented on PR #5418:
URL: https://github.com/apache/hadoop/pull/5418#issuecomment-1440080838
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 8 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 46s | | trunk passed |
| +1 :green_heart: | compile | 1m 38s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 28s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 9s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 16s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 34s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 29m 29s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 32s | | the patch passed |
| +1 :green_heart: | compile | 1m 33s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 33s | | the patch passed |
| +1 :green_heart: | compile | 1m 24s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 57s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 486 unchanged - 21
fixed = 486 total (was 507) |
| +1 :green_heart: | mvnsite | 1m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 26s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 3m 50s |
[/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5418/8/artifact/out/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html)
| hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) |
| +1 :green_heart: | shadedclient | 29m 38s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 245m 20s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5418/8/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 376m 54s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Inconsistent synchronization of
org.apache.hadoop.hdfs.util.AsyncRFAAppender.conversionPattern; locked 50% of
time Unsynchronized access at AsyncRFAAppender.java:50% of time
Unsynchronized access at AsyncRFAAppender.java:[line 124] |
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestObserverNode |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5418/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5418 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 6e959bcb5df3 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git
[GitHub] [hadoop] hadoop-yetus commented on pull request #5418: HADOOP-18631. Migrate Async appenders to log4j properties
hadoop-yetus commented on PR #5418:
URL: https://github.com/apache/hadoop/pull/5418#issuecomment-1440080838
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 8 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 46s | | trunk passed |
| +1 :green_heart: | compile | 1m 38s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 28s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 9s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 16s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 34s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 29m 29s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 32s | | the patch passed |
| +1 :green_heart: | compile | 1m 33s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 33s | | the patch passed |
| +1 :green_heart: | compile | 1m 24s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 57s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 486 unchanged - 21
fixed = 486 total (was 507) |
| +1 :green_heart: | mvnsite | 1m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 26s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 3m 50s |
[/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5418/8/artifact/out/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html)
| hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) |
| +1 :green_heart: | shadedclient | 29m 38s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 245m 20s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5418/8/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 376m 54s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Inconsistent synchronization of
org.apache.hadoop.hdfs.util.AsyncRFAAppender.conversionPattern; locked 50% of
time Unsynchronized access at AsyncRFAAppender.java:50% of time
Unsynchronized access at AsyncRFAAppender.java:[line 124] |
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestObserverNode |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5418/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5418 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 6e959bcb5df3 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0fa413b57800ec7dc07222c61519d570477e87f9 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
[jira] [Commented] (HADOOP-18631) Migrate Async appenders to log4j properties
[
https://issues.apache.org/jira/browse/HADOOP-18631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692204#comment-17692204
]
ASF GitHub Bot commented on HADOOP-18631:
-
hadoop-yetus commented on PR #5418:
URL: https://github.com/apache/hadoop/pull/5418#issuecomment-1440062836
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 8 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 44s | | trunk passed |
| +1 :green_heart: | compile | 1m 31s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 13s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 54s | | trunk passed |
| +1 :green_heart: | shadedclient | 29m 38s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 33s | | the patch passed |
| +1 :green_heart: | compile | 1m 29s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 29s | | the patch passed |
| +1 :green_heart: | compile | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 1s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 486 unchanged - 21
fixed = 486 total (was 507) |
| +1 :green_heart: | mvnsite | 1m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 31s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 3m 50s |
[/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5418/7/artifact/out/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html)
| hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) |
| +1 :green_heart: | shadedclient | 29m 18s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 237m 36s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 52s | | The patch does not
generate ASF License warnings. |
| | | 368m 44s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Inconsistent synchronization of
org.apache.hadoop.hdfs.util.AsyncRFAAppender.conversionPattern; locked 50% of
time Unsynchronized access at AsyncRFAAppender.java:50% of time
Unsynchronized access at AsyncRFAAppender.java:[line 122] |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5418/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5418 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 3d42c92f17d4 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 284d6f90b9482ab4140a24a4129bf298e05890e8 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
[GitHub] [hadoop] hadoop-yetus commented on pull request #5418: HADOOP-18631. Migrate Async appenders to log4j properties
hadoop-yetus commented on PR #5418:
URL: https://github.com/apache/hadoop/pull/5418#issuecomment-1440062836
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 8 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 44s | | trunk passed |
| +1 :green_heart: | compile | 1m 31s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 13s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 54s | | trunk passed |
| +1 :green_heart: | shadedclient | 29m 38s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 33s | | the patch passed |
| +1 :green_heart: | compile | 1m 29s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 29s | | the patch passed |
| +1 :green_heart: | compile | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 1s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 486 unchanged - 21
fixed = 486 total (was 507) |
| +1 :green_heart: | mvnsite | 1m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 31s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 3m 50s |
[/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5418/7/artifact/out/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html)
| hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) |
| +1 :green_heart: | shadedclient | 29m 18s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 237m 36s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 52s | | The patch does not
generate ASF License warnings. |
| | | 368m 44s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Inconsistent synchronization of
org.apache.hadoop.hdfs.util.AsyncRFAAppender.conversionPattern; locked 50% of
time Unsynchronized access at AsyncRFAAppender.java:50% of time
Unsynchronized access at AsyncRFAAppender.java:[line 122] |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5418/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5418 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 3d42c92f17d4 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 284d6f90b9482ab4140a24a4129bf298e05890e8 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5418/7/testReport/ |
| Max. process+thread count | 2264 (vs. ulimit of 5500) |
|
[jira] [Updated] (HADOOP-18639) DockerContainerDeletionTask is not removed from the Nodemanager's statestore when the task is completed.
[ https://issues.apache.org/jira/browse/HADOOP-18639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18639: Labels: pull-request-available (was: ) > DockerContainerDeletionTask is not removed from the Nodemanager's statestore > when the task is completed. > > > Key: HADOOP-18639 > URL: https://issues.apache.org/jira/browse/HADOOP-18639 > Project: Hadoop Common > Issue Type: Bug >Reporter: Sejin Hwang >Priority: Major > Labels: pull-request-available > > YARN NodeManager has two types of deletion tasks: the FileDeletionTask for > deleting log files and the DockerContainerDeletionTask for deleting Docker > containers. > > The FileDeletionTask is removed from the statestore when the task is > completed, but the DockerContainerDeletionTask is not. > Therefore, the DockerContainerDeletionTask accumulates continuously in the > statestore. > > This causes the NodeManager's deletion service to run the accumulated > DockerContainerDeletionTask in the statestore when the NodeManager restarts. > As a result, the FileDeletionTask and DockerContainerDeletionTask are delayed > unnecessarily while processing accumulated tasks, which can cause disk full > issues in environments where a large number of containers are allocated and > released. > I will attach a patch soon -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18639) DockerContainerDeletionTask is not removed from the Nodemanager's statestore when the task is completed.
[ https://issues.apache.org/jira/browse/HADOOP-18639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692199#comment-17692199 ] ASF GitHub Bot commented on HADOOP-18639: - Sejin-Hwang opened a new pull request, #5425: URL: https://github.com/apache/hadoop/pull/5425 ### Description of PR When the DockerContainerDeletionTask is completed, it should be deleted from the NodeManager's statestore, just like the FileDeletionTask. JIRA : https://issues.apache.org/jira/browse/HADOOP-18639 ### How was this patch tested? Unit test added > DockerContainerDeletionTask is not removed from the Nodemanager's statestore > when the task is completed. > > > Key: HADOOP-18639 > URL: https://issues.apache.org/jira/browse/HADOOP-18639 > Project: Hadoop Common > Issue Type: Bug >Reporter: Sejin Hwang >Priority: Major > > YARN NodeManager has two types of deletion tasks: the FileDeletionTask for > deleting log files and the DockerContainerDeletionTask for deleting Docker > containers. > > The FileDeletionTask is removed from the statestore when the task is > completed, but the DockerContainerDeletionTask is not. > Therefore, the DockerContainerDeletionTask accumulates continuously in the > statestore. > > This causes the NodeManager's deletion service to run the accumulated > DockerContainerDeletionTask in the statestore when the NodeManager restarts. > As a result, the FileDeletionTask and DockerContainerDeletionTask are delayed > unnecessarily while processing accumulated tasks, which can cause disk full > issues in environments where a large number of containers are allocated and > released. > I will attach a patch soon -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Sejin-Hwang opened a new pull request, #5425: HADOOP-18639 DockerContainerDeletionTask is not removed from the Nodemanager's statestore when the task is completed
Sejin-Hwang opened a new pull request, #5425: URL: https://github.com/apache/hadoop/pull/5425 ### Description of PR When the DockerContainerDeletionTask is completed, it should be deleted from the NodeManager's statestore, just like the FileDeletionTask. JIRA : https://issues.apache.org/jira/browse/HADOOP-18639 ### How was this patch tested? Unit test added -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18639) DockerContainerDeletionTask is not removed from the Nodemanager's statestore when the task is completed.
[ https://issues.apache.org/jira/browse/HADOOP-18639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sejin Hwang updated HADOOP-18639: - Description: YARN NodeManager has two types of deletion tasks: the FileDeletionTask for deleting log files and the DockerContainerDeletionTask for deleting Docker containers. The FileDeletionTask is removed from the statestore when the task is completed, but the DockerContainerDeletionTask is not. Therefore, the DockerContainerDeletionTask accumulates continuously in the statestore. This causes the NodeManager's deletion service to run the accumulated DockerContainerDeletionTask in the statestore when the NodeManager restarts. As a result, the FileDeletionTask and DockerContainerDeletionTask are delayed unnecessarily while processing accumulated tasks, which can cause disk full issues in environments where a large number of containers are allocated and released. I will attach a patch soon was: YARN NodeManager has two types of deletion tasks: the FileDeletionTask for deleting log files and the DockerContainerDeletionTask for deleting Docker containers. The FileDeletionTask is removed from the statestore when the task is completed, but the DockerContainerDeletionTask is not. Therefore, the DockerContainerDeletionTask accumulates continuously in the statestore. This causes the NodeManager's deletion service to run the accumulated DockerContainerDeletionTask in the statestore when the NodeManager restarts. As a result, the FileDeletionTask and DockerContainerDeletionTask are delayed unnecessarily while processing accumulated tasks, which can cause disk full issues in environments where a large number of containers are allocated and released. > DockerContainerDeletionTask is not removed from the Nodemanager's statestore > when the task is completed. > > > Key: HADOOP-18639 > URL: https://issues.apache.org/jira/browse/HADOOP-18639 > Project: Hadoop Common > Issue Type: Bug >Reporter: Sejin Hwang >Priority: Major > > YARN NodeManager has two types of deletion tasks: the FileDeletionTask for > deleting log files and the DockerContainerDeletionTask for deleting Docker > containers. > > The FileDeletionTask is removed from the statestore when the task is > completed, but the DockerContainerDeletionTask is not. > Therefore, the DockerContainerDeletionTask accumulates continuously in the > statestore. > > This causes the NodeManager's deletion service to run the accumulated > DockerContainerDeletionTask in the statestore when the NodeManager restarts. > As a result, the FileDeletionTask and DockerContainerDeletionTask are delayed > unnecessarily while processing accumulated tasks, which can cause disk full > issues in environments where a large number of containers are allocated and > released. > I will attach a patch soon -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18639) DockerContainerDeletionTask is not removed from the Nodemanager's statestore when the task is completed.
Sejin Hwang created HADOOP-18639: Summary: DockerContainerDeletionTask is not removed from the Nodemanager's statestore when the task is completed. Key: HADOOP-18639 URL: https://issues.apache.org/jira/browse/HADOOP-18639 Project: Hadoop Common Issue Type: Bug Reporter: Sejin Hwang YARN NodeManager has two types of deletion tasks: the FileDeletionTask for deleting log files and the DockerContainerDeletionTask for deleting Docker containers. The FileDeletionTask is removed from the statestore when the task is completed, but the DockerContainerDeletionTask is not. Therefore, the DockerContainerDeletionTask accumulates continuously in the statestore. This causes the NodeManager's deletion service to run the accumulated DockerContainerDeletionTask in the statestore when the NodeManager restarts. As a result, the FileDeletionTask and DockerContainerDeletionTask are delayed unnecessarily while processing accumulated tasks, which can cause disk full issues in environments where a large number of containers are allocated and released. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] liuaer opened a new pull request, #5424: HDFS-16931. Observer nn delete blocks asynchronously when tail OP_DEL…
liuaer opened a new pull request, #5424: URL: https://github.com/apache/hadoop/pull/5424 In our HDFS cluster that holds hundreds of millions of metadata, all of which is EC data. When Observer NN replays the OP_DELETE operation which deletes a large directory, involving tens of millions of metadata. Because the operation is synchronous, the Observer NN is stuck for nearly 50 minutes. [HDFS-16043](https://issues.apache.org/jira/browse/HDFS-16043) has solved the asynchronous deletion on the active NN. This is to solve the asynchronous deletion on the Observer NN. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18258) Merging of S3A Audit Logs
[
https://issues.apache.org/jira/browse/HADOOP-18258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692168#comment-17692168
]
ASF GitHub Bot commented on HADOOP-18258:
-
hadoop-yetus commented on PR #4383:
URL: https://github.com/apache/hadoop/pull/4383#issuecomment-1439940590
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 28s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 35s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 17s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 37s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 23m 56s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 47s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 16s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 28s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 39s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 105m 37s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4383/18/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4383 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile
javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux cc4f13fbc226 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b21304c17bb40b3be690f792b3429c256fe03e49 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4383/18/testReport/ |
| Max. process+thread count | 568 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4383/18/console |
| versions |
[GitHub] [hadoop] hadoop-yetus commented on pull request #4383: HADOOP-18258. Merging of S3A Audit Logs
hadoop-yetus commented on PR #4383:
URL: https://github.com/apache/hadoop/pull/4383#issuecomment-1439940590
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 28s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 35s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 17s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 37s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 23m 56s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 47s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 16s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 28s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 39s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 105m 37s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4383/18/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4383 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile
javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux cc4f13fbc226 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b21304c17bb40b3be690f792b3429c256fe03e49 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4383/18/testReport/ |
| Max. process+thread count | 568 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4383/18/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the
[jira] [Commented] (HADOOP-18636) LocalDirAllocator cannot recover from directory tree deletion during the life of a filesystem client
[
https://issues.apache.org/jira/browse/HADOOP-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692149#comment-17692149
]
ASF GitHub Bot commented on HADOOP-18636:
-
steveloughran merged PR #5412:
URL: https://github.com/apache/hadoop/pull/5412
> LocalDirAllocator cannot recover from directory tree deletion during the life
> of a filesystem client
>
>
> Key: HADOOP-18636
> URL: https://issues.apache.org/jira/browse/HADOOP-18636
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, fs/azure, fs/s3
>Affects Versions: 3.3.4
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Labels: pull-request-available
>
> The s3a and abfs clients use LocalDirAllocator for allocating files in local
> (temporary) storage for buffering blocks to write, and, for the s3a staging
> committer, files being staged.
> When initialized (or when the configuration key value is updated)
> LocalDirAllocator enumerates all directories in the list and calls
> {{mkdirs()}} to create them.
> when you ask actually for a file, it will look for the parent dir, and will
> again call {{mkdirs()}}.
> But before it does that, it looks to see if the dir has any space...if not it
> is excluded from the list of directories with room for data.
> And guess what: directories which don't exist report as having no space. So
> they get excluded -the recreation code doesn't get a chance to run.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #5412: HADOOP-18636 LocalDirAllocator cannot recover from directory tree deletion
steveloughran merged PR #5412: URL: https://github.com/apache/hadoop/pull/5412 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18636) LocalDirAllocator cannot recover from directory tree deletion during the life of a filesystem client
[
https://issues.apache.org/jira/browse/HADOOP-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692147#comment-17692147
]
ASF GitHub Bot commented on HADOOP-18636:
-
steveloughran commented on PR #5412:
URL: https://github.com/apache/hadoop/pull/5412#issuecomment-1439886887
thanks. That checkwrite is really a safety check about whether a dir is
*still* writable; the allocator precreates all parent dirs whenever the config
changes
merging
> LocalDirAllocator cannot recover from directory tree deletion during the life
> of a filesystem client
>
>
> Key: HADOOP-18636
> URL: https://issues.apache.org/jira/browse/HADOOP-18636
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, fs/azure, fs/s3
>Affects Versions: 3.3.4
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Labels: pull-request-available
>
> The s3a and abfs clients use LocalDirAllocator for allocating files in local
> (temporary) storage for buffering blocks to write, and, for the s3a staging
> committer, files being staged.
> When initialized (or when the configuration key value is updated)
> LocalDirAllocator enumerates all directories in the list and calls
> {{mkdirs()}} to create them.
> when you ask actually for a file, it will look for the parent dir, and will
> again call {{mkdirs()}}.
> But before it does that, it looks to see if the dir has any space...if not it
> is excluded from the list of directories with room for data.
> And guess what: directories which don't exist report as having no space. So
> they get excluded -the recreation code doesn't get a chance to run.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5412: HADOOP-18636 LocalDirAllocator cannot recover from directory tree deletion
steveloughran commented on PR #5412: URL: https://github.com/apache/hadoop/pull/5412#issuecomment-1439886887 thanks. That checkwrite is really a safety check about whether a dir is *still* writable; the allocator precreates all parent dirs whenever the config changes merging -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] (HADOOP-17063) S3A deleteObjects hanging/retrying forever
[ https://issues.apache.org/jira/browse/HADOOP-17063 ]
Steve Loughran deleted comment on HADOOP-17063:
-
was (Author: ste...@apache.org):
assuming this is in dir marker deletion, how about we disable retries in that
case -now that recent clients are all happy with retained markers, and because
permissions problems may be cause.
> S3A deleteObjects hanging/retrying forever
> --
>
> Key: HADOOP-17063
> URL: https://issues.apache.org/jira/browse/HADOOP-17063
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.1
> Environment: hadoop 3.2.1
> spark 2.4.4
>
>Reporter: Dyno
>Priority: Minor
> Attachments: jstack_exec-34.log, jstack_exec-40.log,
> jstack_exec-74.log
>
>
> {code}
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:523)
> com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:82)
>
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.putObject(S3ABlockOutputStream.java:446)
>
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.close(S3ABlockOutputStream.java:365)
>
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> org.apache.parquet.hadoop.util.HadoopPositionOutputStream.close(HadoopPositionOutputStream.java:64)
> org.apache.parquet.hadoop.ParquetFileWriter.end(ParquetFileWriter.java:685)
> org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:122)
>
> org.apache.parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:165)
>
> org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.close(ParquetOutputWriter.scala:42)
>
> org.apache.spark.sql.execution.datasources.FileFormatDataWriter.releaseResources(FileFormatDataWriter.scala:57)
>
> org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:74)
>
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:247)
>
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:242)
>
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)
>
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:248)
>
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)
>
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
> org.apache.spark.scheduler.Task.run(Task.scala:123)
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
> org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> {code}
>
> we are using spark 2.4.4 with hadoop 3.2.1 on kubernetes/spark-operator,
> sometimes we see this hang with the stacktrace above. it looks like the
> putObject never return, we have to kill the executor to make the job move
> forward.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17881) S3A DeleteOperation to parallelize POSTing of bulk deletes
[ https://issues.apache.org/jira/browse/HADOOP-17881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692140#comment-17692140 ] Steve Loughran commented on HADOOP-17881: - aa bit of shuffling on a page would be good; maybe if that first page list signalled there'd be a continuation then it could start collecting > 1 page of results before POSTing a delete from a random set of the results. we could also have a central rate limiter of write ops in the S3A FS instance, which could be used to to throttle delete as well as POSTs on job commit, as that is the other bursty operation. The ABFS connector does that for things, like rename() in job commit > S3A DeleteOperation to parallelize POSTing of bulk deletes > -- > > Key: HADOOP-17881 > URL: https://issues.apache.org/jira/browse/HADOOP-17881 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Priority: Major > > Once the need to update the DDB tables is removed, we can't go from a single > POSTed delete at a time to posting a large set of bulk delete operations in > parallel. > The current design is to support incremental update of S3Guard tables, > including handling partial failures. Not a problem anymore. > This will significantly improve delete() performance on directory trees with > many many children/descendants, as it goes from a sequence of children/1000 > POSTs to parallel writes. As each file deleted is still throttled, we will be > limited to 3500 deletes/second with throttling, so throwing a large pool of > workers at the problem would be counter-productive and potentially cause > problems for other applications trying to write down the same directory tree. > But we can do better than one-POST at a time. > Proposed > * if parallel delete is off: no limit > * parallel delete is on, limit #of parallel to 3000/page-size: you'll never > have more updates pending than the write limit of a single shard. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5295: YARN-11404. Add junit5 dependency to hadoop-mapreduce-client-app to fix few unit test failure
steveloughran commented on PR #5295: URL: https://github.com/apache/hadoop/pull/5295#issuecomment-1439816394 ``` Caused by: java.lang.ClassNotFoundException: org.junit.jupiter.api.Assertions at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 14 more ``` problem here is that maven doesn't import transitive dependencies from a -test JAR, so if one module (mapreduce-client) adds a new mandatory dependency (here, junit jupiter stuff, maybe junit-mockito) then everything which imports the suite and uses it will break. We had that when we added assertJ tests to hadoop-common fs contract tests. Fix here is pom changes everywhere the test jar is referenced. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18636) LocalDirAllocator cannot recover from directory tree deletion during the life of a filesystem client
[
https://issues.apache.org/jira/browse/HADOOP-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692120#comment-17692120
]
ASF GitHub Bot commented on HADOOP-18636:
-
steveloughran commented on PR #5412:
URL: https://github.com/apache/hadoop/pull/5412#issuecomment-1439805659
to completely clear: that temp dir creation already happens on the first
invocation of getLocalPathForWrite, this just adds it to every call (this is
cheap if the path exists()).
> LocalDirAllocator cannot recover from directory tree deletion during the life
> of a filesystem client
>
>
> Key: HADOOP-18636
> URL: https://issues.apache.org/jira/browse/HADOOP-18636
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, fs/azure, fs/s3
>Affects Versions: 3.3.4
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Labels: pull-request-available
>
> The s3a and abfs clients use LocalDirAllocator for allocating files in local
> (temporary) storage for buffering blocks to write, and, for the s3a staging
> committer, files being staged.
> When initialized (or when the configuration key value is updated)
> LocalDirAllocator enumerates all directories in the list and calls
> {{mkdirs()}} to create them.
> when you ask actually for a file, it will look for the parent dir, and will
> again call {{mkdirs()}}.
> But before it does that, it looks to see if the dir has any space...if not it
> is excluded from the list of directories with room for data.
> And guess what: directories which don't exist report as having no space. So
> they get excluded -the recreation code doesn't get a chance to run.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5412: HADOOP-18636 LocalDirAllocator cannot recover from directory tree deletion
steveloughran commented on PR #5412: URL: https://github.com/apache/hadoop/pull/5412#issuecomment-1439805659 to completely clear: that temp dir creation already happens on the first invocation of getLocalPathForWrite, this just adds it to every call (this is cheap if the path exists()). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18636) LocalDirAllocator cannot recover from directory tree deletion during the life of a filesystem client
[
https://issues.apache.org/jira/browse/HADOOP-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692119#comment-17692119
]
ASF GitHub Bot commented on HADOOP-18636:
-
steveloughran commented on PR #5412:
URL: https://github.com/apache/hadoop/pull/5412#issuecomment-1439803446
Currently `LocalDirAllocator.getLocalPathForWrite()` will create all target
dirs if absent on first invocation or if the config is changed via
`confChanged()`
On later calls, checkWrite is called after a dir has been selected. This
method contains both parent dir recreation *and* explicit attempt to create a
file under it.
But the parent dir creation is too late for the disk space allocator, which
will have already got 0 as the available space if a dir doesn't exist.
What does this mean? It means that long-lived LocalDirAllocator instances
cannot recover from the deletion of temp directories, even though checkWrite is
meant to be able to do so. Calling mkdirs() earlier *and ignoring the result*
means that if hadoop.tmp.dir has been deleted there will be a best effort
attempt to recreate it before looking for free disk space. By not checking at
the return code we are avoiding changing where things may fail and all the
complications that might bring, so it is not a perfect fix. It's just a lot
simpler than something more sophisticated like handling the special case where
there are no free directories buy triggering a whole new confChanged()-style
rescan in the hope that will bring back a directory or even disk volume.
> LocalDirAllocator cannot recover from directory tree deletion during the life
> of a filesystem client
>
>
> Key: HADOOP-18636
> URL: https://issues.apache.org/jira/browse/HADOOP-18636
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, fs/azure, fs/s3
>Affects Versions: 3.3.4
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Labels: pull-request-available
>
> The s3a and abfs clients use LocalDirAllocator for allocating files in local
> (temporary) storage for buffering blocks to write, and, for the s3a staging
> committer, files being staged.
> When initialized (or when the configuration key value is updated)
> LocalDirAllocator enumerates all directories in the list and calls
> {{mkdirs()}} to create them.
> when you ask actually for a file, it will look for the parent dir, and will
> again call {{mkdirs()}}.
> But before it does that, it looks to see if the dir has any space...if not it
> is excluded from the list of directories with room for data.
> And guess what: directories which don't exist report as having no space. So
> they get excluded -the recreation code doesn't get a chance to run.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18565) AWS SDK V2 - Complete outstanding items
[ https://issues.apache.org/jira/browse/HADOOP-18565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692118#comment-17692118 ] ASF GitHub Bot commented on HADOOP-18565: - ahmarsuhail commented on code in PR #5421: URL: https://github.com/apache/hadoop/pull/5421#discussion_r1114152953 ## hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AProxy.java: ## @@ -20,18 +20,17 @@ import java.io.IOException; -import com.amazonaws.ClientConfiguration; -import com.amazonaws.Protocol; import org.assertj.core.api.Assertions; import org.junit.Test; +import software.amazon.awssdk.http.apache.ProxyConfiguration; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.s3a.impl.AWSClientConfig; import org.apache.hadoop.test.AbstractHadoopTestBase; import static org.apache.hadoop.fs.s3a.Constants.PROXY_HOST; import static org.apache.hadoop.fs.s3a.Constants.PROXY_PORT; import static org.apache.hadoop.fs.s3a.Constants.PROXY_SECURED; -import static org.apache.hadoop.fs.s3a.S3AUtils.initProxySupport; /** Review Comment: @mehakmeet I noticed we introduced a new config option `fs.s3a.proxy.ssl.enabled`, but the port of the proxy is still set using `fs.s3a.connection.ssl.enabled` [here](https://github.com/mehakmeet/hadoop/blob/8d7a7f74e62e4e37a91dc800cfbca698518ec37f/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java#L1323). Was wondering why need the new `fs.s3a.proxy.ssl.enabled` option? can we not use `fs.s3a.connection.ssl.enabled` to set the protocol too? Asking here as this SDK V2 branch uses `fs.s3a.connection.ssl.enabled` to set both protocol + port, but that causes this test to fail as it uses `fs.s3a.proxy.ssl.enabled`. > AWS SDK V2 - Complete outstanding items > --- > > Key: HADOOP-18565 > URL: https://issues.apache.org/jira/browse/HADOOP-18565 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Priority: Major > Labels: pull-request-available > > The following work remains to complete the SDK upgrade work: > * S3A allows users configure to custom signers, add in support for this. > * Remove SDK V1 bundle dependency > * Update `getRegion()` logic to use retries. > * Add in progress listeners for `S3ABlockOutputStream` > * Fix any failing tests. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5412: HADOOP-18636 LocalDirAllocator cannot recover from directory tree deletion
steveloughran commented on PR #5412: URL: https://github.com/apache/hadoop/pull/5412#issuecomment-1439803446 Currently `LocalDirAllocator.getLocalPathForWrite()` will create all target dirs if absent on first invocation or if the config is changed via `confChanged()` On later calls, checkWrite is called after a dir has been selected. This method contains both parent dir recreation *and* explicit attempt to create a file under it. But the parent dir creation is too late for the disk space allocator, which will have already got 0 as the available space if a dir doesn't exist. What does this mean? It means that long-lived LocalDirAllocator instances cannot recover from the deletion of temp directories, even though checkWrite is meant to be able to do so. Calling mkdirs() earlier *and ignoring the result* means that if hadoop.tmp.dir has been deleted there will be a best effort attempt to recreate it before looking for free disk space. By not checking at the return code we are avoiding changing where things may fail and all the complications that might bring, so it is not a perfect fix. It's just a lot simpler than something more sophisticated like handling the special case where there are no free directories buy triggering a whole new confChanged()-style rescan in the hope that will bring back a directory or even disk volume. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
