[jira] [Commented] (HADOOP-17612) Bump default Zookeeper version to 3.7.0
[ https://issues.apache.org/jira/browse/HADOOP-17612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17314414#comment-17314414 ] Viraj Jasani commented on HADOOP-17612: --- Thanks for inputs [~ayushtkn] [~eolivelli]. Keeping an eye on Zookeeper 3.6.3 and Curator 5.1.1 releases, we can start using them once they are out. Zookeeper 3.6.3 RC is [in-progress|https://github.com/apache/zookeeper/releases/tag/release-3.6.3-0]. > Bump default Zookeeper version to 3.7.0 > --- > > Key: HADOOP-17612 > URL: https://issues.apache.org/jira/browse/HADOOP-17612 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > We can bump Zookeeper version to 3.7.0 for trunk. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #2860: Test Pre-Commits.
virajjasani commented on pull request #2860: URL: https://github.com/apache/hadoop/pull/2860#issuecomment-812976534 Thanks for this PR @ayushtkn . TestBlockRecovery is consistently failing and fix in-progress here: #2844 Let me see if I can also take up some other flakies or consistently failing tests. Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17619) Fix DelegationTokenRenewer#updateRenewalTime java doc error.
[
https://issues.apache.org/jira/browse/HADOOP-17619?focusedWorklogId=576524=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-576524
]
ASF GitHub Bot logged work on HADOOP-17619:
---
Author: ASF GitHub Bot
Created on: 04/Apr/21 05:31
Start Date: 04/Apr/21 05:31
Worklog Time Spent: 10m
Work Description: qizhu-lucas commented on a change in pull request #2846:
URL: https://github.com/apache/hadoop/pull/2846#discussion_r606750017
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegationTokenRenewer.java
##
@@ -107,7 +107,7 @@ public boolean equals(final Object that) {
* Set a new time for the renewal.
* It can only be called when the action is not in the queue or any
* collection because the hashCode may change
- * @param newTime the new time
+ * @param delay the renew cycle
Review comment:
Thanks @ayushtkn for good suggestion, it's more reasonable.
Updated this in latest PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 576524)
Time Spent: 1h (was: 50m)
> Fix DelegationTokenRenewer#updateRenewalTime java doc error.
>
>
> Key: HADOOP-17619
> URL: https://issues.apache.org/jira/browse/HADOOP-17619
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Qi Zhu
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The param of updateRenewalTime should be the renew cycle, not the new time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] qizhu-lucas commented on a change in pull request #2846: HADOOP-17619: Fix DelegationTokenRenewer#updateRenewalTime java doc e…
qizhu-lucas commented on a change in pull request #2846:
URL: https://github.com/apache/hadoop/pull/2846#discussion_r606750017
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegationTokenRenewer.java
##
@@ -107,7 +107,7 @@ public boolean equals(final Object that) {
* Set a new time for the renewal.
* It can only be called when the action is not in the queue or any
* collection because the hashCode may change
- * @param newTime the new time
+ * @param delay the renew cycle
Review comment:
Thanks @ayushtkn for good suggestion, it's more reasonable.
Updated this in latest PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a change in pull request #2844: HDFS-15940. Fixing and refactoring tests specific to Block recovery
virajjasani commented on a change in pull request #2844:
URL: https://github.com/apache/hadoop/pull/2844#discussion_r606748973
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery2.java
##
@@ -0,0 +1,464 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.datanode;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.ha.HAServiceProtocol;
+import org.apache.hadoop.hdfs.AppendTestUtil;
+import org.apache.hadoop.hdfs.DFSClient;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.DFSTestUtil;
+import org.apache.hadoop.hdfs.DistributedFileSystem;
+import org.apache.hadoop.hdfs.HdfsConfiguration;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.hdfs.StripedFileTestUtil;
+import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
+import org.apache.hadoop.hdfs.protocol.ErasureCodingPolicy;
+import org.apache.hadoop.hdfs.protocol.LocatedBlock;
+import
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB;
+import org.apache.hadoop.hdfs.server.namenode.FSNamesystem;
+import org.apache.hadoop.hdfs.server.protocol.BlockRecoveryCommand;
+import org.apache.hadoop.hdfs.server.protocol.DatanodeCommand;
+import org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration;
+import org.apache.hadoop.hdfs.server.protocol.HeartbeatResponse;
+import org.apache.hadoop.hdfs.server.protocol.NNHAStatusHeartbeat;
+import org.apache.hadoop.hdfs.server.protocol.NamenodeProtocols;
+import org.apache.hadoop.hdfs.server.protocol.NamespaceInfo;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.util.AutoCloseableLock;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TestName;
+import org.mockito.Mockito;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.apache.hadoop.hdfs.DFSConfigKeys
+.DFS_BLOCK_SIZE_KEY;
Review comment:
Ahh, my bad. Will fix it.
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery1.java
##
@@ -122,9 +108,9 @@
/**
* This tests if sync all replicas in block recovery works correctly.
*/
-public class TestBlockRecovery {
+public class TestBlockRecovery1 {
Review comment:
Sure, sounds good. Will update.
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery1.java
##
@@ -336,191 +322,194 @@ public void testFinalizedReplicas () throws IOException
{
if(LOG.isDebugEnabled()) {
LOG.debug("Running " + GenericTestUtils.getMethodName());
}
-ReplicaRecoveryInfo replica1 = new ReplicaRecoveryInfo(BLOCK_ID,
-REPLICA_LEN1, GEN_STAMP-1, ReplicaState.FINALIZED);
-ReplicaRecoveryInfo replica2 = new ReplicaRecoveryInfo(BLOCK_ID,
-REPLICA_LEN1, GEN_STAMP-2, ReplicaState.FINALIZED);
+ReplicaRecoveryInfo replica1 = new ReplicaRecoveryInfo(BLOCK_ID,
+REPLICA_LEN1, GEN_STAMP - 1, ReplicaState.FINALIZED);
+ReplicaRecoveryInfo replica2 = new ReplicaRecoveryInfo(BLOCK_ID,
+REPLICA_LEN1, GEN_STAMP - 2, ReplicaState.FINALIZED);
Review comment:
Indeed I also did not want to mix checkstyle with actual test fix,
however only after
[GitHub] [hadoop] qizhu-lucas commented on pull request #2829: HDFS-15930: Fix some @param errors in DirectoryScanner.
qizhu-lucas commented on pull request #2829: URL: https://github.com/apache/hadoop/pull/2829#issuecomment-812964703 Thanks @ayushtkn for commit and review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2860: Test Pre-Commits.
hadoop-yetus commented on pull request #2860:
URL: https://github.com/apache/hadoop/pull/2860#issuecomment-812963302
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 45s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 13s | | trunk passed |
| +1 :green_heart: | compile | 5m 12s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 4m 47s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 59s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 14s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 3s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 6m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 16s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 48s | | the patch passed |
| +1 :green_heart: | compile | 5m 37s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 5m 37s | | the patch passed |
| +1 :green_heart: | compile | 4m 51s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 4m 51s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 13s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 43s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 56s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 44s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 7m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 11s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 19s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 237m 3s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2860/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 18m 14s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not
generate ASF License warnings. |
| | | 383m 33s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestBlockRecovery |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2860/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2860 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 38ac97a7b311 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28
05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c10691f9e2ab39f9cea3e4b1c7317480d55484fb |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results |
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17314385#comment-17314385 ] Duo Zhang commented on HADOOP-16206: [~ahussein] So you suggest we do a benchmark first to get a clear evidence of performance gains in log4j2, and then start our work? Or you mean we could just start the work as you think we'd better fully move to log4j2? Thanks. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17612) Bump default Zookeeper version to 3.7.0
[ https://issues.apache.org/jira/browse/HADOOP-17612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17314362#comment-17314362 ] Ayush Saxena commented on HADOOP-17612: --- bq. The best we can do as of this point is bump up Zookeeper to 3.6 and curator to the appropriate version that uses 3.6. That sound like a plan, we can go ahead this way, and if we face some issues, from zk or curator side, we can bug the respective teams and see if they can help us in the upcoming release > Bump default Zookeeper version to 3.7.0 > --- > > Key: HADOOP-17612 > URL: https://issues.apache.org/jira/browse/HADOOP-17612 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > We can bump Zookeeper version to 3.7.0 for trunk. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17619) Fix DelegationTokenRenewer#updateRenewalTime java doc error.
[
https://issues.apache.org/jira/browse/HADOOP-17619?focusedWorklogId=576501=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-576501
]
ASF GitHub Bot logged work on HADOOP-17619:
---
Author: ASF GitHub Bot
Created on: 03/Apr/21 21:57
Start Date: 03/Apr/21 21:57
Worklog Time Spent: 10m
Work Description: ayushtkn commented on a change in pull request #2846:
URL: https://github.com/apache/hadoop/pull/2846#discussion_r606714993
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegationTokenRenewer.java
##
@@ -107,7 +107,7 @@ public boolean equals(final Object that) {
* Set a new time for the renewal.
* It can only be called when the action is not in the queue or any
* collection because the hashCode may change
- * @param newTime the new time
+ * @param delay the renew cycle
Review comment:
Shouldn't this be.
`the renewal time`
rather than `the renew cycle`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 576501)
Time Spent: 50m (was: 40m)
> Fix DelegationTokenRenewer#updateRenewalTime java doc error.
>
>
> Key: HADOOP-17619
> URL: https://issues.apache.org/jira/browse/HADOOP-17619
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Qi Zhu
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The param of updateRenewalTime should be the renew cycle, not the new time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #2846: HADOOP-17619: Fix DelegationTokenRenewer#updateRenewalTime java doc e…
ayushtkn commented on a change in pull request #2846:
URL: https://github.com/apache/hadoop/pull/2846#discussion_r606714993
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegationTokenRenewer.java
##
@@ -107,7 +107,7 @@ public boolean equals(final Object that) {
* Set a new time for the renewal.
* It can only be called when the action is not in the queue or any
* collection because the hashCode may change
- * @param newTime the new time
+ * @param delay the renew cycle
Review comment:
Shouldn't this be.
`the renewal time`
rather than `the renew cycle`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #2833: HDFS-15934: Make DirectoryScanner reconcile blocks batch size and int…
ayushtkn commented on a change in pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833#discussion_r606714508
##
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
##
@@ -873,6 +873,22 @@
+
+ dfs.datanode.reconcile.blocks.batch.size
+ 1000
+ Run reconcile to checkAndUpdate with batch,
Review comment:
can you add some recheck the descriptions for both the configs, the
first line in. both is same, and doesn't make sense to me, recheck once.
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
##
@@ -316,6 +317,18 @@ public DirectoryScanner(FsDatasetSpi dataset,
Configuration conf) {
masterThread =
new ScheduledThreadPoolExecutor(1, new Daemon.DaemonFactory());
+
+reconcileBlocksBatchSize =
+conf.getInt(DFSConfigKeys.
+DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE,
+DFSConfigKeys.
+DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE_DEFAULT);
+
+reconcileBlocksBatchInterval =
+conf.getInt(DFSConfigKeys.
+DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL,
+DFSConfigKeys.
+DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL_DEFAULT);
Review comment:
Add a validation for these configs, if ``reconcileBlocksBatchSize`` and
``reconcileBlocksBatchInterval`` is less than one use default. and add a warn
log message if these values are incorrect something like:
Invalid value configured for < config name>, should be greater than 0, Using
default.
In the end add an Info log for the values being used.
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
##
@@ -836,6 +836,10 @@
public static final int DFS_DATANODE_DIRECTORYSCAN_INTERVAL_DEFAULT =
21600;
public static final String DFS_DATANODE_DIRECTORYSCAN_THREADS_KEY =
"dfs.datanode.directoryscan.threads";
public static final int DFS_DATANODE_DIRECTORYSCAN_THREADS_DEFAULT = 1;
+ public static final String DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE =
"dfs.datanode.reconcile.blocks.batch.size";
+ public static final int DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE_DEFAULT
= 1000;
+ public static final String DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL =
"dfs.datanode.reconcile.blocks.batch.interval";
+ public static final int
DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL_DEFAULT = 2000;
Review comment:
Can we add support for time units?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on pull request #2829: HDFS-15930: Fix some @param errors in DirectoryScanner.
ayushtkn commented on pull request #2829: URL: https://github.com/apache/hadoop/pull/2829#issuecomment-812929534 Thanx @qizhu-lucas for the contribution, @Hexiaoqiao for the review. @qizhu-lucas, I missed your message, sorry. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn merged pull request #2829: HDFS-15930: Fix some @param errors in DirectoryScanner.
ayushtkn merged pull request #2829: URL: https://github.com/apache/hadoop/pull/2829 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #2844: HDFS-15940. Fixing and refactoring tests specific to Block recovery
ayushtkn commented on a change in pull request #2844:
URL: https://github.com/apache/hadoop/pull/2844#discussion_r606712644
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery1.java
##
@@ -336,191 +322,194 @@ public void testFinalizedReplicas () throws IOException
{
if(LOG.isDebugEnabled()) {
LOG.debug("Running " + GenericTestUtils.getMethodName());
}
-ReplicaRecoveryInfo replica1 = new ReplicaRecoveryInfo(BLOCK_ID,
-REPLICA_LEN1, GEN_STAMP-1, ReplicaState.FINALIZED);
-ReplicaRecoveryInfo replica2 = new ReplicaRecoveryInfo(BLOCK_ID,
-REPLICA_LEN1, GEN_STAMP-2, ReplicaState.FINALIZED);
+ReplicaRecoveryInfo replica1 = new ReplicaRecoveryInfo(BLOCK_ID,
+REPLICA_LEN1, GEN_STAMP - 1, ReplicaState.FINALIZED);
+ReplicaRecoveryInfo replica2 = new ReplicaRecoveryInfo(BLOCK_ID,
+REPLICA_LEN1, GEN_STAMP - 2, ReplicaState.FINALIZED);
Review comment:
Can we chunk out the formatting changes in this file? We can restrict
ourselves to only the intended changes, Usually we aren't mixing checkstyle
fixes with the other code changes..
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery1.java
##
@@ -122,9 +108,9 @@
/**
* This tests if sync all replicas in block recovery works correctly.
*/
-public class TestBlockRecovery {
+public class TestBlockRecovery1 {
Review comment:
Lets not bother the name, Let it be as is.
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery2.java
##
@@ -0,0 +1,464 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.datanode;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.ha.HAServiceProtocol;
+import org.apache.hadoop.hdfs.AppendTestUtil;
+import org.apache.hadoop.hdfs.DFSClient;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.DFSTestUtil;
+import org.apache.hadoop.hdfs.DistributedFileSystem;
+import org.apache.hadoop.hdfs.HdfsConfiguration;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.hdfs.StripedFileTestUtil;
+import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
+import org.apache.hadoop.hdfs.protocol.ErasureCodingPolicy;
+import org.apache.hadoop.hdfs.protocol.LocatedBlock;
+import
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB;
+import org.apache.hadoop.hdfs.server.namenode.FSNamesystem;
+import org.apache.hadoop.hdfs.server.protocol.BlockRecoveryCommand;
+import org.apache.hadoop.hdfs.server.protocol.DatanodeCommand;
+import org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration;
+import org.apache.hadoop.hdfs.server.protocol.HeartbeatResponse;
+import org.apache.hadoop.hdfs.server.protocol.NNHAStatusHeartbeat;
+import org.apache.hadoop.hdfs.server.protocol.NamenodeProtocols;
+import org.apache.hadoop.hdfs.server.protocol.NamespaceInfo;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.util.AutoCloseableLock;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TestName;
+import org.mockito.Mockito;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static
[GitHub] [hadoop] ayushtkn commented on a change in pull request #2831: HDFS-15920.Solve the problem that the value of SafeModeMonitor#RECHECK_INTERVAL can be configured.
ayushtkn commented on a change in pull request #2831:
URL: https://github.com/apache/hadoop/pull/2831#discussion_r606710952
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerSafeMode.java
##
@@ -640,7 +644,18 @@ private void reportStatus(String msg, boolean rightNow) {
*/
private class SafeModeMonitor implements Runnable {
/** Interval in msec for checking safe mode. */
-private static final long RECHECK_INTERVAL = 1000;
+private long recheckInterval;
+
+public SafeModeMonitor(Configuration conf) {
+ recheckInterval = conf.getLong(
+ DFS_NAMENODE_SAFEMODE_RECHECK_INTERVAL_KEY,
+ DFS_NAMENODE_SAFEMODE_RECHECK_INTERVAL_DEFAULT);
+ if (recheckInterval < 1) {
+LOG.warn("The current value of recheckInterval is {}, " +
+"this variable should be a positive number.", recheckInterval);
Review comment:
Can you change the message as ```"Invalid value for " +
DFS_NAMENODE_SAFEMODE_RECHECK_INTERVAL_KEY + ". Should be greater than 0, but
is {}", recheckInterval```
Please correct the syntax
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerSafeMode.java
##
@@ -640,7 +644,18 @@ private void reportStatus(String msg, boolean rightNow) {
*/
private class SafeModeMonitor implements Runnable {
/** Interval in msec for checking safe mode. */
-private static final long RECHECK_INTERVAL = 1000;
+private long recheckInterval;
+
+public SafeModeMonitor(Configuration conf) {
+ recheckInterval = conf.getLong(
+ DFS_NAMENODE_SAFEMODE_RECHECK_INTERVAL_KEY,
+ DFS_NAMENODE_SAFEMODE_RECHECK_INTERVAL_DEFAULT);
+ if (recheckInterval < 1) {
+LOG.warn("The current value of recheckInterval is {}, " +
+"this variable should be a positive number.", recheckInterval);
+recheckInterval = DFS_NAMENODE_SAFEMODE_RECHECK_INTERVAL_DEFAULT;
+ }
Review comment:
Add a Log.info("Using {} as SafeModeMonitor Interval", recheckInterval)
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManagerSafeMode.java
##
@@ -230,6 +232,50 @@ public void testCheckSafeMode8() throws Exception {
assertEquals(BMSafeModeStatus.OFF, getSafeModeStatus());
}
+ @Test(timeout = 2)
+ public void testCheckSafeMode9() throws Exception {
+Configuration conf = new HdfsConfiguration();
+try {
+ conf.setLong(DFS_NAMENODE_SAFEMODE_RECHECK_INTERVAL_KEY, 3000);
+ bm = spy(new BlockManager(fsn, false, conf));
+ doReturn(true).when(bm).isGenStampInFuture(any(Block.class));
+ dn = spy(bm.getDatanodeManager());
+ Whitebox.setInternalState(bm, "datanodeManager", dn);
+ // the datanode threshold is always met
+ when(dn.getNumLiveDataNodes()).thenReturn(DATANODE_NUM);
+ bmSafeMode = new BlockManagerSafeMode(bm, fsn, false, conf);
+ bmSafeMode.activate(BLOCK_TOTAL);
+ Whitebox.setInternalState(bmSafeMode, "extension", Integer.MAX_VALUE);
+ setSafeModeStatus(BMSafeModeStatus.PENDING_THRESHOLD);
+ setBlockSafe(BLOCK_THRESHOLD);
+ bmSafeMode.checkSafeMode();
+
+ assertTrue(bmSafeMode.isInSafeMode());
+ assertEquals(BMSafeModeStatus.EXTENSION, getSafeModeStatus());
+
+ GenericTestUtils.waitFor(new Supplier() {
+@Override
+public Boolean get() {
+ Whitebox.setInternalState(bmSafeMode, "extension", 0);
+ return getSafeModeStatus() != BMSafeModeStatus.EXTENSION;
+}
+ }, EXTENSION / 10, EXTENSION * 10);
+
+ assertFalse(bmSafeMode.isInSafeMode());
+ assertEquals(BMSafeModeStatus.OFF, getSafeModeStatus());
+} finally {
+ conf.setLong(DFS_NAMENODE_SAFEMODE_RECHECK_INTERVAL_KEY,
+ DFS_NAMENODE_SAFEMODE_RECHECK_INTERVAL_DEFAULT);
+ bm = spy(new BlockManager(fsn, false, conf));
+ doReturn(true).when(bm).isGenStampInFuture(any(Block.class));
+ dn = spy(bm.getDatanodeManager());
+ Whitebox.setInternalState(bm, "datanodeManager", dn);
+ // the datanode threshold is always met
+ when(dn.getNumLiveDataNodes()).thenReturn(DATANODE_NUM);
+ bmSafeMode = new BlockManagerSafeMode(bm, fsn, false, conf);
+}
Review comment:
The tests have too many warnings due to deprecation. See if you can get
rid of them, If not, just do a assert on the log output, That it contains
``Using as SafeModeMonitor Interval``
``GenericTestUtils`` has a ``LogCapturer`` you can use that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2860: Test Pre-Commits.
hadoop-yetus commented on pull request #2860:
URL: https://github.com/apache/hadoop/pull/2860#issuecomment-812922960
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 23s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 35s | | trunk passed |
| +1 :green_heart: | compile | 5m 36s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 5m 11s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 6s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 21s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 3s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 7m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 57s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 33s | | the patch passed |
| +1 :green_heart: | compile | 4m 49s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 4m 49s | | the patch passed |
| +1 :green_heart: | compile | 4m 27s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 4m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 10s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 55s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 44s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 7m 31s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 33s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 17s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 233m 43s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2860/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 18m 0s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2860/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 380m 39s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestBlockRecovery |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2860/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2860 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux d0d12c140194 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28
05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c10691f9e2ab39f9cea3e4b1c7317480d55484fb |
| Default Java | Private
[GitHub] [hadoop] ayushtkn commented on pull request #2852: MAPREDUCE-7287. Distcp will delete exists file , If we use "--delete …
ayushtkn commented on pull request #2852:
URL: https://github.com/apache/hadoop/pull/2852#issuecomment-812918934
Apart from the checkstyle issue, changes LGTM.
Removal of code shouldn't bother, it is used in ``validatePaths`` which
won't break by passing the actual dest.
In the present code we were passing ``/NONE`` as target which actually
didn't exist considering we would be least bothered with the target path, when
building listing, but the problem seems to happen at
```SimpleCopyListing#computeSourceRootPath``` when the source is a ``solitary
file``, It has check which relies on target:
```
if (solitaryFile) {
if (!targetPathExists || targetFS.isFile(target)) {
return sourceStatus.getPath();
} else {
return sourceStatus.getPath().getParent();
}
```
So, here the ``targetFS.isFile(target)`` returns `false` due to ``FNF`` on
``/NONE`` which wasn't in the case when building listing for ``source``, so for
source the listing is built with ``sourceStatus.getPath();`` and for target
with ``sourceStatus.getPath().getParent()``, hence on comparison in
``CopyCommitter`` file gets deleted.
So, this passing `/NONE` wasn't a good idea, The present fix to pass the
actual destination seems fair enough. Just give a check, guess these constants
can be removed:
These constants can be removed now?
```
/**
* Constants for NONE file deletion
*/
public static final String NONE_PATH_NAME = "/NONE";
public static final Path NONE_PATH = new Path(NONE_PATH_NAME);
public static final Path RAW_NONE_PATH = new Path(
DistCpConstants.HDFS_RESERVED_RAW_DIRECTORY_NAME + NONE_PATH_NAME);
```
Nice Catch!!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2852: MAPREDUCE-7287. Distcp will delete exists file , If we use "--delete …
steveloughran commented on pull request #2852: URL: https://github.com/apache/hadoop/pull/2852#issuecomment-812887968 going to have to pull in people who know more of the distcp code to review this, in particular, someone needs to understand what the code being removed is meant to be doing. Tests look excellent, BTW. @ayushtkn -you got time to look at this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17615) ADLFS: Update SDK version from 2.3.6 to 2.3.9
[ https://issues.apache.org/jira/browse/HADOOP-17615?focusedWorklogId=576435=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-576435 ] ASF GitHub Bot logged work on HADOOP-17615: --- Author: ASF GitHub Bot Created on: 03/Apr/21 15:06 Start Date: 03/Apr/21 15:06 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2842: URL: https://github.com/apache/hadoop/pull/2842#issuecomment-812877682 what's causing the test failures? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 576435) Time Spent: 20m (was: 10m) > ADLFS: Update SDK version from 2.3.6 to 2.3.9 > - > > Key: HADOOP-17615 > URL: https://issues.apache.org/jira/browse/HADOOP-17615 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Affects Versions: 3.3.0 >Reporter: Bilahari T H >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2842: HADOOP-17615. ADLS Gen1: Update adls SDK to 2.3.9
steveloughran commented on pull request #2842: URL: https://github.com/apache/hadoop/pull/2842#issuecomment-812877682 what's causing the test failures? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn opened a new pull request #2860: Test Pre-Commits.
ayushtkn opened a new pull request #2860: URL: https://github.com/apache/hadoop/pull/2860 Checking HDFS Pre-Commit builds, to get hold of the flaky tests. Please ignore for reviewing. If any fix is found for any test, please push directly to the PR, or drop a comment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2859: HDFS-15951. Remove unused parameters in NameNodeProxiesClient
hadoop-yetus commented on pull request #2859:
URL: https://github.com/apache/hadoop/pull/2859#issuecomment-812868839
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 13m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 44s | | trunk passed |
| +1 :green_heart: | compile | 0m 58s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 51s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 58s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 38s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 48s | | the patch passed |
| +1 :green_heart: | compile | 0m 49s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 49s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 46s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 24s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 21s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 21s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 93m 24s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2859/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2859 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 9fea1b902a73 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9c998ad7344e7a9aca44f24fb18e69451d40 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2859/1/testReport/ |
| Max. process+thread count | 675 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2859/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[GitHub] [hadoop] hadoop-yetus commented on pull request #2856: HDFS-15947. Replace deprecated protobuf APIs
hadoop-yetus commented on pull request #2856: URL: https://github.com/apache/hadoop/pull/2856#issuecomment-812863295 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomscut opened a new pull request #2859: HDFS-15951. Remove unused parameters in NameNodeProxiesClient
tomscut opened a new pull request #2859: URL: https://github.com/apache/hadoop/pull/2859 JIRA: [HDFS-15951](https://issues.apache.org/jira/browse/HDFS-15951) Remove unused parameters in org.apache.hadoop.hdfs.NameNodeProxiesClient. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #1971: MAPREDUCE-7276 fix hadoop job fast fail not working issue
hadoop-yetus commented on pull request #1971:
URL: https://github.com/apache/hadoop/pull/1971#issuecomment-812846675
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 21m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 0s | | trunk passed |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 23s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 10s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 23s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 24s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 8m 17s | | hadoop-mapreduce-client-app in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 106m 33s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1971/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/1971 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux ccdba38cb559 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28
05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 51ddb09e57b6e327b42819ca2cc3bb9a9f659eeb |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1971/1/testReport/ |
| Max. process+thread count | 560 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1971/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2858: HDFS-15950. Remove unused hdfs.proto import
hadoop-yetus commented on pull request #2858:
URL: https://github.com/apache/hadoop/pull/2858#issuecomment-812842463
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 20m 56s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 51s | | trunk passed |
| +1 :green_heart: | compile | 1m 0s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 49s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | mvnsite | 0m 56s | | trunk passed |
| +1 :green_heart: | shadedclient | 53m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 48s | | the patch passed |
| +1 :green_heart: | compile | 0m 52s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | cc | 0m 52s | | the patch passed |
| +1 :green_heart: | javac | 0m 52s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | cc | 0m 43s | | the patch passed |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 46s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 4s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 15s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 98m 23s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2858/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2858 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit buflint
bufcompat codespell |
| uname | Linux 5bd2823b76f8 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28
05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d2ccebe2fed75dcbff3974d173b70015f9b54311 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2858/1/testReport/ |
| Max. process+thread count | 514 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2858/1/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2857: HDFS-15949. Fix integer overflow
hadoop-yetus commented on pull request #2857:
URL: https://github.com/apache/hadoop/pull/2857#issuecomment-812833238
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 13m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 51s | | trunk passed |
| +1 :green_heart: | compile | 2m 41s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 3m 14s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | mvnsite | 0m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 52m 39s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 16s | | the patch passed |
| +1 :green_heart: | compile | 2m 32s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | cc | 2m 32s | |
hadoop-hdfs-project_hadoop-hdfs-native-client-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 0 new + 8 unchanged
- 3 fixed = 8 total (was 11) |
| +1 :green_heart: | golang | 2m 32s | | the patch passed |
| +1 :green_heart: | javac | 2m 32s | | the patch passed |
| +1 :green_heart: | compile | 2m 38s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | cc | 2m 38s | |
hadoop-hdfs-project_hadoop-hdfs-native-client-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 0 new
+ 8 unchanged - 3 fixed = 8 total (was 11) |
| +1 :green_heart: | golang | 2m 38s | | the patch passed |
| +1 :green_heart: | javac | 2m 38s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 8s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 31m 55s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 119m 39s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2857/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2857 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell golang |
| uname | Linux b47fa8d047cc 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e93e361f757fe8b8ddb9b3c42c225f8ce5783dc3 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2857/1/testReport/ |
| Max. process+thread count | 609 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2857/1/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail:
[GitHub] [hadoop] GauthamBanasandra commented on pull request #2858: HDFS-15950. Remove unused hdfs.proto import
GauthamBanasandra commented on pull request #2858: URL: https://github.com/apache/hadoop/pull/2858#issuecomment-812832629 This PR fixes a warning reported in PR https://github.com/apache/hadoop/pull/2792. ``` inotify.proto:35:1: warning: Import hdfs.proto is unused. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] GauthamBanasandra opened a new pull request #2858: HDFS-15950. Remove unused hdfs.proto import
GauthamBanasandra opened a new pull request #2858: URL: https://github.com/apache/hadoop/pull/2858 * hdfs.proto is imported in inotify.proto and is unused. Thus, removing it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] GauthamBanasandra commented on pull request #2857: HDFS-15949. Fix integer overflow
GauthamBanasandra commented on pull request #2857: URL: https://github.com/apache/hadoop/pull/2857#issuecomment-812820177 We can observe that `Max int64_t + 1` and `Min int64_t` are equal. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] GauthamBanasandra commented on pull request #2857: HDFS-15949. Fix integer overflow
GauthamBanasandra commented on pull request #2857:
URL: https://github.com/apache/hadoop/pull/2857#issuecomment-812820088
I was able to verify this change using the following standalone C++ program -
```cpp
#include
#include
#include
int main(int argc, char *argv[]) {
std::cout << "Max int64_t = " << std::numeric_limits::max()
<< std::endl;
std::cout << "Max int64_t + 1 = " << std::numeric_limits::max() +
1
<< std::endl;
std::cout << "Min int64_t = " << std::numeric_limits::min()
<< std::endl;
return 0;
}
```
Output -
```
Max int64_t = 9223372036854775807
Max int64_t + 1 = -9223372036854775808
Min int64_t = -9223372036854775808
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] GauthamBanasandra commented on pull request #2857: HDFS-15949. Fix integer overflow
GauthamBanasandra commented on pull request #2857: URL: https://github.com/apache/hadoop/pull/2857#issuecomment-812819946 This PR fixes some warnings reported as part of https://github.com/apache/hadoop/pull/2792. ``` [ 63%] Building CXX object main/native/libhdfspp/tests/CMakeFiles/hdfs_ext_hdfspp_test_shim_static.dir/hdfs_ext_test.cc.o In file included from /mnt/d/projects/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/out/build/WSL-GCC-Debug/main/native/libhdfspp/googletest-src/googletest/include/gtest/gtest.h:375, from /mnt/d/projects/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/out/build/WSL-GCC-Debug/main/native/libhdfspp/googletest-src/googlemock/include/gmock/internal/gmock-internal-utils.h:47, from /mnt/d/projects/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/out/build/WSL-GCC-Debug/main/native/libhdfspp/googletest-src/googlemock/include/gmock/gmock-actions.h:51, from /mnt/d/projects/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/out/build/WSL-GCC-Debug/main/native/libhdfspp/googletest-src/googlemock/include/gmock/gmock.h:59, from /mnt/d/projects/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfspp_mini_dfs.h:24, from /mnt/d/projects/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_ext_test.cc:19: /mnt/d/projects/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_ext_test.cc: In member function ‘virtual void hdfs::HdfsExtTest_TestHosts_Test::TestBody()’: /mnt/d/projects/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_ext_test.cc:456:95: warning: integer overflow in expression of type ‘long int’ results in ‘-9223372036854775808’ [-Woverflow] 456 | EXPECT_EQ(nullptr, hdfsGetHosts(fs, filename.c_str(), 0, std::numeric_limits::max()+1)); | ~~~^~ /mnt/d/projects/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_ext_test.cc:460:92: warning: integer overflow in expression of type ‘long int’ results in ‘-9223372036854775808’ [-Woverflow] 460 | EXPECT_EQ(nullptr, hdfsGetHosts(fs, filename.c_str(), std::numeric_limits::max()+1, std::numeric_limits::max())); | ~~~^~ ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] GauthamBanasandra opened a new pull request #2857: HDFS-15949. Fix integer overflow
GauthamBanasandra opened a new pull request #2857: URL: https://github.com/apache/hadoop/pull/2857 * There are some instance where the overflow is deliberately done in order to get the lower bound. This results in an overflow warning. * We fix this by using the min value directly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
