Re: [VOTE] Release Apache Hadoop 2.8.4 (RC0)

[Mingliang Liu]

+1 (binding)

1. Download src file, and check md5
2. Build package from src
3. Create a 6 node cluster in Docker containers and start the HDFS/YARN
services (The script is caochong ).
4. Verify the running version
5. Run simple HDFS/YARN client/admin commands and verify the output
6. Run example programs wordcount and grep
7. Check NN/DN/RM status and logs

Thanks Junping for driving this!

On Tue, May 8, 2018 at 10:41 AM, 俊平堵  wrote:

> Hi all,
>  I've created the first release candidate (RC0) for Apache Hadoop
> 2.8.4. This is our next maint release to follow up 2.8.3. It includes 77
> important fixes and improvements.
>
> The RC artifacts are available at:
> http://home.apache.org/~junping_du/hadoop-2.8.4-RC0
>
> The RC tag in git is: release-2.8.4-RC0
>
> The maven artifacts are available via repository.apache.org<
> http://repository.apache.org> at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1118
>
> Please try the release and vote; the vote will run for the usual 5
> working days, ending on 5/14/2018 PST time.
>
> Thanks,
>
> Junping
>



-- 
Mingliang Liu

Apache Hadoop qbt Report: trunk+JDK9 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java9-linux-x86/23/

[May 6, 2018 4:15:14 PM] (aajisaka) MAPREDUCE-7093. Use assertEquals instead of 
assertTrue(a == b) in
[May 7, 2018 3:33:14 AM] (wwei) YARN-8025. 
UsersManangers#getComputedResourceLimitForActiveUsers throws
[May 7, 2018 10:54:08 AM] (stevel) HADOOP-15446. WASB: PageBlobInputStream.skip 
breaks HBASE replication.
[May 7, 2018 8:32:27 PM] (xiao) Revert "HDFS-13430. Fix 
TestEncryptionZonesWithKMS failure due to
[May 7, 2018 8:32:27 PM] (xiao) Revert "HADOOP-14445. Delegation tokens are not 
shared between KMS
[May 7, 2018 9:58:52 PM] (aengineer) HDDS-1. Remove SCM Block DB. Contributed 
by Xiaoyu Yao.
[May 7, 2018 10:36:29 PM] (xiao) HDFS-12981. renameSnapshot a Non-Existent 
snapshot to itself should
[May 8, 2018 1:29:50 AM] (xyao) HDDS-27. Fix 
TestStorageContainerManager#testBlockDeletionTra;nsactions.
[May 8, 2018 5:05:19 AM] (sunilg) YARN-5151. [UI2] Support kill application 
from new YARN UI. Contributed
[May 8, 2018 6:07:03 AM] (sunilg) YARN-8251. [UI2] Clicking on Application link 
at the header goes to
[May 8, 2018 6:58:54 AM] (rohithsharmaks) YARN-8253. HTTPS Ats v2 api call 
fails with 'bad HTTP parsed'.
[May 8, 2018 12:41:33 PM] (msingh) HDDS-23. Remove SCMNodeAddressList from 
SCMRegisterRequestProto.
[May 8, 2018 6:34:45 PM] (wangda) YARN-8232. RMContainer lost queue name when 
RM HA happens. (Hu Ziqian
[May 8, 2018 8:30:36 PM] (jlowe) YARN-8207. Docker container launch use popen 
have risk of shell
[May 8, 2018 8:49:41 PM] (billie) YARN-7894. Improve ATS response for DShell 
DS_CONTAINER when container
[May 8, 2018 10:29:54 PM] (fabbri) HADOOP-13649 s3guard: implement time-based 
(TTL) expiry for
[May 9, 2018 1:58:42 AM] (fabbri) HADOOP-15420 s3guard ITestS3GuardToolLocal 
failures in diff tests (Gabor
[May 9, 2018 4:51:41 AM] (msingh) HDDS-33. Ozone : Fix the test logic in
[May 9, 2018 6:30:43 AM] (rohithsharmaks) YARN-8239. [UI2] Clicking on Node 
Manager UI under AM container info /
[May 9, 2018 6:34:31 AM] (rohithsharmaks) YARN-8264. [UI2] GPU Info tab 
disappears if we click any sub link under
[May 9, 2018 6:47:04 AM] (rohithsharmaks) YARN-8260. [UI2] Per-application 
tracking URL is no longer available in
[May 9, 2018 1:36:07 PM] (msingh) HDDS-20. Ozone: Add support for rename key 
within a bucket for rpc
[May 9, 2018 2:50:21 PM] (msingh) HDDS-28. Ozone:Duplicate declaration in
[May 9, 2018 5:32:51 PM] (eyang) YARN-8261.  Fixed a bug in creation of 
localized container directory.   
[May 9, 2018 9:15:51 PM] (mackrorysd) HADOOP-15356. Make HTTP timeout 
configurable in ADLS connector.
[May 9, 2018 11:52:09 PM] (inigoiri) HDFS-13537. TestHdfsHelper does not 
generate jceks path properly for
[May 10, 2018 3:45:46 AM] (bibinchundatt) YARN-8201. Skip stacktrace of few 
exception from ClientRMService.
[May 10, 2018 5:17:48 AM] (vrushali) YARN-8247 Incorrect HTTP status code 
returned by ATSv2 for
[May 10, 2018 6:00:13 AM] (aengineer) HDDS-18. Ozone Shell should use 
RestClient and RpcClient. Contributed by
[May 10, 2018 9:38:08 AM] (aajisaka) HADOOP-15354. hadoop-aliyun & hadoop-azure 
modules to mark hadoop-common
[May 10, 2018 4:31:59 PM] (rkanter) YARN-8202. DefaultAMSProcessor should 
properly check units of requested
[May 10, 2018 4:41:16 PM] (inigoiri) HADOOP-15454. 
TestRollingFileSystemSinkWithLocal fails on Windows.
[May 10, 2018 5:46:55 PM] (haibochen) MAPREDUCE-7095. Race conditions in 
closing FadvisedChunkedFile. (Miklos
[May 10, 2018 6:01:01 PM] (haibochen) YARN-7715. Support NM promotion/demotion 
of running containers. (Miklos
[May 10, 2018 6:36:25 PM] (aengineer) HDDS-30. Fix TestContainerSQLCli. 
Contributed by Shashikant Banerjee.
[May 10, 2018 6:44:14 PM] (aengineer) HDDS-42. Inconsistent module names and 
descriptions. Contributed by Tsz
[May 10, 2018 7:43:13 PM] (aengineer) HDDS-31. Fix TestSCMCli. Contributed by 
Lokesh Jain.
[May 10, 2018 9:49:58 PM] (xyao) HDDS-16. Remove Pipeline from Datanode 
Container Protocol protobuf
[May 10, 2018 11:27:21 PM] (aengineer) HDDS-37. Remove dependency of 
hadoop-hdds-common and
[May 11, 2018 12:08:26 AM] (aengineer) HDDS-34. Remove .meta file during 
creation of container Contributed by
[May 11, 2018 12:24:40 AM] (Bharat) HDDS-43: Rename hdsl to hdds in 
hadoop-ozone/acceptance-test/README.md.
[May 11, 2018 2:05:35 AM] (vinodkv) YARN-8249. Fixed few REST APIs in 
RMWebServices to have static-user
[May 11, 2018 2:47:04 AM] (wwei) YARN-7003. DRAINING state of queues is not 
recovered after RM restart.
[May 11, 2018 6:51:30 AM] (yqlin) HDFS-13346. RBF: Fix synchronization of 
router quota and nameservice
[May 11, 2018 4:47:57 PM] (inigoiri) HDFS-13542. 
TestBlockManager#testNeededReplicationWhileAppending fails
[May 11, 2018 5:20:04 PM] (aengineer) HDDS-47. Add acceptance tests for Ozone 
Shell. Contributed by Lokesh
[May 11, 2018 6:08:45 PM] (aengineer) HDDS-39. Ozone: Compile Ozone/HDFS/Cblock 
protobuf files with proto3
[May 11, 2018 6:23:16 

[jira] [Created] (HDDS-53) Fix TestKey#testPutAndGetKeyWithDnRestart

[Mukul Kumar Singh (JIRA)]

Mukul Kumar Singh created HDDS-53:
-

 Summary: Fix TestKey#testPutAndGetKeyWithDnRestart
 Key: HDDS-53
 URL: https://issues.apache.org/jira/browse/HDDS-53
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Mukul Kumar Singh


Test TestKey#testPutAndGetKeyWithDnRestart is failing with the following 
exception.

{code}
org.apache.hadoop.ozone.client.rest.OzoneException: Exception getting 
XceiverClient: java.util.concurrent.ExecutionException: 
java.net.ConnectException: Connection refused: bc7c9fbfaa65/172.17.0.2:58706
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call(AnnotatedConstructor.java:119)
at 
com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createUsingDefault(StdValueInstantiator.java:270)
at 
com.fasterxml.jackson.databind.deser.std.ThrowableDeserializer.deserializeFromObject(ThrowableDeserializer.java:149)
at 
com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:159)
at 
com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:1611)
at 
com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1219)
at 
org.apache.hadoop.ozone.client.rest.OzoneException.parse(OzoneException.java:265)
at 
org.apache.hadoop.ozone.web.client.OzoneBucket.executeGetKey(OzoneBucket.java:401)
at 
org.apache.hadoop.ozone.web.client.OzoneBucket.getKey(OzoneBucket.java:327)
at 
org.apache.hadoop.ozone.web.client.TestKeys.runTestPutAndGetKeyWithDnRestart(TestKeys.java:309)
at 
org.apache.hadoop.ozone.web.client.TestKeys.testPutAndGetKeyWithDnRestart(TestKeys.java:288)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-52) Fix TestSCMCli#testInfoContainer

[Mukul Kumar Singh (JIRA)]

Mukul Kumar Singh created HDDS-52:
-

 Summary: Fix TestSCMCli#testInfoContainer
 Key: HDDS-52
 URL: https://issues.apache.org/jira/browse/HDDS-52
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Mukul Kumar Singh


The test is failing with the following error 

{code}
java.util.MissingFormatArgumentException: Format specifier '%s'
at java.util.Formatter.format(Formatter.java:2519)
at java.util.Formatter.format(Formatter.java:2455)
at java.lang.String.format(String.java:2940)
at 
org.apache.hadoop.ozone.scm.TestSCMCli.testInfoContainer(TestSCMCli.java:334)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-51) Fix TestDeletedBlockLog#testDeletedBlockTransactions

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-51:
--

 Summary: Fix TestDeletedBlockLog#testDeletedBlockTransactions
 Key: HDDS-51
 URL: https://issues.apache.org/jira/browse/HDDS-51
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Xiaoyu Yao


This was found in recent run. 
https://builds.apache.org/job/PreCommit-HDDS-Build/87/testReport/org.apache.hadoop.hdds.scm.block/TestDeletedBlockLog/testDeletedBlockTransactions/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/779/

[May 11, 2018 2:05:35 AM] (vinodkv) YARN-8249. Fixed few REST APIs in 
RMWebServices to have static-user
[May 11, 2018 2:47:04 AM] (wwei) YARN-7003. DRAINING state of queues is not 
recovered after RM restart.
[May 11, 2018 6:51:30 AM] (yqlin) HDFS-13346. RBF: Fix synchronization of 
router quota and nameservice
[May 11, 2018 4:47:57 PM] (inigoiri) HDFS-13542. 
TestBlockManager#testNeededReplicationWhileAppending fails
[May 11, 2018 5:20:04 PM] (aengineer) HDDS-47. Add acceptance tests for Ozone 
Shell. Contributed by Lokesh
[May 11, 2018 6:08:45 PM] (aengineer) HDDS-39. Ozone: Compile Ozone/HDFS/Cblock 
protobuf files with proto3
[May 11, 2018 6:23:16 PM] (eyang) YARN-8274. Fixed a bug on docker start 
command.Contributed
[May 11, 2018 6:34:00 PM] (haibochen) YARN-8268. Fair scheduler: reservable 
queue is configured both as parent
[May 11, 2018 6:36:52 PM] (aengineer) HDDS-25. Simple async event processing 
for SCM. Contributed by Elek,
[May 11, 2018 7:07:32 PM] (jlowe) YARN-8244. 
TestContainerSchedulerQueuing.testStartMultipleContainers
[May 11, 2018 7:49:05 PM] (billie) YARN-8243. Flex down should remove instance 
with largest component
[May 11, 2018 9:07:55 PM] (haibochen) YARN-3610. FairScheduler: Add 
steady-fair-shares to the REST API
[May 11, 2018 9:24:36 PM] (aengineer) HDDS-40. Separating packaging of 
Ozone/HDDS from the main Hadoop.
[May 11, 2018 11:56:05 PM] (jlowe) YARN-7654. Support ENTRY_POINT for docker 
container. Contributed by Eric




-1 overall


The following subsystems voted -1:
asflicense compile findbugs unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-hdds/common 
   Useless control flow in 
org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$CloseContainerRequestProto$Builder.maybeForceBuilderInitialization()
 At ContainerProtos.java: At ContainerProtos.java:[line 17815] 
   Useless control flow in 
org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$CloseContainerResponseProto$Builder.maybeForceBuilderInitialization()
 At ContainerProtos.java: At ContainerProtos.java:[line 18363] 
   Useless control flow in 
org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$CopyContainerRequestProto$Builder.maybeForceBuilderInitialization()
 At ContainerProtos.java: At ContainerProtos.java:[line 34624] 
   Useless control flow in 
org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$CopyContainerResponseProto$Builder.maybeForceBuilderInitialization()
 At ContainerProtos.java: At ContainerProtos.java:[line 35479] 
   Useless control flow in 
org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$CreateContainerResponseProto$Builder.maybeForceBuilderInitialization()
 At ContainerProtos.java: At ContainerProtos.java:[line 12991] 
   Useless control flow in 
org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$DatanodeBlockID$Builder.maybeForceBuilderInitialization()
 At ContainerProtos.java: At ContainerProtos.java:[line 1112] 
   Useless control flow in 
org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$DeleteChunkResponseProto$Builder.maybeForceBuilderInitialization()
 At ContainerProtos.java: At ContainerProtos.java:[line 30029] 
   Useless control flow in 
org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$DeleteContainerRequestProto$Builder.maybeForceBuilderInitialization()
 At ContainerProtos.java: At ContainerProtos.java:[line 15580] 
   Useless control flow in 
org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$DeleteContainerResponseProto$Builder.maybeForceBuilderInitialization()
 At ContainerProtos.java: At ContainerProtos.java:[line 16042] 
   Useless control flow in 
org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$DeleteKeyResponseProto$Builder.maybeForceBuilderInitialization()
 At ContainerProtos.java: At ContainerProtos.java:[line 23085] 
   Useless control flow in 
org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$KeyValue$Builder.maybeForceBuilderInitialization()
 At ContainerProtos.java: At ContainerProtos.java:[line 1739] 
   Useless control flow in 
org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ListContainerRequestProto$Builder.maybeForceBuilderInitialization()
 At ContainerProtos.java: At ContainerProtos.java:[line 16530] 
   Useless control flow in 
org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ListKeyRequestProto$Builder.maybeForceBuilderInitialization()
 At ContainerProtos.java: At ContainerProtos.java:[line 23608] 
   Useless control flow in 

Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64

[Apache Jenkins Server]

For more details, see https://builds.apache.org/job/hadoop-trunk-win/465/

[May 11, 2018 4:47:57 PM] (inigoiri) HDFS-13542. 
TestBlockManager#testNeededReplicationWhileAppending fails
[May 11, 2018 5:20:04 PM] (aengineer) HDDS-47. Add acceptance tests for Ozone 
Shell. Contributed by Lokesh
[May 11, 2018 6:08:45 PM] (aengineer) HDDS-39. Ozone: Compile Ozone/HDFS/Cblock 
protobuf files with proto3
[May 11, 2018 6:23:16 PM] (eyang) YARN-8274. Fixed a bug on docker start 
command.Contributed
[May 11, 2018 6:34:00 PM] (haibochen) YARN-8268. Fair scheduler: reservable 
queue is configured both as parent
[May 11, 2018 6:36:52 PM] (aengineer) HDDS-25. Simple async event processing 
for SCM. Contributed by Elek,
[May 11, 2018 7:07:32 PM] (jlowe) YARN-8244. 
TestContainerSchedulerQueuing.testStartMultipleContainers
[May 11, 2018 7:49:05 PM] (billie) YARN-8243. Flex down should remove instance 
with largest component
[May 11, 2018 9:07:55 PM] (haibochen) YARN-3610. FairScheduler: Add 
steady-fair-shares to the REST API
[May 11, 2018 9:24:36 PM] (aengineer) HDDS-40. Separating packaging of 
Ozone/HDDS from the main Hadoop.
[May 11, 2018 11:56:05 PM] (jlowe) YARN-7654. Support ENTRY_POINT for docker 
container. Contributed by Eric
[May 12, 2018 5:37:43 AM] (eyang) YARN-8265.  Improve DNS handling on docker IP 
changes.
[May 12, 2018 12:10:29 PM] (msingh) HDDS-21. Add support for rename key within 
a bucket for rest client.




-1 overall


The following subsystems voted -1:
compile mvninstall unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


Specific tests:

Failed junit tests :

   hadoop.security.authentication.util.TestZKSignerSecretProvider 
   hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec 
   hadoop.fs.contract.rawlocal.TestRawlocalContractAppend 
   hadoop.fs.TestDelegationTokenRenewer 
   hadoop.fs.TestFsShellCopy 
   hadoop.fs.TestFsShellList 
   hadoop.fs.TestLocalFileSystem 
   hadoop.fs.TestLocalFSFileContextMainOperations 
   hadoop.fs.TestRawLocalFileSystemContract 
   hadoop.fs.TestTrash 
   hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem 
   hadoop.http.TestHttpServer 
   hadoop.http.TestHttpServerLogs 
   hadoop.io.nativeio.TestNativeIO 
   hadoop.ipc.TestSocketFactory 
   hadoop.metrics2.impl.TestStatsDMetrics 
   hadoop.security.TestGroupsCaching 
   hadoop.security.TestSecurityUtil 
   hadoop.security.TestShellBasedUnixGroupsMapping 
   hadoop.security.token.delegation.TestZKDelegationTokenSecretManager 
   hadoop.security.token.TestDtUtilShell 
   hadoop.util.curator.TestChildReaper 
   hadoop.util.TestNativeCodeLoader 
   hadoop.fs.TestResolveHdfsSymlink 
   hadoop.fs.http.server.TestHttpFSServerWebServer 
   hadoop.hdfs.nfs.nfs3.TestRpcProgramNfs3 
   hadoop.hdfs.nfs.nfs3.TestWrites 
   hadoop.fs.contract.router.TestRouterHDFSContractOpen 
   hadoop.fs.contract.router.TestRouterHDFSContractRename 
   hadoop.fs.contract.router.TestRouterHDFSContractRootDirectory 
   hadoop.fs.contract.router.TestRouterHDFSContractSeek 
   hadoop.fs.contract.router.TestRouterHDFSContractSetTimes 
   hadoop.fs.contract.router.web.TestRouterWebHDFSContractAppend 
   hadoop.fs.contract.router.web.TestRouterWebHDFSContractConcat 
   hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate 
   hadoop.yarn.server.webproxy.amfilter.TestAmFilter 
  

   mvninstall:

   
https://builds.apache.org/job/hadoop-trunk-win/464/artifact/out/patch-mvninstall-root.txt
  [824K]

   compile:

   
https://builds.apache.org/job/hadoop-trunk-win/464/artifact/out/patch-compile-root.txt
  [272K]

   cc:

   
https://builds.apache.org/job/hadoop-trunk-win/464/artifact/out/patch-compile-root.txt
  [272K]

   javac:

   
https://builds.apache.org/job/hadoop-trunk-win/464/artifact/out/patch-compile-root.txt
  [272K]

   unit:

   
https://builds.apache.org/job/hadoop-trunk-win/464/artifact/out/patch-unit-hadoop-common-project_hadoop-auth.txt
  [28K]
   
https://builds.apache.org/job/hadoop-trunk-win/464/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [372K]
   
https://builds.apache.org/job/hadoop-trunk-win/464/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [740K]
   
https://builds.apache.org/job/hadoop-trunk-win/464/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-trunk-win/464/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt
  [20K]
   
https://builds.apache.org/job/hadoop-trunk-win/464/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-nfs.txt
  [16K]
   
https://builds.apache.org/job/hadoop-trunk-win/464/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
  [40K]
   

[jira] [Resolved] (HDFS-8406) Lease recovery continually failed

[Wei-Chiu Chuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HDFS-8406.
---
Resolution: Duplicate

> Lease recovery continually failed
> -
>
> Key: HDFS-8406
> URL: https://issues.apache.org/jira/browse/HDFS-8406
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Keith Turner
>Priority: Major
>  Labels: Accumulo, HBase, SolrCloud
>
> While testing Accumulo on a cluster and killing processes, I ran into a 
> situation where the lease on an accumulo write ahead log in HDFS could not be 
> recovered.   Even restarting HDFS and Accumulo would not fix the problem.
> The following message was seen in an Accumulo tablet server log immediately 
> before the tablet server was killed.
> {noformat}
> 2015-05-14 17:12:37,466 [hdfs.DFSClient] WARN : DFSOutputStream 
> ResponseProcessor exception  for block 
> BP-802741494-10.1.5.6-1431557089849:blk_1073932823_192060
> java.io.IOException: Bad response ERROR for block 
> BP-802741494-10.1.5.6-1431557089849:blk_1073932823_192060 from datanode 
> 10.1.5.9:50010
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:897)
> 2015-05-14 17:12:37,466 [hdfs.DFSClient] WARN : Error Recovery for block 
> BP-802741494-10.1.5.6-1431557089849:blk_1073932823_192060 in pipeline 
> 10.1.5.55:50010, 10.1.5.9:5
> {noformat}
> Before recovering data from a write ahead log, the Accumulo master attempts 
> to recover the lease.   This repeatedly failed with messages like the 
> following.
> {noformat}
> 2015-05-14 17:14:54,301 [recovery.HadoopLogCloser] WARN : Error recovering 
> lease on 
> hdfs://10.1.5.6:1/accumulo/wal/worker11+9997/3a731759-3594-4535-8086-245eed7cd4c2
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException):
>  failed to create file 
> /accumulo/wal/worker11+9997/3a731759-3594-4535-8086-245eed7cd4c2 for 
> DFSClient_NONMAPREDUCE_950713214_16 for client 10.1.5.158 because 
> pendingCreates is non-null but no leases found.
> {noformat}
> Below is some info from the NN logs for the problematic file.
> {noformat}
> [ec2-user@leader2 logs]$ grep 3a731759-3594-4535-8086-245 
> hadoop-ec2-user-namenode-leader2.log 
> 2015-05-14 17:10:46,299 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* 
> allocateBlock: 
> /accumulo/wal/worker11+9997/3a731759-3594-4535-8086-245eed7cd4c2. 
> BP-802741494-10.1.5.6-1431557089849 
> blk_1073932823_192060{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, 
> replicas=[ReplicaUnderConstruction[[DISK]DS-ffe07d7d-0e68-45b8-b3d5-c976f1716481:NORMAL:10.1.5.55:50010|RBW],
>  
> ReplicaUnderConstruction[[DISK]DS-6efec702-3f1f-4ec0-a31f-de947e7e6097:NORMAL:10.1.5.9:50010|RBW],
>  
> ReplicaUnderConstruction[[DISK]DS-5e27df17-abf8-47df-b4bc-c38d0cd426ea:NORMAL:10.1.5.45:50010|RBW]]}
> 2015-05-14 17:10:46,628 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* 
> fsync: /accumulo/wal/worker11+9997/3a731759-3594-4535-8086-245eed7cd4c2 for 
> DFSClient_NONMAPREDUCE_-1128465883_16
> 2015-05-14 17:14:49,288 INFO 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: recoverLease: [Lease.  
> Holder: DFSClient_NONMAPREDUCE_-1128465883_16, pendingcreates: 1], 
> src=/accumulo/wal/worker11+9997/3a731759-3594-4535-8086-245eed7cd4c2 from 
> client DFSClient_NONMAPREDUCE_-1128465883_16
> 2015-05-14 17:14:49,288 INFO 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Recovering [Lease.  
> Holder: DFSClient_NONMAPREDUCE_-1128465883_16, pendingcreates: 1], 
> src=/accumulo/wal/worker11+9997/3a731759-3594-4535-8086-245eed7cd4c2
> 2015-05-14 17:14:49,289 WARN org.apache.hadoop.hdfs.StateChange: DIR* 
> NameSystem.internalReleaseLease: File 
> /accumulo/wal/worker11+9997/3a731759-3594-4535-8086-245eed7cd4c2 has not been 
> closed. Lease recovery is in progress. RecoveryId = 192257 for block 
> blk_1073932823_192060{blockUCState=UNDER_RECOVERY, primaryNodeIndex=2, 
> replicas=[ReplicaUnderConstruction[[DISK]DS-ffe07d7d-0e68-45b8-b3d5-c976f1716481:NORMAL:10.1.5.55:50010|RBW],
>  
> ReplicaUnderConstruction[[DISK]DS-6efec702-3f1f-4ec0-a31f-de947e7e6097:NORMAL:10.1.5.9:50010|RBW],
>  
> ReplicaUnderConstruction[[DISK]DS-5e27df17-abf8-47df-b4bc-c38d0cd426ea:NORMAL:10.1.5.45:50010|RBW]]}
> java.lang.IllegalStateException: Failed to finalize INodeFile 
> 3a731759-3594-4535-8086-245eed7cd4c2 since blocks[0] is non-complete, where 
> blocks=[blk_1073932823_192257{blockUCState=COMMITTED, primaryNodeIndex=2, 
> replicas=[ReplicaUnderConstruction[[DISK]DS-ffe07d7d-0e68-45b8-b3d5-c976f1716481:NORMAL:10.1.5.55:50010|RBW],
>  
> ReplicaUnderConstruction[[DISK]DS-5e27df17-abf8-47df-b4bc-c38d0cd426ea:NORMAL:10.1.5.45:50010|RBW]]}].
> 2015-05-14 17:14:54,292 INFO org.apache.hadoop.ipc.Server: IPC Server