Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64

[Apache Jenkins Server]

For more details, see https://builds.apache.org/job/hadoop-trunk-win/483/

[May 29, 2018 4:00:30 PM] (jlowe) YARN-8338. TimelineService V1.5 doesn't come 
up after HADOOP-15406.
[May 29, 2018 4:11:08 PM] (inigoiri) HADOOP-15497. TestTrash should use proper 
test path to avoid failing on
[May 29, 2018 4:23:11 PM] (wangda) YARN-8339. Service AM should localize 
static/archive resource types to
[May 29, 2018 4:27:36 PM] (wangda) YARN-8369. Javadoc build failed due to 'bad 
use of >'. (Takanobu Asanuma
[May 29, 2018 4:54:06 PM] (aengineer) HDDS-125. Cleanup HDDS CheckStyle issues. 
Contributed by Anu Engineer.
[May 29, 2018 7:15:12 PM] (kihwal) Additional check when unpacking archives. 
Contributed by Wilfred
[May 29, 2018 7:43:17 PM] (jlowe) YARN-8329. Docker client configuration can 
still be set incorrectly.
[May 29, 2018 7:48:50 PM] (aengineer) HDDS-81. Moving ContainerReport inside 
Datanode heartbeat. Contributed
[May 29, 2018 8:38:15 PM] (fabbri) HADOOP-14946 S3Guard testPruneCommandCLI can 
fail. Contributed by Gabor
[May 29, 2018 8:48:55 PM] (aengineer) HDDS-114. Ozone Datanode mbean 
registration fails for StorageLocation.
[May 29, 2018 8:56:58 PM] (eyang) YARN-8362.  Bugfix logic in container retries 
in node manager.  
[May 30, 2018 2:20:22 AM] (fabbri) HADOOP-15480 
AbstractS3GuardToolTestBase.testDiffCommand fails when
[May 30, 2018 8:52:21 AM] (yqlin) HDFS-13626. Fix incorrect username when deny 
the setOwner operation.




-1 overall


The following subsystems voted -1:
compile mvninstall pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


The following subsystems are considered long running:
(runtime bigger than 1h 00m 00s)
unit


Specific tests:

Failed junit tests :

   hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec 
   hadoop.fs.contract.rawlocal.TestRawlocalContractAppend 
   hadoop.fs.TestFsShellCopy 
   hadoop.fs.TestFsShellList 
   hadoop.fs.TestLocalFileSystem 
   hadoop.http.TestHttpServer 
   hadoop.http.TestHttpServerLogs 
   hadoop.io.compress.TestCodec 
   hadoop.io.nativeio.TestNativeIO 
   hadoop.ipc.TestSocketFactory 
   hadoop.metrics2.impl.TestStatsDMetrics 
   hadoop.security.TestGroupsCaching 
   hadoop.security.TestSecurityUtil 
   hadoop.security.TestShellBasedUnixGroupsMapping 
   hadoop.security.token.TestDtUtilShell 
   hadoop.util.TestDiskCheckerWithDiskIo 
   hadoop.util.TestNativeCodeLoader 
   hadoop.util.TestNodeHealthScriptRunner 
   hadoop.hdfs.qjournal.server.TestJournalNode 
   hadoop.hdfs.qjournal.server.TestJournalNodeSync 
   hadoop.hdfs.server.balancer.TestBalancer 
   hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks 
   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS 
   hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestProvidedImpl 
   hadoop.hdfs.server.datanode.TestBlockPoolSliceStorage 
   hadoop.hdfs.server.datanode.TestBlockScanner 
   hadoop.hdfs.server.datanode.TestDataNodeFaultInjector 
   hadoop.hdfs.server.datanode.TestDataNodeUUID 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.hdfs.server.datanode.TestDirectoryScanner 
   hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand 
   hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC 
   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA 
   hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages 
   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics 
   hadoop.hdfs.server.namenode.TestCacheDirectives 
   hadoop.hdfs.server.namenode.TestEditLogRace 
   hadoop.hdfs.server.namenode.TestNameNodeMXBean 
   hadoop.hdfs.server.namenode.TestReencryption 
   hadoop.hdfs.server.namenode.TestReencryptionHandler 
   hadoop.hdfs.server.namenode.TestStartup 
   hadoop.hdfs.TestDatanodeRegistration 
   hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs 
   hadoop.hdfs.TestDFSShell 
   hadoop.hdfs.TestDFSStripedInputStream 
   hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy 
   hadoop.hdfs.TestDFSUpgradeFromImage 
   hadoop.hdfs.TestDistributedFileSystem 
   hadoop.hdfs.TestFetchImage 
   hadoop.hdfs.TestHDFSFileSystemContract 
   hadoop.hdfs.TestLeaseRecovery2 
   hadoop.hdfs.TestLeaseRecoveryStriped 
   hadoop.hdfs.TestLocalDFS 
   hadoop.hdfs.TestPread 
   hadoop.hdfs.TestReadStripedFileWithDecodingCorruptData 
   hadoop.hdfs.TestReadStripedFileWithDecodingDeletedData 
   hadoop.hdfs.TestReconstructStripedFile 
   hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy 
   

[jira] [Created] (HDFS-13644) Backport HDFS-10376 to branch-2

[Yiqun Lin (JIRA)]

Yiqun Lin created HDFS-13644:


 Summary: Backport HDFS-10376 to branch-2
 Key: HDFS-13644
 URL: https://issues.apache.org/jira/browse/HDFS-13644
 Project: Hadoop HDFS
  Issue Type: Improvement
Affects Versions: 2.9.1
 Environment: HDFS-10376 enhances setOwner testing, but didn't backport 
to branch-2. This is worth to have in branch-2.
Reporter: Yiqun Lin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13643) Implement basic async rpc client

[Duo Zhang (JIRA)]

Duo Zhang created HDFS-13643:


 Summary: Implement basic async rpc client
 Key: HDFS-13643
 URL: https://issues.apache.org/jira/browse/HDFS-13643
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ipc
Reporter: Duo Zhang


Implement the basic async rpc client so we can start working on the DFSClient 
implementation ASAP.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-133) Change format of .container files to Yaml

[Bharat Viswanadham (JIRA)]

Bharat Viswanadham created HDDS-133:
---

 Summary: Change format of .container files to Yaml
 Key: HDDS-133
 URL: https://issues.apache.org/jira/browse/HDDS-133
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Bharat Viswanadham
Assignee: Bharat Viswanadham


This Jira changes .container file format to yaml.

Following fields are serialized into .container file:

containerType, containerId, layOutVersion, state, metadata, dbPath, 
containerFilePath, containerDBType



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Apache Hadoop 3.0.3 Release plan

[Yongjun Zhang]

Sorry, forgot to mention Junping Du, who also helped me a lot. Many thanks
Junping!

Thanks.

--Yongjun

On Wed, May 30, 2018 at 11:49 AM, Yongjun Zhang 
wrote:

> Hi,
>
> The build issues are all solved, and I have cut the 3.0.3 branch and close
> to get a build out. Since it's taking me a bit more time (I expect to send
> vote invitation email by today). I would like to send a heads-up notice now.
>
> Thank you all for feedback, and many thanks to Sammi Chen, Andrew Wang,
> Eddy Xu who helped when I tried to solve the build issues.
>
> At this point, please be aware of the existence of branch-3.0,
> branch-3.0.3.
>
> Best,
>
> --Yongjun
>
>
>
> On Sat, May 26, 2018 at 11:52 PM, Yongjun Zhang 
> wrote:
>
>> HI,
>>
>> I did build before cut branch and hit some issues, have not got to the
>> bottom, will cut branch after the build issues are resolved.
>>
>> Thanks.
>>
>> --Yongjun
>>
>> On Sat, May 26, 2018 at 1:46 PM, Yongjun Zhang 
>> wrote:
>>
>>> Hi All,
>>>
>>> I will be working on cutting the 3.0.3 branch and trying a build today.
>>>
>>> Thanks.
>>>
>>> --Yongjun
>>>
>>>
>>>
>>> On Wed, May 23, 2018 at 3:31 PM, Yongjun Zhang 
>>> wrote:
>>>
 Thanks Eric. Sounds good. I may try to see if I can do the branching/RC
 sooner.

 --Yongjun


 On Wed, May 23, 2018 at 2:18 PM, Eric Badger  wrote:

> My thinking is to cut the branch in next couple of days and create RC
> for
> vote at the end of month.
>   >  We plan to cut branch-3.0.3 by the coming Wednesday (May 9th)
> and vote for RC on May 30th
>   I much prefer to wait to cut the branch until just before the
> production of the release and the vote. With so many branches, we 
> sometimes
> miss putting critical bug fixes in unreleased branches if thebranch is
> cut too early.
>
> Echoing Eric Payne, I think we should wait to cut the branch until we
> are actually creating the RC to vote on (i.e. on May 29 or 30 if the vote
> is to be on May 30).
>
> Eric
>
>
>
> On Wed, May 23, 2018 at 4:11 PM, Yongjun Zhang 
> wrote:
>
>> Hi,
>>
>> I have gardened the jiras for 3.0.3, and have the following open
>> issues:
>>
>> https://issues.apache.org/jira/issues/?filter=12343970
>>
>> Two of them are blockers, one of them (YARN-8346) has already got +1
>> for
>> patch, the other (YARN-8108) will take longer time to resolve and it
>> seems
>> we can possibly push it to next release given 3.0.2 also has the
>> issue.
>>
>> My thinking is to cut the branch in next couple of days and create RC
>> for
>> vote at the end of month.
>>
>> Comments are welcome.
>>
>> Thanks,
>>
>> --Yongjun
>>
>> On Tue, May 8, 2018 at 11:40 AM, Vrushali C 
>> wrote:
>>
>> > +1 for including the YARN-7190 patch in 3.0.3 release. This is a
>> fix that
>> > will enable HBase to use Hadoop 3.0.x in the production line.
>> >
>> > thanks
>> > Vrushali
>> >
>> >
>> > On Tue, May 8, 2018 at 10:24 AM, Yongjun Zhang > >
>> > wrote:
>> >
>> >> Thanks Wei-Chiu and Haibo for the feedback!
>> >>
>> >> Good thing is that I have made the following note couple of days
>> ago when
>> >> I
>> >> looked the at branch diff, so we are on the same page:
>> >>
>> >>  496dc57 Revert "YARN-7190. Ensure only NM classpath in 2.x
>> gets TSv2
>> >> related hbase jars, not the user classpath. Contributed by Varun
>> Saxena."
>> >>
>> >> *YARN-7190 is not in 3.0.2,  I will include it in 3.0.3 per* the
>> comment
>> >> below:
>> >> https://issues.apache.org/jira/browse/YARN-7190?focusedComme
>> >> ntId=16457649&
>> >> page=com.atlassian.jira.plugin.system.issuetabpanels
>> >> > entId=16457649=com.atlassian.jira.plugin.system.issuetabpanels>
>>
>> >> :
>> >> comment-tabpanel#comment-16457649
>> >>
>> >>
>> >> In addition, I will revert   https://issues.apache.org/
>> >> jira/browse/HADOOP-13055 from 3.0.3 since it's a feature.
>> >>
>> >>
>> >> Best,
>> >>
>> >> --Yongjun
>> >>
>> >> On Tue, May 8, 2018 at 8:57 AM, Haibo Chen > >
>> >> wrote:
>> >>
>> >> > +1 on adding YARN-7190 to Hadoop 3.0.x despite the fact that it
>> is
>> >> > technically incompatible.
>> >> > It is critical enough to justify being an exception, IMO.
>> >> >
>> >> > Added Rohith and Vrushali
>> >> >
>> >> > On Tue, May 8, 2018 at 6:20 AM, Wei-Chiu Chuang <
>> weic...@apache.org>
>> >> > wrote:
>> >> >
>> >> >> Thanks Yongjun for driving 3.0.3 release!
>> >> >>
>> >> >> IMHO, could we consider adding YARN-7190
>> >> >>  into the
>> list?
>> >> >> I 

Re: Apache Hadoop 3.0.3 Release plan

[Yongjun Zhang]

Hi,

The build issues are all solved, and I have cut the 3.0.3 branch and close
to get a build out. Since it's taking me a bit more time (I expect to send
vote invitation email by today). I would like to send a heads-up notice now.

Thank you all for feedback, and many thanks to Sammi Chen, Andrew Wang,
Eddy Xu who helped when I tried to solve the build issues.

At this point, please be aware of the existence of branch-3.0, branch-3.0.3.

Best,

--Yongjun



On Sat, May 26, 2018 at 11:52 PM, Yongjun Zhang  wrote:

> HI,
>
> I did build before cut branch and hit some issues, have not got to the
> bottom, will cut branch after the build issues are resolved.
>
> Thanks.
>
> --Yongjun
>
> On Sat, May 26, 2018 at 1:46 PM, Yongjun Zhang 
> wrote:
>
>> Hi All,
>>
>> I will be working on cutting the 3.0.3 branch and trying a build today.
>>
>> Thanks.
>>
>> --Yongjun
>>
>>
>>
>> On Wed, May 23, 2018 at 3:31 PM, Yongjun Zhang 
>> wrote:
>>
>>> Thanks Eric. Sounds good. I may try to see if I can do the branching/RC
>>> sooner.
>>>
>>> --Yongjun
>>>
>>>
>>> On Wed, May 23, 2018 at 2:18 PM, Eric Badger  wrote:
>>>
 My thinking is to cut the branch in next couple of days and create RC
 for
 vote at the end of month.
   >  We plan to cut branch-3.0.3 by the coming Wednesday (May 9th) and
 vote for RC on May 30th
   I much prefer to wait to cut the branch until just before the
 production of the release and the vote. With so many branches, we sometimes
 miss putting critical bug fixes in unreleased branches if thebranch is
 cut too early.

 Echoing Eric Payne, I think we should wait to cut the branch until we
 are actually creating the RC to vote on (i.e. on May 29 or 30 if the vote
 is to be on May 30).

 Eric



 On Wed, May 23, 2018 at 4:11 PM, Yongjun Zhang 
 wrote:

> Hi,
>
> I have gardened the jiras for 3.0.3, and have the following open
> issues:
>
> https://issues.apache.org/jira/issues/?filter=12343970
>
> Two of them are blockers, one of them (YARN-8346) has already got +1
> for
> patch, the other (YARN-8108) will take longer time to resolve and it
> seems
> we can possibly push it to next release given 3.0.2 also has the issue.
>
> My thinking is to cut the branch in next couple of days and create RC
> for
> vote at the end of month.
>
> Comments are welcome.
>
> Thanks,
>
> --Yongjun
>
> On Tue, May 8, 2018 at 11:40 AM, Vrushali C 
> wrote:
>
> > +1 for including the YARN-7190 patch in 3.0.3 release. This is a fix
> that
> > will enable HBase to use Hadoop 3.0.x in the production line.
> >
> > thanks
> > Vrushali
> >
> >
> > On Tue, May 8, 2018 at 10:24 AM, Yongjun Zhang 
> > wrote:
> >
> >> Thanks Wei-Chiu and Haibo for the feedback!
> >>
> >> Good thing is that I have made the following note couple of days
> ago when
> >> I
> >> looked the at branch diff, so we are on the same page:
> >>
> >>  496dc57 Revert "YARN-7190. Ensure only NM classpath in 2.x
> gets TSv2
> >> related hbase jars, not the user classpath. Contributed by Varun
> Saxena."
> >>
> >> *YARN-7190 is not in 3.0.2,  I will include it in 3.0.3 per* the
> comment
> >> below:
> >> https://issues.apache.org/jira/browse/YARN-7190?focusedComme
> >> ntId=16457649&
> >> page=com.atlassian.jira.plugin.system.issuetabpanels
> >>  entId=16457649=com.atlassian.jira.plugin.system.issuetabpanels>
>
> >> :
> >> comment-tabpanel#comment-16457649
> >>
> >>
> >> In addition, I will revert   https://issues.apache.org/
> >> jira/browse/HADOOP-13055 from 3.0.3 since it's a feature.
> >>
> >>
> >> Best,
> >>
> >> --Yongjun
> >>
> >> On Tue, May 8, 2018 at 8:57 AM, Haibo Chen 
> >> wrote:
> >>
> >> > +1 on adding YARN-7190 to Hadoop 3.0.x despite the fact that it is
> >> > technically incompatible.
> >> > It is critical enough to justify being an exception, IMO.
> >> >
> >> > Added Rohith and Vrushali
> >> >
> >> > On Tue, May 8, 2018 at 6:20 AM, Wei-Chiu Chuang <
> weic...@apache.org>
> >> > wrote:
> >> >
> >> >> Thanks Yongjun for driving 3.0.3 release!
> >> >>
> >> >> IMHO, could we consider adding YARN-7190
> >> >>  into the list?
> >> >> I understand that it is listed as an incompatible change,
> however,
> >> because
> >> >> of this bug, HBase considers the entire Hadoop 3.0.x line not
> >> production
> >> >> ready. I feel there's not much point releasing any more 3.0.x
> releases
> >> if
> >> >> downstream projects can't pick it up (after the fact that HBase
> is one
> >> 

[jira] [Created] (HDFS-13642) Creating a file with block size smaller than EC policy's cell size should throw

[Xiao Chen (JIRA)]

Xiao Chen created HDFS-13642:


 Summary: Creating a file with block size smaller than EC policy's 
cell size should throw
 Key: HDFS-13642
 URL: https://issues.apache.org/jira/browse/HDFS-13642
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: erasure-coding
Affects Versions: 3.0.0
Reporter: Xiao Chen
Assignee: Xiao Chen
 Attachments: HDFS-13642.01.patch

The following command causes an exception:
{noformat}
hadoop fs -Ddfs.block.size=349696 -put -f lineitem_sixblocks.parquet 
/test-warehouse/tmp123ec
{noformat}

{noformat}
18/05/25 16:00:59 WARN hdfs.DataStreamer: DataStreamer Exception
java.io.IOException: BlockSize 349696 < lastByteOffsetInBlock, #0: 
blk_-9223372036854574256_14634, packet seqno: 7 offsetInBlock: 349696 
lastPacketInBlock: false lastByteOffsetInBlock: 350208
  at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:729)
  at org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:46)
18/05/25 16:00:59 WARN hdfs.DFSOutputStream: Failed: offset=4096, length=512, 
DFSStripedOutputStream:#0: failed, blk_-9223372036854574256_14634
java.io.IOException: BlockSize 349696 < lastByteOffsetInBlock, #0: 
blk_-9223372036854574256_14634, packet seqno: 7 offsetInBlock: 349696 
lastPacketInBlock: false lastByteOffsetInBlock: 350208
  at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:729)
  at org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:46)
{noformat}

Then the streamer is confused and hangs.

The local file is under 6MB, the hdfs file has a RS-3-2-1024k EC policy.

 

Credit to [~tarasbob] for reporting this issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13641) Add metrics for edit log tailin

[Chao Sun (JIRA)]

Chao Sun created HDFS-13641:
---

 Summary: Add metrics for edit log tailin 
 Key: HDFS-13641
 URL: https://issues.apache.org/jira/browse/HDFS-13641
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Chao Sun






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/796/

[May 29, 2018 4:00:30 PM] (jlowe) YARN-8338. TimelineService V1.5 doesn't come 
up after HADOOP-15406.
[May 29, 2018 4:11:08 PM] (inigoiri) HADOOP-15497. TestTrash should use proper 
test path to avoid failing on
[May 29, 2018 4:23:11 PM] (wangda) YARN-8339. Service AM should localize 
static/archive resource types to
[May 29, 2018 4:27:36 PM] (wangda) YARN-8369. Javadoc build failed due to 'bad 
use of >'. (Takanobu Asanuma
[May 29, 2018 4:54:06 PM] (aengineer) HDDS-125. Cleanup HDDS CheckStyle issues. 
Contributed by Anu Engineer.
[May 29, 2018 7:15:12 PM] (kihwal) Additional check when unpacking archives. 
Contributed by Wilfred
[May 29, 2018 7:43:17 PM] (jlowe) YARN-8329. Docker client configuration can 
still be set incorrectly.
[May 29, 2018 7:48:50 PM] (aengineer) HDDS-81. Moving ContainerReport inside 
Datanode heartbeat. Contributed
[May 29, 2018 8:38:15 PM] (fabbri) HADOOP-14946 S3Guard testPruneCommandCLI can 
fail. Contributed by Gabor
[May 29, 2018 8:48:55 PM] (aengineer) HDDS-114. Ozone Datanode mbean 
registration fails for StorageLocation.
[May 29, 2018 8:56:58 PM] (eyang) YARN-8362.  Bugfix logic in container retries 
in node manager.  




-1 overall


The following subsystems voted -1:
asflicense findbugs pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-hdds/common 
   Found reliance on default encoding in 
org.apache.hadoop.utils.MetadataKeyFilters$KeyPrefixFilter.filterKey(byte[], 
byte[], byte[]):in 
org.apache.hadoop.utils.MetadataKeyFilters$KeyPrefixFilter.filterKey(byte[], 
byte[], byte[]): String.getBytes() At MetadataKeyFilters.java:[line 97] 

Failed junit tests :

   hadoop.hdfs.server.datanode.TestDirectoryScanner 
   hadoop.hdfs.client.impl.TestBlockReaderLocal 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints 
   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart 
   hadoop.yarn.client.api.impl.TestAMRMProxy 
   
hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities 
   
hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity 
   hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun 
   hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps 
   
hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction
 
   
hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageSchema 
   
hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageDomain 
   
hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/796/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/796/artifact/out/diff-compile-javac-root.txt
  [332K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/796/artifact/out/diff-checkstyle-root.txt
  [17M]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/796/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/796/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/796/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/796/artifact/out/diff-patch-shelldocs.txt
  [16K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/796/artifact/out/whitespace-eol.txt
  [9.4M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/796/artifact/out/whitespace-tabs.txt
  [1.1M]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/796/artifact/out/xml.txt
  [4.0K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/796/artifact/out/branch-findbugs-hadoop-hdds_client.txt
  [56K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/796/artifact/out/branch-findbugs-hadoop-hdds_common-warnings.html
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/796/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt
  [52K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/796/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt
  [56K]
   

[jira] [Reopened] (HDFS-13640) enable ShortCircuit Read on UC block

[Duo Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-13640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang reopened HDFS-13640:
--

> enable ShortCircuit Read on UC block
> 
>
> Key: HDFS-13640
> URL: https://issues.apache.org/jira/browse/HDFS-13640
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client
>Affects Versions: 3.0.2
>Reporter: Gang Xie
>Priority: Major
>
> The ShortCircuit read is disabled by HDFS-2757 due to the inconsistency of 
> the block states. With this limitation, some streaming/messaging application 
> could not benefit the performance improvement from SCR. In our streaming 
> system whose storage is HDFS, has around 90% read on the last blocks. So, 
> it's necessary to enable SCR on the last block, especially the app could 
> ensure the read after the flush.
> After look into the original issue in HDFS-2757, it could only happen when 
> the read beyond the flush or the local datanode is kicked out from the 
> pipeline. But if there is the data and the visible length of the block covers 
> the read length, we could still read the data from the block, right?  
>  
> I didn't get a completed solution here. Any suggestion could be helpful.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-13640) s

[Duo Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-13640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HDFS-13640.
--
Resolution: Invalid

Accident?

> s
> -
>
> Key: HDFS-13640
> URL: https://issues.apache.org/jira/browse/HDFS-13640
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client
>Affects Versions: 3.0.2
>Reporter: Gang Xie
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13640) s

[Gang Xie (JIRA)]

Gang Xie created HDFS-13640:
---

 Summary: s
 Key: HDFS-13640
 URL: https://issues.apache.org/jira/browse/HDFS-13640
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Gang Xie






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13639) SlotReleaser is not fast enough

[Gang Xie (JIRA)]

Gang Xie created HDFS-13639:
---

 Summary: SlotReleaser is not fast enough
 Key: HDFS-13639
 URL: https://issues.apache.org/jira/browse/HDFS-13639
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client
Affects Versions: 3.0.2, 2.6.0, 2.4.0
 Environment: --

YCSB:
 # {color:#00}records
 recordcount=20
 fieldcount=1
 fieldlength=1000
 operationcount=1000
 
 workload=com.yahoo.ycsb.workloads.CoreWorkload
 
 table=ycsb-test
 columnfamily=C
 readproportion=1
 updateproportion=0
 insertproportion=0
 scanproportion=0
 
 maxscanlength=0
 requestdistribution=zipfian
 
 # default 
 readallfields=true
 writeallfields=true
 
scanlengthdistribution=constant{color}{color:#00}---{color}{color:#00}datanode:{color}{color:#00}-Xmx2048m
 -Xms2048m -Xmn1024m -XX:MaxDirectMemorySize=1024m -XX:MaxPermSize=256m 
-Xloggc:$run_dir/stdout/datanode_gc_${start_time}.log -XX:+DisableExplicitGC 
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$log_dir 
-XX:+PrintGCApplicationStoppedTime -XX:+UseConcMarkSweepGC 
-XX:CMSInitiatingOccupancyFraction=80 -XX:+UseCMSInitiatingOccupancyOnly 
-XX:+CMSParallelRemarkEnabled -XX:+CMSClassUnloadingEnabled 
-XX:CMSMaxAbortablePrecleanTime=1 -XX:+CMSScavengeBeforeRemark 
-XX:+PrintPromotionFailure -XX:+CMSConcurrentMTEnabled 
-XX:+ExplicitGCInvokesConcurrent -XX:+SafepointTimeout -XX:MonitorBound=16384 
-XX:-UseBiasedLocking -verbose:gc -XX:+PrintGCDetails -XX:+PrintHeapAtGC 
-XX:+PrintGCDateStamps{color}{color:#00}{color}{color:#00}regionserver:{color}{color:#00}-Xmx10g
 -Xms10g -XX:MaxDirectMemorySize=10g -XX:MaxGCPauseMillis=150 
-XX:MaxTenuringThreshold=2 -XX:+UnlockExperimentalVMOptions 
-XX:G1NewSizePercent=5 
-Xloggc:$run_dir/stdout/regionserver_gc_${start_time}.log -Xss256k 
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$log_dir -verbose:gc 
-XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCApplicationStoppedTime 
-XX:+PrintHeapAtGC -XX:+PrintGCDateStamps -XX:+PrintAdaptiveSizePolicy 
-XX:+PrintTenuringDistribution -XX:+PrintSafepointStatistics 
-XX:PrintSafepointStatisticsCount=1 -XX:PrintFLSStatistics=1 
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=100 -XX:GCLogFileSize=128m 
-XX:+SafepointTimeout -XX:MonitorBound=16384 -XX:-UseBiasedLocking -XX:+UseG1GC 
-XX:InitiatingHeapOccupancyPercent=65 -XX:+ParallelRefProcEnabled 
-XX:ConcGCThreads=4 -XX:ParallelGCThreads=16 -XX:G1HeapRegionSize=32m 
-XX:G1MixedGCCountTarget=64 
-XX:G1OldCSetRegionThresholdPercent=5{color}{color:#00}block cache is 
disabled:{color}{color:#00} 
 hbase.bucketcache.size
 0.9
 {color}

 
Reporter: Gang Xie
 Fix For: 2.4.0
 Attachments: ShortCircuitCache_new_slotReleaser.diff, 
perf_after_improve_SlotReleaser.png, perf_before_improve_SlotReleaser.png

When test the performance of the ShortCircuit Read of the HDFS with YCSB, we 
find that SlotReleaser of the ShortCircuitCache has some performance issue. The 
problem is that, the qps of the slot releasing could only reach to 1000+ while 
the qps of the slot allocating is ~3000. This means that the replica info on 
datanode could not be released in time, which causes a lot of GCs and finally 
full GCs.

 

The fireflame graph shows that SlotReleaser spends a lot of time to do domain 
socket connecting and throw/catching the exception when close the domain socket 
and its streams. It doesn't make any sense to do the connecting and closing 
each time. Each time when we connect to the domain socket, Datanode allocates a 
new thread to free the slot. There are a lot of initializing work, and it's 
costly. We need reuse the domain socket. 

 

After switch to reuse the domain socket(see diff attached), we get great 
improvement(see the perf):
 # without reusing the domain socket, the get qps of the YCSB getting worse and 
worse, and after about 45 mins, full GC starts. When we reuse the domain 
socket, no full GC found, and the stress test could be finished smoothly, the 
qps of allocating and releasing match.
 # Due to the datanode young GC, without the improvement, the YCSB get qps is 
even smaller than the one with the improvement, ~3700 VS ~4200.

The diff is against 2.4, and I think this issue exists till latest version. I 
doesn't have test env with 2.7 and higher version. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-132) Update SCMNodeStorageStatMap while processing Node Report

[Shashikant Banerjee (JIRA)]

Shashikant Banerjee created HDDS-132:


 Summary: Update SCMNodeStorageStatMap while processing Node Report
 Key: HDDS-132
 URL: https://issues.apache.org/jira/browse/HDDS-132
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: SCM
Reporter: Shashikant Banerjee
Assignee: Shashikant Banerjee
 Fix For: 0.2.1


When the node report is received at SCM, SCMNodeStorageStatMap needs to get 
updated.

In the event of a node/Volume failure, this Map needs to be updated as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org