[jira] [Created] (HDDS-313) Add metrics to containerState Machine

[Mukul Kumar Singh (JIRA)]

Mukul Kumar Singh created HDDS-313:
--

 Summary: Add metrics to containerState Machine
 Key: HDDS-313
 URL: https://issues.apache.org/jira/browse/HDDS-313
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: Ozone Datanode
Reporter: Mukul Kumar Singh


metrics needs to be added to containerStateMachine to keep track of various 
ratis ops like writeStateMachine/readStateMachine/applyTransactions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-312) Add blockIterator to Container

[Bharat Viswanadham (JIRA)]

Bharat Viswanadham created HDDS-312:
---

 Summary: Add blockIterator to Container
 Key: HDDS-312
 URL: https://issues.apache.org/jira/browse/HDDS-312
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Bharat Viswanadham
Assignee: Bharat Viswanadham


This Jira is to add newly added blockIterator to Container and its implemented 
class KeyValueContainer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[PR] Request to PR review

[Xiaoqiao He]

 Hi, HDFS-DEVs,

Anyone have time to help review the following pullrequest? Thanks.

HDFS-12749  DN may not
send block report to NN after NN restart
HDFS-13473  DataNode
update BlockKeys using mode PULL rather than PUSH from NameNode
HDFS-13668 
FSPermissionChecker may throws AIOOE when check if inode has permission

Best regards,
Hexiaoqiao

[jira] [Resolved] (HDFS-13084) [SPS]: Fix the branch review comments

[Rakesh R (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-13084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rakesh R resolved HDFS-13084.
-
   Resolution: Fixed
Fix Version/s: HDFS-10285

I'm closing this issue as {{IntraSPSNameNodeContext}} code implementation 
specifically for the internal SPS service has been removed from this branch. 
Internal SPS mechanism will be discussed and supported via the follow-up Jira 
task HDFS-12226. We have taken care the comments related to this branch via 
HDFS-13097, HDFS-13110, HDFS-13166, HDFS-13381 Jira sub-tasks.

> [SPS]: Fix the branch review comments
> -
>
> Key: HDFS-13084
> URL: https://issues.apache.org/jira/browse/HDFS-13084
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Uma Maheswara Rao G
>Assignee: Rakesh R
>Priority: Major
> Fix For: HDFS-10285
>
>
> Fix the review comments provided by [~daryn]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13784) Metrics sampling period is milliseconds instead of seconds。

[chencan (JIRA)]

chencan created HDFS-13784:
--

 Summary: Metrics sampling period is milliseconds instead of 
seconds。
 Key: HDFS-13784
 URL: https://issues.apache.org/jira/browse/HDFS-13784
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: chencan


Metrics sampling period is milliseconds instead of seconds，this patch modify 
the related configuration file.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13783) Balancer: make balancer to be a long service process for easy to monitor it.

[maobaolong (JIRA)]

maobaolong created HDFS-13783:
-

 Summary: Balancer: make balancer to be a long service process for 
easy to monitor it.
 Key: HDFS-13783
 URL: https://issues.apache.org/jira/browse/HDFS-13783
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: balancer  mover
Affects Versions: 3.0.3
Reporter: maobaolong


If we have a long service process of balancer, like namenode, datanode, we can 
get metrics of balancer, the metrics can tell us the status of balancer, the 
amount of block it has moved, 
We can get or set the balance plan by the balancer webUI. So many things we can 
do if we have a long balancer service process.

So, shall we start to plan the new Balancer? Hope this feature can enter the 
next release of hadoop.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13782) IPFailoverProxyProvider should work with SBN

[Konstantin Shvachko (JIRA)]

Konstantin Shvachko created HDFS-13782:
--

 Summary: IPFailoverProxyProvider should work with SBN
 Key: HDFS-13782
 URL: https://issues.apache.org/jira/browse/HDFS-13782
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: test
Reporter: Konstantin Shvachko


Currently {{ObserverReadProxyProvider}} is based on 
{{ConfiguredFailoverProxyProvider}}. We should also be able perform SBN reads 
in case of {{IPFailoverProxyProvider}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13781) Unit test for standby reads.

[Konstantin Shvachko (JIRA)]

Konstantin Shvachko created HDFS-13781:
--

 Summary: Unit test for standby reads.
 Key: HDFS-13781
 URL: https://issues.apache.org/jira/browse/HDFS-13781
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: test
Reporter: Konstantin Shvachko


Create more unit tests supporting standby reads feature. Let's come up with a 
list of tests that provide sufficient test coverage.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13780) Postpone NameNode state discovery in ObserverReadProxyProvider until the first real RPC call.

[Konstantin Shvachko (JIRA)]

Konstantin Shvachko created HDFS-13780:
--

 Summary: Postpone NameNode state discovery in 
ObserverReadProxyProvider until the first real RPC call.
 Key: HDFS-13780
 URL: https://issues.apache.org/jira/browse/HDFS-13780
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: hdfs-client
Reporter: Konstantin Shvachko


Currently {{ObserverReadProxyProvider}} during instantiation discovers 
Observers by poking known NameNodes and checking their states. This rather 
expensive process can be postponed until the first actual RPC call. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13779) Implement performFailover logic for ObserverReadProxyProvider.

[Konstantin Shvachko (JIRA)]

Konstantin Shvachko created HDFS-13779:
--

 Summary: Implement performFailover logic for 
ObserverReadProxyProvider.
 Key: HDFS-13779
 URL: https://issues.apache.org/jira/browse/HDFS-13779
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: hdfs-client
Reporter: Konstantin Shvachko


Currently {{ObserverReadProxyProvider}} inherits {{performFailover()}} method 
from {{ConfiguredFailoverProxyProvider}}, which simply increments the index and 
switches over to another NameNode. The logic for ORPP should be smart enough to 
choose another observer, otherwise it can switch to a SBN, where reads are 
disallowed, or to an ANN, which defeats the purpose of reads from standby.
This was discussed in HDFS-12976.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13778) In TestStateAlignmentContextWithHA replace artificial AlignmentContextProxyProvider with real ObserverReadProxyProvider.

[Konstantin Shvachko (JIRA)]

Konstantin Shvachko created HDFS-13778:
--

 Summary: In TestStateAlignmentContextWithHA replace artificial 
AlignmentContextProxyProvider with real ObserverReadProxyProvider.
 Key: HDFS-13778
 URL: https://issues.apache.org/jira/browse/HDFS-13778
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: test
Reporter: Konstantin Shvachko


TestStateAlignmentContextWithHA uses an artificial 
AlignmentContextProxyProvider, which was temporary needed for testing. Now that 
we have real ObserverReadProxyProvider it can take over ACPP. This is also 
useful for testing the ORPP.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Apache Hadoop 3.1.1 release plan

[Wangda Tan]

Hi all,

I'm working on RC for 3.1.1 now, if you have any commits to 3.1.x line,
please commit them to branch-3.1.

If it is a blocker for 3.1.1, please let me know so we can decide if we
want to roll another RC or not.

Thanks,
Wangda


On Thu, Jul 19, 2018 at 2:47 PM Wangda Tan  wrote:

> Hi all,
>
> After several blockers of 3.1.1 landed, I think we're pretty close to a
> clean 3.1.1 branch ready for RC.
>
> By far we have two blockers targeted 3.1.1 [1], and there're 420 tickets
> have fix version = 3.1.1 [2]
>
> As we previously communicated for 3.1.1 release date (May 01), we have
> delayed for more than two months, which I want to get 3.1.1 released as
> soon as possible. I just cut branch-3.1.1 for blockers only. Branch-3.1
> will be open for all bug fixes.
>
> I'm going to create RC0 by end of tomorrow or the last blocker get
> resolved, whichever is the later. Please let me know if there are any other
> blockers need to get in 3.1.1.
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.1.1 ORDER
> BY priority DESC
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.1)
> ORDER BY priority DESC
>
> Thanks,
> Wangda
>
>
> On Thu, May 10, 2018 at 6:40 PM Weiwei Yang 
> wrote:
>
>> Hi Wangda
>>
>> I would propose to have https://issues.apache.org/jira/browse/YARN-8015 
>> included
>> in 3.1.1.
>>
>> Once this is done, we get both intra and inter placement constraint
>> covered so users could start to explore this feature. Otherwise the
>> functionality is pretty limited. It has been Patch Available for a while, I
>> just promoted it targeting to 3.1.1. Hope that makes sense.
>>
>> Thanks!
>>
>> --
>> Weiwei
>>
>> On 11 May 2018, 9:02 AM +0800, Wangda Tan , wrote:
>>
>> Hi all,
>>
>> As we previously proposed RC time (May 1st), we want to release 3.1.1
>> sooner if possible. As of now, 3.1.1 has 187 fixes [1] on top of 3.1.0,
>> and
>> there're 10 open blockers/criticals which target to 3.1.1 [2]. I just
>> posted comments to these open criticals/blockers ticket owners asking
>> about
>> statuses.
>>
>> If everybody agrees, I propose start code freeze of branch-3.1 from Sat
>> PDT
>> time this week, only blockers/criticals can be committed to branch-3.1. To
>> avoid the burden of committers, I want to delay cutting branch-3.1.1 as
>> late as possible. If you have any major/minors (For severe issues please
>> update priorities) tickets want to go to 3.1.1, please reply this email
>> thread and we can look at them and make a call together.
>>
>> Please feel free to share your comments and suggestions.
>>
>> Thanks,
>> Wangda
>>
>> [1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop Map/Reduce")
>> AND status = Resolved AND fixVersion = 3.1.1
>> [2] project in (YARN, HADOOP, MAPREDUCE, "Hadoop Development Tools") AND
>> priority in (Blocker, Critical) AND resolution = Unresolved AND "Target
>> Version/s" = 3.1.1 ORDER BY priority DESC
>>
>>
>> On Thu, May 10, 2018 at 5:48 PM, Wangda Tan  wrote:
>>
>> Thanks Brahma/Sunil,
>>
>> For YARN-8265, it is a too big change for 3.1.1, I just removed 3.1.1 from
>> target version.
>> For YARN-8236, it is a severe issue and I think it is close to finish.
>>
>>
>>
>> On Thu, May 10, 2018 at 3:08 AM, Sunil G  wrote:
>>
>>
>> Thanks Brahma.
>> Yes, Billie is reviewing YARN-8265 and I am helping in YARN-8236.
>>
>> - Sunil
>>
>>
>> On Thu, May 10, 2018 at 2:25 PM Brahma Reddy Battula <
>> brahmareddy.batt...@huawei.com> wrote:
>>
>> Thanks Wangda Tan for driving the 3.1.1 release.Yes,This can be better
>> addition to 3.1 line release for improving quality.
>>
>> Looks only following two are pending which are in review state. Hope you
>> are monitoring these two.
>>
>> https://issues.apache.org/jira/browse/YARN-8265
>> https://issues.apache.org/jira/browse/YARN-8236
>>
>>
>>
>> Note : https://issues.apache.org/jira/browse/YARN-8247==> committed
>> branch-3.1
>>
>>
>> -Original Message-
>> From: Wangda Tan [mailto:wheele...@gmail.com]
>> Sent: 19 April 2018 17:49
>> To: Hadoop Common ;
>> mapreduce-...@hadoop.apache.org; Hdfs-dev ;
>> yarn-...@hadoop.apache.org
>> Subject: Apache Hadoop 3.1.1 release plan
>>
>> Hi, All
>>
>> We have released Apache Hadoop 3.1.0 on Apr 06. To further improve the
>> quality of the release, we plan to release 3.1.1 at May 06. The focus of
>> 3.1.1 will be fixing blockers / critical bugs and other enhancements. So
>> far there are 100 JIRAs [1] have fix version marked to 3.1.1.
>>
>> We plan to cut branch-3.1.1 on May 01 and vote for RC on the same day.
>>
>> Please feel free to share your insights.
>>
>> Thanks,
>> Wangda Tan
>>
>> [1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop
>> Map/Reduce") AND fixVersion = 3.1.1
>>
>>
>>
>>

Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64

[Apache Jenkins Server]

For more details, see https://builds.apache.org/job/hadoop-trunk-win/545/

[Aug 1, 2018 11:39:10 AM] (aw) YETUS-46. releasedocmaker should support JIRA 
authentication
[Aug 1, 2018 11:40:17 AM] (aw) YETUS-647. ssh-based git repos do not work 
correctly under docker
[Jul 31, 2018 2:20:40 PM] (msingh) HDDS-279. DeleteBlocks command should not be 
sent for open containers.
[Jul 31, 2018 4:36:34 PM] (inigoiri) YARN-8175. Add support for Node Labels in 
SLS. Contributed by Abhishek
[Jul 31, 2018 6:32:40 PM] (haibochen) YARN-8605. 
TestDominantResourceFairnessPolicy.testModWhileSorting is
[Jul 31, 2018 7:08:00 PM] (wangda) YARN-8418. App local logs could leaked if 
log aggregation fails to
[Jul 31, 2018 10:01:02 PM] (eyang) YARN-8579.  Recover NMToken of previous 
attempted component data.   
[Jul 31, 2018 11:26:09 PM] (bharat) HDDS-271. Create a block iterator to 
iterate blocks in a container.
[Aug 1, 2018 1:44:49 AM] (fabbri) HDFS-13322 fuse dfs - uid persists when 
switching between ticket caches.
[Aug 1, 2018 3:03:30 AM] (sunilg) YARN-8397. Potential thread leak in 
ActivitiesManager. Contributed by
[Aug 1, 2018 3:33:00 AM] (msingh) HDDS-226. Client should update block length 
in OM while committing the
[Aug 1, 2018 5:34:53 AM] (wangda) YARN-8522. Application fails with 
InvalidResourceRequestException. (Zian
[Aug 1, 2018 6:47:18 AM] (sunilg) YARN-8606. Opportunistic scheduling does not 
work post RM failover.
[Aug 1, 2018 8:57:54 AM] (sunilg) YARN-8595. [UI2] Container diagnostic 
information is missing from




-1 overall


The following subsystems voted -1:
compile mvninstall pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


The following subsystems are considered long running:
(runtime bigger than 1h 00m 00s)
unit


Specific tests:

Failed junit tests :

   hadoop.crypto.key.kms.server.TestKMS 
   hadoop.cli.TestAclCLI 
   hadoop.cli.TestAclCLIWithPosixAclInheritance 
   hadoop.cli.TestCacheAdminCLI 
   hadoop.cli.TestCryptoAdminCLI 
   hadoop.cli.TestDeleteCLI 
   hadoop.cli.TestErasureCodingCLI 
   hadoop.cli.TestHDFSCLI 
   hadoop.cli.TestXAttrCLI 
   hadoop.fs.contract.hdfs.TestHDFSContractAppend 
   hadoop.fs.contract.hdfs.TestHDFSContractConcat 
   hadoop.fs.contract.hdfs.TestHDFSContractCreate 
   hadoop.fs.contract.hdfs.TestHDFSContractDelete 
   hadoop.fs.contract.hdfs.TestHDFSContractGetFileStatus 
   hadoop.fs.contract.hdfs.TestHDFSContractMkdir 
   hadoop.fs.contract.hdfs.TestHDFSContractOpen 
   hadoop.fs.contract.hdfs.TestHDFSContractPathHandle 
   hadoop.fs.contract.hdfs.TestHDFSContractRename 
   hadoop.fs.contract.hdfs.TestHDFSContractRootDirectory 
   hadoop.fs.contract.hdfs.TestHDFSContractSeek 
   hadoop.fs.contract.hdfs.TestHDFSContractSetTimes 
   hadoop.fs.loadGenerator.TestLoadGenerator 
   hadoop.fs.permission.TestStickyBit 
   hadoop.fs.shell.TestHdfsTextCommand 
   hadoop.fs.TestEnhancedByteBufferAccess 
   hadoop.fs.TestFcHdfsCreateMkdir 
   hadoop.fs.TestFcHdfsPermission 
   hadoop.fs.TestFcHdfsSetUMask 
   hadoop.fs.TestGlobPaths 
   hadoop.fs.TestHDFSFileContextMainOperations 
   hadoop.fs.TestHDFSMultipartUploader 
   hadoop.fs.TestHdfsNativeCodeLoader 
   hadoop.fs.TestResolveHdfsSymlink 
   hadoop.fs.TestSWebHdfsFileContextMainOperations 
   hadoop.fs.TestSymlinkHdfsDisable 
   hadoop.fs.TestSymlinkHdfsFileContext 
   hadoop.fs.TestSymlinkHdfsFileSystem 
   hadoop.fs.TestUnbuffer 
   hadoop.fs.TestUrlStreamHandler 
   hadoop.fs.TestWebHdfsFileContextMainOperations 
   hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot 
   hadoop.fs.viewfs.TestViewFileSystemHdfs 
   hadoop.fs.viewfs.TestViewFileSystemLinkFallback 
   hadoop.fs.viewfs.TestViewFileSystemLinkMergeSlash 
   hadoop.fs.viewfs.TestViewFileSystemWithAcls 
   hadoop.fs.viewfs.TestViewFileSystemWithTruncate 
   hadoop.fs.viewfs.TestViewFileSystemWithXAttrs 
   hadoop.fs.viewfs.TestViewFsAtHdfsRoot 
   hadoop.fs.viewfs.TestViewFsDefaultValue 
   hadoop.fs.viewfs.TestViewFsFileStatusHdfs 
   hadoop.fs.viewfs.TestViewFsHdfs 
   hadoop.fs.viewfs.TestViewFsWithAcls 
   hadoop.fs.viewfs.TestViewFsWithXAttrs 
   hadoop.hdfs.client.impl.TestBlockReaderLocal 
   hadoop.hdfs.client.impl.TestBlockReaderLocalLegacy 
   hadoop.hdfs.client.impl.TestBlockReaderRemote 
   hadoop.hdfs.client.impl.TestClientBlockVerification 
   hadoop.hdfs.crypto.TestHdfsCryptoStreams 
   hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer 
   hadoop.hdfs.qjournal.client.TestEpochsAreUnique 
   hadoop.hdfs.qjournal.client.TestQJMWithFaults 
   hadoop.hdfs.qjournal.client.TestQuorumJournalManager 
   hadoop.hdfs.qjournal.server.TestJournal 
   hadoop.hdfs.qjournal.server.TestJournalNode 
   

[jira] [Created] (HDDS-311) Clean up pipeline code in Container mapping.

[Ajay Kumar (JIRA)]

Ajay Kumar created HDDS-311:
---

 Summary: Clean up pipeline code in Container mapping.
 Key: HDDS-311
 URL: https://issues.apache.org/jira/browse/HDDS-311
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Ajay Kumar


Originally suggested by [~msingh] in HDDS-298. Jira intends to discuss removal 
of getReplicationPipeline in {{ContainerMapping}} L208, L475,L532 as it creates 
a new pipeline.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13777) [PROVIDED Phase 2] Scheduler in the NN for distributing DNA_BACKUP work.

[Ewan Higgs (JIRA)]

Ewan Higgs created HDFS-13777:
-

 Summary: [PROVIDED Phase 2] Scheduler in the NN for distributing 
DNA_BACKUP work.
 Key: HDFS-13777
 URL: https://issues.apache.org/jira/browse/HDFS-13777
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Ewan Higgs
Assignee: Ewan Higgs


When the SyncService is running, it should periodically take snapshots, make a 
snapshotdiff, and then distribute DNA_BACKUP work to the Datanodes (See 
HDFS-13421). Upon completion of the work, the NN should update the AliasMap.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/855/

[Jul 31, 2018 11:57:51 AM] (nanda) HDDS-305. Datanode 
StateContext#addContainerActionIfAbsent will add
[Jul 31, 2018 2:20:40 PM] (msingh) HDDS-279. DeleteBlocks command should not be 
sent for open containers.
[Jul 31, 2018 4:36:34 PM] (inigoiri) YARN-8175. Add support for Node Labels in 
SLS. Contributed by Abhishek
[Jul 31, 2018 6:32:40 PM] (haibochen) YARN-8605. 
TestDominantResourceFairnessPolicy.testModWhileSorting is
[Jul 31, 2018 7:08:00 PM] (wangda) YARN-8418. App local logs could leaked if 
log aggregation fails to
[Jul 31, 2018 10:01:02 PM] (eyang) YARN-8579.  Recover NMToken of previous 
attempted component data.   
[Jul 31, 2018 11:26:09 PM] (bharat) HDDS-271. Create a block iterator to 
iterate blocks in a container.




-1 overall


The following subsystems voted -1:
asflicense findbugs pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed CTEST tests :

   test_test_libhdfs_threaded_hdfs_static 
   test_libhdfs_threaded_hdfspp_test_shim_static 

Failed junit tests :

   hadoop.security.TestFixKerberosTicketOrder 
   hadoop.hdfs.server.balancer.TestBalancer 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler 
   hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
   hadoop.mapred.TestMRTimelineEventHandling 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/855/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/855/artifact/out/diff-compile-javac-root.txt
  [332K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/855/artifact/out/diff-checkstyle-root.txt
  [4.0K]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/855/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/855/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/855/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/855/artifact/out/diff-patch-shelldocs.txt
  [16K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/855/artifact/out/whitespace-eol.txt
  [9.4M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/855/artifact/out/whitespace-tabs.txt
  [1.1M]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/855/artifact/out/xml.txt
  [4.0K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/855/artifact/out/branch-findbugs-hadoop-hdds_client.txt
  [56K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/855/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt
  [52K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/855/artifact/out/branch-findbugs-hadoop-hdds_framework.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/855/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt
  [56K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/855/artifact/out/branch-findbugs-hadoop-hdds_tools.txt
  [16K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/855/artifact/out/branch-findbugs-hadoop-ozone_client.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/855/artifact/out/branch-findbugs-hadoop-ozone_common.txt
  [28K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/855/artifact/out/branch-findbugs-hadoop-ozone_objectstore-service.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/855/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/855/artifact/out/branch-findbugs-hadoop-ozone_ozonefs.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/855/artifact/out/branch-findbugs-hadoop-ozone_tools.txt
  [4.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/855/artifact/out/diff-javadoc-javadoc-root.txt
  [760K]

   CTEST:

   

[jira] [Created] (HDFS-13776) Add Storage policies related ClientProtocol methods

[Dibyendu Karmakar (JIRA)]

Dibyendu Karmakar created HDFS-13776:


 Summary: Add Storage policies related ClientProtocol methods
 Key: HDFS-13776
 URL: https://issues.apache.org/jira/browse/HDFS-13776
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Dibyendu Karmakar
Assignee: Dibyendu Karmakar


Currently unsetStoragePolicy and getStoragePolicy are not implemented in 
RouterRpcServer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-310) VolumeSet shutdown hoot fails on datanode restart

[Mukul Kumar Singh (JIRA)]

Mukul Kumar Singh created HDDS-310:
--

 Summary: VolumeSet shutdown hoot fails on datanode restart
 Key: HDDS-310
 URL: https://issues.apache.org/jira/browse/HDDS-310
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: Ozone Datanode
Affects Versions: 0.2.1
Reporter: Mukul Kumar Singh
 Fix For: 0.2.1


{code}
2018-08-01 11:01:57,204 WARN org.apache.hadoop.fs.CachingGetSpaceUsed: Thread 
Interrupted waiting to refresh disk information: sleep interrupted
2018-08-01 11:01:57,204 WARN org.apache.hadoop.util.ShutdownHookManager: 
ShutdownHook 'VolumeSet$$Lambda$13/360062456' failed, 
java.util.concurrent.ExecutionException: java.lang.IllegalStateException: 
Shutdown in progress, cannot remove a shutdownHook
java.util.concurrent.ExecutionException: java.lang.IllegalStateException: 
Shutdown in progress, cannot remove a shutdownHook
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at 
org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:68)
Caused by: java.lang.IllegalStateException: Shutdown in progress, cannot remove 
a shutdownHook
at 
org.apache.hadoop.util.ShutdownHookManager.removeShutdownHook(ShutdownHookManager.java:247)
at 
org.apache.hadoop.ozone.container.common.volume.VolumeSet.shutdown(VolumeSet.java:317)
at 
org.apache.hadoop.ozone.container.common.volume.VolumeSet.lambda$initializeVolumeSet$0(VolumeSet.java:170)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-309) closePipelineIfNoOpenContainers should remove pipeline from activePipelines list.

[chencan (JIRA)]

chencan created HDDS-309:


 Summary: closePipelineIfNoOpenContainers should remove pipeline 
from activePipelines list.
 Key: HDDS-309
 URL: https://issues.apache.org/jira/browse/HDDS-309
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: SCM
Reporter: chencan


    Function closePipeline remove pipeline from pipelineMap and 
node2PipelineMap.If closePipeline is called by 
closePipelineIfNoOpenContainers，Are we supposed to remove pipeline from 
activePipelines?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[VOTE] Merge Storage Policy Satisfier (SPS) [HDFS-10285] feature branch to trunk

[Uma Maheswara Rao G]

Hi All,



 From the positive responses from JIRA discussion and no objections from
below DISCUSS thread [1], I am converting it to voting thread.



 Last couple of weeks we spent time on testing the feature and so far it is
working fine. Surendra uploaded a test report at HDFS-10285:  [2]



 In this phase, we provide to run SPS outside of Namenode only and as a
next phase we continue to discuss and work on to enable it as Internal SPS
as explained below. We have got clean QA report on branch and if there are
any static tool comments triggered later while running this thread, we will
make sure to fix them before merge. We committed and continue to improve
the code on trunk. Please refer to HDFS-10285 for discussion details.



 This has been a long effort and we're grateful for the support we've
received from the community. In particular, thanks to Andrew Wang, Anoop
Sam John, Anu Engineer, Chris Douglas, Daryn Sharp, Du Jingcheng , Ewan
Higgs, Jing Zhao, Kai Zheng,  Rakesh R, Ramkrishna , Surendra Singh Lilhore
, Thomas Demoor, Uma Maheswara Rao G, Vinayakumar, Virajith,  Wei Zhou,
Yuanbo Liu. Without these members effort, this feature might not have
reached to this state.



To start with, here is my +1

It will end on 6th Aug.



Regards,

Uma

[1]  https://s.apache.org/bhyu
[2]  https://s.apache.org/AXvL


On Wed, Jun 27, 2018 at 3:21 PM, Uma Maheswara Rao G 
wrote:

> Hi All,
>
>   After long discussions(offline and on JIRA) on SPS, we came to a
> conclusion on JIRA(HDFS-10285) that, we will go ahead with External SPS
> merge in first phase. In this phase process will not be running inside
> Namenode.
>   We will continue discussion on Internal SPS. Current code base supports
> both internal and external option. We have review comments for Internal
> which needs some additional works for analysis and testing etc. We will
> move Internal SPS work to under HDFS-12226 (Follow-on work for SPS in NN)
> We are working on cleanup task HDFS-13076 for the merge. .
> For more clarity on Internal and External SPS proposal thoughts, please
> refer to JIRA HDFS-10285.
>
> If there are no objections with this, I will go ahead for voting soon.
>
> Regards,
> Uma
>
> On Fri, Nov 17, 2017 at 3:16 PM, Uma Maheswara Rao G  > wrote:
>
>> Update: We worked on the review comments and additional JIRAs above
>> mentioned.
>>
>> >1. After the feedbacks from Andrew, Eddy, Xiao in JIRA reviews, we
>> planned to take up the support for recursive API support. HDFS-12291<
>> https://issues.apache.org/jira/browse/HDFS-12291>
>>
>> We provided the recursive API support now.
>>
>> >2. Xattr optimizations HDFS-12225> he.org/jira/browse/HDFS-12225>
>> Improved this portion as well
>>
>> >3. Few other review comments already fixed and committed HDFS-12214<
>> https://issues.apache.org/jira/browse/HDFS-12214>
>> Fixed the comments.
>>
>> We are continuing to test the feature and working so far well. Also we
>> uploaded a combined patch and got the good QA report.
>>
>> If there are no further objections, we would like to go for merge vote
>> tomorrow. Please by default this feature will be disabled.
>>
>> Regards,
>> Uma
>>
>> On Fri, Aug 18, 2017 at 11:27 PM, Gangumalla, Uma <
>> uma.ganguma...@intel.com> wrote:
>>
>>> Hi Andrew,
>>>
>>> >Great to hear. It'd be nice to define which use cases are met by the
>>> current version of SPS, and which will be handled after the merge.
>>> After the discussions in JIRA, we planned to support recursive API as
>>> well. The primary use cases we planned was for Hbase. Please check next
>>> point for use case details.
>>>
>>> >A bit more detail in the design doc on how HBase would use this feature
>>> would also be helpful. Is there an HBase JIRA already?
>>> Please find the usecase details at this comment in JIRA:
>>> https://issues.apache.org/jira/browse/HDFS-10285?focusedComm
>>> entId=16120227=com.atlassian.jira.plugin.system.issueta
>>> bpanels:comment-tabpanel#comment-16120227
>>>
>>> >I also spent some more time with the design doc and posted a few
>>> questions on the JIRA.
>>> Thank you for the reviews.
>>>
>>> To summarize the discussions in JIRA:
>>> 1. After the feedbacks from Andrew, Eddy, Xiao in JIRA reviews, we
>>> planned to take up the support for recursive API support. HDFS-12291<
>>> https://issues.apache.org/jira/browse/HDFS-12291> (Rakesh started the
>>> work on it)
>>> 2. Xattr optimizations HDFS-12225>> he.org/jira/browse/HDFS-12225> (Patch available)
>>> 3. Few other review comments already fixed and committed HDFS-12214<
>>> https://issues.apache.org/jira/browse/HDFS-12214>
>>>
>>> For tracking the follow-up tasks we filed JIRA HDFS-12226, they should
>>> not be critical for merge.
>>>
>>> Regards,
>>> Uma
>>>
>>> From: Andrew Wang >> andrew.w...@cloudera.com>>
>>> Date: Friday, July 28, 2017 at 11:33 AM
>>> To: Uma Gangumalla >> uma.ganguma...@intel.com>>
>>> Cc: "hdfs-dev@hadoop.apache.org" <