Re: [Vote] Merge discussion for Node attribute support feature YARN-3409

2018-09-11 Thread Naganarasimha Garla 
Hi All,
 Voting has been running since 6 days and adding my vote we have 4
binding and 2 non binding +1's with no -1's this voting passes and we will
be merging the branch shortly. Thanks for all who participated in the
discussion and voting thread !

Thanks and Regards,
+ Naga

On Mon, Sep 10, 2018 at 2:50 PM Zian Chen  wrote:

> +1 for merge.
>
> > On Sep 9, 2018, at 10:47 PM, Weiwei Yang  wrote:
> >
> > +1 for the merge
> >
> > On Mon, Sep 10, 2018 at 12:06 PM Rohith Sharma K S <
> > rohithsharm...@apache.org> wrote:
> >
> >> +1 for merge
> >>
> >> -Rohith Sharma K S
> >>
> >> On Wed, 5 Sep 2018 at 18:01, Naganarasimha Garla <
> >> naganarasimha...@apache.org> wrote:
> >>
> >>> Hi All,
> >>>Thanks for feedback folks, based on the positive response
> >> starting
> >>> a Vote thread for merging YARN-3409 to master.
> >>>
> >>> Regards,
> >>> + Naga & Sunil
> >>>
> >>> On Wed, 5 Sep 2018 2:51 am Wangda Tan,  wrote:
> >>>
>  +1 for the merge, it gonna be a great addition to 3.2.0 release.
> Thanks
> >>> to
>  everybody for pushing this feature to complete.
> 
>  Best,
>  Wangda
> 
>  On Tue, Sep 4, 2018 at 8:25 AM Bibinchundatt <
> >> bibin.chund...@huawei.com>
>  wrote:
> 
> > +1 for merge. Fetaure would be a good addition to 3.2 release.
> >
> > --
> > Bibin A Chundatt
> > M: +91-9742095715
> > E: bibin.chund...@huawei.com
> > 2012实验室-印研IT BU分部
> > 2012 Laboratories-IT BU Branch Dept.
> > From:Naganarasimha Garla
> > To:common-...@hadoop.apache.org,Hdfs-dev,yarn-...@hadoop.apache.org,
> > mapreduce-...@hadoop.apache.org,
> > Date:2018-08-29 20:00:44
> > Subject:[Discuss] Merge discussion for Node attribute support feature
> > YARN-3409
> >
> > Hi All,
> >
> > We would like to hear your thoughts on merging “Node Attributes
> >> Support
> >>> in
> > YARN” branch (YARN-3409) [2] into trunk in a few weeks. The goal is
> to
> >>> get
> > it in for HADOOP 3.2.
> >
> > *Major work happened in this branch*
> >
> > YARN-6858. Attribute Manager to store and provide node attributes in
> >> RM
> > YARN-7871. Support Node attributes reporting from NM to RM(
> >> distributed
> > node attributes)
> > YARN-7863. Modify placement constraints to support node attributes
> > YARN-7875. Node Attribute store for storing and recovering attributes
> >
> > *Detailed Design:*
> >
> > Please refer [1] for detailed design document.
> >
> > *Testing Efforts:*
> >
> > We did detailed tests for the feature in the last few weeks.
> > This feature will be enabled only when Node Attributes constraints
> are
> > specified through SchedulingRequest from AM.
> > Manager implementation will help to store and recover Node
> Attributes.
> > This
> > works with existing placement constraints.
> >
> > *Regarding to API stability:*
> >
> > All newly added @Public APIs are @Unstable.
> >
> > Documentation jira [3] could help to provide detailed configuration
> > details. This feature works from end-to-end and we tested this in our
> > local
> > cluster. Branch code is run against trunk and tracked via [4].
> >
> > We would love to get your thoughts before opening a voting thread.
> >
> > Special thanks to a team of folks who worked hard and contributed
> >>> towards
> > this efforts including design discussion / patch / reviews, etc.:
> >> Weiwei
> > Yang, Bibin Chundatt, Wangda Tan, Vinod Kumar Vavilappali,
> >> Konstantinos
> > Karanasos, Arun Suresh, Varun Saxena, Devaraj Kavali, Lei Guo, Chong
> >>> Chen.
> >
> > [1] :
> >
> >
> >>>
> >>
> https://issues.apache.org/jira/secure/attachment/12937633/Node-Attributes-Requirements-Design-doc_v2.pdf
> > [2] : https://issues.apache.org/jira/browse/YARN-3409
> > [3] : https://issues.apache.org/jira/browse/YARN-7865
> > [4] : https://issues.apache.org/jira/browse/YARN-8718
> >
> > Thanks,
> > + Naga & Sunil Govindan
> >
> 
> >>>
> >>
> >
> >
> > --
> > Weiwei Yang
>
>

Re: [VOTE] Release Apache Hadoop 2.8.5 (RC0)

2018-09-11 Thread Eric Badger 
+1 (non-binding)

- Verified all hashes and checksums
- Built from source on macOS 10.13.6, Java 1.8.0u65
- Deployed a pseudo cluster
- Ran some example jobs

Eric

On Tue, Sep 11, 2018 at 1:39 PM, Gabor Bota  wrote:

>   Thanks for the work Junping!
>
>   +1 (non-binding)
>
> - checked out git tag release-2.8.5-RC0
> - built from source on Mac OS X 10.13.6, java version 8.0.181-oracle
> - deployed on a 3 node cluster
> - verified pi job (yarn), teragen, terasort and teravalidate
>
>   Regards,
>   Gabor Bota
>
> On Tue, Sep 11, 2018 at 6:31 PM Eric Payne  invalid>
> wrote:
>
> > Thanks a lot Junping!
> >
> > +1 (binding)
> >
> > Tested the following:
> > - Built from source
> > - Installed on a 7 node, multi-tenant, insecure pseudo cluster, running
> > YARN capacity scheduler
> > - Added a queue via refresh
> > - Verified various GUI pages
> > - Streaming jobs
> > - Cross-queue (Inter) preemption
> > - In-queue (Intra) preemption
> > - Teragen / terasort
> >
> >
> > -Eric
> >
> >
> >
> >
> > On Monday, September 10, 2018, 7:01:46 AM CDT, 俊平堵 <
> junping...@apache.org>
> > wrote:
> >
> >
> >
> >
> >
> > Hi all,
> >
> > I've created the first release candidate (RC0) for Apache
> > Hadoop 2.8.5. This is our next point release to follow up 2.8.4. It
> > includes 33 important fixes and improvements.
> >
> >
> > The RC artifacts are available at:
> > http://home.apache.org/~junping_du/hadoop-2.8.5-RC0
> >
> >
> > The RC tag in git is: release-2.8.5-RC0
> >
> >
> >
> > The maven artifacts are available via repository.apache.org<
> > http://repository.apache.org> at:
> >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1140
> >
> >
> > Please try the release and vote; the vote will run for the usual 5
> > working
> > days, ending on 9/15/2018 PST time.
> >
> >
> > Thanks,
> >
> >
> > Junping
> >
> > -
> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >
> >
>

[jira] [Created] (HDFS-13910) Improve BlockPlacementPolicy for small clusters

2018-09-11 Thread Shweta (JIRA) 
Shweta created HDFS-13910:
-

 Summary: Improve BlockPlacementPolicy for small clusters
 Key: HDFS-13910
 URL: https://issues.apache.org/jira/browse/HDFS-13910
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs
Reporter: Shweta
Assignee: Shweta


>From investigations and a few test occurrences, the NameNode 
>BlockPlacementPolicy’s considerLoad can be bad for small test clusters.
A small (1-node) cluster may trigger a corner case of maxLoad = 0. In this 
case, filtering should not take place. 

When there is no heartbeat, maxLoad will be 0 and the existing logic will 
filter out many nodes that have workload, which should be avoided.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-13900) NameNode: Unable to trigger a roll of the active NN

2018-09-11 Thread JIRA 


 [ 
https://issues.apache.org/jira/browse/HDFS-13900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-13900.

Resolution: Duplicate

> NameNode: Unable to trigger a roll of the active NN
> ---
>
> Key: HDFS-13900
> URL: https://issues.apache.org/jira/browse/HDFS-13900
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: liuhongtong
>Priority: Critical
>
> I have backport Multi-standby NNs to our own hdfs version. I found an issue 
> of EditLog roll.
> h2. Reproducible Steps：
> h3. 1.original state
> nn1 active
> nn2 standby
> nn3 standby
> h3. 2. stop nn1
> h3. 3. new state
> nn1 stopped
> nn2 active
> nn3 standby
> h3. 4. nn3 unable to trigger a roll of the active NN
> [2018-08-22T10:33:38.025+08:00] [WARN] 
> namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java 307) [Edit 
> log tailer] : Unable to trigger a roll of the active NN
> java.net.ConnectException: Call From  to  failed 
> on connection exception: java.net.ConnectException: Connection refused; For 
> more details see:[http://wiki.apache.org/hadoop/ConnectionRefused]
> at sun.reflect.GeneratedConstructorAccessor17.newInstance(Unknown Source)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:722)
> at org.apache.hadoop.ipc.Client.call(Client.java:1536)
> at org.apache.hadoop.ipc.Client.call(Client.java:1463)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:237)
> at com.sun.proxy.$Proxy16.rollEditLog(Unknown Source)
> at 
> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:148)
> at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:301)
> at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:298)
> at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$MultipleNameNodeProxy.call(EditLogTailer.java:414)
> at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:304)
> at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$800(EditLogTailer.java:69)
> at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:346)
> at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:315)
> at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:332)
> at 
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:415)
> at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:328)
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at 
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:521)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:485)
> at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:658)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:756)
> at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:419)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1585)
> at org.apache.hadoop.ipc.Client.call(Client.java:1502)
> ... 14 more



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.5 (RC0)

2018-09-11 Thread Gabor Bota 
  Thanks for the work Junping!

  +1 (non-binding)

- checked out git tag release-2.8.5-RC0
- built from source on Mac OS X 10.13.6, java version 8.0.181-oracle
- deployed on a 3 node cluster
- verified pi job (yarn), teragen, terasort and teravalidate

  Regards,
  Gabor Bota

On Tue, Sep 11, 2018 at 6:31 PM Eric Payne 
wrote:

> Thanks a lot Junping!
>
> +1 (binding)
>
> Tested the following:
> - Built from source
> - Installed on a 7 node, multi-tenant, insecure pseudo cluster, running
> YARN capacity scheduler
> - Added a queue via refresh
> - Verified various GUI pages
> - Streaming jobs
> - Cross-queue (Inter) preemption
> - In-queue (Intra) preemption
> - Teragen / terasort
>
>
> -Eric
>
>
>
>
> On Monday, September 10, 2018, 7:01:46 AM CDT, 俊平堵 
> wrote:
>
>
>
>
>
> Hi all,
>
> I've created the first release candidate (RC0) for Apache
> Hadoop 2.8.5. This is our next point release to follow up 2.8.4. It
> includes 33 important fixes and improvements.
>
>
> The RC artifacts are available at:
> http://home.apache.org/~junping_du/hadoop-2.8.5-RC0
>
>
> The RC tag in git is: release-2.8.5-RC0
>
>
>
> The maven artifacts are available via repository.apache.org<
> http://repository.apache.org> at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1140
>
>
> Please try the release and vote; the vote will run for the usual 5
> working
> days, ending on 9/15/2018 PST time.
>
>
> Thanks,
>
>
> Junping
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (HDDS-433) ContainerStateMachine#readStateMachineData should properly build LogEntryProto

2018-09-11 Thread Lokesh Jain (JIRA) 
Lokesh Jain created HDDS-433:


 Summary: ContainerStateMachine#readStateMachineData should 
properly build LogEntryProto
 Key: HDDS-433
 URL: https://issues.apache.org/jira/browse/HDDS-433
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: Ozone Datanode
Affects Versions: 0.2.1
Reporter: Lokesh Jain
Assignee: Lokesh Jain


ContainerStateMachine#readStateMachineData returns LogEntryProto with index set 
to 0. This leads to exception in Ratis. The LogEntryProto to return should be 
built over the input LogEntryProto.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-12615) Router-based HDFS federation phase 2

2018-09-11 Thread JIRA 


 [ 
https://issues.apache.org/jira/browse/HDFS-12615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-12615.

   Resolution: Fixed
Fix Version/s: 3.2.0

> Router-based HDFS federation phase 2
> 
>
> Key: HDFS-12615
> URL: https://issues.apache.org/jira/browse/HDFS-12615
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
>  Labels: RBF
> Fix For: 3.2.0
>
>
> This umbrella JIRA tracks set of improvements over the Router-based HDFS 
> federation (HDFS-10467).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.5 (RC0)

2018-09-11 Thread Eric Payne 
Thanks a lot Junping!

+1 (binding)

Tested the following:
- Built from source
- Installed on a 7 node, multi-tenant, insecure pseudo cluster, running YARN 
capacity scheduler
- Added a queue via refresh
- Verified various GUI pages
- Streaming jobs
- Cross-queue (Inter) preemption
- In-queue (Intra) preemption
- Teragen / terasort


-Eric




On Monday, September 10, 2018, 7:01:46 AM CDT, 俊平堵  
wrote: 





Hi all,

    I've created the first release candidate (RC0) for Apache
Hadoop 2.8.5. This is our next point release to follow up 2.8.4. It
includes 33 important fixes and improvements.


    The RC artifacts are available at:
http://home.apache.org/~junping_du/hadoop-2.8.5-RC0


    The RC tag in git is: release-2.8.5-RC0



    The maven artifacts are available via repository.apache.org<
http://repository.apache.org> at:

https://repository.apache.org/content/repositories/orgapachehadoop-1140


    Please try the release and vote; the vote will run for the usual 5 working
days, ending on 9/15/2018 PST time.


Thanks,


Junping

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Speakers needed for Apache DC Roadshow

2018-09-11 Thread Rich Bowen 
We need your help to make the Apache Washington DC Roadshow on Dec 4th a 
success.


What do we need most? Speakers!

We're bringing a unique DC flavor to this event by mixing Open Source 
Software with talks about Apache projects as well as OSS CyberSecurity, 
OSS in Government and and OSS Career advice.


Please take a look at: http://www.apachecon.com/usroadshow18/

(Note: You are receiving this message because you are subscribed to one 
or more mailing lists at The Apache Software Foundation.)


Rich, for the ApacheCon Planners

--
rbo...@apache.org
http://apachecon.com
@ApacheCon

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-432) Replication of closed containers is not working

2018-09-11 Thread Elek, Marton (JIRA) 
Elek, Marton created HDDS-432:
-

 Summary: Replication of closed containers is not working
 Key: HDDS-432
 URL: https://issues.apache.org/jira/browse/HDDS-432
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: SCM
Reporter: Elek, Marton
Assignee: Elek, Marton
 Fix For: 0.2.1


Steps to reproduce:

1. Start a cluster with three datanodes:

docker-compose up -d
docker-compose scale datanode=3

2. Create keys:
ozone oz -createVolume /vol1 -user hadoop --quota 1TB --root
ozone oz -createBucket /vol1/bucket
dd if=/dev/zero of=/tmp/test bs=1024000 count=512
ozone oz -putKey /vol1/bucket/file1 -replicationFactor THREE -file /tmp/test

3. Close the containers with scmcli

4. kill a datanode with a replica

{code}
for i in `seq 1 4`; do docker diff ozone_datanode_$i && echo 
""; done
#Choose a datanode with replica
docker kill ozone_datanode_3
{code}
 
5. Wait

6. After a while the last data node should container the chunks (checked with 
docker diff)




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-431) LeaseManager of CloseContainers are not started

2018-09-11 Thread Elek, Marton (JIRA) 
Elek, Marton created HDDS-431:
-

 Summary: LeaseManager of CloseContainers are not started
 Key: HDDS-431
 URL: https://issues.apache.org/jira/browse/HDDS-431
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: SCM
Reporter: Elek, Marton
 Fix For: 0.2.1


You can see the following error in case of a datanode failure:

{code}
scm_1   | 2018-09-11 11:43:46 ERROR SingleThreadExecutor:88 - Error on 
execution message 
org.apache.hadoop.hdds.scm.container.CloseContainerEventHandler$CloseContainerRetryableReq@2aa17d1cscm_1
   | org.apache.hadoop.ozone.lease.LeaseManagerNotRunningException: 
LeaseManager not running.scm_1   | at 
org.apache.hadoop.ozone.lease.LeaseManager.checkStatus(LeaseManager.java:189)scm_1
   |   at 
org.apache.hadoop.ozone.lease.LeaseManager.acquire(LeaseManager.java:112)scm_1  
 |   at 
org.apache.hadoop.ozone.lease.LeaseManager.acquire(LeaseManager.java:97)scm_1   
|at 
org.apache.hadoop.hdds.server.events.EventWatcher.handleStartEvent(EventWatcher.java:128)scm_1
   |   at 
org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:85)scm_1
   |  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)scm_1
   | at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)scm_1
   | at java.lang.Thread.run(Thread.java:748)scm_1   | 
2018-09-11 11:43:46 ERROR SingleThreadExecutor:88 - Error on execution message 
org.apache.hadoop.hdds.scm.container.CloseContainerEventHandler$CloseContainerRetryableReq@2772f338scm_1
   | org.apache.hadoop.ozone.lease.LeaseManagerNotRunningException: 
LeaseManager not running.scm_1   | at 
org.apache.hadoop.ozone.lease.LeaseManager.checkStatus(LeaseManager.java:189)scm_1
   |   at 
org.apache.hadoop.ozone.lease.LeaseManager.acquire(LeaseManager.java:112)scm_1  
 |   at 
org.apache.hadoop.ozone.lease.LeaseManager.acquire(LeaseManager.java:97)scm_1   
|at 
org.apache.hadoop.hdds.server.events.EventWatcher.handleStartEvent(EventWatcher.java:128)scm_1
   |   at 
org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:85)scm_1
   |  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)scm_1
   | at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)scm_1
   | at java.lang.Thread.run(Thread.java:748)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-430) Close piplineActions are duplicated in the HB response

2018-09-11 Thread Elek, Marton (JIRA) 
Elek, Marton created HDDS-430:
-

 Summary: Close piplineActions are duplicated in the HB response
 Key: HDDS-430
 URL: https://issues.apache.org/jira/browse/HDDS-430
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: SCM
 Environment: In case of a datanode failure the closePipline action 
will be sent to the datanodes.

But there are a lot of actions for the same pipline:

{code}
 | pipelineActions {
scm_1   |   pipelineActions {
scm_1   | action: CLOSE
scm_1   | closePipeline {
scm_1   |   pipelineID {
scm_1   | id: "9df11f3a-565f-48bc-81f9-3b6c89876f47"
scm_1   |   }
scm_1   |   reason: PIPELINE_FAILED
scm_1   |   detailedReason: "46e4a0a9-e7d1-4f51-9434-d001b477a804 
has not seen follower/s e81b12d7-8234-4394-b621-dc71af4e3093 for 120331ms"
scm_1   | }
scm_1   |   }
scm_1   |   pipelineActions {
scm_1   | action: CLOSE
scm_1   | closePipeline {
scm_1   |   pipelineID {
scm_1   | id: "9df11f3a-565f-48bc-81f9-3b6c89876f47"
scm_1   |   }
scm_1   |   reason: PIPELINE_FAILED
scm_1   |   detailedReason: "46e4a0a9-e7d1-4f51-9434-d001b477a804 
has not seen follower/s e81b12d7-8234-4394-b621-dc71af4e3093 for 120753ms"
scm_1   | }
scm_1   |   }
scm_1   |   pipelineActions {
scm_1   | action: CLOSE
scm_1   | closePipeline {
scm_1   |   pipelineID {
scm_1   | id: "9df11f3a-565f-48bc-81f9-3b6c89876f47"
scm_1   |   }
scm_1   |   reason: PIPELINE_FAILED
scm_1   |   detailedReason: "46e4a0a9-e7d1-4f51-9434-d001b477a804 
has not seen follower/s e81b12d7-8234-4394-b621-dc71af4e3093 for 121154ms"
scm_1   | }
scm_1   |   }
scm_1   |   pipelineActions {
scm_1   | action: CLOSE
scm_1   | closePipeline {
scm_1   |   pipelineID {
scm_1   | id: "9df11f3a-565f-48bc-81f9-3b6c89876f47"
scm_1   |   }
scm_1   |   reason: PIPELINE_FAILED
scm_1   |   detailedReason: "46e4a0a9-e7d1-4f51-9434-d001b477a804 
has not seen follower/s e81b12d7-8234-4394-b621-dc71af4e3093 for 121554ms"
scm_1   | }
scm_1   |   }
scm_1   |   pipelineActions {
scm_1   | action: CLOSE
scm_1   | closePipeline {
scm_1   |   pipelineID {
scm_1   | id: "9df11f3a-565f-48bc-81f9-3b6c89876f47"
scm_1   |   }
scm_1   |   reason: PIPELINE_FAILED
scm_1   |   detailedReason: "46e4a0a9-e7d1-4f51-9434-d001b477a804 
has not seen follower/s e81b12d7-8234-4394-b621-dc71af4e3093 for 121955ms"
scm_1   | }
scm_1   |   }
scm_1   |   pipelineActions {
scm_1   | action: CLOSE
scm_1   | closePipeline {
scm_1   |   pipelineID {
scm_1   | id: "9df11f3a-565f-48bc-81f9-3b6c89876f47"
scm_1   |   }
scm_1   |   reason: PIPELINE_FAILED
scm_1   |   detailedReason: "46e4a0a9-e7d1-4f51-9434-d001b477a804 
has not seen follower/s e81b12d7-8234-4394-b621-dc71af4e3093 for 122356ms"
scm_1   | }
scm_1   |   }
scm_1   |   pipelineActions {
scm_1   | action: CLOSE
scm_1   | closePipeline {
scm_1   |   pipelineID {
scm_1   | id: "9df11f3a-565f-48bc-81f9-3b6c89876f47"
scm_1   |   }
scm_1   |   reason: PIPELINE_FAILED
scm_1   |   detailedReason: "46e4a0a9-e7d1-4f51-9434-d001b477a804 
has not seen follower/s e81b12d7-8234-4394-b621-dc71af4e3093 for 122756ms"
scm_1   | }
scm_1   |   }
scm_1   |   pipelineActions {
scm_1   | action: CLOSE
scm_1   | closePipeline {
scm_1   |   pipelineID {
scm_1   | id: "9df11f3a-565f-48bc-81f9-3b6c89876f47"
scm_1   |   }
scm_1   |   reason: PIPELINE_FAILED
scm_1   |   detailedReason: "46e4a0a9-e7d1-4f51-9434-d001b477a804 
has not seen follower/s e81b12d7-8234-4394-b621-dc71af4e3093 for 123158ms"
scm_1   | }
scm_1   |   }
scm_1   |   pipelineActions {
scm_1   | action: CLOSE
scm_1   | closePipeline {
scm_1   |   pipelineID {
scm_1   | id: "9df11f3a-565f-48bc-81f9-3b6c89876f47"
scm_1   |   }
scm_1   |   reason: PIPELINE_FAILED
scm_1   |   detailedReason: "46e4a0a9-e7d1-4f51-9434-d001b477a804 
has not seen follower/s e81b12d7-8234-4394-b621-dc71af4e3093 for 123559ms"
scm_1   | }
scm_1   |   }
scm_1   |

[jira] [Created] (HDDS-429) StorageContainerManager lock optimization

2018-09-11 Thread Nanda kumar (JIRA) 
Nanda kumar created HDDS-429:


 Summary: StorageContainerManager lock optimization
 Key: HDDS-429
 URL: https://issues.apache.org/jira/browse/HDDS-429
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
  Components: SCM
Reporter: Nanda kumar
Assignee: Nanda kumar
 Fix For: 0.2.1


Currently, {{StorageContainerManager}} uses {{ReentrantLock}} for 
synchronization. We can replace this with {{ReentrantReadWriteLock}} to get 
better performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-428) OzoneManager lock optimization

2018-09-11 Thread Nanda kumar (JIRA) 
Nanda kumar created HDDS-428:


 Summary: OzoneManager lock optimization
 Key: HDDS-428
 URL: https://issues.apache.org/jira/browse/HDDS-428
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
  Components: OM
Reporter: Nanda kumar
Assignee: Nanda kumar
 Fix For: 0.2.1


Currently, {{OzoneManager}} uses a single lock for everything which impacts the 
performance. We can introduce a separate lock for each resource like 
User/Volume/Bucket which will give us a performance boost.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13909) RBF: Add Cache pool related ClientProtocol apis

2018-09-11 Thread Dibyendu Karmakar (JIRA) 
Dibyendu Karmakar created HDFS-13909:


 Summary: RBF: Add Cache pool related ClientProtocol apis
 Key: HDFS-13909
 URL: https://issues.apache.org/jira/browse/HDFS-13909
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Dibyendu Karmakar
Assignee: Dibyendu Karmakar






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org