[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL

2016-10-05 Thread Rakesh R (JIRA)

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547752#comment-15547752
 ] 

Rakesh R edited comment on ZOOKEEPER-1045 at 10/5/16 7:10 AM:
--

Attached another patch in the [review ticket 
47354|https://reviews.apache.org/r/47354/] addressing [~hanm]'s comments. 
Majorly I've used the {{QuorumConnectionThread}} only if the 
'quorumSaslEnabled' to process the connection requests asynchronously. In the 
existing code path(non-auth) it will continue connecting to the peers 
sequentially. This separation is basically done to give the confidence to all 
by not touching the existing code flow, and keeps a different code path for 
this feature. I hope this change will help to push the feature safely into the 
{{branch3.4}}.

[~fpj], [~cnauroth], [~rgs], do you have some cycles to review the latest 
patch. Appreciate your feedback. Thanks!


was (Author: rakeshr):
Attached another patch in the [review ticket 
47354|https://reviews.apache.org/r/47354/] addressing [~hanm]'s comments. 
Majorly I've used the {{QuorumConnectionThread}} only if the 
'quorumSaslEnabled' to process the connection requests asynchronously. In the 
existing code path(non-auth) it will continue connecting to the peers 
sequentially. This separation is basically done to give the confidence to all 
by not touching the existing code flow, and keeps a different code path for 
this feature. I hope this change will help to push the feature safely in 
{{branch3.4}}.

[~fpj], [~cnauroth], [~rgs], do you have some cycles to review the latest 
patch. Appreciate your feedback. Thanks!

> Support Quorum Peer mutual authentication via SASL
> --
>
> Key: ZOOKEEPER-1045
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1045
> Project: ZooKeeper
>  Issue Type: New Feature
>  Components: server
>Reporter: Eugene Koontz
>Assignee: Rakesh R
>Priority: Critical
> Fix For: 3.4.10, 3.5.3
>
> Attachments: 0001-ZOOKEEPER-1045-br-3-4.patch, 
> 1045_failing_phunt.tar.gz, HOST_RESOLVER-ZK-1045.patch, 
> TEST-org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.txt, 
> ZK-1045-test-case-failure-logs.zip, ZOOKEEPER-1045-00.patch, 
> ZOOKEEPER-1045-Rolling Upgrade Design Proposal.pdf, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045TestValidationDesign.pdf, 
> org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.testRollingUpgrade.log
>
>
> ZOOKEEPER-938 addresses mutual authentication between clients and servers. 
> This bug, on the other hand, is for authentication among quorum peers. 
> Hopefully much of the work done on SASL integration with Zookeeper for 
> ZOOKEEPER-938 can be used as a foundation for this enhancement.
> Review board: https://reviews.apache.org/r/47354/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL

2016-10-05 Thread Rakesh R (JIRA)

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547752#comment-15547752
 ] 

Rakesh R edited comment on ZOOKEEPER-1045 at 10/5/16 7:08 AM:
--

Attached another patch in the [review ticket 
47354|https://reviews.apache.org/r/47354/] addressing [~hanm]'s comments. 
Majorly I've used the {{QuorumConnectionThread}} only if the 
'quorumSaslEnabled' to process the connection requests asynchronously. In the 
existing code path(non-auth) it will continue connecting to the peers 
sequentially. This separation is basically done to give the confidence to all 
by not touching the existing code flow, and keeps a different code path for 
this feature. I hope this change will help to push the feature safely in 
{{branch3.4}}.

[~fpj], [~cnauroth], [~rgs], do you have some cycles to review the latest 
patch. Appreciate your feedback. Thanks!


was (Author: rakeshr):
Attached another patch in the [review ticket 
47354|https://reviews.apache.org/r/47354/] addressing [~hanm]'s comments. 
Majorly I've used the {{QuorumConnectionThread}} only if the 
'quorumSaslEnabled' to process the connection requests asynchronously. In the 
existing code path(non-auth) it will continue connecting to the peers 
sequentially. This change is basically done to give the confidence to all by 
not touching the existing code flow so as to improve the confidence in pushing 
this feature in.

[~fpj], [~cnauroth], [~rgs], do you have some cycles to review the latest 
patch. Appreciate your feedback. Thanks!

> Support Quorum Peer mutual authentication via SASL
> --
>
> Key: ZOOKEEPER-1045
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1045
> Project: ZooKeeper
>  Issue Type: New Feature
>  Components: server
>Reporter: Eugene Koontz
>Assignee: Rakesh R
>Priority: Critical
> Fix For: 3.4.10, 3.5.3
>
> Attachments: 0001-ZOOKEEPER-1045-br-3-4.patch, 
> 1045_failing_phunt.tar.gz, HOST_RESOLVER-ZK-1045.patch, 
> TEST-org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.txt, 
> ZK-1045-test-case-failure-logs.zip, ZOOKEEPER-1045-00.patch, 
> ZOOKEEPER-1045-Rolling Upgrade Design Proposal.pdf, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045TestValidationDesign.pdf, 
> org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.testRollingUpgrade.log
>
>
> ZOOKEEPER-938 addresses mutual authentication between clients and servers. 
> This bug, on the other hand, is for authentication among quorum peers. 
> Hopefully much of the work done on SASL integration with Zookeeper for 
> ZOOKEEPER-938 can be used as a foundation for this enhancement.
> Review board: https://reviews.apache.org/r/47354/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL

2016-09-14 Thread Rakesh R (JIRA)

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491138#comment-15491138
 ] 

Rakesh R edited comment on ZOOKEEPER-1045 at 9/14/16 6:33 PM:
--

Thanks a lot [~phunt], [~shralex], [~hanm] for the discussions and suggestions. 
I've tried an initial attempt to do the authorization using the hostnames from 
{{zoo.cfg}}.  Kindly review and let me know the feedback. To keep the 
implementation simple, this patch expects fqdn should be configured in the 
zoo.cfg. Later this could be enhanced by supporting ipaddress/hostname and 
could use the approach in the patch {{HOST_RESOLVER-ZK-1045.patch}}

bq. 2. in 3.4, create a separate file for the auth list, and link it from 
zoo.cfg, similarly to the way I link the dynamic config file from zoo.cfg. 
As an initial attempt I've used zoo.cfg based approach for the authorized 
hosts. I agree we could enhance this using separate file for the auth list or 
znode approach etc. How about push this patch first and later we could discuss 
and implement solution through another jira.

bq. 3. In 3.5 support dynamic addition/removal of permissions (this may be very 
similar to dynamic reconfig): store the auth list in a znode, 
bq. create a new command for addition/removal/query from the auth list. 
Whenever the auth list is updated, also update the on-disk auth file.
I've plans to raise a separate jira for forward porting the solution to 
branch3.5 through another jira. I will make a note of these points and will 
consider while implementing the same.



was (Author: rakeshr):
Thanks a lot [~phunt], [~shralex], [~hanm] for the discussions and suggestions. 
I've tried an initial attempt to do the authorization using the hostnames from 
{{zoo.cfg}}.  Kindly review and let me know the feedback. To keep the 
implementation simple, this patch expects fqdn should be configured in the 
zoo.cfg. Later this could be enhanced by supporting ipaddress/hostname and 
could use the approach in the patch {{HOST_RESOLVER-ZK-1045.patch}}

bq. 2. in 3.4, create a separate file for the auth list, and link it from 
zoo.cfg, similarly to the way I link the dynamic config file from zoo.cfg. 
This will make updating the file easier in 3.5 (see below).
As an initial attempt I've used zoo.cfg based approach for the authorized 
hosts. I agree we could enhance this using separate file for the auth list or 
znode approach etc. How about push this patch first and later we could discuss 
and implement solution through another jira.

bq. 3. In 3.5 support dynamic addition/removal of permissions (this may be very 
similar to dynamic reconfig): store the auth list in a znode, 
create a new command for addition/removal/query from the auth list. Whenever 
the auth list is updated, also update the on-disk auth file.
I've plans to raise a separate jira for forward porting the solution through 
another jira. I will make a note of these points and will consider while 
implementing the same.


> Support Quorum Peer mutual authentication via SASL
> --
>
> Key: ZOOKEEPER-1045
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1045
> Project: ZooKeeper
>  Issue Type: New Feature
>  Components: server
>Reporter: Eugene Koontz
>Assignee: Rakesh R
>Priority: Critical
> Fix For: 3.4.10, 3.5.3
>
> Attachments: 0001-ZOOKEEPER-1045-br-3-4.patch, 
> 1045_failing_phunt.tar.gz, HOST_RESOLVER-ZK-1045.patch, 
> TEST-org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.txt, 
> ZK-1045-test-case-failure-logs.zip, ZOOKEEPER-1045-00.patch, 
> ZOOKEEPER-1045-Rolling Upgrade Design Proposal.pdf, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045TestValidationDesign.pdf
>
>
> ZOOKEEPER-938 addresses mutual authentication between clients and servers. 
> This bug, on the other hand, is for authentication among quorum peers. 
> Hopefully much of the work done on SASL integration with Zookeeper for 
> ZOOKEEPER-938 can be used as a foundation for this enhancement.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL

2016-09-14 Thread Rakesh R (JIRA)

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491138#comment-15491138
 ] 

Rakesh R edited comment on ZOOKEEPER-1045 at 9/14/16 6:32 PM:
--

Thanks a lot [~phunt], [~shralex], [~hanm] for the discussions and suggestions. 
I've tried an initial attempt to do the authorization using the hostnames from 
{{zoo.cfg}}.  Kindly review and let me know the feedback. To keep the 
implementation simple, this patch expects fqdn should be configured in the 
zoo.cfg. Later this could be enhanced by supporting ipaddress/hostname and 
could use the approach in the patch {{HOST_RESOLVER-ZK-1045.patch}}

bq. 2. in 3.4, create a separate file for the auth list, and link it from 
zoo.cfg, similarly to the way I link the dynamic config file from zoo.cfg. 
This will make updating the file easier in 3.5 (see below).
As an initial attempt I've used zoo.cfg based approach for the authorized 
hosts. I agree we could enhance this using separate file for the auth list or 
znode approach etc. How about push this patch first and later we could discuss 
and implement solution through another jira.

bq. 3. In 3.5 support dynamic addition/removal of permissions (this may be very 
similar to dynamic reconfig): store the auth list in a znode, 
create a new command for addition/removal/query from the auth list. Whenever 
the auth list is updated, also update the on-disk auth file.
I've plans to raise a separate jira for forward porting the solution through 
another jira. I will make a note of these points and will consider while 
implementing the same.



was (Author: rakeshr):
Thanks a lot [~phunt], [~shralex], [~hanm] for the discussions and suggestions. 
I've tried and initial attempt to do the authorization using the hostnames from 
{{zoo.cfg}}.  Kindly review and let me know the feedback. To keep the 
implementation simple, this patch expects fqdn should be configured in the 
zoo.cfg. Later this could be enhanced by supporting ipaddress/hostname and 
could use the approach in the patch {{HOST_RESOLVER-ZK-1045.patch}}

bq. 2. in 3.4, create a separate file for the auth list, and link it from 
zoo.cfg, similarly to the way I link the dynamic config file from zoo.cfg. 
This will make updating the file easier in 3.5 (see below).
As an initial attempt I've used zoo.cfg based approach for the authorized 
hosts. I agree we could enhance this using separate file for the auth list or 
znode approach etc. How about push this patch first and later we could discuss 
and implement solution through another jira.

bq. 3. In 3.5 support dynamic addition/removal of permissions (this may be very 
similar to dynamic reconfig): store the auth list in a znode, 
create a new command for addition/removal/query from the auth list. Whenever 
the auth list is updated, also update the on-disk auth file.
I've plans to raise a separate jira for forward porting the solution through 
another jira. I will make a note of these points and will consider while 
implementing the same.


> Support Quorum Peer mutual authentication via SASL
> --
>
> Key: ZOOKEEPER-1045
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1045
> Project: ZooKeeper
>  Issue Type: New Feature
>  Components: server
>Reporter: Eugene Koontz
>Assignee: Rakesh R
>Priority: Critical
> Fix For: 3.4.10, 3.5.3
>
> Attachments: 0001-ZOOKEEPER-1045-br-3-4.patch, 
> 1045_failing_phunt.tar.gz, HOST_RESOLVER-ZK-1045.patch, 
> TEST-org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.txt, 
> ZK-1045-test-case-failure-logs.zip, ZOOKEEPER-1045-00.patch, 
> ZOOKEEPER-1045-Rolling Upgrade Design Proposal.pdf, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045TestValidationDesign.pdf
>
>
> ZOOKEEPER-938 addresses mutual authentication between clients and servers. 
> This bug, on the other hand, is for authentication among quorum peers. 
> Hopefully much of the work done on SASL integration with Zookeeper for 
> ZOOKEEPER-938 can be used as a foundation for this enhancement.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL

2016-09-08 Thread Rakesh R (JIRA)

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15473057#comment-15473057
 ] 

Rakesh R edited comment on ZOOKEEPER-1045 at 9/8/16 7:22 AM:
-

bq. But authorization should probably happen before, maybe when a reconfig is 
processed or maybe even before that - when a new server boots and tries to 
connect to the leader. Does the leader check new server connections ? If that 
server wasn't initially declared, would it be able to connect ?
Perhaps we should identify the exact place where it needs to do the 
authorization. I can say, this jira is proposing authentication and 
authorization at the beginning of FLE process. That means, when a learner tries 
to connect to remote quorum server, first it will do the authn and authz, on 
success it will proceed to FLE process, otw reject the connecting server. 
Again, after FLE when a Follower/Observer tries to {{connectToLeader}} it does 
authn and authz steps.

As per my above comment, when a new server tries to connect to leader, leader 
will cross check with its quorum server principal list(built from zoo.cfg). So 
here the leader should have the information about the newly connecting server. 
Will the reconfig commit happens before this?


was (Author: rakeshr):
bq. But authorization should probably happen before, maybe when a reconfig is 
processed or maybe even before that - when a new server boots and tries to 
connect to the leader. Does the leader check new server connections ? If that 
server wasn't initially declared, would it be able to connect ?
Perhaps we should identify the exact place where it needs to do the 
authorization. I can say, this jira is proposing authentication and 
authorization at the beginning of FLE process. That means, when a learner tries 
to connect to remote quorum server, first it will do the authn and authz, on 
success it will proceed to FLE process, otw reject the connecting server. 

As per my above comment, when a new server tries to connect to leader, leader 
will cross check with its quorum server principal list(built from zoo.cfg). So 
here the leader should have the information about the newly connecting server. 
Will the reconfig commit happens before this?

> Support Quorum Peer mutual authentication via SASL
> --
>
> Key: ZOOKEEPER-1045
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1045
> Project: ZooKeeper
>  Issue Type: New Feature
>  Components: server
>Reporter: Eugene Koontz
>Assignee: Rakesh R
>Priority: Critical
> Fix For: 3.4.10, 3.5.3
>
> Attachments: 0001-ZOOKEEPER-1045-br-3-4.patch, 
> 1045_failing_phunt.tar.gz, HOST_RESOLVER-ZK-1045.patch, 
> TEST-org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.txt, 
> ZK-1045-test-case-failure-logs.zip, ZOOKEEPER-1045-00.patch, 
> ZOOKEEPER-1045-Rolling Upgrade Design Proposal.pdf, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045TestValidationDesign.pdf
>
>
> ZOOKEEPER-938 addresses mutual authentication between clients and servers. 
> This bug, on the other hand, is for authentication among quorum peers. 
> Hopefully much of the work done on SASL integration with Zookeeper for 
> ZOOKEEPER-938 can be used as a foundation for this enhancement.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL

2016-09-08 Thread Rakesh R (JIRA)

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15472985#comment-15472985
 ] 

Rakesh R edited comment on ZOOKEEPER-1045 at 9/8/16 6:56 AM:
-

Thanks everyone for putting all the use cases and active discussions. Let me 
try to summarize all the problems and proposed solutions.

*Point-1)* ??Prevent another host with the same krb user and realm to join my 
zookeeper cluster, for example {{zk/@EXAMPLE.COM}}??

Like [~phunt], [~hanm] explained we could make use of zoo.cfg as the source of 
authorization information and replace the _HOST part in {{user/_HOST@REALM}}. 
Since admins can configure ZK server details as host name or ipaddress or fqdn 
in zoo.cfg, server should have a mechanism to resolve this to fully qualified 
domain name for this IP address. Sometime back I've attached 
{{HOST_RESOLVER-ZK-1045.patch}} idea (thanks to hadoop project [hadoop 
SecurityUtil 
ref.|https://github.com/apache/hadoop/blob/branch-2.8/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java#L570]),
 which is an independent patch to prepare QuorumServer kerberos principal by 
resolving the host address of ZK server to 
InetAddress.getLocalHost().getCanonicalHostName() and expects principal like 
zk/ho...@example.com. This principal will be used by the quorum peer learner to 
talk to another quorum peer server during FLE. Here the implicit requirement 
is, admin has to ensure that the configured kerberos principal name should be 
resolved to fully qualified domain name for this IP address.

* For authorization every server will compare the full principal name that 
composed of {{user/host@realm}}. For doing this every server will cross check 
with the list of known quorum server principals built from zoo.cfg file.
+Ensemble-1 :-+
quorum server principal list => {{zk/ho...@example.com, zk/ho...@example.com, 
zk/ho...@example.com}}
+Ensemble-2 :-+
quorum server principal list => {{zk/ho...@example.com, zk/ho...@example.com, 
zk/ho...@example.com}}

* For authentication, quorum learner server will get the remote quorum server 
principal name and then do authentication. For example, host1 will get host2 
principal {{zk/ho...@example.com}} and do authenticate.

Does this make sense?

*Point-2)* ??Feature of KDC that it will treat repeated attempts to log in with 
the same Kerberos principal within a short period of time as replay attacks and 
will reject such login requests. Since we are supporting shared Kerberos 
credential, we might hit this issue.??

Good catch, Michael. [~cnauroth], I hope you are pointing me to the hadoop code 
[Client.java#L699|https://github.com/apache/hadoop/blob/branch-2.8/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L699],
 isn't it? I'll try to understand this part and get back to you.

*Point-3)* ??How would we authorize against something that is not 
pre-configured? Basically the dynamic reconfiguration of servers (addition and 
removal). Also, supports upgrade from 3.4 to 3.5 and above.??

[~shralex], IIUC, dynamic reconfig feature is continue using the zoo.cfg 
configuration file to keep the quorum info and while processing the 
reconfiguration request, it will always update the zoo.cfg file and ensure this 
file is uptodate. In that case each server will get the details of newly added 
server and during this time we should accommodate the logic of updating the 
{{quorum server principal list}} with the newly added server or removed server 
details, if any. But there is a case,  server tries to join 
{{zk/@EXAMPLE.COM}}, I think this has to be restricted at the reconfig 
command execution side rather than FLE, probably ZOOKEEPER-2014 jira will help 
to resolve this problem, am I missing anything?


was (Author: rakeshr):
Thanks everyone for putting all the use cases and active discussions. Let me 
try to summarize all the problems and proposed solutions.

*Point-1)* ??Prevent another user from getting Kerberos credentials for 
{{zk/@EXAMPLE.COM}}, and don't want them to be able to join my 
cluster.??

Like [~phunt], [~hanm] explained we could make use of zoo.cfg as the source of 
authorization information and replace the _HOST part in {{user/_HOST@REALM}}. 
Since admins can configure ZK server details as host name or ipaddress or fqdn 
in zoo.cfg, server should have a mechanism to resolve this to fully qualified 
domain name for this IP address. Sometime back I've attached 
{{HOST_RESOLVER-ZK-1045.patch}} idea (thanks to hadoop project [hadoop 
SecurityUtil 
ref.|https://github.com/apache/hadoop/blob/branch-2.8/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java#L570]),
 which is an independent patch to prepare QuorumServer kerberos principal by 
resolving the host address of ZK server to 

[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL

2016-08-12 Thread Eugene Koontz (JIRA)

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15419069#comment-15419069
 ] 

Eugene Koontz edited comment on ZOOKEEPER-1045 at 8/12/16 4:12 PM:
---

Thank you [~rakeshr] for the correction - I'm now using the latest, correct 
patch. 


was (Author: ekoontz):
Thank you ~rakeshr for the correction - I'm now using the latest, correct 
patch. 

> Support Quorum Peer mutual authentication via SASL
> --
>
> Key: ZOOKEEPER-1045
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1045
> Project: ZooKeeper
>  Issue Type: New Feature
>  Components: server
>Reporter: Eugene Koontz
>Assignee: Rakesh R
>Priority: Critical
> Fix For: 3.5.3, 3.4.10
>
> Attachments: 0001-ZOOKEEPER-1045-br-3-4.patch, 
> 1045_failing_phunt.tar.gz, HOST_RESOLVER-ZK-1045.patch, 
> TEST-org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.txt, 
> ZK-1045-test-case-failure-logs.zip, ZOOKEEPER-1045-00.patch, 
> ZOOKEEPER-1045-Rolling Upgrade Design Proposal.pdf, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045TestValidationDesign.pdf
>
>
> ZOOKEEPER-938 addresses mutual authentication between clients and servers. 
> This bug, on the other hand, is for authentication among quorum peers. 
> Hopefully much of the work done on SASL integration with Zookeeper for 
> ZOOKEEPER-938 can be used as a foundation for this enhancement.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL

2016-07-21 Thread Michael Han (JIRA)

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388151#comment-15388151
 ] 

Michael Han edited comment on ZOOKEEPER-1045 at 7/21/16 6:19 PM:
-

[~rakeshr]
There is one flaky test (testRollingUpgrade) in latest (July 12) version of 
patch. Attach the test log. 
Would be good to fix this test to reduce the flaky tests. The rest of tests 
passed my pressure tests.


was (Author: hanm):
[~rakeshr]
There is one flaky tests in latest (July 12) version of patch. Attach the test 
log. 
Would be good to fix this test to reduce the flaky tests. The rest of tests 
passed my pressure tests.

> Support Quorum Peer mutual authentication via SASL
> --
>
> Key: ZOOKEEPER-1045
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1045
> Project: ZooKeeper
>  Issue Type: New Feature
>  Components: server
>Reporter: Eugene Koontz
>Assignee: Rakesh R
>Priority: Critical
> Fix For: 3.4.9, 3.5.3
>
> Attachments: 0001-ZOOKEEPER-1045-br-3-4.patch, 
> 1045_failing_phunt.tar.gz, 
> TEST-org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.txt, 
> ZK-1045-test-case-failure-logs.zip, ZOOKEEPER-1045-00.patch, 
> ZOOKEEPER-1045-Rolling Upgrade Design Proposal.pdf, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045TestValidationDesign.pdf
>
>
> ZOOKEEPER-938 addresses mutual authentication between clients and servers. 
> This bug, on the other hand, is for authentication among quorum peers. 
> Hopefully much of the work done on SASL integration with Zookeeper for 
> ZOOKEEPER-938 can be used as a foundation for this enhancement.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL

2016-07-18 Thread Rakesh R (JIRA)

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383577#comment-15383577
 ] 

Rakesh R edited comment on ZOOKEEPER-1045 at 7/19/16 4:55 AM:
--

bq.  I believe we should do similar, if not for security then for consistency.
Agreed. Previously in this jira, [~dbenediktson] has brought up a case where 
all the ZK hosts run with the same Kerberos principal. So now we have two 
cases, need to support both {{_HOST}} based principal name and {{shared}} 
principal name. I'm assuming there won't be any need of supporting mixture of 
both of these like, few quorum servers with "zkquorum/_HOST@" pattern in their 
principal name and few others are having constant name "zkquorum@". 

Following is an idea to support both the cases. Welcome comments.

*Case-1)* {{_HOST}} based principal -> For example, zkquorum/_h...@example.com
zoo.cfg has the following configuration which has the 'host' information. This 
host address {{addr.getCanonicalHostName()}} will be used to replace the 
{{_HOST}} pattern. We will make use of the existing data structure 
{{QuorumCnxManager#view}} map to get the respective server's host name. While 
connecting to the respective server, first the quorum learner will check 
{{quorum.auth.kerberos.servicePrincipal}} configuration has {{_HOST}} name 
pattern then convert the Kerberos principal name to a valid name by replacing 
the {{_HOST}} part. Myid will be used as the key to get the respective quorum 
server address from the {{#view}}.
{code}
server.0=host1:11222:11223:participant
server.1=host2:11225:11226:participant
server.2=host3:11228:11229:participant 
{code}

*Case-2)* Shared principal -> For example, zkquo...@example.com
While connecting to peer servers, first the quorum learner will check 
{{quorum.auth.kerberos.servicePrincipal}} configuration has {{_HOST}} name 
pattern, if not then will directly use the value as Kerberos principal name and 
continue with the authentication process.

*Case-3* Mixture of {{_HOST}} based and constant principal
This won't be supported. We will supports only two valid principal names, 
either all servers should have "_HOST" based principal name or all servers 
shares same principal name.

bq. I this this is a great idea, however is it possible to move to another 
jira? 
OK, I will push this separately

bq. It looks like we don't have any tests to verify the authz aspect of the 
change? 
Yes, will include test case for this.


was (Author: rakeshr):
bq.  I believe we should do similar, if not for security then for consistency.
Agreed. Previously in this jira, [~dbenediktson] has brought up a case where 
all the ZK hosts run with the same Kerberos principal. So now we have two 
cases, need to support both {{_HOST}} based principal name and {{shared}} 
principal name. I'm assuming there won't be any need of supporting mixture of 
both of these like, few quorum servers with "zkquorum/_HOST@" pattern in their 
principal name and few others are having constant name "zkquorum@". 

Following is an idea to support both the cases. Welcome comments.

*Case-1)* {{_HOST}} based principal -> For example, zkquorum/_h...@example.com
zoo.cfg has the following configuration which has the 'host' information. This 
host address {{addr.getCanonicalHostName()}} will be used to replace the 
{{_HOST}} pattern. We will make use of the existing data structure 
{{QuorumCnxManager#view}} map to get the respective server's host name. While 
connecting to the respective server, first the quorum learner will check 
{{quorum.auth.kerberos.servicePrincipal}} configuration has {{_HOST}} name 
pattern then convert the Kerberos principal name to a valid name by replacing 
the {{_HOST}} part. Myid will be used as the key to get the respective quorum 
server address from the {{#view}}.
{code}
server.0=host1:11222:11223:participant
server.1=host2:11225:11226:participant
server.2=host3:11228:11229:participant 
{code}

*Case-2)* Shared principal -> For example, zkquo...@example.com
While connecting to peer servers, first the quorum learner will check 
{{quorum.auth.kerberos.servicePrincipal}} configuration has {{_HOST}} name 
pattern, if not then will directly use the value as Kerberos principal name and 
continue with the authentication process.

*Case-3* Mixture of {{_HOST}} based and constant principal
ZooKeeper doesn't supports this case. ZooKeeper supports only two valid 
principal names, either all servers should have "_HOST" based principal name or 
all servers shares same principal name.

bq. I this this is a great idea, however is it possible to move to another 
jira? 
OK, I will push this separately

bq. It looks like we don't have any tests to verify the authz aspect of the 
change? 
Yes, will include test case for this.

> Support Quorum Peer mutual authentication via SASL
> --
>
>   

[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL

2016-07-03 Thread Rakesh R (JIRA)

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360796#comment-15360796
 ] 

Rakesh R edited comment on ZOOKEEPER-1045 at 7/4/16 3:36 AM:
-

[~hanm] Yes, exactly. Presently we will be supporting only single(shared) Kerb 
principal across all the servers. We could capture this point clearly in our 
test report documentation and later the same can be used to update {{cwiki 
page}} as well. In future, if anyone has a use case of different Kerb principal 
then we can discuss/extend the implementation to support the same later. IMHO, 
its not required to handle those complex case now. Does that make sense to you?


was (Author: rakeshr):
[~hanm] Yes, exactly. Presently we will be supporting only single(shared) Kerb 
principal across all the servers now. We could capture this point clearly in 
our test report documentation and later the same can be used to update {{cwiki 
page}} as well. In future, if anyone has a use case of different Kerb principal 
then we can discuss/extend the implementation to support the same later. IMHO, 
its not required to handle those complex case now. Does that make sense to you?

> Support Quorum Peer mutual authentication via SASL
> --
>
> Key: ZOOKEEPER-1045
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1045
> Project: ZooKeeper
>  Issue Type: New Feature
>  Components: server
>Reporter: Eugene Koontz
>Assignee: Rakesh R
>Priority: Critical
> Fix For: 3.4.9, 3.5.3
>
> Attachments: 0001-ZOOKEEPER-1045-br-3-4.patch, 
> 1045_failing_phunt.tar.gz, ZK-1045-test-case-failure-logs.zip, 
> ZOOKEEPER-1045-00.patch, ZOOKEEPER-1045-Rolling Upgrade Design Proposal.pdf, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045-br-3-4.patch
>
>
> ZOOKEEPER-938 addresses mutual authentication between clients and servers. 
> This bug, on the other hand, is for authentication among quorum peers. 
> Hopefully much of the work done on SASL integration with Zookeeper for 
> ZOOKEEPER-938 can be used as a foundation for this enhancement.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL

2016-07-03 Thread Michael Han (JIRA)

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360675#comment-15360675
 ] 

Michael Han edited comment on ZOOKEEPER-1045 at 7/3/16 8:46 PM:


bq. For the server-server auth, Kerb principal should be same for all the 
servers to allow communicating each other. 

IIUC, this means that we only support a single (shared) Kerberos principal / 
credential across all servers for server to server communication, and if so, 
the failure of my validation against the case where servers use different 
Kerberos principal is a by design, because I was using different Kerberos 
principals on each server for server to server auth validation. [~rakeshr]


was (Author: hanm):
b.q. For the server-server auth, Kerb principal should be same for all the 
servers to allow communicating each other. 

IIUC, this means that we only support a single (shared) Kerberos principal / 
credential across all servers for server to server communication, and if so, 
the failure of my validation against the case where servers use different 
Kerberos principal is a by design, because I was using different Kerberos 
principals on each server for server to server auth validation. [~rakeshr]

> Support Quorum Peer mutual authentication via SASL
> --
>
> Key: ZOOKEEPER-1045
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1045
> Project: ZooKeeper
>  Issue Type: New Feature
>  Components: server
>Reporter: Eugene Koontz
>Assignee: Rakesh R
>Priority: Critical
> Fix For: 3.4.9, 3.5.3
>
> Attachments: 0001-ZOOKEEPER-1045-br-3-4.patch, 
> 1045_failing_phunt.tar.gz, ZK-1045-test-case-failure-logs.zip, 
> ZOOKEEPER-1045-00.patch, ZOOKEEPER-1045-Rolling Upgrade Design Proposal.pdf, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, 
> ZOOKEEPER-1045-br-3-4.patch
>
>
> ZOOKEEPER-938 addresses mutual authentication between clients and servers. 
> This bug, on the other hand, is for authentication among quorum peers. 
> Hopefully much of the work done on SASL integration with Zookeeper for 
> ZOOKEEPER-938 can be used as a foundation for this enhancement.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL

2016-07-01 Thread Rakesh R (JIRA)

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358379#comment-15358379
 ] 

Rakesh R edited comment on ZOOKEEPER-1045 at 7/1/16 6:03 AM:
-

Thanks [~dbenediktson] for the interest in this work and sharing the use case.
bq. please make sure you support the case where all the ZK hosts run as the 
same Kerberos principal
Yes, you can configure the same Krb credentials for client-server and 
server-server communications. As part of this jira, there is no changes to the 
existing client-server communication path, this will work as it is. I will try 
to add few details about the server-server auth configs.

For the server-server auth, Kerb principal should be same for all the servers 
to allow communicating each other. Since each server will talk to all the other 
servers to form quorum it is required to know each others Krb principal. This 
jira introduces {{QuorumServer}} section where admin can configure the Krb 
principal of self (also it represents the principal of other quorum peer 
server), learner will use this and can contact them. 

In the below example config, should use same 
{{principal="zkquorum/localh...@example.com";}} in all the servers.
{code}
QuorumServer {
   com.sun.security.auth.module.Krb5LoginModule required
   useKeyTab=true
   keyTab="/path/to/keytab"
   storeKey=true
   useTicketCache=false
   debug=false
   principal="zkquorum/localh...@example.com";
};
{code}

Few days back there was a discussion about configuring [different Kerb 
credentials|https://issues.apache.org/jira/browse/ZOOKEEPER-1045?focusedCommentId=15339198=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15339198]
 for client-server(Server) and server-server(QuorumServer/QuorumLearner) 
communications. Please refer this to understand more.

We will be supporting Krb credentials in the following ways. I'd appreciate if 
you can test the same in your env and see its working.
1) all ZK hosts sharing same Kerb principal for both client-server and 
server-server
2) client-server(Server) uses {{principal_1}} and 
server-server(QuorumServer/QuorumLearner) uses {{principal_2}}.

bq.I've validated that server to server Kerberos SASL auth working, when 
servers share same credentials (same service principal name + same full 
qualified domain+ same keytabs) deployed on all nodes.
Thanks [~hanm] for the confirmation.

bq. For the cases where each server has a distinct Kerberos credential, it's 
not working yet. 
[~hanm], please let me the QuorumServer principal values. Could you share the 
{{jaas.config}} of all the servers and the failure logs for better debugging.


was (Author: rakeshr):
Thanks [~dbenediktson] for the interest in this work and sharing the use case.
bq. please make sure you support the case where all the ZK hosts run as the 
same Kerberos principal
Yes, you can configure the same Krb credentials for client-server and 
server-server communications. As part of this jira, there is no changes to the 
existing client-server communication path, this will work as it is. I will try 
to add few details about the server-server auth configs.

For the server-server auth, Kerb principal should be same for all the servers 
to allow communicating each other. Since each server will talk to all the other 
servers to form quorum it is required to know each others Krb principal. This 
jira introduces {{QuorumServer}} section where admin can configure the 
principal of self(also the principal of other quorum peer server), learner will 
use this and can contact them. 

In the below example config, should use same 
{{principal="zkquorum/localh...@example.com";}} in all the servers.
{code}
QuorumServer {
   com.sun.security.auth.module.Krb5LoginModule required
   useKeyTab=true
   keyTab="/path/to/keytab"
   storeKey=true
   useTicketCache=false
   debug=false
   principal="zkquorum/localh...@example.com";
};
{code}

Few days back there was a discussion about configuring [different Kerb 
credentials|https://issues.apache.org/jira/browse/ZOOKEEPER-1045?focusedCommentId=15339198=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15339198]
 for client-server(Server) and server-server(QuorumServer/QuorumLearner) 
communications. Please refer this to understand more.

We will be supporting Krb credentials in the following ways. I'd appreciate if 
you can test the same in your env and see its working.
1) all ZK hosts sharing same Kerb principal for both client-server and 
server-server
2) client-server(Server) uses {{principal_1}} and 
server-server(QuorumServer/QuorumLearner) uses {{principal_2}}.

bq.I've validated that server to server Kerberos SASL auth working, when 
servers share same credentials (same service principal name + same full 
qualified domain+ same keytabs) deployed on all nodes.

[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL

2016-07-01 Thread Rakesh R (JIRA)

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358379#comment-15358379
 ] 

Rakesh R edited comment on ZOOKEEPER-1045 at 7/1/16 6:01 AM:
-

Thanks [~dbenediktson] for the interest in this work and sharing the use case.
bq. please make sure you support the case where all the ZK hosts run as the 
same Kerberos principal
Yes, you can configure the same Krb credentials for client-server and 
server-server communications. As part of this jira, there is no changes to the 
existing client-server communication path, this will work as it is. I will try 
to add few details about the server-server auth configs.

For the server-server auth, Kerb principal should be same for all the servers 
to allow communicating each other. Since each server will talk to all the other 
servers to form quorum it is required to know each others Krb principal. This 
jira introduces {{QuorumServer}} section where admin can configure the 
principal of self(also the principal of other quorum peer server), learner will 
use this and can contact them. 

In the below example config, should use same 
{{principal="zkquorum/localh...@example.com";}} in all the servers.
{code}
QuorumServer {
   com.sun.security.auth.module.Krb5LoginModule required
   useKeyTab=true
   keyTab="/path/to/keytab"
   storeKey=true
   useTicketCache=false
   debug=false
   principal="zkquorum/localh...@example.com";
};
{code}

Few days back there was a discussion about configuring [different Kerb 
credentials|https://issues.apache.org/jira/browse/ZOOKEEPER-1045?focusedCommentId=15339198=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15339198]
 for client-server(Server) and server-server(QuorumServer/QuorumLearner) 
communications. Please refer this to understand more.

We will be supporting Krb credentials in the following ways. I'd appreciate if 
you can test the same in your env and see its working.
1) all ZK hosts sharing same Kerb principal for both client-server and 
server-server
2) client-server(Server) uses {{principal_1}} and 
server-server(QuorumServer/QuorumLearner) uses {{principal_2}}.

bq.I've validated that server to server Kerberos SASL auth working, when 
servers share same credentials (same service principal name + same full 
qualified domain+ same keytabs) deployed on all nodes.
Thanks [~hanm] for the confirmation.

bq. For the cases where each server has a distinct Kerberos credential, it's 
not working yet. 
[~hanm], please let me the QuorumServer principal values. Could you share the 
{{jaas.config}} of all the servers and the failure logs for better debugging.


was (Author: rakeshr):
Thanks [~dbenediktson] for the interest in this work and sharing the use case.
bq. please make sure you support the case where all the ZK hosts run as the 
same Kerberos principal
Yes, you can configure the same Krb credentials for client-server and 
server-server communications. As part of this jira, there is no changes to the 
existing client-server communication path, this will work as it is. I will try 
to add few details about the server-server auth configs.

For the server-server auth, Kerb principal should be same for all the servers 
to allow communicating each other. Since each server will talk to all the other 
servers to form quorum it is required to know each others Krb principal. This 
jira introduces {{QuorumServer}} section where admin can configure the 
principal of other quorum peer server so that the learner can use this and can 
contact them. 

In the below example config, should use same 
{{principal="zkquorum/localh...@example.com";}} in all the servers.
{code}
QuorumServer {
   com.sun.security.auth.module.Krb5LoginModule required
   useKeyTab=true
   keyTab="/path/to/keytab"
   storeKey=true
   useTicketCache=false
   debug=false
   principal="zkquorum/localh...@example.com";
};
{code}

Few days back there was a discussion about configuring [different Kerb 
credentials|https://issues.apache.org/jira/browse/ZOOKEEPER-1045?focusedCommentId=15339198=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15339198]
 for client-server(Server) and server-server(QuorumServer/QuorumLearner) 
communications. Please refer this to understand more.

We will be supporting Krb credentials in the following ways. I'd appreciate if 
you can test the same in your env and see its working.
1) all ZK hosts sharing same Kerb principal for both client-server and 
server-server
2) client-server(Server) uses {{principal_1}} and 
server-server(QuorumServer/QuorumLearner) uses {{principal_2}}.

bq.I've validated that server to server Kerberos SASL auth working, when 
servers share same credentials (same service principal name + same full 
qualified domain+ same keytabs) deployed on all nodes.
Thanks [~hanm] for the confirmation.

[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL

2016-06-30 Thread Rakesh R (JIRA)

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358379#comment-15358379
 ] 

Rakesh R edited comment on ZOOKEEPER-1045 at 7/1/16 4:52 AM:
-

Thanks [~dbenediktson] for the interest in this work and sharing the use case.
bq. please make sure you support the case where all the ZK hosts run as the 
same Kerberos principal
Yes, you can configure the same Krb credentials for client-server and 
server-server communications. As part of this jira, there is no changes to the 
existing client-server communication path, this will work as it is. I will try 
to add few details about the server-server auth configs.

For the server-server auth, Kerb principal should be same for all the servers 
to allow communicating each other. Since each server will talk to all the other 
servers to form quorum it is required to know each others Krb principal. This 
jira introduces {{QuorumServer}} section where admin can configure the 
principal of other quorum peer server so that the learner can use this and can 
contact them. 

In the below example config, should use same 
{{principal="zkquorum/localh...@example.com";}} in all the servers.
{code}
QuorumServer {
   com.sun.security.auth.module.Krb5LoginModule required
   useKeyTab=true
   keyTab="/path/to/keytab"
   storeKey=true
   useTicketCache=false
   debug=false
   principal="zkquorum/localh...@example.com";
};
{code}

Few days back there was a discussion about configuring [different Kerb 
credentials|https://issues.apache.org/jira/browse/ZOOKEEPER-1045?focusedCommentId=15339198=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15339198]
 for client-server(Server) and server-server(QuorumServer/QuorumLearner) 
communications. Please refer this to understand more.

We will be supporting Krb credentials in the following ways. I'd appreciate if 
you can test the same in your env and see its working.
1) all ZK hosts sharing same Kerb principal for both client-server and 
server-server
2) client-server(Server) uses {{principal_1}} and 
server-server(QuorumServer/QuorumLearner) uses {{principal_2}}.

bq.I've validated that server to server Kerberos SASL auth working, when 
servers share same credentials (same service principal name + same full 
qualified domain+ same keytabs) deployed on all nodes.
Thanks [~hanm] for the confirmation.

bq. For the cases where each server has a distinct Kerberos credential, it's 
not working yet. 
[~hanm], please let me the QuorumServer principal values. Could you share the 
{{jaas.config}} of all the servers and the failure logs for better debugging.


was (Author: rakeshr):
Thanks [~dbenediktson] for the interest on this work and sharing the use case.
bq. please make sure you support the case where all the ZK hosts run as the 
same Kerberos principal
Yes, you can configure the same Krb credentials for client-server and 
server-server communications. As part of this jira, there is no changes to the 
existing client-server communication path, this will work as it is. I will try 
to add few details about the server-server auth configs.

For the server-server auth, Kerb principal should be same for all the servers 
to allow communicating each other. Since each server will talk to all the other 
servers to form quorum it is required to know each others Krb principal. This 
jira introduces {{QuorumServer}} section where admin can configure the 
principal of other quorum peer server so that the learner can use this and can 
contact them. 

In the below example config, should use same 
{{principal="zkquorum/localh...@example.com";}} in all the servers.
{code}
QuorumServer {
   com.sun.security.auth.module.Krb5LoginModule required
   useKeyTab=true
   keyTab="/path/to/keytab"
   storeKey=true
   useTicketCache=false
   debug=false
   principal="zkquorum/localh...@example.com";
};
{code}

Few days back there was a discussion about configuring [different Kerb 
credentials|https://issues.apache.org/jira/browse/ZOOKEEPER-1045?focusedCommentId=15339198=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15339198]
 for client-server(Server) and server-server(QuorumServer/QuorumLearner) 
communications. Please refer this to understand more.

We will be supporting Krb credentials in the following ways. I'd appreciate if 
you can test the same in your env and see its working.
1) all ZK hosts sharing same Kerb principal for both client-server and 
server-server
2) client-server(Server) uses {{principal_1}} and 
server-server(QuorumServer/QuorumLearner) uses {{principal_2}}.

bq.I've validated that server to server Kerberos SASL auth working, when 
servers share same credentials (same service principal name + same full 
qualified domain+ same keytabs) deployed on all nodes.
Thanks [~hanm] for the confirmation.

bq. For the