[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547752#comment-15547752 ] Rakesh R edited comment on ZOOKEEPER-1045 at 10/5/16 7:10 AM: -- Attached another patch in the [review ticket 47354|https://reviews.apache.org/r/47354/] addressing [~hanm]'s comments. Majorly I've used the {{QuorumConnectionThread}} only if the 'quorumSaslEnabled' to process the connection requests asynchronously. In the existing code path(non-auth) it will continue connecting to the peers sequentially. This separation is basically done to give the confidence to all by not touching the existing code flow, and keeps a different code path for this feature. I hope this change will help to push the feature safely into the {{branch3.4}}. [~fpj], [~cnauroth], [~rgs], do you have some cycles to review the latest patch. Appreciate your feedback. Thanks! was (Author: rakeshr): Attached another patch in the [review ticket 47354|https://reviews.apache.org/r/47354/] addressing [~hanm]'s comments. Majorly I've used the {{QuorumConnectionThread}} only if the 'quorumSaslEnabled' to process the connection requests asynchronously. In the existing code path(non-auth) it will continue connecting to the peers sequentially. This separation is basically done to give the confidence to all by not touching the existing code flow, and keeps a different code path for this feature. I hope this change will help to push the feature safely in {{branch3.4}}. [~fpj], [~cnauroth], [~rgs], do you have some cycles to review the latest patch. Appreciate your feedback. Thanks! > Support Quorum Peer mutual authentication via SASL > -- > > Key: ZOOKEEPER-1045 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1045 > Project: ZooKeeper > Issue Type: New Feature > Components: server >Reporter: Eugene Koontz >Assignee: Rakesh R >Priority: Critical > Fix For: 3.4.10, 3.5.3 > > Attachments: 0001-ZOOKEEPER-1045-br-3-4.patch, > 1045_failing_phunt.tar.gz, HOST_RESOLVER-ZK-1045.patch, > TEST-org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.txt, > ZK-1045-test-case-failure-logs.zip, ZOOKEEPER-1045-00.patch, > ZOOKEEPER-1045-Rolling Upgrade Design Proposal.pdf, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045TestValidationDesign.pdf, > org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.testRollingUpgrade.log > > > ZOOKEEPER-938 addresses mutual authentication between clients and servers. > This bug, on the other hand, is for authentication among quorum peers. > Hopefully much of the work done on SASL integration with Zookeeper for > ZOOKEEPER-938 can be used as a foundation for this enhancement. > Review board: https://reviews.apache.org/r/47354/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547752#comment-15547752 ] Rakesh R edited comment on ZOOKEEPER-1045 at 10/5/16 7:08 AM: -- Attached another patch in the [review ticket 47354|https://reviews.apache.org/r/47354/] addressing [~hanm]'s comments. Majorly I've used the {{QuorumConnectionThread}} only if the 'quorumSaslEnabled' to process the connection requests asynchronously. In the existing code path(non-auth) it will continue connecting to the peers sequentially. This separation is basically done to give the confidence to all by not touching the existing code flow, and keeps a different code path for this feature. I hope this change will help to push the feature safely in {{branch3.4}}. [~fpj], [~cnauroth], [~rgs], do you have some cycles to review the latest patch. Appreciate your feedback. Thanks! was (Author: rakeshr): Attached another patch in the [review ticket 47354|https://reviews.apache.org/r/47354/] addressing [~hanm]'s comments. Majorly I've used the {{QuorumConnectionThread}} only if the 'quorumSaslEnabled' to process the connection requests asynchronously. In the existing code path(non-auth) it will continue connecting to the peers sequentially. This change is basically done to give the confidence to all by not touching the existing code flow so as to improve the confidence in pushing this feature in. [~fpj], [~cnauroth], [~rgs], do you have some cycles to review the latest patch. Appreciate your feedback. Thanks! > Support Quorum Peer mutual authentication via SASL > -- > > Key: ZOOKEEPER-1045 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1045 > Project: ZooKeeper > Issue Type: New Feature > Components: server >Reporter: Eugene Koontz >Assignee: Rakesh R >Priority: Critical > Fix For: 3.4.10, 3.5.3 > > Attachments: 0001-ZOOKEEPER-1045-br-3-4.patch, > 1045_failing_phunt.tar.gz, HOST_RESOLVER-ZK-1045.patch, > TEST-org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.txt, > ZK-1045-test-case-failure-logs.zip, ZOOKEEPER-1045-00.patch, > ZOOKEEPER-1045-Rolling Upgrade Design Proposal.pdf, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045TestValidationDesign.pdf, > org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.testRollingUpgrade.log > > > ZOOKEEPER-938 addresses mutual authentication between clients and servers. > This bug, on the other hand, is for authentication among quorum peers. > Hopefully much of the work done on SASL integration with Zookeeper for > ZOOKEEPER-938 can be used as a foundation for this enhancement. > Review board: https://reviews.apache.org/r/47354/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491138#comment-15491138 ] Rakesh R edited comment on ZOOKEEPER-1045 at 9/14/16 6:33 PM: -- Thanks a lot [~phunt], [~shralex], [~hanm] for the discussions and suggestions. I've tried an initial attempt to do the authorization using the hostnames from {{zoo.cfg}}. Kindly review and let me know the feedback. To keep the implementation simple, this patch expects fqdn should be configured in the zoo.cfg. Later this could be enhanced by supporting ipaddress/hostname and could use the approach in the patch {{HOST_RESOLVER-ZK-1045.patch}} bq. 2. in 3.4, create a separate file for the auth list, and link it from zoo.cfg, similarly to the way I link the dynamic config file from zoo.cfg. As an initial attempt I've used zoo.cfg based approach for the authorized hosts. I agree we could enhance this using separate file for the auth list or znode approach etc. How about push this patch first and later we could discuss and implement solution through another jira. bq. 3. In 3.5 support dynamic addition/removal of permissions (this may be very similar to dynamic reconfig): store the auth list in a znode, bq. create a new command for addition/removal/query from the auth list. Whenever the auth list is updated, also update the on-disk auth file. I've plans to raise a separate jira for forward porting the solution to branch3.5 through another jira. I will make a note of these points and will consider while implementing the same. was (Author: rakeshr): Thanks a lot [~phunt], [~shralex], [~hanm] for the discussions and suggestions. I've tried an initial attempt to do the authorization using the hostnames from {{zoo.cfg}}. Kindly review and let me know the feedback. To keep the implementation simple, this patch expects fqdn should be configured in the zoo.cfg. Later this could be enhanced by supporting ipaddress/hostname and could use the approach in the patch {{HOST_RESOLVER-ZK-1045.patch}} bq. 2. in 3.4, create a separate file for the auth list, and link it from zoo.cfg, similarly to the way I link the dynamic config file from zoo.cfg. This will make updating the file easier in 3.5 (see below). As an initial attempt I've used zoo.cfg based approach for the authorized hosts. I agree we could enhance this using separate file for the auth list or znode approach etc. How about push this patch first and later we could discuss and implement solution through another jira. bq. 3. In 3.5 support dynamic addition/removal of permissions (this may be very similar to dynamic reconfig): store the auth list in a znode, create a new command for addition/removal/query from the auth list. Whenever the auth list is updated, also update the on-disk auth file. I've plans to raise a separate jira for forward porting the solution through another jira. I will make a note of these points and will consider while implementing the same. > Support Quorum Peer mutual authentication via SASL > -- > > Key: ZOOKEEPER-1045 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1045 > Project: ZooKeeper > Issue Type: New Feature > Components: server >Reporter: Eugene Koontz >Assignee: Rakesh R >Priority: Critical > Fix For: 3.4.10, 3.5.3 > > Attachments: 0001-ZOOKEEPER-1045-br-3-4.patch, > 1045_failing_phunt.tar.gz, HOST_RESOLVER-ZK-1045.patch, > TEST-org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.txt, > ZK-1045-test-case-failure-logs.zip, ZOOKEEPER-1045-00.patch, > ZOOKEEPER-1045-Rolling Upgrade Design Proposal.pdf, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045TestValidationDesign.pdf > > > ZOOKEEPER-938 addresses mutual authentication between clients and servers. > This bug, on the other hand, is for authentication among quorum peers. > Hopefully much of the work done on SASL integration with Zookeeper for > ZOOKEEPER-938 can be used as a foundation for this enhancement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491138#comment-15491138 ] Rakesh R edited comment on ZOOKEEPER-1045 at 9/14/16 6:32 PM: -- Thanks a lot [~phunt], [~shralex], [~hanm] for the discussions and suggestions. I've tried an initial attempt to do the authorization using the hostnames from {{zoo.cfg}}. Kindly review and let me know the feedback. To keep the implementation simple, this patch expects fqdn should be configured in the zoo.cfg. Later this could be enhanced by supporting ipaddress/hostname and could use the approach in the patch {{HOST_RESOLVER-ZK-1045.patch}} bq. 2. in 3.4, create a separate file for the auth list, and link it from zoo.cfg, similarly to the way I link the dynamic config file from zoo.cfg. This will make updating the file easier in 3.5 (see below). As an initial attempt I've used zoo.cfg based approach for the authorized hosts. I agree we could enhance this using separate file for the auth list or znode approach etc. How about push this patch first and later we could discuss and implement solution through another jira. bq. 3. In 3.5 support dynamic addition/removal of permissions (this may be very similar to dynamic reconfig): store the auth list in a znode, create a new command for addition/removal/query from the auth list. Whenever the auth list is updated, also update the on-disk auth file. I've plans to raise a separate jira for forward porting the solution through another jira. I will make a note of these points and will consider while implementing the same. was (Author: rakeshr): Thanks a lot [~phunt], [~shralex], [~hanm] for the discussions and suggestions. I've tried and initial attempt to do the authorization using the hostnames from {{zoo.cfg}}. Kindly review and let me know the feedback. To keep the implementation simple, this patch expects fqdn should be configured in the zoo.cfg. Later this could be enhanced by supporting ipaddress/hostname and could use the approach in the patch {{HOST_RESOLVER-ZK-1045.patch}} bq. 2. in 3.4, create a separate file for the auth list, and link it from zoo.cfg, similarly to the way I link the dynamic config file from zoo.cfg. This will make updating the file easier in 3.5 (see below). As an initial attempt I've used zoo.cfg based approach for the authorized hosts. I agree we could enhance this using separate file for the auth list or znode approach etc. How about push this patch first and later we could discuss and implement solution through another jira. bq. 3. In 3.5 support dynamic addition/removal of permissions (this may be very similar to dynamic reconfig): store the auth list in a znode, create a new command for addition/removal/query from the auth list. Whenever the auth list is updated, also update the on-disk auth file. I've plans to raise a separate jira for forward porting the solution through another jira. I will make a note of these points and will consider while implementing the same. > Support Quorum Peer mutual authentication via SASL > -- > > Key: ZOOKEEPER-1045 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1045 > Project: ZooKeeper > Issue Type: New Feature > Components: server >Reporter: Eugene Koontz >Assignee: Rakesh R >Priority: Critical > Fix For: 3.4.10, 3.5.3 > > Attachments: 0001-ZOOKEEPER-1045-br-3-4.patch, > 1045_failing_phunt.tar.gz, HOST_RESOLVER-ZK-1045.patch, > TEST-org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.txt, > ZK-1045-test-case-failure-logs.zip, ZOOKEEPER-1045-00.patch, > ZOOKEEPER-1045-Rolling Upgrade Design Proposal.pdf, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045TestValidationDesign.pdf > > > ZOOKEEPER-938 addresses mutual authentication between clients and servers. > This bug, on the other hand, is for authentication among quorum peers. > Hopefully much of the work done on SASL integration with Zookeeper for > ZOOKEEPER-938 can be used as a foundation for this enhancement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15473057#comment-15473057 ] Rakesh R edited comment on ZOOKEEPER-1045 at 9/8/16 7:22 AM: - bq. But authorization should probably happen before, maybe when a reconfig is processed or maybe even before that - when a new server boots and tries to connect to the leader. Does the leader check new server connections ? If that server wasn't initially declared, would it be able to connect ? Perhaps we should identify the exact place where it needs to do the authorization. I can say, this jira is proposing authentication and authorization at the beginning of FLE process. That means, when a learner tries to connect to remote quorum server, first it will do the authn and authz, on success it will proceed to FLE process, otw reject the connecting server. Again, after FLE when a Follower/Observer tries to {{connectToLeader}} it does authn and authz steps. As per my above comment, when a new server tries to connect to leader, leader will cross check with its quorum server principal list(built from zoo.cfg). So here the leader should have the information about the newly connecting server. Will the reconfig commit happens before this? was (Author: rakeshr): bq. But authorization should probably happen before, maybe when a reconfig is processed or maybe even before that - when a new server boots and tries to connect to the leader. Does the leader check new server connections ? If that server wasn't initially declared, would it be able to connect ? Perhaps we should identify the exact place where it needs to do the authorization. I can say, this jira is proposing authentication and authorization at the beginning of FLE process. That means, when a learner tries to connect to remote quorum server, first it will do the authn and authz, on success it will proceed to FLE process, otw reject the connecting server. As per my above comment, when a new server tries to connect to leader, leader will cross check with its quorum server principal list(built from zoo.cfg). So here the leader should have the information about the newly connecting server. Will the reconfig commit happens before this? > Support Quorum Peer mutual authentication via SASL > -- > > Key: ZOOKEEPER-1045 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1045 > Project: ZooKeeper > Issue Type: New Feature > Components: server >Reporter: Eugene Koontz >Assignee: Rakesh R >Priority: Critical > Fix For: 3.4.10, 3.5.3 > > Attachments: 0001-ZOOKEEPER-1045-br-3-4.patch, > 1045_failing_phunt.tar.gz, HOST_RESOLVER-ZK-1045.patch, > TEST-org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.txt, > ZK-1045-test-case-failure-logs.zip, ZOOKEEPER-1045-00.patch, > ZOOKEEPER-1045-Rolling Upgrade Design Proposal.pdf, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045TestValidationDesign.pdf > > > ZOOKEEPER-938 addresses mutual authentication between clients and servers. > This bug, on the other hand, is for authentication among quorum peers. > Hopefully much of the work done on SASL integration with Zookeeper for > ZOOKEEPER-938 can be used as a foundation for this enhancement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15472985#comment-15472985 ] Rakesh R edited comment on ZOOKEEPER-1045 at 9/8/16 6:56 AM: - Thanks everyone for putting all the use cases and active discussions. Let me try to summarize all the problems and proposed solutions. *Point-1)* ??Prevent another host with the same krb user and realm to join my zookeeper cluster, for example {{zk/@EXAMPLE.COM}}?? Like [~phunt], [~hanm] explained we could make use of zoo.cfg as the source of authorization information and replace the _HOST part in {{user/_HOST@REALM}}. Since admins can configure ZK server details as host name or ipaddress or fqdn in zoo.cfg, server should have a mechanism to resolve this to fully qualified domain name for this IP address. Sometime back I've attached {{HOST_RESOLVER-ZK-1045.patch}} idea (thanks to hadoop project [hadoop SecurityUtil ref.|https://github.com/apache/hadoop/blob/branch-2.8/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java#L570]), which is an independent patch to prepare QuorumServer kerberos principal by resolving the host address of ZK server to InetAddress.getLocalHost().getCanonicalHostName() and expects principal like zk/ho...@example.com. This principal will be used by the quorum peer learner to talk to another quorum peer server during FLE. Here the implicit requirement is, admin has to ensure that the configured kerberos principal name should be resolved to fully qualified domain name for this IP address. * For authorization every server will compare the full principal name that composed of {{user/host@realm}}. For doing this every server will cross check with the list of known quorum server principals built from zoo.cfg file. +Ensemble-1 :-+ quorum server principal list => {{zk/ho...@example.com, zk/ho...@example.com, zk/ho...@example.com}} +Ensemble-2 :-+ quorum server principal list => {{zk/ho...@example.com, zk/ho...@example.com, zk/ho...@example.com}} * For authentication, quorum learner server will get the remote quorum server principal name and then do authentication. For example, host1 will get host2 principal {{zk/ho...@example.com}} and do authenticate. Does this make sense? *Point-2)* ??Feature of KDC that it will treat repeated attempts to log in with the same Kerberos principal within a short period of time as replay attacks and will reject such login requests. Since we are supporting shared Kerberos credential, we might hit this issue.?? Good catch, Michael. [~cnauroth], I hope you are pointing me to the hadoop code [Client.java#L699|https://github.com/apache/hadoop/blob/branch-2.8/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L699], isn't it? I'll try to understand this part and get back to you. *Point-3)* ??How would we authorize against something that is not pre-configured? Basically the dynamic reconfiguration of servers (addition and removal). Also, supports upgrade from 3.4 to 3.5 and above.?? [~shralex], IIUC, dynamic reconfig feature is continue using the zoo.cfg configuration file to keep the quorum info and while processing the reconfiguration request, it will always update the zoo.cfg file and ensure this file is uptodate. In that case each server will get the details of newly added server and during this time we should accommodate the logic of updating the {{quorum server principal list}} with the newly added server or removed server details, if any. But there is a case, server tries to join {{zk/@EXAMPLE.COM}}, I think this has to be restricted at the reconfig command execution side rather than FLE, probably ZOOKEEPER-2014 jira will help to resolve this problem, am I missing anything? was (Author: rakeshr): Thanks everyone for putting all the use cases and active discussions. Let me try to summarize all the problems and proposed solutions. *Point-1)* ??Prevent another user from getting Kerberos credentials for {{zk/@EXAMPLE.COM}}, and don't want them to be able to join my cluster.?? Like [~phunt], [~hanm] explained we could make use of zoo.cfg as the source of authorization information and replace the _HOST part in {{user/_HOST@REALM}}. Since admins can configure ZK server details as host name or ipaddress or fqdn in zoo.cfg, server should have a mechanism to resolve this to fully qualified domain name for this IP address. Sometime back I've attached {{HOST_RESOLVER-ZK-1045.patch}} idea (thanks to hadoop project [hadoop SecurityUtil ref.|https://github.com/apache/hadoop/blob/branch-2.8/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java#L570]), which is an independent patch to prepare QuorumServer kerberos principal by resolving the host address of ZK server to
[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15419069#comment-15419069 ] Eugene Koontz edited comment on ZOOKEEPER-1045 at 8/12/16 4:12 PM: --- Thank you [~rakeshr] for the correction - I'm now using the latest, correct patch. was (Author: ekoontz): Thank you ~rakeshr for the correction - I'm now using the latest, correct patch. > Support Quorum Peer mutual authentication via SASL > -- > > Key: ZOOKEEPER-1045 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1045 > Project: ZooKeeper > Issue Type: New Feature > Components: server >Reporter: Eugene Koontz >Assignee: Rakesh R >Priority: Critical > Fix For: 3.5.3, 3.4.10 > > Attachments: 0001-ZOOKEEPER-1045-br-3-4.patch, > 1045_failing_phunt.tar.gz, HOST_RESOLVER-ZK-1045.patch, > TEST-org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.txt, > ZK-1045-test-case-failure-logs.zip, ZOOKEEPER-1045-00.patch, > ZOOKEEPER-1045-Rolling Upgrade Design Proposal.pdf, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045TestValidationDesign.pdf > > > ZOOKEEPER-938 addresses mutual authentication between clients and servers. > This bug, on the other hand, is for authentication among quorum peers. > Hopefully much of the work done on SASL integration with Zookeeper for > ZOOKEEPER-938 can be used as a foundation for this enhancement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388151#comment-15388151 ] Michael Han edited comment on ZOOKEEPER-1045 at 7/21/16 6:19 PM: - [~rakeshr] There is one flaky test (testRollingUpgrade) in latest (July 12) version of patch. Attach the test log. Would be good to fix this test to reduce the flaky tests. The rest of tests passed my pressure tests. was (Author: hanm): [~rakeshr] There is one flaky tests in latest (July 12) version of patch. Attach the test log. Would be good to fix this test to reduce the flaky tests. The rest of tests passed my pressure tests. > Support Quorum Peer mutual authentication via SASL > -- > > Key: ZOOKEEPER-1045 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1045 > Project: ZooKeeper > Issue Type: New Feature > Components: server >Reporter: Eugene Koontz >Assignee: Rakesh R >Priority: Critical > Fix For: 3.4.9, 3.5.3 > > Attachments: 0001-ZOOKEEPER-1045-br-3-4.patch, > 1045_failing_phunt.tar.gz, > TEST-org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.txt, > ZK-1045-test-case-failure-logs.zip, ZOOKEEPER-1045-00.patch, > ZOOKEEPER-1045-Rolling Upgrade Design Proposal.pdf, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045TestValidationDesign.pdf > > > ZOOKEEPER-938 addresses mutual authentication between clients and servers. > This bug, on the other hand, is for authentication among quorum peers. > Hopefully much of the work done on SASL integration with Zookeeper for > ZOOKEEPER-938 can be used as a foundation for this enhancement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383577#comment-15383577 ] Rakesh R edited comment on ZOOKEEPER-1045 at 7/19/16 4:55 AM: -- bq. I believe we should do similar, if not for security then for consistency. Agreed. Previously in this jira, [~dbenediktson] has brought up a case where all the ZK hosts run with the same Kerberos principal. So now we have two cases, need to support both {{_HOST}} based principal name and {{shared}} principal name. I'm assuming there won't be any need of supporting mixture of both of these like, few quorum servers with "zkquorum/_HOST@" pattern in their principal name and few others are having constant name "zkquorum@". Following is an idea to support both the cases. Welcome comments. *Case-1)* {{_HOST}} based principal -> For example, zkquorum/_h...@example.com zoo.cfg has the following configuration which has the 'host' information. This host address {{addr.getCanonicalHostName()}} will be used to replace the {{_HOST}} pattern. We will make use of the existing data structure {{QuorumCnxManager#view}} map to get the respective server's host name. While connecting to the respective server, first the quorum learner will check {{quorum.auth.kerberos.servicePrincipal}} configuration has {{_HOST}} name pattern then convert the Kerberos principal name to a valid name by replacing the {{_HOST}} part. Myid will be used as the key to get the respective quorum server address from the {{#view}}. {code} server.0=host1:11222:11223:participant server.1=host2:11225:11226:participant server.2=host3:11228:11229:participant {code} *Case-2)* Shared principal -> For example, zkquo...@example.com While connecting to peer servers, first the quorum learner will check {{quorum.auth.kerberos.servicePrincipal}} configuration has {{_HOST}} name pattern, if not then will directly use the value as Kerberos principal name and continue with the authentication process. *Case-3* Mixture of {{_HOST}} based and constant principal This won't be supported. We will supports only two valid principal names, either all servers should have "_HOST" based principal name or all servers shares same principal name. bq. I this this is a great idea, however is it possible to move to another jira? OK, I will push this separately bq. It looks like we don't have any tests to verify the authz aspect of the change? Yes, will include test case for this. was (Author: rakeshr): bq. I believe we should do similar, if not for security then for consistency. Agreed. Previously in this jira, [~dbenediktson] has brought up a case where all the ZK hosts run with the same Kerberos principal. So now we have two cases, need to support both {{_HOST}} based principal name and {{shared}} principal name. I'm assuming there won't be any need of supporting mixture of both of these like, few quorum servers with "zkquorum/_HOST@" pattern in their principal name and few others are having constant name "zkquorum@". Following is an idea to support both the cases. Welcome comments. *Case-1)* {{_HOST}} based principal -> For example, zkquorum/_h...@example.com zoo.cfg has the following configuration which has the 'host' information. This host address {{addr.getCanonicalHostName()}} will be used to replace the {{_HOST}} pattern. We will make use of the existing data structure {{QuorumCnxManager#view}} map to get the respective server's host name. While connecting to the respective server, first the quorum learner will check {{quorum.auth.kerberos.servicePrincipal}} configuration has {{_HOST}} name pattern then convert the Kerberos principal name to a valid name by replacing the {{_HOST}} part. Myid will be used as the key to get the respective quorum server address from the {{#view}}. {code} server.0=host1:11222:11223:participant server.1=host2:11225:11226:participant server.2=host3:11228:11229:participant {code} *Case-2)* Shared principal -> For example, zkquo...@example.com While connecting to peer servers, first the quorum learner will check {{quorum.auth.kerberos.servicePrincipal}} configuration has {{_HOST}} name pattern, if not then will directly use the value as Kerberos principal name and continue with the authentication process. *Case-3* Mixture of {{_HOST}} based and constant principal ZooKeeper doesn't supports this case. ZooKeeper supports only two valid principal names, either all servers should have "_HOST" based principal name or all servers shares same principal name. bq. I this this is a great idea, however is it possible to move to another jira? OK, I will push this separately bq. It looks like we don't have any tests to verify the authz aspect of the change? Yes, will include test case for this. > Support Quorum Peer mutual authentication via SASL > -- > >
[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360796#comment-15360796 ] Rakesh R edited comment on ZOOKEEPER-1045 at 7/4/16 3:36 AM: - [~hanm] Yes, exactly. Presently we will be supporting only single(shared) Kerb principal across all the servers. We could capture this point clearly in our test report documentation and later the same can be used to update {{cwiki page}} as well. In future, if anyone has a use case of different Kerb principal then we can discuss/extend the implementation to support the same later. IMHO, its not required to handle those complex case now. Does that make sense to you? was (Author: rakeshr): [~hanm] Yes, exactly. Presently we will be supporting only single(shared) Kerb principal across all the servers now. We could capture this point clearly in our test report documentation and later the same can be used to update {{cwiki page}} as well. In future, if anyone has a use case of different Kerb principal then we can discuss/extend the implementation to support the same later. IMHO, its not required to handle those complex case now. Does that make sense to you? > Support Quorum Peer mutual authentication via SASL > -- > > Key: ZOOKEEPER-1045 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1045 > Project: ZooKeeper > Issue Type: New Feature > Components: server >Reporter: Eugene Koontz >Assignee: Rakesh R >Priority: Critical > Fix For: 3.4.9, 3.5.3 > > Attachments: 0001-ZOOKEEPER-1045-br-3-4.patch, > 1045_failing_phunt.tar.gz, ZK-1045-test-case-failure-logs.zip, > ZOOKEEPER-1045-00.patch, ZOOKEEPER-1045-Rolling Upgrade Design Proposal.pdf, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045-br-3-4.patch > > > ZOOKEEPER-938 addresses mutual authentication between clients and servers. > This bug, on the other hand, is for authentication among quorum peers. > Hopefully much of the work done on SASL integration with Zookeeper for > ZOOKEEPER-938 can be used as a foundation for this enhancement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360675#comment-15360675 ] Michael Han edited comment on ZOOKEEPER-1045 at 7/3/16 8:46 PM: bq. For the server-server auth, Kerb principal should be same for all the servers to allow communicating each other. IIUC, this means that we only support a single (shared) Kerberos principal / credential across all servers for server to server communication, and if so, the failure of my validation against the case where servers use different Kerberos principal is a by design, because I was using different Kerberos principals on each server for server to server auth validation. [~rakeshr] was (Author: hanm): b.q. For the server-server auth, Kerb principal should be same for all the servers to allow communicating each other. IIUC, this means that we only support a single (shared) Kerberos principal / credential across all servers for server to server communication, and if so, the failure of my validation against the case where servers use different Kerberos principal is a by design, because I was using different Kerberos principals on each server for server to server auth validation. [~rakeshr] > Support Quorum Peer mutual authentication via SASL > -- > > Key: ZOOKEEPER-1045 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1045 > Project: ZooKeeper > Issue Type: New Feature > Components: server >Reporter: Eugene Koontz >Assignee: Rakesh R >Priority: Critical > Fix For: 3.4.9, 3.5.3 > > Attachments: 0001-ZOOKEEPER-1045-br-3-4.patch, > 1045_failing_phunt.tar.gz, ZK-1045-test-case-failure-logs.zip, > ZOOKEEPER-1045-00.patch, ZOOKEEPER-1045-Rolling Upgrade Design Proposal.pdf, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch, > ZOOKEEPER-1045-br-3-4.patch > > > ZOOKEEPER-938 addresses mutual authentication between clients and servers. > This bug, on the other hand, is for authentication among quorum peers. > Hopefully much of the work done on SASL integration with Zookeeper for > ZOOKEEPER-938 can be used as a foundation for this enhancement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358379#comment-15358379 ] Rakesh R edited comment on ZOOKEEPER-1045 at 7/1/16 6:03 AM: - Thanks [~dbenediktson] for the interest in this work and sharing the use case. bq. please make sure you support the case where all the ZK hosts run as the same Kerberos principal Yes, you can configure the same Krb credentials for client-server and server-server communications. As part of this jira, there is no changes to the existing client-server communication path, this will work as it is. I will try to add few details about the server-server auth configs. For the server-server auth, Kerb principal should be same for all the servers to allow communicating each other. Since each server will talk to all the other servers to form quorum it is required to know each others Krb principal. This jira introduces {{QuorumServer}} section where admin can configure the Krb principal of self (also it represents the principal of other quorum peer server), learner will use this and can contact them. In the below example config, should use same {{principal="zkquorum/localh...@example.com";}} in all the servers. {code} QuorumServer { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/path/to/keytab" storeKey=true useTicketCache=false debug=false principal="zkquorum/localh...@example.com"; }; {code} Few days back there was a discussion about configuring [different Kerb credentials|https://issues.apache.org/jira/browse/ZOOKEEPER-1045?focusedCommentId=15339198=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15339198] for client-server(Server) and server-server(QuorumServer/QuorumLearner) communications. Please refer this to understand more. We will be supporting Krb credentials in the following ways. I'd appreciate if you can test the same in your env and see its working. 1) all ZK hosts sharing same Kerb principal for both client-server and server-server 2) client-server(Server) uses {{principal_1}} and server-server(QuorumServer/QuorumLearner) uses {{principal_2}}. bq.I've validated that server to server Kerberos SASL auth working, when servers share same credentials (same service principal name + same full qualified domain+ same keytabs) deployed on all nodes. Thanks [~hanm] for the confirmation. bq. For the cases where each server has a distinct Kerberos credential, it's not working yet. [~hanm], please let me the QuorumServer principal values. Could you share the {{jaas.config}} of all the servers and the failure logs for better debugging. was (Author: rakeshr): Thanks [~dbenediktson] for the interest in this work and sharing the use case. bq. please make sure you support the case where all the ZK hosts run as the same Kerberos principal Yes, you can configure the same Krb credentials for client-server and server-server communications. As part of this jira, there is no changes to the existing client-server communication path, this will work as it is. I will try to add few details about the server-server auth configs. For the server-server auth, Kerb principal should be same for all the servers to allow communicating each other. Since each server will talk to all the other servers to form quorum it is required to know each others Krb principal. This jira introduces {{QuorumServer}} section where admin can configure the principal of self(also the principal of other quorum peer server), learner will use this and can contact them. In the below example config, should use same {{principal="zkquorum/localh...@example.com";}} in all the servers. {code} QuorumServer { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/path/to/keytab" storeKey=true useTicketCache=false debug=false principal="zkquorum/localh...@example.com"; }; {code} Few days back there was a discussion about configuring [different Kerb credentials|https://issues.apache.org/jira/browse/ZOOKEEPER-1045?focusedCommentId=15339198=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15339198] for client-server(Server) and server-server(QuorumServer/QuorumLearner) communications. Please refer this to understand more. We will be supporting Krb credentials in the following ways. I'd appreciate if you can test the same in your env and see its working. 1) all ZK hosts sharing same Kerb principal for both client-server and server-server 2) client-server(Server) uses {{principal_1}} and server-server(QuorumServer/QuorumLearner) uses {{principal_2}}. bq.I've validated that server to server Kerberos SASL auth working, when servers share same credentials (same service principal name + same full qualified domain+ same keytabs) deployed on all nodes.
[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358379#comment-15358379 ] Rakesh R edited comment on ZOOKEEPER-1045 at 7/1/16 6:01 AM: - Thanks [~dbenediktson] for the interest in this work and sharing the use case. bq. please make sure you support the case where all the ZK hosts run as the same Kerberos principal Yes, you can configure the same Krb credentials for client-server and server-server communications. As part of this jira, there is no changes to the existing client-server communication path, this will work as it is. I will try to add few details about the server-server auth configs. For the server-server auth, Kerb principal should be same for all the servers to allow communicating each other. Since each server will talk to all the other servers to form quorum it is required to know each others Krb principal. This jira introduces {{QuorumServer}} section where admin can configure the principal of self(also the principal of other quorum peer server), learner will use this and can contact them. In the below example config, should use same {{principal="zkquorum/localh...@example.com";}} in all the servers. {code} QuorumServer { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/path/to/keytab" storeKey=true useTicketCache=false debug=false principal="zkquorum/localh...@example.com"; }; {code} Few days back there was a discussion about configuring [different Kerb credentials|https://issues.apache.org/jira/browse/ZOOKEEPER-1045?focusedCommentId=15339198=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15339198] for client-server(Server) and server-server(QuorumServer/QuorumLearner) communications. Please refer this to understand more. We will be supporting Krb credentials in the following ways. I'd appreciate if you can test the same in your env and see its working. 1) all ZK hosts sharing same Kerb principal for both client-server and server-server 2) client-server(Server) uses {{principal_1}} and server-server(QuorumServer/QuorumLearner) uses {{principal_2}}. bq.I've validated that server to server Kerberos SASL auth working, when servers share same credentials (same service principal name + same full qualified domain+ same keytabs) deployed on all nodes. Thanks [~hanm] for the confirmation. bq. For the cases where each server has a distinct Kerberos credential, it's not working yet. [~hanm], please let me the QuorumServer principal values. Could you share the {{jaas.config}} of all the servers and the failure logs for better debugging. was (Author: rakeshr): Thanks [~dbenediktson] for the interest in this work and sharing the use case. bq. please make sure you support the case where all the ZK hosts run as the same Kerberos principal Yes, you can configure the same Krb credentials for client-server and server-server communications. As part of this jira, there is no changes to the existing client-server communication path, this will work as it is. I will try to add few details about the server-server auth configs. For the server-server auth, Kerb principal should be same for all the servers to allow communicating each other. Since each server will talk to all the other servers to form quorum it is required to know each others Krb principal. This jira introduces {{QuorumServer}} section where admin can configure the principal of other quorum peer server so that the learner can use this and can contact them. In the below example config, should use same {{principal="zkquorum/localh...@example.com";}} in all the servers. {code} QuorumServer { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/path/to/keytab" storeKey=true useTicketCache=false debug=false principal="zkquorum/localh...@example.com"; }; {code} Few days back there was a discussion about configuring [different Kerb credentials|https://issues.apache.org/jira/browse/ZOOKEEPER-1045?focusedCommentId=15339198=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15339198] for client-server(Server) and server-server(QuorumServer/QuorumLearner) communications. Please refer this to understand more. We will be supporting Krb credentials in the following ways. I'd appreciate if you can test the same in your env and see its working. 1) all ZK hosts sharing same Kerb principal for both client-server and server-server 2) client-server(Server) uses {{principal_1}} and server-server(QuorumServer/QuorumLearner) uses {{principal_2}}. bq.I've validated that server to server Kerberos SASL auth working, when servers share same credentials (same service principal name + same full qualified domain+ same keytabs) deployed on all nodes. Thanks [~hanm] for the confirmation.
[jira] [Comment Edited] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358379#comment-15358379 ] Rakesh R edited comment on ZOOKEEPER-1045 at 7/1/16 4:52 AM: - Thanks [~dbenediktson] for the interest in this work and sharing the use case. bq. please make sure you support the case where all the ZK hosts run as the same Kerberos principal Yes, you can configure the same Krb credentials for client-server and server-server communications. As part of this jira, there is no changes to the existing client-server communication path, this will work as it is. I will try to add few details about the server-server auth configs. For the server-server auth, Kerb principal should be same for all the servers to allow communicating each other. Since each server will talk to all the other servers to form quorum it is required to know each others Krb principal. This jira introduces {{QuorumServer}} section where admin can configure the principal of other quorum peer server so that the learner can use this and can contact them. In the below example config, should use same {{principal="zkquorum/localh...@example.com";}} in all the servers. {code} QuorumServer { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/path/to/keytab" storeKey=true useTicketCache=false debug=false principal="zkquorum/localh...@example.com"; }; {code} Few days back there was a discussion about configuring [different Kerb credentials|https://issues.apache.org/jira/browse/ZOOKEEPER-1045?focusedCommentId=15339198=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15339198] for client-server(Server) and server-server(QuorumServer/QuorumLearner) communications. Please refer this to understand more. We will be supporting Krb credentials in the following ways. I'd appreciate if you can test the same in your env and see its working. 1) all ZK hosts sharing same Kerb principal for both client-server and server-server 2) client-server(Server) uses {{principal_1}} and server-server(QuorumServer/QuorumLearner) uses {{principal_2}}. bq.I've validated that server to server Kerberos SASL auth working, when servers share same credentials (same service principal name + same full qualified domain+ same keytabs) deployed on all nodes. Thanks [~hanm] for the confirmation. bq. For the cases where each server has a distinct Kerberos credential, it's not working yet. [~hanm], please let me the QuorumServer principal values. Could you share the {{jaas.config}} of all the servers and the failure logs for better debugging. was (Author: rakeshr): Thanks [~dbenediktson] for the interest on this work and sharing the use case. bq. please make sure you support the case where all the ZK hosts run as the same Kerberos principal Yes, you can configure the same Krb credentials for client-server and server-server communications. As part of this jira, there is no changes to the existing client-server communication path, this will work as it is. I will try to add few details about the server-server auth configs. For the server-server auth, Kerb principal should be same for all the servers to allow communicating each other. Since each server will talk to all the other servers to form quorum it is required to know each others Krb principal. This jira introduces {{QuorumServer}} section where admin can configure the principal of other quorum peer server so that the learner can use this and can contact them. In the below example config, should use same {{principal="zkquorum/localh...@example.com";}} in all the servers. {code} QuorumServer { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/path/to/keytab" storeKey=true useTicketCache=false debug=false principal="zkquorum/localh...@example.com"; }; {code} Few days back there was a discussion about configuring [different Kerb credentials|https://issues.apache.org/jira/browse/ZOOKEEPER-1045?focusedCommentId=15339198=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15339198] for client-server(Server) and server-server(QuorumServer/QuorumLearner) communications. Please refer this to understand more. We will be supporting Krb credentials in the following ways. I'd appreciate if you can test the same in your env and see its working. 1) all ZK hosts sharing same Kerb principal for both client-server and server-server 2) client-server(Server) uses {{principal_1}} and server-server(QuorumServer/QuorumLearner) uses {{principal_2}}. bq.I've validated that server to server Kerberos SASL auth working, when servers share same credentials (same service principal name + same full qualified domain+ same keytabs) deployed on all nodes. Thanks [~hanm] for the confirmation. bq. For the