[jira] [Commented] (ZOOKEEPER-3482) SASL (Kerberos) Authentication with SSL for clients and Quorum
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17016984#comment-17016984 ] Mate Szalay-Beko commented on ZOOKEEPER-3482: - I updated my two PRs. I think, this bug report can be closed (after the PRs got merged). In the PRs I added new unit tests and updated the ZooKeeper Admin Guide, and ZooKeeper Programming Guide (these are stored in git). I also updated the wiki here: https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication Together with [~andor] we also tested Kerberos authentication over SSL on real Hadoop clusters, so I am confident to say that both features are working at least on 3.5.5, 3.5.6 and on the current 3.6 and master branches. I know that there is an other requirement in this Jira, namely: allowing client SSL for ZooKeeper without actual authentication (no keystore in client side, only truststore). This is not working now in ZooKeeper, but I don't think it would be a bug... this is simply how the SSL feature was designed originally. (In ZooKeeper SSL was plugged in as an authentication module.) Although I think it is a good idea and we should implement it. We should issue a separate Jira for this. I would be happy to work on this, but not sure when I will have time exactly. > SASL (Kerberos) Authentication with SSL for clients and Quorum > -- > > Key: ZOOKEEPER-3482 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3482 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.5.5 >Reporter: Jörn Franke >Assignee: Mate Szalay-Beko >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > It seems that Kerberos authentication does not work for encrypted connections > of clients and quorum. It seems that only X509 Authentication works. > What I would have expected: > ClientSecurePort is defined > A keystore and truststore are deployed on the ZooKeeper servers > Only a truststore is deployed with the client (to validate the CA of the > server certificate) > Client can authenticate with SASL (Kerberos) > Similarly, it should work for the Quorum SSL connection. > Is there a way to configure this in ZooKeeper? > > Note: Kerberos Authentication for SSL encrypted connection should be used > instead of X509 authentication for this case and not in addition. However, if > it only works in 3.5.5 in addition then I would be interested and willing to > test it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ZOOKEEPER-3482) SASL (Kerberos) Authentication with SSL for clients and Quorum
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014286#comment-17014286 ] Andor Molnar commented on ZOOKEEPER-3482: - [~symat] [~jornfranke] I repeated my test with another cluster and I was able to use SSL and Kerberos in conjunction successfully. I'm still looking at my original report to see the difference, but unfortunately the test cluster has already been destroyed and I cannot see anything obvious now. Anyway we can say that ZooKeeper supports Kerberized client connection on the secure port as of version 3.5.5 Adding new tests is a very good idea and also some sort of documentation about how to set this up properly would also be useful. Thanks. > SASL (Kerberos) Authentication with SSL for clients and Quorum > -- > > Key: ZOOKEEPER-3482 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3482 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.5.5 >Reporter: Jörn Franke >Assignee: Mate Szalay-Beko >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > It seems that Kerberos authentication does not work for encrypted connections > of clients and quorum. It seems that only X509 Authentication works. > What I would have expected: > ClientSecurePort is defined > A keystore and truststore are deployed on the ZooKeeper servers > Only a truststore is deployed with the client (to validate the CA of the > server certificate) > Client can authenticate with SASL (Kerberos) > Similarly, it should work for the Quorum SSL connection. > Is there a way to configure this in ZooKeeper? > > Note: Kerberos Authentication for SSL encrypted connection should be used > instead of X509 authentication for this case and not in addition. However, if > it only works in 3.5.5 in addition then I would be interested and willing to > test it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ZOOKEEPER-3482) SASL (Kerberos) Authentication with SSL for clients and Quorum
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17009816#comment-17009816 ] Mate Szalay-Beko commented on ZOOKEEPER-3482: - update: I managed to create some working unit tests on the master branch. I tested SASL Digest + SSL, and also SASL Kerberos + SSL. So it seems working on the master branch (although due to some conflict, it requires more work to make the tests work on the branch 3.5). the PR for the master branch: https://github.com/apache/zookeeper/pull/1204 I want to work now on the 3.5 branch to see if the problem exists there or not. I also want to reproduce this scenario on a real kerberized server. > SASL (Kerberos) Authentication with SSL for clients and Quorum > -- > > Key: ZOOKEEPER-3482 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3482 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.5.5 >Reporter: Jörn Franke >Assignee: Mate Szalay-Beko >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > It seems that Kerberos authentication does not work for encrypted connections > of clients and quorum. It seems that only X509 Authentication works. > What I would have expected: > ClientSecurePort is defined > A keystore and truststore are deployed on the ZooKeeper servers > Only a truststore is deployed with the client (to validate the CA of the > server certificate) > Client can authenticate with SASL (Kerberos) > Similarly, it should work for the Quorum SSL connection. > Is there a way to configure this in ZooKeeper? > > Note: Kerberos Authentication for SSL encrypted connection should be used > instead of X509 authentication for this case and not in addition. However, if > it only works in 3.5.5 in addition then I would be interested and willing to > test it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ZOOKEEPER-3482) SASL (Kerberos) Authentication with SSL for clients and Quorum
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17008690#comment-17008690 ] Mate Szalay-Beko commented on ZOOKEEPER-3482: - I started to work on this... > SASL (Kerberos) Authentication with SSL for clients and Quorum > -- > > Key: ZOOKEEPER-3482 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3482 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.5.5 >Reporter: Jörn Franke >Assignee: Mate Szalay-Beko >Priority: Major > > It seems that Kerberos authentication does not work for encrypted connections > of clients and quorum. It seems that only X509 Authentication works. > What I would have expected: > ClientSecurePort is defined > A keystore and truststore are deployed on the ZooKeeper servers > Only a truststore is deployed with the client (to validate the CA of the > server certificate) > Client can authenticate with SASL (Kerberos) > Similarly, it should work for the Quorum SSL connection. > Is there a way to configure this in ZooKeeper? > > Note: Kerberos Authentication for SSL encrypted connection should be used > instead of X509 authentication for this case and not in addition. However, if > it only works in 3.5.5 in addition then I would be interested and willing to > test it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ZOOKEEPER-3482) SASL (Kerberos) Authentication with SSL for clients and Quorum
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998314#comment-16998314
]
Mate Szalay-Beko commented on ZOOKEEPER-3482:
-
So AFAICS the expected behaviour would be:
1) be able to use SSL without actual authentication (no keystore in client
side, only truststore)
2) use Kerberos authentication on top of the SSL session
Actually for issue 1 there is an undocumented feature I just found in the code.
But maybe it was trivial for everyone else but me :)
https://github.com/apache/zookeeper/blob/48e5eaadffd8e23d2f47fe3eb0d0437b172dcd39/zookeeper-server/src/main/java/org/apache/zookeeper/common/X509Util.java#L116
It looks like setting {{zookeeper.ssl.clientAuth}} and
{{zookeeper.ssl.quorum.clientAuth}} to {{none}} in the server config should
solve issue 1.
But still it is a question for me if the same configuration parameter would
enable to use SASL on top of SSL. I guess not...
I am happy to work on this ticket after the Holidays. But if someone would like
to start it before, feel free (and please assign the ticket to herself/himself
avoiding double-work).
> SASL (Kerberos) Authentication with SSL for clients and Quorum
> --
>
> Key: ZOOKEEPER-3482
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3482
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.5.5
>Reporter: Jörn Franke
>Priority: Major
>
> It seems that Kerberos authentication does not work for encrypted connections
> of clients and quorum. It seems that only X509 Authentication works.
> What I would have expected:
> ClientSecurePort is defined
> A keystore and truststore are deployed on the ZooKeeper servers
> Only a truststore is deployed with the client (to validate the CA of the
> server certificate)
> Client can authenticate with SASL (Kerberos)
> Similarly, it should work for the Quorum SSL connection.
> Is there a way to configure this in ZooKeeper?
>
> Note: Kerberos Authentication for SSL encrypted connection should be used
> instead of X509 authentication for this case and not in addition. However, if
> it only works in 3.5.5 in addition then I would be interested and willing to
> test it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ZOOKEEPER-3482) SASL (Kerberos) Authentication with SSL for clients and Quorum
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998033#comment-16998033
]
Andor Molnar commented on ZOOKEEPER-3482:
-
A little bit more context:
ZooKeeper server has been setup for both Kerberos auth and secure port enabled:
{panel:title=zoo.cfg}
tickTime=2000
initLimit=10
syncLimit=5
4lw.commands.whitelist=conf,cons,crst,dirs,dump,envi,gtmk,ruok,stmk,srst,srvr,stat,wchs,mntr,isro
dataDir=/var/lib/zookeeper
dataLogDir=/var/lib/zookeeper
clientPort=2181
maxClientCnxns=60
minSessionTimeout=4000
maxSessionTimeout=6
autopurge.purgeInterval=24
autopurge.snapRetainCount=5
quorum.auth.enableSasl=true
quorum.cnxn.threads.size=20
admin.enableServer=false
admin.serverPort=5181
server.1=barbaresco-1.vpc.cloudera.com:3181:4181
leaderServes=yes
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
kerberos.removeHostFromPrincipal=true
kerberos.removeRealmFromPrincipal=true
quorum.auth.kerberos.servicePrincipal=zookeeper/_HOST
quorum.auth.learnerRequireSasl=true
quorum.auth.serverRequireSasl=true
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
sslQuorum=true
ssl.quorum.keyStore.location=/var/run/zookeeper/keystore.jks
ssl.quorum.keyStore.password=topSecret
ssl.quorum.trustStore.location=/var/lib/zookeeper/truststore.jks
ssl.quorum.trustStore.password=topSecret
secureClientPort=2182
ssl.keyStore.location=/var/run/zookeeper/keystore.jks
ssl.keyStore.password=topSecret
ssl.trustStore.location=/var/lib/zookeeper/truststore.jks
ssl.trustStore.password=topSecret
{panel}
Client is started as follows:
{panel:title=zookeeper-client}
CLIENT_JVMFLAGS="-Djava.security.auth.login.config=/etc/zookeeper/conf/jaas.conf
-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
-Dzookeeper.ssl.keyStore.location=/var/lib/zookeeper/keystore.jks
-Dzookeeper.ssl.keyStore.password=topSecret
-Dzookeeper.ssl.trustStore.location=/var/lib/zookeeper/truststore.jks
-Dzookeeper.ssl.trustStore.password=topSecret -Dzookeeper.client.secure=true"
zookeeper-client -server barbaresco-1.vpc.cloudera.com:2182
{panel}
> SASL (Kerberos) Authentication with SSL for clients and Quorum
> --
>
> Key: ZOOKEEPER-3482
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3482
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.5.5
>Reporter: Jörn Franke
>Priority: Major
>
> It seems that Kerberos authentication does not work for encrypted connections
> of clients and quorum. It seems that only X509 Authentication works.
> What I would have expected:
> ClientSecurePort is defined
> A keystore and truststore are deployed on the ZooKeeper servers
> Only a truststore is deployed with the client (to validate the CA of the
> server certificate)
> Client can authenticate with SASL (Kerberos)
> Similarly, it should work for the Quorum SSL connection.
> Is there a way to configure this in ZooKeeper?
>
> Note: Kerberos Authentication for SSL encrypted connection should be used
> instead of X509 authentication for this case and not in addition. However, if
> it only works in 3.5.5 in addition then I would be interested and willing to
> test it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ZOOKEEPER-3482) SASL (Kerberos) Authentication with SSL for clients and Quorum
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998028#comment-16998028
]
Andor Molnar commented on ZOOKEEPER-3482:
-
[~jornfranke]
I confirm that the issue is valid. On my test cluster when Kerberos is enabled
client is unable to connect to the secure port:
{noformat}
2019-12-17 01:43:30,984 [myid:barbaresco-1.vpc.cloudera.com:2182] - WARN
[Thread-39:Login$1@197] - TGT renewal thread has been interrupted and will exit.
2019-12-17 01:43:30,987 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
[main-SendThread(barbaresco-1.vpc.cloudera.com:2182):Login@302] - Client
successfully logged in.
2019-12-17 01:43:30,987 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
[Thread-40:Login$1@135] - TGT refresh thread started.
2019-12-17 01:43:30,987 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
[main-SendThread(barbaresco-1.vpc.cloudera.com:2182):SecurityUtils$1@124] -
Client will use GSSAPI as SASL mechanism.
2019-12-17 01:43:30,988 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
[main-SendThread(barbaresco-1.vpc.cloudera.com:2182):ClientCnxn$SendThread@1112]
- Opening socket connection to server
barbaresco-1.vpc.cloudera.com/10.65.25.98:2182. Will attempt to
SASL-authenticate using Login Context section 'Client'
2019-12-17 01:43:30,988 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
[main-SendThread(barbaresco-1.vpc.cloudera.com:2182):ClientCnxn$SendThread@959]
- Socket connection established, initiating session, client:
/10.65.25.98:45362, server: barbaresco-1.vpc.cloudera.com/10.65.25.98:2182
2019-12-17 01:43:30,989 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
[Thread-40:Login@320] - TGT valid starting at:Tue Dec 17 01:43:30 PST
2019
2019-12-17 01:43:30,989 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
[Thread-40:Login@321] - TGT expires: Thu Jan 16 01:43:30 PST
2020
2019-12-17 01:43:30,989 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
[Thread-40:Login$1@193] - TGT refresh sleeping until: Fri Jan 10 20:23:33 PST
2020
2019-12-17 01:43:30,989 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
[main-SendThread(barbaresco-1.vpc.cloudera.com:2182):ClientCnxn$SendThread@1240]
- Unable to read additional data from server sessionid 0x0, likely server has
closed socket, closing socket connection and attempting reconnect{noformat}
And error on server side:
{noformat}
2019-12-17 01:43:33,002 INFO
org.apache.zookeeper.server.NettyServerCnxnFactory: SSL handler added for
channel: [id: 0xcf37c14b, L:/10.65.25.98:2182 - R:/10.65.25.98:45380]
2019-12-17 01:43:33,003 ERROR
org.apache.zookeeper.server.NettyServerCnxnFactory: Unsuccessful handshake with
session 0x0
2019-12-17 01:43:33,003 WARN
org.apache.zookeeper.server.NettyServerCnxnFactory: Exception caught
io.netty.handler.codec.DecoderException:
io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:
002d7530001000
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:475)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
at
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
at
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:483)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:383)
at
io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748){noformat}
> SASL (Kerberos) Authentication with SSL for clients and Quorum
> --
>
> Key: ZOOKEEPER-3482
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3482
>
[jira] [Commented] (ZOOKEEPER-3482) SASL (Kerberos) Authentication with SSL for clients and Quorum
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970533#comment-16970533 ] Andor Molnar commented on ZOOKEEPER-3482: - [~jornfranke] Would you please be a little more specific about the problem? Please provide ZooKeeper version, config files, client settings and log files. Please also elaborate on what steps have you made before facing the problem, what would be the expected behaviour and what was your experience? > SASL (Kerberos) Authentication with SSL for clients and Quorum > -- > > Key: ZOOKEEPER-3482 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3482 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.5.5 >Reporter: Jörn Franke >Priority: Major > > It seems that Kerberos authentication does not work for encrypted connections > of clients and quorum. It seems that only X509 Authentication works. > What I would have expected: > ClientSecurePort is defined > A keystore and truststore are deployed on the ZooKeeper servers > Only a truststore is deployed with the client (to validate the CA of the > server certificate) > Client can authenticate with SASL (Kerberos) > Similarly, it should work for the Quorum SSL connection. > Is there a way to configure this in ZooKeeper? > > Note: Kerberos Authentication for SSL encrypted connection should be used > instead of X509 authentication for this case and not in addition. However, if > it only works in 3.5.5 in addition then I would be interested and willing to > test it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ZOOKEEPER-3482) SASL (Kerberos) Authentication with SSL for clients and Quorum
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948330#comment-16948330 ] Jan Høydahl commented on ZOOKEEPER-3482: Anyone from ZK team have a comment on this? > SASL (Kerberos) Authentication with SSL for clients and Quorum > -- > > Key: ZOOKEEPER-3482 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3482 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.5.5 >Reporter: Jörn Franke >Priority: Major > > It seems that Kerberos authentication does not work for encrypted connections > of clients and quorum. It seems that only X509 Authentication works. > What I would have expected: > ClientSecurePort is defined > A keystore and truststore are deployed on the ZooKeeper servers > Only a truststore is deployed with the client (to validate the CA of the > server certificate) > Client can authenticate with SASL (Kerberos) > Similarly, it should work for the Quorum SSL connection. > Is there a way to configure this in ZooKeeper? > > Note: Kerberos Authentication for SSL encrypted connection should be used > instead of X509 authentication for this case and not in addition. However, if > it only works in 3.5.5 in addition then I would be interested and willing to > test it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ZOOKEEPER-3482) SASL (Kerberos) Authentication with SSL for clients and Quorum
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898268#comment-16898268 ] Jörn Franke commented on ZOOKEEPER-3482: Please note that some ZooKeeper Clients, e.g. Solr, do not seem to support X509 Authentication+ACLs, but only Digest and SASL (Kerberos). > SASL (Kerberos) Authentication with SSL for clients and Quorum > -- > > Key: ZOOKEEPER-3482 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3482 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.5.5 >Reporter: Jörn Franke >Priority: Major > > It seems that Kerberos authentication does not work for encrypted connections > of clients and quorum. It seems that only X509 Authentication works. > What I would have expected: > ClientSecurePort is defined > A keystore and truststore are deployed on the ZooKeeper servers > Only a truststore is deployed with the client (to validate the CA of the > server certificate) > Client can authenticate with SASL (Kerberos) > Similarly for the Quorum SSL connection. > Is there a way to configure this in ZooKeeeper? -- This message was sent by Atlassian JIRA (v7.6.14#76016)
