We use mTLS with short lived certificates which are provided via an authenticated certificate authority service (CAS) and only work for a specific cluster. In order for someone to get a certificate, they must make a request to CAS which verifies they are allowed to have a certificate for the specific cluster. Once you have a certificate, it only works for a few hours. The client side will periodically provision a new one in a thread, and FileWatcher will swap it in. Further, with host verification the certificate only works on the given host it was provisioned on (a secure host generally)
As I said, when I first planned this work I had considered a feature like you said. I decided not to do it for many of the reasons Andrew said. With the above design, we can trust our connections enough to just trust the username. The username on hbase isn’t used for much beyond table ACL, quotas, and bulk load verification. The only one that mattered for us is table ACL and actually it’s somewhat easier to manage this outside of hbase. We have a coprocessor which inspects the certificate for audit logging, and we can add a table field to the certificate which we could validate. There’s already a field on the RpcCall to get the current certificate. You could use that in a coprocessor to validate the username in your application, as Andrew mentions. On Mon, Jun 10, 2024 at 3:31 AM Andor Molnar <an...@apache.org> wrote: > On Sun, 2024-06-09 at 23:10 -0700, Andrew Purtell wrote: > > > There is middle ground between your position and a blanket trust of > > connection header parameters from any client with a valid > > certificate... > > Certificates can carry additional information, including usernames, > > which > > would be protected by the signature. Sometimes this is done. HBase > > doesn't > > implement it (yet) and if you have a trust architecture that demands > > it, it > > would establish the username as attested by an authority, like a > > kerberos > > ticket. You'd use a netty API presumably in a coprocessor hook to get > > it in > > order to hand off to UGI. Should someone decide this is needed it > > would > > make a good contribution back to the project, somewhere in hbase- > > examples > > perhaps. > > Exactly. That's what I wanted to clarify and pointed what other > projects (like ZooKeeper) are doing under the hood of x509 > authentication. I definitely would like to contribute something similar > for HBase too. > > Thanks for the pointer of hbase-examples. > > Andor > > > > > > > > On Sun, Jun 9, 2024 at 10:57 PM Andrew Purtell <apurt...@apache.org> > > wrote: > > > > > Like I said the successful verification of the certificate implies > > > the > > > client is trustworthy, including what the client supplies in the > > > header. > > > Now, if within your organization, you are distributing trusted > > > certificates > > > to potentially untrustworthy software, that is a you problem, as > > > they say. > > > > > > It is absolutely not true that kerberos authentication is > > > *required* when > > > using mTLS. Frankly I'd prefer you expand on such comments to > > > explain why > > > you might have untrustworthy clients operating in your production > > > as > > > opposed to another organization with competent operations and > > > better > > > controls. Otherwise this flirts with F.U.D. > > > > > > > > > On Sun, Jun 9, 2024 at 10:07 PM Andor Molnar <an...@apache.org> > > > wrote: > > > > > > > Yeah. I think the key point here is that the client certificate > > > > identifies the originating "host" and not the "user", hence we > > > > have the > > > > client hostname verification built-in. The only thing that you > > > > can be > > > > sure about when you do mTLS is that the request is coming from a > > > > legitimate host. > > > > > > > > Therefore you still need a secure authentication method in order > > > > to > > > > prevent legitimate users to impersonate each other. > > > > > > > > For instance, you have Alice and Bob valid users both having > > > > trusted > > > > client certificates. They both can make requests from legitimate > > > > hosts, > > > > but with SIMPLE auth they're free to lie about their valid > > > > usernames. > > > > > > > > As far as I know, the only secure authentication option for HBase > > > > is > > > > Kerberos, so you still have to use it. Using mTLS will prevent > > > > attackers from making requests from ordinary hosts by stoling > > > > Kerberos > > > > tokens. > > > > > > > > Andor > > > > > > > > > > > > > > > > On Sun, 2024-06-09 at 13:45 -0400, Bryan Beaudreault wrote: > > > > > mTLS is totally unrelated to the username. It's whatever you'd > > > > > typically > > > > > have without mTLS. > > > > > > > > > > On Sun, Jun 9, 2024 at 1:38 PM Andor Molnar <an...@apache.org> > > > > > wrote: > > > > > > > > > > > That is a completely fair point and I agree that from > > > > > > security > > > > > > perspective, the approach is safe enough. > > > > > > > > > > > > I'd just like to figure out what is the username in this > > > > > > case? > > > > > > Linux > > > > > > user id? Anything that comes from SASL layer based on the > > > > > > Hadoop > > > > > > stack? > > > > > > > > > > > > Andor > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, 2024-06-07 at 09:30 -0700, Andrew Purtell wrote: > > > > > > > Most users who would employ a mTLS authentication scheme > > > > > > > would > > > > > > > operate with this trust model. The fact the client has a > > > > > > > valid > > > > > > > signed > > > > > > > certificate means it can be trusted, and that trust > > > > > > > includes > > > > > > > supplied > > > > > > > connection metadata like username. Or, if not, then not. > > > > > > > So then a lot of security engineering effort goes in to > > > > > > > protecting > > > > > > > the trust established by certificate distribution, like > > > > > > > using > > > > > > > short > > > > > > > lived certs, and secure distribution methods. > > > > > > > > > > > > > > > On Jun 7, 2024, at 6:34 AM, Bryan Beaudreault < > > > > > > > > bbeaudrea...@apache.org> wrote: > > > > > > > > > > > > > > > > You're sort of correct. We've been using mTLS in prod > > > > > > > > for a > > > > > > > > while > > > > > > > > now, ever > > > > > > > > since the feature was committed. It's true that the > > > > > > > > actual > > > > > > > > HBase > > > > > > > > username > > > > > > > > is not verified with mTLS, however you still can > > > > > > > > authenticate > > > > > > > > the > > > > > > > > connection. The idea behind mTLS is that the certificate > > > > > > > > carries > > > > > > > > the > > > > > > > > authentication -- so a client will need a certificate > > > > > > > > which has > > > > > > > > been signed > > > > > > > > by the same CA (or at least within the CA chain) which > > > > > > > > signed > > > > > > > > the > > > > > > > > server's > > > > > > > > certificate, and vise versa. > > > > > > > > > > > > > > > > For us, if someone has a valid certificate and the mTLS > > > > > > > > authentication > > > > > > > > succeeds, then we just trust their username. Based on how > > > > > > > > we > > > > > > > > use > > > > > > > > HBase in > > > > > > > > our environment, this is perfectly secure for our use- > > > > > > > > case. > > > > > > > > That > > > > > > > > may not > > > > > > > > work for everyone, and I did file a jira to add a feature > > > > > > > > for > > > > > > > > validating > > > > > > > > the username (perhaps pulling from a custom certificate > > > > > > > > property). > > > > > > > > But I > > > > > > > > haven't actually implemented that, and not sure that I > > > > > > > > will > > > > > > > > since > > > > > > > > it works > > > > > > > > as-is for us. > > > > > > > > > > > > > > > > I'm on mobile now so I can't find it, but it should be > > > > > > > > findable > > > > > > > > in > > > > > > > > jira if > > > > > > > > you search the tls-related tickets > > > > > > > > > > > > > > > > > On Fri, Jun 7, 2024 at 8:53 AM Andor Molnar < > > > > > > > > > an...@apache.org > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > Hi Bryan / Hbase devs, > > > > > > > > > > > > > > > > > > Based on the changes when you added mTLS support in > > > > > > > > > HBASE- > > > > > > > > > 27280 > > > > > > > > > [1], > > > > > > > > > only the certificate and hostname verification part > > > > > > > > > were > > > > > > > > > added to > > > > > > > > > the > > > > > > > > > codebase. HBase doesn't actually authenticates the user > > > > > > > > > when > > > > > > > > > mTLS > > > > > > > > > is > > > > > > > > > being used. > > > > > > > > > > > > > > > > > > In other words some other auth method Simple or > > > > > > > > > Kerberos is > > > > > > > > > still > > > > > > > > > needed to identify the HBase user, because mTLS doesn't > > > > > > > > > extract > > > > > > > > > identity information from the client certificate and > > > > > > > > > doesn't > > > > > > > > > map > > > > > > > > > it to > > > > > > > > > an active HBase user. > > > > > > > > > > > > > > > > > > Is that correct? > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > Andor > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/HBASE-27280 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
