On top of everything above I feel strongly to add the 4th point which is
based on Java APIs for TrustManagerFactory.init(KeyStore) (
https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/TrustManagerFactory.html#init(java.security.KeyStore))
and KeyManagerFactory.init(KeyStore, char[]) (
https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory.html#init(java.security.KeyStore,%20char[])
).
4. The above APIs are intended to support providing "trust/key material"
from the user without having to write their own TrustManager/KeyManagers.
To quote from the TrustManagerFactory.init()'s documentation "Initializes
this factory with a source of certificate authorities and related trust
material."
To quote from the KeyManagerFactory.init()'s documentation "Initializes
this factory with a source of key material."
Based on this it is clear that there is a flexibility provided by Java to
to enable developers to provide the required trust/key material loaded from
"anywhere" without requiring them to write custom provider OR trust/key
managers. This same flexibility is reflected in Kafka code also where it
loads the trust/keys from a local file and doesn't require writing a
Provider necessarily. If we do NOT have a custom algorithm, it makes less
sense to write a Provider.
Thanks
Maulin
On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <maulin.vasav...@gmail.com>
wrote:
> Hi Harsha/Colin
>
> I did the sample with a custom Provider for TrustStoreManager and tried
> using ssl.provider Kafka config AND the way KIP-492 is suggesting (by
> adding Provider programmatically instead of relying on
> ssl.provider+java.security. The below sample is followed by my detailed
> findings. I'll appreciate if you can go through it carefully and see if you
> see my point.
>
> package providertest;
>
> import java.security.Provider;
>
> public class MyProvider extends Provider {
>
> private static final String name = "MyProvider";
> private static double version = 1.0d;
> private static String info = "Maulin's SSL Provider v"+version;
>
> public MyProvider() {
> super(name, version, info);
> this.put("TrustManagerFactory.PKIX",
> "providertest.MyTrustManagerFactory");
> }
> }
>
>
>
> *Details:*
>
> KIP-492 documents that it will use Security.addProvider() assuming it will
> add it as position '0' which is not a correct assumption. The
> addProvider()'s documentation says it will add it to the last available
> position. You may want to correct that to say
> Security.insertProviderAt(provider, 1).
>
> Now coming back to our specific discussion,
>
> 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add that
> provider in the provider list via Security.addProvider() the position where
> it gets added doesn't matter (even if you don't end up adding it as first
> entry) since that is the ONLY provider for SPIFFE specific algorithm you
> might have.
>
> We do *not* have custom algorithm for Key/Trust StoreMangers. Which means
> we have to use X509, PKIX etc "Standard Algorithms" ((
> https://docs.oracle.com/javase/7/docs/technotes/guides/security/StandardNames.html))
> in our provider to override the TrustStoreManager (see my sample code) and
> KeyStoreManger and KeyManager. This creates another challenge mentioned in
> the below point.
>
> 2. In order to make our Provider for loading custom TrustStore work, we
> have to add the provider as 'first' in the list since there are others with
> the same algorithm.
>
> However, the programatic way of adding provider
> (Security.insertProviderAt()) is *not* deterministic for ordering since
> different code can call the method for a different provider and depending
> upon the order of the call our provider can be first or pushed down the
> list. This can happen very well in any client application using Kafka. This
> is specially problematic for a case when you want to guarantee order for a
> Provider having "Standard Algorithms".
>
> If we add our provider in java.security file that definitely guarantees
> the order(unless somebody calls removeProvider() which is less likely). But
> if we add our provider in java.security file it will defeat the purpose of
> the KIP-492.
>
> In the gist - Apache Kafka must not rely on "un-deterministic" method to
> rely on Provider ordering.
>
> 3. If we just use existing ssl.provider kafka configuration then our
> provider will be used in SSLContext.getInstance(protocol, provider) call in
> SslFactory.java and if our provider does not have implementation for
> SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we tested it). Example: In
> MyProvider sample above you see that I didn't add SSLContext.TLSv1 as
> "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
> don't have this challenge since you are planning to bypass ssl.provider as
> you mention in the KIP-492.
>
>
> *Overall summary,*
>
> 1. Any provider based mechanisms- a) existing ssl.provider and b)KIP-492,
> for loading key/trust store using "Standard Algorithms" do not work
>
> 2. Approach suggested in our KIP-486 works without any issue and it is
> *not* our context specific solve
>
> 3. Based on above we feel KIP-492 and KIP-486 are complimentary changes
> and not contradicting or redundent.
>
> If you want we can do a joint session somehow to walk through the sample I
> have and various experiments I did. I would encourage you to do similar
> exercise by writing a Provider for "Standard Algorithm" for
> TrustStoreManager (like our needs) and see what you find since only writing
> samples can bring out the complexity/challenges we face.
>
> Thanks
> Maulin
>
> On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
> maulin.vasav...@gmail.com> wrote:
>
>> Just to update - still working on it. Get to work only on and off on it :(
>>
>> On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <maulin.vasav...@gmail.com>
>> wrote:
>>
>>> Hi Harsha
>>>
>>> Let me try to write samples and will let you know.
>>>
>>> Thanks
>>> Maulin
>>>
>>> On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <harsha...@gmail.com> wrote:
>>>
>>>> Hi Maulin,
>>>> With java security providers can be as custom you would like
>>>> it to
>>>> be. If you only want to to implement a custom way of loading the
>>>> keystore and truststore and not implement any protocol/encryption
>>>> handling
>>>> you can leave them empty and no need to implement.
>>>> Have you looked into the links I pasted before?
>>>>
>>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
>>>>
>>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
>>>>
>>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
>>>>
>>>> Can you please tell me which methods are too complex in above to
>>>> implement
>>>> or unnecessary? You are changing anything in SSL/TLS implementations
>>>> provided by
>>>>
>>>> All of the implementations delegating the checks to the default
>>>> implementation anyway.
>>>> Spire agent is an example, its nothing but a GRPC server listening on a
>>>> unix domain socket . Above code is making a RPC call to the local
>>>> daemon to
>>>> get the certificate and keys. The mechanics are pretty much same as what
>>>> you are asking for.
>>>>
>>>> Thanks,
>>>> Harsha
>>>>
>>>> On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
>>>> maulin.vasav...@gmail.com>
>>>> wrote:
>>>>
>>>> > Imagine a scenario like - We know we have a custom KMS and as a Kafka
>>>> owner
>>>> > we want to comply to using that KMS source to load keys/certs. As a
>>>> Kafka
>>>> > owner we know how to integrate with KMS but doesn't necessarily have
>>>> to
>>>> > know anything about cipher suites, algorithms, and SSL/TLS
>>>> implementation.
>>>> > Going the Provider way requires to know lot more than we should,
>>>> isn't it?
>>>> > Not that we would have concern/shy-away knowing those details - but
>>>> if we
>>>> > don't have to - why should we?
>>>> >
>>>> > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
>>>> maulin.vasav...@gmail.com>
>>>> > wrote:
>>>> >
>>>> > > Hi Harsha
>>>> > >
>>>> > > We don't have spire (or similar) agents and we do not have
>>>> keys/certs
>>>> > > locally on any brokers. To elaborate more on my previous email,
>>>> > >
>>>> > > I agree that Java security Providers are used in much broader sense
>>>> - to
>>>> > > have a particular implementation of an algorithm, use specific
>>>> cipher
>>>> > > suites for SSL , OR in our current team's case have a particular
>>>> way to
>>>> > > leverage pre-generated SSL sessions. However, the scope of our KIP
>>>> (486)
>>>> > is
>>>> > > much restricted than that. We merely intend to provide a custom
>>>> > > keystore/truststore for our SSL connections and not really worry
>>>> about
>>>> > > underlying specific SSL/TLS implementation. This simplifies it a
>>>> lot for
>>>> > > us to keep the concerns separate and clear.
>>>> > >
>>>> > > I feel our approach is more complimentary such that it allows for
>>>> using
>>>> > > keystores of choice while retaining the flexibility to use any
>>>> > > underlying/available Provider for actually making the SSL call.
>>>> > >
>>>> > > We agree with KIP-492's approach based on Providers (and Java's
>>>> > > recommendation), but also strongly believe that our approach can
>>>> > compliment
>>>> > > it very effectively for reasons explained above.
>>>> > >
>>>> > > Thanks
>>>> > > Maulin
>>>> > >
>>>> > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <ka...@harsha.io
>>>> >
>>>> > > wrote:
>>>> > >
>>>> > >> Hi Maulin,
>>>> > >>
>>>> > >> On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
>>>> > >> maulin.vasav...@gmail.com>
>>>> > >> wrote:
>>>> > >>
>>>> > >> > Hi Harsha
>>>> > >> >
>>>> > >> > The reason we rejected the SslProvider route is that - we only
>>>> needed
>>>> > a
>>>> > >> > custom way to load keys/certs. Not touch any policy that existing
>>>> > >> Providers
>>>> > >> > govern like SunJSSE Provider.
>>>> > >> >
>>>> > >>
>>>> > >> We have exactly the same requirements to load certs and keys
>>>> through
>>>> > spire
>>>> > >> agent. We used security.provider to do that exactly. I am not sure
>>>> why
>>>> > you
>>>> > >> would be modifying any policies provided by default SunJSSE
>>>> provider.
>>>> > Can
>>>> > >> you give me an example of having custom provider that will
>>>> override an
>>>> > >> existing policy in SunJSSE provider.
>>>> > >>
>>>> > >> As pointed out earlier, this kip
>>>> > >>
>>>> > >>
>>>> >
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>>>> > >> allows
>>>> > >> you to load security.provider through config.
>>>> > >> Take a look at the examples I gave before
>>>> > >>
>>>> > >>
>>>> >
>>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
>>>> > >> It registers KeyManagerFactory and TrustManagerFactory and Keystore
>>>> > >> algorithm.
>>>> > >> Implement your custom way of loading Keystore in here
>>>> > >>
>>>> > >>
>>>> >
>>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
>>>> > >>
>>>> > >> and Trust manager like here
>>>> > >>
>>>> > >>
>>>> >
>>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
>>>> > >>
>>>> > >> In your Kafka client you can set the security.provider to your
>>>> custom
>>>> > >> implementation and with this fix
>>>> > >> https://issues.apache.org/jira/browse/KAFKA-8191 you can set
>>>> > >> keyManagerAlgorigthm and trustManagerAlgorithm configs.
>>>> > >>
>>>> > >> All of this is in your clients and broker side and do not need to
>>>> touch
>>>> > >> any
>>>> > >> policy changes at JVM level. You'll register the providers in the
>>>> > priority
>>>> > >> order and can still have SunJSSE provider and have your custom
>>>> provider
>>>> > to
>>>> > >> implement the key and trust managers.
>>>> > >>
>>>> > >>
>>>> > >>
>>>> > >> The ask here is different than KIP-492. We don't have any need to
>>>> > >> > modify/specify the algorithm parameter. Does that make sense?
>>>> > >> >
>>>> > >>
>>>> > >> The ask in KIP is introducing new interfaces where the KIP's
>>>> > >> goal/motivation can be achieved through the security.provider and
>>>> we
>>>> > >> worked
>>>> > >> on similar goal without touching any Keystore or Truststore
>>>> interfaces.
>>>> > >> My advise is against changing or introducing new interfaces when
>>>> it can
>>>> > >> work through security.provider.
>>>> > >>
>>>> > >> Thanks,
>>>> > >> Harsha
>>>> > >>
>>>> > >>
>>>> > >> Thanks
>>>> > >> > Maulin
>>>> > >> >
>>>> > >> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
>>>> ka...@harsha.io>
>>>> > >> > wrote:
>>>> > >> >
>>>> > >> > In your KIP you added security. provider as rejected alternative
>>>> and
>>>> > >> > specified "its not the correct way". Do you mind explaining why
>>>> its
>>>> > >> not? I
>>>> > >> > didn't find any evidence in Java docs to say so. Contrary to your
>>>> > >> statement
>>>> > >> > it does say in the java docs
>>>> > >> > " However, please note that a provider can be used to implement
>>>> any
>>>> > >> > security service in Java that uses a pluggable architecture with
>>>> a
>>>> > >> choice
>>>> > >> > of implementations that fit underneath."
>>>> > >> >
>>>> > >> > Java Security Providers have been used by other projects to
>>>> provide
>>>> > such
>>>> > >> > integration . I am not sure if you looked into Spiffe project to
>>>> > >> > efficiently distribute certificates but here is an example of
>>>> Java
>>>> > >> provider
>>>> > >> >
>>>> > >> >
>>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>>>> > >> >
>>>> > >>
>>>> >
>>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>>>> > >> > java which
>>>> > >> > obtains certificates from local daemons.
>>>> > >> > These integrations are being used in Tomcat, Jetty etc.. We are
>>>> also
>>>> > >> using
>>>> > >> > Security provider to do the same in our Kafka clusters. So
>>>> unless I
>>>> > see
>>>> > >> > more evidence why security.provider doesn't work for you adding
>>>> new
>>>> > >> > interfaces while there exists more cleaner way of achieving the
>>>> goals
>>>> > of
>>>> > >> > this KIP is unnecessary and breaks the well known security
>>>> interfaces
>>>> > >> > provided by Java itself.
>>>> > >> >
>>>> > >> > Thanks,
>>>> > >> > Harsha
>>>> > >> >
>>>> > >> > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
>>>> ka...@harsha.io
>>>> > >
>>>> > >> > wrote:
>>>> > >> >
>>>> > >> > Hi Maulin,
>>>> > >> > Not sure if you looked at my previous replies. This
>>>> > >> >
>>>> > >> > changes
>>>> > >> >
>>>> > >> > are not required as there is already security Provider to do
>>>> what you
>>>> > >> are
>>>> > >> > proposing. This KIP
>>>> > https://cwiki.apache.org/confluence/display/KAFKA/
>>>> > >> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>>>> also
>>>> > >> > addresses easy registration of such providers.
>>>> > >> >
>>>> > >> > Thanks,
>>>> > >> > Harsha
>>>> > >> >
>>>> > >> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
>>>> > >> <maulin.vasavada@gmail.
>>>> > >> > com> wrote:
>>>> > >> >
>>>> > >> > Bump! Can somebody please review this?
>>>> > >> >
>>>> > >> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
>>>> > >> >
>>>> > >> > maulin.vasav...@gmail.com>
>>>> > >> >
>>>> > >> > wrote:
>>>> > >> >
>>>> > >> > Bump! Can somebody please review this?
>>>> > >> >
>>>> > >> >
>>>> > >>
>>>> > >
>>>> >
>>>>
>>>
