Hi Maulin, thanks for the discussion. As Harsha pointed out, to use the
KIP492, you need to create a new provider, register a *new* custom
algorithm for your keymanager and trustmanager factory implementations.
After this, the kafka server configuration can be done as given below
# Register the provider class with custom algorithm, say CUSTOM
security.provider.classes=com.company.security.CustomProvider
# Register the keymanager and trustmanager algorithms
# These algorithms indicate that the Keymanager and Trustmanagers
registered under the algorithm "CUSTOM" needs to be used
ssl.trustmanager.algorithm=CUSTOM
ssl.keymanager.algorithm=CUSTOM
And the customprovider looks like this...
public class CustomProvider extends Provider {
public CustomProvider() {
super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and TrustStore");
super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory");
super.put("TrustManagerFactory.CUSTOM",
"customTrustManagerFactory");
}
}
The PR for this is in review and can be found here:
https://github.com/apache/kafka/pull/7090
This PR includes the fixed insertProviderAt function call.
On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <ka...@harsha.io> wrote:
> Answers are in-line
>
>
>
> On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
> maulin.vasav...@gmail.com
> > wrote:
>
> > Hi Colin
> >
> >
> > When I refer to "standard" or "custom" algorithms I am following Java
> > security Provider Terminology. You can refer to
> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > StandardNames.html#TrustManagerFactory link I provided earlier in the
> > emails. It says PKIX is the default Algorithm for TrustManagerFactory.
> >
> >
> > 1. For SPIFFE, I am not sure why you are saying 'it does not implement
> > custom algorithms' because the following file clearly indicates that it
> > does use custom algorithm-
> >
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > provider/SpiffeProvider.java#L17
> >
> >
> > Algorithm value:
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > provider/SpiffeProviderConstants.java#L6
> >
> >
> > @Harsha do you want to chime in since you use that provider?
> >
>
> Here the "Custom" Algorithm is not an implementation per say , rather used
> to invoke the custom trust store factory and key manager factory. You are
> not moving away from "standard" alogrithms that are available.
>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/SpiffeTrustManager.java
> As you can see it delegates all the calls of verification of certificate to
> the default implementation available.
> So in our implementation we still use PKIX to verify the certificate
> chain. So you are not losing anything here and Spiffe is not
> reimplementing the verification process.
>
>
> 2. I already mentioned in my 3rd point, in my previous post, why using
> > ssl.provider does NOT work. I updated KIP-486 in "rejected alternatives"
> > also why ssl.provider does not work.
> >
>
> As mentioned before , provider is the right way to go. I am still not
> understanding the gap is.
> If I understand correctly your argument is , provider is going to ask to
> implement a custom algorithm.
> My answer to that is , no you are not re-implementing the algorithm. Please
> check the above link , TrustManager implementation it delegates the calls
> back. There is no need to implement your own here.
>
>
>
> 3. Security.insertProviderAt() comments were based on assumption if KIP-492
> > changes are done and we use that mechanism to configure providers instead
> > of ssl.provider configuration.
> >
>
> KIP-492 has patch available and going through review.
>
>
> Can you read my all the points, I mentioned in my previous post, very
> > carefully? I am covering all the aspects in explaining. I am open to
> still
> > discuss more to clarify any doubts.
> >
>
>
> "3. If we just use existing ssl.provider kafka configuration then our
> provider will be used in SSLContext.getInstance(protocol, provider) call in
> SslFactory.java <http://sslfactory.java/> and if our provider does not
> have
> implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we tested
> it). Example: In MyProvider sample above you see that I didn't add
> SSLContext.TLSv1 as
> "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
> don't have this challenge since you are planning to bypass ssl.provider as
> you mention in the KIP-492."
>
> Yes here you need to pass the protocol that your KeyManager/TrustManager
> registered with and in no way its deviating from TLS RFC spec.
>
> https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/javax/net/ssl/SSLContext.java#L134
>
>
> My suggestion here is for you to implement a simple Security Provider as
> you did before and register a Algorithm. You can use the existing
> implementation in SpiffeProvider and plug in changes where you need to
> retrieve the certificates from by making RPC call.
>
> Run an end-to-end test with Kafka broker coming up with your provider
> making calls to RPC call. You do need to pass the "custom algorithm" that
> you registered in your key manager to invoke the provider.
> I think your concern is this approach is replacing the existing known
> ciphersuites and certificate validation provided by java. Which its not.
>
> Now test the TLS connection you can do so via openssl -s_client options to
> test if everything else is working.
>
> I am happy to share configs that we used if you are interested.
>
> Thanks,
> Harsha
>
>
>
> > Thanks
> > Maulin
> >
> >
> > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cmcc...@apache.org> wrote:
> >
> > Hi Maulin,
> >
> > A lot of JSSE providers don't implement custom algorithms. Spire is a
> good
> > example of a JSSE provider that doesn't, and yet is still useful to many
> > people. Your JSSE provider can work fine even if it doesn't implement a
> > custom algorithm.
> >
> >
> > Maybe I'm missing something, but I don't understand the discussion of
> > Security.insertProviderAt() that you included. SslEngineBuilder doesn't
> use
> > that API to get the security provider. Instead, it calls
> > "SSLContext.getInstance(protocol, provider)", where provider is the name
> > of the provider.
> >
> >
> >
> > best,
> > Colin
> >
> >
> > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
> >
> > On top of everything above I feel strongly to add the 4th point which is
> > based on Java APIs for TrustManagerFactory.init(KeyStore) (
> >
> >
> > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > TrustManagerFactory.html#init(java.security.KeyStore)
> > )
> >
> >
> > and KeyManagerFactory.init(KeyStore, char[]) (
> >
> >
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory.
> > html#init(java.security.KeyStore,%20char[])
> >
> > ).
> >
> >
> > 4. The above APIs are intended to support providing "trust/key material"
> > from the user without having to write their own TrustManager/KeyManagers.
> > To quote from the TrustManagerFactory.init()'s documentation "Initializes
> > this factory with a source of certificate authorities and related trust
> > material."
> > To quote from the KeyManagerFactory.init()'s documentation "Initializes
> > this factory with a source of key material."
> >
> >
> > Based on this it is clear that there is a flexibility provided by Java to
> > to enable developers to provide the required trust/key material loaded
> >
> > from
> >
> > "anywhere" without requiring them to write custom provider OR trust/key
> > managers. This same flexibility is reflected in Kafka code also where it
> > loads the trust/keys from a local file and doesn't require writing a
> > Provider necessarily. If we do NOT have a custom algorithm, it makes less
> > sense to write a Provider.
> >
> >
> > Thanks
> > Maulin
> >
> >
> > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
> >
> > maulin.vasav...@gmail.com>
> >
> > wrote:
> >
> > Hi Harsha/Colin
> >
> > I did the sample with a custom Provider for TrustStoreManager and tried
> > using ssl.provider Kafka config AND the way KIP-492 is suggesting (by
> > adding Provider programmatically instead of relying on ssl.provider+java.
> > security. The below sample is followed by my detailed findings. I'll
> > appreciate if you can go through it carefully and see
> >
> > if you
> >
> > see my point.
> >
> > package providertest;
> >
> > import java.security.Provider <http://java.security.provider/>;
> >
> > public class MyProvider extends Provider {
> >
> >
> > private static final String name = "MyProvider"; private static double
> > version = 1.0d;
> > private static String info = "Maulin's SSL Provider v"+version;
> >
> >
> >
> > public MyProvider() {
> > super(name, version, info);
> > this.put("TrustManagerFactory.PKIX",
> >
> >
> > "providertest.MyTrustManagerFactory");
> >
> >
> > }
> > }
> >
> >
> > *Details:*
> >
> > KIP-492 documents that it will use Security.addProvider() assuming it
> >
> > will
> >
> >
> > add it as position '0' which is not a correct assumption. The
> > addProvider()'s documentation says it will add it to the last available
> > position. You may want to correct that to say
> > Security.insertProviderAt(provider, 1).
> >
> >
> > Now coming back to our specific discussion,
> >
> > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add
> >
> > that
> >
> > provider in the provider list via Security.addProvider() the position
> >
> > where
> >
> > it gets added doesn't matter (even if you don't end up adding it as
> >
> > first
> >
> > entry) since that is the ONLY provider for SPIFFE specific algorithm
> >
> > you
> >
> > might have.
> >
> > We do *not* have custom algorithm for Key/Trust StoreMangers. Which
> >
> > means
> >
> > we have to use X509, PKIX etc "Standard Algorithms" ((
> >
> >
> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > StandardNames.html
> > ))
> >
> >
> > in our provider to override the TrustStoreManager (see my sample code)
> >
> > and
> >
> > KeyStoreManger and KeyManager. This creates another challenge
> >
> > mentioned in
> >
> > the below point.
> >
> > 2. In order to make our Provider for loading custom TrustStore work, we
> > have to add the provider as 'first' in the list since there are others
> >
> > with
> >
> > the same algorithm.
> >
> >
> > However, the programatic way of adding provider
> > (Security.insertProviderAt()) is *not* deterministic for ordering since
> > different code can call the method for a different provider and
> >
> >
> > depending
> >
> > upon the order of the call our provider can be first or pushed down the
> > list. This can happen very well in any client application using Kafka.
> >
> > This
> >
> > is specially problematic for a case when you want to guarantee order
> >
> > for a
> >
> > Provider having "Standard Algorithms".
> >
> > If we add our provider in java.security file that definitely guarantees
> > the order(unless somebody calls removeProvider() which is less
> >
> > likely). But
> >
> > if we add our provider in java.security file it will defeat the
> >
> > purpose of
> >
> > the KIP-492.
> >
> > In the gist - Apache Kafka must not rely on "un-deterministic" method
> >
> > to
> >
> > rely on Provider ordering.
> >
> > 3. If we just use existing ssl.provider kafka configuration then our
> > provider will be used in SSLContext.getInstance(protocol, provider)
> >
> > call in
> >
> > SslFactory.java <http://sslfactory.java/> and if our provider does not
> > have implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we
> > tested it). Example:
> >
> > In
> >
> >
> > MyProvider sample above you see that I didn't add SSLContext.TLSv1 as
> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
> > don't have this challenge since you are planning to bypass
> >
> >
> > ssl.provider as
> >
> > you mention in the KIP-492.
> >
> > *Overall summary,*
> >
> > 1. Any provider based mechanisms- a) existing ssl.provider and
> >
> > b)KIP-492,
> >
> > for loading key/trust store using "Standard Algorithms" do not work
> >
> >
> > 2. Approach suggested in our KIP-486 works without any issue and it is
> > *not* our context specific solve
> >
> >
> > 3. Based on above we feel KIP-492 and KIP-486 are complimentary changes
> > and not contradicting or redundent.
> >
> > If you want we can do a joint session somehow to walk through the
> >
> > sample I
> >
> > have and various experiments I did. I would encourage you to do similar
> > exercise by writing a Provider for "Standard Algorithm" for
> > TrustStoreManager (like our needs) and see what you find since only
> >
> > writing
> >
> > samples can bring out the complexity/challenges we face.
> >
> >
> > Thanks
> > Maulin
> >
> >
> > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada < maulin.vasavada@gmail.
> > com> wrote:
> >
> > Just to update - still working on it. Get to work only on and off on
> >
> > it :(
> >
> > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
> >
> > maulin.vasav...@gmail.com>
> >
> > wrote:
> >
> > Hi Harsha
> >
> > Let me try to write samples and will let you know.
> >
> >
> > Thanks
> > Maulin
> >
> >
> > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <harsha...@gmail.com>
> >
> > wrote:
> >
> >
> > Hi Maulin,
> > With java security providers can be as custom you would
> >
> >
> > like
> >
> >
> > it to
> > be. If you only want to to implement a custom way of loading the keystore
> > and truststore and not implement any protocol/encryption handling
> > you can leave them empty and no need to implement. Have you looked into
> > the links I pasted before?
> >
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> > java
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > SpiffeTrustManager.java
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > java
> >
> >
> > Can you please tell me which methods are too complex in above to
> implement
> > or unnecessary? You are changing anything in SSL/TLS implementations
> > provided by
> >
> >
> >
> > All of the implementations delegating the checks to the default
> > implementation anyway.
> > Spire agent is an example, its nothing but a GRPC server listening
> >
> >
> > on a
> >
> >
> > unix domain socket . Above code is making a RPC call to the local daemon
> > to
> > get the certificate and keys. The mechanics are pretty much same as
> >
> >
> > what
> >
> > you are asking for.
> >
> >
> > Thanks,
> > Harsha
> >
> >
> >
> > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
> maulin.vasav...@gmail.com>
> >
> > wrote:
> >
> >
> > Imagine a scenario like - We know we have a custom KMS and as a
> >
> > Kafka
> >
> > owner
> >
> > we want to comply to using that KMS source to load keys/certs. As
> >
> > a
> >
> > Kafka
> >
> > owner we know how to integrate with KMS but doesn't necessarily
> >
> > have
> >
> > to
> >
> > know anything about cipher suites, algorithms, and SSL/TLS
> >
> > implementation.
> >
> > Going the Provider way requires to know lot more than we should,
> >
> > isn't it?
> >
> > Not that we would have concern/shy-away knowing those details -
> >
> > but
> >
> > if we
> >
> > don't have to - why should we?
> >
> > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
> >
> > maulin.vasav...@gmail.com>
> >
> > wrote:
> >
> > Hi Harsha
> >
> > We don't have spire (or similar) agents and we do not have
> >
> > keys/certs
> >
> > locally on any brokers. To elaborate more on my previous email,
> >
> > I agree that Java security Providers are used in much broader
> >
> > sense
> >
> > - to
> >
> > have a particular implementation of an algorithm, use specific
> >
> > cipher
> >
> > suites for SSL , OR in our current team's case have a
> >
> > particular
> >
> > way to
> >
> > leverage pre-generated SSL sessions. However, the scope of our
> >
> > KIP
> >
> > (486)
> >
> > is
> >
> > much restricted than that. We merely intend to provide a custom
> > keystore/truststore for our SSL connections and not really worry
> >
> > about
> >
> > underlying specific SSL/TLS implementation. This simplifies it
> >
> > a
> >
> > lot for
> >
> > us to keep the concerns separate and clear.
> >
> > I feel our approach is more complimentary such that it allows
> >
> > for
> >
> > using
> >
> > keystores of choice while retaining the flexibility to use any
> > underlying/available Provider for actually making the SSL call.
> >
> > We agree with KIP-492's approach based on Providers (and Java's
> > recommendation), but also strongly believe that our approach can
> >
> > compliment
> >
> > it very effectively for reasons explained above.
> >
> >
> > Thanks
> > Maulin
> >
> >
> > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
> >
> > ka...@harsha.io
> >
> > wrote:
> >
> > Hi Maulin,
> >
> >
> > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada < maulin.vasavada@gmail.
> > com>
> > wrote:
> >
> >
> > Hi Harsha
> >
> > The reason we rejected the SslProvider route is that - we
> >
> > only
> >
> > needed
> >
> > a
> >
> > custom way to load keys/certs. Not touch any policy that
> >
> > existing
> >
> > Providers
> >
> > govern like SunJSSE Provider.
> >
> > We have exactly the same requirements to load certs and keys
> >
> > through
> >
> > spire
> >
> > agent. We used security.provider to do that exactly. I am not
> >
> > sure
> >
> > why
> >
> > you
> >
> > would be modifying any policies provided by default SunJSSE
> >
> > provider.
> >
> > Can
> >
> > you give me an example of having custom provider that will
> >
> > override an
> >
> > existing policy in SunJSSE provider.
> >
> > As pointed out earlier, this kip
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> >
> >
> > allows
> > you to load security.provider through config.
> > Take a look at the examples I gave before
> >
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > java
> >
> > It registers KeyManagerFactory and TrustManagerFactory and
> >
> > Keystore
> >
> >
> > algorithm.
> > Implement your custom way of loading Keystore in here
> >
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> > java
> >
> > and Trust manager like here
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > SpiffeTrustManager.java
> >
> > In your Kafka client you can set the security.provider to your
> >
> > custom
> >
> >
> > implementation and with this fix
> > https://issues.apache.org/jira/browse/KAFKA-8191 you can set
> > keyManagerAlgorigthm and trustManagerAlgorithm configs.
> >
> >
> > All of this is in your clients and broker side and do not need
> >
> > to
> >
> > touch
> >
> >
> > any
> > policy changes at JVM level. You'll register the providers in
> >
> >
> > the
> >
> > priority
> >
> > order and can still have SunJSSE provider and have your custom
> >
> > provider
> >
> > to
> >
> > implement the key and trust managers.
> >
> > The ask here is different than KIP-492. We don't have any need
> >
> > to
> >
> > modify/specify the algorithm parameter. Does that make sense?
> >
> > The ask in KIP is introducing new interfaces where the KIP's
> > goal/motivation can be achieved through the security.provider
> >
> > and
> >
> > we
> >
> >
> > worked
> > on similar goal without touching any Keystore or Truststore
> >
> >
> > interfaces.
> >
> > My advise is against changing or introducing new interfaces
> >
> > when
> >
> > it can
> >
> > work through security.provider.
> >
> >
> > Thanks,
> > Harsha
> >
> >
> > Thanks
> >
> > Maulin
> >
> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
> >
> > ka...@harsha.io>
> >
> > wrote:
> >
> > In your KIP you added security. provider as rejected
> >
> > alternative
> >
> > and
> >
> > specified "its not the correct way". Do you mind explaining
> >
> > why
> >
> > its
> >
> > not? I
> >
> > didn't find any evidence in Java docs to say so. Contrary to
> >
> > your
> >
> > statement
> >
> >
> > it does say in the java docs
> > " However, please note that a provider can be used to
> >
> >
> > implement
> >
> > any
> >
> > security service in Java that uses a pluggable architecture
> >
> > with
> >
> > a
> >
> > choice
> >
> > of implementations that fit underneath."
> >
> > Java Security Providers have been used by other projects to
> >
> > provide
> >
> > such
> >
> > integration . I am not sure if you looked into Spiffe
> >
> > project to
> >
> > efficiently distribute certificates but here is an example of
> >
> > Java
> >
> > provider
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> >
> >
> > java which
> > obtains certificates from local daemons.
> > These integrations are being used in Tomcat, Jetty etc.. We
> >
> >
> > are
> >
> > also
> >
> > using
> >
> > Security provider to do the same in our Kafka clusters. So
> >
> > unless I
> >
> > see
> >
> > more evidence why security.provider doesn't work for you
> >
> > adding
> >
> > new
> >
> > interfaces while there exists more cleaner way of achieving
> >
> > the
> >
> > goals
> >
> > of
> >
> > this KIP is unnecessary and breaks the well known security
> >
> > interfaces
> >
> > provided by Java itself.
> >
> >
> > Thanks,
> > Harsha
> >
> >
> > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
> >
> > ka...@harsha.io
> >
> > wrote:
> >
> >
> > Hi Maulin,
> > Not sure if you looked at my previous replies. This
> >
> >
> > changes
> >
> > are not required as there is already security Provider to do
> >
> > what you
> >
> > are
> >
> > proposing. This KIP
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> >
> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> >
> > also
> >
> > addresses easy registration of such providers.
> >
> >
> > Thanks,
> > Harsha
> >
> >
> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
> >
> > <maulin.vasavada@gmail.
> >
> > com> wrote:
> >
> > Bump! Can somebody please review this?
> >
> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> >
> > maulin.vasav...@gmail.com>
> >
> > wrote:
> >
> > Bump! Can somebody please review this?
> >
> >
>
--
Thanks,
M.Sai Sandeep
