Hi Rajan Your email format doesn't show up correctly. Can you repost to make it more readable?
Thanks Maulin On Wed, Aug 28, 2019 at 2:32 PM Rajan Dhabalia <rdhaba...@apache.org> wrote: > *Hi Harsha/Maulin,I am following KIP-486 and KIP-492 and it seems > https://github.com/apache/kafka/pull/7090 > <https://github.com/apache/kafka/pull/7090> is the right solution when one > wants to register custom factory class for KeyManager and TrustManager. > User can easily configure custom implementation of TrustManager and > KeyManager using factory Provider class.Configuration of the provider is > also simple and straightforward: 1. Create custom provider which defines > factory classes for KeyManager and TrustManagerpublic class CustomProvider > extends Provider { public CustomProvider() > { super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and > TrustStore"); super.put("KeyManagerFactory.CUSTOM", > "customKeyManagerFactory"); > super.put("TrustManagerFactory.CUSTOM","customTrustManagerFactory"); > }} > 1. Register provider at broker > startupjava.security.Security.addProvider(new CustomProvider());However, > this approach is useful when one wants to implement custom TrustManager for > X509 certs by extending X509ExtendedKeyManager and implement various > abstract methods such as: checkClientTrusted, checkServerTrusted, etc..In > JDK, default implementation class of X509ExtendedKeyManager is > X509TrustManagerImpl and one can’t extend or delegate calls to this class > because it’s final and same is applicable for other available providers > such as : BouncyCastleProviderTurstManager/KeyManager mainly serve two > purposes: 1. Provide certs/key2. Perform validationX509TrustManagerImpl > performs various RFC specified validations.#7090 can be helpful when user > has both above asks. However, problem defined at KIP-486 has different ask > where user wants to provide certs/key without copying/implementing Manager > class because all the available Manager classes are final and can’t be > extended or delegated. And security team in most of the companies don’t > allow custom/copying provider in order to get up to date with various RFC > validations provided into standard jdk provider.Many users manage keys and > certs into KMS and sometimes it’s not feasible to copy them to file-system > instead directly use them from the memory. So, KIP-486 provides a custom > way to load keys/certs without implementing security-provider.Thanks,Rajan* > > On Wed, Aug 28, 2019 at 2:18 PM Maulin Vasavada <maulin.vasav...@gmail.com > > > wrote: > > > > > > > Hi Harsha > > > > As we synced-up offline on this topic, we hope you don't have any more > > clarifications that you are seeking. If that is the case, can you please > > help us move this forward and discuss what changes you would expect on > the > > KIP design in order to make it valuable contribution? > > > > Just FYI - we verified our primary design change with the author of Sun's > > X509 Trustmanager implementation and the outcome is that what we are > > proposing makes sense at the heart of it - "Instead of writing > TrustManager > > just plugin the Trust store". We are open to discuss additional changes > > that you/anybody else would like to see on the functionality however. > > > > > > Thanks > > Maulin > > > > On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada < > maulin.vasav...@gmail.com> > > wrote: > > > >> Hi Harsha > >> > >> Any response on my question? I feel this KIP is worth accommodating. > Your > >> help is much appreciated. > >> > >> Thanks > >> Maulin > >> > >> On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada < > >> maulin.vasav...@gmail.com> wrote: > >> > >>> Hi Harsha > >>> > >>> I've examined the SPIFFE provider more and have one question - > >>> > >>> If SPIFFE didn't have a need to do checkSpiffeId() call at the below > >>> location, would you really still write the Provider? *OR* > >>> Would you just use TrustManagerFactory.init(KeyStore) signature to pass > >>> the KeyStore from set of certs returned by spiffeIdManager. > >>> getTrustedCerts()? > >>> > >>> > >>> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/CertificateUtils.java#L100 > >>> > >>> > >>> /** > >>>> * Validates that the SPIFFE ID is present and matches the SPIFFE ID > >>>> configured in > >>>> * the java.security property ssl.spiffe.accept > >>>> * > >>>> * If the authorized spiffe ids list is empty any spiffe id is > authorized > >>>> * > >>>> * @param chain an array of X509Certificate that contains the Peer's > >>>> SVID to be validated > >>>> * @throws CertificateException when either the certificates doesn't > >>>> have a SPIFFE ID or the SPIFFE ID is not authorized > >>>> */ > >>>> static void checkSpiffeId(X509Certificate[] chain) throws > >>>> CertificateException { > >>> > >>> > >>> > >>> Thanks > >>> Maulin > >>> > >>> > >>> On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <ka...@harsha.io> > >>> wrote: > >>> > >>>> Maulin, > >>>> The code parts you are pointing are specific for Spiffe > >>>> and if > >>>> you are talking about validate method which uses PKIX check like any > >>>> other provider does. > >>>> If you want to default to SunJSSE everywhere you can do so by > delegating > >>>> the calls in these methods to SunJSSE provider. > >>>> > >>>> TrustManagerFactory tmf = TrustManagerFactory > >>>> .getInstance(TrustManagerFactory.getDefaultAlgorithm());and use > >>>> tmf.chekServerTrusted() > >>>> or use > >>>> > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/TrustManagerFactory.html#getInstance(java.lang.String)if > >>>> you want a specific provider. > >>>> > >>>> -Harsha > >>>> > >>>> > >>>> On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada < > >>>> maulin.vasav...@gmail.com> > >>>> wrote: > >>>> > >>>> > Okay, so I take that you guys agree that I have to write a 'custom' > >>>> > algorithm and a provider to make it work , correct? > >>>> > > >>>> > Now, for Harsha's comment "Here the 'Custom' Algorithm is not an > >>>> > implementation per say , ..." , I diagree. You can refer to https:// > >>>> > > >>>> > github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/ > >>>> > SpiffeTrustManager.java#L91 and > >>>> > > >>>> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/ > >>>> > provider/CertificateUtils.java#L100 > >>>> > > >>>> > "that code" is the customization you have for the custom way to > check > >>>> > something on top of regular checks. That method is NOT doing custom > >>>> > truststore loading. It is validating/verifying something in the > >>>> "custom" > >>>> > way with spiffeId. > >>>> > I bet that without that you won't have a need of the custom > algorithm > >>>> in > >>>> > the first place. > >>>> > > >>>> > Let me know if you agree to this. > >>>> > > >>>> > Thanks > >>>> > Maulin > >>>> > > >>>> > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <mpr...@gmail.com> > >>>> wrote: > >>>> > > >>>> > Hi Maulin, thanks for the discussion. As Harsha pointed out, to use > >>>> the > >>>> > KIP492, you need to create a new provider, register a *new* custom > >>>> > algorithm for your keymanager and trustmanager factory > >>>> implementations. > >>>> > After this, the kafka server configuration can be done as given > below > >>>> > > >>>> > # Register the provider class with custom algorithm, say CUSTOM > >>>> security. > >>>> > provider.classes=com.company.security.CustomProvider > >>>> > <http://security.provider.classes > >>>> =com.company.security.customprovider/> > >>>> > > >>>> > # Register the keymanager and trustmanager algorithms > >>>> > # These algorithms indicate that the Keymanager and Trustmanagers > >>>> > registered under the algorithm "CUSTOM" needs to be used > >>>> > ssl.trustmanager.algorithm=CUSTOM > >>>> > ssl.keymanager.algorithm=CUSTOM > >>>> > > >>>> > And the customprovider looks like this... > >>>> > > >>>> > public class CustomProvider extends Provider { > >>>> > public CustomProvider() { > >>>> > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and TrustStore"); > >>>> > super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory"); > >>>> > super.put("TrustManagerFactory.CUSTOM", > >>>> > "customTrustManagerFactory"); > >>>> > } > >>>> > } > >>>> > > >>>> > The PR for this is in review and can be found here: > >>>> https://github.com/ > >>>> > apache/kafka/pull/7090 > >>>> > This PR includes the fixed insertProviderAt function call. > >>>> > > >>>> > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani < > ka...@harsha.io> > >>>> > wrote: > >>>> > > >>>> > Answers are in-line > >>>> > > >>>> > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada < > >>>> maulin.vasavada@gmail. > >>>> > com > >>>> > > >>>> > wrote: > >>>> > > >>>> > Hi Colin > >>>> > > >>>> > When I refer to "standard" or "custom" algorithms I am following > Java > >>>> > security Provider Terminology. You can refer to > >>>> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/ > >>>> > StandardNames.html#TrustManagerFactory link I provided earlier in > the > >>>> > emails. It says PKIX is the default Algorithm for > TrustManagerFactory. > >>>> > > >>>> > 1. For SPIFFE, I am not sure why you are saying 'it does not > implement > >>>> > custom algorithms' because the following file clearly indicates that > >>>> it > >>>> > does use custom algorithm- > >>>> > > >>>> > > >>>> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/ > >>>> > > >>>> > provider/SpiffeProvider.java#L17 > >>>> > > >>>> > Algorithm value: > >>>> > > >>>> > > >>>> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/ > >>>> > > >>>> > provider/SpiffeProviderConstants.java#L6 > >>>> > > >>>> > @Harsha do you want to chime in since you use that provider? > >>>> > > >>>> > Here the "Custom" Algorithm is not an implementation per say , > rather > >>>> > > >>>> > used > >>>> > > >>>> > to invoke the custom trust store factory and key manager factory. > You > >>>> are > >>>> > not moving away from "standard" alogrithms that are available. > >>>> > > >>>> > > >>>> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/ > >>>> > provider/SpiffeTrustManager.java > >>>> > > >>>> > As you can see it delegates all the calls of verification of > >>>> certificate > >>>> > > >>>> > to > >>>> > > >>>> > the default implementation available. > >>>> > So in our implementation we still use PKIX to verify the certificate > >>>> > chain. So you are not losing anything here and Spiffe is not > >>>> reimplementing > >>>> > the verification process. > >>>> > > >>>> > 2. I already mentioned in my 3rd point, in my previous post, why > using > >>>> > > >>>> > ssl.provider does NOT work. I updated KIP-486 in "rejected > >>>> > > >>>> > alternatives" > >>>> > > >>>> > also why ssl.provider does not work. > >>>> > > >>>> > As mentioned before , provider is the right way to go. I am still > not > >>>> > understanding the gap is. > >>>> > If I understand correctly your argument is , provider is going to > ask > >>>> to > >>>> > implement a custom algorithm. > >>>> > My answer to that is , no you are not re-implementing the algorithm. > >>>> > > >>>> > Please > >>>> > > >>>> > check the above link , TrustManager implementation it delegates the > >>>> calls > >>>> > back. There is no need to implement your own here. > >>>> > > >>>> > 3. Security.insertProviderAt() comments were based on assumption if > >>>> > > >>>> > KIP-492 > >>>> > > >>>> > changes are done and we use that mechanism to configure providers > >>>> > > >>>> > instead > >>>> > > >>>> > of ssl.provider configuration. > >>>> > > >>>> > KIP-492 has patch available and going through review. > >>>> > > >>>> > Can you read my all the points, I mentioned in my previous post, > very > >>>> > > >>>> > carefully? I am covering all the aspects in explaining. I am open to > >>>> > > >>>> > still > >>>> > > >>>> > discuss more to clarify any doubts. > >>>> > > >>>> > "3. If we just use existing ssl.provider kafka configuration then > our > >>>> > provider will be used in SSLContext.getInstance(protocol, provider) > >>>> call > >>>> > > >>>> > in > >>>> > > >>>> > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/> > >>>> and > >>>> > if our provider does not have > >>>> > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we > >>>> > > >>>> > tested > >>>> > > >>>> > it). Example: In MyProvider sample above you see that I didn't add > >>>> > SSLContext.TLSv1 as > >>>> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider > >>>> you > >>>> > don't have this challenge since you are planning to bypass > >>>> ssl.provider > >>>> > > >>>> > as > >>>> > > >>>> > you mention in the KIP-492." > >>>> > > >>>> > Yes here you need to pass the protocol that your > >>>> KeyManager/TrustManager > >>>> > registered with and in no way its deviating from TLS RFC spec. > >>>> > > >>>> > > >>>> > https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/ > >>>> > javax/net/ssl/SSLContext.java#L134 > >>>> > > >>>> > My suggestion here is for you to implement a simple Security > Provider > >>>> as > >>>> > you did before and register a Algorithm. You can use the existing > >>>> > implementation in SpiffeProvider and plug in changes where you need > to > >>>> > retrieve the certificates from by making RPC call. > >>>> > > >>>> > Run an end-to-end test with Kafka broker coming up with your > provider > >>>> > making calls to RPC call. You do need to pass the "custom algorithm" > >>>> that > >>>> > you registered in your key manager to invoke the provider. I think > >>>> your > >>>> > concern is this approach is replacing the existing known > ciphersuites > >>>> and > >>>> > certificate validation provided by java. Which its not. > >>>> > > >>>> > Now test the TLS connection you can do so via openssl -s_client > >>>> options > >>>> > > >>>> > to > >>>> > > >>>> > test if everything else is working. > >>>> > > >>>> > I am happy to share configs that we used if you are interested. > >>>> > > >>>> > Thanks, > >>>> > Harsha > >>>> > > >>>> > Thanks > >>>> > Maulin > >>>> > > >>>> > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cmcc...@apache.org> > >>>> > > >>>> > wrote: > >>>> > > >>>> > Hi Maulin, > >>>> > > >>>> > A lot of JSSE providers don't implement custom algorithms. Spire is > a > >>>> > > >>>> > good > >>>> > > >>>> > example of a JSSE provider that doesn't, and yet is still useful to > >>>> > > >>>> > many > >>>> > > >>>> > people. Your JSSE provider can work fine even if it doesn't > implement > >>>> a > >>>> > custom algorithm. > >>>> > > >>>> > Maybe I'm missing something, but I don't understand the discussion > of > >>>> > Security.insertProviderAt() that you included. SslEngineBuilder > >>>> doesn't > >>>> > > >>>> > use > >>>> > > >>>> > that API to get the security provider. Instead, it calls > >>>> > "SSLContext.getInstance(protocol, provider)", where provider is the > >>>> > > >>>> > name > >>>> > > >>>> > of the provider. > >>>> > > >>>> > best, > >>>> > Colin > >>>> > > >>>> > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote: > >>>> > > >>>> > On top of everything above I feel strongly to add the 4th point > which > >>>> > > >>>> > is > >>>> > > >>>> > based on Java APIs for TrustManagerFactory.init(KeyStore) ( > >>>> > > >>>> > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/ > >>>> > TrustManagerFactory.html#init(java.security.KeyStore > >>>> > <http://java.security.keystore/>) > >>>> > ) > >>>> > > >>>> > and KeyManagerFactory.init(KeyStore, char[]) ( > >>>> > > >>>> > > >>>> > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory > >>>> . > >>>> > > >>>> > > >>>> > html#init(java.security.KeyStore <http://java.security.keystore/ > >>>> >,%20char[]) > >>>> > > >>>> > > >>>> > ). > >>>> > > >>>> > 4. The above APIs are intended to support providing "trust/key > >>>> > > >>>> > material" > >>>> > > >>>> > from the user without having to write their own > >>>> > > >>>> > TrustManager/KeyManagers. > >>>> > > >>>> > To quote from the TrustManagerFactory.init()'s documentation > >>>> > > >>>> > "Initializes > >>>> > > >>>> > this factory with a source of certificate authorities and related > >>>> trust > >>>> > material." > >>>> > To quote from the KeyManagerFactory.init()'s documentation > >>>> "Initializes > >>>> > this factory with a source of key material." > >>>> > > >>>> > Based on this it is clear that there is a flexibility provided by > Java > >>>> > > >>>> > to > >>>> > > >>>> > to enable developers to provide the required trust/key material > loaded > >>>> > > >>>> > from > >>>> > > >>>> > "anywhere" without requiring them to write custom provider OR > >>>> trust/key > >>>> > managers. This same flexibility is reflected in Kafka code also > where > >>>> > > >>>> > it > >>>> > > >>>> > loads the trust/keys from a local file and doesn't require writing a > >>>> > Provider necessarily. If we do NOT have a custom algorithm, it makes > >>>> > > >>>> > less > >>>> > > >>>> > sense to write a Provider. > >>>> > > >>>> > Thanks > >>>> > Maulin > >>>> > > >>>> > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada < > >>>> > > >>>> > maulin.vasav...@gmail.com> > >>>> > > >>>> > wrote: > >>>> > > >>>> > Hi Harsha/Colin > >>>> > > >>>> > I did the sample with a custom Provider for TrustStoreManager and > >>>> tried > >>>> > using ssl.provider Kafka config AND the way KIP-492 is suggesting > (by > >>>> > adding Provider programmatically instead of relying on > >>>> > > >>>> > ssl.provider+java. > >>>> > > >>>> > security. The below sample is followed by my detailed findings. I'll > >>>> > appreciate if you can go through it carefully and see > >>>> > > >>>> > if you > >>>> > > >>>> > see my point. > >>>> > > >>>> > package providertest; > >>>> > > >>>> > import java.security.Provider <http://java.security.provider/> > >>>> <http:// > >>>> > java.security.provider/>; > >>>> > > >>>> > public class MyProvider extends Provider { > >>>> > > >>>> > private static final String name = "MyProvider"; private static > double > >>>> > version = 1.0d; > >>>> > private static String info = "Maulin's SSL Provider v"+version; > >>>> > > >>>> > public MyProvider() { > >>>> > super(name, version, info); > >>>> > this.put("TrustManagerFactory.PKIX", > >>>> > > >>>> > "providertest.MyTrustManagerFactory"); > >>>> > > >>>> > } > >>>> > } > >>>> > > >>>> > *Details:* > >>>> > > >>>> > KIP-492 documents that it will use Security.addProvider() assuming > it > >>>> > > >>>> > will > >>>> > > >>>> > add it as position '0' which is not a correct assumption. The > >>>> > addProvider()'s documentation says it will add it to the last > >>>> available > >>>> > position. You may want to correct that to say > >>>> > Security.insertProviderAt(provider, 1). > >>>> > > >>>> > Now coming back to our specific discussion, > >>>> > > >>>> > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add > >>>> > > >>>> > that > >>>> > > >>>> > provider in the provider list via Security.addProvider() the > position > >>>> > > >>>> > where > >>>> > > >>>> > it gets added doesn't matter (even if you don't end up adding it as > >>>> > > >>>> > first > >>>> > > >>>> > entry) since that is the ONLY provider for SPIFFE specific algorithm > >>>> > > >>>> > you > >>>> > > >>>> > might have. > >>>> > > >>>> > We do *not* have custom algorithm for Key/Trust StoreMangers. Which > >>>> > > >>>> > means > >>>> > > >>>> > we have to use X509, PKIX etc "Standard Algorithms" (( > >>>> > > >>>> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/ > >>>> > StandardNames.html > >>>> > )) > >>>> > > >>>> > in our provider to override the TrustStoreManager (see my sample > code) > >>>> > > >>>> > and > >>>> > > >>>> > KeyStoreManger and KeyManager. This creates another challenge > >>>> > > >>>> > mentioned in > >>>> > > >>>> > the below point. > >>>> > > >>>> > 2. In order to make our Provider for loading custom TrustStore work, > >>>> we > >>>> > have to add the provider as 'first' in the list since there are > others > >>>> > > >>>> > with > >>>> > > >>>> > the same algorithm. > >>>> > > >>>> > However, the programatic way of adding provider > >>>> > (Security.insertProviderAt()) is *not* deterministic for ordering > >>>> since > >>>> > different code can call the method for a different provider and > >>>> > > >>>> > depending > >>>> > > >>>> > upon the order of the call our provider can be first or pushed down > >>>> the > >>>> > list. This can happen very well in any client application using > Kafka. > >>>> > > >>>> > This > >>>> > > >>>> > is specially problematic for a case when you want to guarantee order > >>>> > > >>>> > for a > >>>> > > >>>> > Provider having "Standard Algorithms". > >>>> > > >>>> > If we add our provider in java.security file that definitely > >>>> guarantees > >>>> > the order(unless somebody calls removeProvider() which is less > >>>> > > >>>> > likely). But > >>>> > > >>>> > if we add our provider in java.security file it will defeat the > >>>> > > >>>> > purpose of > >>>> > > >>>> > the KIP-492. > >>>> > > >>>> > In the gist - Apache Kafka must not rely on "un-deterministic" > method > >>>> > > >>>> > to > >>>> > > >>>> > rely on Provider ordering. > >>>> > > >>>> > 3. If we just use existing ssl.provider kafka configuration then our > >>>> > provider will be used in SSLContext.getInstance(protocol, provider) > >>>> > > >>>> > call in > >>>> > > >>>> > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/> > >>>> and > >>>> > if our provider does not have implementation for > >>>> > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks > >>>> > > >>>> > (we > >>>> > > >>>> > tested it). Example: > >>>> > > >>>> > In > >>>> > > >>>> > MyProvider sample above you see that I didn't add SSLContext.TLSv1 > as > >>>> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider > >>>> you > >>>> > don't have this challenge since you are planning to bypass > >>>> > > >>>> > ssl.provider as > >>>> > > >>>> > you mention in the KIP-492. > >>>> > > >>>> > *Overall summary,* > >>>> > > >>>> > 1. Any provider based mechanisms- a) existing ssl.provider and > >>>> > > >>>> > b)KIP-492, > >>>> > > >>>> > for loading key/trust store using "Standard Algorithms" do not work > >>>> > > >>>> > 2. Approach suggested in our KIP-486 works without any issue and it > is > >>>> > *not* our context specific solve > >>>> > > >>>> > 3. Based on above we feel KIP-492 and KIP-486 are complimentary > >>>> changes > >>>> > and not contradicting or redundent. > >>>> > > >>>> > If you want we can do a joint session somehow to walk through the > >>>> > > >>>> > sample I > >>>> > > >>>> > have and various experiments I did. I would encourage you to do > >>>> similar > >>>> > exercise by writing a Provider for "Standard Algorithm" for > >>>> > TrustStoreManager (like our needs) and see what you find since only > >>>> > > >>>> > writing > >>>> > > >>>> > samples can bring out the complexity/challenges we face. > >>>> > > >>>> > Thanks > >>>> > Maulin > >>>> > > >>>> > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada < > >>>> > > >>>> > maulin.vasavada@gmail. > >>>> > > >>>> > com> wrote: > >>>> > > >>>> > Just to update - still working on it. Get to work only on and off on > >>>> > > >>>> > it :( > >>>> > > >>>> > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada < > >>>> > > >>>> > maulin.vasav...@gmail.com> > >>>> > > >>>> > wrote: > >>>> > > >>>> > Hi Harsha > >>>> > > >>>> > Let me try to write samples and will let you know. > >>>> > > >>>> > Thanks > >>>> > Maulin > >>>> > > >>>> > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <harsha...@gmail.com> > >>>> > > >>>> > wrote: > >>>> > > >>>> > Hi Maulin, > >>>> > With java security providers can be as custom you would > >>>> > > >>>> > like > >>>> > > >>>> > it to > >>>> > be. If you only want to to implement a custom way of loading the > >>>> > > >>>> > keystore > >>>> > > >>>> > and truststore and not implement any protocol/encryption handling > you > >>>> can > >>>> > leave them empty and no need to implement. Have you looked into the > >>>> links I > >>>> > pasted before? > >>>> > > >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ > >>>> > > >>>> > > >>>> > spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore. > >>>> > > >>>> > java > >>>> > > >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ > >>>> > spiffe-security-provider/src/main/java/spiffe/api/provider/ > >>>> > SpiffeTrustManager.java <http://spiffetrustmanager.java/> > >>>> > > >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ > >>>> > > >>>> > > >>>> > spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider. > >>>> > > >>>> > java > >>>> > > >>>> > Can you please tell me which methods are too complex in above to > >>>> > > >>>> > implement > >>>> > > >>>> > or unnecessary? You are changing anything in SSL/TLS implementations > >>>> > provided by > >>>> > > >>>> > All of the implementations delegating the checks to the default > >>>> > implementation anyway. > >>>> > Spire agent is an example, its nothing but a GRPC server listening > >>>> > > >>>> > on a > >>>> > > >>>> > unix domain socket . Above code is making a RPC call to the local > >>>> > > >>>> > daemon > >>>> > > >>>> > to > >>>> > get the certificate and keys. The mechanics are pretty much same as > >>>> > > >>>> > what > >>>> > > >>>> > you are asking for. > >>>> > > >>>> > Thanks, > >>>> > Harsha > >>>> > > >>>> > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada < > >>>> > > >>>> > maulin.vasav...@gmail.com> > >>>> > > >>>> > wrote: > >>>> > > >>>> > Imagine a scenario like - We know we have a custom KMS and as a > >>>> > > >>>> > Kafka > >>>> > > >>>> > owner > >>>> > > >>>> > we want to comply to using that KMS source to load keys/certs. As > >>>> > > >>>> > a > >>>> > > >>>> > Kafka > >>>> > > >>>> > owner we know how to integrate with KMS but doesn't necessarily > >>>> > > >>>> > have > >>>> > > >>>> > to > >>>> > > >>>> > know anything about cipher suites, algorithms, and SSL/TLS > >>>> > > >>>> > implementation. > >>>> > > >>>> > Going the Provider way requires to know lot more than we should, > >>>> > > >>>> > isn't it? > >>>> > > >>>> > Not that we would have concern/shy-away knowing those details - > >>>> > > >>>> > but > >>>> > > >>>> > if we > >>>> > > >>>> > don't have to - why should we? > >>>> > > >>>> > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada < > >>>> > > >>>> > maulin.vasav...@gmail.com> > >>>> > > >>>> > wrote: > >>>> > > >>>> > Hi Harsha > >>>> > > >>>> > We don't have spire (or similar) agents and we do not have > >>>> > > >>>> > keys/certs > >>>> > > >>>> > locally on any brokers. To elaborate more on my previous email, > >>>> > > >>>> > I agree that Java security Providers are used in much broader > >>>> > > >>>> > sense > >>>> > > >>>> > - to > >>>> > > >>>> > have a particular implementation of an algorithm, use specific > >>>> > > >>>> > cipher > >>>> > > >>>> > suites for SSL , OR in our current team's case have a > >>>> > > >>>> > particular > >>>> > > >>>> > way to > >>>> > > >>>> > leverage pre-generated SSL sessions. However, the scope of our > >>>> > > >>>> > KIP > >>>> > > >>>> > (486) > >>>> > > >>>> > is > >>>> > > >>>> > much restricted than that. We merely intend to provide a custom > >>>> > keystore/truststore for our SSL connections and not really worry > >>>> > > >>>> > about > >>>> > > >>>> > underlying specific SSL/TLS implementation. This simplifies it > >>>> > > >>>> > a > >>>> > > >>>> > lot for > >>>> > > >>>> > us to keep the concerns separate and clear. > >>>> > > >>>> > I feel our approach is more complimentary such that it allows > >>>> > > >>>> > for > >>>> > > >>>> > using > >>>> > > >>>> > keystores of choice while retaining the flexibility to use any > >>>> > underlying/available Provider for actually making the SSL call. > >>>> > > >>>> > We agree with KIP-492's approach based on Providers (and Java's > >>>> > recommendation), but also strongly believe that our approach can > >>>> > > >>>> > compliment > >>>> > > >>>> > it very effectively for reasons explained above. > >>>> > > >>>> > Thanks > >>>> > Maulin > >>>> > > >>>> > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani < > >>>> > > >>>> > ka...@harsha.io > >>>> > > >>>> > wrote: > >>>> > > >>>> > Hi Maulin, > >>>> > > >>>> > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada < > >>>> > > >>>> > maulin.vasavada@gmail. > >>>> > > >>>> > com> > >>>> > wrote: > >>>> > > >>>> > Hi Harsha > >>>> > > >>>> > The reason we rejected the SslProvider route is that - we > >>>> > > >>>> > only > >>>> > > >>>> > needed > >>>> > > >>>> > a > >>>> > > >>>> > custom way to load keys/certs. Not touch any policy that > >>>> > > >>>> > existing > >>>> > > >>>> > Providers > >>>> > > >>>> > govern like SunJSSE Provider. > >>>> > > >>>> > We have exactly the same requirements to load certs and keys > >>>> > > >>>> > through > >>>> > > >>>> > spire > >>>> > > >>>> > agent. We used security.provider to do that exactly. I am not > >>>> > > >>>> > sure > >>>> > > >>>> > why > >>>> > > >>>> > you > >>>> > > >>>> > would be modifying any policies provided by default SunJSSE > >>>> > > >>>> > provider. > >>>> > > >>>> > Can > >>>> > > >>>> > you give me an example of having custom provider that will > >>>> > > >>>> > override an > >>>> > > >>>> > existing policy in SunJSSE provider. > >>>> > > >>>> > As pointed out earlier, this kip > >>>> > > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/ > >>>> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config > >>>> > > >>>> > allows > >>>> > you to load security.provider through config. > >>>> > Take a look at the examples I gave before > >>>> > > >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ > >>>> > > >>>> > > >>>> > spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider. > >>>> > > >>>> > java > >>>> > > >>>> > It registers KeyManagerFactory and TrustManagerFactory and > >>>> > > >>>> > Keystore > >>>> > > >>>> > algorithm. > >>>> > Implement your custom way of loading Keystore in here > >>>> > > >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ > >>>> > > >>>> > > >>>> > spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore. > >>>> > > >>>> > java > >>>> > > >>>> > and Trust manager like here > >>>> > > >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ > >>>> > spiffe-security-provider/src/main/java/spiffe/api/provider/ > >>>> > SpiffeTrustManager.java <http://spiffetrustmanager.java/> > >>>> > > >>>> > In your Kafka client you can set the security.provider to your > >>>> > > >>>> > custom > >>>> > > >>>> > implementation and with this fix > >>>> > https://issues.apache.org/jira/browse/KAFKA-8191 you can set > >>>> > keyManagerAlgorigthm and trustManagerAlgorithm configs. > >>>> > > >>>> > All of this is in your clients and broker side and do not need > >>>> > > >>>> > to > >>>> > > >>>> > touch > >>>> > > >>>> > any > >>>> > policy changes at JVM level. You'll register the providers in > >>>> > > >>>> > the > >>>> > > >>>> > priority > >>>> > > >>>> > order and can still have SunJSSE provider and have your custom > >>>> > > >>>> > provider > >>>> > > >>>> > to > >>>> > > >>>> > implement the key and trust managers. > >>>> > > >>>> > The ask here is different than KIP-492. We don't have any need > >>>> > > >>>> > to > >>>> > > >>>> > modify/specify the algorithm parameter. Does that make sense? > >>>> > > >>>> > The ask in KIP is introducing new interfaces where the KIP's > >>>> > goal/motivation can be achieved through the security.provider > >>>> > > >>>> > and > >>>> > > >>>> > we > >>>> > > >>>> > worked > >>>> > on similar goal without touching any Keystore or Truststore > >>>> > > >>>> > interfaces. > >>>> > > >>>> > My advise is against changing or introducing new interfaces > >>>> > > >>>> > when > >>>> > > >>>> > it can > >>>> > > >>>> > work through security.provider. > >>>> > > >>>> > Thanks, > >>>> > Harsha > >>>> > > >>>> > Thanks > >>>> > > >>>> > Maulin > >>>> > > >>>> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani < > >>>> > > >>>> > ka...@harsha.io> > >>>> > > >>>> > wrote: > >>>> > > >>>> > In your KIP you added security. provider as rejected > >>>> > > >>>> > alternative > >>>> > > >>>> > and > >>>> > > >>>> > specified "its not the correct way". Do you mind explaining > >>>> > > >>>> > why > >>>> > > >>>> > its > >>>> > > >>>> > not? I > >>>> > > >>>> > didn't find any evidence in Java docs to say so. Contrary to > >>>> > > >>>> > your > >>>> > > >>>> > statement > >>>> > > >>>> > it does say in the java docs > >>>> > " However, please note that a provider can be used to > >>>> > > >>>> > implement > >>>> > > >>>> > any > >>>> > > >>>> > security service in Java that uses a pluggable architecture > >>>> > > >>>> > with > >>>> > > >>>> > a > >>>> > > >>>> > choice > >>>> > > >>>> > of implementations that fit underneath." > >>>> > > >>>> > Java Security Providers have been used by other projects to > >>>> > > >>>> > provide > >>>> > > >>>> > such > >>>> > > >>>> > integration . I am not sure if you looked into Spiffe > >>>> > > >>>> > project to > >>>> > > >>>> > efficiently distribute certificates but here is an example of > >>>> > > >>>> > Java > >>>> > > >>>> > provider > >>>> > > >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ > >>>> > > >>>> > > >>>> > spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider. > >>>> > > >>>> > java which > >>>> > obtains certificates from local daemons. > >>>> > These integrations are being used in Tomcat, Jetty etc.. We > >>>> > > >>>> > are > >>>> > > >>>> > also > >>>> > > >>>> > using > >>>> > > >>>> > Security provider to do the same in our Kafka clusters. So > >>>> > > >>>> > unless I > >>>> > > >>>> > see > >>>> > > >>>> > more evidence why security.provider doesn't work for you > >>>> > > >>>> > adding > >>>> > > >>>> > new > >>>> > > >>>> > interfaces while there exists more cleaner way of achieving > >>>> > > >>>> > the > >>>> > > >>>> > goals > >>>> > > >>>> > of > >>>> > > >>>> > this KIP is unnecessary and breaks the well known security > >>>> > > >>>> > interfaces > >>>> > > >>>> > provided by Java itself. > >>>> > > >>>> > Thanks, > >>>> > Harsha > >>>> > > >>>> > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani < > >>>> > > >>>> > ka...@harsha.io > >>>> > > >>>> > wrote: > >>>> > > >>>> > Hi Maulin, > >>>> > Not sure if you looked at my previous replies. This > >>>> > > >>>> > changes > >>>> > > >>>> > are not required as there is already security Provider to do > >>>> > > >>>> > what you > >>>> > > >>>> > are > >>>> > > >>>> > proposing. This KIP > >>>> > > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/ > >>>> > > >>>> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config > >>>> > > >>>> > also > >>>> > > >>>> > addresses easy registration of such providers. > >>>> > > >>>> > Thanks, > >>>> > Harsha > >>>> > > >>>> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada > >>>> > > >>>> > <maulin.vasavada@gmail. > >>>> > > >>>> > com> wrote: > >>>> > > >>>> > Bump! Can somebody please review this? > >>>> > > >>>> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada < > >>>> > > >>>> > maulin.vasav...@gmail.com> > >>>> > > >>>> > wrote: > >>>> > > >>>> > Bump! Can somebody please review this? > >>>> > > >>>> > -- > >>>> > Thanks, > >>>> > M.Sai Sandeep > >>>> > > >>>> > > >>>> > >>> > On Wed, Aug 28, 2019 at 2:18 PM Maulin Vasavada <maulin.vasav...@gmail.com > > > wrote: > > > > > > > ---------- Forwarded message --------- > > From: Maulin Vasavada <maulin.vasav...@gmail.com> > > Date: Wed, Aug 28, 2019 at 1:51 PM > > Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and > > TrustStore > > To: <dev@kafka.apache.org> > > > > > > Hi Harsha > > > > As we synced-up offline on this topic, we hope you don't have any more > > clarifications that you are seeking. If that is the case, can you please > > help us move this forward and discuss what changes you would expect on > the > > KIP design in order to make it valuable contribution? > > > > Just FYI - we verified our primary design change with the author of Sun's > > X509 Trustmanager implementation and the outcome is that what we are > > proposing makes sense at the heart of it - "Instead of writing > TrustManager > > just plugin the Trust store". We are open to discuss additional changes > > that you/anybody else would like to see on the functionality however. > > > > > > Thanks > > Maulin > > > > On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada < > maulin.vasav...@gmail.com> > > wrote: > > > >> Hi Harsha > >> > >> Any response on my question? I feel this KIP is worth accommodating. > Your > >> help is much appreciated. > >> > >> Thanks > >> Maulin > >> > >> On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada < > >> maulin.vasav...@gmail.com> wrote: > >> > >>> Hi Harsha > >>> > >>> I've examined the SPIFFE provider more and have one question - > >>> > >>> If SPIFFE didn't have a need to do checkSpiffeId() call at the below > >>> location, would you really still write the Provider? *OR* > >>> Would you just use TrustManagerFactory.init(KeyStore) signature to pass > >>> the KeyStore from set of certs returned by spiffeIdManager. > >>> getTrustedCerts()? > >>> > >>> > >>> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/CertificateUtils.java#L100 > >>> > >>> > >>> /** > >>>> * Validates that the SPIFFE ID is present and matches the SPIFFE ID > >>>> configured in > >>>> * the java.security property ssl.spiffe.accept > >>>> * > >>>> * If the authorized spiffe ids list is empty any spiffe id is > authorized > >>>> * > >>>> * @param chain an array of X509Certificate that contains the Peer's > >>>> SVID to be validated > >>>> * @throws CertificateException when either the certificates doesn't > >>>> have a SPIFFE ID or the SPIFFE ID is not authorized > >>>> */ > >>>> static void checkSpiffeId(X509Certificate[] chain) throws > >>>> CertificateException { > >>> > >>> > >>> > >>> Thanks > >>> Maulin > >>> > >>> > >>> On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <ka...@harsha.io> > >>> wrote: > >>> > >>>> Maulin, > >>>> The code parts you are pointing are specific for Spiffe > >>>> and if > >>>> you are talking about validate method which uses PKIX check like any > >>>> other provider does. > >>>> If you want to default to SunJSSE everywhere you can do so by > delegating > >>>> the calls in these methods to SunJSSE provider. > >>>> > >>>> TrustManagerFactory tmf = TrustManagerFactory > >>>> .getInstance(TrustManagerFactory.getDefaultAlgorithm());and use > >>>> tmf.chekServerTrusted() > >>>> or use > >>>> > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/TrustManagerFactory.html#getInstance(java.lang.String)if > >>>> you want a specific provider. > >>>> > >>>> -Harsha > >>>> > >>>> > >>>> On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada < > >>>> maulin.vasav...@gmail.com> > >>>> wrote: > >>>> > >>>> > Okay, so I take that you guys agree that I have to write a 'custom' > >>>> > algorithm and a provider to make it work , correct? > >>>> > > >>>> > Now, for Harsha's comment "Here the 'Custom' Algorithm is not an > >>>> > implementation per say , ..." , I diagree. You can refer to https:// > >>>> > > >>>> > github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/ > >>>> > SpiffeTrustManager.java#L91 and > >>>> > > >>>> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/ > >>>> > provider/CertificateUtils.java#L100 > >>>> > > >>>> > "that code" is the customization you have for the custom way to > check > >>>> > something on top of regular checks. That method is NOT doing custom > >>>> > truststore loading. It is validating/verifying something in the > >>>> "custom" > >>>> > way with spiffeId. > >>>> > I bet that without that you won't have a need of the custom > algorithm > >>>> in > >>>> > the first place. > >>>> > > >>>> > Let me know if you agree to this. > >>>> > > >>>> > Thanks > >>>> > Maulin > >>>> > > >>>> > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <mpr...@gmail.com> > >>>> wrote: > >>>> > > >>>> > Hi Maulin, thanks for the discussion. As Harsha pointed out, to use > >>>> the > >>>> > KIP492, you need to create a new provider, register a *new* custom > >>>> > algorithm for your keymanager and trustmanager factory > >>>> implementations. > >>>> > After this, the kafka server configuration can be done as given > below > >>>> > > >>>> > # Register the provider class with custom algorithm, say CUSTOM > >>>> security. > >>>> > provider.classes=com.company.security.CustomProvider > >>>> > <http://security.provider.classes > >>>> =com.company.security.customprovider/> > >>>> > > >>>> > # Register the keymanager and trustmanager algorithms > >>>> > # These algorithms indicate that the Keymanager and Trustmanagers > >>>> > registered under the algorithm "CUSTOM" needs to be used > >>>> > ssl.trustmanager.algorithm=CUSTOM > >>>> > ssl.keymanager.algorithm=CUSTOM > >>>> > > >>>> > And the customprovider looks like this... > >>>> > > >>>> > public class CustomProvider extends Provider { > >>>> > public CustomProvider() { > >>>> > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and TrustStore"); > >>>> > super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory"); > >>>> > super.put("TrustManagerFactory.CUSTOM", > >>>> > "customTrustManagerFactory"); > >>>> > } > >>>> > } > >>>> > > >>>> > The PR for this is in review and can be found here: > >>>> https://github.com/ > >>>> > apache/kafka/pull/7090 > >>>> > This PR includes the fixed insertProviderAt function call. > >>>> > > >>>> > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani < > ka...@harsha.io> > >>>> > wrote: > >>>> > > >>>> > Answers are in-line > >>>> > > >>>> > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada < > >>>> maulin.vasavada@gmail. > >>>> > com > >>>> > > >>>> > wrote: > >>>> > > >>>> > Hi Colin > >>>> > > >>>> > When I refer to "standard" or "custom" algorithms I am following > Java > >>>> > security Provider Terminology. You can refer to > >>>> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/ > >>>> > StandardNames.html#TrustManagerFactory link I provided earlier in > the > >>>> > emails. It says PKIX is the default Algorithm for > TrustManagerFactory. > >>>> > > >>>> > 1. For SPIFFE, I am not sure why you are saying 'it does not > implement > >>>> > custom algorithms' because the following file clearly indicates that > >>>> it > >>>> > does use custom algorithm- > >>>> > > >>>> > > >>>> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/ > >>>> > > >>>> > provider/SpiffeProvider.java#L17 > >>>> > > >>>> > Algorithm value: > >>>> > > >>>> > > >>>> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/ > >>>> > > >>>> > provider/SpiffeProviderConstants.java#L6 > >>>> > > >>>> > @Harsha do you want to chime in since you use that provider? > >>>> > > >>>> > Here the "Custom" Algorithm is not an implementation per say , > rather > >>>> > > >>>> > used > >>>> > > >>>> > to invoke the custom trust store factory and key manager factory. > You > >>>> are > >>>> > not moving away from "standard" alogrithms that are available. > >>>> > > >>>> > > >>>> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/ > >>>> > provider/SpiffeTrustManager.java > >>>> > > >>>> > As you can see it delegates all the calls of verification of > >>>> certificate > >>>> > > >>>> > to > >>>> > > >>>> > the default implementation available. > >>>> > So in our implementation we still use PKIX to verify the certificate > >>>> > chain. So you are not losing anything here and Spiffe is not > >>>> reimplementing > >>>> > the verification process. > >>>> > > >>>> > 2. I already mentioned in my 3rd point, in my previous post, why > using > >>>> > > >>>> > ssl.provider does NOT work. I updated KIP-486 in "rejected > >>>> > > >>>> > alternatives" > >>>> > > >>>> > also why ssl.provider does not work. > >>>> > > >>>> > As mentioned before , provider is the right way to go. I am still > not > >>>> > understanding the gap is. > >>>> > If I understand correctly your argument is , provider is going to > ask > >>>> to > >>>> > implement a custom algorithm. > >>>> > My answer to that is , no you are not re-implementing the algorithm. > >>>> > > >>>> > Please > >>>> > > >>>> > check the above link , TrustManager implementation it delegates the > >>>> calls > >>>> > back. There is no need to implement your own here. > >>>> > > >>>> > 3. Security.insertProviderAt() comments were based on assumption if > >>>> > > >>>> > KIP-492 > >>>> > > >>>> > changes are done and we use that mechanism to configure providers > >>>> > > >>>> > instead > >>>> > > >>>> > of ssl.provider configuration. > >>>> > > >>>> > KIP-492 has patch available and going through review. > >>>> > > >>>> > Can you read my all the points, I mentioned in my previous post, > very > >>>> > > >>>> > carefully? I am covering all the aspects in explaining. I am open to > >>>> > > >>>> > still > >>>> > > >>>> > discuss more to clarify any doubts. > >>>> > > >>>> > "3. If we just use existing ssl.provider kafka configuration then > our > >>>> > provider will be used in SSLContext.getInstance(protocol, provider) > >>>> call > >>>> > > >>>> > in > >>>> > > >>>> > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/> > >>>> and > >>>> > if our provider does not have > >>>> > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we > >>>> > > >>>> > tested > >>>> > > >>>> > it). Example: In MyProvider sample above you see that I didn't add > >>>> > SSLContext.TLSv1 as > >>>> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider > >>>> you > >>>> > don't have this challenge since you are planning to bypass > >>>> ssl.provider > >>>> > > >>>> > as > >>>> > > >>>> > you mention in the KIP-492." > >>>> > > >>>> > Yes here you need to pass the protocol that your > >>>> KeyManager/TrustManager > >>>> > registered with and in no way its deviating from TLS RFC spec. > >>>> > > >>>> > > >>>> > https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/ > >>>> > javax/net/ssl/SSLContext.java#L134 > >>>> > > >>>> > My suggestion here is for you to implement a simple Security > Provider > >>>> as > >>>> > you did before and register a Algorithm. You can use the existing > >>>> > implementation in SpiffeProvider and plug in changes where you need > to > >>>> > retrieve the certificates from by making RPC call. > >>>> > > >>>> > Run an end-to-end test with Kafka broker coming up with your > provider > >>>> > making calls to RPC call. You do need to pass the "custom algorithm" > >>>> that > >>>> > you registered in your key manager to invoke the provider. I think > >>>> your > >>>> > concern is this approach is replacing the existing known > ciphersuites > >>>> and > >>>> > certificate validation provided by java. Which its not. > >>>> > > >>>> > Now test the TLS connection you can do so via openssl -s_client > >>>> options > >>>> > > >>>> > to > >>>> > > >>>> > test if everything else is working. > >>>> > > >>>> > I am happy to share configs that we used if you are interested. > >>>> > > >>>> > Thanks, > >>>> > Harsha > >>>> > > >>>> > Thanks > >>>> > Maulin > >>>> > > >>>> > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cmcc...@apache.org> > >>>> > > >>>> > wrote: > >>>> > > >>>> > Hi Maulin, > >>>> > > >>>> > A lot of JSSE providers don't implement custom algorithms. Spire is > a > >>>> > > >>>> > good > >>>> > > >>>> > example of a JSSE provider that doesn't, and yet is still useful to > >>>> > > >>>> > many > >>>> > > >>>> > people. Your JSSE provider can work fine even if it doesn't > implement > >>>> a > >>>> > custom algorithm. > >>>> > > >>>> > Maybe I'm missing something, but I don't understand the discussion > of > >>>> > Security.insertProviderAt() that you included. SslEngineBuilder > >>>> doesn't > >>>> > > >>>> > use > >>>> > > >>>> > that API to get the security provider. Instead, it calls > >>>> > "SSLContext.getInstance(protocol, provider)", where provider is the > >>>> > > >>>> > name > >>>> > > >>>> > of the provider. > >>>> > > >>>> > best, > >>>> > Colin > >>>> > > >>>> > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote: > >>>> > > >>>> > On top of everything above I feel strongly to add the 4th point > which > >>>> > > >>>> > is > >>>> > > >>>> > based on Java APIs for TrustManagerFactory.init(KeyStore) ( > >>>> > > >>>> > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/ > >>>> > TrustManagerFactory.html#init(java.security.KeyStore > >>>> > <http://java.security.keystore/>) > >>>> > ) > >>>> > > >>>> > and KeyManagerFactory.init(KeyStore, char[]) ( > >>>> > > >>>> > > >>>> > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory > >>>> . > >>>> > > >>>> > > >>>> > html#init(java.security.KeyStore <http://java.security.keystore/ > >>>> >,%20char[]) > >>>> > > >>>> > > >>>> > ). > >>>> > > >>>> > 4. The above APIs are intended to support providing "trust/key > >>>> > > >>>> > material" > >>>> > > >>>> > from the user without having to write their own > >>>> > > >>>> > TrustManager/KeyManagers. > >>>> > > >>>> > To quote from the TrustManagerFactory.init()'s documentation > >>>> > > >>>> > "Initializes > >>>> > > >>>> > this factory with a source of certificate authorities and related > >>>> trust > >>>> > material." > >>>> > To quote from the KeyManagerFactory.init()'s documentation > >>>> "Initializes > >>>> > this factory with a source of key material." > >>>> > > >>>> > Based on this it is clear that there is a flexibility provided by > Java > >>>> > > >>>> > to > >>>> > > >>>> > to enable developers to provide the required trust/key material > loaded > >>>> > > >>>> > from > >>>> > > >>>> > "anywhere" without requiring them to write custom provider OR > >>>> trust/key > >>>> > managers. This same flexibility is reflected in Kafka code also > where > >>>> > > >>>> > it > >>>> > > >>>> > loads the trust/keys from a local file and doesn't require writing a > >>>> > Provider necessarily. If we do NOT have a custom algorithm, it makes > >>>> > > >>>> > less > >>>> > > >>>> > sense to write a Provider. > >>>> > > >>>> > Thanks > >>>> > Maulin > >>>> > > >>>> > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada < > >>>> > > >>>> > maulin.vasav...@gmail.com> > >>>> > > >>>> > wrote: > >>>> > > >>>> > Hi Harsha/Colin > >>>> > > >>>> > I did the sample with a custom Provider for TrustStoreManager and > >>>> tried > >>>> > using ssl.provider Kafka config AND the way KIP-492 is suggesting > (by > >>>> > adding Provider programmatically instead of relying on > >>>> > > >>>> > ssl.provider+java. > >>>> > > >>>> > security. The below sample is followed by my detailed findings. I'll > >>>> > appreciate if you can go through it carefully and see > >>>> > > >>>> > if you > >>>> > > >>>> > see my point. > >>>> > > >>>> > package providertest; > >>>> > > >>>> > import java.security.Provider <http://java.security.provider/> > >>>> <http:// > >>>> > java.security.provider/>; > >>>> > > >>>> > public class MyProvider extends Provider { > >>>> > > >>>> > private static final String name = "MyProvider"; private static > double > >>>> > version = 1.0d; > >>>> > private static String info = "Maulin's SSL Provider v"+version; > >>>> > > >>>> > public MyProvider() { > >>>> > super(name, version, info); > >>>> > this.put("TrustManagerFactory.PKIX", > >>>> > > >>>> > "providertest.MyTrustManagerFactory"); > >>>> > > >>>> > } > >>>> > } > >>>> > > >>>> > *Details:* > >>>> > > >>>> > KIP-492 documents that it will use Security.addProvider() assuming > it > >>>> > > >>>> > will > >>>> > > >>>> > add it as position '0' which is not a correct assumption. The > >>>> > addProvider()'s documentation says it will add it to the last > >>>> available > >>>> > position. You may want to correct that to say > >>>> > Security.insertProviderAt(provider, 1). > >>>> > > >>>> > Now coming back to our specific discussion, > >>>> > > >>>> > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add > >>>> > > >>>> > that > >>>> > > >>>> > provider in the provider list via Security.addProvider() the > position > >>>> > > >>>> > where > >>>> > > >>>> > it gets added doesn't matter (even if you don't end up adding it as > >>>> > > >>>> > first > >>>> > > >>>> > entry) since that is the ONLY provider for SPIFFE specific algorithm > >>>> > > >>>> > you > >>>> > > >>>> > might have. > >>>> > > >>>> > We do *not* have custom algorithm for Key/Trust StoreMangers. Which > >>>> > > >>>> > means > >>>> > > >>>> > we have to use X509, PKIX etc "Standard Algorithms" (( > >>>> > > >>>> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/ > >>>> > StandardNames.html > >>>> > )) > >>>> > > >>>> > in our provider to override the TrustStoreManager (see my sample > code) > >>>> > > >>>> > and > >>>> > > >>>> > KeyStoreManger and KeyManager. This creates another challenge > >>>> > > >>>> > mentioned in > >>>> > > >>>> > the below point. > >>>> > > >>>> > 2. In order to make our Provider for loading custom TrustStore work, > >>>> we > >>>> > have to add the provider as 'first' in the list since there are > others > >>>> > > >>>> > with > >>>> > > >>>> > the same algorithm. > >>>> > > >>>> > However, the programatic way of adding provider > >>>> > (Security.insertProviderAt()) is *not* deterministic for ordering > >>>> since > >>>> > different code can call the method for a different provider and > >>>> > > >>>> > depending > >>>> > > >>>> > upon the order of the call our provider can be first or pushed down > >>>> the > >>>> > list. This can happen very well in any client application using > Kafka. > >>>> > > >>>> > This > >>>> > > >>>> > is specially problematic for a case when you want to guarantee order > >>>> > > >>>> > for a > >>>> > > >>>> > Provider having "Standard Algorithms". > >>>> > > >>>> > If we add our provider in java.security file that definitely > >>>> guarantees > >>>> > the order(unless somebody calls removeProvider() which is less > >>>> > > >>>> > likely). But > >>>> > > >>>> > if we add our provider in java.security file it will defeat the > >>>> > > >>>> > purpose of > >>>> > > >>>> > the KIP-492. > >>>> > > >>>> > In the gist - Apache Kafka must not rely on "un-deterministic" > method > >>>> > > >>>> > to > >>>> > > >>>> > rely on Provider ordering. > >>>> > > >>>> > 3. If we just use existing ssl.provider kafka configuration then our > >>>> > provider will be used in SSLContext.getInstance(protocol, provider) > >>>> > > >>>> > call in > >>>> > > >>>> > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/> > >>>> and > >>>> > if our provider does not have implementation for > >>>> > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks > >>>> > > >>>> > (we > >>>> > > >>>> > tested it). Example: > >>>> > > >>>> > In > >>>> > > >>>> > MyProvider sample above you see that I didn't add SSLContext.TLSv1 > as > >>>> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider > >>>> you > >>>> > don't have this challenge since you are planning to bypass > >>>> > > >>>> > ssl.provider as > >>>> > > >>>> > you mention in the KIP-492. > >>>> > > >>>> > *Overall summary,* > >>>> > > >>>> > 1. Any provider based mechanisms- a) existing ssl.provider and > >>>> > > >>>> > b)KIP-492, > >>>> > > >>>> > for loading key/trust store using "Standard Algorithms" do not work > >>>> > > >>>> > 2. Approach suggested in our KIP-486 works without any issue and it > is > >>>> > *not* our context specific solve > >>>> > > >>>> > 3. Based on above we feel KIP-492 and KIP-486 are complimentary > >>>> changes > >>>> > and not contradicting or redundent. > >>>> > > >>>> > If you want we can do a joint session somehow to walk through the > >>>> > > >>>> > sample I > >>>> > > >>>> > have and various experiments I did. I would encourage you to do > >>>> similar > >>>> > exercise by writing a Provider for "Standard Algorithm" for > >>>> > TrustStoreManager (like our needs) and see what you find since only > >>>> > > >>>> > writing > >>>> > > >>>> > samples can bring out the complexity/challenges we face. > >>>> > > >>>> > Thanks > >>>> > Maulin > >>>> > > >>>> > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada < > >>>> > > >>>> > maulin.vasavada@gmail. > >>>> > > >>>> > com> wrote: > >>>> > > >>>> > Just to update - still working on it. Get to work only on and off on > >>>> > > >>>> > it :( > >>>> > > >>>> > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada < > >>>> > > >>>> > maulin.vasav...@gmail.com> > >>>> > > >>>> > wrote: > >>>> > > >>>> > Hi Harsha > >>>> > > >>>> > Let me try to write samples and will let you know. > >>>> > > >>>> > Thanks > >>>> > Maulin > >>>> > > >>>> > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <harsha...@gmail.com> > >>>> > > >>>> > wrote: > >>>> > > >>>> > Hi Maulin, > >>>> > With java security providers can be as custom you would > >>>> > > >>>> > like > >>>> > > >>>> > it to > >>>> > be. If you only want to to implement a custom way of loading the > >>>> > > >>>> > keystore > >>>> > > >>>> > and truststore and not implement any protocol/encryption handling > you > >>>> can > >>>> > leave them empty and no need to implement. Have you looked into the > >>>> links I > >>>> > pasted before? > >>>> > > >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ > >>>> > > >>>> > > >>>> > spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore. > >>>> > > >>>> > java > >>>> > > >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ > >>>> > spiffe-security-provider/src/main/java/spiffe/api/provider/ > >>>> > SpiffeTrustManager.java <http://spiffetrustmanager.java/> > >>>> > > >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ > >>>> > > >>>> > > >>>> > spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider. > >>>> > > >>>> > java > >>>> > > >>>> > Can you please tell me which methods are too complex in above to > >>>> > > >>>> > implement > >>>> > > >>>> > or unnecessary? You are changing anything in SSL/TLS implementations > >>>> > provided by > >>>> > > >>>> > All of the implementations delegating the checks to the default > >>>> > implementation anyway. > >>>> > Spire agent is an example, its nothing but a GRPC server listening > >>>> > > >>>> > on a > >>>> > > >>>> > unix domain socket . Above code is making a RPC call to the local > >>>> > > >>>> > daemon > >>>> > > >>>> > to > >>>> > get the certificate and keys. The mechanics are pretty much same as > >>>> > > >>>> > what > >>>> > > >>>> > you are asking for. >
