[
https://issues.apache.org/jira/browse/WICKET-6154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Grigorov updated WICKET-6154:
------------------------------------
Description:
We observed a performance problem in production. As could be seen by thread
dumps, many threads were waiting for the same lock, an instance of
com.oracle.security.ucrypto.UcryptoProvider.
The base class java.security.Provider.java extends java.util.Properties and
many methos are synchronized(). So calling lots of operations on the provider
results in a bottleneck.
The calls stacks waiting for the lock were the following (line numbers for
Wicket 6.0.22, but from looking at the code I expect the same problems should
happen even for master).
First the frames close to the bottom of the stack, they are always the same:
at
org.apache.wicket.core.request.mapper.CryptoMapper.encryptEntireUrl(CryptoMapper.java:313)
at
org.apache.wicket.core.request.mapper.CryptoMapper.encryptUrl(CryptoMapper.java:295)
at
de.postbank.rff.application.rai.security.RaiCryptoMapper.mapHandler(RaiCryptoMapper.java:62)
at org.apache.wicket.request.cycle.RequestCycle.mapUrlFor(RequestCycle.java:429)
at org.apache.wicket.request.cycle.RequestCycle.urlFor(RequestCycle.java:529)
at org.apache.wicket.Component.urlFor(Component.java:3374)
at org.apache.wicket.markup.html.link.Link.getURL(Link.java:327)
Now the four most frequent detail stacks which sit above the common stack:
at java.util.Hashtable.get(Hashtable.java:433)
- waiting on (a com.oracle.security.ucrypto.UcryptoProvider@0xHEXADDR)
at java.util.Properties.getProperty(Properties.java:951)
at java.security.Provider.getProperty(Provider.java:390)
at java.security.Security.getProviderProperty(Security.java:262)
at java.security.Security.isCriterionSatisfied(Security.java:914)
at java.security.Security.getProvidersNotUsingCache(Security.java:889)
at java.security.Security.getAllQualifyingCandidates(Security.java:877)
at java.security.Security.getProviders(Security.java:625)
at java.security.Security.getProviders(Security.java:552)
at org.apache.wicket.util.crypt.SunJceCrypt.<init>(SunJceCrypt.java:81)
at
org.apache.wicket.core.util.crypt.KeyInSessionSunJceCryptFactory.createCrypt(KeyInSessionSunJceCryptFactory.java:92)
at
org.apache.wicket.core.util.crypt.KeyInSessionSunJceCryptFactory.newCrypt(KeyInSessionSunJceCryptFactory.java:82)
at java.security.Provider.getService(Provider.java:680)
- waiting on (a com.oracle.security.ucrypto.UcryptoProvider@0xHEXADDR)
at sun.security.jca.ProviderList.getService(ProviderList.java:331)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:157)
at java.security.Security.getImpl(Security.java:695)
at java.security.MessageDigest.getInstance(MessageDigest.java:167)
at com.sun.crypto.provider.PBECipherCore.<init>(PBECipherCore.java:75)
at
com.sun.crypto.provider.PBEWithMD5AndDESCipher.<init>(PBEWithMD5AndDESCipher.java:61)
at sun.reflect.GeneratedConstructorAccessor108.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.security.Provider$Service.newInstance(Provider.java:1240)
at javax.crypto.Cipher.chooseProvider(Cipher.java:850)
at javax.crypto.Cipher.init(Cipher.java:1374)
at javax.crypto.Cipher.init(Cipher.java:1308)
at org.apache.wicket.util.crypt.SunJceCrypt.createCipher(SunJceCrypt.java:133)
at org.apache.wicket.util.crypt.SunJceCrypt.crypt(SunJceCrypt.java:114)
at
org.apache.wicket.util.crypt.AbstractCrypt.encryptStringToByteArray(AbstractCrypt.java:172)
at
org.apache.wicket.util.crypt.AbstractCrypt.encryptUrlSafe(AbstractCrypt.java:87)
at
org.apache.wicket.core.request.mapper.CryptoMapper.encryptEntireUrl(CryptoMapper.java:313)
at java.security.Provider.getService(Provider.java:680)
- waiting on (a com.oracle.security.ucrypto.UcryptoProvider@0xHEXADDR)
at sun.security.jca.ProviderList$ServiceList.tryGet(ProviderList.java:443)
at sun.security.jca.ProviderList$ServiceList.access$200(ProviderList.java:375)
at sun.security.jca.ProviderList$ServiceList$1.hasNext(ProviderList.java:485)
at javax.crypto.Cipher.getInstance(Cipher.java:502)
at org.apache.wicket.util.crypt.SunJceCrypt.createCipher(SunJceCrypt.java:132)
at org.apache.wicket.util.crypt.SunJceCrypt.crypt(SunJceCrypt.java:114)
at
org.apache.wicket.util.crypt.AbstractCrypt.encryptStringToByteArray(AbstractCrypt.java:172)
at
org.apache.wicket.util.crypt.AbstractCrypt.encryptUrlSafe(AbstractCrypt.java:87)
at
org.apache.wicket.core.request.mapper.CryptoMapper.encryptEntireUrl(CryptoMapper.java:313)
at java.security.Provider.getService(Provider.java:680)
- waiting on (a com.oracle.security.ucrypto.UcryptoProvider@0xHEXADDR)
at sun.security.jca.ProviderList$ServiceList.tryGet(ProviderList.java:436)
at sun.security.jca.ProviderList$ServiceList.access$200(ProviderList.java:375)
at sun.security.jca.ProviderList$ServiceList$1.hasNext(ProviderList.java:485)
at javax.crypto.SecretKeyFactory.nextSpi(SecretKeyFactory.java:292)
at javax.crypto.SecretKeyFactory.<init>(SecretKeyFactory.java:120)
at javax.crypto.SecretKeyFactory.getInstance(SecretKeyFactory.java:159)
at
org.apache.wicket.util.crypt.SunJceCrypt.generateSecretKey(SunJceCrypt.java:152)
at org.apache.wicket.util.crypt.SunJceCrypt.crypt(SunJceCrypt.java:112)
at
org.apache.wicket.util.crypt.AbstractCrypt.encryptStringToByteArray(AbstractCrypt.java:172)
at
org.apache.wicket.util.crypt.AbstractCrypt.encryptUrlSafe(AbstractCrypt.java:87)
at
org.apache.wicket.core.request.mapper.CryptoMapper.encryptEntireUrl(CryptoMapper.java:313)
There's a couple of possible improvements/solutions:
In SunJceCrypt.java when creating a new instance there's a check
Security.getProviders("Cipher." + cryptMethod).length > 0
The result of the check could be cached in a static
ConcurrentSkipListSet<String>. Note that the method Security.getProviders() is
in one of the synchronized stacks above.
The SecretKey derived from the string types session key is created in
SunJceCrypt using method generateSecretKey() every time a crypt() call is
executed. Since the SecretKey only depends on the cryptMethod and the string
key, and furthermore it is serializable and should be thread-safe (read-only)
it could be generated once and cached in the session. Creation of the SecretKey
is another of the synchronized stacks above.
Furthemore one could cache the cipher per session, more precisely one instance
created for encrypt mode and one for decrypt mode. Unfortunately the
MetaDataKey based way of putting stuff into the session only allows
serializable objects and Cipher is not serializable. The use of such cached
ciphers per session would need to be synchronized on the cipher object, but
that looks much better than stressing the global provider lock.
Regards,
Rainer
was:
We observed a performance problem in production. As could be seen by thread
dumps, many threads were waiting for the same lock, an instance of
com.oracle.security.ucrypto.UcryptoProvider.
The base class java.security.Provider.java extends java.util.Properties and
many methos are synchronized(). So calling lots of operations on the provider
results in a bottleneck.
The calls stacks waiting for the lock were the following (line numbers for
Wicket 6.0.22, but from looking at the code I expect the same problems should
happen even for master).
First the frames close to the bottom of the stack, they are always the same:
at
org.apache.wicket.core.request.mapper.CryptoMapper.encryptEntireUrl(CryptoMapper.java:313)
at
org.apache.wicket.core.request.mapper.CryptoMapper.encryptUrl(CryptoMapper.java:295)
at
de.postbank.rff.application.rai.security.RaiCryptoMapper.mapHandler(RaiCryptoMapper.java:62)
at org.apache.wicket.request.cycle.RequestCycle.mapUrlFor(RequestCycle.java:429)
at org.apache.wicket.request.cycle.RequestCycle.urlFor(RequestCycle.java:529)
at org.apache.wicket.Component.urlFor(Component.java:3374)
at org.apache.wicket.markup.html.link.Link.getURL(Link.java:327)
Now the four most frequent detail stacks which sit above the common stack:
at java.util.Hashtable.get(Hashtable.java:433)
- waiting on (a com.oracle.security.ucrypto.UcryptoProvider@0xHEXADDR)
at java.util.Properties.getProperty(Properties.java:951)
at java.security.Provider.getProperty(Provider.java:390)
at java.security.Security.getProviderProperty(Security.java:262)
at java.security.Security.isCriterionSatisfied(Security.java:914)
at java.security.Security.getProvidersNotUsingCache(Security.java:889)
at java.security.Security.getAllQualifyingCandidates(Security.java:877)
at java.security.Security.getProviders(Security.java:625)
at java.security.Security.getProviders(Security.java:552)
at org.apache.wicket.util.crypt.SunJceCrypt.<init>(SunJceCrypt.java:81)
at
org.apache.wicket.core.util.crypt.KeyInSessionSunJceCryptFactory.createCrypt(KeyInSessionSunJceCryptFactory.java:92)
at
org.apache.wicket.core.util.crypt.KeyInSessionSunJceCryptFactory.newCrypt(KeyInSessionSunJceCryptFactory.java:82)
at java.security.Provider.getService(Provider.java:680)
- waiting on (a com.oracle.security.ucrypto.UcryptoProvider@0xHEXADDR)
at sun.security.jca.ProviderList.getService(ProviderList.java:331)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:157)
at java.security.Security.getImpl(Security.java:695)
at java.security.MessageDigest.getInstance(MessageDigest.java:167)
at com.sun.crypto.provider.PBECipherCore.<init>(PBECipherCore.java:75)
at
com.sun.crypto.provider.PBEWithMD5AndDESCipher.<init>(PBEWithMD5AndDESCipher.java:61)
at sun.reflect.GeneratedConstructorAccessor108.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.security.Provider$Service.newInstance(Provider.java:1240)
at javax.crypto.Cipher.chooseProvider(Cipher.java:850)
at javax.crypto.Cipher.init(Cipher.java:1374)
at javax.crypto.Cipher.init(Cipher.java:1308)
at org.apache.wicket.util.crypt.SunJceCrypt.createCipher(SunJceCrypt.java:133)
at org.apache.wicket.util.crypt.SunJceCrypt.crypt(SunJceCrypt.java:114)
at
org.apache.wicket.util.crypt.AbstractCrypt.encryptStringToByteArray(AbstractCrypt.java:172)
at
org.apache.wicket.util.crypt.AbstractCrypt.encryptUrlSafe(AbstractCrypt.java:87)
at
org.apache.wicket.core.request.mapper.CryptoMapper.encryptEntireUrl(CryptoMapper.java:313)
at java.security.Provider.getService(Provider.java:680)
- waiting on (a com.oracle.security.ucrypto.UcryptoProvider@0xHEXADDR)
at sun.security.jca.ProviderList$ServiceList.tryGet(ProviderList.java:443)
at sun.security.jca.ProviderList$ServiceList.access$200(ProviderList.java:375)
at sun.security.jca.ProviderList$ServiceList$1.hasNext(ProviderList.java:485)
at javax.crypto.Cipher.getInstance(Cipher.java:502)
at org.apache.wicket.util.crypt.SunJceCrypt.createCipher(SunJceCrypt.java:132)
at org.apache.wicket.util.crypt.SunJceCrypt.crypt(SunJceCrypt.java:114)
at
org.apache.wicket.util.crypt.AbstractCrypt.encryptStringToByteArray(AbstractCrypt.java:172)
at
org.apache.wicket.util.crypt.AbstractCrypt.encryptUrlSafe(AbstractCrypt.java:87)
at
org.apache.wicket.core.request.mapper.CryptoMapper.encryptEntireUrl(CryptoMapper.java:313)
at java.security.Provider.getService(Provider.java:680)
- waiting on (a com.oracle.security.ucrypto.UcryptoProvider@0xHEXADDR)
at sun.security.jca.ProviderList$ServiceList.tryGet(ProviderList.java:436)
at sun.security.jca.ProviderList$ServiceList.access$200(ProviderList.java:375)
at sun.security.jca.ProviderList$ServiceList$1.hasNext(ProviderList.java:485)
at javax.crypto.SecretKeyFactory.nextSpi(SecretKeyFactory.java:292)
at javax.crypto.SecretKeyFactory.<init>(SecretKeyFactory.java:120)
at javax.crypto.SecretKeyFactory.getInstance(SecretKeyFactory.java:159)
at
org.apache.wicket.util.crypt.SunJceCrypt.generateSecretKey(SunJceCrypt.java:152)
at org.apache.wicket.util.crypt.SunJceCrypt.crypt(SunJceCrypt.java:112)
at
org.apache.wicket.util.crypt.AbstractCrypt.encryptStringToByteArray(AbstractCrypt.java:172)
at
org.apache.wicket.util.crypt.AbstractCrypt.encryptUrlSafe(AbstractCrypt.java:87)
at
org.apache.wicket.core.request.mapper.CryptoMapper.encryptEntireUrl(CryptoMapper.java:313)
There's a couple of possible improvements/solutions:
In SunJceCrypt.java when creating a new instance there's a check
Security.getProviders("Cipher." + cryptMethod).length > 0
The result of the check could be cached in a static
ConcurrentSkipListSet<String>. Note that the method Security.getProviders() is
in one of the synchronized stacks above.
The SecretKey derived from the string types session key is created in
SunJceCrypt using method generateSecretKey() every time a crypt() call is
executed. Since the SecretKey only depends on the cryptMethod and the string
key, and furthermore it is serializable and should be thread-safe (read-only)
it could be generated once and cached in the session. Creation of the SecretKey
is another of the synchronized stacks above.
Furthemore one could cache the cipher per session, more precisely one instance
created for encrypt mode and one for decrypt mode. Unfortunately the
MetaDataKey based way of putting stuff into the session only allows
serializable objects and Cipher is not serializable. The use of such cached
ciphers per session would need to be synchronized on the cipher object, but
that looks much worse than stressing the global provider lock.
Regards,
Rainer
> Performance bottleneck when using KeyInSessionSunJceCryptFactory
> ----------------------------------------------------------------
>
> Key: WICKET-6154
> URL: https://issues.apache.org/jira/browse/WICKET-6154
> Project: Wicket
> Issue Type: Bug
> Components: wicket
> Reporter: Rainer Jung
>
> We observed a performance problem in production. As could be seen by thread
> dumps, many threads were waiting for the same lock, an instance of
> com.oracle.security.ucrypto.UcryptoProvider.
> The base class java.security.Provider.java extends java.util.Properties and
> many methos are synchronized(). So calling lots of operations on the provider
> results in a bottleneck.
> The calls stacks waiting for the lock were the following (line numbers for
> Wicket 6.0.22, but from looking at the code I expect the same problems should
> happen even for master).
> First the frames close to the bottom of the stack, they are always the same:
> at
> org.apache.wicket.core.request.mapper.CryptoMapper.encryptEntireUrl(CryptoMapper.java:313)
> at
> org.apache.wicket.core.request.mapper.CryptoMapper.encryptUrl(CryptoMapper.java:295)
> at
> de.postbank.rff.application.rai.security.RaiCryptoMapper.mapHandler(RaiCryptoMapper.java:62)
> at
> org.apache.wicket.request.cycle.RequestCycle.mapUrlFor(RequestCycle.java:429)
> at org.apache.wicket.request.cycle.RequestCycle.urlFor(RequestCycle.java:529)
> at org.apache.wicket.Component.urlFor(Component.java:3374)
> at org.apache.wicket.markup.html.link.Link.getURL(Link.java:327)
> Now the four most frequent detail stacks which sit above the common stack:
> at java.util.Hashtable.get(Hashtable.java:433)
> - waiting on (a com.oracle.security.ucrypto.UcryptoProvider@0xHEXADDR)
> at java.util.Properties.getProperty(Properties.java:951)
> at java.security.Provider.getProperty(Provider.java:390)
> at java.security.Security.getProviderProperty(Security.java:262)
> at java.security.Security.isCriterionSatisfied(Security.java:914)
> at java.security.Security.getProvidersNotUsingCache(Security.java:889)
> at java.security.Security.getAllQualifyingCandidates(Security.java:877)
> at java.security.Security.getProviders(Security.java:625)
> at java.security.Security.getProviders(Security.java:552)
> at org.apache.wicket.util.crypt.SunJceCrypt.<init>(SunJceCrypt.java:81)
> at
> org.apache.wicket.core.util.crypt.KeyInSessionSunJceCryptFactory.createCrypt(KeyInSessionSunJceCryptFactory.java:92)
> at
> org.apache.wicket.core.util.crypt.KeyInSessionSunJceCryptFactory.newCrypt(KeyInSessionSunJceCryptFactory.java:82)
> at java.security.Provider.getService(Provider.java:680)
> - waiting on (a com.oracle.security.ucrypto.UcryptoProvider@0xHEXADDR)
> at sun.security.jca.ProviderList.getService(ProviderList.java:331)
> at sun.security.jca.GetInstance.getInstance(GetInstance.java:157)
> at java.security.Security.getImpl(Security.java:695)
> at java.security.MessageDigest.getInstance(MessageDigest.java:167)
> at com.sun.crypto.provider.PBECipherCore.<init>(PBECipherCore.java:75)
> at
> com.sun.crypto.provider.PBEWithMD5AndDESCipher.<init>(PBEWithMD5AndDESCipher.java:61)
> at sun.reflect.GeneratedConstructorAccessor108.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at java.security.Provider$Service.newInstance(Provider.java:1240)
> at javax.crypto.Cipher.chooseProvider(Cipher.java:850)
> at javax.crypto.Cipher.init(Cipher.java:1374)
> at javax.crypto.Cipher.init(Cipher.java:1308)
> at org.apache.wicket.util.crypt.SunJceCrypt.createCipher(SunJceCrypt.java:133)
> at org.apache.wicket.util.crypt.SunJceCrypt.crypt(SunJceCrypt.java:114)
> at
> org.apache.wicket.util.crypt.AbstractCrypt.encryptStringToByteArray(AbstractCrypt.java:172)
> at
> org.apache.wicket.util.crypt.AbstractCrypt.encryptUrlSafe(AbstractCrypt.java:87)
> at
> org.apache.wicket.core.request.mapper.CryptoMapper.encryptEntireUrl(CryptoMapper.java:313)
> at java.security.Provider.getService(Provider.java:680)
> - waiting on (a com.oracle.security.ucrypto.UcryptoProvider@0xHEXADDR)
> at sun.security.jca.ProviderList$ServiceList.tryGet(ProviderList.java:443)
> at sun.security.jca.ProviderList$ServiceList.access$200(ProviderList.java:375)
> at sun.security.jca.ProviderList$ServiceList$1.hasNext(ProviderList.java:485)
> at javax.crypto.Cipher.getInstance(Cipher.java:502)
> at org.apache.wicket.util.crypt.SunJceCrypt.createCipher(SunJceCrypt.java:132)
> at org.apache.wicket.util.crypt.SunJceCrypt.crypt(SunJceCrypt.java:114)
> at
> org.apache.wicket.util.crypt.AbstractCrypt.encryptStringToByteArray(AbstractCrypt.java:172)
> at
> org.apache.wicket.util.crypt.AbstractCrypt.encryptUrlSafe(AbstractCrypt.java:87)
> at
> org.apache.wicket.core.request.mapper.CryptoMapper.encryptEntireUrl(CryptoMapper.java:313)
> at java.security.Provider.getService(Provider.java:680)
> - waiting on (a com.oracle.security.ucrypto.UcryptoProvider@0xHEXADDR)
> at sun.security.jca.ProviderList$ServiceList.tryGet(ProviderList.java:436)
> at sun.security.jca.ProviderList$ServiceList.access$200(ProviderList.java:375)
> at sun.security.jca.ProviderList$ServiceList$1.hasNext(ProviderList.java:485)
> at javax.crypto.SecretKeyFactory.nextSpi(SecretKeyFactory.java:292)
> at javax.crypto.SecretKeyFactory.<init>(SecretKeyFactory.java:120)
> at javax.crypto.SecretKeyFactory.getInstance(SecretKeyFactory.java:159)
> at
> org.apache.wicket.util.crypt.SunJceCrypt.generateSecretKey(SunJceCrypt.java:152)
> at org.apache.wicket.util.crypt.SunJceCrypt.crypt(SunJceCrypt.java:112)
> at
> org.apache.wicket.util.crypt.AbstractCrypt.encryptStringToByteArray(AbstractCrypt.java:172)
> at
> org.apache.wicket.util.crypt.AbstractCrypt.encryptUrlSafe(AbstractCrypt.java:87)
> at
> org.apache.wicket.core.request.mapper.CryptoMapper.encryptEntireUrl(CryptoMapper.java:313)
> There's a couple of possible improvements/solutions:
> In SunJceCrypt.java when creating a new instance there's a check
> Security.getProviders("Cipher." + cryptMethod).length > 0
> The result of the check could be cached in a static
> ConcurrentSkipListSet<String>. Note that the method Security.getProviders()
> is in one of the synchronized stacks above.
> The SecretKey derived from the string types session key is created in
> SunJceCrypt using method generateSecretKey() every time a crypt() call is
> executed. Since the SecretKey only depends on the cryptMethod and the string
> key, and furthermore it is serializable and should be thread-safe (read-only)
> it could be generated once and cached in the session. Creation of the
> SecretKey is another of the synchronized stacks above.
> Furthemore one could cache the cipher per session, more precisely one
> instance created for encrypt mode and one for decrypt mode. Unfortunately the
> MetaDataKey based way of putting stuff into the session only allows
> serializable objects and Cipher is not serializable. The use of such cached
> ciphers per session would need to be synchronized on the cipher object, but
> that looks much better than stressing the global provider lock.
> Regards,
> Rainer
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
