One of the long standing issues with the security manager support is that the overhead when there is no security manager is non-zero. Two specific examples are (i) the overhead of defineClass (assuming the defining loader is a SecureClassLoader) as it involves a callback to get the permissions for the protection domain and (ii) the overhead of AccessController.doPrivileged in order to execute an action on a privileged stack.
The bigger question is of course whether it is interesting to run with a security manager in 2017. It's clearly still important for environments that download potentially malicious code to execute in a sandox but that isn't most applications these days. We have seen a few cases where applications set a security manager in order to enforce some policy, like preventing plugins calling System.exit but these are cases that would be better served with other solutions.
I would like to explore changes to the API and implementation that would allow us to eliminate some of the overhead when not running with a security manager. Long term then it would be interesting to explore degrading and eventually dropping the security manager but that is beyond the scope of what I would like to discuss here. Sean Mullan and Jeff Nisewanger ran an interesting BOF at JavaOne 2017 on this topic and I believe are planning to do a survey at some point to understand the current usage of the security manager.
For now I would like to explore what it would take to eliminate some of the overhead. A big challenge is the System.setSecurityManager API as allows a security manager to be set in a running VM. This means that classes loaded before a security manager is set may be involved in permission checks that happen after a security manager is set. Similarly, privileged actions may have started before the security manager is set. The complications around this go away if there was some way to know that a security manager can never be turned on in a running system.
To get started, I've put a prototype of an initial proposal on the cr site [1]. The summary of the proposal is:
1. A "disallow security manager" mode that is opt-in to disallow setSecurityManager being used to set a security manager in a running VM. Opt-in means it has not impact on existing code.
2. Deprecates setSecurityManager to discourage further usage and allow for the possibility of making "disallow security manager" the default in the future. If that were to become the default then it would have no impact on deployments that set the security manager on the command line at startup.
3. When running in "disallow security manager" mode, AccessController.doPrivileged invokes actions directly and SecureClassLoader.defineClass skips the callback to get permissions for the protection domain. This part helps startup and runtime performance.
It's important not to get hung up on the details in the webrev, the important thing is understand if the security folks on this mailing list are open to a run mode that prevents a security manager being set in a running system.
-Alan. [1] http://cr.openjdk.java.net/~alanb/8191053/webrev/