Re: compression?
Chris, Weird, when going thru IIS to TC, it's not compressed: HTTP/1.1 200 200 Content-Type: application/json;charset=UTF-8 Content-Language: en-US Server: Microsoft-IIS/10.0 Date: Fri, 23 Jul 2021 16:34:30 GMT Content-Length: 3210105 That has likely nothing to do with TC. It's an IIS or reverse proxy thing. IIS fetches the requested resource from TC, acting as an HTTP client (or are you using AJP with IIS?). It gets a compressed response. What does IIS do next? It just uncompresses the resource and sends it back as response to its client (your browser). So, it's up to IIS or its reverse proxy module (whatever) to fix that. Maybe there is an option to tell IIS not to uncompress the response. Or you could add compression to IIS a well. Yes, in that case, TC compresses, IIS uncompresses and compresses again... :( Carsten - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Questions about Integrated Windows Authentication
Sorry Mark, I've clicked the wrong button in my mail client :( On 28.06.2021 15:29, Mark Thomas wrote: Note that Tomcat 7 is no longer supported. I guess it's nearly the same for all versions of Tomcat. That looks more like some form of configuration issue but I always found the Kerberos error message rather hard to decipher. AFAIK, the Kerberos is working fine. This error occurs in JNDIRealm's getPrincipal method. One log line before, Kerberos reports Found ticket for HTTP/apps.atlas-03t.gvsn.local@GVSN.LOCAL to go to krbtgt/GVSN.LOCAL@GVSN.LOCAL expiring on Thu Jun 24 18:26:05 CEST 2021 So, there is a ticket. However, JNDIRealm cannot use it or the ticket does not allow binding to the directory with that user. I'm not understanding the whole process, so I was asking if someone has more glue on that. 2. Fallback Authenticator It has been mentioned before. There is this on the Wiki: https://cwiki.apache.org/confluence/display/TOMCAT/SSLWithFORMFallback Will have a look at that. It's basically what I was thinking about adding a fallback to SpnegoAuthenticator only. As with most enhancements, whether it is accepted is going to depend largely on the benefit it brings vs how complex / invasive the code is. For sure. Rémy mentioned he was looking for a development project. Maybe this could be it. I guess, Rémy was taking my user attributes Realm extension as development project... You might be able to authenticate external users in a reverse proxy and have it pass the user ID to Tomcat rather than have Tomcat do the authentication. I read about that somewhere some months ago. However, I don't know how to get the authentication from the reverse proxy (my Tomcat already runs behind an Apache HTTPD using mod_proxy_ajp) to Tomcat? Finally, Tomcat needs the Principal and a couple of roles for authorization (including my additional user attributes). Passing the user ID only is likely not sufficient. Could you please describe that in more detail or point me to some sites to learn more about that? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Questions about Integrated Windows Authentication
Hi there, I have two questions about Tomcat's Integrated Windows Authentication: Tomcat is stuck on version 7.0.52 on an outdated Ubuntu 14.04 LTS. 1. useDelegatedCredential = true I'm using JNDIRalm together with the SPNEGO authenticator. If the Realm's option 'useDelegatedCredential' is set to true, I'm getting exception: javax.naming.AuthenticationException: GSSAPI [Root exception is javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]]; remaining name 'xxx.yyy.zzz...' Everything works fine when not using delegated credentials, but configured connectionName and connectionPassword. What's the reason for that? Is it a Tomcat configuration issue? Or, is the client (Google Chrome) not sending enough (credential?) information: There is Chrome's Policy option 'AuthNegotiateDelegateWhitelist' (deprecated, replaced by 'AuthNegotiateDelegateAllowlist') which must be configured in order to delegate the user's identity. However, setting any of these policy settings to true does not help. See https://www.chromium.org/developers/design-documents/http-authentication Is it a limitation/setting in Active Directory Server? The exception occurs after SpnegoAuthenticator has contacted ADS trough Kerberos. Is the response obtained from that Kerberos call not suitable for using delegated credentials? 2. Fallback Authenticator We have some clients connected through VPN, whose Windows sessions are not logged on to the Active Directory's Windows Domain, so Integrated Windows Authentication cannot work. SpnegoAuthenticator reports 'No authorization header sent by client'. However, I've not yet found a way to fall back to e. g. FORM authentication for those clients. AFAIK, there is no way to do this with Tomcat, since a Context can only have one single authenticator valve. Oracle's WebLogic Server support configuring more than one authentication method, by adding something like CLIENT-CERT,BASIC into web.xml. What about adding support for that or something similar in Tomcat? A CombinedAuthenticator (like with CombinedRealm) could be a solution. That could instantiate other required Authenticator valves and pass the request from one to the other until authentication succeeds. Those valves must not necessarily be queued in the container's pipeline, but could be called by the CombinedAuthenticator valve. Thats likely not too simple but it could be done. Are you open to such a solution? I believe that only the SpnegoAuthenticator needs such a fallback, as it uses mechanisms that you can't just add to any client. (In contrast, you could always distribute a X509 certificate or use FORM, BASIC, or DIGEST login. But you can't add your client to a Windows Domain just in order to log in to an application.) Adding a fallback mechanism to SpnegoAuthenticator only may be much easier. What other solutions do you know? Carsten What other solutions do you know? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Subclassing JNDIRealm to return a custom Principal
Roberto, On 21.06.2021 18:31, Roberto Benedetti wrote: Hello, in our product we subclassed JNDIRealm to return a custom Principal with attributes retrieved from Active Directory (bug 65391 is going to add support for that). We overrode authenticate(DirContext, String, String) to retrieve the attributes and return the custom Principal. In Tomcat 9.0.39 the method has changed to authenticate(JNDIConnection, String, String) and we cannot override it to retrieve the attributes because JNDIConnection.context is not visible to subclasses. Is there a work-around for that? Couldn't that new feature be an alternative for running your own implementation of the JNDIRealm? https://github.com/apache/tomcat/pull/428 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Where does Tomcat 9 (apt managed / standard config) store sessions in Ubuntu 18?
Gustavo, On 20.06.21 04:54, Gustavo Almeida wrote: "Configuration Reference - The Manager Component" says: "A Manager element MAY be nested inside a Context component. If it is not included, a default Manager configuration will be created automatically" My Context has no nested Manager element. So, guessing I'm using by default the Standard Manager Implementation, where does it store active sessions, if it does at all? Olaf is right. StandardManager keeps active sessions in Memory. However, when its _Persistence Across Restarts_ features comes into play (https://tomcat.apache.org/tomcat-10.0-doc/config/manager.html#Persistence_Across_Restarts), it persists active sessions to disk in order to preserve these across a restart of either the context or the whole server. Configuration option 'pathname' specifies, where on disk sessions are persisted. StandardManager serializes all sessions into a single file specified by 'pathname'. This defaulted to "SESSIONS.ser" and was a file path relative to the context's TEMP directory. However, on 2020-02-18 that was changed to null, effectively disabling the Session Persistence Feature by default due to possible performance issues with a large number of sessions. If you want it back, you could set 'pathname' to an absolute path of your choice. However, on Ubuntu, Tomcat runs in a sandbox and by default only has write access to a defined list of directories (see file tomcat9.service) as well as to /tmp and /cache. Option 'pathname' can also take a path relative to the context's work directory, which is in /var/lib/tomcat9/work (a symlink to /var/cache/tomcat9/). There, under Catalina/localhost> is a directory named after the context. That is the context's working directory. This is where "SESSIONS.ser" used to be. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Enhancement: Additional user attributes queried by (some) realms
On 01/06/2021 10:18, Mark Thomas wrote: I don't know if you can. I suspect not. By all means see if you can. I'm mildly curious to find out the answer. Whether you can or not, you don't need to. I found nothing to re-trigger the Travis CI build so far. However, now the CI test is 'successful'. Did you do anything about it? Carsten - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Enhancement: Additional user attributes queried by (some) realms
Mark, On 01/06/2021 09:28, Mark Thomas wrote: We have been seeing that a lot lately. As far as I can tell, it is an issue with Travis CI. Can you use the PR anyway? Can/must I re-trigger the Travis build? Carsten - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Encoding of LocalStrings_xy.properties files
Mark, On 01/06/2021 09:15, Mark Thomas wrote: Start Tomcat with: catalina jpda run (or start but I typically use run as I nearly always want to see what is logged to the console) In Eclipse go to Debug > Debug Configurations > Remote Java Application > New Configuration. Browse to the project and then click Debug. Tomcat's default jpda config matches Eclipse's so so should then have a remote debug session set up with your Tomcat instance. Trying that soon. Many thanks. Carsten - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Enhancement: Additional user attributes queried by (some) realms
Chris, On 28/05/2021 23:16, Christopher Schultz wrote: Yeah, about that... https://openjdk.java.net/jeps/411 IMO this is a Bad Thing for Java. If someone was looking for a reason to abandon the whole Java ecosystem, this would be it. Well, we had a good run. Now we can all run node.js, Python, or Go where security is not a problem because the languages are "safe" so nothing Bad can happen, right? *facepalm* Safe all the way down... including type safety :-p For now, Tomcat can rely on the SecurityManager doing its job. That means we only need to rely on the encapsulation strategies the language and the standard library provide, which are (currently) sufficient. I will try to come up with a solution that uses defensive copying whenever possible. That will include a couple of hard-coded special cases as well as try to use Serializable if available. I will not consider Cloneable due to the risk of shallow copies. If none of the tried methods will work, getAttribute(String name) should return the result of the object's toString() method. We'll likely "loose" some of the requested attributes that way but, need not care about preventing reflection with a (soon missing) SecurityManager. Carsten - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Enhancement: Additional user attributes queried by (some) realms
Mark, On 27/05/2021 18:56, Carsten Klein wrote: Concerning removal of class UserDatabaseRealm.UserDatabasePrincipal: I will provide a PR and file a corresponding issue in Bugzilla soon. My PR and Bugzilla issue are present. However, Travis CI build failed on arm64 architecture for the PR with "An error occurred while generating the build script." All other tests/jobs (including my local builds) were fine. Currently, I have no idea how to deal with that. Carsten - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Encoding of LocalStrings_xy.properties files
Mark, On 28/05/2021 10:35, Mark Thomas wrote: No doubt that UTF-8 is the better encoding for messages and language files. And yes, my Eclipse actually does not use the version built by Ant. I use the start-tomcat.launch configuration file for starting Tomcat. Actually it only takes a startup-class name. So, it must obviously use the JARs built by Eclipse. The trick is, that in the build.xml file, you are actually converting message files: Simple. However, you do that after having them copied. While copying, you use filtering-copy and specify ISO-8859-1 as the file's encoding: [...] Should be UTF-8 now? Back to the Eclipse. I guess there is not much difference between calling Ant from the console and using Eclipse's Ant support (Run As -> Ant build). But, how to start that with support for debugging in Eclipse (may be a dumb questing, I know)? Carsten - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Encoding of LocalStrings_xy.properties files
Hi there, I'm facing character set encoding problems in quite a recent Tomcat 10 setup. I noticed that with the http://localhost:8080/manager/html application in a browser (my browser) set to German language. My Tomcat runs from within Eclipse, built with the official build.xml file. I'm using my forked cklein05/tomcat GitHub repository, which is nearly up to date with your main branch. In the Manager application, there are texts which contain German umlauts, like "Lösche Sitzungen" (Expire sessions, aka htmlManagerServlet.appsExpire). These buttons now have captions that look like "Lösche Sitzungen". Obviously that's an UTF-8 <-> ISO--y conversion issue. I'm pretty sure that my setup is not causing that problems. After digging into GitHub, I found that recently someone converted many (or all) messages files to UTF-8: https://github.com/apache/tomcat/commit/90fe08bdee0494110bb8145d2f067b61f74ae429 However, since these language files are actually java.util.Properties files, these must be encoded as ISO-8859-1: https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html#load-java.io.InputStream- That's also true for more recent versions of Java. The language files are actually Properties files in a (according do Javadoc) "simple line-oriented format". These must be loaded with the Properties.load method(s) and must always be in ISO-8859-1. In contrast, there are XML-based Properties files, that must be loaded with method(s) loadFromXML(...). Only these must be encoded in UTF-8. Although editing international language files in ISO-8859-1 requires many \u escapes and is a hassle, for my mind, converting these plain-text language files to UFT-8 was likely not a good idea. But why don't others report that problem? Am I overlooking something? According to my explanation above, that problem is neither limited to German language nor to the Manager application. It should occur with any language using non-ascii characters (> 127) and with all localized text resources Tomcat is using. Carsten - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Enhancement: Additional user attributes queried by (some) realms
Chris, Mark, On 27/05/2021 22:11, Christopher Schultz wrote: After re-reading this, you mentioned reflection while asking how much we trust in Collections.unmodifiableMap(). I didn't get that right, my bad. However, I thought of reflection in order to implement a deep copy mechanism. Maybe anyone already coded one on jpm? :) If we return Collections.unmodifiableMap() to return Map that makes things simpler. But how much do we trust Collections.unmodifiableMap if the underlying data are security-sensitive? With reflection, is it possible to monkey your way through the references and swap-out the underlying references? That might be Bad. The question is, what kind of security problem are we actually trying to prevent? It is clearly not leak of information, since the administrator is responsible for what attributes are configured to be gathered. The administrator should clearly not provide sensitive information that should not be seen by the public. If so, his bad :-p Needless to say that I plan to deny querying passwords with that new Realm feature? Attribute names, for which I know they refer to fields containing passwords (like whats set for DataSourceRealm's option userCredCol), will *never* be queried. So, security here is about preventing that someone can modify that information during that Principal's lifetime. Can you modify an immutable object with reflection? Not sure. But, you could always use JNI and a C++ module to do nearly everything with the bytes in memory. Should we think that far? That would be a killer argument against using any sensitive data with Tomcat, Java or even computers in general :-) Carsten - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Enhancement: Additional user attributes queried by (some) realms
Chris, Mark, On 27/05/2021 22:11, Christopher Schultz wrote: What's the primary use-case for these kinds of attributes? This has been described in detail here: http://mail-archives.apache.org/mod_mbox/tomcat-users/202104.mbox/ajax/%3Cb9a2a913-f00f-f5bf-ca05-8ea4f8663ca9%40datagis.com%3E Trying brief: Basically, it is about getting some user-friendly user related attributes, like display name, department, phone nuber, Email address into our applications. Typically, such attributes live in the same user database, that has already been configured for the Realm for authentication and authorization. That database might be an SQL database (DataSourceRealm) or any directory server (JNDIRealm). Those attributes are often intended for display only (which user is logged on?). Another common case is sending mails from an application, which requires a valid From: address of the logged on user. The Realm will have a new configuration option, e. g. additionalAttributes, which takes a comma separated field (attribute) list. The Realm, which already knows how to access that user database, is the responsible for querying that attributes from the "user table" in that database (e. g. in JNDI, that is the User's DirContext). Without knowing anything about SQL or JNDI/LDAP queries, users can simply configure, what additional user attributes they like to get. Without that, users need to implement a "session initialized"-hook and access that user database "manually" and need some knowledge of that database. Also, credentials for accessing that database must be provided separately for that hook method. Unfortunately, Clonable and Serializable have both fallen out of favor in the Java world because both ideas seem to have some serious issues. If we pick one versus the other, we may have a lot of push-back. Too bad. In order to get it bullet proof, my idea is to have a real deep copy cloning mechanism in Java. That may be even quite slow, since we could check for a couple of simple but fast cases up front (e. g. objects that are by design immutable, like String, or some other commonly used objects, for which we can provide a fast clone with a copy constructor). Most of the attributes will likely be strings or be of any of the commonly used types so, only few object must go through the slow deep copy clone mechanism. Actually, my idea was to use Serializable for that. What's the point with Serializable? Isn't Tomcat using that for clustering/HA as well? The documentation of Serializable is quite promising... If we restrict attribute values to Strings, is that too limiting? I consider that a last resort option, only. Maybe we could agree on String plus a limited number of well-known, commonly used and (by design) immutable objects like Number, Boolean, Date (is Date actually immutable?). If we return Collections.unmodifiableMap() to return Map that makes things simpler. But how much do we trust Collections.unmodifiableMap if the underlying data are security-sensitive? With reflection, is it possible to monkey your way through the references and swap-out the underlying references? That might be Bad. Reflection was another idea. This my slow down things noticeably but, if we have enough well-known fast-cloning object types defined, that should not be too bad. On npm there are likely ten thousands of packages that support deep copying JavaScript objects. Where's the jpm for Java? Just kidding... However, the deep copy problem should have been solved already, likely many many times. Isn't there a common utility function available for Java somewhere on the Internet? What about Apache Commons? Can we use these in Tomcat? Carsten - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Enhancement: Additional user attributes queried by (some) realms
Mark, On 27/05/2021 18:19, Mark Thomas wrote: I will note that it isn't uncommon to have to log out and back in again to pick up newly allocated groups/roles in other environments. Yes, you are right. Didn't see it that way so far. We're talking about live updates for a session during its lifetime, which we can surely drop. Only few other Realms will support this (if any). If we go the removal route then I'd treat that as a separate PR / bugzilla issue so any discussion remains focussed on the relevant issue. The removal route seems simple. Both the UserDatabasePrincipal and the hasRole()-overwrite can be removed. That should do the trick. I will provide a PR and file a corresponding issue in Bugzilla soon. Carsten - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Enhancement: Additional user attributes queried by (some) realms
On 27/05/2021 10:59, Mark Thomas wrote:
As far as I can tell, removing UserDatabasePrincipal, relying on
GenericPrincipal and User remaining an internal object not exposed via
the Servlet API would achieve the same result with less code.
At this point I am looking for a reason not to remove
UserDatabasePrincipal and I'm not seeing one.
Since User and Role instances support dynamic updates (have a look at
methods addGroup/Role and removeGroup/Role), the special handling in
UserDatabaseRealm.hasRole() is required. Although that feature is
currently not used by Tomcat (is it?), those possible live updates must
be taken into account in the code.
If the UserDatabase was immutable, we could simply go ahead with the
pre-resolved effective roles list and GenericPrincipal. But actually,
those effective roles may be wrong after an update of the UserDatabase.
So, I suggest not resolving effective roles up front in
UserDatabaseRealm.getPrincipal() and use null as the roles list when
constructing the new GenericPrincipal instance.
The special handling in UserDatabaseRealm.hasRole() is triggered, if the
Principal is a GenericPrincipal with a userPrincipal of type
UserDatabasePrincipal:
if (principal instanceof GenericPrincipal) {
GenericPrincipal gp = (GenericPrincipal) principal;
if (gp.getUserPrincipal() instanceof UserDatabasePrincipal) {
principal = database.findUser(gp.getName());
}
}
The current test requires an extra UserDatabasePrincipal class. Making
that class support attributes and serializable is possible.
However, a simple default/package visible boolean flag
'userDatabasePrincipal' in GenericPrincipal would work as well and lets
us finally drop class UserDatabasePrincipal.
I think it would be worth handling this is a separate commit to give
folks the chance to review it before proceeding to add attribute support.
+1
Are you talking about a separate commit in my additional-user-attributes
branch? Or is that worth an extra BUG/case/issue?
I suspect the users that worry about that sort of thing aren't using the
UserDatabaseRealm but it would be nice to fix that anyway.
That will be fixed after having added attribute support either way.
Carsten
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Enhancement: Additional user attributes queried by (some) realms
Hi Mark,
thanks for sharing your ideas :)
On 26/05/2021 19:56, Mark Thomas wrote:
Given that the attributes may well be security related, you would need
to make sure neither the Map nor any of the keys/values could be
modified. Protecting the Map is easy. Protecting the keys/values is a
little trickier. For that reason I'd lean towards the solution below.
Oh yes, these attributes should likely be immutable. Since I still
believe that Enumerations are kind of uncomfortable (and outdated?),
what about strictly relying on Collections.unmodifiableMap?
But, I'm ok with getAttribute(String name) and getAttributeNames() pair
as well. You decide :)
My first idea was to allow users to put their own attributes as well.
But then, we'd need to track Realm-sourced attribute names and make
those read-only in a setAttribute method. Since users could as well put
their stuff into session attributes, making the whole map immutable is
possible and clearly much simpler.
2. Should I add attributes-related public methods to the
TomcatPrincipal interface as well?
Yes. So far the only Tomcat specific extensions are for SPNEGO but this
is in the same category. The other option would be a new interface but I
don't see a need for that.
Since UserDatabasePrincipal is a private inner class of
UserDatabaseRealm, no instanceof checks are possible to determine
whether attributes may be available or not. So, I was thinking of a new
interface
public interface AttributedPrincipal extends Principal {
public Object getAttribute(String name);
Enumeration getAttributeNames();
}
That interface has a slightly more meaningful name for consumers of
additional user attributes. But adding these methods to TomcatPrincipal
is good as well.
3. Class UserDatabasePrincipal in UserDatabaseRealm
Why does UserDatabaseRealm pass a userPrincipal of type
UserDatabasePrincipal? Can't we just drop that and do it like
JNDIRealm or DataSourceRealm?
I don't see any obvious reason. I'll do some digging in the source
history to see if I can find out why. Absent a good reason, I'd say
drop it.
There is a good reason for it, but I think it should be possible to drop
it.
It is there because the UserDatabaseRealm supports the concepts of
groups. Users can have roles assigned directly or users can be assigned
to a group and inherit the roles of the group. This means hasRole() is a
little more complicated and the UserDatabasePrincipal is used to
determine if this additional processing is required.
I think this could be replaced by a
"org.apache.catalina.realm.UserDatabaseRealm.groups" attribute which
would remove the need for the dedicated UserDatabasePrincipal
Honestly, I don't see that in the code. What I see is, that
UserDatabaseRealm resolves roles coming from groups up front in its
getPrincipal method. After that, it just creates an ordinary
GenericPrincipal instance with a single list of effective roles.
The associated UserDatabasePrincipal does nothing special with
hasRole(). Its only method is getName(). So, it just removes or "hides"
any special GenericPrincipal methods like hasRole() and getRoles() from
the user (aka from the Servlet code, internally Tomcat still uses the
surrounding GenericPrincipal, e. g. for authorization).
So, the only thing UserDatabasePrincipal does, is to hide the fact that
groups have already been resolved to a single list of effective roles,
the Principal is working with during its lifetime. Did I overlook something?
Actually, when declaring the attributes-related methods in a public
interface (either TomcatPrincipal or AttributedPrincipal), adding that
interface to UserDatabasePrincipal is not expensive. So, we could always
decide to leave that UserDatabasePrincipal in place and let it implement
that interface.
However, in order to not "pollute" UserDatabasePrincipal with unneeded
methods, I'd vote for using a new
org.apache.catalina.AttributedPrincipal interface.
Additionally, class UserDatabasePrincipal is NOT serializable. That
means, it gets dropped when sessions and principals get persisted and
reloaded. That applies to session persistence across restarts (with
persistAuthentication set to true) and likely to clustering/HA. That's
not fatal, but after a restart or when running on a different cluster
node, users suddenly get a full-blown GenericPrincipal instance when
they call Request.getUserPrincipal(). However, since nobody complained
about it, the UserDatabasePrincipal is likely not so important :-p
Carsten
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Enhancement: Additional user attributes queried by (some) realms
Hi there,
as already discussed here:
http://mail-archives.apache.org/mod_mbox/tomcat-users/202104.mbox/ajax/%3Cb9a2a913-f00f-f5bf-ca05-8ea4f8663ca9%40datagis.com%3E
I'm implementing an enhancement for querying configurable extra user
attributes through some of the Realm classes from the "user database"
configured for that Realm.
Before starting off, I want do clarify some questions.
The implementation basically of consists two parts. One is to enable the
GenericPrincipal class (and/or TomcatPrincipal interface) to hold these
additional user attributes. The other part is to query these attributes
from the Realm's configured user database an to store these in the newly
created Principal instance.
The second part is quite clear and changes are only local to the Realm
classes. But some decisions must be made for the first part.
The Principal's attributes will be stored in a Map hash
map. Since attributes may come from various sources, using an
Object-typed value seems appropriate to me. That map will also be added
to internal class SerializablePrincipal in order to be persisted properly.
Now, my questions:
1. How to access the Principal's new attributes
Simplest is to provide a getter method, that actually returns the map
(optionally with a read-only parameter):
/**
* The set of additional attributes associated with the user represented
* by this Principal.
*/
protected final Map attributes;
/**
* Returns the user's additional attributes as a Map.
*
* @param readonly
*if true, the returned map is an unmodifiable
*view of the map; otherwise the underlying map is returned
*unchanged
* @return the user's additional attributes as an unmodifiable Map
*/
public Map getAttributes(boolean readonly) {
if (readonly) {
return Collections.unmodifiableMap(attributes);
}
return attributes;
}
/**
* Returns the user's additional attributes as an unmodifiable Map.
*
* @return the user's additional attributes as an unmodifiable Map
*/
public Map getAttributes() {
return getAttributes(true);
}
However, in the jakarta.servlet classes, there are also some entities
that have attributes. There, attributes are accessed by these methods:
Object getAttribute(String name);
Enumeration getAttributeNames();
void setAttribute(String name, Object o);
Would you, for the sake of uniformity, prefer using these more
servlet-spec like methods?
2. Should I add attributes-related public methods to the TomcatPrincipal
interface as well?
According to the documentation of TomcatPrincipal I likely should:
Defines additional methods implemented by {@link Principal}s created by
Tomcat's standard {@link Realm} implementations.
However, this interface seems only being used with GSS (most other
instanceof tests are performed against GenericPrincipal).
So, I guess the attributes stuff should better NOT be added to
TomcatPrincipal?
3. Class UserDatabasePrincipal in UserDatabaseRealm
Although neither documented nor used in Tomcat's shipped standard
tomcat-users.xml, the UserDatabaseRealm supports the additional
attribute "fullname" for a user entry. This makes this Realm a good
candidate for supporting additional user attributes (actually, the
authenticated user's display name was the initial intention for this
enhancement).
The UserDatabaseRealm creates a Principal with an explicitly set
userPrincipal (why? e.g. DataSourceRealm and JNDIRealm don't) of type
UserDatabasePrincipal. The latter does not extend GenericPrincipal class
and so, will not have the new attributes-related methods by default.
However, since Request.getUserPrincipal returns that "inner"
userPrincipal by calling getUserPrincipal(), attributes should be
available for that UserDatabasePrincipal as well.
My Questions:
Why does UserDatabaseRealm pass a userPrincipal of type
UserDatabasePrincipal? Can't we just drop that and do it like JNDIRealm
or DataSourceRealm?
If not, I need to add the attributes-related methods to the
UserDatabasePrincipal as well.
Carsten
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Getting additional attributes for logged on users
Hi there,
are there other comments on that? I'd like to implement this and want to
provide a CR On GitHub, if there is a fair chance, that this enhancement
will make it into Tomcat :)
Trying to summarize what has been discussed so far: (read the other
posts for more information)
The basic idea is to obtain additional user attributes and providing
these to the application through an "attributes" map of the Principal
class (e. g. the user's display name, e-mail address, etc.).
Since such user (-related) data is often available in data stores also
used for authentication, it shall be up to the Realm to query these
extra attributes. (Additionally, the Realm already has access data and
credentials to access that data store and also creates and populates the
Principal instance.)
The provided extra user information is not crucial for the Realm and its
authentication and authorization process.
Olaf Kock was suggesting to agree upon a set of "standard" additional
user attributes, which the Realm should query for by default. However,
for at least these reasons, I believe that this is just not possible:
1. Every application's needs are different
2. Every user data store is different
3. Cultural differences may require different attributes to form names
or addresses, for example
In contrast to that, my initial idea was to specify the desired extra
user attributes as a comma separated list though a new configuration
option of the Realm (called "extraAttributes" or "userAttributes"). This
list is nearly equivalent to a SELECT clause in SQL. However, that
"SELECT clause" shall work as well with Realms not based on SQL.
Not all Realms can/shall support that optional new feature. For example,
JAASRealm, which is merely a wrapper around an arbitrary authentication
and authorization process behind the curtain, likely cannot participate
in this.
The queried attributes shall be provided to the application through a
new "attributes" map of the Principal instance. Accessors to this map
should be declared in either TomcatPrincipal or GenericPrincipal. Still
to discuss, what accessors to implement:
Map getUserAttributesMap() {}
vs.
Object getUserAttribute(String name)
String[] getUserAttributeNames()
void setUserAttribute(String name, Object o)
void removeUserAttribute(String name)
The attribute names specified in the configuration option (SELECT
clause) must be the actual field names of the attribute, of course.
These also form the names/keys of the attributes in the Principal's
attribute map (could/should be prefixed with a namespace or prefix):
Realm-Configuration:
userAttributes="attr1, displayName, dept_no"
Using the attributes:
String s1 = principal.getUserAttribute("attr1")
// with a namespace/prefix (configurable and with a default value?)
String s2 = principal.getUserAttribute("realm.usrattr.displayName")
I believe that implementing the discussed enhancement is not too
difficult, since only (some of) the Realm classes as well as the
Principal implementation must be modified (TomcatPrincipal and/or
GenericPrincipal). Also, I do not see any security related risks.
Carsten
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Getting additional attributes for logged on users
Hi Olaf,
also, I only picked your statements for an answer.
On 08.04.21 13:53, Olaf Kock wrote:
It would be cool, absolutely.
Even cooler if "the desired extra attributes" could be agreed upon - or
even what would be desirable in user management (is user + roles enough?
That requires redeployment any time the system gets more picky with
permissions.
Personally, I work with Liferay, where roles are runtime-configurable
and map to permissions. The permissions are what's checked in the
application code, and the applications couldn't care less about number
of roles or path through which a permission is assigned to a user.
I'm not sure whether I got you right. For my mind, my idea has nothing
to do with authorization, roles and permissions. Also, I don't think
that the set of desired extra attributes is something that could be
agreed upon. It's rather very specific for a project and/or customer.
The basic idea is just to get some of the user's attributes/properties
for (any/free) usage by the application. Actually this proposal only
(ab)uses the Realm, since it already has configured access to a storage,
that one might call the 'user database'.
However, these attributes are only used by the application. These are
not used for authorization or grant of permissions.
You first state "its simplest form", then state that more complex
solutions are likely not needed.
I'd argue that the current form is "the simplest form", as it provides
identifiers that can be used to look up extra information according to
the application's need.
What can be simpler than a comma separated list of field names? If we
can agree upon the set of desired user attributes (which we can't
surely), no configuration will be required. That's would be simpler but
won't work, since every user database is different and desires are too.
I don't want to tear down your idea - only reflect that "only a little
bit more" will be barely enough for the next person coming around, and
end up being a really complex beast. And if you look at all the
different available attributes for users that make sense in some, but
not in other countries/cultures, you'll soon be implementing your own
anyway:
* What makes up the name? First/Last Name? Including middle name (common
in US, almost nonsensical in Germany)? There are cultures with just a
single name
* Gender: How many choices and what will they be named (by law, Germany
now has 3 genders)
* Address: I've recently sent out letters to international addresses,
and boy was it hard to ask for the right information to go to the
address sticker.
* Department: Makes sense in a business environment, not at all in a
community environment
That's why I recommend that the desired attributes must be specified in
a list. That list can be seen as the SELECT clause of an SQL statement
(but this will work with an Active Directory Server and likely
tomcat-users.xml as well).
In one of our projects, we authenticate users against the customer's
Active Directory Server. The sAMAccountName attribute is used as the
user's logon name. That's what the principal returns in its getName()
method. For example, we output that name in the footer of printed
reports. On reports created by me, 'c.klein' is shown.
In the ADS, my user class also has these attributes: (among many others)
displayName: ext. Klein, Carsten
givenName: Carsten
name:ext. Klein, Carsten
sn: Klein
(ext. = I'm an external user)
So, in order to show a more user-friendly name on the reports, I'd
configure the extraAttributes list to contain attribute 'displayName'.
Since our reports are template driven, I could as well retrieve fields
'givenNamen' and 'sn' and build my on report specific display name like
'${givenName} ${sn}' (with the knowledge in mind, that it's a German
customer and names in Germany typically are written that way).
While you name some samples that sound agreeable on first sight, I don't
think that they're agreeable enough to impose them on people unless
provided voluntarily and under arbitrary names. Which ends up with the
cost of a huge framework in order to potentially save a single database
request per log in.
I guess, we will never find a bunch of attributes everybody can agree
with in all countries, companies and projects. So, there will likely
never be a mechanism that provides extra user attributes voluntarily.
But that was never announced. Some minimal and simple configuration is
still required.
For the same reason these attributes cannot be provided under arbitrary
names. But we could simply use the configured field names (which must be
the actual field names):
Config:
extraAttributes="displayName, givenName, sn"
Code: (no safe type checks here)
Principal p = ((TomcatPrincipal) request.getUserPrincipal());
out.writeln(p.getAttribute("displayName"));
It's not about saving a single database request per log in (at least not
from a performance point of view,
Getting additional attributes for logged on users
Hi there, from a servlet, the logged on user is represented by a Principal instance, which has a getName() to get the logon name of that user. In most cases, that is a more or less cryptic name like c.klein or m.scott. Some of our customers like the name of the currently logged on user be displayed in the application's GUI. We all know that from many popular applications. While relying on the Servlet specs only, we can just show that non-user-friendly logon name. Our customers prefer seeing the user's display name. Other attributes of the user, like e-mail address, phone number, department etc., is often welcome (if not required) information during an application's lifetime. It's not too hard to hook into session initialization and get the required extra information, for example through JDBC or JNDI, if the user database is actually an Active Directory Server. That information can then be stored in the user's session attributes map. However, one has to implement that information retrieval "by hand" for every application and/or customer. Even worse, typically credentials are required to access the storage in which that information lives. If it's in the application's 'standard' database, there's a good chance to have a Tomcat connection pool. However, in general, you need credentials and access data to get these additional attributes (credentials and JDBC URL may be configured in Context or Servlet parameters). Does anybody know a more general way to get such extra user attributes? My proposal: Typically, those desired extra attributes are stored in the user's record in the user database. That may be a SQL database or may be an Active Directory Server (or any other directory), which is already used for authentication and authorization. Wouldn't it be cool to make the Realm get us that extra information? Why the realm? First, in order to perform authentication, the Realm already has access to the user database. So, access data and credentials must not be configured twice at different places. Second, the Realm knows how to query the user database and does that already for retrieving roles and passwords. Getting some more fields shouldn't be a big deal. Third, the Realm actually creates and initializes the Principal instance so, it should be easy to store these extra user attributes in an extended version of the TomcatPrincipal class. A simple MapString> should be sufficient, but for the sake of uniformity, the well known methods getAttribute, getAttributeNames, removeAttribute and setAttribute may be a better option. That's clearly something not all Realms can/shall support. Likely JAASRealm will not, since it uses custom Principal classes. But JNDIRealm, JDBCRealm and DataSourceRealm (not yet sure about UserDatabaseRealm) could easily support that feature. In its simplest form, these Realms get a new configuration property 'extraAttributes', which takes a comma separated list of field names to retrieve. Implicitly, for an SQL-based Realm, these fields are queried from the 'userTable' table. The JNDIRealm tries to find these attributes from the user's entry in the directory, of course. More complex configurations are possible (but likely not needed). I'm curious what you think about it :) Carsten - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] programming style or mental process ?
Guten Morgen Peter :) Guten Morgen Carsten and All [...] I may add, that a French Café au lait (pronounced Olé, which is not French :-O ) is also an option. Maybe my German coffee expertise is kind of narrow and shaped by personal experience (having my coffee w/o milk)... OLE? Isn't it something used in Windows programming? *lol* Obviously, coding and coffee are very closely related... Carsten - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] programming style or mental process ?
(And dutch people. Where are they in this discussion by the way ? (but
they have only one type of coffee I think)).
Dutch people may only have one type of coffee (actually I don't know).
But remember, Dutch people have 'Coffee Shops' offering stuff far beyond
coffee... :) Is there a relation between that and usage of Yoda style?
Germans used to drink so called 'filter coffee' for decades, which
today, even if served hot, many people would call 'cold coffee' ('kalter
Kaffee' in German), an idiom that could be translated to English as 'an
old hat').
Now, thanks to companies like De'Longhi or Seaco (which is now owned by
Philips and so is actually Dutch), most of us prefer Italian coffee
types like Espresso, Cappuccino or Latte Macchiato.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Adding regular expression support to CORS filter
Chris, On 9/28/20 02:40, Christopher Schultz wrote: Carsten, On 9/27/20 05:53, Carsten Klein wrote: Any comments on that? Is it worth preparing a PR? Regular expressions are fairly expensive. Yes, but my measurements of the HashSet-lookups were wrong, since hashValue() of a String gets cached, so, while measuring in a loop, the hash value gets computed only once. Setting up a fair test is challenging (using new strings in the loop causes memory allocations and GCs). I ended with additionally calling a clone of the String's hashValue() function in the loop. Now, testing an origin with HashSet.contains(origin) (current solution) takes about 19 ns for a cache-miss (empty list in bucket) and 30 ns for a cache-hit) on my box. In contrast, evaluating a regular expression takes about 120 ns; so these are about 6 times slower (NOT 25 times as stated in my last mail). The creation of a new Matcher instance each time is a significant bottleneck (in my measurement loop): reusing the same Matcher instance and resetting it with a new input string makes the test taking only about 75 ns (that's only about 4 times slower that the HashSet test). So, regular expressions are not as bad as we were thinking? If there is a way to build the code such that some subset of wildcards can be serviced without regex (and of course exact matches without using regex), then I'm at least a +0 on this. I never intended to implement tests for exact matches with regular expressions. Configured "exact" origins (those without wildcards and not enclosed between slashes) will still be tested with HashSet.contains, of course. So, there's no change (considering performance) for Tomcat setups only using "exactly" defined allowed origins. If someone uses the new "inexact" origin specifiers, will it not be comprehensible that these are more expensive (from a performance point of view)? It may seem like over-engineering, but maybe creating a Matcher interface and a Factory (I know, I know) which produces Matchers which implement the optimal strategy for a particular value would be a good way to do this. A single matcher could be used for really simple values and maybe a MultipleMatcher could be used for multiple different value-checks. Something like that will likely lead to the highest-performing filter processing. I did some tests on that. As I don't believe, that a (self-made) NFA-based solution could outperform Java's regular expression engine, I was looking for a different (simpler) algorithm. I started with a rule-driven globbing algorithm, that supports ? (any char except "."), # (any digit, using Character.isDigit) $ (any letter, using Character.isAlphabetic) as well as literal character sequences. (Actually, I followed your suggestion. Your Matchers are my Rules together with a piece of code in a switch-case block. Using real classes/instances for the matchers requires method invocations and maybe instanceof tests. Both are adding extra overhead, so I decided to use a more C-like approach.) That simple algorithm takes about 42 ns and so, is still 2 times slower than the HashSet test. I already made more than half the way down to support * and ** multi-matching wildcards. That implementation uses non-recursive backtracking, similar to the algorithms described at https://www.codeproject.com/Articles/5163931/Fast-String-Matching-with-Wildcards-Globs-and-Giti With * and ** partly in place, time consumption is about 50 ns. The code additions for making the algorithm work on the many edge cases will very likely add more nanoseconds to the test so, we may soon end at 60 ns or even more. That's almost the same time required for evaluating a regular expression (without the time needed to create the Matcher instance). The algorithm is optimized and uses only a few method calls and no OOP constructs (by using the Rules, which are like beans that specify what to match next, the whole logic can be implemented in a single method). But it's still not much faster than a Java regular expression test. I don't believe, that it's worth to (self-)implement such a rather complex (error prone) and hard to understand (and maintain) algorithm, if it's not significantly faster than real Java regular expressions. Anyhow, if performance should not degrade due to using wildcards in the allowed origins, the goal is not just to be better than Java regular expressions, but to be close to the HashSet test (~19 ns). And, if real regular expressions shall be used as well (enclosed between slashes), that all will not help much for these. That's why I wanted to combine this with a HashMap-based (LRU-)cache, so that regular expressions must only be evaluated if the current request's origin is not yet in the cache. This cache's performance nearly equals the performance of the HashSet test (depending on whether real LRU is used or not). That way,
Re: Adding regular expression support to CORS filter
Any comments on that? Is it worth preparing a PR?
Adding regular expression support to CORS filter
Hi there,
I'd like to contribute a CORS filter enhancement, making it accept both
wildcard-based and 'regular expression'-based expressions for its
allowed origins list.
I know this from a project based on Jetty, which has support for, at
least, simple wildcard matching (*). Specifying multiple allowed origins
with one pattern comes quite handy if there are numerous developer
machines that all access one single server from their browsers.
The implementation shall support two flavors of expressions:
- Java Pattern Class based expressions
Enclosed between slashes (/.../) these bring the full power of regular
expressions to the filter's allowed origins list configuration. This
will also support specifying pattern flags by appending single
characters after the terminating slash (like in /^foo.bar$/i for
case-insensitive matching).
- Wildcard-based simple expressions
With a much simpler syntax, these expressions are more compact and may
be more intuitive for people not familiar with regular expressions.
Although less powerful than *real* regular expressions, the special
characters supported should provide enough options to specify allowed
origins efficiently:
? matches any single character except the domain separator (.)
# matches any single digit
* matches any number of any characters except the domain
separator (.) including none
** matches any number of any characters including none (this also
matches the domain separator and so matches several sub-
domains)
(Technically, any number > 1 of consecutive asterisks are
treated as a single ** pattern.)
[abc] not yet sure about character classes
[a-z]
[^abc]
Wildcard-based expressions are implemented by regular expressions as
well. Such an expression is turned into an *anchored* regular expression
during initialization e.g.
http://?*.devzone.intra ==> ^http://[^.][^.]*\.devzone\.intra$
Of course, it is still possible to specify literal origins, as well as
the sole '*' to allow access from any origin. So, the value of the
'cors.allowed.origins' initialization parameter may look like this:
https://www.apache.org,
http://?*.devzone.intra,
/:\/\/staginghost-\d{1,3}.mycompany.corp$/i
As you can see, *real* Java regular expressions are not anchored by
default. The above one is end anchored only (making it match any protocol).
Obviously, forward slashes withing the regular expression must be
escaped in order not to end the expression prematurely.
The current CORS filter implementation uses a HashSet to
determine whether a given origin is valid or not. That's a quite fast
solution. Since evaluating a regular expression is much more expensive
(~25 times slower), a sort of caching mechanism is required.
The idea is to transparently add positive matches (allowed origins) to
the same HashSet that already contains all literally specified
allowed origins. Since these positives form a (rather small) countable
set (in practice) we could simply just add these without worrying about
cache removal. There is no difference in memory consumption compared to
the current implementation: all allowed origins must be stored in that
hash set.
There *may* be more disallowed origins than allowed ones so, another
cache for non-matching origins is certainly a good idea. However, this
adds extra memory consumption and that cache should not grow with no
limit. The idea is to use a LinkedHashMap with access order ordering
mode (accessOrder = true). This makes the map an LRU-cache, removing the
least recently used entry when a defined maximum capacity is reached
while adding a new entry. Maybe that cache's capacity should be
configurable.
Write access to that caches must be synchronized. At current, I tend to
use a ReadWriteLock.
So, the new algorithm for determining whether an origin is allowed or
not is like so:
private boolean isOriginAllowed(final String origin) {
if (anyOriginAllowed) {
return true;
}
if (allowedOrigins.contains(origin)) {
return true;
}
if (notAllowedOrigins.containsKey(origin)) {
return false;
}
// synchronized block starts here (using a ReadWriteLock)
boolean result = false;
for (Pattern pattern : allowedOriginPatterns) {
if (pattern.matcher(origin).matches()) {
allowedOrigins.add(origin);
break;
}
}
if (!result) {
// origin is definitely not allowed
notAllowedOrigins.put(origin, null);
}
// release lock here
return result;
}
As you can see, there are not more than N extra evaluations of a regular
expression required for an allowed origin during the filter's lifetime
(when the origin is used for the first time), N being the number of
different expressions configured.
The same is true for not allowed origins if the LRU cache is large and
if there are requests from only few different
Re: How to encrypt db password in tomcat context.xml
Hi Jürgen and Olaf,
I can really understand Jürgen's intentions. The core problem is not
security but administrators and so called security panels in
"professional" (non-open source related) companies. I really know this
from my own experiences. Maybe that's a German problem, since Germans
are said to be over-correct? Sometimes, that turns into paranoia...
(I'm from Germany, so I know this circumstances quite well, sounds
like Jürgen is German as well...)
True is, that there are administrators, which have very little
knowledge of software in general and security. Those tend to stick to
their personal categorical rules, which often are far off from what is
considered sensible by real IT and software professionals.
Furthermore, there are "security" panels, working out policies for a
company. These often only consist of people with very little *real* IT
an security knowledge.
The (sad) point is, that the policies passed by such a council are
actually valid and no one ever again asks whether these make sense or
are *correct* from a security professional's point of view. Changing
user passwords on a regular basis (e.g. after 90 days) still today is
one prominent example of that.
So, in order to make Tomcat fit into such "professional" company
environments easily (w/o requiring people to implement their own data
source class), it may be a good idea to add such a "encrypted
password" feature to Tomcat. Consider this as pure "syntactic sugar"
and keep in mind, that it's NOT a security feature (need to emphasis
that in the docs).
My idea is, that *all* passwords read by Tomcat MAY be
encrypted/obfuscated with a small number of algorithms. The algorithm
applied to the password could be prefixed like Jürgen suggested:
password="+duk6<7v@LD#"(plain, no encryption)
password="base64:K2R1azY8N3ZATEQj" (base64 obfuscation)
password="3des:hkph5ewjEwv70CvTB16v/w==" (3DES with hard-coded key,
expressed as base64 string)
The decoding algorithm could be implemented in a common util method
String decodePassword(String password) in Tomcat, so it could easily
be called from all those places at which Tomcat actually reads a
password.
Also, a small separate tool should be provided, which encodes such
passwords (like htpasswd does for httpd). However, it should be
sufficient to just print the encoded password to standard out. Then,
the user is responsible for copy and pasting it into the config file.
I offer my help for writing such an enhancement, since I believe that
it's a way to make Tomcat more out-of-the-box usable in such
"professional" company's environments (for some people it may be the
only way).
Again, I know this is NOT a security feature as it adds no extra
security to Tomcat. However, I may make some administrators and CEOs
happy, that are solely guided by questionable policies they don't
understand.
Some ideas on that?
Carsten
On 28.06.2020 23:49, Jürgen Weber wrote:
I'd just put some nice password as byte[] into Tomcat's source code
and provide a way to have passwords in the configs encrypted with that
nice password.
Use properties replacement so that in the xml config you have
${db.password} and in conf/catalina.properties you put the password
there.
So one could have samething like db.pass=3des: in
catalina.properties
Greetings, Juergen
Am So., 28. Juni 2020 um 21:19 Uhr schrieb Olaf Kock :
On 28.06.20 19:50, Jürgen Weber wrote:
>>>> I would like to know how to encrypt and decrypt the database
password in
>>>> context.xml when the application is running which also allow
me to change
>>>> the db password for the purpose of security.
>> https://cwiki.apache.org/confluence/display/TOMCAT/Password
> Well, I know a chief open source app server that has the password to
> decrypt all passwords buried in its open source, and I know auditors
> who are good if root cannot read passwords at first sight. The
> reasoning behind that is that running java -jar someappserverlib.jar
> -decrypt is a deliberate act that a god guy root does not do. So a
> hidden password is a step better, even if not in the cryptographic
> sense.
Hi Jürgen,
I don't get your point here. Are you arguing that the linked wiki
article is incorrect, insufficient or invalid?
Because I believe that the article documents how to implement everything
that you describe on your own, and gives arguments for why this is not
implemented out of the box.
Best,
Olaf
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For addition
Re: Should Tomcat 10 enable response compression by default?
Although I believe that buggy clients are no longer a problem today, compression may introduce complications when Tomcat runs behind a reverse proxy as it is often the case. If your front-end server (e.g. Apache) needs to modify the responses (e.g. with mod_proxy_http), you'll end up with a quite expensive compress -> decompress -> compress processing chain on the back end. That is obviously no good idea from a performance and resource consumption point of view. So, I'm actually with Rémy. Leave it off by default and point out that there is an "on" available by explicitly writing compression="off" to the connector. Carsten On Tue, Jun 9, 2020 at 10:20 PM Mark Thomas wrote: Hi all, An enhancement has been opened to enable response compression by default: https://bz.apache.org/bugzilla/show_bug.cgi?id=64431 In short, the proposal is to change the default for the Connector's compression attribute from "off" to "on". This would be for Tomcat 10 onwards only. The following would be unchanged: - compressibleMimeType - compressionMinSize - noCompressionStrongETag It would be helpful to know what the range of views of the user community are on this proposal. So, thoughts? Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Enhancement: New option 'persistAuthentication' for session manager
Hi there, what to do next to get things moving? Seems like there's nothing more to implement for the addition right now. Should I just create a PR in the apache/tomcat repository? And/or post some lines to the dev mailing list? Carsten - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Enhancement: New option 'persistAuthentication' for session manager
Jonathan, On Tue, Feb 18, 2020 at 5:29 PM Jonathan S. Fisher wrote: For older versions of tomcat, I'd suggest adding an additional option that causes GenericPrincipal to drop the password after authentication is complete, by default if persistAuthentication is enabled, which alleviates your concern. Tomcat's GenericPrincipal implements Serializable only as of the 8.5 branch. However, the latest 8.5 version uses a helper class SerializablePrincipal (private static class in GenericPrincipal). In both 8.5 and 9.0 branch, this class also serializes the password, whereas in current 10.0 branch, there is no longer a password available in GenericPrincipal. Although it would be a good idea to remove the password field generally from GenericPrincipal even in 8.5 and 9.0 (as in 10.0), it's likely easiest to just remove it from SerializablePrincipal in the older versions. So, even if the user's password is stored in memory, its no longer persisted to storage and does not blame session and authentication persistence. Carsten - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Enhancement: New option 'persistAuthentication' for session manager
Chris, there is a new PR #2, I messed up my branch and had to setup a new one... Still a bit new to GIT... https://github.com/cklein05/tomcat/pull/2 Actually, I decided to just add the String array. That's not too bad either, right? Have a look at the latest code and comment in PR #2. Sorry for the confusion :-( Carsten - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Enhancement: New option 'persistAuthentication' for session manager
Hi there, had to re-setup my branch... Nevertheless, updated PR is available now. Some last things... The pattern for default sessionAttributeValueClassNameFilter must even be extended to match String arrays as well (roles are stored that way). In order to keep the pattern smaller, one option is to allow arrays for all "base types": Current pattern: "java\\.lang\\.(?:Boolean|Integer|Long|Number|String)" New pattern (short version): "(\\[L)?java\\.lang\\.(?:Boolean|Integer|Long|Number|String);?" + "|org\\.apache\\.catalina\\.realm\\.GenericPrincipal\\$SerializablePrincipal" In terms of CVE-2016-0714, an array of any of these types is not really more insecure, right? Otherwise, I could add String arrays to the default pattern explicitly: Current pattern: "java\\.lang\\.(?:Boolean|Integer|Long|Number|String)" New pattern: "java\\.lang\\.(?:Boolean|Integer|Long|Number|String)" + "|org\\.apache\\.catalina\\.realm\\.GenericPrincipal\\$SerializablePrincipal" + "|\\[Ljava.lang.String;" Shall I update these patterns in context.xml files of the manager and host-manager apps as well? BTW, thanks a lot to all or you for supporting me :) Carsten - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Enhancement: New option 'persistAuthentication' for session manager
Hi there, most of the issues discussed with/suggested by Mark Thomas should be in place now. The renamed PR should now show a much better code basis for further discussions. b) Please add a changelog entry for this addition. Still looking for the change log file... Carsten - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Enhancement: New option 'persistAuthentication' for session manager
j) At a minimum, new Manager attributes need to be added here: https://github.com/apache/tomcat/blob/master/webapps/docs/config/manager.xml Here's an online preview of the updated Manger documentation: http://office.datagis.com/pub/tomcat-9.0.x/docs/config/manager.html These are the changes: - added persistAuthentication for both StandardManager and PersistentManager - updated default regular expression pattern for sessionAttributeValueClassNameFilter (both StandardManager and PersistentManager) - added some notes under Special Features \ Persistence Across Restarts Native speakers (English!) are encouraged to drop some comments focusing on language and translation... :-) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Enhancement: New option 'persistAuthentication' for session manager
Open the pull request in your own fork... this link should work: https://github.com/cklein05/tomcat/compare/cklein05:master...cklein05:session-manager-persist-authentication?expand=1 Done. Carsten - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Enhancement: New option 'persistAuthentication' for session manager
Mark, Please don't be put off by the number of comments and suggested changes. I think the core idea is sound and meets a valid requirement that some users have. To some extent, the volume of comments reflects that fact I'm responding to a clear proposal and explanation. This is a good thing in my eyes. In the order I thought of them: a) Please don't add author tags. ASF policy is not to add them. The ones you see in the Tomcat codebase pre-date that policy. You will be credited in the changelog. Which brings me on to... +1 OK b) Please add a changelog entry for this addition. Where is the changelog? Sorry, didn't find anything so far... c) Please add this new attribute to the documentation. +1 OK 1. The new boolean option 'persistAuthentication' is implemented in ManagerBase and so can be configured for StandardManager or PersistentManager (defaults to false). Also added this to mbeans-descriptor.xml, so it's writable through JMX. +1 2. That option is set for any new StandardSession upon creation (in method ManagerBase.createSession(String)). Once a session got this option, it's not being changed during the session's lifetime. d) Why do it this way as opposed to looking at the current setting in the Manager when the session is persisted? That was my first idea, too. However, the StandardSession only knows a Manager (interface). I could have added the get/setPersistAuthentication methods to that interface (but didn't want to introduce such far-reaching changes) or need to test with instanceof for every session being written. Furthermore, some locking mechanism would be a good idea while sessions are stored, so that the option cannot be changed while a bunch of sessions is currently persisted (StandardManager synchronizes with sessions itself, however, PersistentManager does not). Actually a new lock monitor would be ideal. On the other side, one will likely not re-decide whether sessions shall be persisted or not from one day to another. I guess, that setting will remain quit constant for a certain installation, so my solution seemed not much worse. However, a simple instanceof ManageBase is also cheap and saves the setters and getters in StandardSession. Shall I refactor it acoordingly? e) I am very much against using system properties for configuration except as a last resort. I don't see why PRINCIPAL_SERIALIZABLE_CHECK can't be a property on the Manager. Although see g) below first. No longer needed, if a simple instanceof Serializable is enough. My idea was, that this option is true by default (a Principal is not a big class and the dry-run test should be fast enough). So, I did not see that this option was worth a config option (initially, that was just an internal constant). 3. In StandardSession, method doWriteObject writes authentication information (authType and principal) only, if 'persistAuthentication' is true, the session is authenticated (authType or principal != null) and principal is serializable. f) Would the code be cleaner if these were always written and null values used if authentication persistence is disabled? The idea was to be able to read the old-style object stream as well (a file created with the version without that enhancement). But, this can be achieved even without the Boolean tag. I will make that a bit more straight. 4. The "is principal serializable" test is, by default, a deep (expensive?) check, which actually serializes the whole principal object to an ObjectOutputStream backed by a new NullInputStream, which just discards all bytes written. This check can be skipped by setting system property org.apache.catalina.session.StandardSession.PRINCIPAL_SERIALIZABLE_CHECK to false (defaults to true, however). If skipped, the principal is considered serializable if (principal instanceof Serializable). That's how the session's attribute values are checked for being serializable or not. However, that is odd, if such a serialized object has a child object which is not serializable. In that case, the ObjectOutputStream is left in an inconsistent state and no so called fatal exception code is written to the stream (that is, when reading such a stream, no WriteAbortedException is not thrown for such an error). g) The instanceof Serializable check should be sufficient. It a class has that and then is not serializable that is a bug but not one I think the Manager needs to protect itself from beyond catching and logging the exception (and inserting null into the object stream). +1 OK 5. A Boolean object is used as a tag/marker that determines, whether authentication information is present id the stream or not. If none of the above conditions are met, both authType and principal are not serialized (than, only the initial Boolean false marker has been emitted to the stream). BTW, the Boolean false marker is not even required (if there is no authentication information in the stream) since the reading
Re: Enhancement: New option 'persistAuthentication' for session manager
On Tue, Feb 18, 2020 at 9:19 AM Carsten Klein wrote: Rémy, Can you describe an actual use case for this ? Without clustering, I don't understand why the auth persistence is useful at all [when using clustering, the delta manager persists that auth information]. To be honest, that's also the case for session persistence itself, which does not provide a QoS level that is still relevant today. But the feature is already there [maybe it could be considered for removal in 10 actually ...]. Why is session persistence across restarts not still relevant today? Tomcat, as well as likely any other servlet container, stops/starts a context when actually a reload (aka reconfigure) action would be required (I guess, that's specified servlet specs, right?). A context needs to be completely restarted for any configuration change (oh, we need to add a new allowed host to the CORS filter, etc.). Restarting, and so loosing all runtime data (the session), makes this simple stop/start approach (from a programmer's point of view) difficult to use in a production environment. I often find myself doing such configuration changes in the evening in order not to drop logged on users (that's when I'd better enjoy some freetime *g*). That's why I believe, that session persistence across restarts is required still today. Of course, there may be frameworks, including clustering, that can do a lot for you. However, I'm talking about Tomcat, not other frameworks. Why shouldn't Tomcat on its own provide a production-system-ready "reconfigure" action? It did in the past, why dropping it? That feature seems not too buggy... Also, for me, setting up a Tomcat cluster with at least two machines, only for getting session persistence during reconfiguring the nodes seems a bit overdrawn. In our setups we typically have one server machine, serving between 10 and 50 users at a time. I just can't tell my customer to double the hardware just in order to keep users logged on while we are working on the configuration. Maybe you think it's "overdrawn", but right now you have very poor QoS. What happens while the webapp/server is away ? Do you think the user will not see anything bad ? [aka: please don't do anything until the restart is done ...] Plus what happens if you have 1s of sessions ? It gets slow and some have complained about it in the past. Etc etc. Rémy, in our case, the app runs standalone in the browser and is fully AJAX driven, so, if the server restarts in some 10 seconds, the user does not even recognize. Maybe one or other request to the server (loading/saving some data) is slow, since it waits for the server to be up again. In very rare cases, it may run into a timeout, in which case the app comes up with a message box "Server is currently not available, please try again in some minutes." BTW, session persistence can simply be disabled even today, by setting StandardManager's pathname to en empty string. Yes, with 1s of sessions persistence may be a bottleneck (even with PersistentManager). However, actually, we do not have so many sessions. Maybe that is the general point. There are so many scenarios, Tomcat is used in. So, who will actually really know, which features are good and which are bad? It always depends on the actual scenario. Of course, security is important and software vendors must ensure to publish secure (-by-default) software. I agree, that saving the principal to disk for some seconds may be a security risk for some apps and some people. That's why the new feature is configurable and defaults to off. However, as a server admin, I'm able to secure the installation that way, so that nobody external is able to read such session files. Of course, with the root password, you are always able to do lot of bad things (includes attaching to the JVM and reading passwords directly from memory). So, when security is an issue, NOT saving passwords at all could be a first step :-p That's not how things should happen today. Whats your proposal? How should things happen today? It may be the case, that today, many setups use WAR files, including code and configuration, which are deployed at regular intervals or at scheduled deploy times. However, that's not how we work. We like to be able to react contemporary when a problem in a production system comes up (our customers like that, too). So, that's my plea for session persistence across restarts, as implemented in recent versions. Nearly the same use case applies to persisting authentication information as well. The feature is there right now, so you're good ... In our setups, we use the session mainly for authentication. The problem that comes with not persisting authentication may not be obvious with simple (browser driven) BASIC authentication, since the Browser sends those Base64 encoded credentials with each request. Using session based authentication (send credentials o
Re: Enhancement: New option 'persistAuthentication' for session manager
Rémy, Can you describe an actual use case for this ? Without clustering, I don't understand why the auth persistence is useful at all [when using clustering, the delta manager persists that auth information]. To be honest, that's also the case for session persistence itself, which does not provide a QoS level that is still relevant today. But the feature is already there [maybe it could be considered for removal in 10 actually ...]. Why is session persistence across restarts not still relevant today? Tomcat, as well as likely any other servlet container, stops/starts a context when actually a reload (aka reconfigure) action would be required (I guess, that's specified servlet specs, right?). A context needs to be completely restarted for any configuration change (oh, we need to add a new allowed host to the CORS filter, etc.). Restarting, and so loosing all runtime data (the session), makes this simple stop/start approach (from a programmer's point of view) difficult to use in a production environment. I often find myself doing such configuration changes in the evening in order not to drop logged on users (that's when I'd better enjoy some freetime *g*). That's why I believe, that session persistence across restarts is required still today. Of course, there may be frameworks, including clustering, that can do a lot for you. However, I'm talking about Tomcat, not other frameworks. Why shouldn't Tomcat on its own provide a production-system-ready "reconfigure" action? It did in the past, why dropping it? That feature seems not too buggy... Also, for me, setting up a Tomcat cluster with at least two machines, only for getting session persistence during reconfiguring the nodes seems a bit overdrawn. In our setups we typically have one server machine, serving between 10 and 50 users at a time. I just can't tell my customer to double the hardware just in order to keep users logged on while we are working on the configuration. It may be the case, that today, many setups use WAR files, including code and configuration, which are deployed at regular intervals or at scheduled deploy times. However, that's not how we work. We like to be able to react contemporary when a problem in a production system comes up (our customers like that, too). So, that's my plea for session persistence across restarts, as implemented in recent versions. Nearly the same use case applies to persisting authentication information as well. In our setups, we use the session mainly for authentication. The problem that comes with not persisting authentication may not be obvious with simple (browser driven) BASIC authentication, since the Browser sends those Base64 encoded credentials with each request. Using session based authentication (send credentials once and rely that your session remains authenticated) is more modern and likely a bit more secure, right? However, in that scenario, it's really bad, if your authentication has gone after the context has been restarted. In that case, you may be forwarded to the login page (when using FORM authentication). In our case, our AJAX-style JavaScript client just logs you off and all your work is gone (that's like if your browser crashes occasionally and has no "restore previously opened tabs" feature...). In our case, authentication information is the only valuable data in the session (we don't save any attributes). So, isn't persisting authentication information (for the sake of completeness) a feature of its own? In other words, why not persisting authentication information? That data is the only data, that currently is NOT persisted. Why not? Comments in the code only mention, that authType and principal are transient. But why? If there is a session persistence feature in Tomcat, shouldn't it persist the session as a whole (at least, as much as possible)? Since recent versions of Tomcat's default GenericPrincipal do no longer save passwords, persisting authType and principal seems not being harmful to me. Again, session persistence across restarts is just a "workaround" for the missing "reconfigure" action. For me, it's the tribute for using the much simpler stop/start schema in servlet containers (in contrast to a real reconfigure action, which often tends to get quite complex). Typically, persisted sessions do not remain on disk for a long time - only while restarting. Mark Thomas suggested to make that an optional feature, which is off by default. That's what I've done. However, at that time it was not obvious (at least to us) that passwords are no longer saved in GenericPrincipal. With that in mind, I would even vote for making that new option defaulting to true. Carsten - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Enhancement: New option 'persistAuthentication' for session manager
Jonathan, I'm not quite sure whether it's not too early for a PR... :-) I was waiting for some remarks on my code, prior to officially release a PR. Can't you just do a Compare on GitHub? Carsten Can you open a PR so we can diff your changes? Very excited to see this! We used a workaround in a Valve to add the authentication information into the session. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Enhancement: New option 'persistAuthentication' for session manager
Hi there, finally, I got my first Tomcat enhancement ready. You can view its code at my Tomcat fork on GitHub: https://github.com/cklein05/tomcat/tree/session-manager-persist-authentication Before I'm opening an enhancement in Tomcat's Bugzilla, maybe, Mark and Christopher (or whoever else is interested), could you please have a quick look at the code? BTW, recent versions of Tomcat do no longer save and store passwords in GenericPrincipal. That makes using this enhancement even less critical. Some brief description of what I've done: 1. The new boolean option 'persistAuthentication' is implemented in ManagerBase and so can be configured for StandardManager or PersistentManager (defaults to false). Also added this to mbeans-descriptor.xml, so it's writable through JMX. 2. That option is set for any new StandardSession upon creation (in method ManagerBase.createSession(String)). Once a session got this option, it's not being changed during the session's lifetime. 3. In StandardSession, method doWriteObject writes authentication information (authType and principal) only, if 'persistAuthentication' is true, the session is authenticated (authType or principal != null) and principal is serializable. 4. The "is principal serializable" test is, by default, a deep (expensive?) check, which actually serializes the whole principal object to an ObjectOutputStream backed by a new NullInputStream, which just discards all bytes written. This check can be skipped by setting system property org.apache.catalina.session.StandardSession.PRINCIPAL_SERIALIZABLE_CHECK to false (defaults to true, however). If skipped, the principal is considered serializable if (principal instanceof Serializable). That's how the session's attribute values are checked for being serializable or not. However, that is odd, if such a serialized object has a child object which is not serializable. In that case, the ObjectOutputStream is left in an inconsistent state and no so called fatal exception code is written to the stream (that is, when reading such a stream, no WriteAbortedException is not thrown for such an error). 5. A Boolean object is used as a tag/marker that determines, whether authentication information is present id the stream or not. If none of the above conditions are met, both authType and principal are not serialized (than, only the initial Boolean false marker has been emitted to the stream). BTW, the Boolean false marker is not even required (if there is no authentication information in the stream) since the reading code works fine without any Boolean in the stream. So emitting Boolean false for signalling "no auth info" is actually optional (we could consider omitting it). 6. When sessions are loaded, ManagerBase provides a org.apache.catalina.util.CustomObjectInputStream instance to read sessions from. That instance is configured with the session's sessionAttributeValueClassNamePattern property. This essentially defines the classes, session attribute values may consist of. This pattern defaults to "java\\.lang\\.(?:Boolean|Integer|Long|Number|String)", so only attributes with these simple types can be loaded from a session. That filter pattern is only in effect, if a security manager is active, or if a pattern has been configured for the manager (e.g. in context.xml). Currently, however, all session data (including its base properties like creationTime, isNew, isValid etc) is loaded with that filter mechanism in place. Since those so called 'scalar instance properties' actually only consist of those simple types, that was not a problem. However, loading the serialized principal from the object stream is now subject to that filter mechanism (BTW, HA's DeltaManager and DeltaSession just do not utilize the CustomObjectInputStream). Since, as the name implies, the sessionAttributeValueClassNamePattern applies to attribute values only, I decided to give the CustomObjectInputStream a boolean 'filterActive' property, which is set to true, just before StandardSession starts deserializing attributes. The initial value of 'filterActive' can be specified though the constructor, to which both StandardManager and StoreBase pass false (actually, only StandardManager and PersistenManager (through StoreBase) do use the CustomObjectInputStream class). 7. Conclusion - Tomcat still builds with that enhancement in place - The enhancement obviously works as expected - Minor decisions can still be made (e.g. pull NullInputStream into a new file, maybe in ...catalina.util) - What about the Tomcat documentation? Where to document all this? Carsten - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 7.x.x, 8.x.x, 8.5.x and 9.x.x: Session serialization w/o authentication related information
On 01/12/2019 23:04, Mark Thomas wrote: I'm with you. And likely our setup is special in a way. However, I've rarely seen that you have to re-enter credentials in a professional web application like Google or Facebook, for example. Yes. But if those apps were running on Tomcat I doubt that a) they'd be running on a single instance and b) they'd be using Tomcat's in memory session management. Yes, my comparison was not really fair. Just wanted to point out, that re-entering credentials is no longer common in today's applications. As an option, disabled by default I don't see why not. Agreed. A good place to start would be to open a Bugzilla enhancement request against 9.0.x (we always develop for the latest version first and then back-port). Makes sense. After that, we accept patches in Bugzilla or Pull Requests at GitHub. Expect a fair amount a feedback on your first version (my first patch needed to be completely re-written) but a couple of tips to reduce the rework: - enable checkstyle and make sure there are no errors when building - generally, follow the style of the code you are editing (the Tomcat code style is not always consistent) So, I will give that a try. Hope to come up with a patch within the next couple of weeks. The patch itself seems not to be too challenging. Thanks for your advice and tips. Carsten - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
