Re: Fwd: Tomcat 8 and Java 8
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Greg, On 4/16/15 12:06 PM, Greg Huber wrote: ..To follow on, for others going grey also. The file:/ seems to be from a :/ (colon) on the class path, and the combination of running it from /etc/rc.d/init.d makes it not work. Not working -classpath :/opt/apache-tomcat/apache-tomcat-8.0.21/bin/bootstrap.jar:/opt/apache - -tomcat/apache-tomcat-8.0.21/bin/tomcat-juli.jar Working -classpath /opt/apache-tomcat/apache-tomcat-8.0.21/bin/bootstrap.jar:/opt/apache- tomcat/apache-tomcat-8.0.21/bin/tomcat-juli.jar I guess we need to check for file:/ so we do not scan the whole server. Whoops. My first reaction is don't do that, but it seems like a stray colon shouldn't really be valid in the classpath, so Tomcat probably ought to ignore (empty string) classpath entries. Could you file a Bugzilla issue? https://bz.apache.org/bugzilla/enter_bug.cgi?product=Tomcat%209 - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v2 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJVMJ68AAoJEBzwKT+lPKRYYqEP/jihtZvF2zQPA7dF+l4HNZ6r ZndnLNIh1FFC6HCeo/LKA2IWr9ISRw+2g5Iuftaz9RJRmdmCP2fYUrODwlZOLbU4 Z9vj1Pc4pCXDKJ2mxTAK1hWMWTP3FwqIM02iw7dli8hiCBs7TM97gtU6u/AhpKYS l9Fd+ynZM377zCnvoBSQ+3C744Roat9a6XZQvvAcIdWkC7ELVCXB7KNF4xtUHO4r ezPUWzbPV6GaE5Y81kxoRIKQOGUq+/JX3q2jaH3b5zVXMWmsVfQY68Mo3UyjpMJ1 ymiPZIYXAKJrXrY5SWhGSB6wzzx6nJVesINQB0O2ykZW/SjiXwyZpVqiMPf0tOmP +xIAaNRFXgT6eH7Rs8+U0n8b+fSZ3uJuqB7zZDWvK0xSwjB1bYt2vRBRH28L+o1x B2tDaDlJQfhsNji78/AiqiYiJcLrXzsJiB/niuVQSggTuGiOUtiHfY2BpbXCRnoU 4oIU5Rety5NKO6oXeB++ZUYF97QY5OfhTjLLrXxv8ehsUG74jZlrO7xDUIgmY86+ /Mw9EYmBWXk9l0qNBk6TqK8N8244rwvLZHmtGiKzhih+Iw+uIP+mjICjW+++Aw3j BRmTf/MyrdD2gIO7h/m0GGOpJsAugQ9IJu7UPDC4g4vZBtWUtUEY9lGzzxcOdAoS utxJvmUG9Ha9WfBsRyBV =OIAY -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: RFE: read keystorePass from file
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Luca, On 4/16/15 4:16 PM, Luca Menegus wrote: Hi Chris, let me describe what I'm trying to do in more details. Goals: * Completely automate tomcat deployment, and make sure deployment is reproducible and testable in different environment (hence we use ansible+git to manage installation and _configuration for tomcat) * Do not make stupid thing (like storing passwords or keys in SCM) just because we want to automate * Be as secure as possible I completely understand. (Working) implementation: I install and configure tomcat with ansible (server.xml is an ansible template, on every run ansible re-generates the template and compare it to the deployed version, so it needs to be able to read the deployed server.xml) Then I generate a random pin file, the keystore and the private keys (and relates cert requests) with ansible on the target host. (Thanks to Jan suggestion) I can now instruct tomcat to read the pin from the pin file. I do not need to store pin/keys in SCM So what have you bought yourself? If Ansible can templatize server.xml, why not simply have it enter the auto-generated pin into the server.xml as it's deployed? I really don't understand why it must be a separate file. Does anyone have access to the Tomcat server who should never see the pin for the keystore? If so, don't give them file-permission-access to the server.xml file. Are you worried about Tomcat being compromised and revealing the contents of server.xml? The Tomcat process needs access to the external pin file, so you haven't protected against this attack, either. I'm running on linux so I can install the following audit rule: auditctl -w [pin file] -p warx -F auid!=tomcat -F auid!=4294967295 so *at least* I'm sure that access to the pin file made by anybody but the tomcat user would be logged (so I know when I was compromised and need to rekey the cert) So why not simply use that rule with server.xml? This rule doesn't trigger when I am comparing the deployed server.xml with the expected version or the deployed keystore (as to check the presence of the key I do not need the key pin). So you trust Ansible to build-out and deploy the server, but not to read the pin file? Ansible is the *source* of the pin file, so you'd better trust it. It only trigger when I generate the initial pin, somebody compromise my box, I need to renew certs (which are all events I like to be informed about) How do you detect compromise? Now back to the tomcat relevant suff: (at least) in my setup I find it valuable to be able to read secrets from files I can protect with specific audit rules. Again, why not simply protect server.xml with such audit rules? Jan suggested me a viable (and easily implemented) way to do it [1], so I'm ok. I'm glad he was able to help. Konstantin's suggestion 2 days prior had the same solution (you just had to follow the link he posted). Now the question is, given how easy is to implement Jan solution do you think my use case deserves an RFE (something like adding keyPassFile attribute to the Connector )? I don't yet see a justifiable position of how it improves security in any way. I'm happy to implement the feature myself if you can explain how it improves security in any meaningful way. I do not know tomcat enough to answer, but if you say so I'll post a BUG. It's definitely an enhancement if anything. Regards and thanks for your help, Luca [1] - Add -Dorg.apache.tomcat.util.digester.PROPERTY_SOURCE=FilePropertySource to CATALINA_OPTS - deploy a jar in tomcat lib with the following class: public class FilePropertySource implements org.apache.tomcat.util.IntrospectionUtils.PropertySource { @Override public String getProperty(String name) { if (name.startsWith(file://)) { try { return readFile(name.substring(7)); } catch (IOException e) { throw new RuntimeException(Unable to derefercence property + name, e); } } return null; } String readFile(String fileName) throws IOException { FileReader br = new FileReader(fileName); try { StringBuilder sb = new StringBuilder(); char[] buff = new char[4096]; int read = -1; while ((read = br.read(buff)) != -1) { sb.append(buff, 0, read); } return sb.toString(); } finally { br.close(); This finally block can throw an NPE. Also note that an IOException thrown from br.close() can cause any in-flight exception to be lost. } } } - in server.xml use it like ${file:///bla/bla/bla} - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v2 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJVMKDMAAoJEBzwKT+lPKRYkEAP+gNOF42ZJdVZh5Ut++fS7kJt 8/yaZxO9D1xgyRbjolUDsTH/ERVL8NGd6Y4DnWOJ5N0cbuC1WtSVQTffGJx4CQu3 F6ljizu1yyyzWUFedFunqWB4zBZB4UwOifwOQZXS/sEtLk23eeazuKf/xpBLM3yK Xq6GjwJDr+y/zmDMMqu2eXV74WnHU1OEkzhKlr14419lCOr/ij1TEK0o6UabfISx J4KR414pT7ry9yYQVtLR/FLh6MDeExuCnO2XHN+U7jQs8K1lRlqCi72cnHU3a3xs
Re: Add member to static memebership cluster during runtime
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chris, On 4/16/15 9:59 AM, Christoph Prybila wrote: Tomcat: 7.0.59 OS: Win7 Professional SP1 Java: 1.7.0_67 I want to create a cluster and I need to use static membership. But the number of nodes in my cluster is not static. Is it possible to add members to such a static membership cluster without restarting the other already running tomcat instances? I figured that in order to add a new member I would need to specify it in the StaticMembershipInterceptor of all the other nodes. But editing the server.xml of a running tomcat instance had no effect. Tomcat will not auto-reload server.xml; there are just too many moving parts to reload it quickly, so you have to bounce the whole server. Oddly enough, I was just talking to someone yesterday about how static membership ought to be updatable via at least JMX. I'm not sure, but it's possible that static membership is /already updatable via JMX. Are you comfortable connecting to a local Tomcat using jconsole or VisualVM and browsing around the Catalina tree? I don't have a clustered setup handy to inspect myself to see if it's already possible. If it's /not/ currently possible, it really should be. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v2 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJVMJ3yAAoJEBzwKT+lPKRYroIQAKh86Eh4GeE3tR/sG8kKeGLg +5nmMqBD5I/uRacUN8llzOjxN2kZA2f5uwxYgiSTmZE8oQVP/a+M6PT0F4pCZMkZ 8/kQDzc/QgcMLWdRC3ApO8GJRX09nes3hbgyTcKUdzqXHOcc5SfZSaESC/euCyZz xSj/G3052FadK0zSKDDnD6JXJ5U55fGyPUew/7x+ERR0ZY/CG7SH0d2PVhm+iDKO vaeCpSFKCakZ87XGp5PchvjXQeFnfqV0K60RCJ8N8EmbINdS9xxn2OcBX9jRaetb Ptp+fUIKXrB5Jdz8riA+jU5vlVB6WtKC333GyNzGHD6/aTSxAOhpV3uuA/oSq7QI B+7HzHWa51AI7GIkKIU5E2zSI0+Y75rIMW6KEyhqiUAatxcAHiC3jAGVG+py6pJ9 uHjLVCV+Jk/DmymL0N0jkA8SwqD2byDbhx92InFzp/eawfEe3NMTGnYrHfGTwJnT wc/9YL1eDHkweM9CDq0z3yvKUiyJz47IppuhsWVD6peE44ibQD/8tMVU7BI2Pleo FGuJI3KX7dInMO66bRCUkGvf1FKlnDrhTsA+O7XXVlQJbFwxFRkbm7sTGvUu0QLG 0EdJmC/fuCnmgKXmmff0rZlmZ7X6XeMan0IlSUR3gf9ry8NBaTAQun7C5M77dtOl VQMrzMZ/xZ9EmHK6nuBq =OpTW -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Fwd: Tomcat 8 and Java 8
Chris, My startup script (many years old) uses: CLASSPATH=$CLASSPATH:$CATALINA_HOME/bin/bootstrap.jar So its quite possible if the $CLASSPATH is empty we get the :/ The odd thing is why does it work when run from the command line but not from /etc/rc.d/init.d as both use case have the colon. https://bz.apache.org/bugzilla/show_bug.cgi?id=57823 Cheers Greg On 17 April 2015 at 06:48, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Greg, On 4/16/15 12:06 PM, Greg Huber wrote: ..To follow on, for others going grey also. The file:/ seems to be from a :/ (colon) on the class path, and the combination of running it from /etc/rc.d/init.d makes it not work. Not working -classpath :/opt/apache-tomcat/apache-tomcat-8.0.21/bin/bootstrap.jar:/opt/apache - -tomcat/apache-tomcat-8.0.21/bin/tomcat-juli.jar Working -classpath /opt/apache-tomcat/apache-tomcat-8.0.21/bin/bootstrap.jar:/opt/apache- tomcat/apache-tomcat-8.0.21/bin/tomcat-juli.jar I guess we need to check for file:/ so we do not scan the whole server. Whoops. My first reaction is don't do that, but it seems like a stray colon shouldn't really be valid in the classpath, so Tomcat probably ought to ignore (empty string) classpath entries. Could you file a Bugzilla issue? https://bz.apache.org/bugzilla/enter_bug.cgi?product=Tomcat%209 - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v2 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJVMJ68AAoJEBzwKT+lPKRYYqEP/jihtZvF2zQPA7dF+l4HNZ6r ZndnLNIh1FFC6HCeo/LKA2IWr9ISRw+2g5Iuftaz9RJRmdmCP2fYUrODwlZOLbU4 Z9vj1Pc4pCXDKJ2mxTAK1hWMWTP3FwqIM02iw7dli8hiCBs7TM97gtU6u/AhpKYS l9Fd+ynZM377zCnvoBSQ+3C744Roat9a6XZQvvAcIdWkC7ELVCXB7KNF4xtUHO4r ezPUWzbPV6GaE5Y81kxoRIKQOGUq+/JX3q2jaH3b5zVXMWmsVfQY68Mo3UyjpMJ1 ymiPZIYXAKJrXrY5SWhGSB6wzzx6nJVesINQB0O2ykZW/SjiXwyZpVqiMPf0tOmP +xIAaNRFXgT6eH7Rs8+U0n8b+fSZ3uJuqB7zZDWvK0xSwjB1bYt2vRBRH28L+o1x B2tDaDlJQfhsNji78/AiqiYiJcLrXzsJiB/niuVQSggTuGiOUtiHfY2BpbXCRnoU 4oIU5Rety5NKO6oXeB++ZUYF97QY5OfhTjLLrXxv8ehsUG74jZlrO7xDUIgmY86+ /Mw9EYmBWXk9l0qNBk6TqK8N8244rwvLZHmtGiKzhih+Iw+uIP+mjICjW+++Aw3j BRmTf/MyrdD2gIO7h/m0GGOpJsAugQ9IJu7UPDC4g4vZBtWUtUEY9lGzzxcOdAoS utxJvmUG9Ha9WfBsRyBV =OIAY -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Configuring Tomcat manager application upload limit
On Wed, Apr 15, 2015 at 08:33:00AM -0500, Mark Thomas wrote: On 15/04/2015 05:28, Dominic Hargreaves wrote: I am running the Tomcat manager application via a Debian package (tomcat8-admin), which deploys the webapp from /usr/share/tomcat8-admin/manager. We ran into a problem hitting th maximum upload size (configured as the multipart-config element in the HTMLManager servlet block). This is easy to fix if you're willing to edit Debian-packaged files, but this is incorrect: files in /usr/share are owned exclusively by dpkg. Is there a way to override this element from somewhere in tomcat's conf directory (which is symlinked to /etc/tomcat8 in Debian)? No. The web application setting would take priority. Okay, thanks for confirming. If it's not possible to override this at the moment, then is this a bug in Tomcat, or the Debian packaging? I'd say the bug is in the Debian packaging (but I would say that wouldn't I). Without know how Tomcat is packaged in Debian, I would expect the following: - the Manager web application to be a separate, optional package Yep, it is. - the web.xml to be placed in an appropriate location for users to be able to edit it. I took a quick look through the web.xml for the Manager app. There are plenty of settings I wouldn't expect to change but there are a handful of things that users might want to tweak. In theory, it should be possible to move those settings to context.xml but that would require code changes and would mean do things that were contrary to the spirit of the servlet spec so there would likely be resistance to such changes from the Tomcat community. One option would be for Debian to use external entities for the configurable elements and place the file that defines those entities somewhere where users can edit it. Okay, that's an interesting idea if it's possible to agree on a sane set :) Thanks for your feedback! Cheers, Dominic. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Add member to static memebership cluster during runtime
Am 17.04.2015 um 07:45 schrieb Christopher Schultz: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chris, On 4/16/15 9:59 AM, Christoph Prybila wrote: Tomcat: 7.0.59 OS: Win7 Professional SP1 Java: 1.7.0_67 I want to create a cluster and I need to use static membership. But the number of nodes in my cluster is not static. Is it possible to add members to such a static membership cluster without restarting the other already running tomcat instances? I figured that in order to add a new member I would need to specify it in the StaticMembershipInterceptor of all the other nodes. But editing the server.xml of a running tomcat instance had no effect. Tomcat will not auto-reload server.xml; there are just too many moving parts to reload it quickly, so you have to bounce the whole server. Oddly enough, I was just talking to someone yesterday about how static membership ought to be updatable via at least JMX. I'm not sure, but it's possible that static membership is /already updatable via JMX. Are you comfortable connecting to a local Tomcat using jconsole or VisualVM and browsing around the Catalina tree? I don't have a clustered setup handy to inspect myself to see if it's already possible. If it's /not/ currently possible, it really should be. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v2 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJVMJ3yAAoJEBzwKT+lPKRYroIQAKh86Eh4GeE3tR/sG8kKeGLg +5nmMqBD5I/uRacUN8llzOjxN2kZA2f5uwxYgiSTmZE8oQVP/a+M6PT0F4pCZMkZ 8/kQDzc/QgcMLWdRC3ApO8GJRX09nes3hbgyTcKUdzqXHOcc5SfZSaESC/euCyZz xSj/G3052FadK0zSKDDnD6JXJ5U55fGyPUew/7x+ERR0ZY/CG7SH0d2PVhm+iDKO vaeCpSFKCakZ87XGp5PchvjXQeFnfqV0K60RCJ8N8EmbINdS9xxn2OcBX9jRaetb Ptp+fUIKXrB5Jdz8riA+jU5vlVB6WtKC333GyNzGHD6/aTSxAOhpV3uuA/oSq7QI B+7HzHWa51AI7GIkKIU5E2zSI0+Y75rIMW6KEyhqiUAatxcAHiC3jAGVG+py6pJ9 uHjLVCV+Jk/DmymL0N0jkA8SwqD2byDbhx92InFzp/eawfEe3NMTGnYrHfGTwJnT wc/9YL1eDHkweM9CDq0z3yvKUiyJz47IppuhsWVD6peE44ibQD/8tMVU7BI2Pleo FGuJI3KX7dInMO66bRCUkGvf1FKlnDrhTsA+O7XXVlQJbFwxFRkbm7sTGvUu0QLG 0EdJmC/fuCnmgKXmmff0rZlmZ7X6XeMan0IlSUR3gf9ry8NBaTAQun7C5M77dtOl VQMrzMZ/xZ9EmHK6nuBq =OpTW -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org I already tried to configure the cluster via jmx. After connecting to a tomcat instance, there was indeed a Cluster object in the tree but I was not able to add/remove existing members there. This cluster object contains a group called Member where all members of the cluster are listed. Therefore I was able to query information about the members. Unfortunately the member objects do not offer any operations at all. I do not have much experience with jmx but I think that means that the member-entries are read-only and it is currently not possible to configure the membership via jmx. The cluster object itself offers the operations start, stop, setProperty and send which as far as I understood also do not give any possibility to add new members. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Add member to static memebership cluster during runtime
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chris, On 4/17/15 5:33 AM, Christoph Prybila wrote: Am 17.04.2015 um 07:45 schrieb Christopher Schultz: Chris, On 4/16/15 9:59 AM, Christoph Prybila wrote: Tomcat: 7.0.59 OS: Win7 Professional SP1 Java: 1.7.0_67 I want to create a cluster and I need to use static membership. But the number of nodes in my cluster is not static. Is it possible to add members to such a static membership cluster without restarting the other already running tomcat instances? I figured that in order to add a new member I would need to specify it in the StaticMembershipInterceptor of all the other nodes. But editing the server.xml of a running tomcat instance had no effect. Tomcat will not auto-reload server.xml; there are just too many moving parts to reload it quickly, so you have to bounce the whole server. Oddly enough, I was just talking to someone yesterday about how static membership ought to be updatable via at least JMX. I'm not sure, but it's possible that static membership is /already updatable via JMX. Are you comfortable connecting to a local Tomcat using jconsole or VisualVM and browsing around the Catalina tree? I don't have a clustered setup handy to inspect myself to see if it's already possible. If it's /not/ currently possible, it really should be. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org I already tried to configure the cluster via jmx. After connecting to a tomcat instance, there was indeed a Cluster object in the tree but I was not able to add/remove existing members there. This cluster object contains a group called Member where all members of the cluster are listed. Therefore I was able to query information about the members. Unfortunately the member objects do not offer any operations at all. I do not have much experience with jmx but I think that means that the member-entries are read-only and it is currently not possible to configure the membership via jmx. The cluster object itself offers the operations start, stop, setProperty and send which as far as I understood also do not give any possibility to add new members. Agreed. Could you file an enhancement request in Bugzilla? Thanks! - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v2 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJVMQBsAAoJEBzwKT+lPKRYPYgP/A0wrm3zsAt3F9ZAb9+49L0d OELtNDCNfn0iy1uXSEdmke0amOSUKF8FrO7QFdf2BhFtU9IQDxQBX/a/+1oeMMCA 7SMtsNP99yGcl0IropbPpHIrDTP+VJ8dhzva+MEBerPO2glcyns6FvrqW595w/7u 67TdJ+NDrfKj6J9yg1A2xCX96T7ugMgQM1sLxbevXL2CcXQENaLzaZ3WXbKX2cJ1 R03iHq2+zaBjp9fX6sSLlF7khAZ+XMWpgkTRDQiGXb150XKgwmjFkdMsgFz97RPX BQi2tGt6kpagBMsT7J7cpIk8wgJsgnQlZQhIkdwXis4r2QDm1O0UEcjDNzoUFL73 GIDK1K9tcxhMm23bo793KXk5u+qWMKlvmrnJ5K1vL/tMskQ+kAqaD/v6n2vVCAd9 AJCMEY0CoADcVx3v1nSVqH4w+8e0b3wtDehx9wTwgSICWfTkAmu1s41eVB8NELDZ kuokk8Y0t4H3vowCHpnEO7GpXhssBNWNBG+AkM1/i0caOE6ckFH0L8A0nXdxIErM IZ4iyrJVQooHaNmaCoU6ZurOnVb66uEYmPJnCBZcWRtaqfMdvEc4VJAK6sTcnVIA 8t7U9q54cITEXu8MuRCtkrubLjDbueBzMmocsWbE2wsg/ocEbqkaOqgV8Ozk4qZx Npb+77NpR7HtUujPorfC =kD2C -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: GC Issue on Tomcat6
On 4/17/2015 8:32 AM, Subhro Paul wrote: Hi Team, For our client website we have two Tomcat servers. Both servers are having same configuration which is 16GB RAM, 8CPUS and Linux 6.5 OS. Tomcat 6 and JAVA 6 are installed in both systems. Yesterday, using Jconsole we observed that frequency of GC on one server was very high which was 5 GC hit per Minute and on other server it was 1 GC hit per 3 Minutes. But today we observed that on both of the servers GC frequency is same. Is this a know secnario or the server is facing any issue? It was probably just receiving more traffic and/or doing more work of some other kind. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: GC Issue on Tomcat6
http://ca.linkedin.com/in/ameermawia Toronto, ON Phone:647-262-4380 On Fri, Apr 17, 2015 at 8:47 AM, David kerber dcker...@verizon.net wrote: On 4/17/2015 8:32 AM, Subhro Paul wrote: Hi Team, For our client website we have two Tomcat servers. Both servers are having same configuration which is 16GB RAM, 8CPUS and Linux 6.5 OS. Tomcat 6 and JAVA 6 are installed in both systems. Yesterday, using Jconsole we observed that frequency of GC on one server was very high which was 5 GC hit per Minute and on other server it was 1 GC hit per 3 Minutes. But today we observed that on both of the servers GC frequency is same. Garbage Collection in Oracle Hot spot JVM takes place in generational manner. GC on young generation is invoked much more frequently, while Full GC is invoked once in a while. In any case, frequency is much higher than what you have mentioned. I'm not sure which GC(Young/Full) you are talking about? What pattern of GC you have earlier observed? Is this a know secnario or the server is facing any issue? GC issue has nothing to do with Tomcat per say. It is the memory footprint of your application which will dictate GC behavior. In any case, if your application is facing some memory bottle-neck, it is better to pay attention on the percentage your JVM is spending in GCing your application. If that percentage is very high, say ~90%, that means your application is facing sever memory constrain, your application is almost stalled as JVM has been trying very high to recover some space. Most probably in that case, your JVM will die throwing OOM error in sometime. It was probably just receiving more traffic and/or doing more work of some other kind. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
GC Issue on Tomcat6
Hi Team, For our client website we have two Tomcat servers. Both servers are having same configuration which is 16GB RAM, 8CPUS and Linux 6.5 OS. Tomcat 6 and JAVA 6 are installed in both systems. Yesterday, using Jconsole we observed that frequency of GC on one server was very high which was 5 GC hit per Minute and on other server it was 1 GC hit per 3 Minutes. But today we observed that on both of the servers GC frequency is same. Is this a know secnario or the server is facing any issue? Thanks Regards Subhro Paul =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you
Re: Tomcat threads dependency on net.core.somaxconn value on Linux
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Satish, On 4/17/15 7:20 AM, satish jupalli wrote: I would like to support a bust in traffic we are planning to increase the umber of threads by adding executor pool with misSpareThread to 100 and accept count to 5. However, I'm trying to understand the correlation between the tomcat connector acceptCount and ThreadCount to the Linux config net.core.somaxconn(number of incoming connections) which defaults to 128. The thread count isn't really a part of this discussion: only the accept count and /proc/sys/net/core/somaxx Does it makes sense to increase the thread count alone with out increasing net.core.somaxconn value? Probably not; Linux limits the accept queue using that value and will not allow client code to exceed that limit. https://computing.llnl.gov/linux/slurm/high_throughput.html Below is the config that we are planning to use on Tomcat 7.0.42 on Linux 5.x. (You should upgrade to 7.0.62 as soon as it feasible for you.) Executor name=tomcatThreadPool namePrefix=tomcat-http- minSpareThreads=100 maxThreads=300/ Connector port=8080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 enableLookups=false keepAliveTimeout=30 maxKeepAliveRequests=1 acceptCount=5 socketBuffer=10240 executor=tomcatThreadPool compression=on compressableMimeType=text/html, text/xml xpoweredBy=false server=false / Any inputs regarding the Tomcat threads and Linux net.core.somaxconn will be greatly appreciated to support many concurrent connections. If you use acceptCount=5, Tomcat will ask for that socket queue size and simply won't get it. You should definitely raise the value of somaxx on Linux. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v2 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJVMQG4AAoJEBzwKT+lPKRYFuMQAMMbBv7v5jOTHz302TMYNbMB sc1qH9IuV7Z1DrqAOE+yN19xdzt+7cakNl5YWomHr/U3ePNianLeqY+27WYYUm9V gTw/kHqBOUYfWttnlFzDCBidUDPw6m3TnhFq8Oia5UBdfPh4IQDR+zxF9FCkncUi 51evAzsp4gzAOMjDUmxMQMCfYQML1l+VX56Za4RT4S6z2L6HCxyjeVYQSc2nsLnU Y6G3X0ccomxszHhb5GuU82lldgKjw5BDubkp57/fxBajZ6QcWSU7sr+HlaoAUfLz +p/PSJhyXDMxlKqGmGsBK8BMMsK5H26yup5LtpKkLDEJuUYSv5N1rsklmQCbUuXk 0pBzFmVKQG2CRh2miutBC7Vr5l3AOP5ghV2uOOvuCBg+Sg8pzImase2m9Nw9Z1sh XnLrrJq7a/nvd4C2jRT/y01AJNfv0qJG95RjBxoVdoQ629fjK05Z2MB7avL2Qzwq 8JpcwhkfhVAWW87JVAIMPluILNuhCCg/LDQSVj4twdtGBy181E29yNSfmIRkq33j 2Jc83/gLgNU+maM4RzXKWzFYpk8ccLMhHsbh8txqswK+rV8XDfQTLZzxy43IawCK Nc1rBnWCVQOPMxjumfZWuZtdAy/HlSpw1otRVkXNVONOCLwMcAggLbjeO1TFJiJj MArVB8ONarkbkDts2quG =vSf+ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Configuring Tomcat manager application upload limit
2015-04-17 13:52 GMT+03:00 Mark Thomas ma...@apache.org: On 17 April 2015 09:37:40 BST, Dominic Hargreaves d...@earth.li wrote: On Wed, Apr 15, 2015 at 08:33:00AM -0500, Mark Thomas wrote: On 15/04/2015 05:28, Dominic Hargreaves wrote: I am running the Tomcat manager application via a Debian package (tomcat8-admin), which deploys the webapp from /usr/share/tomcat8-admin/manager. We ran into a problem hitting th maximum upload size (configured as the multipart-config element in the HTMLManager servlet block). This is easy to fix if you're willing to edit Debian-packaged files, but this is incorrect: files in /usr/share are owned exclusively by dpkg. Is there a way to override this element from somewhere in tomcat's conf directory (which is symlinked to /etc/tomcat8 in Debian)? No. The web application setting would take priority. Okay, thanks for confirming. If it's not possible to override this at the moment, then is this a bug in Tomcat, or the Debian packaging? I'd say the bug is in the Debian packaging (but I would say that wouldn't I). Without know how Tomcat is packaged in Debian, I would expect the following: - the Manager web application to be a separate, optional package Yep, it is. - the web.xml to be placed in an appropriate location for users to be able to edit it. I took a quick look through the web.xml for the Manager app. There are plenty of settings I wouldn't expect to change but there are a handful of things that users might want to tweak. In theory, it should be possible to move those settings to context.xml but that would require code changes and would mean do things that were contrary to the spirit of the servlet spec so there would likely be resistance to such changes from the Tomcat community. One option would be for Debian to use external entities for the configurable elements and place the file that defines those entities somewhere where users can edit it. Okay, that's an interesting idea if it's possible to agree on a sane set :) Thanks for your feedback! Another option (I think) is to use the property replacement feature. I.e. ${property.name} in web.xml and property.name=foo in catalina.properties The only question is if this works in web.xml which I don't recall and I don't have easy access to the source to check. If it doesn't work we should be able to fix that easily. +1. I am sure that it works in web.xml. Also things to note: 1) It is recommended to configure an IP filter (RemoteAddrValve) for manager web application in its context file. http://tomcat.apache.org/tomcat-8.0-doc/security-howto.html#Securing_Management_Applications 2) It should be possible to shadow (override) WEB-INF/web.xml of an application by configuring a PreResources element in its context file, http://tomcat.apache.org/tomcat-8.0-doc/config/resources.html 3) You can rename the application, configure it, pack into a war and upload via the default manager app. The name of the manager application does not matter. It can be renamed. (If you are running with java SecurityManager enabled, it will require additional configuration in policy file, and to configure context file for the application). Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Issue with a principal and remote_user
I posted this on the dev list but I must have placed it on the wrong list... I am running IIS+jk_connect+Tomcat 7.0.59 but this issue was replicated on Tomcat 5.5.36. We are using a security filter from a 3rd party that is failing to engage while requests are sent over AJP via jk_connect. I was able to trace the issue to the 3rd party checking for previously authenticated principal via HttpServletRequest.getUserPrincipal(). Regular call via HTTP connector returns null. Call over jk_connect returns CoyotePrinciapal object but the getName() on it is . The whole issue starts in the jk_isapi_plugin.c where GET_SERVER_VARIABLE_VALUE(REMOTE_USER, s-remote_user); This macro is defined as #define GET_SERVER_VARIABLE_VALUE(name, place) \ do { \ (place) = dup_server_value(private_data-lpEcb, \ (name), \ private_data-p); \ } while(0) dup_server_value is static char *dup_server_value(LPEXTENSION_CONTROL_BLOCK lpEcb, const char *name, jk_pool_t *p) { DWORD sz = HDR_BUFFER_SIZE; char buf[HDR_BUFFER_SIZE]; char *dp; if (lpEcb-GetServerVariable(lpEcb-ConnID, (LPSTR)name, buf, sz)) return jk_pool_strdup(p, buf); and jk_pool_strdup starts as char *jk_pool_strdup(jk_pool_t *p, const char *s) { char *rc = NULL; if (s p) { size_t size = strlen(s); if (!size) { return ; } So essentially GetServerVariable(REMOTE_USER, buf, sz) returns TRUE and sets buf[0]=0 and sz to 0 indicating no REMOTE_USER is present. However, this is converted to by jk_pool_strdup and sent over AJP to Tomcat as a remote_user with size of 0 bytes. Since a remote_user field IS sent to Tomcat, it creates a CoyotePrincipal object with a principal name of empty string. There is a problem somewhere: two requests over two connectors generate two different principal objects (null and empty CoyotePrincipal). If I'd to put a finger, I would say the issue is with the IIS connector converting empty REMOTE_USER value to instead of NULL and thus sending empty remote_user to Tomcat. But a case can be made for Tomcat's AJP processor to check if the incoming remote_user value is and not create a phoney CoyotePrincipal either I'd like to raise an issue but I want to submit it into the correct component. George - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Add member to static memebership cluster during runtime
Am 17.04.2015 um 14:45 schrieb Christopher Schultz: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chris, On 4/17/15 5:33 AM, Christoph Prybila wrote: Am 17.04.2015 um 07:45 schrieb Christopher Schultz: Chris, On 4/16/15 9:59 AM, Christoph Prybila wrote: Tomcat: 7.0.59 OS: Win7 Professional SP1 Java: 1.7.0_67 I want to create a cluster and I need to use static membership. But the number of nodes in my cluster is not static. Is it possible to add members to such a static membership cluster without restarting the other already running tomcat instances? I figured that in order to add a new member I would need to specify it in the StaticMembershipInterceptor of all the other nodes. But editing the server.xml of a running tomcat instance had no effect. Tomcat will not auto-reload server.xml; there are just too many moving parts to reload it quickly, so you have to bounce the whole server. Oddly enough, I was just talking to someone yesterday about how static membership ought to be updatable via at least JMX. I'm not sure, but it's possible that static membership is /already updatable via JMX. Are you comfortable connecting to a local Tomcat using jconsole or VisualVM and browsing around the Catalina tree? I don't have a clustered setup handy to inspect myself to see if it's already possible. If it's /not/ currently possible, it really should be. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org I already tried to configure the cluster via jmx. After connecting to a tomcat instance, there was indeed a Cluster object in the tree but I was not able to add/remove existing members there. This cluster object contains a group called Member where all members of the cluster are listed. Therefore I was able to query information about the members. Unfortunately the member objects do not offer any operations at all. I do not have much experience with jmx but I think that means that the member-entries are read-only and it is currently not possible to configure the membership via jmx. The cluster object itself offers the operations start, stop, setProperty and send which as far as I understood also do not give any possibility to add new members. Agreed. Could you file an enhancement request in Bugzilla? Thanks! - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v2 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJVMQBsAAoJEBzwKT+lPKRYPYgP/A0wrm3zsAt3F9ZAb9+49L0d OELtNDCNfn0iy1uXSEdmke0amOSUKF8FrO7QFdf2BhFtU9IQDxQBX/a/+1oeMMCA 7SMtsNP99yGcl0IropbPpHIrDTP+VJ8dhzva+MEBerPO2glcyns6FvrqW595w/7u 67TdJ+NDrfKj6J9yg1A2xCX96T7ugMgQM1sLxbevXL2CcXQENaLzaZ3WXbKX2cJ1 R03iHq2+zaBjp9fX6sSLlF7khAZ+XMWpgkTRDQiGXb150XKgwmjFkdMsgFz97RPX BQi2tGt6kpagBMsT7J7cpIk8wgJsgnQlZQhIkdwXis4r2QDm1O0UEcjDNzoUFL73 GIDK1K9tcxhMm23bo793KXk5u+qWMKlvmrnJ5K1vL/tMskQ+kAqaD/v6n2vVCAd9 AJCMEY0CoADcVx3v1nSVqH4w+8e0b3wtDehx9wTwgSICWfTkAmu1s41eVB8NELDZ kuokk8Y0t4H3vowCHpnEO7GpXhssBNWNBG+AkM1/i0caOE6ckFH0L8A0nXdxIErM IZ4iyrJVQooHaNmaCoU6ZurOnVb66uEYmPJnCBZcWRtaqfMdvEc4VJAK6sTcnVIA 8t7U9q54cITEXu8MuRCtkrubLjDbueBzMmocsWbE2wsg/ocEbqkaOqgV8Ozk4qZx Npb+77NpR7HtUujPorfC =kD2C -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Done. Thanks for the feedback. https://bz.apache.org/bugzilla/show_bug.cgi?id=57827 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Quinta dos Três Rios
Hi Dave, nice to speak to you just now. The place I mentioned in Portugal is: Quinta dos Três Rios http://www.minola.co.uk/ ...a treasure trove. :-) Hope we'll be able to bump into you soon. All the best, Dave
Re: Configuring Tomcat manager application upload limit
On 17 April 2015 09:37:40 BST, Dominic Hargreaves d...@earth.li wrote: On Wed, Apr 15, 2015 at 08:33:00AM -0500, Mark Thomas wrote: On 15/04/2015 05:28, Dominic Hargreaves wrote: I am running the Tomcat manager application via a Debian package (tomcat8-admin), which deploys the webapp from /usr/share/tomcat8-admin/manager. We ran into a problem hitting th maximum upload size (configured as the multipart-config element in the HTMLManager servlet block). This is easy to fix if you're willing to edit Debian-packaged files, but this is incorrect: files in /usr/share are owned exclusively by dpkg. Is there a way to override this element from somewhere in tomcat's conf directory (which is symlinked to /etc/tomcat8 in Debian)? No. The web application setting would take priority. Okay, thanks for confirming. If it's not possible to override this at the moment, then is this a bug in Tomcat, or the Debian packaging? I'd say the bug is in the Debian packaging (but I would say that wouldn't I). Without know how Tomcat is packaged in Debian, I would expect the following: - the Manager web application to be a separate, optional package Yep, it is. - the web.xml to be placed in an appropriate location for users to be able to edit it. I took a quick look through the web.xml for the Manager app. There are plenty of settings I wouldn't expect to change but there are a handful of things that users might want to tweak. In theory, it should be possible to move those settings to context.xml but that would require code changes and would mean do things that were contrary to the spirit of the servlet spec so there would likely be resistance to such changes from the Tomcat community. One option would be for Debian to use external entities for the configurable elements and place the file that defines those entities somewhere where users can edit it. Okay, that's an interesting idea if it's possible to agree on a sane set :) Thanks for your feedback! Another option (I think) is to use the property replacement feature. I.e. ${property.name} in web.xml and property.name=foo in catalina.properties The only question is if this works in web.xml which I don't recall and I don't have easy access to the source to check. If it doesn't work we should be able to fix that easily. Mark Cheers, Dominic. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat threads dependency on net.core.somaxconn value on Linux
Hi, I would like to support a bust in traffic we are planning to increase the umber of threads by adding executor pool with misSpareThread to 100 and accept count to 5. However, I'm trying to understand the correlation between the tomcat connector acceptCount and ThreadCount to the Linux config net.core.somaxconn(number of incoming connections) which defaults to 128. Does it makes sense to increase the thread count alone with out increasing net.core.somaxconn value? Below is the config that we are planning to use on Tomcat 7.0.42 on Linux 5.x. Executor name=tomcatThreadPool namePrefix=tomcat-http- minSpareThreads=100 maxThreads=300/ Connector port=8080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 enableLookups=false keepAliveTimeout=30 maxKeepAliveRequests=1 acceptCount=5 socketBuffer=10240 executor=tomcatThreadPool compression=on compressableMimeType=text/html, text/xml xpoweredBy=false server=false / Any inputs regarding the Tomcat threads and Linux net.core.somaxconn will be greatly appreciated to support many concurrent connections. Regards Satish Jupalli
Re: GC Issue on Tomcat6
From: Ameer Mawia ameer.ma...@gmail.com To: Tomcat Users List users@tomcat.apache.org Date: 04/17/2015 06:29 PM Subject:Re: GC Issue on Tomcat6 http://ca.linkedin.com/in/ameermawia Toronto, ON Phone:647-262-4380 On Fri, Apr 17, 2015 at 8:47 AM, David kerber dcker...@verizon.net wrote: On 4/17/2015 8:32 AM, Subhro Paul wrote: Hi Team, For our client website we have two Tomcat servers. Both servers are having same configuration which is 16GB RAM, 8CPUS and Linux 6.5 OS. Tomcat 6 and JAVA 6 are installed in both systems. Yesterday, using Jconsole we observed that frequency of GC on one server was very high which was 5 GC hit per Minute and on other server it was 1 GC hit per 3 Minutes. But today we observed that on both of the servers GC frequency is same. Garbage Collection in Oracle Hot spot JVM takes place in generational manner. GC on young generation is invoked much more frequently, while Full GC is invoked once in a while. In any case, frequency is much higher than what you have mentioned. I'm not sure which GC(Young/Full) you are talking about? What pattern of GC you have earlier observed? Is this a know secnario or the server is facing any issue? GC issue has nothing to do with Tomcat per say. It is the memory footprint of your application which will dictate GC behavior. In any case, if your application is facing some memory bottle-neck, it is better to pay attention on the percentage your JVM is spending in GCing your application. If that percentage is very high, say ~90%, that means your application is facing sever memory constrain, your application is almost stalled as JVM has been trying very high to recover some space. Most probably in that case, your JVM will die throwing OOM error in sometime. It was probably just receiving more traffic and/or doing more work of some other kind. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Dear Ameer, I am talking about Young GC which is PS Scavenge for both of the servers. Full GC was Running 1/Hour. Can you please assist how to check the % of memory utilization by GC through JConsole or any other process. Thanks Regards, Subhro Paul =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you
Re: GC Issue on Tomcat6
On Fri, Apr 17, 2015 at 10:07 AM, Subhro Paul subhro.p...@tcs.com wrote: From: Ameer Mawia ameer.ma...@gmail.com To: Tomcat Users List users@tomcat.apache.org Date: 04/17/2015 06:29 PM Subject:Re: GC Issue on Tomcat6 http://ca.linkedin.com/in/ameermawia Toronto, ON Phone:647-262-4380 On Fri, Apr 17, 2015 at 8:47 AM, David kerber dcker...@verizon.net wrote: On 4/17/2015 8:32 AM, Subhro Paul wrote: Hi Team, For our client website we have two Tomcat servers. Both servers are having same configuration which is 16GB RAM, 8CPUS and Linux 6.5 OS. Tomcat 6 and JAVA 6 are installed in both systems. Yesterday, using Jconsole we observed that frequency of GC on one server was very high which was 5 GC hit per Minute and on other server it was 1 GC hit per 3 Minutes. But today we observed that on both of the servers GC frequency is same. Garbage Collection in Oracle Hot spot JVM takes place in generational manner. GC on young generation is invoked much more frequently, while Full GC is invoked once in a while. In any case, frequency is much higher than what you have mentioned. I'm not sure which GC(Young/Full) you are talking about? What pattern of GC you have earlier observed? Is this a know secnario or the server is facing any issue? GC issue has nothing to do with Tomcat per say. It is the memory footprint of your application which will dictate GC behavior. In any case, if your application is facing some memory bottle-neck, it is better to pay attention on the percentage your JVM is spending in GCing your application. If that percentage is very high, say ~90%, that means your application is facing sever memory constrain, your application is almost stalled as JVM has been trying very high to recover some space. Most probably in that case, your JVM will die throwing OOM error in sometime. It was probably just receiving more traffic and/or doing more work of some other kind. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Dear Ameer, I am talking about Young GC which is PS Scavenge for both of the servers. Full GC was Running 1/Hour. Can you please assist how to check the % of memory utilization by GC through JConsole or any other process. Subhro, please reply inline. People prefer that way here. It makes reading the thread easier. I was talking about percentage utilization of CPU Time by GC Threads. Getting % Memory usage is also very helpful. There is a number of way to check Memory utilization, stats: - JConsole Memory tab. - Console output. You will have to turn on the JVM flag to dump these details. - -XX:+PrintGCDetails -- This also give time spent in a particular GC operation. - Thread dump( Kill -3 Unix machine) Also, you can get some fine tuned information about time usage by your application threads, using these flags: - -XX:+PrintGCApplicationConcurrentTime - -XX:+PrintGCApplicationStoppedTime Out of curiosity, how did you get this information Full GC was Running 1/Hour. I guess that is too few run. Thanks Regards, Subhro Paul =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you -- http://ca.linkedin.com/in/ameermawia Toronto, ON
Re: Global JNDI resources lookup behavior difference between version 6.0.39 and 6.0.41/7.0.55
Issue fixed: https://github.com/psi-probe/psi-probe/commit/9f233d0f0ccb02062f2b094710897b2a551d0ca5 2014-09-04 11:07 GMT-03:00 Robert Anderson ranom...@gmail.com: 2014-09-04 10:57 GMT-03:00 Christopher Schultz ch...@christopherschultz.net: I don't know. You could inspect a Subversion log between the two. It's no fun, but it will show you what actually changed instead of what was documented to have been changed. It's exactly what I'm doing at this moment. :)