Re: [DBCP] troubleshooting pool activity (tomcat version)
First thing to do is thank you again for taking the time to help me. Apache has great communities. On 3/22/2018 5:38 PM, Phil Steitz wrote: > You must be looking at documentation describing how to use the > alternative pool mentioned above (tomcat-jdbc). The config you > posted is correct for DBCP. I'm looking at Tomcat documentation. https://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html The tomcat is the one included with Liferay 6.2. It is 7.0.42. > Don't look at DBCP 2 code for troubleshooting the code you are > running. Either look at the repackaged sources inside the tomcat > source, or find the version in the tomcat build files and go to the > old DBCP / pool sources referenced there. I have figured that out. Felt pretty dumb when I realized I wasn't looking at code from the correct Tomcat version. > Of course, you *should* upgrade both TC and the DBCP it ships so you > *can* look at that (much better) code. See below for one reason why. I hear what you're saying, and don't disagree ... but this is not the kind of environment where I can just do an upgrade, even though upgrading might make it all better. We didn't download Tomcat -- we downloaded Liferay, which came with a specific version of Tomcat already included. Upgrading any significant component (liferay, tomcat, and others) runs the risk that when we restart the service, our web application won't work any more. For any upgrade, we have to spend a lot of resources trying the upgrade in a staging environment, so we can be sure that everything still works. Because that's very time-consuming, we tend to not do a lot of upgrading, at least of significant components, and our versions get REALLY old. This is also why I'm hesitant to move away from Tomcat's DBCP implementation to Commons DBCP (particularly version 2), even though that's exactly what I want to do. Switching to a different library might work seamlessly ... or it might completely break the application. Our customers get REALLY irritated when the websites we've built for them don't work! > One thing that could be going on is that in the old 1.x DBCP, > abandoned connection removal only happens when borrows are > attempted. So if you check out a lot of connections, abandon them > and don't ask for more, they won't get closed as abandoned until you > borrow a new one. In DBCP 2, the removeAbandoned property is split > into two different properties: removeAbandonedOnBorrow (the old > behavior) and removeAbandonedOnMaintenance. The second one makes > abandoned connection removal run on pool maintenance (so will not > have to wait until a borrow is attempted). I don't know if anyone needs me to actually back up and describe what's happening that led me down this rabbit hole, but that's what I'm going to do: The master MySQL server in our environment has a max connection limit configured at 600 connections. Every now and then, we start getting website failures, because all the connections are in use and the connection pools can't make any more connections. Looking at the connections on the MySQL side, the vast majority are idle (the command is "Sleep" on the server processlist), and have been idle for several hours. There are five main webservers and a handful of ancillary systems that also connect to the database. When the problem happens, the connection count from each webserver has gotten up near 100, and sometimes over 100. The surplus of connections are definitely the ones configured in Tomcat. Liferay has its own DB config for its own data (using c3p0 for pooling), and although I often see a higher number of connections to that database than I would excpect, I've never seen the idle time on those connections above one minute, so I'm not concerned about that pool, beyond some minor tweaks. The frequency of the connection-related failures has been increasing, so in response, I have set up monitoring that will send us an alarm when the server reaches 550 connections. This has allowed us to kill idle connections and prevent customer-visible problems a couple of times already, but we still have a fundamental issue to correct. I do not yet have any information that indicates whether Tomcat's DBCP thinks those connections are idle or active. I have reason to suspect that they are active, and have not been returned to the pool (closed). I've worked out a way with one of our developers to add logging that displays the active and idle connection counts, but it's not yet in production. If those connections were idle, as the MySQL server thinks they are, it really seems like DBCP would be choosing to re-use a connection that it's already got, instead of trying to create a brand new one and failing. So I am chasing abandoned connection removal. We have it configured, but it's not working. The config is lacking things I think it needs, but as far as I could tell, there is enough for abandoned connection removal to work. I suspect it's not working because
Re: [DBCP] troubleshooting pool activity (tomcat version)
On 3/21/18 12:15 PM, Shawn Heisey wrote: > On 3/21/2018 1:31 AM, Mark Thomas wrote: >>> and that we need to >>> change the factory on our pool definitions. >> I believe not. > Tomcat's documentation seems to disagree with this point. It > specifically says to use org.apache.tomcat.jdbc.pool.DataSourceFactory, > and doesn't list any other valid choices. You must be looking at documentation describing how to use the alternative pool mentioned above (tomcat-jdbc). The config you posted is correct for DBCP. > But we have > org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory in our > configuration. I've found questions from people whose Tomcat > installations didn't even contain that class, where the answers said to > switch to the factory in the documentation. Apparently deb-based > packages of Tomcat do not contain the latter class. > > I found a historical document for 7.0.42 with Google and it also shows > that the factory should be set to the DataSourceFactory, not the > BasicDataSourceFactory that we have. > > I do see "removeAbandoned" in the property constants in the 7.0 source > code for BasicDataSourceFactory I had thought that it wasn't there, > but I realized that I was looking at trunk code when I came to that > conclusion, and that BasicDataSourceFactory is in a dbcp2 package in > trunk. Don't look at DBCP 2 code for troubleshooting the code you are running. Either look at the repackaged sources inside the tomcat source, or find the version in the tomcat build files and go to the old DBCP / pool sources referenced there. Of course, you *should* upgrade both TC and the DBCP it ships so you *can* look at that (much better) code. See below for one reason why. > So I'm not sure what to think, but I can say that the abandoned > connection handling does not appear to actually be working. So either > my configuration is wrong, or the factory that we are using is ignoring > part of the config. One thing that could be going on is that in the old 1.x DBCP, abandoned connection removal only happens when borrows are attempted. So if you check out a lot of connections, abandon them and don't ask for more, they won't get closed as abandoned until you borrow a new one. In DBCP 2, the removeAbandoned property is split into two different properties: removeAbandonedOnBorrow (the old behavior) and removeAbandonedOnMaintenance. The second one makes abandoned connection removal run on pool maintenance (so will not have to wait until a borrow is attempted). Phil > > Thanks, > Shawn > > p.s. I sent an earlier draft of this message, but did so from the wrong > email address. If the moderators decide to allow that message through, > then you may see an almost-duplicate of this message. > > - > To unsubscribe, e-mail: user-unsubscr...@commons.apache.org > For additional commands, e-mail: user-h...@commons.apache.org > > - To unsubscribe, e-mail: user-unsubscr...@commons.apache.org For additional commands, e-mail: user-h...@commons.apache.org
[crypto]
I’m building Apache Commons Crypto from source on Mac OS HighSierra (10.13.3) and the CryptoRandom tests fail, apparently because of failure to load the underlying random number generation library. All other tests pass. I successfully built and tested on Ubuntu 16.04 in a similar development environment, so the problem seems isolated to my Mac. Has anyone had a similar experience, or have any ideas why the Mac is exhibiting this behavior? I’ve tested two versions of openssl, and both versions produced identical failures. Best wishes, Alex [INFO] Scanning for projects... [INFO] [INFO] [INFO] Building Apache Commons Crypto 1.1.0-SNAPSHOT [INFO] [INFO] [INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-maven-3) @ commons-crypto --- [INFO] [INFO] --- build-helper-maven-plugin:3.0.0:parse-version (parse-version) @ commons-crypto --- [INFO] [INFO] --- maven-resources-plugin:2.7:resources (create-version-file) @ commons-crypto --- [INFO] Using 'iso-8859-1' encoding to copy filtered resources. [INFO] Copying 1 resource [INFO] [INFO] --- maven-antrun-plugin:1.8:run (javadoc.resources) @ commons-crypto --- [INFO] Executing tasks main: [copy] Copying 2 files to /Users/Alex/Documents/git_repos/commons-crypto/target/apidocs/META-INF [INFO] Executed tasks [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles) @ commons-crypto --- [INFO] [INFO] --- buildnumber-maven-plugin:1.4:create (default) @ commons-crypto --- [INFO] Executing: /bin/sh -c cd '/Users/Alex/Documents/git_repos/commons-crypto' && 'git' 'rev-parse' '--verify' 'HEAD' [INFO] Working directory: /Users/Alex/Documents/git_repos/commons-crypto [INFO] Storing buildNumber: d69ef9556f44c076034add15feef1c5513248404 at timestamp: 2018-03-22 17:12:10-0400 [INFO] Storing buildScmBranch: master [INFO] [INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ commons-crypto --- [INFO] Using 'iso-8859-1' encoding to copy filtered resources. [INFO] Copying 1 resource [INFO] [INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ commons-crypto --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 38 source files to /Users/Alex/Documents/git_repos/commons-crypto/target/classes [INFO] [INFO] --- maven-antrun-plugin:1.8:run (make) @ commons-crypto --- [INFO] Executing tasks make: [exec] "/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/bin/javah" -force -classpath target/classes -o target/jni-classes/org/apache/commons/crypto/random/OpenSslCryptoRandomNative.h org.apache.commons.crypto.random.OpenSslCryptoRandomNative [exec] gcc -arch x86_64 -Ilib/inc_mac -I/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/include -O2 -fPIC -mmacosx-version-min=10.5 -fvisibility=hidden -I/usr/local/include -I/usr/local/Cellar/openssl/1.0.2n/include -Ilib/include -I/usr/include -I"src/main/native/org/apache/commons/crypto/" -I"/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/include/darwin" -I"target/jni-classes/org/apache/commons/crypto/cipher" -I"target/jni-classes/org/apache/commons/crypto/random" -c src/main/native/org/apache/commons/crypto/random/OpenSslCryptoRandomNative.c -o target/commons-crypto-1.1.0-SNAPSHOT-Mac-x86_64/OpenSslCryptoRandomNative.o [exec] "/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/bin/javah" -force -classpath target/classes -o target/jni-classes/org/apache/commons/crypto/cipher/OpenSslNative.h org.apache.commons.crypto.cipher.OpenSslNative [exec] gcc -arch x86_64 -Ilib/inc_mac -I/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/include -O2 -fPIC -mmacosx-version-min=10.5 -fvisibility=hidden -I/usr/local/include -I/usr/local/Cellar/openssl/1.0.2n/include -Ilib/include -I/usr/include -I"src/main/native/org/apache/commons/crypto/" -I"/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/include/darwin" -I"target/jni-classes/org/apache/commons/crypto/cipher" -I"target/jni-classes/org/apache/commons/crypto/random" -c src/main/native/org/apache/commons/crypto/cipher/OpenSslNative.c -o target/commons-crypto-1.1.0-SNAPSHOT-Mac-x86_64/OpenSslNative.o [exec] "/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/bin/javah" -force -classpath target/classes -o target/jni-classes/org/apache/commons/crypto/OpenSslInfoNative.h org.apache.commons.crypto.OpenSslInfoNative [exec] gcc -arch x86_64 -Ilib/inc_mac -I/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/include -O2 -fPIC -mmacosx-version-min=10.5 -fvisibility=hidden -I/usr/local/include -I/usr/local/Cellar/openssl/1.0.2n/include -Ilib/include -I/usr/include -I"src/main/native/org/apache/commons/crypto/"
[Daemon] StdOut and StdErr use extra newline
I'm using the daemon to wrap my Jetty based service. I've redirected stdout and stderr to the Daemon via the relevant configuration params. However, both files end up with 2 newline characters (/r/n/r/n) which results in a very unreadable log... especially when dealing with stacktraces... Any idea what could cause that? BTW, when using Jetty without the wrapper, Jetty's stdout and err do not suffer from this behavior. thanks, GBa.
