Tomcat connection pool bleeding under heavy load
Hello! We have received an issue from our customer that after some time in production they get empty connection pool. After some time we managed to reproduce their problem in our test environment with smaller number of pool connections. So generally we have a problem that if connection pool is under heavy load (meaning most of the time all the connections are taken - active) it starts to loose database connections (we call it bleed): they are lost from the perspective of Tomcat but they are very much alive when we check then on the database side. We are using Apache Tomcat 7.0.47 running on java 1.6. Database is SQL Server 2008R2 and we are using Hibernate as persistence provider. Our configuration is defined in server.xml in the GlobalNamingResources section (referenced from context.xml): Resource name=jdbc/ChiDS auth=Container type=javax.sql.DataSource factory=org.apache.tomcat.jdbc.pool.DataSourceFactory driverClassName=com.microsoft.sqlserver.jdbc.SQLServerDriver url=jdbc:sqlserver://*;databaseName=chiTestDB username=* password=* initialSize=2 maxActive=5 maxIdle=4 minIdle=2 maxWait=15000 timeBetweenEvictionRunsMillis=3 testOnBorrow=true testOnReturn=false testOnIdle=false validationQuery=SELECT 1 validationInterval=3 removeAbandoned=true removeAbandonedTimeout=60 useEquals=false/ I will explain in detail what happens. For better understanding I am also providing graph of idle and active connections from Java VisualVM MBeans viewer: http://img11.imageshack.us/img11/6888/fm4j.jpg So we start the server and connect clients to the server that generate small database load. After that we order half of our clients to start generating operations that result in medium database load (I think around 300 transactions per second). We can see from the graph that connection pool is behaving normally (sometimes all 5 connections are taken, sometimes not - if we would run test like this all day everything would be OK). So after 7 minutes we increase the load by 100% (other half of clients start to do operations). The load is bigger than 5 pool connections can handle: pretty much all the time all 5 connections are active so we can soon expect that server application logic will start to receive exceptions that jdbc connection is not available (this should happen if application logic does not receive pool connection in 15 seconds). Indeed this happens and the following exception is thrown: aused by: org.hibernate.exception.GenericJDBCException: Cannot open connection at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:140) at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:128) at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66) at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:52) at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:449) at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167) at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:142) at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:85) at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1463) at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:60) ... 18 more Caused by: java.sql.SQLException: Pool wait interrupted. at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:655) at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) at org.hibernate.connection.DatasourceConnectionProvider.getConnection(DatasourceConnectionProvider.java:92) at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446) ... 23 more Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1024) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253) at org.apache.tomcat.jdbc.pool.FairBlockingQueue.poll(FairBlockingQueue.java:151) at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:650) But here something goes wrong because active connections count drops by 1 and never goes back even though database load is still the same. Judging from the graph at this moment pool is giving only 4 pool connections to the application. Pretty soon the same exception is thrown again reducing the active pool count by 1. And then very quickly same exception is thrown 3 more times and pool has no active or idle connections. After this another type of
Re: EL 3.0 Streams Edge Case - Am I wrong or is Tomcat wrong?
Being convinced now that this is a Tomcat bug in violation of the spec--and not
something I am doing wrong--I'm going to go ahead and file a bug now.
N
On Nov 17, 2013, at 9:32 AM, Nick Williams wrote:
On Nov 17, 2013, at 6:00 AM, Konstantin Kolinko wrote:
2013/11/17 Nick Williams nicho...@nicholaswilliams.net:
I have an EL 3.0 edge case that I need help understanding. Am I doing
something wrong (I don't think so) or is the Tomcat 8.0 implementation
missing something?
Consider the following EL expression:
${users.stream()
.filter(u - fn:contains(u.username, '1'))
.sorted((u1, u2) - u1.lastName.compareTo(u2.lastName) == 0 ?
u1.firstName.compareTo(u2.firstName) : u1.lastName.compareTo(u2.lastName))
.toList()}
This works as expected. However, it results in potentially evaluating
u1.lastName.compareTo(u2.lastName) twice. My understanding is that the
right-hand side of a lambda expression can be any valid EL expression, so I
believe this should also work:
${users.stream()
.filter(u - fn:contains(u.username, '1'))
.sorted((u1, u2) - x = u1.lastName.compareTo(u2.lastName); x
== 0 ? u1.firstName.compareTo(u2.firstName) : x)
.toList()}
However, this doesn't evaluate. I get the following error instead:
org.apache.el.parser.ParseException: Encountered = = at line 3,
column 38.
Was expecting one of:
. ...
) ...
etc ...
What if you add ( ) ?
What operator has higher priority, - or ; ?
- has higher priority than both = and ;, according to the spec. In
this particular case, I'm not sure whether that means parenthesis are
absolutely required or not. However, I can confirm that adding parenthesis
here solves this problem, so perhaps that's what I was doing wrong for this
error. This expression now works:
${users.stream()
.filter(u - fn:contains(u.username, '1'))
.sorted((u1, u2) - (x = u1.lastName.compareTo(u2.lastName); x
== 0 ? u1.firstName.compareTo(u2.firstName) : x))
.toList()}
Next I tried to reduce the properties present in each user using the stream
map method. Once again, with the understanding that the right-hand side
of a lambda expression can be any valid EL expression, I use an EL Map
literal to construct a reduced set of properties:
${users.stream()
.filter(u - fn:contains(u.username, '1'))
.map(u - {'username':u.username, 'first':u.firstName,
'last':u.lastName})
.sorted((u1, u2) - u1.lastName.compareTo(u2.lastName) == 0 ?
u1.firstName.compareTo(u2.firstName) : u1.lastName.compareTo(u2.lastName))
.toList()}
However, that doesn't work and I get this error:
org.apache.el.parser.ParseException: Encountered EOF at line 3, column
88.
Was expecting one of:
. ...
) ...
etc ...
I do not understand the above. Can you provide a simple test case?
Which one of the expressions does not work? Can you remove the others?
So, as mentioned above, the following expression works:
${users.stream()
.filter(u - fn:contains(u.username, '1'))
.sorted((u1, u2) - (x = u1.lastName.compareTo(u2.lastName); x
== 0 ? u1.firstName.compareTo(u2.firstName) : x))
.toList()}
If I now add the map operation to it, I get the EOF error. Nothing else
about the expression changed:
${users.stream()
.filter(u - fn:contains(u.username, '1'))
.sorted((u1, u2) - (x = u1.lastName.compareTo(u2.lastName);
x == 0 ? u1.firstName.compareTo(u2.firstName) : x))
.map(u - {'username':u.username, 'first':u.firstName,
'last':u.lastName})
.toList()}
javax.el.ELException: Failed to parse the expression [${users.stream()
.filter(u - fn:contains(u.username, '1'))
.sorted((u1, u2) - (x = u1.lastName.compareTo(u2.lastName);
x == 0 ? u1.firstName.compareTo(u2.firstName) : x))
.map(u - {'username':u.username, 'first':u.firstName,
'last':u.lastName}]
...
root cause
org.apache.el.parser.ParseException: Encountered EOF at line 6, column
38. Was expecting one of:
. ...
) ...
etc...
Notice that it thinks the expression is ending after the closing } of the
map-literal. Now the example in 2.3.6.4 of the specification alludes to the
fact that my use of the map-literal here is correct. In that example they use
a list-literal instead (.map(p-[p.name, p.unitPrice])). I tried changing to
use a list-literal instead of a map-literal, and that's when I got the
NumberFormatException described earlier. I _believe_ both should be legal;
the specification clearly intends that at least using the list-literal should
be legal. Neither work in Tomcat.
Does this make sense?
Section 2.3.6.4 of
RE: [OT] RE: Baked-in context paths
Hi, -Original Message- From: Konstantin Preißer [mailto:kpreis...@apache.org] Sent: Wednesday, November 13, 2013 6:14 PM To: 'Tomcat Users List' Subject: RE: [OT] RE: Baked-in context paths Hi Jeffrey, -Original Message- From: Jeffrey Janner [mailto:jeffrey.jan...@polydyne.com] Sent: Wednesday, November 13, 2013 5:14 PM To: 'Tomcat Users List' Subject: RE: [OT] RE: Baked-in context paths The thing is, the garbage characters at the end don't exist in the email I see as I type. They are either hidden in the editor or getting added by Outlook (2007) or Exchange (2010). They are added by the list mailer of this mailing list, not by your mail client - it is the the following lines: - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org The problem here is that your client sent the mail content as Base64, and the list mailer adds these lines as a separate Base64 block which (in my opinion) is probably wrong - it produces invalid Base64 data, which causes such junk characters on some mailing clients (and an empty mail on my client). This also happens with some other mails like the one from Buildbot on the Dev list - these are are also sent as Base64 which then show blank for me. Should I report this problem somewhere, e.g. on the Apache Infra JIRA? Note that this problem does not always appear - I think it happens only if (original_message_length_in_bytes % 3 != 0), because then the last 4 characters sequence does not represent 3 bytes, but only 1 or 2, and therefore has some = characters at the end. However, in the other case, the last 4 characters represent full 3 bytes, so another Base64 block can be added without corrupting the data. E.g., if you look at this mail from Buildbot [1], you can see that the bottom contains the To unsubscribe... text added by the mailing list. This mail displayed correctly in my mail client. If you look at the raw message, you can see that the first Base64 block represents 429 bytes, and as 429 % 3 == 0 there isn't a problem. If you look at this mail from Buildbot [2], then the bottom doesn't contain the To unsubscribe lines. In the raw message, you can see the first Base64 block represents 403 bytes (403 % 3 == 1), so it contains == at the end, and the added Base64 block makes the Base64 invalid. This email displayed blank in my client. [1] http://mail-archives.apache.org/mod_mbox/tomcat-dev/201311.mbox/%3C20131118012335.AB0A6C04A4%40aegis.apache.org%3E [2] http://mail-archives.apache.org/mod_mbox/tomcat-dev/201311.mbox/%3C20131118032322.48526C04A4%40aegis.apache.org%3E Regards, Konstantin Preißer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Curious difference in connection behaviour on database side DBCP vs. JDBC?
Hello, We have recently migrated from dbcp pool to the newer tomcat-jdbc pool. As I understand, it is supposed to be almost a drop in replacement for dbcp. Everything works great except once small thing. Its a bit difficult for me to explain but our DBA is really annoyed by it... With the new jdbc configuration, idle connections never really turn into normal idle on the database side. The DBS looks at the connections on the db side with sp_whoisactive (excellent improvement of the sp_whoX procedures by Adam Machanic). It should just show the statement/queries that are actually running. After that the connection should be empty and idle and thus not be visible... But with the new JDBC pool it shows the last statement/query executed in the connection and it only changes when a new query comes from the application (see below)* I therefore know that the connection gets reused as an idle connection should but this is not the expected behaviour. It should not linger on th db server like it does.. I automatically would assume there is something in the code not closing statements properly and/or connection leak or some such... BUT When using the old dbcp pool configuration all connections behaved as expected .. .statement/query get run and the connection turns to a normal idle connection with no information regarding earlier queries/statements... therefore it never shows up in sp_whoisactive which is expected. For example: rebooting the application an not doing anything (test env. no users no nothing) if I execute sp_whoisactive I currently see the default startup query from the app * 00 00:31:59.793 333 ?query -- update Account set loggedIn = 0 where loggedIn = 1 --? Sleeping Now this is a short simple fast query and it shouldn't show here but its shows as been hanging around for half an hour. It doesn't hang around like this if I revert to the old dbcp configuration... - the issue Am I missing something fundamental in expected behaviour or should I just go shout at the devs (who will only say it worked with the old pool so it should work the same now)... We are using tomcat 6.32 / jdk1.6.0_26 with MS SQL server 2012 Old dbcp conf, everything works fully as expected Resource name=database1 auth=Container maxActive=50 maxIdle=8 username=* password= driverClassName=com.inet.tds.TdsDriver type=javax.sql.DataSource defaultTransactionIsolation=READ_UNCOMMITTED url=jdbc:inetdae7://sqlserver12:1433/database1_dev timeBetweenEvictionRunsMillis=30 removeAbandoned=true removeAbandonedTimeout=600 maxWait=1/ JDBC configuration with tomca-jdbc.jar taken from tomcat7.42, has idle connections with statement/query info in them, so that the dba never really sees them as fully idle Resource name=database1 auth=Container maxActive=50 maxIdle=10 minIdle=2 initialSize=2 username=* password=* driverClassName=com.inet.tds.TdsDriver type=javax.sql.DataSource factory=org.apache.tomcat.jdbc.pool.DataSourceFactory defaultTransactionIsolation=READ_UNCOMMITTED jdbcInterceptors=ConnectionState;StatementFinalizer defaultAutoCommit=true url=jdbc:inetdae7://sqlserver12:1433/database1_dev testOnBorrow=true validationQuery=SELECT 1 timeBetweenEvictionRunsMillis=1 removeAbandoned=true removeAbandonedTimeout=1800 maxWait=1/ I have googled, read documentation, browsed forums but never seen anyone having an issue like this... and I admit, its nothing major. I'm just very curious about the source to this difference in behaviour (and can I do something configuration wise to resolve it?). Best regards --- Carl Boberg
Re: Curious difference in connection behaviour on database side DBCP vs. JDBC?
On Nov 18, 2013, at 9:48 AM, Carl Boberg carl.bob...@memnonnetworks.com wrote: Hello, We have recently migrated from dbcp pool to the newer tomcat-jdbc pool. As I understand, it is supposed to be almost a drop in replacement for dbcp. *Almost* is the key word here. It's very similar, but there are differences (typically with default values). Most of the time these don't matter, but occasionally you'll bump into them. Everything works great except once small thing. Its a bit difficult for me to explain but our DBA is really annoyed by it... With the new jdbc configuration, idle connections never really turn into normal idle on the database side. The DBS looks at the connections on the db side with sp_whoisactive (excellent improvement of the sp_whoX procedures by Adam Machanic). It should just show the statement/queries that are actually running. After that the connection should be empty and idle and thus not be visible... But with the new JDBC pool it shows the last statement/query executed in the connection and it only changes when a new query comes from the application (see below)* I therefore know that the connection gets reused as an idle connection should but this is not the expected behaviour. It should not linger on th db server like it does.. Sorry, I'm not certain what you mean here. I don't use MSSQL, but perhaps someone else does and can comment. I automatically would assume there is something in the code not closing statements properly and/or connection leak or some such… Entirely possible. A good test here would be to deploy a trivial application that you know is closing connections properly. If that still exhibits the same behavior then you can rule out the application, plus you'll have a good test application that you can use to replicate the problem. Probably also worth enabling logAbandoned, so that you can see if you're getting abandoned connections with either configuration. BUT When using the old dbcp pool configuration all connections behaved as expected .. .statement/query get run and the connection turns to a normal idle connection with no information regarding earlier queries/statements... therefore it never shows up in sp_whoisactive which is expected. It might help if you can be more specific about what these stored procedures are doing. What are they looking at? What is different about the data that they are looking at for DBCP tomcat-jdbc? Also if you could correlate this data to what is shown in JMX for the pool, that would help. For example, if the database shows 5 connections with 3 idle, what does the connection pool show in JMX? Does this match? For example: rebooting the application an not doing anything (test env. no users no nothing) if I execute sp_whoisactive I currently see the default startup query from the app * 00 00:31:59.793 333 ?query -- update Account set loggedIn = 0 where loggedIn = 1 --? Sleeping Now this is a short simple fast query and it shouldn't show here but its shows as been hanging around for half an hour. It doesn't hang around like this if I revert to the old dbcp configuration... - the issue Am I missing something fundamental in expected behaviour or should I just go shout at the devs (who will only say it worked with the old pool so it should work the same now)... We are using tomcat 6.32 / jdk1.6.0_26 with MS SQL server 2012 Old dbcp conf, everything works fully as expected Resource name=database1 auth=Container maxActive=50 maxIdle=8 username=* password= driverClassName=com.inet.tds.TdsDriver type=javax.sql.DataSource defaultTransactionIsolation=READ_UNCOMMITTED url=jdbc:inetdae7://sqlserver12:1433/database1_dev timeBetweenEvictionRunsMillis=30 removeAbandoned=true removeAbandonedTimeout=600 maxWait=1/ In this config, minIdle is going to be 0. That's the default listed here. https://commons.apache.org/proper/commons-dbcp/configuration.html In the config below, you're setting minIdle to 2. That means your new configuration will always have at least two connections to the database server, whereas the old one could idle down to zero. Some other differences in configuration: - old configuration doesn't specify a validation query - the timeBetweenEvictionRunsMillis is quite different (30 vs 1) - remove abandoned timeout is different by a factor of 3 - maxIdle is 8 vs 10 - initial size is 0 vs 2 Probably worth making the configurations as similar as possible. JDBC configuration with tomca-jdbc.jar taken from tomcat7.42, has idle connections with statement/query info in them, so that the dba never really sees them as fully idle Resource name=database1 auth=Container maxActive=50 maxIdle=10 minIdle=2 initialSize=2
Re: EL 3.0 Streams Edge Case - Am I wrong or is Tomcat wrong?
2013/11/17 Nick Williams nicho...@nicholaswilliams.net:
On Nov 17, 2013, at 6:00 AM, Konstantin Kolinko wrote:
2013/11/17 Nick Williams nicho...@nicholaswilliams.net:
I have an EL 3.0 edge case that I need help understanding. Am I doing
something wrong (I don't think so) or is the Tomcat 8.0 implementation
missing something?
Consider the following EL expression:
${users.stream()
.filter(u - fn:contains(u.username, '1'))
.sorted((u1, u2) - u1.lastName.compareTo(u2.lastName) == 0 ?
u1.firstName.compareTo(u2.firstName) : u1.lastName.compareTo(u2.lastName))
.toList()}
This works as expected. However, it results in potentially evaluating
u1.lastName.compareTo(u2.lastName) twice. My understanding is that the
right-hand side of a lambda expression can be any valid EL expression, so I
believe this should also work:
${users.stream()
.filter(u - fn:contains(u.username, '1'))
.sorted((u1, u2) - x = u1.lastName.compareTo(u2.lastName); x
== 0 ? u1.firstName.compareTo(u2.firstName) : x)
.toList()}
However, this doesn't evaluate. I get the following error instead:
org.apache.el.parser.ParseException: Encountered = = at line 3,
column 38.
Was expecting one of:
. ...
) ...
etc ...
What if you add ( ) ?
What operator has higher priority, - or ; ?
- has higher priority than both = and ;, according to the spec. In
this particular case, I'm not sure whether that means parenthesis are
absolutely required or not. However, I can confirm that adding parenthesis
here solves this problem, so perhaps that's what I was doing wrong for this
error. This expression now works:
${users.stream()
.filter(u - fn:contains(u.username, '1'))
.sorted((u1, u2) - (x = u1.lastName.compareTo(u2.lastName); x
== 0 ? u1.firstName.compareTo(u2.firstName) : x))
.toList()}
Tomcat is performing correctly here.
(For a reference: the priorities are specified by Ch.1.16 of EL 3.0).
Next I tried to reduce the properties present in each user using the stream
map method. Once again, with the understanding that the right-hand side
of a lambda expression can be any valid EL expression, I use an EL Map
literal to construct a reduced set of properties:
${users.stream()
.filter(u - fn:contains(u.username, '1'))
.map(u - {'username':u.username, 'first':u.firstName,
'last':u.lastName})
.sorted((u1, u2) - u1.lastName.compareTo(u2.lastName) == 0 ?
u1.firstName.compareTo(u2.firstName) : u1.lastName.compareTo(u2.lastName))
.toList()}
However, that doesn't work and I get this error:
org.apache.el.parser.ParseException: Encountered EOF at line 3, column
88.
Was expecting one of:
. ...
) ...
etc ...
I do not understand the above. Can you provide a simple test case?
Which one of the expressions does not work? Can you remove the others?
So, as mentioned above, the following expression works:
${users.stream()
.filter(u - fn:contains(u.username, '1'))
.sorted((u1, u2) - (x = u1.lastName.compareTo(u2.lastName); x
== 0 ? u1.firstName.compareTo(u2.firstName) : x))
.toList()}
If I now add the map operation to it, I get the EOF error. Nothing else
about the expression changed:
${users.stream()
.filter(u - fn:contains(u.username, '1'))
.sorted((u1, u2) - (x = u1.lastName.compareTo(u2.lastName);
x == 0 ? u1.firstName.compareTo(u2.firstName) : x))
.map(u - {'username':u.username, 'first':u.firstName,
'last':u.lastName})
.toList()}
javax.el.ELException: Failed to parse the expression [${users.stream()
.filter(u - fn:contains(u.username, '1'))
.sorted((u1, u2) - (x = u1.lastName.compareTo(u2.lastName);
x == 0 ? u1.firstName.compareTo(u2.firstName) : x))
.map(u - {'username':u.username, 'first':u.firstName,
'last':u.lastName}]
...
root cause
org.apache.el.parser.ParseException: Encountered EOF at line 6, column
38. Was expecting one of:
. ...
) ...
etc...
Notice that it thinks the expression is ending after the closing } of the
map-literal. Now the example in 2.3.6.4 of the specification alludes to the
fact that my use of the map-literal here is correct. In that example they use
a list-literal instead (.map(p-[p.name, p.unitPrice])). I tried changing to
use a list-literal instead of a map-literal, and that's when I got the
NumberFormatException described earlier. I _believe_ both should be legal;
the specification clearly intends that at least using the list-literal should
be legal. Neither work in Tomcat.
Does this make sense?
The early '}' terminating the EL
Re: EL 3.0 Streams Edge Case - Am I wrong or is Tomcat wrong?
On Nov 18, 2013, at 12:05 PM, Konstantin Kolinko wrote:
2013/11/17 Nick Williams nicho...@nicholaswilliams.net:
On Nov 17, 2013, at 6:00 AM, Konstantin Kolinko wrote:
2013/11/17 Nick Williams nicho...@nicholaswilliams.net:
I have an EL 3.0 edge case that I need help understanding. Am I doing
something wrong (I don't think so) or is the Tomcat 8.0 implementation
missing something?
Consider the following EL expression:
${users.stream()
.filter(u - fn:contains(u.username, '1'))
.sorted((u1, u2) - u1.lastName.compareTo(u2.lastName) == 0 ?
u1.firstName.compareTo(u2.firstName) : u1.lastName.compareTo(u2.lastName))
.toList()}
This works as expected. However, it results in potentially evaluating
u1.lastName.compareTo(u2.lastName) twice. My understanding is that the
right-hand side of a lambda expression can be any valid EL expression, so
I believe this should also work:
${users.stream()
.filter(u - fn:contains(u.username, '1'))
.sorted((u1, u2) - x = u1.lastName.compareTo(u2.lastName); x
== 0 ? u1.firstName.compareTo(u2.firstName) : x)
.toList()}
However, this doesn't evaluate. I get the following error instead:
org.apache.el.parser.ParseException: Encountered = = at line 3,
column 38.
Was expecting one of:
. ...
) ...
etc ...
What if you add ( ) ?
What operator has higher priority, - or ; ?
- has higher priority than both = and ;, according to the spec. In
this particular case, I'm not sure whether that means parenthesis are
absolutely required or not. However, I can confirm that adding parenthesis
here solves this problem, so perhaps that's what I was doing wrong for this
error. This expression now works:
${users.stream()
.filter(u - fn:contains(u.username, '1'))
.sorted((u1, u2) - (x = u1.lastName.compareTo(u2.lastName); x
== 0 ? u1.firstName.compareTo(u2.firstName) : x))
.toList()}
Tomcat is performing correctly here.
(For a reference: the priorities are specified by Ch.1.16 of EL 3.0).
Next I tried to reduce the properties present in each user using the
stream map method. Once again, with the understanding that the
right-hand side of a lambda expression can be any valid EL expression, I
use an EL Map literal to construct a reduced set of properties:
${users.stream()
.filter(u - fn:contains(u.username, '1'))
.map(u - {'username':u.username, 'first':u.firstName,
'last':u.lastName})
.sorted((u1, u2) - u1.lastName.compareTo(u2.lastName) == 0 ?
u1.firstName.compareTo(u2.firstName) : u1.lastName.compareTo(u2.lastName))
.toList()}
However, that doesn't work and I get this error:
org.apache.el.parser.ParseException: Encountered EOF at line 3, column
88.
Was expecting one of:
. ...
) ...
etc ...
I do not understand the above. Can you provide a simple test case?
Which one of the expressions does not work? Can you remove the others?
So, as mentioned above, the following expression works:
${users.stream()
.filter(u - fn:contains(u.username, '1'))
.sorted((u1, u2) - (x = u1.lastName.compareTo(u2.lastName); x
== 0 ? u1.firstName.compareTo(u2.firstName) : x))
.toList()}
If I now add the map operation to it, I get the EOF error. Nothing else
about the expression changed:
${users.stream()
.filter(u - fn:contains(u.username, '1'))
.sorted((u1, u2) - (x = u1.lastName.compareTo(u2.lastName);
x == 0 ? u1.firstName.compareTo(u2.firstName) : x))
.map(u - {'username':u.username, 'first':u.firstName,
'last':u.lastName})
.toList()}
javax.el.ELException: Failed to parse the expression [${users.stream()
.filter(u - fn:contains(u.username, '1'))
.sorted((u1, u2) - (x = u1.lastName.compareTo(u2.lastName);
x == 0 ? u1.firstName.compareTo(u2.firstName) : x))
.map(u - {'username':u.username, 'first':u.firstName,
'last':u.lastName}]
...
root cause
org.apache.el.parser.ParseException: Encountered EOF at line 6, column
38. Was expecting one of:
. ...
) ...
etc...
Notice that it thinks the expression is ending after the closing } of the
map-literal. Now the example in 2.3.6.4 of the specification alludes to the
fact that my use of the map-literal here is correct. In that example they
use a list-literal instead (.map(p-[p.name, p.unitPrice])). I tried
changing to use a list-literal instead of a map-literal, and that's when I
got the NumberFormatException described earlier. I _believe_ both should be
legal; the specification clearly intends that at least using the
list-literal should be legal. Neither work in Tomcat.
Does this make
Re: EL 3.0 Streams Edge Case - Am I wrong or is Tomcat wrong?
2013/11/18 Nick Williams nicho...@nicholaswilliams.net:
Regarding the list() example,
map(u - [u.username, u.firstName, u.lastName]) creates a
ListObject with 3 elements and you are asking for lastName
property on that list.
No, this is not correct. The lambda expression u - [u.username,
u.firstName, u.lastName] RETURNS a ListObject. But u is a User. I am
creating a ListObject where the first element is the user's username, the
second element is the user's first name, and the third element is the user's
last name. That is completely valid. It is also essentially identical to
Section 2.3.6.4's example p-[p.name, p.unitPrice].
It is no wonder that it results in NumberFormatException. (Whether
there should be other handling for wrong property name here, I do not
know. One should look into what resolvers are being used here).
It should not result in any exception. It should work, because the syntax is
valid, as explained above. :-)
You apply map() which replaces each user with a list [u.username,
u.firstName, u.lastName].
Then you try to apply sorted() to that list. That sorted() fails as it
cannot evaluate u1.lastName.compareTo(u2.lastName).
As I said - provide a simple example.
Section 2.3.6.4 of the specification uses the following example, where a
LIST literal is used as the right-hand side of the mapping lambda
expression:
products.stream().filter(p-p.unitPrice = 10).
.map(p-[p.name, p.unitPrice])
.toList()
I tried to use this exact syntax, as shown in the spec, with my example:
${users.stream()
.filter(u - fn:contains(u.username, '1'))
.map(u - [u.username, u.firstName, u.lastName])
.sorted((u1, u2) - u1.lastName.compareTo(u2.lastName) == 0
? u1.firstName.compareTo(u2.firstName) :
u1.lastName.compareTo(u2.lastName))
.toList()}
And now I get this lovely error:
javax.el.ELException: java.lang.NumberFormatException: For input string:
lastName
javax.el.BeanELResolver.invoke(BeanELResolver.java:185)
org.apache.jasper.el.JasperELResolver.invoke(JasperELResolver.java:147)
org.apache.el.parser.AstValue.getValue(AstValue.java:158)
...
Best regards,
Konstantin Kolinko
Thanks,
Nick
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mySQL connector error
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Phillip,
On 11/17/13, 1:56 PM, Philipp Kraus wrote:
Hello,
I'm new with Tomcat, so I hope for some help. I try to run
www.icescrum.org on my Tomcat 7 on Ubuntu 12.04 with OpenJDK. I
have installed Tomcat with apt-get and also I have installed the
libmysql-java package. On starting Tomcat it reports on a
stacktrace log, that the mySQL connector is not found
(java.lang.ClassNotFoundException: com.mysql.jdbc.Driver). So I
have try to create a symlink in Tomcats lib dir to the jar file and
try to modify the properties of the search paths:
common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,/var/lib/tomcat7/common/classes,/var/lib/tomcat7/common/*.jar
server.loader=/var/lib/tomcat7/server/classes,/var/lib/tomcat7/server/*.jar
shared.loader=/var/lib/tomcat7/shared/classes,/var/lib/tomcat7/shared/*.jar
The JDBC mySQL Jar is stored under /usr/share/java with two
symlinks, I have tried to create a symlink from Tomcats lib dir
(/usr/share/tomcat7/lib) to ../../java/mysql.jar but the mySQL is
also not loaded. The CATALINA_HOME is set to /usr/share/tomcat7 and
the CATALINA_BASE is /var/lib/tomcat7, so imho it should be worked
How can I create a working IceScrum with mySQL and a shared mySQL
connector?
Can you post a copy/paste of the *somplete* output from ps aux from
your Tomcat process? It will show all the paths used.
Also, what does ls -l /usr/share/tomcat7/lib/mysql-*.jar show?
What about ls -l /usr/share/java/mysql*.jar?
Package-managed Java-related packages are usually a complete mess.
You're going to hear a lot of suggestions (including from myself) that
you abandon that mess and install everything yourself. You are likely
to get more updated versions, too, which is nice.
- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.15 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJSilviAAoJEBzwKT+lPKRYFg8QAKa6cFcBofbla/83qHPdquXa
00YDerswDtn3B7Fk9A8+Bjg4/rcVHxb45EChg2apirxd8hish4JLrslcj2FCywG1
QCZOnfV3ubRvFOFjO6Kk1jSjb8n8MZXUUz9G/9Vt/+EsxhTNrn9W5ojLivtqs2vU
L5NoH8G9nGhiwIXvYZRY5/tb0p1paIppIn4+UyJKsP2+fwV0N/YrFPeoGHjk9Yg4
aMy4JRRlY+tdmiS0vDUHw+652CaxJAOszgi3nZi6GZK2dfCRkB+INOTMmMk9ekN2
vyFtmDvYmyT6ugY+FF1WA1z4jXIzpLSQTdjun2VOakBxVX6/VIKtgN7gG0OM1SrC
PuAGqP71i518Wp1+MAN5bGD2ND3Fw46mJ9V9bfUunZyeWP7WNNSJlpykW0LCtWPO
9HVGhRsrKS7rkRq+JCFJ+VTQcknQJaPEblR3KQxsrqsHsIbzumQ3GNsU7L1m5mfj
UF8+xNW8OMjNGUcaK5stmFW6JuPHSir93h9gIs5RXQ3GWz/p3VC2W8F9hXR0AQG9
wti9yYHWNqtkQ7ZLZDSCj8EMv67JIF659dHPG8Lj/7wiqDDDg8j/yOMmQ5j7hXxr
LOSq1qQPzIYbJsKiOEEGjfm9itDkHEiPatsU8Dc6gq8NK0cTOu5qxvPRZgGUfyxI
uwy+iZ/ncEYaERr/4a54
=SJET
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] RE: Baked-in context paths
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Konstantin, On 11/18/13, 9:35 AM, Konstantin Preißer wrote: Hi, -Original Message- From: Konstantin Preißer [mailto:kpreis...@apache.org] Sent: Wednesday, November 13, 2013 6:14 PM To: 'Tomcat Users List' Subject: RE: [OT] RE: Baked-in context paths Hi Jeffrey, -Original Message- From: Jeffrey Janner [mailto:jeffrey.jan...@polydyne.com] Sent: Wednesday, November 13, 2013 5:14 PM To: 'Tomcat Users List' Subject: RE: [OT] RE: Baked-in context paths The thing is, the garbage characters at the end don't exist in the email I see as I type. They are either hidden in the editor or getting added by Outlook (2007) or Exchange (2010). They are added by the list mailer of this mailing list, not by your mail client - it is the the following lines: - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org The problem here is that your client sent the mail content as Base64, and the list mailer adds these lines as a separate Base64 block which (in my opinion) is probably wrong - it produces invalid Base64 data, which causes such junk characters on some mailing clients (and an empty mail on my client). This also happens with some other mails like the one from Buildbot on the Dev list - these are are also sent as Base64 which then show blank for me. Should I report this problem somewhere, e.g. on the Apache Infra JIRA? Note that this problem does not always appear - I think it happens only if (original_message_length_in_bytes % 3 != 0), because then the last 4 characters sequence does not represent 3 bytes, but only 1 or 2, and therefore has some = characters at the end. However, in the other case, the last 4 characters represent full 3 bytes, so another Base64 block can be added without corrupting the data. E.g., if you look at this mail from Buildbot [1], you can see that the bottom contains the To unsubscribe... text added by the mailing list. This mail displayed correctly in my mail client. If you look at the raw message, you can see that the first Base64 block represents 429 bytes, and as 429 % 3 == 0 there isn't a problem. If you look at this mail from Buildbot [2], then the bottom doesn't contain the To unsubscribe lines. In the raw message, you can see the first Base64 block represents 403 bytes (403 % 3 == 1), so it contains == at the end, and the added Base64 block makes the Base64 invalid. This email displayed blank in my client. [1] http://mail-archives.apache.org/mod_mbox/tomcat-dev/201311.mbox/%3C20131118012335.AB0A6C04A4%40aegis.apache.org%3E [2] http://mail-archives.apache.org/mod_mbox/tomcat-dev/201311.mbox/%3C20131118032322.48526C04A4%40aegis.apache.org%3E Interesting observation. I think you should report it to infra@ - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.15 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJSilwZAAoJEBzwKT+lPKRYsP4QAMmYqPQXLx/r979z3xkfTZLL hnYZTW2oYayEBY8Ooip0/Hs8qoJFeKo5AcRU9DrNM/dDa7H+lsUEeP3/6REFhdBr 7xTNJu5mVK0I0A/UK22HVc3UIyTKwRW9viA4n28Dk9Mn5znvXX9tGL0YgfPdfo+5 KhPGll0p5W15cz7TPC4rIxOXk6u5519gSOiJolL4xCEWNLbrwvIjtjbEHxXl06xt CZ9PE4Y7G1EBhyCMGKcfW3AsKoXI23omv8w4lEfXq2Y3VCoLr/IRZSIX7RpONC5i Rrf+TabcIMwjk+R0E4FyqBXMVYsf/bI57ZPf/hYq5ioywiMW4QkWnrjoCPlx5rme fSQ/U7jA8lFZ8D6sCeopqHZTTFP2b9oerT4cKTLrRU4hXYzGKv6BFKSkl7haLh7s qBUeZGjw3o3t+ylkwFpn4rwPNqooo+E6I2IkNWhglg+peVFac4lfKpxCG4i1snct vy0efGrTy8Tqy5thh5tNnqzRzIxzDzoWQNHWu5MYEMJaWY9i8zA/786PhQsrOlDE c1SE37uKhVIoL97tFjmtK7+M3jJuJCFCwFJgq7QehmVCUm9wg/UvY+zYFcMPTI2C jEi101NYSgT4ECw/Ix7K8AAxpd813xIBLfOTIWVJISMHBysUxYWTUA/a0f3D7QFb rnPSDU0H7LY2jZVLiL0d =sQjX -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: EL 3.0 Streams Edge Case - Am I wrong or is Tomcat wrong?
On Nov 18, 2013, at 12:21 PM, Konstantin Kolinko wrote:
2013/11/18 Nick Williams nicho...@nicholaswilliams.net:
Regarding the list() example,
map(u - [u.username, u.firstName, u.lastName]) creates a
ListObject with 3 elements and you are asking for lastName
property on that list.
No, this is not correct. The lambda expression u - [u.username,
u.firstName, u.lastName] RETURNS a ListObject. But u is a User. I am
creating a ListObject where the first element is the user's username, the
second element is the user's first name, and the third element is the user's
last name. That is completely valid. It is also essentially identical to
Section 2.3.6.4's example p-[p.name, p.unitPrice].
It is no wonder that it results in NumberFormatException. (Whether
there should be other handling for wrong property name here, I do not
know. One should look into what resolvers are being used here).
It should not result in any exception. It should work, because the syntax is
valid, as explained above. :-)
You apply map() which replaces each user with a list [u.username,
u.firstName, u.lastName].
Then you try to apply sorted() to that list. That sorted() fails as it
cannot evaluate u1.lastName.compareTo(u2.lastName).
As I said - provide a simple example.
Okay, I see where the confusion is. Indeed, when I moved map to after
sorted the list-literal started working. So it's only the map-literal that is
failing. My apologies; I'll revise the bug.
Section 2.3.6.4 of the specification uses the following example, where a
LIST literal is used as the right-hand side of the mapping lambda
expression:
products.stream().filter(p-p.unitPrice = 10).
.map(p-[p.name, p.unitPrice])
.toList()
I tried to use this exact syntax, as shown in the spec, with my example:
${users.stream()
.filter(u - fn:contains(u.username, '1'))
.map(u - [u.username, u.firstName, u.lastName])
.sorted((u1, u2) - u1.lastName.compareTo(u2.lastName) == 0
? u1.firstName.compareTo(u2.firstName) :
u1.lastName.compareTo(u2.lastName))
.toList()}
And now I get this lovely error:
javax.el.ELException: java.lang.NumberFormatException: For input string:
lastName
javax.el.BeanELResolver.invoke(BeanELResolver.java:185)
org.apache.jasper.el.JasperELResolver.invoke(JasperELResolver.java:147)
org.apache.el.parser.AstValue.getValue(AstValue.java:158)
...
Best regards,
Konstantin Kolinko
Thanks,
Nick
Nick
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Intermittent Digest Authentication User Lockout
I am running into an intermittent problem with Digest-Authentication. This is with tomcat 7.0.39 The issue appears to be that clients will occasionally get locked out for 5 minutes. The problem appears to happen with there is a combination of good password and then bad password, or the other way round. We have also seen the problem happen when our load balancer is not sticky. My understanding is that digest-auth really should not work if the load-balancer is not sticky since there need to be information sent from the server to the client in order to make the authentication. We have since made our load balancer sticky, hoping that this would resolve the issue. Actually, I should make a clarification here. It’s not “clients” that are getting locked out. It is “users”. Once a user gets into a bad state the account gets locked out until a 5 minute period goes by. Looking at the tomcat source code, I see DigestAuthenticator.java line 147: protected long nonceValidity = 5 * 60 * 1000; Sorry if this sounds confused - I’m confused. I can say this. We’re seeing users get locked out for 5 minutes at a time. Having the load balancer not being sticky would definitely cause the problem, but after making them sticky, we still see the problem with at least one client program. The client programs are mostly non-webbrowser based. Thanks - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Intermittent Digest Authentication User Lockout
I’ll answer my own question. Short story: Everything was working as designed. The problem, as we now understand it, was in fact due to the LockOutRealm http://tomcat.apache.org/tomcat-7.0-doc/config/realm.html#LockOut_Realm_-_org.apache.catalina.realm.LockOutRealm The load balancer not being sticky was causing bad authentications to be sent to the backend tomcats. Eventually, enough bad requests were being sent to the backends that they would each lock out. Sometimes, by luck authentications would work due to the Load Balancer sending requests back to the same backend as the originating response. That all makes sense. Has nothing to do with the nonceValidity timeout. Also, for some reason, the client software would try 20 times before giving up on a bad password. Not sure why it did that, but the effect was to lockout the back end tomcat. We need to modify the client to prevent this from happening. Thanks, -Bruce On Nov 18, 2013, at 10:29 AM, Bruce Weertman br...@iris.washington.edu wrote: I am running into an intermittent problem with Digest-Authentication. This is with tomcat 7.0.39 The issue appears to be that clients will occasionally get locked out for 5 minutes. The problem appears to happen with there is a combination of good password and then bad password, or the other way round. We have also seen the problem happen when our load balancer is not sticky. My understanding is that digest-auth really should not work if the load-balancer is not sticky since there need to be information sent from the server to the client in order to make the authentication. We have since made our load balancer sticky, hoping that this would resolve the issue. Actually, I should make a clarification here. It’s not “clients” that are getting locked out. It is “users”. Once a user gets into a bad state the account gets locked out until a 5 minute period goes by. Looking at the tomcat source code, I see DigestAuthenticator.java line 147: protected long nonceValidity = 5 * 60 * 1000; Sorry if this sounds confused - I’m confused. I can say this. We’re seeing users get locked out for 5 minutes at a time. Having the load balancer not being sticky would definitely cause the problem, but after making them sticky, we still see the problem with at least one client program. The client programs are mostly non-webbrowser based. Thanks - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat connection pool bleeding under heavy load
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Marko, On 11/18/13, 3:48 AM, marko lugarič wrote: We have received an issue from our customer that after some time in production they get empty connection pool. After some time we managed to reproduce their problem in our test environment with smaller number of pool connections. So generally we have a problem that if connection pool is under heavy load (meaning most of the time all the connections are taken - active) it starts to loose database connections (we call it bleed): they are lost from the perspective of Tomcat but they are very much alive when we check then on the database side. We are using Apache Tomcat 7.0.47 running on java 1.6. Database is SQL Server 2008R2 and we are using Hibernate as persistence provider. Our configuration is defined in server.xml in the GlobalNamingResources section (referenced from context.xml): Resource name=jdbc/ChiDS auth=Container type=javax.sql.DataSource factory=org.apache.tomcat.jdbc.pool.DataSourceFactory driverClassName=com.microsoft.sqlserver.jdbc.SQLServerDriver url=jdbc:sqlserver://*;databaseName=chiTestDB username=* password=* initialSize=2 maxActive=5 maxIdle=4 minIdle=2 maxWait=15000 timeBetweenEvictionRunsMillis=3 testOnBorrow=true testOnReturn=false testOnIdle=false validationQuery=SELECT 1 validationInterval=3 removeAbandoned=true removeAbandonedTimeout=60 useEquals=false/ I'm just curious: why are you using useEquals=false here? I've wondered what the use-case is for this configuration attribute. I will explain in detail what happens. For better understanding I am also providing graph of idle and active connections from Java VisualVM MBeans viewer: http://img11.imageshack.us/img11/6888/fm4j.jpg So we start the server and connect clients to the server that generate small database load. After that we order half of our clients to start generating operations that result in medium database load (I think around 300 transactions per second). We can see from the graph that connection pool is behaving normally (sometimes all 5 connections are taken, sometimes not - if we would run test like this all day everything would be OK). So after 7 minutes we increase the load by 100% (other half of clients start to do operations). The load is bigger than 5 pool connections can handle: pretty much all the time all 5 connections are active so we can soon expect that server application logic will start to receive exceptions that jdbc connection is not available (this should happen if application logic does not receive pool connection in 15 seconds). Indeed this happens and the following exception is thrown: Caused by: org.hibernate.exception.GenericJDBCException: Cannot open connection at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:140) at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:128) at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66) at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:52) at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:449) at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167) at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:142) at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:85) at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1463) at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:60) ... 18 more Caused by: java.sql.SQLException: Pool wait interrupted. at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:655) at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) at org.hibernate.connection.DatasourceConnectionProvider.getConnection(DatasourceConnectionProvider.java:92) at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446) ... 23 more Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1024) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253) at org.apache.tomcat.jdbc.pool.FairBlockingQueue.poll(FairBlockingQueue.java:151) at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:650) Good. It seems like that is the appropriate result given the situation you describe. But here something goes wrong because active connections count drops by 1 and never goes back even though database load is still the same. Judging from the graph at this
Re: Tomcat connection pool bleeding under heavy load
In PooledConnection#borrow(int,String,String) when handling InterruptedException, the code there does: Thread.interrupted();//clear the flag, and bail out but if the comment is correct, then the code is wrong (should be Thread.isInterrupted()). However, the entire line is probably not necessary - using the java.util.concurrent contracts, a threads' interruption status is usually cleared when InterruptedException is thrown. The 'leakiness' of the pool is actually in org.apache.tomcat.jdbc.pool.FairBlockingQueue#poll() If the line (!c.await(timeout, unit)) exits via an InterruptedException, the the 'exchangecountdownlatch' is never removed from the waitlist ... connections will be 'exchanged' with threads which will never rendezvous. But why is InterruptedException being thrown anyway? That'll be because the Tomcat worker thread trying to acquire a connection gets issued aWorkerThread.interrupt() for taking too long. So adjusting some timeout settings (to make Tomcat more patient than the connection pool) should give a workaround. DBCP had a better grasp of what was going on when reporting this. cheers, David David Bullock Machaira Enterprises Pty Ltd PO Box 31 Canowindra NSW 2804 02 6344 1100 http://machaira.com.au/ On 18 November 2013 19:48, marko lugarič marko.luga...@gmail.com wrote: Hello! We have received an issue from our customer that after some time in production they get empty connection pool. After some time we managed to reproduce their problem in our test environment with smaller number of pool connections. So generally we have a problem that if connection pool is under heavy load (meaning most of the time all the connections are taken - active) it starts to loose database connections (we call it bleed): they are lost from the perspective of Tomcat but they are very much alive when we check then on the database side. We are using Apache Tomcat 7.0.47 running on java 1.6. Database is SQL Server 2008R2 and we are using Hibernate as persistence provider. Our configuration is defined in server.xml in the GlobalNamingResources section (referenced from context.xml): Resource name=jdbc/ChiDS auth=Container type=javax.sql.DataSource factory=org.apache.tomcat.jdbc.pool.DataSourceFactory driverClassName=com.microsoft.sqlserver.jdbc.SQLServerDriver url=jdbc:sqlserver://*;databaseName=chiTestDB username=* password=* initialSize=2 maxActive=5 maxIdle=4 minIdle=2 maxWait=15000 timeBetweenEvictionRunsMillis=3 testOnBorrow=true testOnReturn=false testOnIdle=false validationQuery=SELECT 1 validationInterval=3 removeAbandoned=true removeAbandonedTimeout=60 useEquals=false/ I will explain in detail what happens. For better understanding I am also providing graph of idle and active connections from Java VisualVM MBeans viewer: http://img11.imageshack.us/img11/6888/fm4j.jpg So we start the server and connect clients to the server that generate small database load. After that we order half of our clients to start generating operations that result in medium database load (I think around 300 transactions per second). We can see from the graph that connection pool is behaving normally (sometimes all 5 connections are taken, sometimes not - if we would run test like this all day everything would be OK). So after 7 minutes we increase the load by 100% (other half of clients start to do operations). The load is bigger than 5 pool connections can handle: pretty much all the time all 5 connections are active so we can soon expect that server application logic will start to receive exceptions that jdbc connection is not available (this should happen if application logic does not receive pool connection in 15 seconds). Indeed this happens and the following exception is thrown: aused by: org.hibernate.exception.GenericJDBCException: Cannot open connection at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:140) at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:128) at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66) at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:52) at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:449) at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167) at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:142) at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:85) at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1463) at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:60) ... 18 more Caused by: java.sql.SQLException: Pool wait interrupted. at
Re: setting the text or binary buffer size for websockets
Upgraded my environment to 8RC5 and this feature works for me. Don't know how much help this is, but here's my deployment descriptor: ?xml version=1.0 encoding=UTF-8? web-app xmlns=http://java.sun.com/xml/ns/javaee; xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation=http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd; version=3.0 display-nameFERMI Framework Test Application/display-name context-param param-nameorg.apache.tomcat.websocket.textBufferSize/param-name param-value10240/param-value /context-param /web-app -Igor. On Sun, Nov 17, 2013 at 7:15 AM, Johan Compagner jcompag...@servoy.comwrote: Exactly which version of Tomcat 7 are you using? currently testing it on 8 RC5 I can test on 7, but i guess thats the +/- the same code? Latest 8 RC is fine but then why are you looking at the Tomcat 7 docs? first that i found, and also this http://tomcat.apache.org/tomcat-8.0-doc/web-socket-howto.html is the same i will look at it a bit more then, but for my current tests it didn't have any effect as far as i could see.
RE: Tomcat connection pool bleeding under heavy load
From: David Bullock [mailto:david.bull...@machaira.com.au] Subject: Re: Tomcat connection pool bleeding under heavy load In PooledConnection#borrow(int,String,String) when handling InterruptedException, the code there does: Thread.interrupted();//clear the flag, and bail out but if the comment is correct, then the code is wrong (should be Thread.isInterrupted()). No, Thread.interrupted() is a static method that checks if the _current_ thread has been interrupted. Calling Thread.isInterrupted() would be illegal, since that is an instance method, and is used primarily to determine if some other thread has been interrupted. Also, the isInterrupted() method does not clear the interrupted flag for the target thread, whereas Thread.interrupted() does clear it for the current thread. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
