This issue occurred again tonight. Nothing of note in the logs again, with
plenty of jvm memory + disk space left. It just will start redirecting
users to the login page as if a failed login occurred, but without
displaying any error messages.
Typing in an incorrect password is caught and the error is displayed as
expected with an authentication failed audit. Typing in a correct password
will log a successful authentication, but not generate/validate any service
tickets in the audit.
This is not related to any LDAP connectivity (this was my reason for
updating in the first place) as I also tested on local password storage
during this outage as well. Swapping ticketing systems does not seem to
have helped.
Any suggestions of what else I should try? Would getting out of tomcat and
running it under a separate container help?
On Friday, September 23, 2016 at 3:45:49 PM UTC-7, Jeffrey Wong wrote:
>
> On suggestions from another user with similar issues using JPA, I have
> changed and deployed CAS using a hazelcast ticketing database.
>
> I'll let you know if I have any success with this configuration.
>
> On Thursday, September 22, 2016 at 11:55:16 AM UTC-7, Jeffrey Wong wrote:
>>
>> Hi again,
>>
>> It's been about a month and have regularly screened the JVM memory - it
>> looks fine since the memory bumps, running CAS v4.2.4.
>>
>> However, the server fell over again (using the JPA ticket registry), with
>> the same behavior: upon entering correct credentials, a user is redirected
>> back to the login page rather than logging in. I'm not sure where to go
>> from here to ensure a more reliable service, and would like to hear your
>> input.
>>
>> While digging, I've found three types of exceptions in the logs:
>>
>> 1: deadlocks
>> SEVERE: Servlet.service() for servlet [cas] in context with path [/cas]
>> threw exception [Request processing failed; nested exception is
>> org.springframework.webflow.execution.ActionExecutionException: Exception
>> thrown executing
>> org.jasig.cas.web.flow.GenerateServiceTicketAction@4805bd13 in state
>> 'generateServiceTicket' of flow 'login' -- action execution attributes were
>> 'map[[empty]]'] with root cause
>> com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException:
>> Deadlock found when trying to get lock; try restarting transaction
>> [265/9592]
>> at
>> sun.reflect.GeneratedConstructorAccessor127.newInstance(Unknown Source)
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
>> at com.mysql.jdbc.Util.getInstance(Util.java:387)
>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:946)
>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3878)
>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3814)
>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2478)
>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2625)
>> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2551)
>> at
>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861)
>> at
>> com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2073)
>> at
>> com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1751)
>> at
>> com.mysql.jdbc.PreparedStatement.executeBatchInternal(PreparedStatement.java:1257)
>> at
>> com.mysql.jdbc.StatementImpl.executeBatch(StatementImpl.java:959)
>> at
>> com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeBatch(NewProxyPreparedStatement.java:2544)
>>
>> 2: badly formatted keys
>> SEVERE: Servlet.service() for servlet [cas] in context with path [/cas]
>> threw exception [Request processing failed; nested exception is
>> org.springframework.webflow.execution.repository.BadlyFormattedFlowExecution
>> KeyException: Badly formatted flow execution key '', the expected format
>> is '<uuid>_<base64-encoded-flow-state>'] with root cause
>> org.springframework.webflow.execution.repository.BadlyFormattedFlowExecutionKeyException:
>>
>> Badly formatted flow execution key '', the expected format is
>> '<uuid>_<base64-encoded-flow-state>'
>> at
>> org.jasig.spring.webflow.plugin.ClientFlowExecutionKey.parse(ClientFlowExecutionKey.java:102)
>> at
>> org.jasig.spring.webflow.plugin.ClientFlowExecutionRepository.parseFlowExecutionKey(ClientFlowExecutionRepository.java:74)
>> at
>> org.springframework.webflow.executor.FlowExecutorImpl.resumeExecution(FlowExecutorImpl.java:164)
>> at
>> org.springframework.webflow.mvc.servlet.FlowHandlerAdapter.handle(FlowHandlerAdapter.java:228)
>> at
>> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
>> at
>> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
>> at
>> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
>> at
>> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
>> at
>> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
>> at
>> org.apereo.cas.security.ResponseHeadersEnforcementFilter.doFilter(ResponseHeadersEnforcementFilter.java:238)
>> at
>> org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
>>
>> 3: constraint violations
>> Sep 22, 2016 9:48:20 AM org.apache.catalina.core.StandardWrapperValve
>> invoke
>> SEVERE: Servlet.service() for servlet [cas] in context with path [/cas]
>> threw exception [Request processing failed; nested exception is
>> org.springframework.webflow.execution.ActionExecutionException: Exception th
>> rown executing
>> org.jasig.cas.web.flow.GenerateServiceTicketAction@4805bd13 in state
>> 'generateServiceTicket' of flow 'login' -- action execution attributes were
>> 'map[[empty]]'] with root cause
>> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
>> Cannot add or update a child row: a foreign key constraint fails
>> (`cas`.`SERVICETICKET`, CONSTRAINT `FK(TGTID)` FOREIG
>> N KEY (`ticketGrantingTicket_ID`) REFERENCES `TICKETGRANTINGTICKET`
>> (`ID`))
>> at
>> sun.reflect.GeneratedConstructorAccessor131.newInstance(Unknown Source)
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
>> at com.mysql.jdbc.Util.getInstance(Util.java:387)
>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:932)
>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3878)
>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3814)
>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2478)
>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2625)
>> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2551)
>> at
>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861)
>> at
>> com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2073)
>> at
>> com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1751)
>> at
>> com.mysql.jdbc.PreparedStatement.executeBatchInternal(PreparedStatement.java:1257)
>> at
>> com.mysql.jdbc.StatementImpl.executeBatch(StatementImpl.java:959)
>> at
>> com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeBatch(NewProxyPreparedStatement.java:2544)
>> at
>> org.hibernate.engine.jdbc.batch.internal.BatchingBatch.performExecution(BatchingBatch.java:110)
>> at
>> org.hibernate.engine.jdbc.batch.internal.BatchingBatch.doExecuteBatch(BatchingBatch.java:97)
>> at
>> org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl.execute(AbstractBatchImpl.java:147)
>> at
>> org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.executeBatch(JdbcCoordinatorImpl.java:214)
>> at
>> org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:470)
>>
>> I'm hoping something is just mis-configured in my config, and wondering
>> if the exceptions here give any clue as to what they might be. Exceptions 1
>> and 3 seem like something is strange with my JPA registry. I'm using C3P0
>> as my jdbc connection pool ('com.mchange:c3p0:0.9.5.2'), and
>> mysql-connector ('mysql:mysql-connector-java:5.1.38').
>>
>> Thanks,
>> Jeff
>>
>> On Thursday, August 25, 2016 at 10:56:42 PM UTC-7, Misagh Moayyed wrote:
>>>
>>>
>>> Ah, JVM stats. Thanks for that tip: I really wasn't looking at my memory
>>> closely enough, and I still had a -Xmx128m set (tomcat7 default in ubuntu).
>>> I bumped it to 1g (actually this time, confirmed via /status this time) so
>>> I'm hoping this helps resolve.
>>>
>>> Ah! That should do it. I am surprised you were able to last a few months
>>> with that setting. My preference is somewhere between 2g and 4g.
>>>
>>>
>>> /status wasn't working for localhost by default, but this was due to
>>> IPv6 on my end. Would you be open to changing the
>>> default cas.securityContext.adminpages.ip=(127\.0\.0\.1|0:0:0:0:0:0:0:1) to
>>> support IPv6 localhost?
>>>
>>> No problem. File an issue please.
>>>
>>>
>>> I'm planning on throwing on a lot more memory monitoring near future.
>>> I'm assuming this would be a very likely root cause for these sorts off
>>> issues - sorry I didn't catch that earlier.
>>>
>>> Hoping this is the ticket (hurr hurr) - I'll let you know if anything
>>> else interesting comes up, but I really appreciate your support!
>>>
>>> No worries. All I did was to echo back what you already knew.
>>>
>>>
--
- CAS gitter chatroom: https://gitter.im/apereo/cas
- CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html
- CAS documentation website: https://apereo.github.io/cas
- CAS project website: https://github.com/apereo/cas
---
You received this message because you are subscribed to the Google Groups "CAS
Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/a/apereo.org/d/msgid/cas-user/52a1362d-e821-476a-a10d-83d317d79ecd%40apereo.org.
