Thanks Dale for the detailed responses. I don't disagree with your
suggestion as we making plans to migrate our platform to a React
architecture, but our current application is robust and eliminating all
ExecuteAndWait actions would be a monumental task - one that doesn't
make sense in this scenario. That said, I should have some time to
investigate this issue further over the next few days and will report
back.
------ Original Message ------
From "Dale Newfield" <d...@newfield.org>
To "Dale Newfield" <d...@newfield.org>
Cc "Struts Users Mailing List" <user@struts.apache.org>
Date 9/24/2023 12:32:08 PM
Subject Re: Issue with ExecuteAndWait when using Spring Session (Redis)
Depending on your deployment architecture, this exec and wait may no longer be the right
strategy. Yes, by managing the executor pool you can bound how many non-http-serving
threads (jobs) are executing concurrently on your web server's host, but it is all
premised on using your own hardware. If running "in the cloud", beware that
this doesn't lead you to overprovison (overspend on) always-on infrastructure...and thus
limit your ability to scale dynamically for variable load requirements.
These days I'd capture the data describing the pending work items somewhere
(db/s3/kinesis/lots of other possibilities) and ensure some resilient scheme
translates that into a broadcast message for each. Probably subscribe a queue
to that broadcast and use that to trigger the work in parallel. If queue
listeners are executing on ec2/ecs/eks, perhaps scale as that queue size grows.
Or most likely execute the steps with serverless lambdas, potentially with
massive concurrency and low cost. Today there are lots of alternatives,
depending on the workload: step functions, etc. Usually you can find a way to
ensure at-least-once execution of each task, so this other side of the coin is
to make the work idempotent so duplicative execution cannot introduce issues.
Perhaps a DLQ for problematic jobs. Make sure something monitors that DLQ.
When exec-and-wait was originally implemented we lived in an only intermittently
connected world and owned all the hardware. I used it to sync up work done by a
disconnected instance with "the real, system" upon reconnection. (For me this
meant photographers at parties, building engagement after the party by virtualizing the
real world connections. It displayed photos live during the event, and allowed
assistants trailing behind the photographers to tag event-goers in those photos (by
scanning barcodes on their wristbands/entering account ids/collecting email addresses on
the spot to create accounts). This allowed us to email event photos to the subjects in
them by the next morning, and gave participants an easy way to connect with each other
afterwards just by having a photographer take their picture together. At the time,
facial recognition was just becoming possible and was also leveraged. But back to the
technical side this meant I ran either ran a full instance (db/struts web server/etc) at
the front door managing barcode assignments to match prepaid ticket holders, and
supporting the whole event via wifi (I'd plug in repeaters as necessary thru the venue
and/or stick battery-powered ones in the back pockets of security guards) or that I ran a
full instance on the device carried around by the photographer's assistant.
This was all before tablets/smartphones/social networks/etc. Today we live in a
very different world--always connected, cloud based, etc. Some
tools/techniques still apply. Many now have much better solutions (more
reliable/resilient/scalable/simpler; less expensive to
build/launch/operate/etc.). YMMV.
-Dale
On Sep 23, 2023, at 9:52 PM, Dale Newfield <d...@newfield.org> wrote:
If it runs again then look at the logic where it's gotta decide whether this is the
initial call or if the job is already running and this p my try just a status check. Is
it looking at the request to figure that out? If so, and the expected info isn't found,
walk the data path to figure out where it is getting lost (sounds like the session filter
might be a good place to start). Is it looking in a database/shared memory/file system?
Perhaps these are not in sync? Perhaps the long running job is doing everything in a
single transaction, not committed until the very end, and the "I'm running!"
flag set within the transaction is not visible unless the query is READ_UNCOMMITTED?
This run/not decision is critical, so it should be determined atomically. Perhaps one
transaction for the decision (which either concludes "report status" or ensures
its record is visible to any other query) and others for each separable unit of work
within the long running job?
Oh, yeah, you said Redis. I've somehow managed to avoid that, so I don't know what persistent
data check/update can be done atomically, but that's where you're hanging your hat. Maybe the
response to your "is anyone else already running this job?" query is being cached, and so
the first "no" response is being returned every time?
So many places this decision could go wrong in a distributed platform. Every possible
race condition will eventually happen. Can you confidently say that two concurrent
requests on distinct hosts will always result in a single "winner"?
(Then later on there's a "liveness" detector you'll need to figure out in case
the winner fails. Perhaps it should periodically update that record to show when it was
last known to be running? (How far into the job it is right then would also be great to
snapshot.) if that time stamp is older than X, is it safe to assume it failed and to
elect another leader to take its place? Make sure time zone differences on different
hosts can't impact the timeout checks. Is it safe, even in the face of a network
disconnection, so eventually the evicted leader can reappear? Would one of them detect
the issue and kill itself?)
-Dale
On Sep 23, 2023, at 4:23 PM, Burton Rhodes <burtonrho...@gmail.com> wrote:
I am attempting to implement a centralized session store to an existing application
using Spring Session (Redis) and Struts. I have everything working on a basic level, but
I am running into an issue when a Struts Action uses ExecuteAndWait. The refresh
attempts don't seem to pick up the running action using the provided Struts Token (at
least I think this is the case). So on each refresh, the action seems to run again as if
it was the first time duplicating database inserts, etc. It becomes an infinite process,
and Struts never returns the final "SUCCESS" result JSP page to the browser.
It's worth noting that if I disable the Spring Session filter, everything works fine.
Anyone have a clue as to what might be going on or how I might begin to troubleshoot?
I've included a thread dump so you can see the filters involved. The
"SessionRepositoryFilter" is where the HttpSession is wrapped by Spring (Redis)
and is before any Struts filters. I've also included a basic action definition that is
causing the issue (although this is happening on all actions that use the ExecuteAndWait
logic). Any help is appreciated.
** Example Action Definition **
<action name="ContactBatchCategories_update"
class="com.afs.web.struts.action.contact.ContactBatchCategoriesAction" method="update">
<interceptor-ref name="myDefaultStack"/>
<interceptor-ref name="openSessionExecuteAndWaitInterceptor" >
<param name="delay">0</param>
</interceptor-ref>
<result
name="input">/struts/search/actions/contactBatchCategories_modal.jsp</result>
<result name="wait">/struts/common/progressMonitorWait_modal.jsp</result>
<result>/struts/common/progressMonitorSuccess_modal.jsp</result>
</action>
** Thread Dump **
java.lang.Thread.State: RUNNABLE
at
com.afs.web.struts.action.contact.ContactBatchCategoriesAction.prepare(ContactBatchCategoriesAction.java:62)
at
com.opensymphony.xwork2.interceptor.PrepareInterceptor.doIntercept(PrepareInterceptor.java:171)
at
com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:99)
at
com.opensymphony.xwork2.DefaultActionInvocation.executeConditional(DefaultActionInvocation.java:299)
at
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:253)
at
org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:154)
at
com.opensymphony.xwork2.DefaultActionInvocation.executeConditional(DefaultActionInvocation.java:299)
at
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:253)
at
com.afs.web.config.struts.StrutsAccountInterceptor.intercept(StrutsAccountInterceptor.java:67)
at
com.opensymphony.xwork2.DefaultActionInvocation.executeConditional(DefaultActionInvocation.java:299)
at
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:253)
at
com.afs.web.config.struts.StrutsSessionInterceptor.intercept(StrutsSessionInterceptor.java:44)
at
com.opensymphony.xwork2.DefaultActionInvocation.executeConditional(DefaultActionInvocation.java:299)
at
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:253)
at
com.afs.web.common.struts.interceptor.ExceptionInterceptor.intercept(ExceptionInterceptor.java:33)
at
com.opensymphony.xwork2.DefaultActionInvocation.executeConditional(DefaultActionInvocation.java:299)
at
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:253)
at
org.apache.struts2.factory.StrutsActionProxy.execute(StrutsActionProxy.java:48)
at
org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:651)
at
org.apache.struts2.dispatcher.ExecuteOperations.executeAction(ExecuteOperations.java:79)
at
org.apache.struts2.dispatcher.filter.StrutsPrepareAndExecuteFilter.handleRequest(StrutsPrepareAndExecuteFilter.java:157)
at
org.apache.struts2.dispatcher.filter.StrutsPrepareAndExecuteFilter.tryHandleRequest(StrutsPrepareAndExecuteFilter.java:140)
at
org.apache.struts2.dispatcher.filter.StrutsPrepareAndExecuteFilter.doFilter(StrutsPrepareAndExecuteFilter.java:128)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at
com.afs.web.config.filter.MyAppSessionFilter.doFilter(MyAppSessionFilter.java:85)
at
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
at
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:352)
at
org.springframework.security.web.access.intercept.AuthorizationFilter.doFilter(AuthorizationFilter.java:100)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
at
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
at
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
at
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:131)
at
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:85)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
at
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
at
org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:110)
at
org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
at
org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:164)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
at
org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
at
org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:151)
at
org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:129)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
at
org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227)
at
org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
at
org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107)
at
org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
at
org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
at
org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
at
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117)
at
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
at
org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
at
org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
at
org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:225)
at
org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:190)
at
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
at
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at
org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter.doFilterInternal(OpenEntityManagerInViewFilter.java:186)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at
org.eclipse.jetty.websocket.servlet.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:170)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at
org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:142)
at
org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82)
at
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
at
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:51)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at org.eclipse.jetty.server.Server.handle(Server.java:563)
at
org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)
at
org.eclipse.jetty.server.HttpChannel$$Lambda$1353.1731450897.dispatch(Unknown
Source:-1)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
at
org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416)
at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385)
at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272)
at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140)
at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy$$Lambda$1338.510268842.run(Unknown
Source:-1)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
at java.lang.Thread.run(Thread.java:834)
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@struts.apache.org
For additional commands, e-mail: user-h...@struts.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@struts.apache.org
For additional commands, e-mail: user-h...@struts.apache.org
