Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded parms
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mark, On 12/12/19 04:38, Mark Thomas wrote: > On 11/12/2019 19:17, Christopher Schultz wrote: >> On 12/11/19 13:01, Mark Thomas wrote: > > > >>> Setting RECYCLE_FACADES=true does exactly that. Continued used >>> by the app triggers an NPE since the underlying >>> request/response is no longer there. Hence my question. >> >> RequestFacade currently has a number of methods that perform >> null-checks on the "request" member, but not all methods do this. >> Exampl e: >> >> @Override public ServletContext getServletContext() { if (request >> == null) { throw new IllegalStateException( >> sm.getString("requestFacade.nullRequest")); } >> >> return request.getServletContext(); } >> >> Negative example: >> >> @Override public AsyncContext startAsync() throws >> IllegalStateException { return request.startAsync(); } >> >> Should these be aligned? > > Probably. It will only change the exception from an NPE to ISE but > I guess we should be consistent here. > >> It seems to me that if recycleFacades="false" then there really >> isn't any reason to wrap the request (response, etc.) in a facade >> object at all, is there? > > It makes it harder for a malicious app to get at the internal > object. That's what I thought. So, if you trust the webapp and there are no held-reference bugs, then the facades are completely superfluous. Right now, the only option is either to re-use facades (which is potentially dangerous if the webapp has held-reference bugs) or recycle them (which is wasteful, GC-wise). There is no option to NOT use them. I'm just wondering if there is an opportunity for simpler execution in a "known trusted" configuration -- which is very common. - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3yXBAACgkQHPApP6U8 pFj9HhAAg3UuJ93OPeDggzdTXHj1uafJUVttma50XsQqHGw9IzXjKp0y68j1c/ro Ii2Rm+/HnGyQG4tMk0ILTkkI6b9235Y8ywk7tTCVeiZppIgcnGH29MkARKakjXyW Q95OAOKoUQ2Bo0z7Hl8jJCi/Ri+VAyV6O98q4nVKh2sx0a99wE+Q2GNBxqH2621r AAGg9Bm9/GOQYOx04Lr6nEwLyWQJy8MyD9rYX5lr0wmh16AFj4l4Tey8oEFEsNuW AUMflBgxuQV5BIICQPwSmHvZ4F0Poo4qxtSbO8N5vryecIxk2xxPvNMuK5GnQ87N PepT0i9Pi/UAqK1P7MH6djc9Kz2ps2KyGU9kbYri0EqhST0lefTiBBXPUwxyv9PM eDOvwAnk80Fp3Jt/KG9ygVuujmmm4qLWFk2CcauZhB08XQSUGxmIqNgTNLwFut0O Kj5JhzJ5Patr+0nohywwDLQ1Cr3DYqvbz0hCjmWIAM0OIVMbIX4aGyQpetyU/imL b5RNLcWeMqt5uM+gtQ7aQbBhwbuZpMVWXqPV4AREpPI0ek7HJlgNPZlTHVr6KrVo i9lAZfYvtbkLDNs2JJitYDhv4TqbvH3X3jh9lXmgzuUirrmq1R+9tP92N/MUepHE rlwyx4DESz5CiutWZksjzJHRK4aqLmUnFB0vZ8pMfyeLXXzUcfc= =DZCT -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded parms
On 11/12/2019 19:17, Christopher Schultz wrote: > On 12/11/19 13:01, Mark Thomas wrote: >> Setting RECYCLE_FACADES=true does exactly that. Continued used by >> the app triggers an NPE since the underlying request/response is no >> longer there. Hence my question. > > RequestFacade currently has a number of methods that perform > null-checks on the "request" member, but not all methods do this. Exampl > e: > > @Override > public ServletContext getServletContext() { > if (request == null) { > throw new IllegalStateException( > sm.getString("requestFacade.nullRequest")); > } > > return request.getServletContext(); > } > > Negative example: > > @Override > public AsyncContext startAsync() throws IllegalStateException { > return request.startAsync(); > } > > Should these be aligned? Probably. It will only change the exception from an NPE to ISE but I guess we should be consistent here. > It seems to me that if recycleFacades="false" then there really isn't > any reason to wrap the request (response, etc.) in a facade object at > all, is there? It makes it harder for a malicious app to get at the internal object. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded parms
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mark, On 12/11/19 13:01, Mark Thomas wrote: > On 11/12/2019 17:43, Christopher Schultz wrote: > > > >>> I'm not sure why we need this over and above the RequestFacade >>> object. >> >> The RequestFacade is intended to protect the container from the >> application, right? > > It can cut both ways. > >> And they are (usually) re-used. If a RequestFacade object is >> retained by the application and used later, a later >> client-request may be confused with the one the application was >> expectin g> In the solution I presented, the facades are one-time >> use and will stay dead once they are killed. The application >> can't make a mistake and read a later client-request or write to >> a later client-response. > > Setting RECYCLE_FACADES=true does exactly that. Continued used by > the app triggers an NPE since the underlying request/response is no > longer there. Hence my question. RequestFacade currently has a number of methods that perform null-checks on the "request" member, but not all methods do this. Exampl e: @Override public ServletContext getServletContext() { if (request == null) { throw new IllegalStateException( sm.getString("requestFacade.nullRequest")); } return request.getServletContext(); } Negative example: @Override public AsyncContext startAsync() throws IllegalStateException { return request.startAsync(); } Should these be aligned? It seems to me that if recycleFacades="false" then there really isn't any reason to wrap the request (response, etc.) in a facade object at all, is there? Is there any opportunity to further reduce memory footprint by just NOT creating these objects at all? - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3xQK4ACgkQHPApP6U8 pFjeYg/+MBc+iKgAtAmlf6D6MzgODhoGmGwibpPxsyssQ4jcmLPGUtwF/SVR5wtD 5QI+Nk3waVO0m9AUas0dSbEZnqYXrj0/yIqg2f8IpjKhwfayHfc0XQ7h+NMffK3o WFpgFmIHb6L1nKBz7UCdLh4aAP4+AYzVH/yaw7aHNa1etZ4BH8CFcnEuzeaezU34 X5pYvP1vKtGae5kv5TuDKPll6UeA3xBSHhj6IJI8CSSwazPnOiwOITMPftCiAQ/3 qE4wR9AcYtdG8w+szAgpSIDkwqkEWGCTDtwxaTpfb4QZoxSCAcLRI8kYoUDkwV/u lPgV647lM619r5yeEJ/aLaH/b11mr3dgWustu1+N7xupyosCLpgaRdbsr6ZBu+WY ITalE2BSNMfjrV9RI45SVGfnWQ8RHzbmt9d5+oOyzo434zIx9pYY7IUEk8mSQ513 eb/D4Ap5BI2M2niOrVs6RnV9WEkvHdYq0/R23xO8LsmyyuAlBS3HuNrnK15z7Yva a2FhZSSD7guPUXQWgB78HhabfhpvYSwVL/T5oghYMe3Q13Q6NMoY/lbzb7GEmZyr FZy1sTnPXb0ScApqHVhKkXUgmWck/x/WDtsGf7NeGZV0uNQsLrdSURxCv1kbwlND GajyDd4stFGPlJx5ycb7ci6vZqVqNGCTWgED/jPQNkGHGbiIgU4= =Ygte -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded parms
On 11/12/2019 17:43, Christopher Schultz wrote: >> I'm not sure why we need this over and above the RequestFacade >> object. > > The RequestFacade is intended to protect the container from the > application, right? It can cut both ways. > And they are (usually) re-used. If a RequestFacade > object is retained by the application and used later, a later > client-request may be confused with the one the application was expectin > g> > In the solution I presented, the facades are one-time use and will > stay dead once they are killed. The application can't make a mistake > and read a later client-request or write to a later client-response. Setting RECYCLE_FACADES=true does exactly that. Continued used by the app triggers an NPE since the underlying request/response is no longer there. Hence my question. Mark > > - -chris > -BEGIN PGP SIGNATURE- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3xKskACgkQHPApP6U8 > pFhZyBAAtLxars/ZWmm5pRUkAg4yamIGLOflCBWE38j01NzGVmfibczkToyOkrIX > qLRSzlLWQsX3x3559I5gD5o9KAwixBCO0M+oCCGzOAmnnmXVs7dPvTnLYJKrJMW0 > U811tR9cNpNAN+rrdLhafYe+O8gHNsbuAOwLOnePc3lt/Q1ze7/kNYLsKDnrt8GG > lQKLE5Tv6mYr6tDnio9I9GjYnXP+G4swSvIL74o4swhzTLG5dzMAELOWtvBTBlPR > MoZvM6T1vIizhYf8xMMOCJuyMVxHm4kbesAlgAVdaff1pCqf9UjezrM/j+g7H0m9 > 5t6bC13EB/uKKGvz6sb1zJHr3yPiX+KRdiRaL59X96oMlKEZhBlySD1fk7KldU6W > 62N9u6l+TrrGG3aKOrIEPYLdIsMNrbPlPDirissnCygkFPMDRKNOjH9xs5CZqV1G > aXI45uC9gUtC4wCdNnSYE+f+uosDdY7WYTWdru0GuDDod/iOOOxNfPbJAyqE8t1w > G8KF+22rJvVxmehd78UL2LQtsYeOjLsZ2HJQAa7LwRZN+TCRhv2efJMIWncPbRuP > GjVxnoRBrkvFSnGNoP2ZKqP0lN7Jrf470NWr4wQHD1r1P5p8Z4suilViS77gxpSK > LQlO+bpl08BrOMY4f7+oRjeLA0FhwA8rNNc7LeTdxmud0LPN9Ro= > =gHas > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded parms
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mark, On 12/10/19 10:36, Mark Thomas wrote: > On 10/12/2019 14:27, Christopher Schultz wrote: > > > >> Would using org.apache.catalina.connector.RECYCLE_FACADES=true >> have made this problem go away? Or would the behavior have been >> the same, just less dangerous? > > I think it would have triggered some NPEs in the background > thread. > >> I'm wondering if Tomcat could or should have another safety >> feature to help catch this sort of thing in development. In all >> my development environments, I have the JDBC connection pool size >> set to a fixed maximum of 1 connection. This means that any >> potential deadlocks in the application due to sloppy >> connection-management will cause pretty early because we'll get >> pool-fetch timeouts, missing-return-connection errors, etc. >> >> Request object reuse has a measurable positive effect on >> performance in production, > > It would be worth confirming that is still the case for the Request > and Response objects. I suspect it is but it would be good to get > some recent, hard data. > >> but in development probably doesn't matter quite so much. In the >> same way that WebappClassLoader becomes inert when the >> application has been stopped, perhaps we could "shut-down" >> request / response / session objects that have been loaned to a >> request-processor thread. >> >> Something like this: > > I'm not sure why we need this over and above the RequestFacade > object. The RequestFacade is intended to protect the container from the application, right? And they are (usually) re-used. If a RequestFacade object is retained by the application and used later, a later client-request may be confused with the one the application was expectin g. In the solution I presented, the facades are one-time use and will stay dead once they are killed. The application can't make a mistake and read a later client-request or write to a later client-response. - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3xKskACgkQHPApP6U8 pFhZyBAAtLxars/ZWmm5pRUkAg4yamIGLOflCBWE38j01NzGVmfibczkToyOkrIX qLRSzlLWQsX3x3559I5gD5o9KAwixBCO0M+oCCGzOAmnnmXVs7dPvTnLYJKrJMW0 U811tR9cNpNAN+rrdLhafYe+O8gHNsbuAOwLOnePc3lt/Q1ze7/kNYLsKDnrt8GG lQKLE5Tv6mYr6tDnio9I9GjYnXP+G4swSvIL74o4swhzTLG5dzMAELOWtvBTBlPR MoZvM6T1vIizhYf8xMMOCJuyMVxHm4kbesAlgAVdaff1pCqf9UjezrM/j+g7H0m9 5t6bC13EB/uKKGvz6sb1zJHr3yPiX+KRdiRaL59X96oMlKEZhBlySD1fk7KldU6W 62N9u6l+TrrGG3aKOrIEPYLdIsMNrbPlPDirissnCygkFPMDRKNOjH9xs5CZqV1G aXI45uC9gUtC4wCdNnSYE+f+uosDdY7WYTWdru0GuDDod/iOOOxNfPbJAyqE8t1w G8KF+22rJvVxmehd78UL2LQtsYeOjLsZ2HJQAa7LwRZN+TCRhv2efJMIWncPbRuP GjVxnoRBrkvFSnGNoP2ZKqP0lN7Jrf470NWr4wQHD1r1P5p8Z4suilViS77gxpSK LQlO+bpl08BrOMY4f7+oRjeLA0FhwA8rNNc7LeTdxmud0LPN9Ro= =gHas -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded parms
On Tue, Dec 10, 2019 at 4:36 PM Mark Thomas wrote: > On 10/12/2019 14:27, Christopher Schultz wrote: > > > > > Would using org.apache.catalina.connector.RECYCLE_FACADES=true have > > made this problem go away? Or would the behavior have been the same, > > just less dangerous? > > I think it would have triggered some NPEs in the background thread. > > > I'm wondering if Tomcat could or should have another safety feature to > > help catch this sort of thing in development. In all my development > > environments, I have the JDBC connection pool size set to a fixed > > maximum of 1 connection. This means that any potential deadlocks in > > the application due to sloppy connection-management will cause pretty > > early because we'll get pool-fetch timeouts, missing-return-connection > > errors, etc. > > > > Request object reuse has a measurable positive effect on performance > > in production, > > It would be worth confirming that is still the case for the Request and > Response objects. I suspect it is but it would be good to get some > recent, hard data. > It would generate more GC per request and it is measurably slower on a best case scenario with a static file (the worst would be starting to use charsets and char input/output, or "large" Servlet buffers - the default is 8KB but people often increase it). Quick test: add request = null; line 305 of CoyoteAdapter. -10% with ab -k on tomcat.gif > > > but in development probably doesn't matter quite so > > much. In the same way that WebappClassLoader becomes inert when the > > application has been stopped, perhaps we could "shut-down" request / > > response / session objects that have been loaned to a > > request-processor thread. > > > > Something like this: > > I'm not sure why we need this over and above the RequestFacade object. > Rémy
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded parms
On 10/12/2019 14:27, Christopher Schultz wrote: > Would using org.apache.catalina.connector.RECYCLE_FACADES=true have > made this problem go away? Or would the behavior have been the same, > just less dangerous? I think it would have triggered some NPEs in the background thread. > I'm wondering if Tomcat could or should have another safety feature to > help catch this sort of thing in development. In all my development > environments, I have the JDBC connection pool size set to a fixed > maximum of 1 connection. This means that any potential deadlocks in > the application due to sloppy connection-management will cause pretty > early because we'll get pool-fetch timeouts, missing-return-connection > errors, etc. > > Request object reuse has a measurable positive effect on performance > in production, It would be worth confirming that is still the case for the Request and Response objects. I suspect it is but it would be good to get some recent, hard data. > but in development probably doesn't matter quite so > much. In the same way that WebappClassLoader becomes inert when the > application has been stopped, perhaps we could "shut-down" request / > response / session objects that have been loaned to a > request-processor thread. > > Something like this: I'm not sure why we need this over and above the RequestFacade object. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded parms
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mark, On 12/10/19 03:26, Mark Thomas wrote: > On 09/12/2019 19:54, Jerry Malcolm wrote: > > > >> Thanks to all of the responders on this problem. From the >> beginning of this problem, I was convinced that I was doing >> something bad wrong to cause this. I think now I have finally >> identified the 'bad thing'. Periodically the model code that runs >> in the request determines that I need to spin off another thread >> to do some background work. That thread needs a few parameters >> from the request object. So I pass the request object into the >> thread object. I realize now that doing that is a horrible thing >> to do. I was not aware until yesterday that request objects are >> pooled and recycled. So I figured holding onto the request >> object a little longer before it is discarded for that thread to >> use was not going to be a problem. Obviously, that was >> incorrect. > > Yep, that'll do it. > >> From what I can deduce, the main request thread finishes and >> returns. The request object recycle code runs and the object is >> returned to the pool. However, then the async thread later gets >> a parm and must apparently still be able to use the request >> object even though it's back in the pool. The >> 'parametersProcessed' gets set to true. Then the next request >> comes in, and that request object is assigned with >> parametersProcessed=true. > > I think this analysis is correct. > >> I'm assuming the fix is to get everything I need out of the >> request object and pass those parm values to the thread before >> returning instead of passing the request object itself to the >> thread. > > Correct. > >> Does all of this sound plausible as the source for this. The >> good news is that I'm still teachable... :-) > > Very plausible. > > Kudos for figuring this out so quickly - and for the very clear > write up. Would using org.apache.catalina.connector.RECYCLE_FACADES=true have made this problem go away? Or would the behavior have been the same, just less dangerous? I'm wondering if Tomcat could or should have another safety feature to help catch this sort of thing in development. In all my development environments, I have the JDBC connection pool size set to a fixed maximum of 1 connection. This means that any potential deadlocks in the application due to sloppy connection-management will cause pretty early because we'll get pool-fetch timeouts, missing-return-connection errors, etc. Request object reuse has a measurable positive effect on performance in production, but in development probably doesn't matter quite so much. In the same way that WebappClassLoader becomes inert when the application has been stopped, perhaps we could "shut-down" request / response / session objects that have been loaned to a request-processor thread. Something like this: Request realRequest = ... // fetch pooled request object HttpServletRequest req; RecyclableWrapper wrapper; if(developmentSafetyMode) { wrapper = new RecyclableWrapper(realRequest); req = (HttpServletRequest)Proxy.newProxyInstance(MigrateConfiguration.class.ge tClassLoader(), new Class[] { HttpServletRequest.class }, wrapper); } else { req = realRequest; } // dispatch request if(null != wrapper) { wrapper.recycle(); } And then the invocation handler class: static class RecyclableWrapper implements InvocationHandler { private T target; private boolean recycled; public RecyclableWrapper(T target) { this.target = target; } public void recycle() { recycled = true; } @Override public Object invoke(Object proxy, Method method, Object[] args) throws Throwable { if(recycled) throw new InvocationTargetException((Throwable)null, "Object has been recycled."); return method.invoke(target, args); } } This would catch these kinds of errors very early, and we could use a very specific error message that is easy to find using e.g. Google. OR just put a URL to a wiki page explaining the problem directly in the error message. I suppose this could even be done as a Valve (or Filter) so it's completely optional and trivially configurable. If a Valve, we'd have to wrap the Request instead of HttpServletRequest. I've only shown how to wrap the request above, but of course the response, session, and maybe even the streams should be wrapped, too, to be completely safe. Is anyone interested in this kind of thing being packaged with Tomcat? - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3vqzkACgkQHPApP6U8 pFg2qg/+PhgRgGxELAoqv/ZHPBjpv5IutL1X+uLa52Nl3NyuIPSWTFV7SB+ORE0l 5x8DVjW7MR7OUuu4Ptg2RPVmrufWDSANfujtJ9RS/ugzAii33O+3MIY2zHAZgfMk AKmcIQ3eGK4Ep1SByPTvD7nK3nmIlSzn7Z6d4bBeSawOFYcb1i1zs026gxfgTnnq x98+bl52YCDi9wCZk8G
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded parms
On 09/12/2019 19:54, Jerry Malcolm wrote: > Thanks to all of the responders on this problem. From the beginning of > this problem, I was convinced that I was doing something bad wrong to > cause this. I think now I have finally identified the 'bad thing'. > Periodically the model code that runs in the request determines that I > need to spin off another thread to do some background work. That thread > needs a few parameters from the request object. So I pass the request > object into the thread object. I realize now that doing that is a > horrible thing to do. I was not aware until yesterday that request > objects are pooled and recycled. So I figured holding onto the request > object a little longer before it is discarded for that thread to use was > not going to be a problem. Obviously, that was incorrect. Yep, that'll do it. > From what I can deduce, the main request thread finishes and returns. > The request object recycle code runs and the object is returned to the > pool. However, then the async thread later gets a parm and must > apparently still be able to use the request object even though it's back > in the pool. The 'parametersProcessed' gets set to true. Then the next > request comes in, and that request object is assigned with > parametersProcessed=true. I think this analysis is correct. > I'm assuming the fix is to get everything I need out of the request > object and pass those parm values to the thread before returning instead > of passing the request object itself to the thread. Correct. > Does all of this sound plausible as the source for this. The good news > is that I'm still teachable... :-) Very plausible. Kudos for figuring this out so quickly - and for the very clear write up. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded parms
On 12/9/2019 5:39 AM, Konstantin Kolinko wrote: вс, 8 дек. 2019 г. в 08:09, Jerry Malcolm : I have ajax code that sends requests to TC in a REST-style process. I send the parms url-encoded in the body. This has worked untouched literally for years. I have some new data objects in my db that "should be" sending the same type of requests through the same javascript routines. But for some inexplicable reason, the HttpServletRequest object is randomly deciding to not process the parms. When I try to enumerate the parms, I get none. Any parm I request comes back not found. I added some code to read the body myself (request.getReader(), etc). When the parms are available as it normally works, the reader is empty, which is what I would expect since it's been read by the request obj. But when the request object tells me I have no parms, I can read the entire url-encoded parm string from the reader, which if I understand things, means the request object never tried to read the stream, unless it somehow restores the stream after a read (??). But the important point I determined is that the parms are indeed present in the body... just not processed. [...] Thanks to all of the responders on this problem. From the beginning of this problem, I was convinced that I was doing something bad wrong to cause this. I think now I have finally identified the 'bad thing'. Periodically the model code that runs in the request determines that I need to spin off another thread to do some background work. That thread needs a few parameters from the request object. So I pass the request object into the thread object. I realize now that doing that is a horrible thing to do. I was not aware until yesterday that request objects are pooled and recycled. So I figured holding onto the request object a little longer before it is discarded for that thread to use was not going to be a problem. Obviously, that was incorrect. From what I can deduce, the main request thread finishes and returns. The request object recycle code runs and the object is returned to the pool. However, then the async thread later gets a parm and must apparently still be able to use the request object even though it's back in the pool. The 'parametersProcessed' gets set to true. Then the next request comes in, and that request object is assigned with parametersProcessed=true. I'm assuming the fix is to get everything I need out of the request object and pass those parm values to the thread before returning instead of passing the request object itself to the thread. Does all of this sound plausible as the source for this. The good news is that I'm still teachable... :-) Jerry - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded parms
вс, 8 дек. 2019 г. в 08:09, Jerry Malcolm : > > I have ajax code that sends requests to TC in a REST-style process. I > send the parms url-encoded in the body. This has worked untouched > literally for years. I have some new data objects in my db that "should > be" sending the same type of requests through the same javascript > routines. But for some inexplicable reason, the HttpServletRequest > object is randomly deciding to not process the parms. When I try to > enumerate the parms, I get none. Any parm I request comes back not > found. I added some code to read the body myself (request.getReader(), > etc). When the parms are available as it normally works, the reader is > empty, which is what I would expect since it's been read by the request > obj. But when the request object tells me I have no parms, I can read > the entire url-encoded parm string from the reader, which if I > understand things, means the request object never tried to read the > stream, unless it somehow restores the stream after a read (??). But > the important point I determined is that the parms are indeed present in > the body... just not processed. > > [...] I usually have the following in the pattern of AccessLogValve in my configurations: [%{org.apache.catalina.parameter_parse_failed}r %{org.apache.catalina.parameter_parse_failed_reason}r] Those request attributes are set in Tomcat whenever a problem is encountered by parameter parsing, e.g. an IOException if a client aborts the request. (The methods to process parameters in Servlet API do not have a way to report any errors). Those attributes can be used by org.apache.catalina.filters.FailedRequestFilter http://tomcat.apache.org/tomcat-9.0-doc/config/filter.html#Failed_Request_Filter You may look where those attributes are set in the source code. Mark wrote: > Issues like this can be caused if a reference to a request or response > is retained longer than it should be. You can try setting: > -Dorg.apache.catalina.connector.RECYCLE_FACADES=true +1. https://cwiki.apache.org/confluence/x/yColBg https://cwiki.apache.org/confluence/display/TOMCAT/Troubleshooting+and+Diagnostics#TroubleshootingandDiagnostics-CommonTroubleshootingScenario > I fired up my Windows laptop TC 9.x and got the exact same symptoms. You may also try with Tomcat 9.0.30 - release candidate is available and is currently being voted - see dev@ mailing list. Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded parms
On 09/12/2019 00:44, Jerry Malcolm wrote: > Mark, thank you so much for the info. I downloaded the TC source and > got everything up and running in Eclipse. After getting familiar with > the paths when it worked normally, I was able to single-step a failure. > I added a request.getParameter( "abc" ) as the first line of the JSP. > It steps into the RequestFacade then to the Catalina Request object. I > see the lazy parsing you described. It checks "parametersParsed". In > this error case, parametersParsed is already set to true, so it skips > parsing. However this is the first time here for this request, and the > parameters have not been parsed, and the getParameter returns null. > > I grepped the source tree to see if there were any other places where > parametersParsed is being set to true. The parseParameters method is > the only place. Being the first line in the JSP, I'm not aware of any > other code above the JSP that would be causing the parsing. And it > appears the only way out of the parser method is with an error condition > or successful parse, which neither appears to happen, implying it's > never being called in the error condition... leading to the possibility > that parametersParsed is somehow incorrectly set to true coming in. > > I saw some code about recycling the object instance. I haven't dug into > any of that code. Is there any possible way that a request object could > be recycled without being wiped clean first? That should not be possible. > I see the recycle method > where everything is cleaned out. But I guess it could somehow throw an > exception while cleaning it out. I realize this is a long shot. But I'm > just grasping at straws as to why the object would indicate with no > errors that the parameters had been parsed when they hadn't. You could out a try/catch around the recycle method but I have a hard time trying to thing of a scenario that would throw an Exception there. > I guess my next step is to figure out how to breakpoint when request > objects are allocated from the pool and trace it all the way up to my > JSP line. Can you give me a pointer to the class(es) that handle > request object pool mgmt? The reference chain is: Processor -> CoyoteRequest -> Request. Processors are pooled and allocated starting here: https://github.com/apache/tomcat/blob/master/java/org/apache/coyote/AbstractProtocol.java#L772 The CoyoteRequest and Response object are final fields on the Processor: https://github.com/apache/tomcat/blob/master/java/org/apache/coyote/AbstractProcessor.java#L64 The Request and Response are held as notes on the corresponding Coyote objects: https://github.com/apache/tomcat/blob/master/java/org/apache/catalina/connector/CoyoteAdapter.java#L303 > Also, I've seen several places in the code where it says "if debug" or > something like that. I've set logging to FINEST for everything, but I > believe there are debug statements that aren't showing. Is there > another way to enable 'debug'? That should be sufficient. Can you provide a specific example? > I'm resisting setting up a full rebuild environment of TC where I can > put in println statements. But I guess if that's required, I can do > it. If you'd like to talk me out that, I'm listening That is often where I end up investigating issues such as this. I usually try and turn those println statements into useful debug logging so I need less of them next time. Issues like this can be caused if a reference to a request or response is retained longer than it should be. You can try setting: -Dorg.apache.catalina.connector.RECYCLE_FACADES=true Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded parms
On 12/8/2019 6:23 AM, Mark Thomas wrote: On 08/12/2019 12:19, Mark Thomas wrote: Jerry, From your description, it sounds as if there is something about those requests that either isn't quite right or triggers an edge vase bug in Tomcat's request parsing. I'm happy to provide some pointers to help you debug this. I'll use 9.0.x links since I assume you prefer to debug this locally. Your starting point for debugging should probably be this class: Parameter parsing is lazy. It doesn't kick in until the application requests a parametyer Mark, thank you so much for the info. I downloaded the TC source and got everything up and running in Eclipse. After getting familiar with the paths when it worked normally, I was able to single-step a failure. I added a request.getParameter( "abc" ) as the first line of the JSP. It steps into the RequestFacade then to the Catalina Request object. I see the lazy parsing you described. It checks "parametersParsed". In this error case, parametersParsed is already set to true, so it skips parsing. However this is the first time here for this request, and the parameters have not been parsed, and the getParameter returns null. I grepped the source tree to see if there were any other places where parametersParsed is being set to true. The parseParameters method is the only place. Being the first line in the JSP, I'm not aware of any other code above the JSP that would be causing the parsing. And it appears the only way out of the parser method is with an error condition or successful parse, which neither appears to happen, implying it's never being called in the error condition... leading to the possibility that parametersParsed is somehow incorrectly set to true coming in. I saw some code about recycling the object instance. I haven't dug into any of that code. Is there any possible way that a request object could be recycled without being wiped clean first? I see the recycle method where everything is cleaned out. But I guess it could somehow throw an exception while cleaning it out. I realize this is a long shot. But I'm just grasping at straws as to why the object would indicate with no errors that the parameters had been parsed when they hadn't. I guess my next step is to figure out how to breakpoint when request objects are allocated from the pool and trace it all the way up to my JSP line. Can you give me a pointer to the class(es) that handle request object pool mgmt? Also, I've seen several places in the code where it says "if debug" or something like that. I've set logging to FINEST for everything, but I believe there are debug statements that aren't showing. Is there another way to enable 'debug'? I'm resisting setting up a full rebuild environment of TC where I can put in println statements. But I guess if that's required, I can do it. If you'd like to talk me out that, I'm listening Thx again. Jerry Please ignore this. I managed to hit send halfway though. A complete response is on the way... Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded parms
On 08/12/2019 12:19, Mark Thomas wrote: > Jerry, > > From your description, it sounds as if there is something about those > requests that either isn't quite right or triggers an edge vase bug in > Tomcat's request parsing. > > I'm happy to provide some pointers to help you debug this. I'll use > 9.0.x links since I assume you prefer to debug this locally. > > Your starting point for debugging should probably be this class: > > Parameter parsing is lazy. It doesn't kick in until the application > requests a parametyer Please ignore this. I managed to hit send halfway though. A complete response is on the way... Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded parms
Jerry, >From your description, it sounds as if there is something about those requests that either isn't quite right or triggers an edge case bug in Tomcat's request parsing. I'm happy to provide some pointers to help you debug this. I'll use 9.0.x links since I assume you prefer to debug this locally. Your starting point for debugging should probably be this class: https://github.com/apache/tomcat/blob/master/java/org/apache/catalina/connector/Request.java Parameter parsing is lazy. It doesn't kick in until the application calls one of the methods that needs the parameters to be parsed. Parsing is then performed in this method: https://github.com/apache/tomcat/blob/master/java/org/apache/catalina/connector/Request.java#L3147 You'll see there are various tests at the beginning of that method. Best guess, one of them is failing. (The code in 8.5.x is almost identical if you prefer to work with 8.5.x) HTH, Mark On 08/12/2019 05:08, Jerry Malcolm wrote: > I have ajax code that sends requests to TC in a REST-style process. I > send the parms url-encoded in the body. This has worked untouched > literally for years. I have some new data objects in my db that "should > be" sending the same type of requests through the same javascript > routines. But for some inexplicable reason, the HttpServletRequest > object is randomly deciding to not process the parms. When I try to > enumerate the parms, I get none. Any parm I request comes back not > found. I added some code to read the body myself (request.getReader(), > etc). When the parms are available as it normally works, the reader is > empty, which is what I would expect since it's been read by the request > obj. But when the request object tells me I have no parms, I can read > the entire url-encoded parm string from the reader, which if I > understand things, means the request object never tried to read the > stream, unless it somehow restores the stream after a read (??). But > the important point I determined is that the parms are indeed present in > the body... just not processed. > > I know this has to be something I'm doing wrong. It only occurs on one > or two new 'product items' on my web app. But it is pretty consistent > on those. I'll refresh 4 or 5 times, and the next time I get the > no-parms situation. I hit refresh again, and it works again. I've > analyzed all the parms and parm values to see if anything strange sticks > out. Nothing looks out of the ordinary. It's a pretty large parm set > (~158 parms). But I don't think that's anywhere near the largest set > I've sent over the years. I've dumped all of the headers for both > success and fail. No differences. The fail still shows the correct > content type and correct content length. Just no parms. I also checked > the logs to see if any exceptions are being thrown. Nothing. And all > of my code executes as normal until it can't find necessary parms and > then it spirals Hard drive has tons of space. And this doesn't > sound like a heap problem since it fails consistently on 2 out of 200+ > data objects and works fine on all of the other objects. Those two > objects should only differ from the others in > name/description/price/etc. Obviously, there's something else > different. I just can't figure it out. > > My production env is TC 8.5.xx on AWS Linux. Just to test, I fired up > my Windows laptop TC 9.x and got the exact same symptoms. > > I'm not expecting anyone to solve my problem off the top of their head > (but I won't refuse that if someone really knows what's happening). > What I really need is just a general overview of how the request object > works regarding url-encoded body parameters (and the impl class TC > uses). Is there simply some code in the impl that checks for > application/x-www-form-urlencoded and reads the buffer, parses the > parms, and stores them in a hash? Where does that occur in the > lifecycle of the request object? If I can just find the code that does > this function in the source, perhaps I can figure out what might make it > randomly decide to ignore this specific set of parms. Or maybe I can > fire everything up in an Eclipse environment and do some debugging. But > a java source file name for the request object impl class and a bit of > functional understanding will really help me along the way. > > Thx > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded parms
Jerry, >From your description, it sounds as if there is something about those requests that either isn't quite right or triggers an edge vase bug in Tomcat's request parsing. I'm happy to provide some pointers to help you debug this. I'll use 9.0.x links since I assume you prefer to debug this locally. Your starting point for debugging should probably be this class: Parameter parsing is lazy. It doesn't kick in until the application requests a parametyer On 08/12/2019 05:08, Jerry Malcolm wrote: > I have ajax code that sends requests to TC in a REST-style process. I > send the parms url-encoded in the body. This has worked untouched > literally for years. I have some new data objects in my db that "should > be" sending the same type of requests through the same javascript > routines. But for some inexplicable reason, the HttpServletRequest > object is randomly deciding to not process the parms. When I try to > enumerate the parms, I get none. Any parm I request comes back not > found. I added some code to read the body myself (request.getReader(), > etc). When the parms are available as it normally works, the reader is > empty, which is what I would expect since it's been read by the request > obj. But when the request object tells me I have no parms, I can read > the entire url-encoded parm string from the reader, which if I > understand things, means the request object never tried to read the > stream, unless it somehow restores the stream after a read (??). But > the important point I determined is that the parms are indeed present in > the body... just not processed. > > I know this has to be something I'm doing wrong. It only occurs on one > or two new 'product items' on my web app. But it is pretty consistent > on those. I'll refresh 4 or 5 times, and the next time I get the > no-parms situation. I hit refresh again, and it works again. I've > analyzed all the parms and parm values to see if anything strange sticks > out. Nothing looks out of the ordinary. It's a pretty large parm set > (~158 parms). But I don't think that's anywhere near the largest set > I've sent over the years. I've dumped all of the headers for both > success and fail. No differences. The fail still shows the correct > content type and correct content length. Just no parms. I also checked > the logs to see if any exceptions are being thrown. Nothing. And all > of my code executes as normal until it can't find necessary parms and > then it spirals Hard drive has tons of space. And this doesn't > sound like a heap problem since it fails consistently on 2 out of 200+ > data objects and works fine on all of the other objects. Those two > objects should only differ from the others in > name/description/price/etc. Obviously, there's something else > different. I just can't figure it out. > > My production env is TC 8.5.xx on AWS Linux. Just to test, I fired up > my Windows laptop TC 9.x and got the exact same symptoms. > > I'm not expecting anyone to solve my problem off the top of their head > (but I won't refuse that if someone really knows what's happening). > What I really need is just a general overview of how the request object > works regarding url-encoded body parameters (and the impl class TC > uses). Is there simply some code in the impl that checks for > application/x-www-form-urlencoded and reads the buffer, parses the > parms, and stores them in a hash? Where does that occur in the > lifecycle of the request object? If I can just find the code that does > this function in the source, perhaps I can figure out what might make it > randomly decide to ignore this specific set of parms. Or maybe I can > fire everything up in an Eclipse environment and do some debugging. But > a java source file name for the request object impl class and a bit of > functional understanding will really help me along the way. > > Thx > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
ServletRequest Obj Randomly not Processing x-www-form-urlencoded parms
I have ajax code that sends requests to TC in a REST-style process. I send the parms url-encoded in the body. This has worked untouched literally for years. I have some new data objects in my db that "should be" sending the same type of requests through the same javascript routines. But for some inexplicable reason, the HttpServletRequest object is randomly deciding to not process the parms. When I try to enumerate the parms, I get none. Any parm I request comes back not found. I added some code to read the body myself (request.getReader(), etc). When the parms are available as it normally works, the reader is empty, which is what I would expect since it's been read by the request obj. But when the request object tells me I have no parms, I can read the entire url-encoded parm string from the reader, which if I understand things, means the request object never tried to read the stream, unless it somehow restores the stream after a read (??). But the important point I determined is that the parms are indeed present in the body... just not processed. I know this has to be something I'm doing wrong. It only occurs on one or two new 'product items' on my web app. But it is pretty consistent on those. I'll refresh 4 or 5 times, and the next time I get the no-parms situation. I hit refresh again, and it works again. I've analyzed all the parms and parm values to see if anything strange sticks out. Nothing looks out of the ordinary. It's a pretty large parm set (~158 parms). But I don't think that's anywhere near the largest set I've sent over the years. I've dumped all of the headers for both success and fail. No differences. The fail still shows the correct content type and correct content length. Just no parms. I also checked the logs to see if any exceptions are being thrown. Nothing. And all of my code executes as normal until it can't find necessary parms and then it spirals Hard drive has tons of space. And this doesn't sound like a heap problem since it fails consistently on 2 out of 200+ data objects and works fine on all of the other objects. Those two objects should only differ from the others in name/description/price/etc. Obviously, there's something else different. I just can't figure it out. My production env is TC 8.5.xx on AWS Linux. Just to test, I fired up my Windows laptop TC 9.x and got the exact same symptoms. I'm not expecting anyone to solve my problem off the top of their head (but I won't refuse that if someone really knows what's happening). What I really need is just a general overview of how the request object works regarding url-encoded body parameters (and the impl class TC uses). Is there simply some code in the impl that checks for application/x-www-form-urlencoded and reads the buffer, parses the parms, and stores them in a hash? Where does that occur in the lifecycle of the request object? If I can just find the code that does this function in the source, perhaps I can figure out what might make it randomly decide to ignore this specific set of parms. Or maybe I can fire everything up in an Eclipse environment and do some debugging. But a java source file name for the request object impl class and a bit of functional understanding will really help me along the way. Thx - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org