Re: Pausing all thread but one.
Tullio, On 10/29/20 04:42, Tullio Bettinazzi wrote: OK tks for the X-Y explanation. I need to reload a few caches and when theese caches are reloaded nobody can access them. This is my problem. Your caches must be thread-safe, otherwise you would have serious problems. (You might want to go check on that, now.) The solution is simple: in your cache-reload thread, just loop through your caches, locking each one, emptying it, re-filling it, and then unlocking it. All will be well, and you don't have to pause any threads. What will happen is any in-flight requests will simply pause waiting for you, which is exactly how they would have behaved if you had "paused" the threads and then let them go. So the user's experience is the same, and your code is simpler and safer. -chris - Messaggio Originale - Da: "Christopher Schultz" A: users@tomcat.apache.org Inviato: 28/10/2020 20:13 Oggetto: Re: Pausing all thread but one. Tullio, On 10/28/20 06:27, Tullio Bettinazzi wrote: > I need to perform some maintenance operations pausing all user thread > for a small but meaningfull time (say 30 secs). Ugh. Why? What do you need to do that can't be handled with "proper" synchronization? > Is it possible to pause all user threads ? Maybe. It's very dangerous, though. https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.oracle.com%2Fjavase%2F8%2Fdocs%2Ftechnotes%2Fguides%2Fconcurrency%2FthreadPrimitiveDeprecation.html=af66902b=887af54c=y=n > How ? I'll leave that as an exercise for the reader at this point. > Any suggestion ? Yes: think if a better way to do what you are trying to do. This sounds like an X-Y problem[1]. You are trying to do X and you have decided that you need to do Y in order to accomplish X. So you are asking about Y. Why not just ask about X instead? -chris [1] https://urlsand.esvalabs.com/?u=http%3A%2F%2Fxyproblem.info%2F=af66902b=9b6e4fa6=y=n - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- This message has been checked by Libraesva ESG and is found to be clean. Follow this link to mark it as spam: https://gw1.idrnet.it/action/B465842741.A19C7/learn-spam - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: NGINX + tomcat 8.0.35 (110: Connection timed out)
Ayub,
On 10/28/20 23:28, Ayub Khan wrote:
During high load of 16k requests per minute, we notice below error in log.
[error] 2437#2437: *13335389 upstream timed out (110: Connection timed
out) while reading response header from upstream, server: jahez.net,
request: "GET /serviceContext/ServiceName?callback= HTTP/1.1", upstream: "
http://127.0.0.1:8080/serviceContext/ServiceName
Below is the flow of requests:
cloudflare-->AWS ALB--> NGINX--> Tomcat-->Elastic-search
I'm curious about why you are using all of cloudflare and ALB and nginx.
Seems like any one of those could provide what you are getting from all
3 of them.
In NGINX we have the below config
location /serviceContext/ServiceName{
proxy_pass http://localhost:8080/serviceContext/ServiceName;
proxy_http_version 1.1;
proxy_set_headerConnection $connection_upgrade;
proxy_set_headerUpgrade $http_upgrade;
proxy_set_headerHost $host;
proxy_set_headerX-Real-IP $remote_addr;
proxy_set_headerX-Forwarded-For $proxy_add_x_forwarded_for;
proxy_buffers 16 16k;
proxy_buffer_size 32k;
}
What is the maximum number of simultaneous requests that one nginx
instance will accept? What is the maximum number of simultaneous proxied
requests one nginx instance will make to a back-end Tomcat node? How
many nginx nodes do you have? How many Tomcat nodes?
below is tomcat connector config
50,000 threads is a LOT of threads.
We monitor the open file using *watch "sudo ls /proc/`cat
/var/run/tomcat8.pid`/fd/ | wc -l" *the number of tomcat open files keeps
increasing slowing the responses. the only option to recover from this is
to restart tomcat.
So this looks like Linux (/proc filesystem). Linux kernels have a 16-bit
pid space which means a theoretical max pid of 65535. In practice, the
max pid is actually to be found here:
$ cat /proc/sys/kernel/pid_max
32768
(on my Debian Linux system, 4.9.0-era kernel)
Each thread takes a pid. 50k threads means more than the maximum allowed
on the OS. So you will eventually hit some kind of serious problem with
that many threads.
How many fds do you get in the process before Tomcat grinds to a halt?
What does the CPU usage look like? The process I/O? Disk usage? What
does a thread dump look like (if you have the disk space to dump it!)?
Why do you need that many threads?
-chris
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk "Can not determine the proper size for pid_t" on macOS 10.15.7
Brian, On 10/28/20 21:24, Paquin, Brian wrote: Chris, On Oct 27, 2020, at 12:31 PM, Christopher Schultz wrote: Brian On 10/26/20 15:33, Paquin, Brian wrote: I’m trying to build httpd and mod_jk for the first time on a macOS 10.15.7 box. XCode 12.1 is installed and I was able to compile OpenSSL 1.1.1g. I got an error “Can not determine the proper size for pid_t” when compiling httpd (v2.4.46) with included apr (v1.7.0). This issue https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbz.apache.org%2Fbugzilla%2Fshow_bug.cgi%3Fid%3D64753data=04%7C01%7Cbrian.paquin%40yale.edu%7C4a009b9f4c19439afc4708d87a95d92e%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637394131299938700%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=WdzfXIIBvFQzON0aA2Q3EHqNyHK2bFqXpGYm2aEyi1A%3Dreserved=0 provided a diff patch that adds “#include ” in a number of locations. Applying this patch allowed me to compile httpd! Weird. pid_it is defined in and yet the patch adds to fix it. I don't have access to my Catalina machine right now, but my clang-based Mojave machine still says to use when you "man getpid": " GETPID(2) BSD System Calls Manual NAME getpid, getppid -- get parent or calling process identification SYNOPSIS #include pid_t getpid(void); " $ cc --version Apple LLVM version 10.0.1 (clang-1001.0.46.4) Target: x86_64-apple-darwin18.7.0 Thread model: posix InstalledDir: /Library/Developer/CommandLineTools/usr/bin Now I am trying to compile mod_jk (v1.2.48), and I get the same error. Does someone have a patch file I can use to get around this issue? $ ./configure CFLAGS='-arch x86_64' APXSLDFLAGS='-arch x86_64' --with-apxs=/usr/local/apache2/bin/apxs $ make Making all in common /usr/local/apache-2.4.46/build/libtool --silent --mode=compile gcc -I. -I/usr/local/apache-2.4.46/include -arch x86_64 -DHAVE_CONFIG_H -arch x86_64 -DHAVE_APR -I/usr/local/apache-2.4.46/include -I/usr/local/apache-2.4.46/include -arch x86_64 -DHAVE_CONFIG_H -DDARWIN -DSIGPROCMASK_SETS_THREAD_MASK -DDARWIN_10 -c jk_ajp12_worker.c -o jk_ajp12_worker.lo In file included from jk_ajp12_worker.c:25: In file included from ./jk_ajp12_worker.h:26: In file included from ./jk_logger.h:26: In file included from ./jk_global.h:340: ./jk_types.h:56:2: error: Can not determine the proper size for pid_t #error Can not determine the proper size for pid_t ^ ./jk_types.h:62:2: error: Can not determine the proper size for pthread_t #error Can not determine the proper size for pthread_t ^ 2 errors generated. make[1]: *** [jk_ajp12_worker.lo] Error 1 make: *** [all-recursive] Error 1 $ I'm sorry, I have no idea how configure does its magic. The auto-generated jk_types.h looks like a hand-wavy template to me. You can probably hack it briefly by running "configure" (which you already did) and then hand-editing include/jk_types.h (ignoring the warning NOT to hand-edit it!) and manually adding: #include to the top. I added the line above to ./tomcat-connectors-1.2.48-src/native/common/jk_types.h and tried running make again but got the same error. I do not have an include directory in ./tomcat-connectors-1.2.48-src/native/. Any other suggestions? Sorry, native/common/jk_types.h was the file I meant. Can you just post the whole content of that file? I'm guessing that you have something like: typedef jk_pid_t @pid_t@ ; leftover from 'configure' not finding the right definition. We may have to adjust that line, too. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Our webapp is running very slowly on one particular customer box
James, On 10/28/20 16:40, James H. H. Lampert wrote: First, thanks once again, Mr. Schultz, for getting back to me. I noticed something rather promising: it seems that maxThreads for the Port 443 connector were set at 150 for System "A" (problem box), but 400 for System "J" (box that's quite happy). I've restarted Tomcat with the maxThreads bumped up to 400, and so far, it seems much happier than it was. That could have been the problem all along. Hmm. That doesn't sound very satisfying to me, honestly. Allowing *more* load uses *less* GC and/or fewer page-faults? Seems fishy. My colleagues and I also observed that yesterday, when we did *not* shut down and restart, the slowdown and the nearly-full "tenured-SOA" portion of the heap eventually resolved itself, which suggests that it wasn't a memory leak in any even remotely conventional sense of the term. That's a Good Thing, but also not very satisfying when you just want it to stop sucking and let your users get work done :) The page-faulting is a virtual memory term: on an AS/400, the entire combined total of main storage and disk is addressable (the concept is called "Single-Level Store"), and virtual storage paging is built into the OS at a very low level; a "page fault" is when a process finds tries to access something that's been paged out to disk. Yes, this is the common definition of a page-fault, not just an AS/400 thing. Good to know for sure that AS/400 doesn't re-define that term, though :) How long has the process on System J been running? How about System A (before you restarted the JVM)? As to the private memory pool, it's not that the subsystem is restricted to its private pool; rather, everything else is kept *out* of that private pool. It still has full access to the "Machine" and "Base" shared pools. Okay, so it's like a guaranteed-minimum memory space? -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Weirdest Tomcat Behavior Ever?
> -Original Message- > From: Mark Thomas > Sent: Thursday, October 29, 2020 5:45 AM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > On 28/10/2020 20:32, Mark Thomas wrote: > > > > > I have the off-list mail and will start looking at the logs shortly. > > Progress. I think. I'll start with the following summary of the log data. > > Source Time Activity > > pcap15:14:25.375451 SYN proxy to Tomcat > pcap15:14:25.375493 SYN, ACK Tomcat to proxy > pcap15:14:25.375839 ACK proxy to Tomcat > pcap15:14:25.375892 GET request proxy to Tomcat > pcap15:14:25.375911 ACK Tomcat to proxy > debug 15:14:25.376 o.a.c.http11.InternalOutputBuffer.init > pcap15:14:25.376777 FIN Tomcat to proxy > pcap15:14:25.377036 FIN, ACK proxy to Tomcat > pcap15:14:25.377048 ACK Tomcat proxy > debug 15:14:25.383 o.a.c.http11.InternalOutputBuffer.commit > debug 15:14:25.383 o.a.c.http11.InternalOutputBuffer$1.doWrite > debug 15:14:25.383 o.a.c.http11.InternalOutputBuffer$1.nextRequest > debug 15:14:25.383 o.a.c.http11.InternalOutputBuffer$1.recycle > > Tomcat writes the request body to the buffer but when Tomcat tries to flush > those bytes out to the network it finds that the socket has been closed. That > normally indicates that the client has dropped the connection. This is > sufficiently common that Tomcat swallows the exception. However, we > know (from the pcap file) that the client did not drop the connection, Tomcat > did. > That's the first hard evidence of where the problem lies. I feel like we're zeroing in on it. > What is strange here is that with BIO is there is a 1-2-1 relationship between > threads and sockets for the life of the socket. While I can see how a retained > reference could truncate a response (including the > headers) I don't yet see how the socket could get closed. > > I think more debug logging is required. I am currently working on that. > I'll apply the new patch and restart the tomcat this evening. Just to be safe, I'm only applying it to one of the tomcat instances. --Eric > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org Disclaimer : This email and any files transmitted with it are confidential and intended solely for intended recipients. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management. Warning: Although Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
Re: Weirdest Tomcat Behavior Ever?
On 29/10/2020 10:44, Mark Thomas wrote: > I think more debug logging is required. I am currently working on that. v2 of the debug logging patch is: http://home.apache.org/~markt/dev/v7.0.72-custom-patch-v2/ Full disclosure: This patch is more invasive. I had to wrap the java.net.Socket objects to ensure I catch all attempts to close the socket. That patch should be safe but it goes further than just adding some log messages. Source code for patches: https://github.com/markt-asf/tomcat/commits/debug-7.0.72 Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Weirdest Tomcat Behavior Ever?
On 28/10/2020 20:32, Mark Thomas wrote: > I have the off-list mail and will start looking at the logs shortly. Progress. I think. I'll start with the following summary of the log data. Source Time Activity pcap15:14:25.375451 SYN proxy to Tomcat pcap15:14:25.375493 SYN, ACK Tomcat to proxy pcap15:14:25.375839 ACK proxy to Tomcat pcap15:14:25.375892 GET request proxy to Tomcat pcap15:14:25.375911 ACK Tomcat to proxy debug 15:14:25.376 o.a.c.http11.InternalOutputBuffer.init pcap15:14:25.376777 FIN Tomcat to proxy pcap15:14:25.377036 FIN, ACK proxy to Tomcat pcap15:14:25.377048 ACK Tomcat proxy debug 15:14:25.383 o.a.c.http11.InternalOutputBuffer.commit debug 15:14:25.383 o.a.c.http11.InternalOutputBuffer$1.doWrite debug 15:14:25.383 o.a.c.http11.InternalOutputBuffer$1.nextRequest debug 15:14:25.383 o.a.c.http11.InternalOutputBuffer$1.recycle Tomcat writes the request body to the buffer but when Tomcat tries to flush those bytes out to the network it finds that the socket has been closed. That normally indicates that the client has dropped the connection. This is sufficiently common that Tomcat swallows the exception. However, we know (from the pcap file) that the client did not drop the connection, Tomcat did. What is strange here is that with BIO is there is a 1-2-1 relationship between threads and sockets for the life of the socket. While I can see how a retained reference could truncate a response (including the headers) I don't yet see how the socket could get closed. I think more debug logging is required. I am currently working on that. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Pausing all thread but one.
OK tks for the X-Y explanation. I need to reload a few caches and when theese caches are reloaded nobody can access them. This is my problem. Tks Dott. Tullio Bettinazzi Responsabile R D Axioma - Tel. +39 039 22511 www.axioma.it Trattamento dei dati personali conforme ai sensi del Regolamento Europeo 2016/679 - Messaggio Originale - Da: "Christopher Schultz" ch...@christopherschultz.net A: users@tomcat.apache.org Inviato: 28/10/2020 20:13 Oggetto: Re: Pausing all thread but one. Tullio, On 10/28/20 06:27, Tullio Bettinazzi wrote: I need to perform some maintenance operations pausing all user thread for a small but meaningfull time (say 30 secs). Ugh. Why? What do you need to do that can't be handled with "proper" synchronization? Is it possible to pause all user threads ? Maybe. It's very dangerous, though. https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.oracle.com%2Fjavase%2F8%2Fdocs%2Ftechnotes%2Fguides%2Fconcurrency%2FthreadPrimitiveDeprecation.htmle=af66902bh=887af54cf=yp=n How ? I'll leave that as an exercise for the reader at this point. Any suggestion ? Yes: think if a better way to do what you are trying to do. This sounds like an X-Y problem[1]. You are trying to do X and you have decided that you need to do Y in order to accomplish X. So you are asking about Y. Why not just ask about X instead? -chris [1] https://urlsand.esvalabs.com/?u=http%3A%2F%2Fxyproblem.info%2Fe=af66902bh=9b6e4fa6f=yp=n - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- This message has been checked by Libraesva ESG and is found to be clean. Follow this link to mark it as spam: https://gw1.idrnet.it/action/B465842741.A19C7/learn-spam - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
