Re: [OT] Trying to determine the minimum heap required for an operation
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 John, On 7/6/20 16:12, john.e.gr...@wellsfargo.com.INVALID wrote: > Chris, >> John, >> >> On 7/6/20 11:48, john.e.gr...@wellsfargo.com.INVALID wrote: >>> Chris, >>> >>> >>>> -Original Message- From: Christopher Schultz >>>> Sent: Monday, July 06, 2020 >>>> 10:21 AM To: Tomcat Users List >>>> Subject: [OT] Trying to determine the minimum heap required >>>> for an operation >>>> >>> All, >>> >>> Definitely off-topic, but it's the kind of weird thing someone >>> here might have experience with. >>> >>> I have an offline operation I'm considering bringing "inside" >>> my web-based application. My only concern is memory usage: it >>> requires that a bunch of data be loaded from a db into memory >>> and then analyzed. It doesn't take long to execute -- maybe 10 >>> seconds or so, so the memory can be released back to the rest >>> of the application. >>> >>> I've instrumented the command-line process with a thread which >>> runs every .5sec and captures the used-memory, maintaining a >>> high-water mark and reporting it after the whole operation is >>> done. The first time I ran it (with no specific JVM >>> memory-related settings), it reported that the high-water mark >>> was ~450MiB. >>> >>> I figured that was higher than necessary, and probably just >>> represented a lazy GC with loads of memory, so I constrained >>> the process using -Xmx64M. That resulted in a 16MiB high-water >>> mark. I tried again with -Xmx8M and the high-water mark became >>> 5MiB. >>> >>> Is there a particularly good way to force the GC to be as >>> aggressive as possible to see how low I can go, or should I >>> just keep playing-around with the -Xmx setting. >>> >>> Another option is to serialize my in-memory structure to the >>> disk to get a sense of the size in-memory, though it's really >>> not the same -- it will at least get me in the right ballpark. >>> >>> Any suggestions? >>> >>> Thanks, -chris >>>> >>>> --- - -- >>>> >>>> >> >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>> For additional commands, e-mail: >>>> users-h...@tomcat.apache.org >>> >>> >>> I guess I’m that person with the weird experiences. >>> >>> Is memory or CPU in short supply? If not, I don't think you'll >>> have >> a problem. This isn't 1997 anymore. I do think you should run a >> realistic load test, however. >> >> No specific problem exists, but this is a multi-user web >> application. Usually somewhere around 500 - 1000 users logged-in >> at once. Session size is typically quite low -- only a handful of >> small objects present with lots of sharing of "large" objects and >> structures. Heap size is set to max 1GiB on each server and >> memory usage shows a beautiful sawtooth pattern hovering around >> ~400MiB for days at a time. >> >> I will certainly limit the number of these operations that can >> occur at once, and they should be relatively rare. My test >> example was using a small data set, but the size of the data-set >> varies wildly with the client, so I have to be careful for the >> larger ones. >> >> Busting the heap isn't something I'd like to have happen. >> >>> To me the most important GC metric is time spent per >> minute/hour/etc. The next most important metric is individual >> pause durations. Through testing you'll see what those numbers >> are. I work with some large apps that have multi-GB heaps and >> it's rare to see GC time being over 1-2%. IOW, 600-1200ms per >> minute. Often it's a fraction of a percent. With those small >> numbers you're talking about, I don't think you'll have any >> trouble in this area unless the server is very heavily loaded. >> >> Actually, I'm not super concerned about performance of the GC >> itself. I was just wondering if there was a way to ask the JVM >> "if you *had* to accomplish this task with the smallest possible >> heap, what would it be?" >> >>> Be sure to enable verbose GC. In java 8, it's something like >>> this: >>> >>
RE: [OT] Trying to determine the minimum heap required for an operation
Chris, > John, > > On 7/6/20 11:48, john.e.gr...@wellsfargo.com.INVALID wrote: > > Chris, > > > > > >> -Original Message- From: Christopher Schultz > >> Sent: Monday, July 06, 2020 10:21 AM > >> To: Tomcat Users List Subject: [OT] Trying > >> to determine the minimum heap required for an operation > >> > > All, > > > > Definitely off-topic, but it's the kind of weird thing someone here > > might have experience with. > > > > I have an offline operation I'm considering bringing "inside" my > > web-based application. My only concern is memory usage: it requires > > that a bunch of data be loaded from a db into memory and then > > analyzed. It doesn't take long to execute -- maybe 10 seconds or so, > > so the memory can be released back to the rest of the application. > > > > I've instrumented the command-line process with a thread which runs > > every .5sec and captures the used-memory, maintaining a high-water > > mark and reporting it after the whole operation is done. The first > > time I ran it (with no specific JVM memory-related settings), it > > reported that the high-water mark was ~450MiB. > > > > I figured that was higher than necessary, and probably just > > represented a lazy GC with loads of memory, so I constrained the > > process using -Xmx64M. That resulted in a 16MiB high-water mark. I > > tried again with -Xmx8M and the high-water mark became 5MiB. > > > > Is there a particularly good way to force the GC to be as aggressive > > as possible to see how low I can go, or should I just keep > > playing-around with the -Xmx setting. > > > > Another option is to serialize my in-memory structure to the disk to > > get a sense of the size in-memory, though it's really not the same -- > > it will at least get me in the right ballpark. > > > > Any suggestions? > > > > Thanks, -chris > >> > >> - > >> > >> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > > > > > > I guess I’m that person with the weird experiences. > > > > Is memory or CPU in short supply? If not, I don't think you'll have > a problem. This isn't 1997 anymore. I do think you should run a realistic > load > test, however. > > No specific problem exists, but this is a multi-user web application. > Usually somewhere around 500 - 1000 users logged-in at once. Session size is > typically quite low -- only a handful of small objects present with lots of > sharing of "large" objects and structures. Heap size is set to max 1GiB on > each > server and memory usage shows a beautiful sawtooth pattern hovering > around ~400MiB for days at a time. > > I will certainly limit the number of these operations that can occur at once, > and they should be relatively rare. My test example was using a small data > set, but the size of the data-set varies wildly with the client, so I have to > be > careful for the larger ones. > > Busting the heap isn't something I'd like to have happen. > > > To me the most important GC metric is time spent per > minute/hour/etc. The next most important metric is individual pause > durations. Through testing you'll see what those numbers are. I work with > some large apps that have multi-GB heaps and it's rare to see GC time being > over 1-2%. IOW, 600-1200ms per minute. Often it's a fraction of a percent. > With those small numbers you're talking about, I don't think you'll have any > trouble in this area unless the server is very heavily loaded. > > Actually, I'm not super concerned about performance of the GC itself. > I was just wondering if there was a way to ask the JVM "if you *had* to > accomplish this task with the smallest possible heap, what would it be?" > > > Be sure to enable verbose GC. In java 8, it's something like > > this: > > > > -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps > - -Xloggc:/path/to/gc.log > > > > Run tests with and without the changes. You can analyze the GC output > > with tools like GCEasy and GCViewer. > Sure. > > Again, I'm more concerned with the overhead that will be required for a > particular operation, so I can predict when running such an operation might > end up endangering the application server's heap -- and therefore the > logged-in users. > &
Re: [OT] Trying to determine the minimum heap required for an operation
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 John, On 7/6/20 11:48, john.e.gr...@wellsfargo.com.INVALID wrote: > Chris, > > >> -Original Message- From: Christopher Schultz >> Sent: Monday, July 06, 2020 10:21 >> AM To: Tomcat Users List Subject: [OT] >> Trying to determine the minimum heap required for an operation >> > All, > > Definitely off-topic, but it's the kind of weird thing someone here > might have experience with. > > I have an offline operation I'm considering bringing "inside" my > web-based application. My only concern is memory usage: it requires > that a bunch of data be loaded from a db into memory and then > analyzed. It doesn't take long to execute -- maybe 10 seconds or > so, so the memory can be released back to the rest of the > application. > > I've instrumented the command-line process with a thread which runs > every .5sec and captures the used-memory, maintaining a high-water > mark and reporting it after the whole operation is done. The first > time I ran it (with no specific JVM memory-related settings), it > reported that the high-water mark was ~450MiB. > > I figured that was higher than necessary, and probably just > represented a lazy GC with loads of memory, so I constrained the > process using -Xmx64M. That resulted in a 16MiB high-water mark. I > tried again with -Xmx8M and the high-water mark became 5MiB. > > Is there a particularly good way to force the GC to be as > aggressive as possible to see how low I can go, or should I just > keep playing-around with the -Xmx setting. > > Another option is to serialize my in-memory structure to the disk > to get a sense of the size in-memory, though it's really not the > same -- it will at least get me in the right ballpark. > > Any suggestions? > > Thanks, -chris >> >> - >> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org > > > I guess I’m that person with the weird experiences. > > Is memory or CPU in short supply? If not, I don't think you'll > have a problem. This isn't 1997 anymore. I do think you should run a realistic load test, however. No specific problem exists, but this is a multi-user web application. Usually somewhere around 500 - 1000 users logged-in at once. Session size is typically quite low -- only a handful of small objects present with lots of sharing of "large" objects and structures. Heap size is set to max 1GiB on each server and memory usage shows a beautiful sawtooth pattern hovering around ~400MiB for days at a time. I will certainly limit the number of these operations that can occur at once, and they should be relatively rare. My test example was using a small data set, but the size of the data-set varies wildly with the client, so I have to be careful for the larger ones. Busting the heap isn't something I'd like to have happen. > To me the most important GC metric is time spent per minute/hour/etc. The next most important metric is individual pause durations. Through testing you'll see what those numbers are. I work with some large apps that have multi-GB heaps and it's rare to see GC time being over 1-2%. IOW, 600-1200ms per minute. Often it's a fraction of a percent. With those small numbers you're talking about, I don't think you'll have any trouble in this area unless the server is very heavily loaded. Actually, I'm not super concerned about performance of the GC itself. I was just wondering if there was a way to ask the JVM "if you *had* to accomplish this task with the smallest possible heap, what would it be?" > Be sure to enable verbose GC. In java 8, it's something like > this: > > -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps - -Xloggc:/path/to/gc.log > > Run tests with and without the changes. You can analyze the GC > output with tools like GCEasy and GCViewer. Sure. Again, I'm more concerned with the overhead that will be required for a particular operation, so I can predict when running such an operation might end up endangering the application server's heap -- and therefore the logged-in users. Theoretically, if the thread hits a heap-full error, the thread will experience an OOME, release it's (temporary) large object tree, and the GC will be able to recover, but after an OOME it's never a great plan to trust the JVM for very long. - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8DTR4ACgkQHPApP6U8 pFiAkQ//cWQ/CL35LJcRIervUhnByPXg/TN1
Re: [OT] Trying to determine the minimum heap required for an operation
Hello Christopher, On Mon, Jul 6, 2020, 8:50 PM Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > All, > > Definitely off-topic, but it's the kind of weird thing someone here > might have experience with. > > I have an offline operation I'm considering bringing "inside" my > web-based application. My only concern is memory usage: it requires > that a bunch of data be loaded from a db into memory and then > analyzed. It doesn't take long to execute -- maybe 10 seconds or so, > so the memory can be released back to the rest of the application. > > I've instrumented the command-line process with a thread which runs > every .5sec and captures the used-memory, maintaining a high-water > mark and reporting it after the whole operation is done. The first > time I ran it (with no specific JVM memory-related settings), it > reported that the high-water mark was ~450MiB. > > I figured that was higher than necessary, and probably just > represented a lazy GC with loads of memory, so I constrained the > process using -Xmx64M. That resulted in a 16MiB high-water mark. I > tried again with -Xmx8M and the high-water mark became 5MiB. > > Is there a particularly good way to force the GC to be as aggressive > as possible to see how low I can go, or should I just keep > playing-around with the -Xmx setting. > Looks like you wanted to track live objects generated by your program. I don't recall a JMX metrics of JVM to get that. If someway you can trigger couple of Full GCs during processing and logging everything on a GC log, then you'll get that. So, reducing -Xmx will help after some iterations. One thing you can try using is -Dsun.rmi.dgc.client.gcInterval and server flags and set them to 1 sec. That way you will get Full GC every 1 sec. It worked properly with Parallel GC and doesn't work with G1 in JDK 1.8. You can try and see if serves your purpose. More on that property can be found at https://docs.oracle.com/javase/8/docs/technotes/guides/rmi/sunrmiproperties.html > > Another option is to serialize my in-memory structure to the disk to > get a sense of the size in-memory, though it's really not the same -- > it will at least get me in the right ballpark. > > Any suggestions? > > Thanks, > - -chris > -BEGIN PGP SIGNATURE- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8DQT8ACgkQHPApP6U8 > pFhvJA/+N8CfjjWvBwkXSpAW6gbozyqgcxx0zt5z9TEbC4viCZNAQchlh0WE1jxF > dQL2NS6138VNOn45QfVLru7jcVQdk6loRSK4Dxe02neAD6sEwe0v3/zsuu7CDu4x > Ln3fwohp+5YNxHAUGc4ssGtw8cilShSSLnJCHwG3mxA+grxg6QvVRVqCxV0b+sCE > ocH0MirON5G7b7zETZohtm5lPcghwDy5SBQ4fVo3mLDjUGR8woGr8SL820pQ3BuY > rjGrJ7SHxq+rVnhOrtX6c4YdEebhjR963385kwPf1ND0GoeCp8Yk/LgySxBRPAbh > 2Kt0UTlbK7wYSDii6kVag1Ayrt5gCyUSrHndvVIl6SdI5gLWfZDbTB3+fvHNg+k5 > x/+Xx/YPvDbXv+b7CtO663uIKV+24iaVq94W+0NVacp3P0YmAmK1CZ9ggs7HQ/SC > uu3R1wRo4yp7eWszhgfpwPHJBvb9Krtfsr8P6rhs5Ry03pzblkmzzTRCvsE85UEZ > 96RN1OGx2YfPEM4+EN9+rxB1hcElLT8V420MAZd9Jx2n8JmJqdZl6DxJ7vgtvKKj > 0Y60VaC211M7tzq2zZ5Sh3th3X2tePPoeJQH/vYrreM4snlM8Mt22eQ7jVFri4bY > F+mu+8DGP3csWmY16nZ0SQ+ZDUS4E9yEplOHq1YKKyHSYGHjn88= > =u12k > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
RE: [OT] Trying to determine the minimum heap required for an operation
Chris, > -Original Message- > From: Christopher Schultz > Sent: Monday, July 06, 2020 10:21 AM > To: Tomcat Users List > Subject: [OT] Trying to determine the minimum heap required for an > operation > > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > All, > > Definitely off-topic, but it's the kind of weird thing someone here might have > experience with. > > I have an offline operation I'm considering bringing "inside" my web-based > application. My only concern is memory usage: it requires that a bunch of > data be loaded from a db into memory and then analyzed. It doesn't take > long to execute -- maybe 10 seconds or so, so the memory can be released > back to the rest of the application. > > I've instrumented the command-line process with a thread which runs every > .5sec and captures the used-memory, maintaining a high-water mark and > reporting it after the whole operation is done. The first time I ran it (with > no > specific JVM memory-related settings), it reported that the high-water mark > was ~450MiB. > > I figured that was higher than necessary, and probably just represented a > lazy GC with loads of memory, so I constrained the process using -Xmx64M. > That resulted in a 16MiB high-water mark. I tried again with -Xmx8M and the > high-water mark became 5MiB. > > Is there a particularly good way to force the GC to be as aggressive as > possible to see how low I can go, or should I just keep playing-around with > the -Xmx setting. > > Another option is to serialize my in-memory structure to the disk to get a > sense of the size in-memory, though it's really not the same -- it will at > least > get me in the right ballpark. > > Any suggestions? > > Thanks, > - -chris > -BEGIN PGP SIGNATURE- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8DQT8ACgkQHPApP > 6U8 > pFhvJA/+N8CfjjWvBwkXSpAW6gbozyqgcxx0zt5z9TEbC4viCZNAQchlh0WE1jx > F > dQL2NS6138VNOn45QfVLru7jcVQdk6loRSK4Dxe02neAD6sEwe0v3/zsuu7CDu > 4x > Ln3fwohp+5YNxHAUGc4ssGtw8cilShSSLnJCHwG3mxA+grxg6QvVRVqCxV0b+ > sCE > ocH0MirON5G7b7zETZohtm5lPcghwDy5SBQ4fVo3mLDjUGR8woGr8SL820pQ3 > BuY > rjGrJ7SHxq+rVnhOrtX6c4YdEebhjR963385kwPf1ND0GoeCp8Yk/LgySxBRPAbh > 2Kt0UTlbK7wYSDii6kVag1Ayrt5gCyUSrHndvVIl6SdI5gLWfZDbTB3+fvHNg+k5 > x/+Xx/YPvDbXv+b7CtO663uIKV+24iaVq94W+0NVacp3P0YmAmK1CZ9ggs7HQ > /SC > uu3R1wRo4yp7eWszhgfpwPHJBvb9Krtfsr8P6rhs5Ry03pzblkmzzTRCvsE85UEZ > 96RN1OGx2YfPEM4+EN9+rxB1hcElLT8V420MAZd9Jx2n8JmJqdZl6DxJ7vgtvKKj > 0Y60VaC211M7tzq2zZ5Sh3th3X2tePPoeJQH/vYrreM4snlM8Mt22eQ7jVFri4bY > F+mu+8DGP3csWmY16nZ0SQ+ZDUS4E9yEplOHq1YKKyHSYGHjn88= > =u12k > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org I guess I’m that person with the weird experiences. Is memory or CPU in short supply? If not, I don't think you'll have a problem. This isn't 1997 anymore. I do think you should run a realistic load test, however. To me the most important GC metric is time spent per minute/hour/etc. The next most important metric is individual pause durations. Through testing you'll see what those numbers are. I work with some large apps that have multi-GB heaps and it's rare to see GC time being over 1-2%. IOW, 600-1200ms per minute. Often it's a fraction of a percent. With those small numbers you're talking about, I don't think you'll have any trouble in this area unless the server is very heavily loaded. Be sure to enable verbose GC. In java 8, it's something like this: -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/path/to/gc.log Run tests with and without the changes. You can analyze the GC output with tools like GCEasy and GCViewer. John
[OT] Trying to determine the minimum heap required for an operation
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 All, Definitely off-topic, but it's the kind of weird thing someone here might have experience with. I have an offline operation I'm considering bringing "inside" my web-based application. My only concern is memory usage: it requires that a bunch of data be loaded from a db into memory and then analyzed. It doesn't take long to execute -- maybe 10 seconds or so, so the memory can be released back to the rest of the application. I've instrumented the command-line process with a thread which runs every .5sec and captures the used-memory, maintaining a high-water mark and reporting it after the whole operation is done. The first time I ran it (with no specific JVM memory-related settings), it reported that the high-water mark was ~450MiB. I figured that was higher than necessary, and probably just represented a lazy GC with loads of memory, so I constrained the process using -Xmx64M. That resulted in a 16MiB high-water mark. I tried again with -Xmx8M and the high-water mark became 5MiB. Is there a particularly good way to force the GC to be as aggressive as possible to see how low I can go, or should I just keep playing-around with the -Xmx setting. Another option is to serialize my in-memory structure to the disk to get a sense of the size in-memory, though it's really not the same -- it will at least get me in the right ballpark. Any suggestions? Thanks, - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8DQT8ACgkQHPApP6U8 pFhvJA/+N8CfjjWvBwkXSpAW6gbozyqgcxx0zt5z9TEbC4viCZNAQchlh0WE1jxF dQL2NS6138VNOn45QfVLru7jcVQdk6loRSK4Dxe02neAD6sEwe0v3/zsuu7CDu4x Ln3fwohp+5YNxHAUGc4ssGtw8cilShSSLnJCHwG3mxA+grxg6QvVRVqCxV0b+sCE ocH0MirON5G7b7zETZohtm5lPcghwDy5SBQ4fVo3mLDjUGR8woGr8SL820pQ3BuY rjGrJ7SHxq+rVnhOrtX6c4YdEebhjR963385kwPf1ND0GoeCp8Yk/LgySxBRPAbh 2Kt0UTlbK7wYSDii6kVag1Ayrt5gCyUSrHndvVIl6SdI5gLWfZDbTB3+fvHNg+k5 x/+Xx/YPvDbXv+b7CtO663uIKV+24iaVq94W+0NVacp3P0YmAmK1CZ9ggs7HQ/SC uu3R1wRo4yp7eWszhgfpwPHJBvb9Krtfsr8P6rhs5Ry03pzblkmzzTRCvsE85UEZ 96RN1OGx2YfPEM4+EN9+rxB1hcElLT8V420MAZd9Jx2n8JmJqdZl6DxJ7vgtvKKj 0Y60VaC211M7tzq2zZ5Sh3th3X2tePPoeJQH/vYrreM4snlM8Mt22eQ7jVFri4bY F+mu+8DGP3csWmY16nZ0SQ+ZDUS4E9yEplOHq1YKKyHSYGHjn88= =u12k -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org