Re: [ovirt-users] Can I reduce the Java heap size of engine-backup???
On 12/31/2015 10:42 AM, Juan Hernández wrote: > On 12/31/2015 08:48 AM, Yedidyah Bar David wrote: >> On Wed, Dec 30, 2015 at 7:50 PM, John Florian wrote: >>> On 12/29/2015 02:02 AM, Yedidyah Bar David wrote: On Tue, Dec 29, 2015 at 12:51 AM, John Florian wrote: > I'm trying to run the engine-backup script via a Bacula job using the > RunScript option so that the engine-backup dumps its output someplace > where Bacula will collect it once engine-backup finishes. However the > job is failing and with enough digging I eventually learned the script > was writing the following in /tmp/hs_err_pid5789.log: > > # > # There is insufficient memory for the Java Runtime Environment to > continue. > # Native memory allocation (mmap) failed to map 2555904 bytes for > committing reserved memory. > # Possible reasons: > # The system is out of physical RAM or swap space > # In 32 bit mode, the process size limit was hit > # Possible solutions: > # Reduce memory load on the system > # Increase physical memory or swap space > # Check if swap backing store is full > # Use 64 bit Java on a 64 bit OS > # Decrease Java heap size (-Xmx/-Xms) > # Decrease number of Java threads > # Decrease Java thread stack sizes (-Xss) > # Set larger code cache with -XX:ReservedCodeCacheSize= > # This output file may be truncated or incomplete. > # > # Out of Memory Error (os_linux.cpp:2627), pid=5789, tid=140709998221056 > # > # JRE version: (8.0_65-b17) (build ) > # Java VM: OpenJDK 64-Bit Server VM (25.65-b01 mixed mode linux-amd64 > compressed oops) > # Failed to write core dump. Core dumps have been disabled. To enable > core dumping, try "ulimit -c unlimited" before starting Java again > # > > > So is there any good way to reduce the Java heap size? I mean I know > what -Xmx does, but where might I try setting it, ideally so that it > affects the engine-backup only? Any idea of good setting for a very > small environment with a dozen VMs? engine-backup does not directly call nor need java. AFAICS it only calls it indirectly as part of some other initialization by running java-home [1], which is a script that decides what JAVA_HOME to use for the engine. This script only runs 'java -version', which imo should not need that much memory. Perhaps there is something else I do not fully understand, such as bacula severely limiting available resources for the process it runs, or something like that. If you only want to debug it, and not as a recommended final solution, you can create a script [2] which only outputs the needed java home. Simply run [1] and make [2] echo the same thing. If [2] exists, [1] will only run it and nothing else, as you can see inside it. I do not think this will work - quite likely engine-backup will fail shortly later, if indeed it gets access to so little memory. Please report back. Thanks and good luck, [1] /usr/share/ovirt-engine/bin/java-home [2] /usr/share/ovirt-engine/bin/java-home.local >>> Thanks for the info and response Didi. Doing the above did allow the >>> backup to run successfully. >> OK. >> >>> I had also replaced the Bacula RunScript >>> with "bash -c ulimit" which reported unlimited but I don't play with >>> those types of limits enough to know if that's correctly reporting to >>> what engine-backup is constrained. >> And was this enough? >> >>> I did occur to me that perhaps a >>> better way to learn of any such constraints would be to query Bacula's >>> file daemon (the only necessary Bacula component running on client >>> systems that are getting backed up) since I suspect it must be this >>> component that's actually spawning the RunScript client side. From the >>> Bacula Director (server side) I queried the status of the client which >>> is my oVirt engine and it reports: >>> >>> europa.doubledog.org-fd Version: 5.2.13 (19 February 2013) >>> x86_64-redhat-linux-gnu redhat (Core) >>> Daemon started 28-Dec-15 16:08. Jobs: run=2 running=0. >>> Heap: heap=32,768 smbytes=190,247 max_bytes=1,599,864 bufs=100 >>> max_bufs=6,758 >>> Sizeof: boffset_t=8 size_t=8 debug=0 trace=0 >>> >>> Alas, I know of no way to increase any of the bacula-fd limits. If I >>> dead-end here, perhaps I'll query the Bacula mailing lists. >> For both yourself and for others, I think it's best to continue with >> this route. >> >> Also note that I have no idea how much memory pg_dump might need on >> a larger database, also including dwh which tends to get larger faster >> than the engine's. >> >>> Meanwhile I tried the following for a more permanent solution but this >>> failed same as before: >>> >>> # diff -u java-home.orig-3.6.1.3 java-home >>> --- java-home.orig-3.6.1.3 2015-12-10 13:07:44.0 -0500 >>> +++ java-home
Re: [ovirt-users] Can I reduce the Java heap size of engine-backup???
On 12/31/2015 08:48 AM, Yedidyah Bar David wrote: > On Wed, Dec 30, 2015 at 7:50 PM, John Florian wrote: >> On 12/29/2015 02:02 AM, Yedidyah Bar David wrote: >>> On Tue, Dec 29, 2015 at 12:51 AM, John Florian >>> wrote: I'm trying to run the engine-backup script via a Bacula job using the RunScript option so that the engine-backup dumps its output someplace where Bacula will collect it once engine-backup finishes. However the job is failing and with enough digging I eventually learned the script was writing the following in /tmp/hs_err_pid5789.log: # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 2555904 bytes for committing reserved memory. # Possible reasons: # The system is out of physical RAM or swap space # In 32 bit mode, the process size limit was hit # Possible solutions: # Reduce memory load on the system # Increase physical memory or swap space # Check if swap backing store is full # Use 64 bit Java on a 64 bit OS # Decrease Java heap size (-Xmx/-Xms) # Decrease number of Java threads # Decrease Java thread stack sizes (-Xss) # Set larger code cache with -XX:ReservedCodeCacheSize= # This output file may be truncated or incomplete. # # Out of Memory Error (os_linux.cpp:2627), pid=5789, tid=140709998221056 # # JRE version: (8.0_65-b17) (build ) # Java VM: OpenJDK 64-Bit Server VM (25.65-b01 mixed mode linux-amd64 compressed oops) # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # So is there any good way to reduce the Java heap size? I mean I know what -Xmx does, but where might I try setting it, ideally so that it affects the engine-backup only? Any idea of good setting for a very small environment with a dozen VMs? >>> engine-backup does not directly call nor need java. >>> >>> AFAICS it only calls it indirectly as part of some other initialization >>> by running java-home [1], which is a script that decides what JAVA_HOME >>> to use for the engine. This script only runs 'java -version', which imo >>> should not need that much memory. Perhaps there is something else I do >>> not fully understand, such as bacula severely limiting available resources >>> for the process it runs, or something like that. >>> >>> If you only want to debug it, and not as a recommended final solution, >>> you can create a script [2] which only outputs the needed java home. >>> Simply run [1] and make [2] echo the same thing. If [2] exists, [1] will >>> only run it and nothing else, as you can see inside it. >>> >>> I do not think this will work - quite likely engine-backup will fail >>> shortly later, if indeed it gets access to so little memory. Please >>> report back. Thanks and good luck, >>> >>> [1] /usr/share/ovirt-engine/bin/java-home >>> [2] /usr/share/ovirt-engine/bin/java-home.local >> Thanks for the info and response Didi. Doing the above did allow the >> backup to run successfully. > > OK. > >> I had also replaced the Bacula RunScript >> with "bash -c ulimit" which reported unlimited but I don't play with >> those types of limits enough to know if that's correctly reporting to >> what engine-backup is constrained. > > And was this enough? > >> I did occur to me that perhaps a >> better way to learn of any such constraints would be to query Bacula's >> file daemon (the only necessary Bacula component running on client >> systems that are getting backed up) since I suspect it must be this >> component that's actually spawning the RunScript client side. From the >> Bacula Director (server side) I queried the status of the client which >> is my oVirt engine and it reports: >> >> europa.doubledog.org-fd Version: 5.2.13 (19 February 2013) >> x86_64-redhat-linux-gnu redhat (Core) >> Daemon started 28-Dec-15 16:08. Jobs: run=2 running=0. >> Heap: heap=32,768 smbytes=190,247 max_bytes=1,599,864 bufs=100 >> max_bufs=6,758 >> Sizeof: boffset_t=8 size_t=8 debug=0 trace=0 >> >> Alas, I know of no way to increase any of the bacula-fd limits. If I >> dead-end here, perhaps I'll query the Bacula mailing lists. > > For both yourself and for others, I think it's best to continue with > this route. > > Also note that I have no idea how much memory pg_dump might need on > a larger database, also including dwh which tends to get larger faster > than the engine's. > >> >> Meanwhile I tried the following for a more permanent solution but this >> failed same as before: >> >> # diff -u java-home.orig-3.6.1.3 java-home >> --- java-home.orig-3.6.1.3 2015-12-10 13:07:44.0 -0500 >> +++ java-home 2015-12-30 12:12:45.779462769 -0500 >> @@ -13,7 +13,7 @@ >> local ret=1 >> >> if [ -x "${dir}/bin/java" ]; then >> -
Re: [ovirt-users] Can I reduce the Java heap size of engine-backup???
On Wed, Dec 30, 2015 at 7:50 PM, John Florian wrote: > On 12/29/2015 02:02 AM, Yedidyah Bar David wrote: >> On Tue, Dec 29, 2015 at 12:51 AM, John Florian >> wrote: >>> I'm trying to run the engine-backup script via a Bacula job using the >>> RunScript option so that the engine-backup dumps its output someplace >>> where Bacula will collect it once engine-backup finishes. However the >>> job is failing and with enough digging I eventually learned the script >>> was writing the following in /tmp/hs_err_pid5789.log: >>> >>> # >>> # There is insufficient memory for the Java Runtime Environment to continue. >>> # Native memory allocation (mmap) failed to map 2555904 bytes for >>> committing reserved memory. >>> # Possible reasons: >>> # The system is out of physical RAM or swap space >>> # In 32 bit mode, the process size limit was hit >>> # Possible solutions: >>> # Reduce memory load on the system >>> # Increase physical memory or swap space >>> # Check if swap backing store is full >>> # Use 64 bit Java on a 64 bit OS >>> # Decrease Java heap size (-Xmx/-Xms) >>> # Decrease number of Java threads >>> # Decrease Java thread stack sizes (-Xss) >>> # Set larger code cache with -XX:ReservedCodeCacheSize= >>> # This output file may be truncated or incomplete. >>> # >>> # Out of Memory Error (os_linux.cpp:2627), pid=5789, tid=140709998221056 >>> # >>> # JRE version: (8.0_65-b17) (build ) >>> # Java VM: OpenJDK 64-Bit Server VM (25.65-b01 mixed mode linux-amd64 >>> compressed oops) >>> # Failed to write core dump. Core dumps have been disabled. To enable >>> core dumping, try "ulimit -c unlimited" before starting Java again >>> # >>> >>> >>> So is there any good way to reduce the Java heap size? I mean I know >>> what -Xmx does, but where might I try setting it, ideally so that it >>> affects the engine-backup only? Any idea of good setting for a very >>> small environment with a dozen VMs? >> engine-backup does not directly call nor need java. >> >> AFAICS it only calls it indirectly as part of some other initialization >> by running java-home [1], which is a script that decides what JAVA_HOME >> to use for the engine. This script only runs 'java -version', which imo >> should not need that much memory. Perhaps there is something else I do >> not fully understand, such as bacula severely limiting available resources >> for the process it runs, or something like that. >> >> If you only want to debug it, and not as a recommended final solution, >> you can create a script [2] which only outputs the needed java home. >> Simply run [1] and make [2] echo the same thing. If [2] exists, [1] will >> only run it and nothing else, as you can see inside it. >> >> I do not think this will work - quite likely engine-backup will fail >> shortly later, if indeed it gets access to so little memory. Please >> report back. Thanks and good luck, >> >> [1] /usr/share/ovirt-engine/bin/java-home >> [2] /usr/share/ovirt-engine/bin/java-home.local > Thanks for the info and response Didi. Doing the above did allow the > backup to run successfully. OK. > I had also replaced the Bacula RunScript > with "bash -c ulimit" which reported unlimited but I don't play with > those types of limits enough to know if that's correctly reporting to > what engine-backup is constrained. And was this enough? > I did occur to me that perhaps a > better way to learn of any such constraints would be to query Bacula's > file daemon (the only necessary Bacula component running on client > systems that are getting backed up) since I suspect it must be this > component that's actually spawning the RunScript client side. From the > Bacula Director (server side) I queried the status of the client which > is my oVirt engine and it reports: > > europa.doubledog.org-fd Version: 5.2.13 (19 February 2013) > x86_64-redhat-linux-gnu redhat (Core) > Daemon started 28-Dec-15 16:08. Jobs: run=2 running=0. > Heap: heap=32,768 smbytes=190,247 max_bytes=1,599,864 bufs=100 > max_bufs=6,758 > Sizeof: boffset_t=8 size_t=8 debug=0 trace=0 > > Alas, I know of no way to increase any of the bacula-fd limits. If I > dead-end here, perhaps I'll query the Bacula mailing lists. For both yourself and for others, I think it's best to continue with this route. Also note that I have no idea how much memory pg_dump might need on a larger database, also including dwh which tends to get larger faster than the engine's. > > Meanwhile I tried the following for a more permanent solution but this > failed same as before: > > # diff -u java-home.orig-3.6.1.3 java-home > --- java-home.orig-3.6.1.3 2015-12-10 13:07:44.0 -0500 > +++ java-home 2015-12-30 12:12:45.779462769 -0500 > @@ -13,7 +13,7 @@ > local ret=1 > > if [ -x "${dir}/bin/java" ]; then > - local version="$("${dir}/bin/java" -version 2>&1 | sed \ > + local version="$("${dir}/bin/java" -Xmx 8 -version 2>&1 > | sed \ > -
Re: [ovirt-users] Can I reduce the Java heap size of engine-backup???
On 12/29/2015 02:02 AM, Yedidyah Bar David wrote: > On Tue, Dec 29, 2015 at 12:51 AM, John Florian wrote: >> I'm trying to run the engine-backup script via a Bacula job using the >> RunScript option so that the engine-backup dumps its output someplace >> where Bacula will collect it once engine-backup finishes. However the >> job is failing and with enough digging I eventually learned the script >> was writing the following in /tmp/hs_err_pid5789.log: >> >> # >> # There is insufficient memory for the Java Runtime Environment to continue. >> # Native memory allocation (mmap) failed to map 2555904 bytes for >> committing reserved memory. >> # Possible reasons: >> # The system is out of physical RAM or swap space >> # In 32 bit mode, the process size limit was hit >> # Possible solutions: >> # Reduce memory load on the system >> # Increase physical memory or swap space >> # Check if swap backing store is full >> # Use 64 bit Java on a 64 bit OS >> # Decrease Java heap size (-Xmx/-Xms) >> # Decrease number of Java threads >> # Decrease Java thread stack sizes (-Xss) >> # Set larger code cache with -XX:ReservedCodeCacheSize= >> # This output file may be truncated or incomplete. >> # >> # Out of Memory Error (os_linux.cpp:2627), pid=5789, tid=140709998221056 >> # >> # JRE version: (8.0_65-b17) (build ) >> # Java VM: OpenJDK 64-Bit Server VM (25.65-b01 mixed mode linux-amd64 >> compressed oops) >> # Failed to write core dump. Core dumps have been disabled. To enable >> core dumping, try "ulimit -c unlimited" before starting Java again >> # >> >> >> So is there any good way to reduce the Java heap size? I mean I know >> what -Xmx does, but where might I try setting it, ideally so that it >> affects the engine-backup only? Any idea of good setting for a very >> small environment with a dozen VMs? > engine-backup does not directly call nor need java. > > AFAICS it only calls it indirectly as part of some other initialization > by running java-home [1], which is a script that decides what JAVA_HOME > to use for the engine. This script only runs 'java -version', which imo > should not need that much memory. Perhaps there is something else I do > not fully understand, such as bacula severely limiting available resources > for the process it runs, or something like that. > > If you only want to debug it, and not as a recommended final solution, > you can create a script [2] which only outputs the needed java home. > Simply run [1] and make [2] echo the same thing. If [2] exists, [1] will > only run it and nothing else, as you can see inside it. > > I do not think this will work - quite likely engine-backup will fail > shortly later, if indeed it gets access to so little memory. Please > report back. Thanks and good luck, > > [1] /usr/share/ovirt-engine/bin/java-home > [2] /usr/share/ovirt-engine/bin/java-home.local Thanks for the info and response Didi. Doing the above did allow the backup to run successfully. I had also replaced the Bacula RunScript with "bash -c ulimit" which reported unlimited but I don't play with those types of limits enough to know if that's correctly reporting to what engine-backup is constrained. I did occur to me that perhaps a better way to learn of any such constraints would be to query Bacula's file daemon (the only necessary Bacula component running on client systems that are getting backed up) since I suspect it must be this component that's actually spawning the RunScript client side. From the Bacula Director (server side) I queried the status of the client which is my oVirt engine and it reports: europa.doubledog.org-fd Version: 5.2.13 (19 February 2013) x86_64-redhat-linux-gnu redhat (Core) Daemon started 28-Dec-15 16:08. Jobs: run=2 running=0. Heap: heap=32,768 smbytes=190,247 max_bytes=1,599,864 bufs=100 max_bufs=6,758 Sizeof: boffset_t=8 size_t=8 debug=0 trace=0 Alas, I know of no way to increase any of the bacula-fd limits. If I dead-end here, perhaps I'll query the Bacula mailing lists. Meanwhile I tried the following for a more permanent solution but this failed same as before: # diff -u java-home.orig-3.6.1.3 java-home --- java-home.orig-3.6.1.3 2015-12-10 13:07:44.0 -0500 +++ java-home 2015-12-30 12:12:45.779462769 -0500 @@ -13,7 +13,7 @@ local ret=1 if [ -x "${dir}/bin/java" ]; then - local version="$("${dir}/bin/java" -version 2>&1 | sed \ + local version="$("${dir}/bin/java" -Xmx 8 -version 2>&1 | sed \ -e 's/^openjdk version "1\.8\.0.*/VERSION_OK/' \ -e 's/^java version "1\.7\.0.*/VERSION_OK/' \ -e 's/^OpenJDK .*(.*).*/VENDOR_OK/' \ If this script is merely checking the validity of the JRE/JDK, should it not be possible to have a test on the rpm details first and only proceed as it does now if that doesn't work? The current tests should work w/o much regard for how the JRE/JDK got installed, but if it wa
Re: [ovirt-users] Can I reduce the Java heap size of engine-backup???
On Tue, Dec 29, 2015 at 12:51 AM, John Florian wrote: > I'm trying to run the engine-backup script via a Bacula job using the > RunScript option so that the engine-backup dumps its output someplace > where Bacula will collect it once engine-backup finishes. However the > job is failing and with enough digging I eventually learned the script > was writing the following in /tmp/hs_err_pid5789.log: > > # > # There is insufficient memory for the Java Runtime Environment to continue. > # Native memory allocation (mmap) failed to map 2555904 bytes for > committing reserved memory. > # Possible reasons: > # The system is out of physical RAM or swap space > # In 32 bit mode, the process size limit was hit > # Possible solutions: > # Reduce memory load on the system > # Increase physical memory or swap space > # Check if swap backing store is full > # Use 64 bit Java on a 64 bit OS > # Decrease Java heap size (-Xmx/-Xms) > # Decrease number of Java threads > # Decrease Java thread stack sizes (-Xss) > # Set larger code cache with -XX:ReservedCodeCacheSize= > # This output file may be truncated or incomplete. > # > # Out of Memory Error (os_linux.cpp:2627), pid=5789, tid=140709998221056 > # > # JRE version: (8.0_65-b17) (build ) > # Java VM: OpenJDK 64-Bit Server VM (25.65-b01 mixed mode linux-amd64 > compressed oops) > # Failed to write core dump. Core dumps have been disabled. To enable > core dumping, try "ulimit -c unlimited" before starting Java again > # > > > So is there any good way to reduce the Java heap size? I mean I know > what -Xmx does, but where might I try setting it, ideally so that it > affects the engine-backup only? Any idea of good setting for a very > small environment with a dozen VMs? engine-backup does not directly call nor need java. AFAICS it only calls it indirectly as part of some other initialization by running java-home [1], which is a script that decides what JAVA_HOME to use for the engine. This script only runs 'java -version', which imo should not need that much memory. Perhaps there is something else I do not fully understand, such as bacula severely limiting available resources for the process it runs, or something like that. If you only want to debug it, and not as a recommended final solution, you can create a script [2] which only outputs the needed java home. Simply run [1] and make [2] echo the same thing. If [2] exists, [1] will only run it and nothing else, as you can see inside it. I do not think this will work - quite likely engine-backup will fail shortly later, if indeed it gets access to so little memory. Please report back. Thanks and good luck, [1] /usr/share/ovirt-engine/bin/java-home [2] /usr/share/ovirt-engine/bin/java-home.local -- Didi ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] Can I reduce the Java heap size of engine-backup???
I'm trying to run the engine-backup script via a Bacula job using the RunScript option so that the engine-backup dumps its output someplace where Bacula will collect it once engine-backup finishes. However the job is failing and with enough digging I eventually learned the script was writing the following in /tmp/hs_err_pid5789.log: # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 2555904 bytes for committing reserved memory. # Possible reasons: # The system is out of physical RAM or swap space # In 32 bit mode, the process size limit was hit # Possible solutions: # Reduce memory load on the system # Increase physical memory or swap space # Check if swap backing store is full # Use 64 bit Java on a 64 bit OS # Decrease Java heap size (-Xmx/-Xms) # Decrease number of Java threads # Decrease Java thread stack sizes (-Xss) # Set larger code cache with -XX:ReservedCodeCacheSize= # This output file may be truncated or incomplete. # # Out of Memory Error (os_linux.cpp:2627), pid=5789, tid=140709998221056 # # JRE version: (8.0_65-b17) (build ) # Java VM: OpenJDK 64-Bit Server VM (25.65-b01 mixed mode linux-amd64 compressed oops) # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # So is there any good way to reduce the Java heap size? I mean I know what -Xmx does, but where might I try setting it, ideally so that it affects the engine-backup only? Any idea of good setting for a very small environment with a dozen VMs? -- John Florian ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users