Re: [ovirt-users] Can I reduce the Java heap size of engine-backup???

[John Florian]

On 12/31/2015 10:42 AM, Juan Hernández wrote:
> On 12/31/2015 08:48 AM, Yedidyah Bar David wrote:
>> On Wed, Dec 30, 2015 at 7:50 PM, John Florian  wrote:
>>> On 12/29/2015 02:02 AM, Yedidyah Bar David wrote:
 On Tue, Dec 29, 2015 at 12:51 AM, John Florian  
 wrote:
> I'm trying to run the engine-backup script via a Bacula job using the
> RunScript option so that the engine-backup dumps its output someplace
> where Bacula will collect it once engine-backup finishes.  However the
> job is failing and with enough digging I eventually learned the script
> was writing the following in /tmp/hs_err_pid5789.log:
>
> #
> # There is insufficient memory for the Java Runtime Environment to 
> continue.
> # Native memory allocation (mmap) failed to map 2555904 bytes for
> committing reserved memory.
> # Possible reasons:
> #   The system is out of physical RAM or swap space
> #   In 32 bit mode, the process size limit was hit
> # Possible solutions:
> #   Reduce memory load on the system
> #   Increase physical memory or swap space
> #   Check if swap backing store is full
> #   Use 64 bit Java on a 64 bit OS
> #   Decrease Java heap size (-Xmx/-Xms)
> #   Decrease number of Java threads
> #   Decrease Java thread stack sizes (-Xss)
> #   Set larger code cache with -XX:ReservedCodeCacheSize=
> # This output file may be truncated or incomplete.
> #
> #  Out of Memory Error (os_linux.cpp:2627), pid=5789, tid=140709998221056
> #
> # JRE version:  (8.0_65-b17) (build )
> # Java VM: OpenJDK 64-Bit Server VM (25.65-b01 mixed mode linux-amd64
> compressed oops)
> # Failed to write core dump. Core dumps have been disabled. To enable
> core dumping, try "ulimit -c unlimited" before starting Java again
> #
>
>
> So is there any good way to reduce the Java heap size?  I mean I know
> what -Xmx does, but where might I try setting it, ideally so that it
> affects the engine-backup only?  Any idea of good setting for a very
> small environment with a dozen VMs?
 engine-backup does not directly call nor need java.

 AFAICS it only calls it indirectly as part of some other initialization
 by running java-home [1], which is a script that decides what JAVA_HOME
 to use for the engine. This script only runs 'java -version', which imo
 should not need that much memory. Perhaps there is something else I do
 not fully understand, such as bacula severely limiting available resources
 for the process it runs, or something like that.

 If you only want to debug it, and not as a recommended final solution,
 you can create a script [2] which only outputs the needed java home.
 Simply run [1] and make [2] echo the same thing. If [2] exists, [1] will
 only run it and nothing else, as you can see inside it.

 I do not think this will work - quite likely engine-backup will fail
 shortly later, if indeed it gets access to so little memory. Please
 report back. Thanks and good luck,

 [1] /usr/share/ovirt-engine/bin/java-home
 [2] /usr/share/ovirt-engine/bin/java-home.local
>>> Thanks for the info and response Didi.  Doing the above did allow the
>>> backup to run successfully.
>> OK.
>>
>>>  I had also replaced the Bacula RunScript
>>> with "bash -c ulimit" which reported unlimited but I don't play with
>>> those types of limits enough to know if that's correctly reporting to
>>> what engine-backup is constrained.
>> And was this enough?
>>
>>>  I did occur to me that perhaps a
>>> better way to learn of any such constraints would be to query Bacula's
>>> file daemon (the only necessary Bacula component running on client
>>> systems that are getting backed up) since I suspect it must be this
>>> component that's actually spawning the RunScript client side.  From the
>>> Bacula Director (server side) I queried the status of the client which
>>> is my oVirt engine and it reports:
>>>
>>> europa.doubledog.org-fd Version: 5.2.13 (19 February 2013)
>>> x86_64-redhat-linux-gnu redhat (Core)
>>> Daemon started 28-Dec-15 16:08. Jobs: run=2 running=0.
>>>  Heap: heap=32,768 smbytes=190,247 max_bytes=1,599,864 bufs=100
>>> max_bufs=6,758
>>>  Sizeof: boffset_t=8 size_t=8 debug=0 trace=0
>>>
>>> Alas, I know of no way to increase any of the bacula-fd limits.  If I
>>> dead-end here, perhaps I'll query the Bacula mailing lists.
>> For both yourself and for others, I think it's best to continue with
>> this route.
>>
>> Also note that I have no idea how much memory pg_dump might need on
>> a larger database, also including dwh which tends to get larger faster
>> than the engine's.
>>
>>> Meanwhile I tried the following for a more permanent solution but this
>>> failed same as before:
>>>
>>> # diff -u java-home.orig-3.6.1.3 java-home
>>> --- java-home.orig-3.6.1.3  

Re: [ovirt-users] Can I reduce the Java heap size of engine-backup???

[Juan Hernández]

On 12/31/2015 08:48 AM, Yedidyah Bar David wrote:
> On Wed, Dec 30, 2015 at 7:50 PM, John Florian  wrote:
>> On 12/29/2015 02:02 AM, Yedidyah Bar David wrote:
>>> On Tue, Dec 29, 2015 at 12:51 AM, John Florian  
>>> wrote:
 I'm trying to run the engine-backup script via a Bacula job using the
 RunScript option so that the engine-backup dumps its output someplace
 where Bacula will collect it once engine-backup finishes.  However the
 job is failing and with enough digging I eventually learned the script
 was writing the following in /tmp/hs_err_pid5789.log:

 #
 # There is insufficient memory for the Java Runtime Environment to 
 continue.
 # Native memory allocation (mmap) failed to map 2555904 bytes for
 committing reserved memory.
 # Possible reasons:
 #   The system is out of physical RAM or swap space
 #   In 32 bit mode, the process size limit was hit
 # Possible solutions:
 #   Reduce memory load on the system
 #   Increase physical memory or swap space
 #   Check if swap backing store is full
 #   Use 64 bit Java on a 64 bit OS
 #   Decrease Java heap size (-Xmx/-Xms)
 #   Decrease number of Java threads
 #   Decrease Java thread stack sizes (-Xss)
 #   Set larger code cache with -XX:ReservedCodeCacheSize=
 # This output file may be truncated or incomplete.
 #
 #  Out of Memory Error (os_linux.cpp:2627), pid=5789, tid=140709998221056
 #
 # JRE version:  (8.0_65-b17) (build )
 # Java VM: OpenJDK 64-Bit Server VM (25.65-b01 mixed mode linux-amd64
 compressed oops)
 # Failed to write core dump. Core dumps have been disabled. To enable
 core dumping, try "ulimit -c unlimited" before starting Java again
 #


 So is there any good way to reduce the Java heap size?  I mean I know
 what -Xmx does, but where might I try setting it, ideally so that it
 affects the engine-backup only?  Any idea of good setting for a very
 small environment with a dozen VMs?
>>> engine-backup does not directly call nor need java.
>>>
>>> AFAICS it only calls it indirectly as part of some other initialization
>>> by running java-home [1], which is a script that decides what JAVA_HOME
>>> to use for the engine. This script only runs 'java -version', which imo
>>> should not need that much memory. Perhaps there is something else I do
>>> not fully understand, such as bacula severely limiting available resources
>>> for the process it runs, or something like that.
>>>
>>> If you only want to debug it, and not as a recommended final solution,
>>> you can create a script [2] which only outputs the needed java home.
>>> Simply run [1] and make [2] echo the same thing. If [2] exists, [1] will
>>> only run it and nothing else, as you can see inside it.
>>>
>>> I do not think this will work - quite likely engine-backup will fail
>>> shortly later, if indeed it gets access to so little memory. Please
>>> report back. Thanks and good luck,
>>>
>>> [1] /usr/share/ovirt-engine/bin/java-home
>>> [2] /usr/share/ovirt-engine/bin/java-home.local
>> Thanks for the info and response Didi.  Doing the above did allow the
>> backup to run successfully.
> 
> OK.
> 
>>  I had also replaced the Bacula RunScript
>> with "bash -c ulimit" which reported unlimited but I don't play with
>> those types of limits enough to know if that's correctly reporting to
>> what engine-backup is constrained.
> 
> And was this enough?
> 
>>  I did occur to me that perhaps a
>> better way to learn of any such constraints would be to query Bacula's
>> file daemon (the only necessary Bacula component running on client
>> systems that are getting backed up) since I suspect it must be this
>> component that's actually spawning the RunScript client side.  From the
>> Bacula Director (server side) I queried the status of the client which
>> is my oVirt engine and it reports:
>>
>> europa.doubledog.org-fd Version: 5.2.13 (19 February 2013)
>> x86_64-redhat-linux-gnu redhat (Core)
>> Daemon started 28-Dec-15 16:08. Jobs: run=2 running=0.
>>  Heap: heap=32,768 smbytes=190,247 max_bytes=1,599,864 bufs=100
>> max_bufs=6,758
>>  Sizeof: boffset_t=8 size_t=8 debug=0 trace=0
>>
>> Alas, I know of no way to increase any of the bacula-fd limits.  If I
>> dead-end here, perhaps I'll query the Bacula mailing lists.
> 
> For both yourself and for others, I think it's best to continue with
> this route.
> 
> Also note that I have no idea how much memory pg_dump might need on
> a larger database, also including dwh which tends to get larger faster
> than the engine's.
> 
>>
>> Meanwhile I tried the following for a more permanent solution but this
>> failed same as before:
>>
>> # diff -u java-home.orig-3.6.1.3 java-home
>> --- java-home.orig-3.6.1.3  2015-12-10 13:07:44.0 -0500
>> +++ java-home   2015-12-30 12:12:45.779462769 -0500
>> @@ -13,7 +13,7 @@
>> local ret=1
>>

Re: [ovirt-users] Can I reduce the Java heap size of engine-backup???

[Yedidyah Bar David]

On Wed, Dec 30, 2015 at 7:50 PM, John Florian  wrote:
> On 12/29/2015 02:02 AM, Yedidyah Bar David wrote:
>> On Tue, Dec 29, 2015 at 12:51 AM, John Florian  
>> wrote:
>>> I'm trying to run the engine-backup script via a Bacula job using the
>>> RunScript option so that the engine-backup dumps its output someplace
>>> where Bacula will collect it once engine-backup finishes.  However the
>>> job is failing and with enough digging I eventually learned the script
>>> was writing the following in /tmp/hs_err_pid5789.log:
>>>
>>> #
>>> # There is insufficient memory for the Java Runtime Environment to continue.
>>> # Native memory allocation (mmap) failed to map 2555904 bytes for
>>> committing reserved memory.
>>> # Possible reasons:
>>> #   The system is out of physical RAM or swap space
>>> #   In 32 bit mode, the process size limit was hit
>>> # Possible solutions:
>>> #   Reduce memory load on the system
>>> #   Increase physical memory or swap space
>>> #   Check if swap backing store is full
>>> #   Use 64 bit Java on a 64 bit OS
>>> #   Decrease Java heap size (-Xmx/-Xms)
>>> #   Decrease number of Java threads
>>> #   Decrease Java thread stack sizes (-Xss)
>>> #   Set larger code cache with -XX:ReservedCodeCacheSize=
>>> # This output file may be truncated or incomplete.
>>> #
>>> #  Out of Memory Error (os_linux.cpp:2627), pid=5789, tid=140709998221056
>>> #
>>> # JRE version:  (8.0_65-b17) (build )
>>> # Java VM: OpenJDK 64-Bit Server VM (25.65-b01 mixed mode linux-amd64
>>> compressed oops)
>>> # Failed to write core dump. Core dumps have been disabled. To enable
>>> core dumping, try "ulimit -c unlimited" before starting Java again
>>> #
>>>
>>>
>>> So is there any good way to reduce the Java heap size?  I mean I know
>>> what -Xmx does, but where might I try setting it, ideally so that it
>>> affects the engine-backup only?  Any idea of good setting for a very
>>> small environment with a dozen VMs?
>> engine-backup does not directly call nor need java.
>>
>> AFAICS it only calls it indirectly as part of some other initialization
>> by running java-home [1], which is a script that decides what JAVA_HOME
>> to use for the engine. This script only runs 'java -version', which imo
>> should not need that much memory. Perhaps there is something else I do
>> not fully understand, such as bacula severely limiting available resources
>> for the process it runs, or something like that.
>>
>> If you only want to debug it, and not as a recommended final solution,
>> you can create a script [2] which only outputs the needed java home.
>> Simply run [1] and make [2] echo the same thing. If [2] exists, [1] will
>> only run it and nothing else, as you can see inside it.
>>
>> I do not think this will work - quite likely engine-backup will fail
>> shortly later, if indeed it gets access to so little memory. Please
>> report back. Thanks and good luck,
>>
>> [1] /usr/share/ovirt-engine/bin/java-home
>> [2] /usr/share/ovirt-engine/bin/java-home.local
> Thanks for the info and response Didi.  Doing the above did allow the
> backup to run successfully.

OK.

>  I had also replaced the Bacula RunScript
> with "bash -c ulimit" which reported unlimited but I don't play with
> those types of limits enough to know if that's correctly reporting to
> what engine-backup is constrained.

And was this enough?

>  I did occur to me that perhaps a
> better way to learn of any such constraints would be to query Bacula's
> file daemon (the only necessary Bacula component running on client
> systems that are getting backed up) since I suspect it must be this
> component that's actually spawning the RunScript client side.  From the
> Bacula Director (server side) I queried the status of the client which
> is my oVirt engine and it reports:
>
> europa.doubledog.org-fd Version: 5.2.13 (19 February 2013)
> x86_64-redhat-linux-gnu redhat (Core)
> Daemon started 28-Dec-15 16:08. Jobs: run=2 running=0.
>  Heap: heap=32,768 smbytes=190,247 max_bytes=1,599,864 bufs=100
> max_bufs=6,758
>  Sizeof: boffset_t=8 size_t=8 debug=0 trace=0
>
> Alas, I know of no way to increase any of the bacula-fd limits.  If I
> dead-end here, perhaps I'll query the Bacula mailing lists.

For both yourself and for others, I think it's best to continue with
this route.

Also note that I have no idea how much memory pg_dump might need on
a larger database, also including dwh which tends to get larger faster
than the engine's.

>
> Meanwhile I tried the following for a more permanent solution but this
> failed same as before:
>
> # diff -u java-home.orig-3.6.1.3 java-home
> --- java-home.orig-3.6.1.3  2015-12-10 13:07:44.0 -0500
> +++ java-home   2015-12-30 12:12:45.779462769 -0500
> @@ -13,7 +13,7 @@
> local ret=1
>
> if [ -x "${dir}/bin/java" ]; then
> -   local version="$("${dir}/bin/java" -version 2>&1 | sed \
> +   local version="$("${dir}/bin/java" -Xmx 8 

Re: [ovirt-users] Can I reduce the Java heap size of engine-backup???

[John Florian]

On 12/29/2015 02:02 AM, Yedidyah Bar David wrote:
> On Tue, Dec 29, 2015 at 12:51 AM, John Florian  wrote:
>> I'm trying to run the engine-backup script via a Bacula job using the
>> RunScript option so that the engine-backup dumps its output someplace
>> where Bacula will collect it once engine-backup finishes.  However the
>> job is failing and with enough digging I eventually learned the script
>> was writing the following in /tmp/hs_err_pid5789.log:
>>
>> #
>> # There is insufficient memory for the Java Runtime Environment to continue.
>> # Native memory allocation (mmap) failed to map 2555904 bytes for
>> committing reserved memory.
>> # Possible reasons:
>> #   The system is out of physical RAM or swap space
>> #   In 32 bit mode, the process size limit was hit
>> # Possible solutions:
>> #   Reduce memory load on the system
>> #   Increase physical memory or swap space
>> #   Check if swap backing store is full
>> #   Use 64 bit Java on a 64 bit OS
>> #   Decrease Java heap size (-Xmx/-Xms)
>> #   Decrease number of Java threads
>> #   Decrease Java thread stack sizes (-Xss)
>> #   Set larger code cache with -XX:ReservedCodeCacheSize=
>> # This output file may be truncated or incomplete.
>> #
>> #  Out of Memory Error (os_linux.cpp:2627), pid=5789, tid=140709998221056
>> #
>> # JRE version:  (8.0_65-b17) (build )
>> # Java VM: OpenJDK 64-Bit Server VM (25.65-b01 mixed mode linux-amd64
>> compressed oops)
>> # Failed to write core dump. Core dumps have been disabled. To enable
>> core dumping, try "ulimit -c unlimited" before starting Java again
>> #
>>
>>
>> So is there any good way to reduce the Java heap size?  I mean I know
>> what -Xmx does, but where might I try setting it, ideally so that it
>> affects the engine-backup only?  Any idea of good setting for a very
>> small environment with a dozen VMs?
> engine-backup does not directly call nor need java.
>
> AFAICS it only calls it indirectly as part of some other initialization
> by running java-home [1], which is a script that decides what JAVA_HOME
> to use for the engine. This script only runs 'java -version', which imo
> should not need that much memory. Perhaps there is something else I do
> not fully understand, such as bacula severely limiting available resources
> for the process it runs, or something like that.
>
> If you only want to debug it, and not as a recommended final solution,
> you can create a script [2] which only outputs the needed java home.
> Simply run [1] and make [2] echo the same thing. If [2] exists, [1] will
> only run it and nothing else, as you can see inside it.
>
> I do not think this will work - quite likely engine-backup will fail
> shortly later, if indeed it gets access to so little memory. Please
> report back. Thanks and good luck,
>
> [1] /usr/share/ovirt-engine/bin/java-home
> [2] /usr/share/ovirt-engine/bin/java-home.local
Thanks for the info and response Didi.  Doing the above did allow the
backup to run successfully.  I had also replaced the Bacula RunScript
with "bash -c ulimit" which reported unlimited but I don't play with
those types of limits enough to know if that's correctly reporting to
what engine-backup is constrained.  I did occur to me that perhaps a
better way to learn of any such constraints would be to query Bacula's
file daemon (the only necessary Bacula component running on client
systems that are getting backed up) since I suspect it must be this
component that's actually spawning the RunScript client side.  From the
Bacula Director (server side) I queried the status of the client which
is my oVirt engine and it reports:

europa.doubledog.org-fd Version: 5.2.13 (19 February 2013) 
x86_64-redhat-linux-gnu redhat (Core)
Daemon started 28-Dec-15 16:08. Jobs: run=2 running=0.
 Heap: heap=32,768 smbytes=190,247 max_bytes=1,599,864 bufs=100
max_bufs=6,758
 Sizeof: boffset_t=8 size_t=8 debug=0 trace=0

Alas, I know of no way to increase any of the bacula-fd limits.  If I
dead-end here, perhaps I'll query the Bacula mailing lists.

Meanwhile I tried the following for a more permanent solution but this
failed same as before:

# diff -u java-home.orig-3.6.1.3 java-home
--- java-home.orig-3.6.1.3  2015-12-10 13:07:44.0 -0500
+++ java-home   2015-12-30 12:12:45.779462769 -0500
@@ -13,7 +13,7 @@
local ret=1

if [ -x "${dir}/bin/java" ]; then
-   local version="$("${dir}/bin/java" -version 2>&1 | sed \
+   local version="$("${dir}/bin/java" -Xmx 8 -version 2>&1
| sed \
-e 's/^openjdk version "1\.8\.0.*/VERSION_OK/' \
-e 's/^java version "1\.7\.0.*/VERSION_OK/' \
-e 's/^OpenJDK .*(.*).*/VENDOR_OK/' \


If this script is merely checking the validity of  the JRE/JDK, should
it not be possible to have a test on the rpm details first and only
proceed as it does now if that doesn't work?  The current tests should
work w/o much regard for how the JRE/JDK 

Re: [ovirt-users] Can I reduce the Java heap size of engine-backup???

[Yedidyah Bar David]

On Tue, Dec 29, 2015 at 12:51 AM, John Florian  wrote:
> I'm trying to run the engine-backup script via a Bacula job using the
> RunScript option so that the engine-backup dumps its output someplace
> where Bacula will collect it once engine-backup finishes.  However the
> job is failing and with enough digging I eventually learned the script
> was writing the following in /tmp/hs_err_pid5789.log:
>
> #
> # There is insufficient memory for the Java Runtime Environment to continue.
> # Native memory allocation (mmap) failed to map 2555904 bytes for
> committing reserved memory.
> # Possible reasons:
> #   The system is out of physical RAM or swap space
> #   In 32 bit mode, the process size limit was hit
> # Possible solutions:
> #   Reduce memory load on the system
> #   Increase physical memory or swap space
> #   Check if swap backing store is full
> #   Use 64 bit Java on a 64 bit OS
> #   Decrease Java heap size (-Xmx/-Xms)
> #   Decrease number of Java threads
> #   Decrease Java thread stack sizes (-Xss)
> #   Set larger code cache with -XX:ReservedCodeCacheSize=
> # This output file may be truncated or incomplete.
> #
> #  Out of Memory Error (os_linux.cpp:2627), pid=5789, tid=140709998221056
> #
> # JRE version:  (8.0_65-b17) (build )
> # Java VM: OpenJDK 64-Bit Server VM (25.65-b01 mixed mode linux-amd64
> compressed oops)
> # Failed to write core dump. Core dumps have been disabled. To enable
> core dumping, try "ulimit -c unlimited" before starting Java again
> #
>
>
> So is there any good way to reduce the Java heap size?  I mean I know
> what -Xmx does, but where might I try setting it, ideally so that it
> affects the engine-backup only?  Any idea of good setting for a very
> small environment with a dozen VMs?

engine-backup does not directly call nor need java.

AFAICS it only calls it indirectly as part of some other initialization
by running java-home [1], which is a script that decides what JAVA_HOME
to use for the engine. This script only runs 'java -version', which imo
should not need that much memory. Perhaps there is something else I do
not fully understand, such as bacula severely limiting available resources
for the process it runs, or something like that.

If you only want to debug it, and not as a recommended final solution,
you can create a script [2] which only outputs the needed java home.
Simply run [1] and make [2] echo the same thing. If [2] exists, [1] will
only run it and nothing else, as you can see inside it.

I do not think this will work - quite likely engine-backup will fail
shortly later, if indeed it gets access to so little memory. Please
report back. Thanks and good luck,

[1] /usr/share/ovirt-engine/bin/java-home
[2] /usr/share/ovirt-engine/bin/java-home.local
-- 
Didi
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users