Still surviving. Memory usage did not seem to increase above 891M on
the Walrus, even after I threw more instances on it.
I still do not really like the memory increase, but it seems we are good
enough to go. Great work.
--
walrus reports java.lang.OutOfMemoryError: Java heap space
hggdh kirkland: No OOMs. I ran about 4,000 instances against it, and
no OOM. mem usage on the Walrus still sounds high, but it survived
--
walrus reports java.lang.OutOfMemoryError: Java heap space
https://bugs.launchpad.net/bugs/565101
You received this bug notification because you are a member
** Description changed:
stress-testing, 400 instances to be started in total. After a while the
walrus starts reporting out-of-memory, and all new instances fail.
Ubuntu-server 2010-04-16, UEC 2010-04-16.
ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: eucalyptus-walrus
There was another place where memory was being leaked. This was
happening after a register requests, so that over time, after a number
of register requests, Walrus would OOM. The issue was with the
interaction between the CLC and Walrus.
This is fixed in revno 1227.
--
walrus reports
Thanks, Neil.
Carlos,
I have uploaded eucalyptus_1.6.2-0ubuntu31~ppa3_source.changes to the
~ubuntu-virt-ppa. Please give this another round of testing when your
GA-candidate tests are done. Once we can conclusively show that this
solves the OOM, we can apply for SRU approval.
--
walrus
starting the setup for topo2 (the most sensitive to the issue, all
components in different machines).
for the CLC:
ubu...@cempedak:~$ dpkg -l euca\* uec\*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
|/
First 1,000-instances run just finished. No OOMs, but:
- memory usage on the Walrus seems pretty constant around ~420M
- memory usage on the CLC is still increasing; at the end of the 1,000 runs it
was at 870M, suggesting the end is near.
I have saved the logs at lp:~hggdh2/+junk/uec-qa, under
Hum. Still up. After another 800 runs, CLC is using 438M, Walrus is
using 891M. Still trying to bring it down.
--
walrus reports java.lang.OutOfMemoryError: Java heap space
https://bugs.launchpad.net/bugs/565101
You received this bug notification because you are a member of Ubuntu
Server Team,
I just rab a sequence of 400 instances on topo2
(CLC;Walrus;CC;SC;NC-1;NC-2). Topo2 was has been, on my experience, the
easiest one to get the OOMs; usually I would see both CLC and Walrus
increasing memory in a similar way, and starting to spit out OOMs at
about the 750M of resident memory
Ran another 400-instances, without restarting Eucalyptus. Got OOMs on
the Walrus, but not on the CLC. For the record, here's the CLC
eucalyptus package details:
ubu...@cempedak:~/uec-testing-scripts$ dpkg -l euca\* uec\*
Desired=Unknown/Install/Remove/Purge/Hold
|
Logs were pushed to lp:~hggdh2/+junk/uec-qa, under
lucid/20100426.1/topo2-1.tar (the output of the test, plus the contents
of /var/log/* for each machine).
--
walrus reports java.lang.OutOfMemoryError: Java heap space
https://bugs.launchpad.net/bugs/565101
You received this bug notification
revno: 1225
committer: Neil Soman
branch nick: 1.6.2
timestamp: Sat 2010-04-24 12:09:26 -0700
message:
fixes LP: #565101
** Changed in: eucalyptus
Importance: Undecided
Yes, my opinion is it *will* use more heap memory when used under load
and that needs to be configured, no matter what.
Maybe there is a bug in eucalyptus code that makes it consume much more
memory than it should, and it should never be necessary to raise it over
the default heap size. Maybe
Thierry, some comments:
*Heap size*
0. The machines on the test rig all have 10G of main memory;
1. when we started (no -Xmx) we would get OutOfMemory when the eucalyptus (CLC,
Walrus, CC) resident memory -- as reported by 'top' was at about 800M;
2. When I tried -Xmx384m I observed OutOfMemory
Note from Dan on the weekly call...
He notes that Thierry's analysis of how heap size is determined by the
JVM doesn't apply to Eucalyptus, as they actually set it themselves,
rather than relying on the JVM to decide.
--
walrus reports java.lang.OutOfMemoryError: Java heap space
** Branch linked: lp:~ubuntu-core-dev/eucalyptus/ubuntu
--
walrus reports java.lang.OutOfMemoryError: Java heap space
https://bugs.launchpad.net/bugs/565101
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to eucalyptus in ubuntu.
--
Java programs have a bad habit of not being able to use the memory but
rather preselect an amount of memory usage at JVM startup, which makes
it difficult to have a sane default in packaging. It's the same for
Tomcat, where we explicitly give people the knobs to tune it after the
fact and
Yesterday night I bounced the whole rig (to start clean), and submitted
a 2,000 instances run. Run ended at 03:30 UTC. Right now it is about
09:10 UTC, and the Walrus shows 1,166M of resident memory.
I am not sure how the garbage collection works, but it really does not
look like it *is* working,
Sorry, times above are EDT, not UTC.
--
walrus reports java.lang.OutOfMemoryError: Java heap space
https://bugs.launchpad.net/bugs/565101
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to eucalyptus in ubuntu.
--
Ubuntu-server-bugs
Okay, so Thierry, the best way to solve this then is to set this value
to a sane default in a variable in the Upstart script, document it,
and point users to adjust that variable if they hit this barrier?
--
walrus reports java.lang.OutOfMemoryError: Java heap space
It sounds like a sane approach, yes. I have a feeling that this memory
creep may be related to errors starting instances, causing the related
Java-instantiated classes to be kept. I also do not remember seeing a
sequence of failures so big as what I am seeing now at or before Beta1.
--
walrus
** Changed in: eucalyptus (Ubuntu)
Status: New = In Progress
** Changed in: eucalyptus (Ubuntu)
Importance: Undecided = High
** Changed in: eucalyptus (Ubuntu)
Milestone: None = ubuntu-10.04
** Changed in: eucalyptus (Ubuntu)
Assignee: (unassigned) = Dustin Kirkland
The suggested fix is to up the jvm's memory to 384M in
/etc/init/eucalyptus.conf:
=== modified file 'debian/eucalyptus-common.eucalyptus.upstart'
--- debian/eucalyptus-common.eucalyptus.upstart 2010-02-04 01:00:15 +
+++ debian/eucalyptus-common.eucalyptus.upstart 2010-04-20 16:59:34 +
@@
Awaiting testing feedback from Carlos, before committing.
--
walrus reports java.lang.OutOfMemoryError: Java heap space
https://bugs.launchpad.net/bugs/565101
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to eucalyptus in ubuntu.
--
Got some more out-of-memory errors.
ubu...@mabolo:/var/log/eucalyptus$ ps aux | grep euca
root 1023 0.0 0.0 16828 1052 ?Ss Apr19 0:00 avahi-publish
-s Walrus _eucalyptus._tcp 8773 txtvers=1 protovers=1.5.0 type=walrus
ipaddr=10.55.55.5
root 14733 0.0 0.0 16688 928
** Attachment added: CLC logs
http://launchpadlibrarian.net/44916919/clc-logs.tar.gz
--
walrus reports java.lang.OutOfMemoryError: Java heap space
https://bugs.launchpad.net/bugs/565101
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to
Copying this bug upstream, as the simple configuration change didn't
solve the problem for us.
** Also affects: eucalyptus
Importance: Undecided
Status: New
--
walrus reports java.lang.OutOfMemoryError: Java heap space
https://bugs.launchpad.net/bugs/565101
You received this bug
running with -Xmx1024m seems to have reached a steady-state at about
930M of RES. At least with our rig I cannot open sessions faster enough
to cause more increases.
--
walrus reports java.lang.OutOfMemoryError: Java heap space
https://bugs.launchpad.net/bugs/565101
You received this bug
Good information, Carlos.
So perhaps upping our default JVM heap to 1024M would alleviate some
of the pain, but perhaps not heal the wound.
Let's see what upstream Eucalyptus has to say...
** Changed in: eucalyptus (Ubuntu Lucid)
Milestone: ubuntu-10.04 = lucid-updates
--
walrus reports
** Attachment added: Dependencies.txt
http://launchpadlibrarian.net/44492702/Dependencies.txt
** Attachment added: EucalyptusCloudDebugLog.gz
http://launchpadlibrarian.net/44492703/EucalyptusCloudDebugLog.gz
** Attachment added: EucalyptusCloudOutputLog.gz
30 matches
Mail list logo
