FWIW: I think we have at least one open ticket on this issue (break up
submits so that we don't overflow PHP and/or apache).
On May 21, 2008, at 2:36 PM, Ethan Mallove wrote:
On Wed, May/21/2008 06:46:06PM, Pavel Shamis (Pasha) wrote:
I sent it directly to your email. Please check.
Thanks,
Pasha
Got it. Thanks. It's a PHP memory overload issue.
(Apparently I didn't look far back enough in the httpd
error_logs.) See below.
Ethan Mallove wrote:
On Wed, May/21/2008 05:19:44PM, Pavel Shamis (Pasha) wrote:
Jeff Squyres wrote:
Are we running into http max memory problems or http max upload
size
problems again?
I guess it is some server side issue, you need to check the
/var/log/httpd/* log on the server.
The only thing I found in the httpd logs
(/var/log/httpd/www.open-mpi.org/error_log*) was this PHP
warning, which I don't think would result in lost results:
PHP Warning: array_shift(): The argument should be an array
in .../submit/index.php on line 1683
I haven't received any emailed Postgres errors either. When
were these results submitted? I searched for "mellanox" over
the past four days. It seem the results aren't buried in
here, because I see no test run failures ...
http://www.open-mpi.org/mtt/index.php?do_redir=659
I'm assuming you're running with two Reporter INI sections:
Textfile and MTTDatabase? Can you send some MTT client
--verbose/--debug output from the below runs?
Thanks,
Ethan
On May 21, 2008, at 5:28 AM, Pavel Shamis (Pasha) wrote:
Hi,
Here is test result from my last mtt run:
+-------------+----------------+------+------+----------+------+
| Phase | Section | Pass | Fail | Time out | Skip |
+-------------+----------------+------+------+----------+------+
| MPI install | ompi/gcc | 1 | 0 | 0 | 0 |
| MPI install | ompi/intel-9.0 | 1 | 0 | 0 | 0 |
| Test Build | trivial | 1 | 0 | 0 | 0 |
| Test Build | trivial | 1 | 0 | 0 | 0 |
| Test Build | intel-suite | 1 | 0 | 0 | 0 |
| Test Build | intel-suite | 1 | 0 | 0 | 0 |
| Test Build | imb | 1 | 0 | 0 | 0 |
| Test Build | imb | 1 | 0 | 0 | 0 |
| Test Build | presta | 1 | 0 | 0 | 0 |
| Test Build | presta | 1 | 0 | 0 | 0 |
| Test Build | osu_benchmarks | 1 | 0 | 0 | 0 |
| Test Build | osu_benchmarks | 1 | 0 | 0 | 0 |
| Test Build | netpipe | 1 | 0 | 0 | 0 |
| Test Build | netpipe | 1 | 0 | 0 | 0 |
| Test Run | trivial | 64 | 0 | 0 | 0 |
| Test Run | trivial | 64 | 0 | 0 | 0 |
| Test Run | intel-suite | 3179 | 165 | 400 | 0 |
| Test Run | intel-suite | 492 | 0 | 0 | 0 |
+-------------+----------------+------+------+----------+------+
In the OMPI MTT DB (http://www.open-mpi.org/mtt) I found the
follow
"test run" results:
| Test Run | trivial | 64 | 0 | 0 | 0 |
| Test Run | trivial | 64 | 0 | 0 | 0 |
| Test Run | intel-suite | 492 | 0 | 0 | 0 |
And I can not find this one:
| Test Run | intel-suite | 3179 | 165 | 400 | 0 |
Some missing results are in mttdb_debug_file.16.txt (and
17.txt), which are the largest .txt files of the bunch. 8
variants isn't that much, but maybe it causes a problem when
there's lots of stderr/stdout? I'm surprised
submit/index.php barfs on files this size:
$ ls -l
...
-rw-r--r-- 1 em162155 staff 956567 May 21 14:21
mttdb_debug_file.16.inc.gz
-rw-r--r-- 1 em162155 staff 9603132 May 21 14:09
mttdb_debug_file.16.txt
...
$ client/mtt-submit -h www.open-mpi.org -f mttdb_debug_file.16.txt -
z -u sun -p sun4sun -d
LWP::UserAgent::new: ()
LWP::UserAgent::proxy: http
Filelist: $VAR1 = 'mttdb_debug_file.16.txt';
LWP::MediaTypes::read_media_types: Reading media types from /ws/
ompi-tools/lib/perl5/5.8.8/LWP/media.types
LWP::MediaTypes::read_media_types: Reading media types from /usr/
perl5/site_perl/5.8.4/LWP/media.types
LWP::MediaTypes::read_media_types: Reading media types from /home/
em162155/.mime.types
LWP::UserAgent::request: ()
LWP::UserAgent::send_request: POST http://www.open-mpi.org/mtt/submit/index.php
LWP::UserAgent::_need_proxy: Not proxied
LWP::Protocol::http::request: ()
LWP::UserAgent::request: Simple response: OK
$ tail -f /var/log/httpd/www.open-mpi.org/error_log | grep -w submit
...
[client 192.18.128.5] PHP Fatal error: Allowed memory size of
33554432 bytes exhausted (tried to allocate 14 bytes) in /nfs/rontok/
xraid/data/osl/www/www.open-mpi.org/mtt/submit/index.php on line 1559
...
We'll have to somehow be more efficient on these loops.
E.g., line 1559:
foreach (array_keys($_POST) as $k) {
Maybe if we broke $_POST up into multiple parts (e.g.,
$_POST_1, $_POST_2, ... $_POST_N)? Maybe we could do
something more efficient than array_keys here? I'm not sure.
The only workaround on the client side would be to break up
the runs. Maybe do a single MPI Install at a time? Do
ompi/gcc then ompi/intel-9.0 as seperate invocations of the
MTT client.
Sorry :-(
-Ethan
From the log I see that all tests results were submitted
successfully.
Can you please check ?
Thanks,
Pasha
_______________________________________________
mtt-users mailing list
mtt-us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
_______________________________________________
mtt-users mailing list
mtt-us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
_______________________________________________
mtt-users mailing list
mtt-us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
--
Jeff Squyres
Cisco Systems
