On Wed, May/21/2008 06:46:06PM, Pavel Shamis (Pasha) wrote: > I sent it directly to your email. Please check. > Thanks, > Pasha
Got it. Thanks. It's a PHP memory overload issue. (Apparently I didn't look far back enough in the httpd error_logs.) See below. > > Ethan Mallove wrote: > > On Wed, May/21/2008 05:19:44PM, Pavel Shamis (Pasha) wrote: > > > >> Jeff Squyres wrote: > >> > >>> Are we running into http max memory problems or http max upload size > >>> problems again? > >>> > >> I guess it is some server side issue, you need to check the > >> /var/log/httpd/* log on the server. > >> > > > > The only thing I found in the httpd logs > > (/var/log/httpd/www.open-mpi.org/error_log*) was this PHP > > warning, which I don't think would result in lost results: > > > > PHP Warning: array_shift(): The argument should be an array in > > .../submit/index.php on line 1683 > > > > I haven't received any emailed Postgres errors either. When > > were these results submitted? I searched for "mellanox" over > > the past four days. It seem the results aren't buried in > > here, because I see no test run failures ... > > > > http://www.open-mpi.org/mtt/index.php?do_redir=659 > > > > I'm assuming you're running with two Reporter INI sections: > > Textfile and MTTDatabase? Can you send some MTT client > > --verbose/--debug output from the below runs? > > > > Thanks, > > Ethan > > > > > >>> On May 21, 2008, at 5:28 AM, Pavel Shamis (Pasha) wrote: > >>> > >>> > >>>> Hi, > >>>> > >>>> Here is test result from my last mtt run: > >>>> +-------------+----------------+------+------+----------+------+ > >>>> | Phase | Section | Pass | Fail | Time out | Skip | > >>>> +-------------+----------------+------+------+----------+------+ > >>>> | MPI install | ompi/gcc | 1 | 0 | 0 | 0 | > >>>> | MPI install | ompi/intel-9.0 | 1 | 0 | 0 | 0 | > >>>> | Test Build | trivial | 1 | 0 | 0 | 0 | > >>>> | Test Build | trivial | 1 | 0 | 0 | 0 | > >>>> | Test Build | intel-suite | 1 | 0 | 0 | 0 | > >>>> | Test Build | intel-suite | 1 | 0 | 0 | 0 | > >>>> | Test Build | imb | 1 | 0 | 0 | 0 | > >>>> | Test Build | imb | 1 | 0 | 0 | 0 | > >>>> | Test Build | presta | 1 | 0 | 0 | 0 | > >>>> | Test Build | presta | 1 | 0 | 0 | 0 | > >>>> | Test Build | osu_benchmarks | 1 | 0 | 0 | 0 | > >>>> | Test Build | osu_benchmarks | 1 | 0 | 0 | 0 | > >>>> | Test Build | netpipe | 1 | 0 | 0 | 0 | > >>>> | Test Build | netpipe | 1 | 0 | 0 | 0 | > >>>> | Test Run | trivial | 64 | 0 | 0 | 0 | > >>>> | Test Run | trivial | 64 | 0 | 0 | 0 | > >>>> | Test Run | intel-suite | 3179 | 165 | 400 | 0 | > >>>> | Test Run | intel-suite | 492 | 0 | 0 | 0 | > >>>> +-------------+----------------+------+------+----------+------+ > >>>> > >>>> In the OMPI MTT DB (http://www.open-mpi.org/mtt) I found the follow > >>>> "test run" results: > >>>> | Test Run | trivial | 64 | 0 | 0 | 0 | > >>>> | Test Run | trivial | 64 | 0 | 0 | 0 | > >>>> | Test Run | intel-suite | 492 | 0 | 0 | 0 | > >>>> > >>>> And I can not find this one: > >>>> | Test Run | intel-suite | 3179 | 165 | 400 | 0 | Some missing results are in mttdb_debug_file.16.txt (and 17.txt), which are the largest .txt files of the bunch. 8 variants isn't that much, but maybe it causes a problem when there's lots of stderr/stdout? I'm surprised submit/index.php barfs on files this size: $ ls -l ... -rw-r--r-- 1 em162155 staff 956567 May 21 14:21 mttdb_debug_file.16.inc.gz -rw-r--r-- 1 em162155 staff 9603132 May 21 14:09 mttdb_debug_file.16.txt ... $ client/mtt-submit -h www.open-mpi.org -f mttdb_debug_file.16.txt -z -u sun -p sun4sun -d LWP::UserAgent::new: () LWP::UserAgent::proxy: http Filelist: $VAR1 = 'mttdb_debug_file.16.txt'; LWP::MediaTypes::read_media_types: Reading media types from /ws/ompi-tools/lib/perl5/5.8.8/LWP/media.types LWP::MediaTypes::read_media_types: Reading media types from /usr/perl5/site_perl/5.8.4/LWP/media.types LWP::MediaTypes::read_media_types: Reading media types from /home/em162155/.mime.types LWP::UserAgent::request: () LWP::UserAgent::send_request: POST http://www.open-mpi.org/mtt/submit/index.php LWP::UserAgent::_need_proxy: Not proxied LWP::Protocol::http::request: () LWP::UserAgent::request: Simple response: OK $ tail -f /var/log/httpd/www.open-mpi.org/error_log | grep -w submit ... [client 192.18.128.5] PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 14 bytes) in /nfs/rontok/xraid/data/osl/www/www.open-mpi.org/mtt/submit/index.php on line 1559 ... We'll have to somehow be more efficient on these loops. E.g., line 1559: foreach (array_keys($_POST) as $k) { Maybe if we broke $_POST up into multiple parts (e.g., $_POST_1, $_POST_2, ... $_POST_N)? Maybe we could do something more efficient than array_keys here? I'm not sure. The only workaround on the client side would be to break up the runs. Maybe do a single MPI Install at a time? Do ompi/gcc then ompi/intel-9.0 as seperate invocations of the MTT client. Sorry :-( -Ethan > >>>> > >>>> From the log I see that all tests results were submitted successfully. > >>>> > >>>> Can you please check ? > >>>> > >>>> Thanks, > >>>> > >>>> Pasha > >>>> _______________________________________________ > >>>> mtt-users mailing list > >>>> mtt-us...@open-mpi.org > >>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users > >>>> > >>> > >> _______________________________________________ > >> mtt-users mailing list > >> mtt-us...@open-mpi.org > >> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users > >> > > > > > > _______________________________________________ > mtt-users mailing list > mtt-us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
