On Wed, May/21/2008 09:53:11PM, Pavel Shamis (Pasha) wrote: > Oops, in the "MTT server side problem" we discussed other issue. > > But anyway I did not see the problem on my server after the upgrade :)
We took *some* steps to alleviate the PHP memory overload problem (e.g., r668, and then r1119), but evidently there's more work to do :-) > > Pasha > > Pavel Shamis (Pasha) wrote: >> I had similar problem on my server. I upgraded the server to latest trunk >> and the problem disappear. >> (see "MTT server side problem" thread). >> >> Pasha >> >> Jeff Squyres wrote: >> >>> FWIW: I think we have at least one open ticket on this issue (break up >>> submits so that we don't overflow PHP and/or apache). https://svn.open-mpi.org/trac/mtt/ticket/221 -Ethan >>> >>> On May 21, 2008, at 2:36 PM, Ethan Mallove wrote: >>> >>> >>>> On Wed, May/21/2008 06:46:06PM, Pavel Shamis (Pasha) wrote: >>>> >>>>> I sent it directly to your email. Please check. >>>>> Thanks, >>>>> Pasha >>>>> >>>> Got it. Thanks. It's a PHP memory overload issue. >>>> (Apparently I didn't look far back enough in the httpd >>>> error_logs.) See below. >>>> >>>> >>>>> Ethan Mallove wrote: >>>>> >>>>>> On Wed, May/21/2008 05:19:44PM, Pavel Shamis (Pasha) wrote: >>>>>> >>>>>> >>>>>>> Jeff Squyres wrote: >>>>>>> >>>>>>> >>>>>>>> Are we running into http max memory problems or http max upload size >>>>>>>> problems again? >>>>>>>> >>>>>>>> >>>>>>> I guess it is some server side issue, you need to check the >>>>>>> /var/log/httpd/* log on the server. >>>>>>> >>>>>>> >>>>>> The only thing I found in the httpd logs >>>>>> (/var/log/httpd/www.open-mpi.org/error_log*) was this PHP >>>>>> warning, which I don't think would result in lost results: >>>>>> >>>>>> PHP Warning: array_shift(): The argument should be an array in >>>>>> .../submit/index.php on line 1683 >>>>>> >>>>>> I haven't received any emailed Postgres errors either. When >>>>>> were these results submitted? I searched for "mellanox" over >>>>>> the past four days. It seem the results aren't buried in >>>>>> here, because I see no test run failures ... >>>>>> >>>>>> http://www.open-mpi.org/mtt/index.php?do_redir=659 >>>>>> >>>>>> I'm assuming you're running with two Reporter INI sections: >>>>>> Textfile and MTTDatabase? Can you send some MTT client >>>>>> --verbose/--debug output from the below runs? >>>>>> >>>>>> Thanks, >>>>>> Ethan >>>>>> >>>>>> >>>>>> >>>>>>>> On May 21, 2008, at 5:28 AM, Pavel Shamis (Pasha) wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> Here is test result from my last mtt run: >>>>>>>>> +-------------+----------------+------+------+----------+------+ >>>>>>>>> | Phase | Section | Pass | Fail | Time out | Skip | >>>>>>>>> +-------------+----------------+------+------+----------+------+ >>>>>>>>> | MPI install | ompi/gcc | 1 | 0 | 0 | 0 | >>>>>>>>> | MPI install | ompi/intel-9.0 | 1 | 0 | 0 | 0 | >>>>>>>>> | Test Build | trivial | 1 | 0 | 0 | 0 | >>>>>>>>> | Test Build | trivial | 1 | 0 | 0 | 0 | >>>>>>>>> | Test Build | intel-suite | 1 | 0 | 0 | 0 | >>>>>>>>> | Test Build | intel-suite | 1 | 0 | 0 | 0 | >>>>>>>>> | Test Build | imb | 1 | 0 | 0 | 0 | >>>>>>>>> | Test Build | imb | 1 | 0 | 0 | 0 | >>>>>>>>> | Test Build | presta | 1 | 0 | 0 | 0 | >>>>>>>>> | Test Build | presta | 1 | 0 | 0 | 0 | >>>>>>>>> | Test Build | osu_benchmarks | 1 | 0 | 0 | 0 | >>>>>>>>> | Test Build | osu_benchmarks | 1 | 0 | 0 | 0 | >>>>>>>>> | Test Build | netpipe | 1 | 0 | 0 | 0 | >>>>>>>>> | Test Build | netpipe | 1 | 0 | 0 | 0 | >>>>>>>>> | Test Run | trivial | 64 | 0 | 0 | 0 | >>>>>>>>> | Test Run | trivial | 64 | 0 | 0 | 0 | >>>>>>>>> | Test Run | intel-suite | 3179 | 165 | 400 | 0 | >>>>>>>>> | Test Run | intel-suite | 492 | 0 | 0 | 0 | >>>>>>>>> +-------------+----------------+------+------+----------+------+ >>>>>>>>> >>>>>>>>> In the OMPI MTT DB (http://www.open-mpi.org/mtt) I found the follow >>>>>>>>> "test run" results: >>>>>>>>> | Test Run | trivial | 64 | 0 | 0 | 0 | >>>>>>>>> | Test Run | trivial | 64 | 0 | 0 | 0 | >>>>>>>>> | Test Run | intel-suite | 492 | 0 | 0 | 0 | >>>>>>>>> >>>>>>>>> And I can not find this one: >>>>>>>>> | Test Run | intel-suite | 3179 | 165 | 400 | 0 | >>>>>>>>> >>>> Some missing results are in mttdb_debug_file.16.txt (and >>>> 17.txt), which are the largest .txt files of the bunch. 8 >>>> variants isn't that much, but maybe it causes a problem when >>>> there's lots of stderr/stdout? I'm surprised >>>> submit/index.php barfs on files this size: >>>> >>>> $ ls -l >>>> ... >>>> -rw-r--r-- 1 em162155 staff 956567 May 21 14:21 >>>> mttdb_debug_file.16.inc.gz >>>> -rw-r--r-- 1 em162155 staff 9603132 May 21 14:09 >>>> mttdb_debug_file.16.txt >>>> ... >>>> >>>> $ client/mtt-submit -h www.open-mpi.org -f mttdb_debug_file.16.txt -z >>>> -u sun -p sun4sun -d >>>> LWP::UserAgent::new: () >>>> LWP::UserAgent::proxy: http >>>> >>>> Filelist: $VAR1 = 'mttdb_debug_file.16.txt'; >>>> LWP::MediaTypes::read_media_types: Reading media types from >>>> /ws/ompi-tools/lib/perl5/5.8.8/LWP/media.types >>>> LWP::MediaTypes::read_media_types: Reading media types from >>>> /usr/perl5/site_perl/5.8.4/LWP/media.types >>>> LWP::MediaTypes::read_media_types: Reading media types from >>>> /home/em162155/.mime.types >>>> LWP::UserAgent::request: () >>>> LWP::UserAgent::send_request: POST >>>> http://www.open-mpi.org/mtt/submit/index.php >>>> LWP::UserAgent::_need_proxy: Not proxied >>>> LWP::Protocol::http::request: () >>>> LWP::UserAgent::request: Simple response: OK >>>> >>>> $ tail -f /var/log/httpd/www.open-mpi.org/error_log | grep -w submit >>>> ... >>>> [client 192.18.128.5] PHP Fatal error: Allowed memory size of 33554432 >>>> bytes exhausted (tried to allocate 14 bytes) in >>>> /nfs/rontok/xraid/data/osl/www/www.open-mpi.org/mtt/submit/index.php on >>>> line 1559 >>>> ... >>>> >>>> We'll have to somehow be more efficient on these loops. >>>> E.g., line 1559: >>>> >>>> foreach (array_keys($_POST) as $k) { >>>> >>>> Maybe if we broke $_POST up into multiple parts (e.g., >>>> $_POST_1, $_POST_2, ... $_POST_N)? Maybe we could do >>>> something more efficient than array_keys here? I'm not sure. >>>> >>>> The only workaround on the client side would be to break up >>>> the runs. Maybe do a single MPI Install at a time? Do >>>> ompi/gcc then ompi/intel-9.0 as seperate invocations of the >>>> MTT client. >>>> >>>> Sorry :-( >>>> >>>> -Ethan >>>> >>>> >>>> >>>>>>>>> From the log I see that all tests results were submitted >>>>>>>>> successfully. >>>>>>>>> >>>>>>>>> Can you please check ? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Pasha >>>>>>>>> _______________________________________________ >>>>>>>>> mtt-users mailing list >>>>>>>>> mtt-us...@open-mpi.org >>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users >>>>>>>>> >>>>>>>>> >>>>>>> _______________________________________________ >>>>>>> mtt-users mailing list >>>>>>> mtt-us...@open-mpi.org >>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users >>>>>>> >>>>>>> >>>>>> >>>>> _______________________________________________ >>>>> mtt-users mailing list >>>>> mtt-us...@open-mpi.org >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users >>>>> >>> >> >> _______________________________________________ >> mtt-users mailing list >> mtt-us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users >> >> >
