Hi Josh,
I ported the error handling mechanism from submit/index.php to to the
database.inc. Please review.
Thanks,
Pasha
Josh Hursey wrote:
Pasha,
I'm looking at the patch a bit closer and even though at a high level
the do_pg_connect, do_pg_query, simple_select, and select functions do
the same thing the versions in submit/index.php have some additional
error handling mechanisms that the ones in database.inc do not have.
Specifically they send email when the functions fail with messages
indicating what failed so corrections can be made.
So though I agree that we should unify the functionality I cannot
recommend this patch since it will result in losing useful error
handling functionality. Maybe there is another way to clean this up to
preserve the error reporting.
-- Josh
On May 7, 2008, at 11:56 AM, Pavel Shamis (Pasha) wrote:
Hi Josh,
I had the original problem with some old revision from trunk.
Today I updated the server to latest revision from trunk + the patch
and everything looks good.
Can I commit the patch ?
Pasha
Ethan Mallove wrote:
On Wed, May/07/2008 06:04:07PM, Pavel Shamis (Pasha) wrote:
Hi Josh.
Looking at the patch I'm a little bit conserned. The
"get_table_fields()" is, as you mentioned, no longer used so
should be removed. However the other functions are critical to the
submission script particularly 'do_pg_connect' which opens the
connection to the backend database.
All the functions are implemented in $topdir/database.inc file.
And the "database.inc" implementation is better because it use
password and username from config.ini. The original implementation
from submit/index use
hardcoded values defined in the file.
Are you using the current development trunk (mtt/trunk) or the
stable release branch (mtt/branches/ompi-core-testers)?
trunk
Can you send us the error messages that you were receiving?
1. On client side I see ""*** WARNING: MTTDatabase client did not
get a serial"
As result of the error some of MTT results is not visible via the
web reporter
2. On server side I found follow error message:
[client 10.4.3.214] PHP Fatal error: Allowed memory size of
33554432 bytes exhausted (tried to allocate 23592960
bytes) in /.autodirect/swgwork/MTT/mtt/submit/index.php(79) :
eval()'d code on line 77515
[Mon May 05 19:26:05 2008] [notice] caught SIGTERM, shutting down
[Mon May 05 19:30:54 2008] [notice] suEXEC mechanism enabled
(wrapper: /usr/sbin/suexec)
[Mon May 05 19:30:54 2008] [notice] Digest: generating secret for
digest authentication ...
[Mon May 05 19:30:54 2008] [notice] Digest: done
[Mon May 05 19:30:54 2008] [notice] LDAP: Built with OpenLDAP LDAP SDK
[Mon May 05 19:30:54 2008] [notice] LDAP: SSL support unavailable
My memory limit in php.ini file was set on 256MB !
Looks like PHP is actually using a 32MB limit ("Allowed
memory size of 33554432 ..."). Does a (Apache?) daemon need
to be restarted for the php.ini file to take effect? To
check your settings, this little PHP script will print an
HTML page of all the active system settings (search on
"memory_limit").
<?php
phpinfo();
?>
-Ethan
Regards,
Pasha
Cheers,
Josh
On May 7, 2008, at 4:49 AM, Pavel Shamis (Pasha) wrote:
Hi,
I upgraded the server side (the mtt is still running , so don't
know if the problem was resolved)
During upgrade I had some problem with the submit/index.php
script, it had some duplicated functions and some of them were
broken.
Please review the attached patch.
Pasha
Ethan Mallove wrote:
On Tue, May/06/2008 06:29:33PM, Pavel Shamis (Pasha) wrote:
I'm not sure which cron jobs you're referring to. Do you
mean these?
https://svn.open-mpi.org/trac/mtt/browser/trunk/server/php/cron
I talked about this one:
https://svn.open-mpi.org/trac/mtt/wiki/ServerMaintenance
I'm guessing you would only be concerned with the below
periodic-maintenance.pl script, which just runs
ANALYZE/VACUUM queries. I think you can start that up
whenever you want (and it should optimize the Reporter).
https://svn.open-mpi.org/trac/mtt/browser/trunk/server/sql/cron/periodic-maintenance.pl
-Ethan
The only thing there are the regular
mtt-resu...@open-mpi.org email alerts and some out-of-date
DB monitoring junk. You can ignore that stuff.
Josh, are there some nightly (DB
pruning/cleaning/vacuuming?) cron jobs that Pasha should be
running?
-Ethan
Thanks.
Ethan Mallove wrote:
Hi Pasha,
I thought this issue was solved in r1119 (see below). Do you
have the latest mtt/server scripts?
https://svn.open-mpi.org/trac/mtt/changeset/1119/trunk/server/php/submit
-Ethan
On Tue, May/06/2008 03:26:43PM, Pavel Shamis (Pasha) wrote:
About the issue:
1. On client side I see ""*** WARNING: MTTDatabase client
did not get a serial"
As result of the error some of MTT results is not visible
via the web reporter
2. On server side I found follow error message:
[client 10.4.3.214] PHP Fatal error: Allowed memory size
of 33554432 bytes exhausted (tried to allocate 23592960
bytes) in /.autodirect/swgwork/MTT/mtt/submit/index.php(79)
: eval()'d code on line 77515
[Mon May 05 19:26:05 2008] [notice] caught SIGTERM,
shutting down
[Mon May 05 19:30:54 2008] [notice] suEXEC mechanism
enabled (wrapper: /usr/sbin/suexec)
[Mon May 05 19:30:54 2008] [notice] Digest: generating
secret for digest authentication ...
[Mon May 05 19:30:54 2008] [notice] Digest: done
[Mon May 05 19:30:54 2008] [notice] LDAP: Built with
OpenLDAP LDAP SDK
[Mon May 05 19:30:54 2008] [notice] LDAP: SSL support
unavailable
My memory limit in php.ini file was set on 256MB !
Any ideas ?
Thanks.
--
Pavel Shamis (Pasha)
Mellanox Technologies
_______________________________________________
mtt-users mailing list
mtt-us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
--
Pavel Shamis (Pasha)
Mellanox Technologies
_______________________________________________
mtt-users mailing list
mtt-us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
--
Pavel Shamis (Pasha)
Mellanox Technologies
_______________________________________________
mtt-users mailing list
mtt-us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
--
Pavel Shamis (Pasha)
Mellanox Technologies
Index: submit/index.php
===================================================================
--- submit/index.php (revision 1200)
+++ submit/index.php (working copy)
@@ -1,6 +1,7 @@
<?php
# Copyright (c) 2006 Cisco Systems, Inc. All rights reserved.
# Copyright (c) 2006-2007 Sun Microsystems, Inc. All rights
reserved.
+# Copyright (c) 2008 Mellanox Technologies. All rights
reserved.
#
#
@@ -24,8 +25,7 @@ if (file_exists("$topdir/config.inc")) {
ini_set("memory_limit", "32M");
$topdir = '..';
-$ompi_home = '/l/osl/www/doc/www.open-mpi.org';
-include_once("$ompi_home/dbpassword.inc");
+include_once("$topdir/database.inc");
include_once("$topdir/reporter.inc");
@@ -1465,60 +1465,6 @@ function get_table_indexes($table_name,
return simple_select($sql_cmd);
}
-# Function used to determine which _POST fields
-# to INSERT. Prevent non-existent fields from being
-# INSERTed
-function get_table_fields($table_name) {
-
- global $dbname;
- global $id;
-
- # These indexes are special in that they link phases
- # together and hence, can and do show up in _POST
- if ($table_name == "test_build")
- $special_indexes = array("mpi_install$id");
- elseif ($table_name == "test_run")
- $special_indexes = array("test_build$id");
-
- # Crude way to tell whether a field is an index
- $is_not_index_clause =
- "\n\t (table_name = '$table_name' AND NOT " .
- "\n\t (data_type = 'integer' AND " .
- "\n\t column_name ~ '_id$' AND " .
- "\n\t table_catalog = '$dbname'))";
-
- $is_special_index_clause =
- "\n\t (table_name = '$table_name' AND " .
- "\n\t (column_name = '$special_indexes[0]'))";
-
- $is_index_columns = array(
- "column_name",
- "data_type",
- "column_default");
-
- $sql_cmd = "\n SELECT " . join(",",$is_index_columns) .
- "\n\t FROM information_schema.columns WHERE " .
- "\n\t " .
- $is_not_index_clause . " OR " .
- $is_special_index_clause . ';';
-
- do_pg_connect();
-
- # This table will be easier to manage if it's
- # keyed by column, instead of index
- $tmp = array();
- $arr = array();
- $arr = select($sql_cmd);
-
- foreach ($is_index_columns as $col) {
- $tmp[$col] = array();
- for ($i = 0; $i < sizeof($arr); $i++) {
- $tmp[$col][] = $arr[$i][$col];
- }
- }
- return $tmp;
-}
-
#
# Useful display of all the parameters posted.
# Warning: this function could explode the memory footprint
causing the
@@ -1622,90 +1568,6 @@ function is_null_($var) {
######################################################################
-function do_pg_connect() {
-
- global $dbname;
- global $user;
- global $pass;
- global $pgsql_conn;
- static $connected = false;
-
- if (!$connected) {
- $pgsql_conn =
- pg_connect("host=localhost port=5432 dbname=$dbname
user=$user password=$pass");
-
- # Exit if we cannot connect
- if (!$pgsql_conn) {
- mtt_abort("\nCould not connect to the $dbname
database; " .
- "submit this run later.");
- }
- else {
- $connected = true;
-
- # Serialize all transactions by default - Safer that
way.
- do_pg_query("SET SESSION CHARACTERISTICS AS
TRANSACTION ISOLATION LEVEL SERIALIZABLE", false);
- }
-
- # pg_set_error_verbosity($pgsql_conn,
PGSQL_ERRORS_VERBOSE); # PHP 5 needed
- # pg_trace($_ENV['HOME'] . "/pgsql.trace", 'w',
$pgsql_conn);
- # debug("\npg_options: " .
var_export(pg_options($pgsql_conn)));
- }
-}
-
-function do_pg_query($cmd, $silent) {
- $db_res = null;
-
- do_pg_connect();
-
- debug("\nSQL: $cmd\n");
- if (! ($db_res = pg_query($cmd))) {
- $out = "\nSQL QUERY: " . $cmd .
- "\nSQL ERROR: " . pg_last_error() .
- "\nSQL ERROR: " . pg_result_error();
-
- # Some errors are unsurprising, allow for silence in
- # such cases
- if (! $silent) {
- mtt_error($out);
- mtt_send_mail($out);
- }
- }
- debug("\nDatabase rows affected: " .
pg_affected_rows($db_res) . "\n");
-
- return $db_res;
-}
-
-# Fetch scalar value
-function select_scalar($cmd) {
-
- $set = array();
- $set = simple_select($cmd);
- return array_shift($set);
-}
-
-# Fetch 1D array
-function simple_select($cmd) {
-
- do_pg_connect();
-
- $rows = null;
-
- debug("\nSQL: $cmd\n");
- if (! ($result = pg_query($cmd))) {
- $out = "\nSQL QUERY: " . $cmd .
- "\nSQL ERROR: " . pg_last_error() .
- "\nSQL ERROR: " . pg_result_error();
- mtt_error($out);
- mtt_send_mail($out);
- }
- $max = pg_num_rows($result);
- for ($i = 0; $i < $max; ++$i) {
- $row = pg_fetch_array($result, $i, PGSQL_NUM);
- $rows[] = $row[0];
- }
- return $rows;
-}
-
# Fetch an associative hash (column name => value)
function associative_select($cmd) {
@@ -1722,21 +1584,6 @@ function associative_select($cmd) {
return pg_fetch_array($result);
}
-# Fetch 2D array
-function select($cmd) {
- do_pg_connect();
-
- debug("\nSQL: $cmd\n");
- if (! ($result = pg_query($cmd))) {
- $out = "\nSQL QUERY: " . $cmd .
- "\nSQL ERROR: " . pg_last_error() .
- "\nSQL ERROR: " . pg_result_error();
- mtt_error($out);
- mtt_send_mail($out);
- }
- return pg_fetch_all($result);
-}
-
######################################################################
# Function for reporting errors back to the client
--
Pavel Shamis (Pasha)
Mellanox Technologies
_______________________________________________
mtt-users mailing list
mtt-us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
--
Pavel Shamis (Pasha)
Mellanox Technologies
--
Pavel Shamis (Pasha)
Mellanox Technologies
Index: database.inc
===================================================================
--- database.inc (revision 1200)
+++ database.inc (working copy)
@@ -3,6 +3,7 @@
#
# Copyright (c) 2006 Sun Microsystems, Inc.
# All rights reserved.
+# Copyright (c) 2008 Mellanox Technologies. All rights reserved.
# $COPYRIGHT$
#
# Additional copyrights may follow
@@ -100,10 +101,13 @@ function do_pg_connect() {
$pgsql_conn =
pg_connect("host=localhost port=5432 dbname=$mtt_database_name
user=$mtt_database_username password=$mtt_database_password");
if (!$pgsql_conn) {
- print("\nCould not connect to '$dbname' database.");
- exit;
+ mtt_abort("\nCould not connect to the $dbname database; " .
+ "submit this run later.");
+ } else {
+ $connected = true;
+# Serialize all transactions by default - Safer that way.
+ do_pg_query("SET SESSION CHARACTERISTICS AS TRANSACTION ISOLATION
LEVEL SERIALIZABLE", false);
}
- $connected = true;
}
}
@@ -113,12 +117,17 @@ function do_pg_query($cmd, $silent) {
$start = time();
- debug_sql($cmd, 2);
+ debug("\nSQL: $cmd\n");
if (! ($db_res = pg_query($cmd))) {
+ $out = "\nSQL QUERY: " . $cmd .
+ "\nSQL ERROR: " . pg_last_error() .
+ "\nSQL ERROR: " . pg_result_error();
+
+# Some errors are unsurprising, allow for silence in
+# such cases
if (! $silent) {
- print("\npostgres: " .
- pg_last_error() . "\n" .
- pg_result_error());
+ mtt_error($out);
+ mtt_send_mail($out);
}
}
@@ -137,21 +146,9 @@ function do_pg_query($cmd, $silent) {
# Fetch scalar value
function select_scalar($cmd) {
- do_pg_connect();
-
- $rows = null;
-
- debug_sql($cmd, 2);
- if (! ($result = pg_query($cmd))) {
- print("\npostgres: " . pg_last_error() . "\n" .
- pg_result_error());
- }
- $max = pg_num_rows($result);
- for ($i = 0; $i < $max; ++$i) {
- $row = pg_fetch_array($result, $i, PGSQL_NUM);
- $rows[] = $row[0];
- }
- return array_shift($rows);
+ $set = array();
+ $set = simple_select($cmd);
+ return array_shift($set);
}
# Fetch 1D array
@@ -160,10 +157,13 @@ function simple_select($cmd) {
$rows = null;
- debug_sql($cmd, 2);
+ debug("\nSQL: $cmd\n");
if (! ($result = pg_query($cmd))) {
- print("\npostgres: " . pg_last_error() . "\n" .
- pg_result_error());
+ $out = "\nSQL QUERY: " . $cmd .
+ "\nSQL ERROR: " . pg_last_error() .
+ "\nSQL ERROR: " . pg_result_error();
+ mtt_error($out);
+ mtt_send_mail($out);
}
$max = pg_num_rows($result);
for ($i = 0; $i < $max; ++$i) {
@@ -177,10 +177,13 @@ function simple_select($cmd) {
function select($cmd) {
do_pg_connect();
- debug_sql($cmd, 2);
+ debug("\nSQL: $cmd\n");
if (! ($result = pg_query($cmd))) {
- print("\npostgres: " . pg_last_error() . "\n" .
- pg_result_error());
+ $out = "\nSQL QUERY: " . $cmd .
+ "\nSQL ERROR: " . pg_last_error() .
+ "\nSQL ERROR: " . pg_result_error();
+ mtt_error($out);
+ mtt_send_mail($out);
}
return pg_fetch_all($result);
}
Index: submit/index.php
===================================================================
--- submit/index.php (revision 1200)
+++ submit/index.php (working copy)
@@ -1,6 +1,7 @@
<?php
# Copyright (c) 2006 Cisco Systems, Inc. All rights reserved.
# Copyright (c) 2006-2007 Sun Microsystems, Inc. All rights reserved.
+# Copyright (c) 2008 Mellanox Technologies. All rights reserved.
#
#
@@ -24,8 +25,7 @@ if (file_exists("$topdir/config.inc")) {
ini_set("memory_limit", "32M");
$topdir = '..';
-$ompi_home = '/l/osl/www/doc/www.open-mpi.org';
-include_once("$ompi_home/dbpassword.inc");
+include_once("$topdir/database.inc");
include_once("$topdir/reporter.inc");
@@ -1465,60 +1465,6 @@ function get_table_indexes($table_name,
return simple_select($sql_cmd);
}
-# Function used to determine which _POST fields
-# to INSERT. Prevent non-existent fields from being
-# INSERTed
-function get_table_fields($table_name) {
-
- global $dbname;
- global $id;
-
- # These indexes are special in that they link phases
- # together and hence, can and do show up in _POST
- if ($table_name == "test_build")
- $special_indexes = array("mpi_install$id");
- elseif ($table_name == "test_run")
- $special_indexes = array("test_build$id");
-
- # Crude way to tell whether a field is an index
- $is_not_index_clause =
- "\n\t (table_name = '$table_name' AND NOT " .
- "\n\t (data_type = 'integer' AND " .
- "\n\t column_name ~ '_id$' AND " .
- "\n\t table_catalog = '$dbname'))";
-
- $is_special_index_clause =
- "\n\t (table_name = '$table_name' AND " .
- "\n\t (column_name = '$special_indexes[0]'))";
-
- $is_index_columns = array(
- "column_name",
- "data_type",
- "column_default");
-
- $sql_cmd = "\n SELECT " . join(",",$is_index_columns) .
- "\n\t FROM information_schema.columns WHERE " .
- "\n\t " .
- $is_not_index_clause . " OR " .
- $is_special_index_clause . ';';
-
- do_pg_connect();
-
- # This table will be easier to manage if it's
- # keyed by column, instead of index
- $tmp = array();
- $arr = array();
- $arr = select($sql_cmd);
-
- foreach ($is_index_columns as $col) {
- $tmp[$col] = array();
- for ($i = 0; $i < sizeof($arr); $i++) {
- $tmp[$col][] = $arr[$i][$col];
- }
- }
- return $tmp;
-}
-
#
# Useful display of all the parameters posted.
# Warning: this function could explode the memory footprint causing the
@@ -1622,90 +1568,6 @@ function is_null_($var) {
######################################################################
-function do_pg_connect() {
-
- global $dbname;
- global $user;
- global $pass;
- global $pgsql_conn;
- static $connected = false;
-
- if (!$connected) {
- $pgsql_conn =
- pg_connect("host=localhost port=5432 dbname=$dbname user=$user
password=$pass");
-
- # Exit if we cannot connect
- if (!$pgsql_conn) {
- mtt_abort("\nCould not connect to the $dbname database; " .
- "submit this run later.");
- }
- else {
- $connected = true;
-
- # Serialize all transactions by default - Safer that way.
- do_pg_query("SET SESSION CHARACTERISTICS AS TRANSACTION ISOLATION
LEVEL SERIALIZABLE", false);
- }
-
- # pg_set_error_verbosity($pgsql_conn, PGSQL_ERRORS_VERBOSE); # PHP 5
needed
- # pg_trace($_ENV['HOME'] . "/pgsql.trace", 'w', $pgsql_conn);
- # debug("\npg_options: " . var_export(pg_options($pgsql_conn)));
- }
-}
-
-function do_pg_query($cmd, $silent) {
- $db_res = null;
-
- do_pg_connect();
-
- debug("\nSQL: $cmd\n");
- if (! ($db_res = pg_query($cmd))) {
- $out = "\nSQL QUERY: " . $cmd .
- "\nSQL ERROR: " . pg_last_error() .
- "\nSQL ERROR: " . pg_result_error();
-
- # Some errors are unsurprising, allow for silence in
- # such cases
- if (! $silent) {
- mtt_error($out);
- mtt_send_mail($out);
- }
- }
- debug("\nDatabase rows affected: " . pg_affected_rows($db_res) . "\n");
-
- return $db_res;
-}
-
-# Fetch scalar value
-function select_scalar($cmd) {
-
- $set = array();
- $set = simple_select($cmd);
- return array_shift($set);
-}
-
-# Fetch 1D array
-function simple_select($cmd) {
-
- do_pg_connect();
-
- $rows = null;
-
- debug("\nSQL: $cmd\n");
- if (! ($result = pg_query($cmd))) {
- $out = "\nSQL QUERY: " . $cmd .
- "\nSQL ERROR: " . pg_last_error() .
- "\nSQL ERROR: " . pg_result_error();
- mtt_error($out);
- mtt_send_mail($out);
- }
- $max = pg_num_rows($result);
- for ($i = 0; $i < $max; ++$i) {
- $row = pg_fetch_array($result, $i, PGSQL_NUM);
- $rows[] = $row[0];
- }
- return $rows;
-}
-
# Fetch an associative hash (column name => value)
function associative_select($cmd) {
@@ -1722,21 +1584,6 @@ function associative_select($cmd) {
return pg_fetch_array($result);
}
-# Fetch 2D array
-function select($cmd) {
- do_pg_connect();
-
- debug("\nSQL: $cmd\n");
- if (! ($result = pg_query($cmd))) {
- $out = "\nSQL QUERY: " . $cmd .
- "\nSQL ERROR: " . pg_last_error() .
- "\nSQL ERROR: " . pg_result_error();
- mtt_error($out);
- mtt_send_mail($out);
- }
- return pg_fetch_all($result);
-}
-
######################################################################
# Function for reporting errors back to the client
