[SCM] CTDB repository - branch master updated - ctdb-1.13-181-g6cf6a9b

2012-05-20 Thread Ronnie Sahlberg
The branch, master has been updated
   via  6cf6a9b071bd8dd730717ca07ff73bf247bb (commit)
   via  9ed58fef4991725f75509433496f4d5ffae0ae87 (commit)
  from  5aba53e6adcfcd7edbdac9e30aa5fcba176aca00 (commit)

http://gitweb.samba.org/?p=ctdb.git;a=shortlog;h=master


- Log -
commit 6cf6a9b071bd8dd730717ca07ff73bf247bb
Author: Ronnie Sahlberg 
Date:   Mon May 21 14:01:04 2012 +1000

DOCS: Document the new tunables to produce warnings if databases grow 
unexpectedly big.

commit 9ed58fef4991725f75509433496f4d5ffae0ae87
Author: Ronnie Sahlberg 
Date:   Mon May 21 13:11:38 2012 +1000

DEBUG: Add checks for and print debug messages when 1) a database contains 
very many records, 2) when a database is very big, 3) when a single record is 
very big.

Add tunables to control when to log these instances and allow it to be 
completely turned off by setting the threshold to 0

---

Summary of changes:
 doc/ctdbd.1|   21 +-
 doc/ctdbd.1.html   |  163 ++--
 doc/ctdbd.1.xml|   28 
 include/ctdb_private.h |3 +
 server/ctdb_recover.c  |   17 +
 server/ctdb_tunables.c |5 +-
 6 files changed, 158 insertions(+), 79 deletions(-)


Changeset truncated at 500 lines:

diff --git a/doc/ctdbd.1 b/doc/ctdbd.1
index be527aa..60abf03 100644
--- a/doc/ctdbd.1
+++ b/doc/ctdbd.1
@@ -1,13 +1,13 @@
 '\" t
 .\" Title: ctdbd
 .\"Author: [FIXME: author] [see http://docbook.sf.net/el/author]
-.\" Generator: DocBook XSL Stylesheets v1.76.1 
-.\"  Date: 03/23/2012
+.\" Generator: DocBook XSL Stylesheets v1.75.2 
+.\"  Date: 05/21/2012
 .\"Manual: CTDB - clustered TDB database
 .\"Source: ctdb
 .\"  Language: English
 .\"
-.TH "CTDBD" "1" "03/23/2012" "ctdb" "CTDB \- clustered TDB database"
+.TH "CTDBD" "1" "05/21/2012" "ctdb" "CTDB \- clustered TDB database"
 .\" -
 .\" * Define some portability stuff
 .\" -
@@ -483,6 +483,21 @@ When you enable this tunable, CTDB will no longer attempt 
to recover the cluster
 Default: 0
 .PP
 When set to 1, ctdb will allow ip addresses to be failed over onto this 
node\&. Any ip addresses that the node currently hosts will remain on the node 
but no new ip addresses can be failed over onto the node\&.
+.SS "DBRecordCountWarn"
+.PP
+Default: 10
+.PP
+When set to non\-zero, ctdb will log a warning when we try to recover a 
database with more than this many records\&. This will produce a warning if a 
database grows uncontrollably with orphaned records\&.
+.SS "DBRecordSizeWarn"
+.PP
+Default: 1000
+.PP
+When set to non\-zero, ctdb will log a warning when we try to recover a 
database where a single record is bigger than this\&. This will produce a 
warning if a database record grows uncontrollably with orphaned sub\-records\&.
+.SS "DBSizeWarn"
+.PP
+Default: 10
+.PP
+When set to non\-zero, ctdb will log a warning when we try to recover a 
database bigger than this\&. This will produce a warning if a database grows 
uncontrollably\&.
 .SS "VerboseMemoryNames"
 .PP
 Default: 0
diff --git a/doc/ctdbd.1.html b/doc/ctdbd.1.html
index 95f1e75..0530b22 100644
--- a/doc/ctdbd.1.html
+++ b/doc/ctdbd.1.html
@@ -1,4 +1,4 @@
-ctdbdNamectdbd — The CTDB cluster 
daemonSynopsisctdbd ctdbd  [-? --help] [-d --debug=] 
{--dbdir=} {--dbdir-persistent=} 
[--event-script-dir=] [-i --interactive] 
[--listen=
] [--logfile=] [--lvs] {--nlist=} [--no-lmaster] [--no-recmaster] [--nosetsched] {--notification-script=} [--public-add resses=] [--public-interface=] {--reclock=} [--single-public-ip=
] [--socket=] [--start-as-disabled] [--start-as-stopped] [--syslog] [--log-ringbuf-size=] [--torture] [--transport=] [--usage]DESCRIPTION +ctdbdNamectdbd — The CTDB cluster daemonSynopsisctdbd ctdbd [-? --help] [-d --debug=] {--dbdir=} {--dbdir-persistent=} [--event-script-dir=] [-i --interactive] [--listen=
] [--logfile=] [--lvs] {--nlist=} [--no-lmaster] [--no-recmaster] [--nosetsched] {--notification-script=} [--public-add resses=] [--public-interface=] {--reclock=} [--single-public-ip=
] [--socket=] [--start-as-disabled] [--start-as-stopped] [--syslog] [--log-ringbuf-size=] [--torture] [--transport=] [--usage]DESCRIPTION ctdbd is the main ctdb daemon. ctdbd provides a clustered version of the TDB database with automatic rebuild/recovery of the da

[SCM] CTDB repository - branch 1.13 updated - ctdb-1.43-3-gb826044

2012-05-20 Thread Ronnie Sahlberg
The branch, 1.13 has been updated
   via  b8260448c192c3866bffb99b475a4b3de57f38b3 (commit)
   via  4cd124c5759771c18c3277bf83aa277373b43942 (commit)
  from  b5f260d84b3989c5da5afb3010861f838dc59f1a (commit)

http://gitweb.samba.org/?p=ctdb.git;a=shortlog;h=1.13


- Log -
commit b8260448c192c3866bffb99b475a4b3de57f38b3
Author: Ronnie Sahlberg 
Date:   Mon May 21 14:01:04 2012 +1000

DOCS: Document the new tunables to produce warnings if databases grow 
unexpectedly big.

commit 4cd124c5759771c18c3277bf83aa277373b43942
Author: Ronnie Sahlberg 
Date:   Mon May 21 13:11:38 2012 +1000

DEBUG: Add checks for and print debug messages when 1) a database contains 
very many records, 2) when a database is very big, 3) when a single record is 
very big.

Add tunables to control when to log these instances and allow it to be 
completely turned off by setting the threshold to 0

---

Summary of changes:
 doc/ctdbd.1|   21 +-
 doc/ctdbd.1.html   |  163 ++--
 doc/ctdbd.1.xml|   28 
 include/ctdb_private.h |3 +
 server/ctdb_recover.c  |   17 +
 server/ctdb_tunables.c |5 +-
 6 files changed, 158 insertions(+), 79 deletions(-)


Changeset truncated at 500 lines:

diff --git a/doc/ctdbd.1 b/doc/ctdbd.1
index be527aa..60abf03 100644
--- a/doc/ctdbd.1
+++ b/doc/ctdbd.1
@@ -1,13 +1,13 @@
 '\" t
 .\" Title: ctdbd
 .\"Author: [FIXME: author] [see http://docbook.sf.net/el/author]
-.\" Generator: DocBook XSL Stylesheets v1.76.1 
-.\"  Date: 03/23/2012
+.\" Generator: DocBook XSL Stylesheets v1.75.2 
+.\"  Date: 05/21/2012
 .\"Manual: CTDB - clustered TDB database
 .\"Source: ctdb
 .\"  Language: English
 .\"
-.TH "CTDBD" "1" "03/23/2012" "ctdb" "CTDB \- clustered TDB database"
+.TH "CTDBD" "1" "05/21/2012" "ctdb" "CTDB \- clustered TDB database"
 .\" -
 .\" * Define some portability stuff
 .\" -
@@ -483,6 +483,21 @@ When you enable this tunable, CTDB will no longer attempt 
to recover the cluster
 Default: 0
 .PP
 When set to 1, ctdb will allow ip addresses to be failed over onto this 
node\&. Any ip addresses that the node currently hosts will remain on the node 
but no new ip addresses can be failed over onto the node\&.
+.SS "DBRecordCountWarn"
+.PP
+Default: 10
+.PP
+When set to non\-zero, ctdb will log a warning when we try to recover a 
database with more than this many records\&. This will produce a warning if a 
database grows uncontrollably with orphaned records\&.
+.SS "DBRecordSizeWarn"
+.PP
+Default: 1000
+.PP
+When set to non\-zero, ctdb will log a warning when we try to recover a 
database where a single record is bigger than this\&. This will produce a 
warning if a database record grows uncontrollably with orphaned sub\-records\&.
+.SS "DBSizeWarn"
+.PP
+Default: 10
+.PP
+When set to non\-zero, ctdb will log a warning when we try to recover a 
database bigger than this\&. This will produce a warning if a database grows 
uncontrollably\&.
 .SS "VerboseMemoryNames"
 .PP
 Default: 0
diff --git a/doc/ctdbd.1.html b/doc/ctdbd.1.html
index 95f1e75..0530b22 100644
--- a/doc/ctdbd.1.html
+++ b/doc/ctdbd.1.html
@@ -1,4 +1,4 @@
-ctdbdNamectdbd — The CTDB cluster 
daemonSynopsisctdbd ctdbd  [-? --help] [-d --debug=] 
{--dbdir=} {--dbdir-persistent=} 
[--event-script-dir=] [-i --interactive] 
[--listen=
] [--logfile=] [--lvs] {--nlist=} [--no-lmaster] [--no-recmaster] [--nosetsched] {--notification-script=} [--public-add resses=] [--public-interface=] {--reclock=} [--single-public-ip=
] [--socket=] [--start-as-disabled] [--start-as-stopped] [--syslog] [--log-ringbuf-size=] [--torture] [--transport=] [--usage]DESCRIPTION +ctdbdNamectdbd — The CTDB cluster daemonSynopsisctdbd ctdbd [-? --help] [-d --debug=] {--dbdir=} {--dbdir-persistent=} [--event-script-dir=] [-i --interactive] [--listen=
] [--logfile=] [--lvs] {--nlist=} [--no-lmaster] [--no-recmaster] [--nosetsched] {--notification-script=} [--public-add resses=] [--public-interface=] {--reclock=} [--single-public-ip=
] [--socket=] [--start-as-disabled] [--start-as-stopped] [--syslog] [--log-ringbuf-size=] [--torture] [--transport=] [--usage]DESCRIPTION ctdbd is the main ctdb daemon. ctdbd provides a clustered version of the TDB database with automatic rebuild/recovery of the databa

[SCM] CTDB repository - branch master updated - ctdb-1.13-179-g5aba53e

2012-05-20 Thread Ronnie Sahlberg
The branch, master has been updated
   via  5aba53e6adcfcd7edbdac9e30aa5fcba176aca00 (commit)
  from  f59b40b3f8ea3da8ffb8601bc025e83c237072d5 (commit)

http://gitweb.samba.org/?p=ctdb.git;a=shortlog;h=master


- Log -
commit 5aba53e6adcfcd7edbdac9e30aa5fcba176aca00
Author: Ronnie Sahlberg 
Date:   Mon May 21 09:17:05 2012 +1000

TEVENT: Add back tracking of long runnig  events to the local copy of 
tevent library

---

Summary of changes:
 lib/tevent/tevent_epoll.c|4 +++
 lib/tevent/tevent_internal.h |2 +
 lib/tevent/tevent_select.c   |4 +++
 lib/tevent/tevent_standard.c |4 +++
 lib/tevent/tevent_util.c |   52 ++
 5 files changed, 66 insertions(+), 0 deletions(-)


Changeset truncated at 500 lines:

diff --git a/lib/tevent/tevent_epoll.c b/lib/tevent/tevent_epoll.c
index f5a69dd..7c09dd7 100644
--- a/lib/tevent/tevent_epoll.c
+++ b/lib/tevent/tevent_epoll.c
@@ -242,6 +242,8 @@ static void epoll_change_event(struct epoll_event_context 
*epoll_ev, struct teve
}
 }
 
+extern pid_t ctdbd_pid;
+
 /*
   event loop handling using epoll
 */
@@ -264,7 +266,9 @@ static int epoll_event_loop(struct epoll_event_context 
*epoll_ev, struct timeval
return 0;
}
 
+   if (getpid() == ctdbd_pid) tevent_before_wait(epoll_ev->ev);
ret = epoll_wait(epoll_ev->epoll_fd, events, MAXEVENTS, timeout);
+   if (getpid() == ctdbd_pid) tevent_after_wait(epoll_ev->ev);
 
if (ret == -1 && errno == EINTR && epoll_ev->ev->signal_events) {
if (tevent_common_check_signal(epoll_ev->ev)) {
diff --git a/lib/tevent/tevent_internal.h b/lib/tevent/tevent_internal.h
index 7f1d876..707bfab 100644
--- a/lib/tevent/tevent_internal.h
+++ b/lib/tevent/tevent_internal.h
@@ -313,3 +313,5 @@ bool tevent_poll_init(void);
 #ifdef HAVE_EPOLL
 bool tevent_epoll_init(void);
 #endif
+void tevent_before_wait(struct tevent_context *ev);
+void tevent_after_wait(struct tevent_context *ev);
diff --git a/lib/tevent/tevent_select.c b/lib/tevent/tevent_select.c
index 51c1dec..6fa38ce 100644
--- a/lib/tevent/tevent_select.c
+++ b/lib/tevent/tevent_select.c
@@ -130,6 +130,8 @@ static struct tevent_fd *select_event_add_fd(struct 
tevent_context *ev, TALLOC_C
return fde;
 }
 
+extern pid_t ctdbd_pid;
+
 /*
   event loop handling using select()
 */
@@ -167,7 +169,9 @@ static int select_event_loop_select(struct 
select_event_context *select_ev, stru
return 0;
}
 
+   if (getpid() == ctdbd_pid) tevent_before_wait(select_ev->ev);
selrtn = select(select_ev->maxfd+1, &r_fds, &w_fds, NULL, tvalp);
+   if (getpid() == ctdbd_pid) tevent_after_wait(select_ev->ev);
 
if (selrtn == -1 && errno == EINTR &&
select_ev->ev->signal_events) {
diff --git a/lib/tevent/tevent_standard.c b/lib/tevent/tevent_standard.c
index 534576c..4041b6a 100644
--- a/lib/tevent/tevent_standard.c
+++ b/lib/tevent/tevent_standard.c
@@ -256,6 +256,8 @@ static void epoll_change_event(struct std_event_context 
*std_ev, struct tevent_f
}
 }
 
+extern pid_t ctdbd_pid;
+
 /*
   event loop handling using epoll
 */
@@ -278,7 +280,9 @@ static int epoll_event_loop(struct std_event_context 
*std_ev, struct timeval *tv
return 0;
}
 
+   if (getpid() == ctdbd_pid) tevent_before_wait(std_ev->ev);
ret = epoll_wait(std_ev->epoll_fd, events, MAXEVENTS, timeout);
+   if (getpid() == ctdbd_pid) tevent_after_wait(std_ev->ev);
 
if (ret == -1 && errno == EINTR && std_ev->ev->signal_events) {
if (tevent_common_check_signal(std_ev->ev)) {
diff --git a/lib/tevent/tevent_util.c b/lib/tevent/tevent_util.c
index 16af8f3..e3a1605 100644
--- a/lib/tevent/tevent_util.c
+++ b/lib/tevent/tevent_util.c
@@ -105,3 +105,55 @@ bool ev_set_close_on_exec(int fd)
 #endif
return false;
 }
+
+
+static struct timeval tevent_before_wait_ts;
+static struct timeval tevent_after_wait_ts;
+
+/*
+ * measure the time difference between multiple arrivals
+ * to the point where we wait for new events to come in
+ *
+ * allows to measure how long it takes to work on a 
+ * event
+ */
+void tevent_before_wait(struct tevent_context *ev) {
+
+   struct timeval diff;
+   struct timeval now = tevent_timeval_current();
+
+   if (!tevent_timeval_is_zero(&tevent_after_wait_ts)) {
+   diff = tevent_timeval_until(&tevent_after_wait_ts, &now);
+   if (diff.tv_sec > 3) {
+   tevent_debug(ev, TEVENT_DEBUG_ERROR,  __location__ 
+" Handling event took %d seconds!",
+(int) diff.tv_sec);
+   }
+   }
+
+   tevent_before_wait_ts = tevent_timeval_current();
+
+}
+
+/*
+ * measure how long the select()/epoll() call too

[SCM] CTDB repository - branch 1.13 updated - ctdb-1.43-1-gb5f260d

2012-05-20 Thread Ronnie Sahlberg
The branch, 1.13 has been updated
   via  b5f260d84b3989c5da5afb3010861f838dc59f1a (commit)
  from  f66d2b30eedc755cb8bc8b1c1e18141c6ee0c946 (commit)

http://gitweb.samba.org/?p=ctdb.git;a=shortlog;h=1.13


- Log -
commit b5f260d84b3989c5da5afb3010861f838dc59f1a
Author: Ronnie Sahlberg 
Date:   Mon May 21 09:17:05 2012 +1000

TEVENT: Add back tracking of long runnig  events to the local copy of 
tevent library

---

Summary of changes:
 lib/tevent/tevent_epoll.c|4 +++
 lib/tevent/tevent_internal.h |2 +
 lib/tevent/tevent_select.c   |4 +++
 lib/tevent/tevent_standard.c |4 +++
 lib/tevent/tevent_util.c |   52 ++
 5 files changed, 66 insertions(+), 0 deletions(-)


Changeset truncated at 500 lines:

diff --git a/lib/tevent/tevent_epoll.c b/lib/tevent/tevent_epoll.c
index f5a69dd..7c09dd7 100644
--- a/lib/tevent/tevent_epoll.c
+++ b/lib/tevent/tevent_epoll.c
@@ -242,6 +242,8 @@ static void epoll_change_event(struct epoll_event_context 
*epoll_ev, struct teve
}
 }
 
+extern pid_t ctdbd_pid;
+
 /*
   event loop handling using epoll
 */
@@ -264,7 +266,9 @@ static int epoll_event_loop(struct epoll_event_context 
*epoll_ev, struct timeval
return 0;
}
 
+   if (getpid() == ctdbd_pid) tevent_before_wait(epoll_ev->ev);
ret = epoll_wait(epoll_ev->epoll_fd, events, MAXEVENTS, timeout);
+   if (getpid() == ctdbd_pid) tevent_after_wait(epoll_ev->ev);
 
if (ret == -1 && errno == EINTR && epoll_ev->ev->signal_events) {
if (tevent_common_check_signal(epoll_ev->ev)) {
diff --git a/lib/tevent/tevent_internal.h b/lib/tevent/tevent_internal.h
index 7f1d876..707bfab 100644
--- a/lib/tevent/tevent_internal.h
+++ b/lib/tevent/tevent_internal.h
@@ -313,3 +313,5 @@ bool tevent_poll_init(void);
 #ifdef HAVE_EPOLL
 bool tevent_epoll_init(void);
 #endif
+void tevent_before_wait(struct tevent_context *ev);
+void tevent_after_wait(struct tevent_context *ev);
diff --git a/lib/tevent/tevent_select.c b/lib/tevent/tevent_select.c
index 51c1dec..6fa38ce 100644
--- a/lib/tevent/tevent_select.c
+++ b/lib/tevent/tevent_select.c
@@ -130,6 +130,8 @@ static struct tevent_fd *select_event_add_fd(struct 
tevent_context *ev, TALLOC_C
return fde;
 }
 
+extern pid_t ctdbd_pid;
+
 /*
   event loop handling using select()
 */
@@ -167,7 +169,9 @@ static int select_event_loop_select(struct 
select_event_context *select_ev, stru
return 0;
}
 
+   if (getpid() == ctdbd_pid) tevent_before_wait(select_ev->ev);
selrtn = select(select_ev->maxfd+1, &r_fds, &w_fds, NULL, tvalp);
+   if (getpid() == ctdbd_pid) tevent_after_wait(select_ev->ev);
 
if (selrtn == -1 && errno == EINTR &&
select_ev->ev->signal_events) {
diff --git a/lib/tevent/tevent_standard.c b/lib/tevent/tevent_standard.c
index 534576c..4041b6a 100644
--- a/lib/tevent/tevent_standard.c
+++ b/lib/tevent/tevent_standard.c
@@ -256,6 +256,8 @@ static void epoll_change_event(struct std_event_context 
*std_ev, struct tevent_f
}
 }
 
+extern pid_t ctdbd_pid;
+
 /*
   event loop handling using epoll
 */
@@ -278,7 +280,9 @@ static int epoll_event_loop(struct std_event_context 
*std_ev, struct timeval *tv
return 0;
}
 
+   if (getpid() == ctdbd_pid) tevent_before_wait(std_ev->ev);
ret = epoll_wait(std_ev->epoll_fd, events, MAXEVENTS, timeout);
+   if (getpid() == ctdbd_pid) tevent_after_wait(std_ev->ev);
 
if (ret == -1 && errno == EINTR && std_ev->ev->signal_events) {
if (tevent_common_check_signal(std_ev->ev)) {
diff --git a/lib/tevent/tevent_util.c b/lib/tevent/tevent_util.c
index 16af8f3..e3a1605 100644
--- a/lib/tevent/tevent_util.c
+++ b/lib/tevent/tevent_util.c
@@ -105,3 +105,55 @@ bool ev_set_close_on_exec(int fd)
 #endif
return false;
 }
+
+
+static struct timeval tevent_before_wait_ts;
+static struct timeval tevent_after_wait_ts;
+
+/*
+ * measure the time difference between multiple arrivals
+ * to the point where we wait for new events to come in
+ *
+ * allows to measure how long it takes to work on a 
+ * event
+ */
+void tevent_before_wait(struct tevent_context *ev) {
+
+   struct timeval diff;
+   struct timeval now = tevent_timeval_current();
+
+   if (!tevent_timeval_is_zero(&tevent_after_wait_ts)) {
+   diff = tevent_timeval_until(&tevent_after_wait_ts, &now);
+   if (diff.tv_sec > 3) {
+   tevent_debug(ev, TEVENT_DEBUG_ERROR,  __location__ 
+" Handling event took %d seconds!",
+(int) diff.tv_sec);
+   }
+   }
+
+   tevent_before_wait_ts = tevent_timeval_current();
+
+}
+
+/*
+ * measure how long the select()/epoll() call took
+ 

autobuild: intermittent test failure detected

2012-05-20 Thread autobuild
The autobuild test system has detected an intermittent failing test in 
the current master tree.

The autobuild log of the failure is available here:

   http://git.samba.org/autobuild.flakey/2012-05-21-0209/flakey.log

The samba3 build logs are available here:

   http://git.samba.org/autobuild.flakey/2012-05-21-0209/samba3.stderr
   http://git.samba.org/autobuild.flakey/2012-05-21-0209/samba3.stdout

The source4 build logs are available here:

   http://git.samba.org/autobuild.flakey/2012-05-21-0209/samba4.stderr
   http://git.samba.org/autobuild.flakey/2012-05-21-0209/samba4.stdout
  
The top commit at the time of the failure was:

commit 1d53e57bbfc6ea7ef4f18f41a9cba61f19d66793
Author: Matthieu Patou 
Date:   Fri May 11 14:25:49 2012 -0700

s4-dsdb: allow modification of some deleted object if the show-deleted 
control is presented

Autobuild-User: Matthieu Patou 
Autobuild-Date: Sat May 19 20:28:01 CEST 2012 on sn-devel-104


autobuild: intermittent test failure detected

2012-05-20 Thread autobuild
The autobuild test system has detected an intermittent failing test in 
the current master tree.

The autobuild log of the failure is available here:

   http://git.samba.org/autobuild.flakey/2012-05-20-1209/flakey.log

The samba3 build logs are available here:

   http://git.samba.org/autobuild.flakey/2012-05-20-1209/samba3.stderr
   http://git.samba.org/autobuild.flakey/2012-05-20-1209/samba3.stdout

The source4 build logs are available here:

   http://git.samba.org/autobuild.flakey/2012-05-20-1209/samba4.stderr
   http://git.samba.org/autobuild.flakey/2012-05-20-1209/samba4.stdout
  
The top commit at the time of the failure was:

commit 1d53e57bbfc6ea7ef4f18f41a9cba61f19d66793
Author: Matthieu Patou 
Date:   Fri May 11 14:25:49 2012 -0700

s4-dsdb: allow modification of some deleted object if the show-deleted 
control is presented

Autobuild-User: Matthieu Patou 
Autobuild-Date: Sat May 19 20:28:01 CEST 2012 on sn-devel-104