Jim Davis wrote:
I brought down my test system, flushed all of the databases, brought up
the sequoia's with INFO logging only and re-ran our 3600 submission
ingest test (1 submission per second). Test hit a hung state
on controllers within a minute of starting. I have attached files
containing snapshots from each controller for dump scheduler queues,
dump backend schema <backend> /locks, and show backend * along
with the errors from the full_cluster.log for each controller. I
checked my application/jboss server logs and I see no errors when these
locks occur. As far the Jboss and my application can tell, everything
is fine and
they are just waiting for database responses. I see the same conditions
on the controllers as before. A duplicate primary key violation during
a write to our application database table which results in a single
write request in queue on one controller and the same request plus
additional write requests on the remaining two controllers.
I could not attach the full_cluster.logs from each controller as they
caused this email message to exceed the maximum email attachement
allowed. I have grep'd out the error messages from each controller's
full_cluster.log instead. If you need additional information from the
logs, please let me know.
Thankyou for all of your assistance,
Jim D.
Emmanuel Cecchet wrote:
Hi Jim,
Sorry for the late reply.
I am using the appia configuration for group communications. I was
having issues with jgroups when I starting really loading the system
with alot of transactions.
Do you have a significant number of writes or manipulating large
amount of data in inserts (like large objects) ?
As I am not a java developer, my knowledge comes from repeated trial
and error. Because my developers cannot reproduce this problem in
their development environment when they go straight to
a postgres data server instead via sequoia, I am left to my own
devices to find a working solution if I am to get to keep my sequoia
configuration for our new production system. I am willing to try
anything to get past this problem. I have DEBUG turned on
everywhere... I have my virtualdatabase configuration set to 30
seconds idle timeout and 10 seconds wait.
Setting to DEBUG mode will significantly slowdown the controller and
add latency to every single query. This might increase lock contention
and induce deadlocks that would not be seen otherwise when accessing
directly the database where lock contention would be much less important.
You should also check in the log if you don't see any warning message
about table names that are not found in the schema. That could lead to
locking conflicts.
Stored procedures could also introduce artificial deadlocks. Do you
use any?
I would appreciate any suggestions you could send my way for how to
proceed.
If would be helpful to dump all controller queues the next time you
get a deadlock. You should be able to get a snapshot of running
queries and transactions. We could match the ids with what is in your
debug log to see if anything wrong happened with any of these queries
or transactions.
Keep us posted with your progress,
Emmanuel
ANGe(admin) > dump backend schema postgresql-n114 /locks
public.aiplocation
Lock owner transaction id: 131
public.orders
No lock
public.angedescriptiveinfo
No lock
public.submissioncontentbean
Lock owner transaction id: 157
public.angedescriptiveinfo_contentfiles
No lock
public.lnslink
No lock
public.submissionbean
Lock owner transaction id: 157
public.ftpdsalocations
No lock
ANGe(admin) > dump backend schema postgresql-n115 /locks
public.ftpdsalocations
No lock
public.aiplocation
Lock owner transaction id: 209
public.lnslink
No lock
public.angedescriptiveinfo
No lock
public.submissioncontentbean
Lock owner transaction id: 208
public.angedescriptiveinfo_contentfiles
No lock
public.orders
No lock
public.submissionbean
Lock owner transaction id: 208
ANGe(admin) > dump scheduler queues
Active transactions: 8
Transaction id list: 208 209 210 211 212 213 214 215
Pending read requests: 0
Read request id list:
Pending write requests: 7
Write request id list: 618 624 632 627 612 629 628
ANGe(admin) > show backend *
+--------------------------+------------------------------------------+
| Backend Name | postgresql-n114 |
| Driver | org.postgresql.Driver |
| URL | jdbc:postgresql://10.10.101.14:3070/ange |
| Active transactions | 18 |
| Pending Requests | 1 |
| Read Enabled | true |
| Write Enabled | true |
| Is Initialized | true |
| Static Schema | false |
| Connection Managers | 1 |
| Total Active Connections | 18 |
| Persistent Connections | 0 |
| Total Requests | 418 |
| Total Transactions | 136 |
| Last known checkpoint | <unknown> |
+--------------------------+------------------------------------------+
| Backend Name | postgresql-n115 |
| Driver | org.postgresql.Driver |
| URL | jdbc:postgresql://10.10.101.15:3070/ange |
| Active transactions | 7 |
| Pending Requests | 0 |
| Read Enabled | true |
| Write Enabled | true |
| Is Initialized | true |
| Static Schema | false |
| Connection Managers | 1 |
| Total Active Connections | 11 |
| Persistent Connections | 0 |
| Total Requests | 410 |
| Total Transactions | 166 |
| Last known checkpoint | <unknown> |
+--------------------------+------------------------------------------+
ANGe(admin) > dump scheduler queues
Active transactions: 8
Transaction id list: 208 209 210 211 213 214 212 215
Pending read requests: 0
Read request id list:
Pending write requests: 1
Write request id list: 612
ANGe(admin) > show backend *
+--------------------------+------------------------------------------+
| Backend Name | postgresql-n117 |
| Driver | org.postgresql.Driver |
| URL | jdbc:postgresql://10.10.101.17:3070/ange |
| Active transactions | 8 |
| Pending Requests | 0 |
| Read Enabled | true |
| Write Enabled | true |
| Is Initialized | true |
| Static Schema | false |
| Connection Managers | 1 |
| Total Active Connections | 10 |
| Persistent Connections | 0 |
| Total Requests | 267 |
| Total Transactions | 167 |
| Last known checkpoint | <unknown> |
+--------------------------+------------------------------------------+
| Backend Name | postgresql-n118 |
| Driver | org.postgresql.Driver |
| URL | jdbc:postgresql://10.10.101.18:3070/ange |
| Active transactions | 8 |
| Pending Requests | 0 |
| Read Enabled | true |
| Write Enabled | true |
| Is Initialized | true |
| Static Schema | false |
| Connection Managers | 1 |
| Total Active Connections | 10 |
| Persistent Connections | 0 |
| Total Requests | 267 |
| Total Transactions | 167 |
| Last known checkpoint | <unknown> |
+--------------------------+------------------------------------------+
ANGe(admin) > dump backend schema postgresql-n117 /locks
public.submissioncontentbean
Lock owner transaction id: 208
public.orders
No lock
public.aiplocation
Lock owner transaction id: 209
public.angedescriptiveinfo
No lock
public.angedescriptiveinfo_contentfiles
No lock
public.ftpdsalocations
No lock
public.lnslink
No lock
public.submissionbean
Lock owner transaction id: 208
ANGe(admin) > dump backend schema postgresql-n118 /locks
public.submissioncontentbean
Lock owner transaction id: 208
public.angedescriptiveinfo
No lock
public.angedescriptiveinfo_contentfiles
No lock
public.orders
No lock
public.submissionbean
Lock owner transaction id: 208
public.lnslink
No lock
public.ftpdsalocations
No lock
public.aiplocation
Lock owner transaction id: 209
ANGe(admin) > dump scheduler queues
Active transactions: 8
Transaction id list: 208 209 210 211 213 214 212 215
Pending read requests: 0
Read request id list:
Pending write requests: 7
Write request id list: 618 624 632 627 612 629 628
ANGe(admin) > show backend *
+--------------------------+------------------------------------------+
| Backend Name | postgresql-n120 |
| Driver | org.postgresql.Driver |
| URL | jdbc:postgresql://10.10.101.20:3070/ange |
| Active transactions | 4 |
| Pending Requests | 1 |
| Read Enabled | true |
| Write Enabled | true |
| Is Initialized | true |
| Static Schema | false |
| Connection Managers | 1 |
| Total Active Connections | 9 |
| Persistent Connections | 0 |
| Total Requests | 214 |
| Total Transactions | 129 |
| Last known checkpoint | <unknown> |
+--------------------------+------------------------------------------+
| Backend Name | postgresql-n121 |
| Driver | org.postgresql.Driver |
| URL | jdbc:postgresql://10.10.101.21:3070/ange |
| Active transactions | 2 |
| Pending Requests | 0 |
| Read Enabled | true |
| Write Enabled | true |
| Is Initialized | true |
| Static Schema | false |
| Connection Managers | 1 |
| Total Active Connections | 7 |
| Persistent Connections | 0 |
| Total Requests | 261 |
| Total Transactions | 161 |
| Last known checkpoint | <unknown> |
+--------------------------+------------------------------------------+
ANGe(admin) > dump backend schema postgresql-n120 /locks
public.submissioncontentbean
Lock owner transaction id: 159
public.angedescriptiveinfo_contentfiles
No lock
public.aiplocation
Lock owner transaction id: 151
public.angedescriptiveinfo
No lock
public.ftpdsalocations
No lock
public.orders
No lock
public.lnslink
No lock
public.submissionbean
Lock owner transaction id: 159
ANGe(admin) > dump backend schema postgresql-n121 /locks
public.ftpdsalocations
No lock
public.submissioncontentbean
Lock owner transaction id: 208
public.angedescriptiveinfo
No lock
public.aiplocation
Lock owner transaction id: 209
public.orders
No lock
public.lnslink
No lock
public.angedescriptiveinfo_contentfiles
No lock
public.submissionbean
Lock owner transaction id: 208
2007-01-12 17:33:53,337 ERROR sequoia.controller.scheduler Wait failed in
suspendWrites (java.lang.InterruptedException)
2007-01-12 17:33:53,337 ERROR sequoia.controller.scheduler Wait failed in
suspendWrites (java.lang.InterruptedException)
2007-01-12 17:33:53,337 ERROR controller.virtualdatabase.ANGe Failed to wait
for writes to complete
2007-01-12 17:33:53,337 ERROR controller.virtualdatabase.ANGe Failed to wait
for writes to complete
2007-01-12 17:33:53,357 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:33:53,357 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:33:53,363 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:33:53,363 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:33:53,369 ERROR sequoia.controller.scheduler Unexpected negative
suspendedWrites in AbstractScheduler.resumeWrites()
2007-01-12 17:33:53,369 ERROR sequoia.controller.scheduler Unexpected negative
suspendedWrites in AbstractScheduler.resumeWrites()
2007-01-12 17:33:53,369 ERROR sequoia.controller.scheduler Unexpected negative
suspendedTransactions in AbstractScheduler.resumeNewTransactions()
2007-01-12 17:33:53,369 ERROR sequoia.controller.scheduler Unexpected negative
suspendedTransactions in AbstractScheduler.resumeNewTransactions()
2007-01-12 17:33:53,369 ERROR sequoia.controller.scheduler Unexpected negative
suspendedPersistentConnections in
AbstractScheduler.resumeNewPersistentConnections()
2007-01-12 17:33:53,369 ERROR sequoia.controller.scheduler Unexpected negative
suspendedPersistentConnections in
AbstractScheduler.resumeNewPersistentConnections()
2007-01-12 17:42:17,979 ERROR sequoia.controller.scheduler Wait failed in
suspendWrites (java.lang.InterruptedException)
2007-01-12 17:42:17,979 ERROR sequoia.controller.scheduler Wait failed in
suspendWrites (java.lang.InterruptedException)
2007-01-12 17:42:17,979 ERROR controller.virtualdatabase.ANGe Failed to wait
for writes to complete
2007-01-12 17:42:17,979 ERROR controller.virtualdatabase.ANGe Failed to wait
for writes to complete
2007-01-12 17:42:17,981 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:42:17,981 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:42:17,985 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:42:17,985 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:42:17,990 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:42:17,990 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:42:17,994 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:42:17,994 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:42:17,998 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:42:17,998 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-16 15:24:02,595 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:24:02,595 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:24:22,264 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:24:22,264 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:24:22,781 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:24:22,781 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:24:22,845 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:24:22,845 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:39:45,339 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-16 15:39:45,339 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-11 18:12:25,562 ERROR continuent.sequoia.enduser Virtual database ANGe
failed to restore dump postgresql-n117 (because of : Trying to access a
non-existing backend postgresql-n114_dump_18131107)
2007-01-11 18:13:03,091 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-11 18:13:03,091 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-11 18:13:03,558 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-11 18:13:03,558 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-11 18:13:03,633 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-11 18:13:03,633 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-11 18:13:15,771 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-11 18:13:15,771 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-11 18:13:16,526 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-11 18:13:16,526 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-11 18:13:16,590 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-11 18:13:16,590 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-12 09:26:03,432 ERROR controller.loadbalancer.RAIDb1 write request 1657
failed:
Backend ANGe - BackendWorkerThread for backend 'postgresql-n117' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n118' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
2007-01-12 09:26:03,432 ERROR controller.loadbalancer.RAIDb1 write request 1657
failed:
Backend ANGe - BackendWorkerThread for backend 'postgresql-n117' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n118' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n117' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n118' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n117' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n118' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n117' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n118' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n117' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n118' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
2007-01-12 09:26:03,484 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 09:26:03,484 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 09:26:03,871 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 09:26:03,871 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 09:26:04,809 ERROR controller.loadbalancer.RAIDb1 write request 1659
failed:
Backend ANGe - BackendWorkerThread for backend 'postgresql-n118' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n117' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
2007-01-12 09:26:04,809 ERROR controller.loadbalancer.RAIDb1 write request 1659
failed:
Backend ANGe - BackendWorkerThread for backend 'postgresql-n118' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n117' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n118' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n117' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n118' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n117' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n118' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n117' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n118' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n117' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
2007-01-12 09:26:04,828 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 09:26:04,828 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 09:26:07,184 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 09:26:07,184 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:33:53,467 ERROR sequoia.controller.scheduler Wait failed in
suspendWrites (java.lang.InterruptedException)
2007-01-12 17:33:53,467 ERROR sequoia.controller.scheduler Wait failed in
suspendWrites (java.lang.InterruptedException)
2007-01-12 17:33:53,467 ERROR controller.virtualdatabase.ANGe Failed to wait
for writes to complete
2007-01-12 17:33:53,467 ERROR controller.virtualdatabase.ANGe Failed to wait
for writes to complete
2007-01-12 17:33:53,482 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:33:53,482 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:33:53,488 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:33:53,488 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:33:53,503 ERROR sequoia.controller.scheduler Unexpected negative
suspendedWrites in AbstractScheduler.resumeWrites()
2007-01-12 17:33:53,503 ERROR sequoia.controller.scheduler Unexpected negative
suspendedWrites in AbstractScheduler.resumeWrites()
2007-01-12 17:33:53,503 ERROR sequoia.controller.scheduler Unexpected negative
suspendedTransactions in AbstractScheduler.resumeNewTransactions()
2007-01-12 17:33:53,503 ERROR sequoia.controller.scheduler Unexpected negative
suspendedTransactions in AbstractScheduler.resumeNewTransactions()
2007-01-12 17:33:53,504 ERROR sequoia.controller.scheduler Unexpected negative
suspendedPersistentConnections in
AbstractScheduler.resumeNewPersistentConnections()
2007-01-12 17:33:53,504 ERROR sequoia.controller.scheduler Unexpected negative
suspendedPersistentConnections in
AbstractScheduler.resumeNewPersistentConnections()
2007-01-12 17:42:18,119 ERROR sequoia.controller.scheduler Wait failed in
suspendWrites (java.lang.InterruptedException)
2007-01-12 17:42:18,119 ERROR sequoia.controller.scheduler Wait failed in
suspendWrites (java.lang.InterruptedException)
2007-01-12 17:42:18,119 ERROR controller.virtualdatabase.ANGe Failed to wait
for writes to complete
2007-01-12 17:42:18,119 ERROR controller.virtualdatabase.ANGe Failed to wait
for writes to complete
2007-01-12 17:42:18,121 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:42:18,121 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:42:18,124 ERROR controller.virtualdatabase.ANGe Disable backend
[EMAIL PROTECTED], [EMAIL PROTECTED] was not found in total order queue,
posting out of order (disable all
backends-10.10.101.19:25323-20070112174156486-0500)
2007-01-12 17:42:18,124 ERROR controller.virtualdatabase.ANGe Disable backend
[EMAIL PROTECTED], [EMAIL PROTECTED] was not found in total order queue,
posting out of order (disable all
backends-10.10.101.19:25323-20070112174156486-0500)
2007-01-12 17:42:18,125 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:42:18,125 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:42:18,131 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:42:18,131 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:42:18,133 ERROR controller.virtualdatabase.ANGe Suspend activity
was not found in total order queue, posting out of order
2007-01-12 17:42:18,133 ERROR controller.virtualdatabase.ANGe Suspend activity
was not found in total order queue, posting out of order
2007-01-12 17:42:18,134 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:42:18,134 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:42:18,137 ERROR controller.virtualdatabase.ANGe Disable backend
[] was not found in total order queue, posting out of order
(shutdown-10.10.101.19:25323-20070112174218078-0500)
2007-01-12 17:42:18,137 ERROR controller.virtualdatabase.ANGe Disable backend
[] was not found in total order queue, posting out of order
(shutdown-10.10.101.19:25323-20070112174218078-0500)
2007-01-12 17:42:18,138 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:42:18,138 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:42:18,235 ERROR controller.jmx.MBeanServer Register mbean of type
sequoia:name=ANGe,type=VirtualDataBase failed with error
sequoia:type=VirtualDataBase,name=ANGe ({2})
2007-01-12 17:42:18,235 ERROR controller.jmx.MBeanServer Register mbean of type
sequoia:name=ANGe,type=VirtualDataBase failed with error
sequoia:type=VirtualDataBase,name=ANGe ({2})
2007-01-12 17:42:18,235 ERROR controller.virtualdatabase.ANGe Failed to
unregister the mbean of virtual database ANGe
2007-01-12 17:42:18,235 ERROR controller.virtualdatabase.ANGe Failed to
unregister the mbean of virtual database ANGe
2007-01-16 15:25:55,085 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:25:55,085 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:25:55,572 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:25:55,572 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:25:55,631 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:25:55,631 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:26:17,565 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:26:17,565 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:26:18,086 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:26:18,086 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:26:18,146 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:26:18,146 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:39:45,323 ERROR controller.loadbalancer.RAIDb1 write request 612
failed:
Backend ANGe - BackendWorkerThread for backend 'postgresql-n117' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissionbean_pkey")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n118' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissionbean_pkey")
2007-01-16 15:39:45,323 ERROR controller.loadbalancer.RAIDb1 write request 612
failed:
Backend ANGe - BackendWorkerThread for backend 'postgresql-n117' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissionbean_pkey")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n118' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissionbean_pkey")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n117' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissionbean_pkey")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n118' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissionbean_pkey")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n117' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissionbean_pkey")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n118' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissionbean_pkey")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n117' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissionbean_pkey")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n118' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissionbean_pkey")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n117' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissionbean_pkey")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n118' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissionbean_pkey")
2007-01-16 15:39:45,383 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-16 15:39:45,383 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-11 18:14:13,912 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-11 18:14:13,912 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-11 18:14:14,621 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-11 18:14:14,621 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-11 18:14:14,692 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-11 18:14:14,692 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-11 18:14:25,348 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-11 18:14:25,348 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-11 18:14:26,067 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-11 18:14:26,067 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-11 18:14:26,155 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-11 18:14:26,155 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-12 09:26:03,598 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 09:26:03,598 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 09:26:03,617 ERROR controller.loadbalancer.RAIDb1 write request 1657
failed:
Backend ANGe - BackendWorkerThread for backend 'postgresql-n120' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n121' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
2007-01-12 09:26:03,617 ERROR controller.loadbalancer.RAIDb1 write request 1657
failed:
Backend ANGe - BackendWorkerThread for backend 'postgresql-n120' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n121' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n120' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n121' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n120' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n121' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n120' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n121' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n120' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n121' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
2007-01-12 09:26:03,637 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 09:26:03,637 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 09:26:05,111 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 09:26:05,111 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 09:26:07,107 ERROR controller.loadbalancer.RAIDb1 write request 1659
failed:
Backend ANGe - BackendWorkerThread for backend 'postgresql-n120' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n121' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
2007-01-12 09:26:07,107 ERROR controller.loadbalancer.RAIDb1 write request 1659
failed:
Backend ANGe - BackendWorkerThread for backend 'postgresql-n120' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n121' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n120' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n121' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n120' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n121' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n120' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n121' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n120' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
Backend ANGe - BackendWorkerThread for backend 'postgresql-n121' with RAIDb
level:1 failed (ERROR: duplicate key violates unique constraint
"submissioncontentbean_contentlabel_key")
2007-01-12 09:26:07,131 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 09:26:07,131 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-12 17:33:53,325 ERROR sequoia.controller.scheduler Wait failed in
suspendWrites (java.lang.InterruptedException)
2007-01-12 17:33:53,325 ERROR sequoia.controller.scheduler Wait failed in
suspendWrites (java.lang.InterruptedException)
2007-01-12 17:33:53,325 ERROR controller.virtualdatabase.ANGe Failed to wait
for writes to complete
2007-01-12 17:33:53,325 ERROR controller.virtualdatabase.ANGe Failed to wait
for writes to complete
2007-01-12 17:36:31,471 ERROR continuent.sequoia.enduser Virtual database ANGe
failed to enable database backend postgresql-n120 from checkpoint (because of :
Cannot enable backend postgresql-n120 from a known state. Resynchronize this
backend by restoring a dump.)
2007-01-12 17:36:46,596 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-12 17:36:46,596 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-12 17:36:47,099 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-12 17:36:47,099 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-12 17:36:47,163 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-12 17:36:47,163 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-12 17:36:47,178 ERROR controller.backend.BackendStateListener Could not
store information for backend: postgresql-n120
2007-01-12 17:36:55,725 ERROR continuent.sequoia.enduser Virtual database ANGe
failed to enable database backend postgresql-n120 from checkpoint (because of :
Cannot enable backend postgresql-n120 from a known state. Resynchronize this
backend by restoring a dump.)
2007-01-16 15:27:10,587 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:27:10,587 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:27:11,140 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:27:11,140 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:27:11,249 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:27:11,249 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:27:22,775 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:27:22,775 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:27:23,290 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:27:23,290 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:27:23,355 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:27:23,355 INFO backup.backupers.NativeCommandExec Command
"expect - " logged 0 errors and terminated with exitcode 0
2007-01-16 15:39:45,422 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
2007-01-16 15:39:45,422 ERROR continuent.hedera.adapters PullPushAdapter: Error
while reading from channel
begin:vcard
fn:Jim Davis
n:Davis;Jim
org:Atmospheric Sciences Data Center;NASA Langley Research Center
adr;dom:;;;Hampton;VA
email;internet:[EMAIL PROTECTED]
title:SSAI, Systems Engineer
tel;work:757-864-7525
version:2.1
end:vcard
_______________________________________________
Sequoia mailing list
[email protected]
https://forge.continuent.org/mailman/listinfo/sequoia
