I am seeing failed writes to a postgresql database backend remain in the
write queue on the controller. The duplicate key error message for the
corresponding
write only appears on one of the 3 controllers. But the two sister
controllers have the same request id 10577 in the scheduler queue along
with any other
write requests which arrived after the 10577 request. Is this normal
behavior? How can I clear a failed write from the controller's write
queue? My three controllers
basically just start queueing any addtional writes after the duplicate
key write occurs. Any assistance with resolving this issue would be
greatly appreciated.
I am running raid1b configuration with 3 controllers. Each controller
has two postgresql dataserver backends.
Below are log snippets of what I am seeing:
Controller where initial duplicate key error is recorded:
ANGe(admin) > dump scheduler queues
Active transactions: 7
Transaction id list: 3800 3802 3803 3804 3805 3806 3807
Pending read requests: 0
Read request id list:
Pending write requests: 1
Write request id list: 10577
ANGe(admin) > dump request 10577
Request id: 10577
query: insert into SubmissionBean (state, startTime, submissionType,
submissionURL, producerName, statusInfo, producerRepresentativeName,
lastUpdateTime, id) values (?, ?, ?, ?, ?, ?, ?, ?, ?)
parameters: <!%I|0|!%><!%T|2007-01-09
19:41:38.089|!%><!%S|submissionType1|!%><!%S|ftp://n140.cluster.larc.nasa.gov/874b3edb-f853-4fbe-8687-71641e532106_Mon_Jan_08_23_04_51_EST_2007.notify|!%><!%S|edos|!%><!%S||!%><!%S|edosUser|!%><!%T|2007-01-09
19:41:38.089|!%><!%S|1da2a231-aa57-41d1-833d-1c1d999979e4|!%>
login: user
autocommit: false
transaction id: 3802
cacheable status: CACHEABLE
isolation level: TRANSACTION_UNDEFINED
start time: 1168389699937
end time: 0
timeout in seconds: 0
locked tables:
submissionbean
submissioncontentbean
persistent connection id: 14
client ip address: n104.cluster.larc.nasa.gov/10.10.101.4
full_cluster.log entries for request 10577:
full_cluster.log:2007-01-09 19:41:41,472 ERROR
controller.loadbalancer.RAIDb1 write request 10577 failed:
full_cluster.log:Backend ANGe - BackendWorkerThread for backend
'postgresql-n120' with RAIDb level:1 failed (ERROR: duplicate key
violates unique constraint "submissionbean_pkey")
full_cluster.log:Backend ANGe - BackendWorkerThread for backend
'postgresql-n121' with RAIDb level:1 failed (ERROR: duplicate key
violates unique constraint "submissionbean_pkey")
full_cluster.log:2007-01-09 19:41:41,472 ERROR
controller.loadbalancer.RAIDb1 write request 10577 failed:
full_cluster.log:Backend ANGe - BackendWorkerThread for backend
'postgresql-n120' with RAIDb level:1 failed (ERROR: duplicate key
violates unique constraint "submissionbean_pkey")
full_cluster.log:Backend ANGe - BackendWorkerThread for backend
'postgresql-n121' with RAIDb level:1 failed (ERROR: duplicate key
violates unique constraint "submissionbean_pkey")
full_cluster.log:Backend ANGe - BackendWorkerThread for backend
'postgresql-n120' with RAIDb level:1 failed (ERROR: duplicate key
violates unique constraint "submissionbean_pkey")
full_cluster.log:Backend ANGe - BackendWorkerThread for backend
'postgresql-n121' with RAIDb level:1 failed (ERROR: duplicate key
violates unique constraint "submissionbean_pkey")
full_cluster.log:Backend ANGe - BackendWorkerThread for backend
'postgresql-n120' with RAIDb level:1 failed (ERROR: duplicate key
violates unique constraint "submissionbean_pkey")
full_cluster.log:Backend ANGe - BackendWorkerThread for backend
'postgresql-n121' with RAIDb level:1 failed (ERROR: duplicate key
violates unique constraint "submissionbean_pkey")
full_cluster.log:Backend ANGe - BackendWorkerThread for backend
'postgresql-n120' with RAIDb level:1 failed (ERROR: duplicate key
violates unique constraint "submissionbean_pkey")
full_cluster.log:Backend ANGe - BackendWorkerThread for backend
'postgresql-n121' with RAIDb level:1 failed (ERROR: duplicate key
violates unique constraint "submissionbean_pkey")
full_cluster.log:Backend ANGe - BackendWorkerThread for backend
'postgresql-n120' with RAIDb level:1 failed (ERROR: duplicate key
violates unique constraint "submissionbean_pkey")
full_cluster.log:Backend ANGe - BackendWorkerThread for backend
'postgresql-n121' with RAIDb level:1 failed (ERROR: duplicate key
violates unique constraint "submissionbean_pkey")
2nd Controller where no duplicate key error is recored but request is
queued:
ANGe(admin) > dump scheduler queues
Active transactions: 7
Transaction id list: 3800 3802 3803 3804 3805 3806 3807
Pending read requests: 0
Read request id list:
Pending write requests: 6
Write request id list: 10586 10593 10591 10581 10587 10577
3rd controller where no duplicate error is recorded but request is queued:
ANGe(admin) > dump scheduler queues
Active transactions: 8
Transaction id list: 3703 3800 3802 3803 3804 3805 3806 3807
Pending read requests: 1
Read request id list: 10283
Pending write requests: 6
Write request id list: 10586 10593 10591 10581 10587 10577
Any suggestions on how I can recover when this happens?
Thanks for your assistance,
begin:vcard
fn:Jim Davis
n:Davis;Jim
org:Atmospheric Sciences Data Center;NASA Langley Research Center
adr;dom:;;;Hampton;VA
email;internet:[EMAIL PROTECTED]
title:SSAI, Systems Engineer
tel;work:757-864-7525
version:2.1
end:vcard
_______________________________________________
Sequoia mailing list
[email protected]
https://forge.continuent.org/mailman/listinfo/sequoia
