Jim Davis wrote:
I noticed that the logging information I provided from the test re-run
did not have the actual insert for the duplicate key violations which
started the whole problem. So, I grep'd out for each transaction
on each controller which had the duplicate error (n119 and n123 -
controller n116 did not have a duplicate key error message logged).
I truly appreciate all of your efforts,
Jim D.
Emmanuel Cecchet wrote:
Hi Jim,
Sorry for the late reply.
I am using the appia configuration for group communications. I was
having issues with jgroups when I starting really loading the system
with alot of transactions.
Do you have a significant number of writes or manipulating large
amount of data in inserts (like large objects) ?
As I am not a java developer, my knowledge comes from repeated trial
and error. Because my developers cannot reproduce this problem in
their development environment when they go straight to
a postgres data server instead via sequoia, I am left to my own
devices to find a working solution if I am to get to keep my sequoia
configuration for our new production system. I am willing to try
anything to get past this problem. I have DEBUG turned on
everywhere... I have my virtualdatabase configuration set to 30
seconds idle timeout and 10 seconds wait.
Setting to DEBUG mode will significantly slowdown the controller and
add latency to every single query. This might increase lock contention
and induce deadlocks that would not be seen otherwise when accessing
directly the database where lock contention would be much less important.
You should also check in the log if you don't see any warning message
about table names that are not found in the schema. That could lead to
locking conflicts.
Stored procedures could also introduce artificial deadlocks. Do you
use any?
I would appreciate any suggestions you could send my way for how to
proceed.
If would be helpful to dump all controller queues the next time you
get a deadlock. You should be able to get a snapshot of running
queries and transactions. We could match the ids with what is in your
debug log to see if anything wrong happened with any of these queries
or transactions.
Keep us posted with your progress,
Emmanuel
2007-01-12 09:26:03,567 DEBUG controller.virtualdatabase.ANGe
handleMessageSingleThreaded (class
org.continuent.sequoia.controller.virtualdatabase.protocol.DistributedStatementExecuteUpdate):
W 1657 569 insert into SubmissionContentBean (contentURL, contentLabel,
submission_id, id) values (?, ?, ?,
?)/<!%S|ftp://n140.cluster.larc.nasa.gov/47754ed2-707b-4fb8-8820-13ee92876143_LATIS1HOUR_Step_00000005_Sub_00000000_00000000|!%><!%S|47754ed2-707b-4fb8-8820-13ee92876143_LATIS1HOUR_Step_00000005_Sub_00000000_00000000|!%><!%S|254afd37-d4ac-4c08-9d4f-cfa99d4399c9|!%><!%S|437a636f-0db3-441a-80ac-dc3441baf60b|!%>
2007-01-12 09:26:03,567 DEBUG controller.virtualdatabase.ANGe
handleMessageSingleThreaded (class
org.continuent.sequoia.controller.virtualdatabase.protocol.DistributedStatementExecuteUpdate):
W 1657 569 insert into SubmissionContentBean (contentURL, contentLabel,
submission_id, id) values (?, ?, ?,
?)/<!%S|ftp://n140.cluster.larc.nasa.gov/47754ed2-707b-4fb8-8820-13ee92876143_LATIS1HOUR_Step_00000005_Sub_00000000_00000000|!%><!%S|47754ed2-707b-4fb8-8820-13ee92876143_LATIS1HOUR_Step_00000005_Sub_00000000_00000000|!%><!%S|254afd37-d4ac-4c08-9d4f-cfa99d4399c9|!%><!%S|437a636f-0db3-441a-80ac-dc3441baf60b|!%>
2007-01-12 09:26:03,567 DEBUG controller.virtualdatabase.ANGe
10.10.101.23:25323: Scheduling distributedRequest 1657 from
Member(address=/10.10.101.16:54230, uid=10.10.101.16:54230)
2007-01-12 09:26:03,567 DEBUG controller.virtualdatabase.ANGe
10.10.101.23:25323: Scheduling distributedRequest 1657 from
Member(address=/10.10.101.16:54230, uid=10.10.101.16:54230)
2007-01-12 09:26:03,567 INFO distributedvirtualdatabase.request.ANGe W 1657
569 insert into SubmissionContentBean (contentURL, contentLabel, submission_id,
id) values (?, ?, ?,
?)/<!%S|ftp://n140.cluster.larc.nasa.gov/47754ed2-707b-4fb8-8820-13ee92876143_LATIS1HOUR_Step_00000005_Sub_00000000_00000000|!%><!%S|47754ed2-707b-4fb8-8820-13ee92876143_LATIS1HOUR_Step_00000005_Sub_00000000_00000000|!%><!%S|254afd37-d4ac-4c08-9d4f-cfa99d4399c9|!%><!%S|437a636f-0db3-441a-80ac-dc3441baf60b|!%>
2007-01-12 09:26:03,567 DEBUG controller.virtualdatabase.ANGe
handleMessageMultiThreaded (class
org.continuent.sequoia.controller.virtualdatabase.protocol.DistributedStatementExecuteUpdate):
W 1657 569 insert into SubmissionContentBean (contentURL, contentLabel,
submission_id, id) values (?, ?, ?,
?)/<!%S|ftp://n140.cluster.larc.nasa.gov/47754ed2-707b-4fb8-8820-13ee92876143_LATIS1HOUR_Step_00000005_Sub_00000000_00000000|!%><!%S|47754ed2-707b-4fb8-8820-13ee92876143_LATIS1HOUR_Step_00000005_Sub_00000000_00000000|!%><!%S|254afd37-d4ac-4c08-9d4f-cfa99d4399c9|!%><!%S|437a636f-0db3-441a-80ac-dc3441baf60b|!%>
2007-01-12 09:26:03,567 DEBUG controller.virtualdatabase.ANGe
handleMessageMultiThreaded (class
org.continuent.sequoia.controller.virtualdatabase.protocol.DistributedStatementExecuteUpdate):
W 1657 569 insert into SubmissionContentBean (contentURL, contentLabel,
submission_id, id) values (?, ?, ?,
?)/<!%S|ftp://n140.cluster.larc.nasa.gov/47754ed2-707b-4fb8-8820-13ee92876143_LATIS1HOUR_Step_00000005_Sub_00000000_00000000|!%><!%S|47754ed2-707b-4fb8-8820-13ee92876143_LATIS1HOUR_Step_00000005_Sub_00000000_00000000|!%><!%S|254afd37-d4ac-4c08-9d4f-cfa99d4399c9|!%><!%S|437a636f-0db3-441a-80ac-dc3441baf60b|!%>
2007-01-12 09:26:03,568 DEBUG controller.virtualdatabase.ANGe
10.10.101.23:25323: Executing distributedRequest 1657 from
Member(address=/10.10.101.16:54230, uid=10.10.101.16:54230)
2007-01-12 09:26:03,568 DEBUG controller.virtualdatabase.ANGe
10.10.101.23:25323: Executing distributedRequest 1657 from
Member(address=/10.10.101.16:54230, uid=10.10.101.16:54230)
2007-01-12 09:26:03,568 DEBUG controller.RequestManager.ANGe Scheduling write
request (1657): insert into SubmissionContentBean (conte...
2007-01-12 09:26:03,568 DEBUG controller.RequestManager.ANGe Scheduling write
request (1657): insert into SubmissionContentBean (conte...
2007-01-12 09:26:03,569 DEBUG controller.RequestManager.ANGe Load balancing
write request (1657,569): insert into SubmissionContentBean (conte...
2007-01-12 09:26:03,569 DEBUG controller.RequestManager.ANGe Load balancing
write request (1657,569): insert into SubmissionContentBean (conte...
2007-01-12 09:26:03,606 DEBUG sequoia.controller.loadbalancer Request 1657
execute on 2 backends
2007-01-12 09:26:03,606 DEBUG sequoia.controller.loadbalancer Request 1657
execute on 2 backends
2007-01-12 09:26:03,617 DEBUG sequoia.controller.recoverylog INSERT INTO
RECOVERY VALUES(1580,user,insert into SubmissionContentBean (contentURL,
contentLabel, submission_id, id) values (?, ?, ?,
?),<!%S|ftp://n140.cluster.larc.nasa.gov/47754ed2-707b-4fb8-8820-13ee92876143_LATIS1HOUR_Step_00000005_Sub_00000000_00000000|!%><!%S|47754ed2-707b-4fb8-8820-13ee92876143_LATIS1HOUR_Step_00000005_Sub_00000000_00000000|!%><!%S|254afd37-d4ac-4c08-9d4f-cfa99d4399c9|!%><!%S|437a636f-0db3-441a-80ac-dc3441baf60b|!%>,T,569,1657,E,38,0)
2007-01-12 09:26:03,617 DEBUG sequoia.controller.recoverylog INSERT INTO
RECOVERY VALUES(1580,user,insert into SubmissionContentBean (contentURL,
contentLabel, submission_id, id) values (?, ?, ?,
?),<!%S|ftp://n140.cluster.larc.nasa.gov/47754ed2-707b-4fb8-8820-13ee92876143_LATIS1HOUR_Step_00000005_Sub_00000000_00000000|!%><!%S|47754ed2-707b-4fb8-8820-13ee92876143_LATIS1HOUR_Step_00000005_Sub_00000000_00000000|!%><!%S|254afd37-d4ac-4c08-9d4f-cfa99d4399c9|!%><!%S|437a636f-0db3-441a-80ac-dc3441baf60b|!%>,T,569,1657,E,38,0)
2007-01-12 09:26:03,617 ERROR controller.loadbalancer.RAIDb1 write request 1657
failed:
2007-01-12 09:26:03,617 ERROR controller.loadbalancer.RAIDb1 write request 1657
failed:
2007-01-12 09:26:03,620 WARN controller.RequestManager.ANGe SQLException while
executing distributed write request write request 1657 failed:
java.sql.SQLException: write request 1657 failed:
2007-01-12 09:26:03,620 WARN controller.RequestManager.ANGe SQLException while
executing distributed write request write request 1657 failed:
java.sql.SQLException: write request 1657 failed:
2007-01-12 09:26:04,783 DEBUG controller.virtualdatabase.ANGe
handleMessageSingleThreaded (class
org.continuent.sequoia.controller.virtualdatabase.protocol.DistributedStatementExecuteUpdate):
W 1659 571 insert into SubmissionContentBean (contentURL, contentLabel,
submission_id, id) values (?, ?, ?,
?)/<!%S|ftp://n140.cluster.larc.nasa.gov/b8b5c1c3-4430-4465-8853-f1a5f882154d.APES|!%><!%S|b8b5c1c3-4430-4465-8853-f1a5f882154d.APES|!%><!%S|b1d65046-1de4-4cf0-b5e1-db9fc46bed9a|!%><!%S|7ca3f9ea-5ab4-4c90-806a-5d6b55fe1e00|!%>
2007-01-12 09:26:04,783 DEBUG controller.virtualdatabase.ANGe
handleMessageSingleThreaded (class
org.continuent.sequoia.controller.virtualdatabase.protocol.DistributedStatementExecuteUpdate):
W 1659 571 insert into SubmissionContentBean (contentURL, contentLabel,
submission_id, id) values (?, ?, ?,
?)/<!%S|ftp://n140.cluster.larc.nasa.gov/b8b5c1c3-4430-4465-8853-f1a5f882154d.APES|!%><!%S|b8b5c1c3-4430-4465-8853-f1a5f882154d.APES|!%><!%S|b1d65046-1de4-4cf0-b5e1-db9fc46bed9a|!%><!%S|7ca3f9ea-5ab4-4c90-806a-5d6b55fe1e00|!%>
2007-01-12 09:26:04,783 DEBUG controller.virtualdatabase.ANGe
10.10.101.19:25323: Scheduling distributedRequest 1659 from
Member(address=/10.10.101.16:54230, uid=10.10.101.16:54230)
2007-01-12 09:26:04,783 DEBUG controller.virtualdatabase.ANGe
10.10.101.19:25323: Scheduling distributedRequest 1659 from
Member(address=/10.10.101.16:54230, uid=10.10.101.16:54230)
2007-01-12 09:26:04,783 INFO distributedvirtualdatabase.request.ANGe W 1659
571 insert into SubmissionContentBean (contentURL, contentLabel, submission_id,
id) values (?, ?, ?,
?)/<!%S|ftp://n140.cluster.larc.nasa.gov/b8b5c1c3-4430-4465-8853-f1a5f882154d.APES|!%><!%S|b8b5c1c3-4430-4465-8853-f1a5f882154d.APES|!%><!%S|b1d65046-1de4-4cf0-b5e1-db9fc46bed9a|!%><!%S|7ca3f9ea-5ab4-4c90-806a-5d6b55fe1e00|!%>
2007-01-12 09:26:04,783 DEBUG controller.virtualdatabase.ANGe
handleMessageMultiThreaded (class
org.continuent.sequoia.controller.virtualdatabase.protocol.DistributedStatementExecuteUpdate):
W 1659 571 insert into SubmissionContentBean (contentURL, contentLabel,
submission_id, id) values (?, ?, ?,
?)/<!%S|ftp://n140.cluster.larc.nasa.gov/b8b5c1c3-4430-4465-8853-f1a5f882154d.APES|!%><!%S|b8b5c1c3-4430-4465-8853-f1a5f882154d.APES|!%><!%S|b1d65046-1de4-4cf0-b5e1-db9fc46bed9a|!%><!%S|7ca3f9ea-5ab4-4c90-806a-5d6b55fe1e00|!%>
2007-01-12 09:26:04,783 DEBUG controller.virtualdatabase.ANGe
handleMessageMultiThreaded (class
org.continuent.sequoia.controller.virtualdatabase.protocol.DistributedStatementExecuteUpdate):
W 1659 571 insert into SubmissionContentBean (contentURL, contentLabel,
submission_id, id) values (?, ?, ?,
?)/<!%S|ftp://n140.cluster.larc.nasa.gov/b8b5c1c3-4430-4465-8853-f1a5f882154d.APES|!%><!%S|b8b5c1c3-4430-4465-8853-f1a5f882154d.APES|!%><!%S|b1d65046-1de4-4cf0-b5e1-db9fc46bed9a|!%><!%S|7ca3f9ea-5ab4-4c90-806a-5d6b55fe1e00|!%>
2007-01-12 09:26:04,783 DEBUG controller.virtualdatabase.ANGe
10.10.101.19:25323: Executing distributedRequest 1659 from
Member(address=/10.10.101.16:54230, uid=10.10.101.16:54230)
2007-01-12 09:26:04,783 DEBUG controller.virtualdatabase.ANGe
10.10.101.19:25323: Executing distributedRequest 1659 from
Member(address=/10.10.101.16:54230, uid=10.10.101.16:54230)
2007-01-12 09:26:04,784 DEBUG controller.RequestManager.ANGe Scheduling write
request (1659): insert into SubmissionContentBean (conte...
2007-01-12 09:26:04,784 DEBUG controller.RequestManager.ANGe Scheduling write
request (1659): insert into SubmissionContentBean (conte...
2007-01-12 09:26:04,785 DEBUG controller.RequestManager.ANGe Load balancing
write request (1659,571): insert into SubmissionContentBean (conte...
2007-01-12 09:26:04,785 DEBUG controller.RequestManager.ANGe Load balancing
write request (1659,571): insert into SubmissionContentBean (conte...
2007-01-12 09:26:04,785 DEBUG sequoia.controller.loadbalancer Request 1659
execute on 2 backends
2007-01-12 09:26:04,785 DEBUG sequoia.controller.loadbalancer Request 1659
execute on 2 backends
2007-01-12 09:26:04,804 DEBUG sequoia.controller.recoverylog INSERT INTO
RECOVERY VALUES(1584,user,insert into SubmissionContentBean (contentURL,
contentLabel, submission_id, id) values (?, ?, ?,
?),<!%S|ftp://n140.cluster.larc.nasa.gov/b8b5c1c3-4430-4465-8853-f1a5f882154d.APES|!%><!%S|b8b5c1c3-4430-4465-8853-f1a5f882154d.APES|!%><!%S|b1d65046-1de4-4cf0-b5e1-db9fc46bed9a|!%><!%S|7ca3f9ea-5ab4-4c90-806a-5d6b55fe1e00|!%>,T,571,1659,E,2,0)
2007-01-12 09:26:04,804 DEBUG sequoia.controller.recoverylog INSERT INTO
RECOVERY VALUES(1584,user,insert into SubmissionContentBean (contentURL,
contentLabel, submission_id, id) values (?, ?, ?,
?),<!%S|ftp://n140.cluster.larc.nasa.gov/b8b5c1c3-4430-4465-8853-f1a5f882154d.APES|!%><!%S|b8b5c1c3-4430-4465-8853-f1a5f882154d.APES|!%><!%S|b1d65046-1de4-4cf0-b5e1-db9fc46bed9a|!%><!%S|7ca3f9ea-5ab4-4c90-806a-5d6b55fe1e00|!%>,T,571,1659,E,2,0)
2007-01-12 09:26:04,809 ERROR controller.loadbalancer.RAIDb1 write request 1659
failed:
2007-01-12 09:26:04,809 ERROR controller.loadbalancer.RAIDb1 write request 1659
failed:
2007-01-12 09:26:04,812 WARN controller.RequestManager.ANGe SQLException while
executing distributed write request write request 1659 failed:
java.sql.SQLException: write request 1659 failed:
2007-01-12 09:26:04,812 WARN controller.RequestManager.ANGe SQLException while
executing distributed write request write request 1659 failed:
java.sql.SQLException: write request 1659 failed:
begin:vcard
fn:Jim Davis
n:Davis;Jim
org:Atmospheric Sciences Data Center;NASA Langley Research Center
adr;dom:;;;Hampton;VA
email;internet:[EMAIL PROTECTED]
title:SSAI, Systems Engineer
tel;work:757-864-7525
version:2.1
end:vcard
_______________________________________________
Sequoia mailing list
[email protected]
https://forge.continuent.org/mailman/listinfo/sequoia
