Thanks for replying Chuck.
This code is importing EOs from a text file in a background process. This code
is used a *lot* and maybe after importing millions of EOs across multiple
files/users in a week, I might get this error. Retrying an import will usually
result in the same error. A restart of the instance will then usually "fix" the
problem.
In development, I have used the same code to import 5 million records without
this error, so the inability to reproduce this in development and the fact that
it may happen once a week or once a month in deployment makes it a "important
but not urgent" thorn in my side.
The Setup: A pool of eight OSCs are reserved for background thread EOF
processing to ensure no "delays" for normal page EOF activity. EditingContexts
are recycled every 250 records imported. One instance is available as a
different app name and generally has one user on that instance for production
processing work and that does the bulk of large imports (mailing list data for
fulfillment of online customer orders). This error always (IIRC) happens in
that admin instance, so the admin app instance is not under heavy concurrent
user load at all (though the database server might be).
Here is where I catch the error (the IllegalStateException) and throw/report
it. This is where the OSCs and ECs info is coming from
(ERX*.outstandingLockDescription()). Any suggestions for additional state I
should try and capture here to troubleshoot this:
try {
ec.saveChanges();
} catch (EOGeneralAdaptorException e) {
throw new
RuntimeException(WKEOUtils.generalAdaptorExceptionInfo(e));
} catch (IllegalStateException e) {
// This is to try and gather enough info to solve the
PITA occasional "failed to provide primary keys" error
String msg = ExceptionUtils.getRootCauseMessage(e);
StringBuilder b = new StringBuilder();
b.append("Root Cause: ").append(msg).append("; ");
b.append(ERXEC.outstandingLockDescription());
b.append(ERXObjectStoreCoordinator.outstandingLockDescription());
throw new RuntimeException(b.toString(), e);
}
Meanwhile, I have disabled ERXJDBCAdaptor on the admin instance as a first
"process of elimination" step, and I may have to wait for a month to see if we
get this error to even decide if that was a *probable* contributor.
The entire group of machines (about 8 including DB/apache/other and a few WO)
is on its own subnet with a fairly pricey Cisco router and while we have not
had noticeable network issues at the time this happens or any other time.
I will park it for today since I have other "urgent" tasks today and get back
to it after vacation, however one thing I have not done is retried the save. So
I don't want to change too much in one go and then not know which change solved
the issue. We will see how it does not with ERXJDBCAdaptor out of the equation.
What I am really hoping for out of this discussion is a list of "state to
capture and log" - I could even capture a bunch of state before I try and save,
and then log that and more state if I get the error since it almost always
happens in this one place in the code.
-Kieran
On Jul 13, 2010, at 1:15 PM, Chuck Hill wrote:
>
> On Jul 13, 2010, at 9:45 AM, Kieran Kelleher wrote:
>
>> Trying to track this down is like a needle in a haystack and I can never
>> reproduce it in development. Any ideas on a strategy to capture the root
>> cause?
>>
>> Error processing file:
>> /tmp/JGZ36CHBY17ZGH7FH518CNCBY_BdrWithSuppression.txt. Erroneous results for
>> import are: isSuccess = false;
>> messages = null;
>> userInfo = null
>> java.lang.RuntimeException: Root Cause: IllegalStateException: Adaptor
>> er.extensions.jdbc.erxjdbcadap...@2ff88688 failed to provide new primary
>> keys for entity 'CTDataRecord'; Currently 0 active ECs : {})No open editing
>> contexts (of 0)0 active ObjectStoreCoordinators : {})No open
>> ObjectStoreCoordinator (of 0)
>
> It seems odd that there are not OSCs and no ECs. Is that perhaps an
> indication of the root problem? Are these getting garbage collected out from
> under your tasks? Or are those messages just sketchy?
>
> The next thing I would do is to take a look in ERXJDBCAdaptor, JDBCAdaptor,
> and the plugin for your database to see why PK generation might fail.
>
> How often does this happen? Does it match up with any network outages
> between the app and database servers?
>
>
> Chuck
>
>
>> at
>> wk.webobjects.eof.ImportEnterpriseObjectData.processFile(ImportEnterpriseObjectData.java:156)
>> at
>> wk.webobjects.eof.ImportEnterpriseObjectData.run(ImportEnterpriseObjectData.java:364)
>> at
>> wk.webobjects.eof.ImportEnterpriseObjectData.call(ImportEnterpriseObjectData.java:371)
>> at
>> wk.webobjects.eof.ImportEnterpriseObjectData.call(ImportEnterpriseObjectData.java:1)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:637)
>> Caused by: java.lang.IllegalStateException: Adaptor
>> er.extensions.jdbc.erxjdbcadap...@2ff88688 failed to provide new primary
>> keys for entity 'CTDataRecord'
>> at
>> com.webobjects.eoaccess.EODatabaseContext.prepareForSaveWithCoordinator(EODatabaseContext.java:5885)
>> at
>> com.webobjects.eocontrol.EOObjectStoreCoordinator.saveChangesInEditingContext(EOObjectStoreCoordinator.java:409)
>> at
>> com.webobjects.eocontrol.EOEditingContext.saveChanges(EOEditingContext.java:3226)
>> at er.extensions.eof.ERXEC._saveChanges(ERXEC.java:1085)
>> at er.extensions.eof.ERXEC.saveChanges(ERXEC.java:1007)
>> at
>> wk.webobjects.eof.ImportEnterpriseObjectData.processFile(ImportEnterpriseObjectData.java:147)
>> ... 8 more
>>
>> _______________________________________________
>> Do not post admin requests to the list. They will be ignored.
>> Webobjects-dev mailing list ([email protected])
>> Help/Unsubscribe/Update your Subscription:
>> http://lists.apple.com/mailman/options/webobjects-dev/chill%40global-village.net
>>
>> This email sent to [email protected]
>
> --
> Chuck Hill Senior Consultant / VP Development
>
> Practical WebObjects - for developers who want to increase their overall
> knowledge of WebObjects or who are trying to solve specific problems.
> http://www.global-village.net/products/practical_webobjects
>
>
>
>
>
>
>
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list ([email protected])
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com
This email sent to [email protected]
