On Jul 14, 2010, at 6:39 AM, Kieran Kelleher wrote: > Thanks for replying Chuck. > > This code is importing EOs from a text file in a background process. This > code is used a *lot* and maybe after importing millions of EOs across > multiple files/users in a week, I might get this error. Retrying an import > will usually result in the same error. A restart of the instance will then > usually "fix" the problem.
Have you checked for running out of memory?
> In development, I have used the same code to import 5 million records without
> this error, so the inability to reproduce this in development and the fact
> that it may happen once a week or once a month in deployment makes it a
> "important but not urgent" thorn in my side.
>
> The Setup: A pool of eight OSCs are reserved for background thread EOF
> processing to ensure no "delays" for normal page EOF activity.
> EditingContexts are recycled every 250 records imported. One instance is
> available as a different app name and generally has one user on that instance
> for production processing work and that does the bulk of large imports
> (mailing list data for fulfillment of online customer orders). This error
> always (IIRC) happens in that admin instance, so the admin app instance is
> not under heavy concurrent user load at all (though the database server might
> be).
>
> Here is where I catch the error (the IllegalStateException) and throw/report
> it. This is where the OSCs and ECs info is coming from
> (ERX*.outstandingLockDescription()). Any suggestions for additional state I
> should try and capture here to troubleshoot this:
>
> try {
> ec.saveChanges();
> } catch (EOGeneralAdaptorException e) {
> throw new
> RuntimeException(WKEOUtils.generalAdaptorExceptionInfo(e));
> } catch (IllegalStateException e) {
> // This is to try and gather enough info to solve the
> PITA occasional "failed to provide primary keys" error
> String msg = ExceptionUtils.getRootCauseMessage(e);
> StringBuilder b = new StringBuilder();
> b.append("Root Cause: ").append(msg).append("; ");
> b.append(ERXEC.outstandingLockDescription());
>
> b.append(ERXObjectStoreCoordinator.outstandingLockDescription());
> throw new RuntimeException(b.toString(), e);
> }
>
> Meanwhile, I have disabled ERXJDBCAdaptor on the admin instance as a first
> "process of elimination" step, and I may have to wait for a month to see if
> we get this error to even decide if that was a *probable* contributor.
I am not sure that disabling this is the path to proceed on. I'd be taking a
look in there, in EOF, and in your database plugin to identify any potential
failure points and then try to get logging in there to log if the failure
happens.
> The entire group of machines (about 8 including DB/apache/other and a few WO)
> is on its own subnet with a fairly pricey Cisco router and while we have not
> had noticeable network issues at the time this happens or any other time.
Have you checked in the system logs for momentary network problems?
> I will park it for today since I have other "urgent" tasks today and get back
> to it after vacation, however one thing I have not done is retried the save.
> So I don't want to change too much in one go and then not know which change
> solved the issue. We will see how it does not with ERXJDBCAdaptor out of the
> equation.
>
> What I am really hoping for out of this discussion is a list of "state to
> capture and log" - I could even capture a bunch of state before I try and
> save, and then log that and more state if I get the error since it almost
> always happens in this one place in the code.
The symptoms look more to me like a low level database, JDBC, or network error
that causes an exception deep in EOF. EOF is not handling this properly and
getting into a confused state.
Chuck
> On Jul 13, 2010, at 1:15 PM, Chuck Hill wrote:
>
>>
>> On Jul 13, 2010, at 9:45 AM, Kieran Kelleher wrote:
>>
>>> Trying to track this down is like a needle in a haystack and I can never
>>> reproduce it in development. Any ideas on a strategy to capture the root
>>> cause?
>>>
>>> Error processing file:
>>> /tmp/JGZ36CHBY17ZGH7FH518CNCBY_BdrWithSuppression.txt. Erroneous results
>>> for import are: isSuccess = false;
>>> messages = null;
>>> userInfo = null
>>> java.lang.RuntimeException: Root Cause: IllegalStateException: Adaptor
>>> er.extensions.jdbc.erxjdbcadap...@2ff88688 failed to provide new primary
>>> keys for entity 'CTDataRecord'; Currently 0 active ECs : {})No open editing
>>> contexts (of 0)0 active ObjectStoreCoordinators : {})No open
>>> ObjectStoreCoordinator (of 0)
>>
>> It seems odd that there are not OSCs and no ECs. Is that perhaps an
>> indication of the root problem? Are these getting garbage collected out
>> from under your tasks? Or are those messages just sketchy?
>>
>> The next thing I would do is to take a look in ERXJDBCAdaptor, JDBCAdaptor,
>> and the plugin for your database to see why PK generation might fail.
>>
>> How often does this happen? Does it match up with any network outages
>> between the app and database servers?
>>
>>
>> Chuck
>>
>>
>>> at
>>> wk.webobjects.eof.ImportEnterpriseObjectData.processFile(ImportEnterpriseObjectData.java:156)
>>> at
>>> wk.webobjects.eof.ImportEnterpriseObjectData.run(ImportEnterpriseObjectData.java:364)
>>> at
>>> wk.webobjects.eof.ImportEnterpriseObjectData.call(ImportEnterpriseObjectData.java:371)
>>> at
>>> wk.webobjects.eof.ImportEnterpriseObjectData.call(ImportEnterpriseObjectData.java:1)
>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:637)
>>> Caused by: java.lang.IllegalStateException: Adaptor
>>> er.extensions.jdbc.erxjdbcadap...@2ff88688 failed to provide new primary
>>> keys for entity 'CTDataRecord'
>>> at
>>> com.webobjects.eoaccess.EODatabaseContext.prepareForSaveWithCoordinator(EODatabaseContext.java:5885)
>>> at
>>> com.webobjects.eocontrol.EOObjectStoreCoordinator.saveChangesInEditingContext(EOObjectStoreCoordinator.java:409)
>>> at
>>> com.webobjects.eocontrol.EOEditingContext.saveChanges(EOEditingContext.java:3226)
>>> at er.extensions.eof.ERXEC._saveChanges(ERXEC.java:1085)
>>> at er.extensions.eof.ERXEC.saveChanges(ERXEC.java:1007)
>>> at
>>> wk.webobjects.eof.ImportEnterpriseObjectData.processFile(ImportEnterpriseObjectData.java:147)
>>> ... 8 more
>>>
>>> _______________________________________________
>>> Do not post admin requests to the list. They will be ignored.
>>> Webobjects-dev mailing list ([email protected])
>>> Help/Unsubscribe/Update your Subscription:
>>> http://lists.apple.com/mailman/options/webobjects-dev/chill%40global-village.net
>>>
>>> This email sent to [email protected]
>>
>> --
>> Chuck Hill Senior Consultant / VP Development
>>
>> Practical WebObjects - for developers who want to increase their overall
>> knowledge of WebObjects or who are trying to solve specific problems.
>> http://www.global-village.net/products/practical_webobjects
>>
>>
>>
>>
>>
>>
>>
>
--
Chuck Hill Senior Consultant / VP Development
Practical WebObjects - for developers who want to increase their overall
knowledge of WebObjects or who are trying to solve specific problems.
http://www.global-village.net/products/practical_webobjects
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list ([email protected]) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to [email protected]
