Thank you Dave, Ben and Mark for the responses so far. If you can help us further to figure out what's going on, we would be grateful. It has us stumped. We're doing a production pilot of Sync from our central server to 2 nearby child sites in the same district, after successful internal testing. One site is synched through a cell modem, and the notebook running OpenMRS at the other site is brought back each night to sync through our LAN. (We expect to deploy both modalities, so we are piloting them both.) If it goes well, we want to roll out Sync (in stages) to our full network consisting of child nodes at our district hospitals and grandchild nodes in the clinics of those districts. We would be replacing Remote Form Entry, which we are currently using.
Everything went fine in both directions between our root parent server and the two production pilot children for over a week. Then the root server mysteriously stopped entering new sync rows in the database, even though it continued to be used heavily. We didn't change anything. Now our children databases are out of sync with the root, and it looks like we have to do a complete backup/restore to get them synchronized again. We would love to expand Sync to all our other sites, but not if we might have more such problems!
Rebooting the parent server made it start synching again perfectly (except that the databases are now out of sync from during the failure period -- so any new data synched from server to child blocks the queue if it is dependent on data that missed being synched.) I don't expect there are any configuration problems since we never changed the configuration between working - not working - working again. We didn't change UUIDs, configuration, not a single thing.
Here is the configuration we have from parent to child (taken from parent server) and child to parent (taken from child server):
[From Parent server] Objects NOT to Send/Export:
org.openmrs.Form
org.openmrs.module.reporting
org.openmrs.scheduler.TaskDefinition
org.openmrs.Cohort
org.openmrs.scheduler
org.openmrs.FormField
org.openmrs.hl7
org.openmrs.Field
org.openmrs.reporting
org.openmrs.logic.token.TokenRegistration
org.openmrs.GlobalProperty
org.openmrs.module.formentry
org.openmrs.module.patientflags
org.openmrs.module.dataintegrity
org.openmrs.module.logic
org.openmrs.api.db.SerializedObject
org.openmrs.notification.Alert
org.openmrs.logic
[From Parent server] Objects NOT to Receive/Import:
org.openmrs.Form
org.openmrs.module.reporting
org.openmrs.scheduler.TaskDefinition
org.openmrs.Cohort
org.openmrs.module.htmlformentry
org.openmrs.scheduler
org.openmrs.FormField
org.openmrs.hl7
org.openmrs.Field
org.openmrs.reporting
org.openmrs.logic.token.TokenRegistration
org.openmrs.GlobalProperty
org.openmrs.Concept
org.openmrs.module.formentry
org.openmrs.module.patientflags
org.openmrs.module.dataintegrity
org.openmrs.module.logic
org.openmrs.api.db.SerializedObject
org.openmrs.notification.Alert
org.openmrs.logic
[From Child server] Objects NOT to Send/Export:
org.openmrs.Field
org.openmrs.Concept
org.openmrs.api.db.SerializedObject
org.openmrs.scheduler
org.openmrs.module.logic
org.openmrs.FormField
org.openmrs.Form
org.openmrs.logic
org.openmrs.module.reporting
org.openmrs.GlobalProperty
org.openmrs.api.db.SerializedObject
org.openmrs.module.htmlformentry
org.openmrs.Cohort
org.openmrs.module.formentry
org.openmrs.logic.token.TokenRegistration
org.openmrs.scheduler.TaskDefinition
org.openmrs.reporting
org.openmrs.hl7
org.openmrs.notification.Alert
[From Child server] Objects NOT to Receive/Import:
org.openmrs.Field
org.openmrs.api.db.SerializedObject
org.openmrs.scheduler
org.openmrs.module.logic
org.openmrs.FormField
org.openmrs.Form
org.openmrs.logic
org.openmrs.module.dataintergrity
org.openmrs.module.patientflags
org.openmrs.module.reporting
org.openmrs.GlobalProperty
org.openmrs.api.db.SerializedObject
org.openmrs.Cohort
org.openmrs.module.formentry
org.openmrs.logic.token.TokenRegistration
org.openmrs.scheduler.TaskDefinition
org.openmrs.reporting
org.openmrs.hl7
org.openmrs.notification.Alert
We can see new sync_record rows created on our parent system until one with record_id 23722, timestamped 2012-03-28 09:28:20. Then there were no more sync_record rows created until after we rebooted. The following catalina.out entries from our parent server seem to show the last two parent sync_records 23721 and 23722 before they stopped, followed by a successful sync operation from the child server Rabuor:
INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,720| In method SyncService.updateSyncRecord. Arguments: SyncRecord=SyncRecord(23721) contains org.openmrs.Privilege,
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,720| Exiting method updateSyncRecord
INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,722| In method SyncService.updateSyncRecord. Arguments: SyncRecord=SyncRecord(23722) contains org.openmrs.Privilege,
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,722| Exiting method updateSyncRecord
INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,724| In method SyncService.saveRemoteServer. Arguments: RemoteServer=RemoteServer(1): Rabuor,
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,726| Exiting method saveRemoteServer
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,847| Exiting method saveEncounter
INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,848| In method HL7Service.encounterCreated. Arguments: Encounter=Encounter: [1171134 Tue Mar 27 00:00:00 EAT 2012 ADULTRETURN null 51216 Revised Adult Follow-Up Form num Obs: [Obs #51749135, Obs #51749164, Obs #51749129, Obs #51749108, Obs #51749111, Obs #51749131, Obs #51749130, Obs #51749125, Obs #51749124, Obs #51749115, Obs #51749126, Obs #51749121, Obs #51749123, Obs #51749119, Obs #51749153, Obs #51749122, Obs #51749148, Obs #51749149, Obs #51749150, Obs #51749151, Obs #51749147, Obs #51749140, Obs #51749141] num Orders: 0 ],
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,849| Exiting method encounterCreated
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,849| Exiting method processHL7Message
I've looked through catalina.out after this point, but can find no clue as to why the sync_record rows stopped being recorded. Is there something in particular I should look for?
Meanwhile on the child servers, new sync_records continued to be entered, and they were all synchronized successfully with the parent. It was only the flow from parent to both children that was interrupted because no new sync_records were being generated by the parent.
When we realized that nothing was moving from parent to child, we checked sync_record and sync_server_record tables and noticed that no new records were being generated by the parent. We verified that all child encounters were present on the parent, but no parent encounters had been transferred to the children after the last sync_record. We restarted the OpenMRS Ubuntu server, and the sync_records started being added again. There has been no problem since in the last few days, except new records that were dependent on unsynchronized objects have also caused problems. New sync_record are now being added again on the parent, starting from just after the system reboot.
We would greatly appreciate any further advice you can give us to understand this problem.
Thanks,
Jim
---------------------------------
At 11:32 PM 02-04-12, Ben Wolfe wrote:
Also see https://wiki.openmrs.org/display/docs/Sync+Module+Technical+Overview for a description of the tables.
To add to Dave's questions:
Did you set the "do not sync" objects to anything likw * or org.openmrs.* per chance?
Ben
On Mon, Apr 2, 2012 at 4:16 PM, Dave Thomas <[email protected]> wrote:
- Hi. If you're not seeing any new records in sync_server_record
on the parent, you're in trouble. This is the table that is used to
keep track of what records have propagated to what child servers.
Without rows being inserted into this table, no data will ever propagate
to the children. In your sync overview page on the parent, can you
actually see the child servers, and are they configured correctly?
- Did you change the server uuids at all, either on the children or on the parent?
- On the parent, the state field in the sync_record table isn't important. Its the state field in the sync_server_record that governs propagation.
- d
- On Mon, Apr 2, 2012 at 8:05 AM, Erick Mugoma <[email protected] > wrote:
- Hi Ben and Mark
- Am just wondering what may or may not be happening with the sync_server_record and sync_record tables on the central/parent server.
- Plse help me understand something, I expect that every time a new record is created on the parent, the record details are logged to the above tables and you can even use these tables to track if the record was committed, sent, new or failed. Why is it that when I do a query to check if there any records in the sync_server_record table I can't see any record for example I should be able to see today's records (02-04-12) since the clean -up was last done on 31 march 2012. Any I deas on what is not Happenning?
- On the parent sync_record table, the state field is not being updated. The state column value is 'NEW' for all the records. I expect that the state should change once the record is committed to all the children. Any reason why this is not so?
- When I look at the same table on the child servers, the state column value is 'COMMITTED'
- Looking at the History Changes on the UI What I see are old changes like last updated is 28-03-2012. That is last week wed when we have new records being entered even today (02-04-12) and should be able to see them under history changes. Any Explanation why I can't see the newest/latest updates.
- Thanks
- Erick
- Am just wondering what may or may not be happening with the sync_server_record and sync_record tables on the central/parent server.
- Click here to unsubscribe from OpenMRS Developers' mailing list
- Did you change the server uuids at all, either on the children or on the parent?
Click here to unsubscribe from OpenMRS Developers' mailing list
Click here to unsubscribe from OpenMRS Developers' mailing list
