Jim & Erick- Did you notice Mike's email to the dev and implementer's lists yesterday? We discovered a serious bug where installing the Metadata Sharing module disables the sync interceptor, which would result in behavior like you were seeing. Do you have the Metadata Sharing module installed?
This bug caused us to lose a few days of syncing for our Haiti deployment, though we were able to recover with some quick, hacky code that we wrote... were you able to recover? Take care, Mark From: [email protected] [mailto:[email protected]] On Behalf Of Jim Grace Sent: Thursday, April 05, 2012 4:44 AM To: [email protected] Subject: Re: [OPENMRS-DEV] SYNC Module Question Hi All, Thank you Dave, Ben and Mark for the responses so far. If you can help us further to figure out what's going on, we would be grateful. It has us stumped. We're doing a production pilot of Sync from our central server to 2 nearby child sites in the same district, after successful internal testing. One site is synched through a cell modem, and the notebook running OpenMRS at the other site is brought back each night to sync through our LAN. (We expect to deploy both modalities, so we are piloting them both.) If it goes well, we want to roll out Sync (in stages) to our full network consisting of child nodes at our district hospitals and grandchild nodes in the clinics of those districts. We would be replacing Remote Form Entry, which we are currently using. Everything went fine in both directions between our root parent server and the two production pilot children for over a week. Then the root server mysteriously stopped entering new sync rows in the database, even though it continued to be used heavily. We didn't change anything. Now our children databases are out of sync with the root, and it looks like we have to do a complete backup/restore to get them synchronized again. We would love to expand Sync to all our other sites, but not if we might have more such problems! Rebooting the parent server made it start synching again perfectly (except that the databases are now out of sync from during the failure period -- so any new data synched from server to child blocks the queue if it is dependent on data that missed being synched.) I don't expect there are any configuration problems since we never changed the configuration between working - not working - working again. We didn't change UUIDs, configuration, not a single thing. Here is the configuration we have from parent to child (taken from parent server) and child to parent (taken from child server): [From Parent server] Objects NOT to Send/Export: org.openmrs.Form org.openmrs.module.reporting org.openmrs.scheduler.TaskDefinition org.openmrs.Cohort org.openmrs.scheduler org.openmrs.FormField org.openmrs.hl7 org.openmrs.Field org.openmrs.reporting org.openmrs.logic.token.TokenRegistration org.openmrs.GlobalProperty org.openmrs.module.formentry org.openmrs.module.patientflags org.openmrs.module.dataintegrity org.openmrs.module.logic org.openmrs.api.db.SerializedObject org.openmrs.notification.Alert org.openmrs.logic [From Parent server] Objects NOT to Receive/Import: org.openmrs.Form org.openmrs.module.reporting org.openmrs.scheduler.TaskDefinition org.openmrs.Cohort org.openmrs.module.htmlformentry org.openmrs.scheduler org.openmrs.FormField org.openmrs.hl7 org.openmrs.Field org.openmrs.reporting org.openmrs.logic.token.TokenRegistration org.openmrs.GlobalProperty org.openmrs.Concept org.openmrs.module.formentry org.openmrs.module.patientflags org.openmrs.module.dataintegrity org.openmrs.module.logic org.openmrs.api.db.SerializedObject org.openmrs.notification.Alert org.openmrs.logic [From Child server] Objects NOT to Send/Export: org.openmrs.Field org.openmrs.Concept org.openmrs.api.db.SerializedObject org.openmrs.scheduler org.openmrs.module.logic org.openmrs.FormField org.openmrs.Form org.openmrs.logic org.openmrs.module.reporting org.openmrs.GlobalProperty org.openmrs.api.db.SerializedObject org.openmrs.module.htmlformentry org.openmrs.Cohort org.openmrs.module.formentry org.openmrs.logic.token.TokenRegistration org.openmrs.scheduler.TaskDefinition org.openmrs.reporting org.openmrs.hl7 org.openmrs.notification.Alert [From Child server] Objects NOT to Receive/Import: org.openmrs.Field org.openmrs.api.db.SerializedObject org.openmrs.scheduler org.openmrs.module.logic org.openmrs.FormField org.openmrs.Form org.openmrs.logic org.openmrs.module.dataintergrity org.openmrs.module.patientflags org.openmrs.module.reporting org.openmrs.GlobalProperty org.openmrs.api.db.SerializedObject org.openmrs.Cohort org.openmrs.module.formentry org.openmrs.logic.token.TokenRegistration org.openmrs.scheduler.TaskDefinition org.openmrs.reporting org.openmrs.hl7 org.openmrs.notification.Alert We can see new sync_record rows created on our parent system until one with record_id 23722, timestamped 2012-03-28 09:28:20. Then there were no more sync_record rows created until after we rebooted. The following catalina.out entries from our parent server seem to show the last two parent sync_records 23721 and 23722 before they stopped, followed by a successful sync operation from the child server Rabuor: INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,720| In method SyncService.updateSyncRecord. Arguments: SyncRecord=SyncRecord(23721) contains org.openmrs.Privilege, INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,720| Exiting method updateSyncRecord INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,722| In method SyncService.updateSyncRecord. Arguments: SyncRecord=SyncRecord(23722) contains org.openmrs.Privilege, INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,722| Exiting method updateSyncRecord INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,724| In method SyncService.saveRemoteServer. Arguments: RemoteServer=RemoteServer(1): Rabuor, INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,726| Exiting method saveRemoteServer INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,847| Exiting method saveEncounter INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,848| In method HL7Service.encounterCreated. Arguments: Encounter=Encounter: [1171134 Tue Mar 27 00:00:00 EAT 2012 ADULTRETURN null 51216 Revised Adult Follow-Up Form num Obs: [Obs #51749135, Obs #51749164, Obs #51749129, Obs #51749108, Obs #51749111, Obs #51749131, Obs #51749130, Obs #51749125, Obs #51749124, Obs #51749115, Obs #51749126, Obs #51749121, Obs #51749123, Obs #51749119, Obs #51749153, Obs #51749122, Obs #51749148, Obs #51749149, Obs #51749150, Obs #51749151, Obs #51749147, Obs #51749140, Obs #51749141] num Orders: 0 ], INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,849| Exiting method encounterCreated INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,849| Exiting method processHL7Message I've looked through catalina.out after this point, but can find no clue as to why the sync_record rows stopped being recorded. Is there something in particular I should look for? Meanwhile on the child servers, new sync_records continued to be entered, and they were all synchronized successfully with the parent. It was only the flow from parent to both children that was interrupted because no new sync_records were being generated by the parent. When we realized that nothing was moving from parent to child, we checked sync_record and sync_server_record tables and noticed that no new records were being generated by the parent. We verified that all child encounters were present on the parent, but no parent encounters had been transferred to the children after the last sync_record. We restarted the OpenMRS Ubuntu server, and the sync_records started being added again. There has been no problem since in the last few days, except new records that were dependent on unsynchronized objects have also caused problems. New sync_record are now being added again on the parent, starting from just after the system reboot. We would greatly appreciate any further advice you can give us to understand this problem. Thanks, Jim --------------------------------- At 11:32 PM 02-04-12, Ben Wolfe wrote: Also see https://wiki.openmrs.org/display/docs/Sync+Module+Technical+Overview for a description of the tables. To add to Dave's questions: Did you set the "do not sync" objects to anything likw * or org.openmrs.* per chance? Ben On Mon, Apr 2, 2012 at 4:16 PM, Dave Thomas <[email protected]<mailto:[email protected]>> wrote: Hi. If you're not seeing any new records in sync_server_record on the parent, you're in trouble. This is the table that is used to keep track of what records have propagated to what child servers. Without rows being inserted into this table, no data will ever propagate to the children. In your sync overview page on the parent, can you actually see the child servers, and are they configured correctly? Did you change the server uuids at all, either on the children or on the parent? On the parent, the state field in the sync_record table isn't important. Its the state field in the sync_server_record that governs propagation. d On Mon, Apr 2, 2012 at 8:05 AM, Erick Mugoma <[email protected]<mailto:[email protected]> > wrote: Hi Ben and Mark Am just wondering what may or may not be happening with the sync_server_record and sync_record tables on the central/parent server. Plse help me understand something, I expect that every time a new record is created on the parent, the record details are logged to the above tables and you can even use these tables to track if the record was committed, sent, new or failed. Why is it that when I do a query to check if there any records in the sync_server_record table I can't see any record for example I should be able to see today's records (02-04-12) since the clean -up was last done on 31 march 2012. Any I deas on what is not Happenning? On the parent sync_record table, the state field is not being updated. The state column value is 'NEW' for all the records. I expect that the state should change once the record is committed to all the children. Any reason why this is not so? When I look at the same table on the child servers, the state column value is 'COMMITTED' Looking at the History Changes on the UI What I see are old changes like last updated is 28-03-2012. That is last week wed when we have new records being entered even today (02-04-12) and should be able to see them under history changes. Any Explanation why I can't see the newest/latest updates. Thanks Erick ________________________________ Click here to unsubscribe<mailto:[email protected]?body=SIGNOFF%20openmrs-devel-l> from OpenMRS Developers' mailing list ________________________________ Click here to unsubscribe<mailto:[email protected]?body=SIGNOFF%20openmrs-devel-l> from OpenMRS Developers' mailing list ________________________________ Click here to unsubscribe<mailto:[email protected]?body=SIGNOFF%20openmrs-devel-l> from OpenMRS Developers' mailing list _________________________________________ To unsubscribe from OpenMRS Developers' mailing list, send an e-mail to [email protected] with "SIGNOFF openmrs-devel-l" in the body (not the subject) of your e-mail. [mailto:[email protected]?body=SIGNOFF%20openmrs-devel-l]
