If the children were successfully propagating data to the parent (can you verify that you can look up data on the parent after sync stopped working that originated from a child server), but no rows were inserted into sync_server_record, i'd guess that this is an issue with core around interceptors failing to register (or be re-registered). Were there any failed module loads, or module stopping and starting, on the day sync stopped working?
d On Thu, Apr 5, 2012 at 7:12 AM, Mark Goodrich <[email protected]> wrote: > Could you check to see if sync_import records were still being created > after 3-28-2012?**** > > ** ** > > Could you create a ticket for this bug and upload the entire catalina.out > log for 3-28-2012 and I will see if I can identify anything out of the > ordinary? Do you know if you installed any new modules on the parent > server at this time?**** > > ** ** > > Mark**** > > ** ** > > ** ** > > *From:* [email protected] [mailto:[email protected]] *On Behalf Of *Jim Grace > *Sent:* Thursday, April 05, 2012 4:44 AM > *To:* [email protected] > *Subject:* Re: [OPENMRS-DEV] SYNC Module Question**** > > ** ** > > Hi All, > > Thank you Dave, Ben and Mark for the responses so far. If you can help us > further to figure out what's going on, we would be grateful. It has us > stumped. We're doing a production pilot of Sync from our central server to > 2 nearby child sites in the same district, after successful internal > testing. One site is synched through a cell modem, and the notebook running > OpenMRS at the other site is brought back each night to sync through our > LAN. (We expect to deploy both modalities, so we are piloting them both.) > If it goes well, we want to roll out Sync (in stages) to our full network > consisting of child nodes at our district hospitals and grandchild nodes in > the clinics of those districts. We would be replacing Remote Form Entry, > which we are currently using. > > Everything went fine in both directions between our root parent server and > the two production pilot children for over a week. Then the root server > mysteriously stopped entering new sync rows in the database, even though it > continued to be used heavily. We didn't change anything. Now our children > databases are out of sync with the root, and it looks like we have to do a > complete backup/restore to get them synchronized again. We would love to > expand Sync to all our other sites, but not if we might have more such > problems! > > Rebooting the parent server made it start synching again perfectly (except > that the databases are now out of sync from during the failure period -- so > any new data synched from server to child blocks the queue if it is > dependent on data that missed being synched.) I don't expect there are any > configuration problems since we never changed the configuration between > working - not working - working again. We didn't change UUIDs, > configuration, not a single thing. > > Here is the configuration we have from parent to child (taken from parent > server) and child to parent (taken from child server): > > [From Parent server] Objects NOT to Send/Export: > org.openmrs.Form > org.openmrs.module.reporting > org.openmrs.scheduler.TaskDefinition > org.openmrs.Cohort > org.openmrs.scheduler > org.openmrs.FormField > org.openmrs.hl7 > org.openmrs.Field > org.openmrs.reporting > org.openmrs.logic.token.TokenRegistration > org.openmrs.GlobalProperty > org.openmrs.module.formentry > org.openmrs.module.patientflags > org.openmrs.module.dataintegrity > org.openmrs.module.logic > org.openmrs.api.db.SerializedObject > org.openmrs.notification.Alert > org.openmrs.logic > > [From Parent server] Objects NOT to Receive/Import: > org.openmrs.Form > org.openmrs.module.reporting > org.openmrs.scheduler.TaskDefinition > org.openmrs.Cohort > org.openmrs.module.htmlformentry > org.openmrs.scheduler > org.openmrs.FormField > org.openmrs.hl7 > org.openmrs.Field > org.openmrs.reporting > org.openmrs.logic.token.TokenRegistration > org.openmrs.GlobalProperty > org.openmrs.Concept > org.openmrs.module.formentry > org.openmrs.module.patientflags > org.openmrs.module.dataintegrity > org.openmrs.module.logic > org.openmrs.api.db.SerializedObject > org.openmrs.notification.Alert > org.openmrs.logic > > [From Child server] Objects NOT to Send/Export: > org.openmrs.Field > org.openmrs.Concept > org.openmrs.api.db.SerializedObject > org.openmrs.scheduler > org.openmrs.module.logic > org.openmrs.FormField > org.openmrs.Form > org.openmrs.logic > org.openmrs.module.reporting > org.openmrs.GlobalProperty > org.openmrs.api.db.SerializedObject > org.openmrs.module.htmlformentry > org.openmrs.Cohort > org.openmrs.module.formentry > org.openmrs.logic.token.TokenRegistration > org.openmrs.scheduler.TaskDefinition > org.openmrs.reporting > org.openmrs.hl7 > org.openmrs.notification.Alert > > [From Child server] Objects NOT to Receive/Import: > org.openmrs.Field > org.openmrs.api.db.SerializedObject > org.openmrs.scheduler > org.openmrs.module.logic > org.openmrs.FormField > org.openmrs.Form > org.openmrs.logic > org.openmrs.module.dataintergrity > org.openmrs.module.patientflags > org.openmrs.module.reporting > org.openmrs.GlobalProperty > org.openmrs.api.db.SerializedObject > org.openmrs.Cohort > org.openmrs.module.formentry > org.openmrs.logic.token.TokenRegistration > org.openmrs.scheduler.TaskDefinition > org.openmrs.reporting > org.openmrs.hl7 > org.openmrs.notification.Alert > > We can see new sync_record rows created on our parent system until one > with record_id 23722, timestamped 2012-03-28 09:28:20. Then there were no > more sync_record rows created until after we rebooted. The following > catalina.out entries from our parent server seem to show the last two > parent sync_records 23721 and 23722 before they stopped, followed by a > successful sync operation from the child server Rabuor: > > INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,720| In method > SyncService.updateSyncRecord. Arguments: SyncRecord=SyncRecord(23721) > contains org.openmrs.Privilege, > INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,720| Exiting method > updateSyncRecord > INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,722| In method > SyncService.updateSyncRecord. Arguments: SyncRecord=SyncRecord(23722) > contains org.openmrs.Privilege, > INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,722| Exiting method > updateSyncRecord > INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,724| In method > SyncService.saveRemoteServer. Arguments: RemoteServer=RemoteServer(1): > Rabuor, > INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,726| Exiting method > saveRemoteServer > INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,847| Exiting method > saveEncounter > INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,848| In method > HL7Service.encounterCreated. Arguments: Encounter=Encounter: [1171134 Tue > Mar 27 00:00:00 EAT 2012 ADULTRETURN null 51216 Revised Adult Follow-Up > Form num Obs: [Obs #51749135, Obs #51749164, Obs #51749129, Obs #51749108, > Obs #51749111, Obs #51749131, Obs #51749130, Obs #51749125, Obs #51749124, > Obs #51749115, Obs #51749126, Obs #51749121, Obs #51749123, Obs #51749119, > Obs #51749153, Obs #51749122, Obs #51749148, Obs #51749149, Obs #51749150, > Obs #51749151, Obs #51749147, Obs #51749140, Obs #51749141] num Orders: 0 > ], > INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,849| Exiting method > encounterCreated > INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,849| Exiting method > processHL7Message > > I've looked through catalina.out after this point, but can find no clue as > to why the sync_record rows stopped being recorded. Is there something in > particular I should look for? > > Meanwhile on the child servers, new sync_records continued to be entered, > and they were all synchronized successfully with the parent. It was only > the flow from parent to both children that was interrupted because no new > sync_records were being generated by the parent. > > When we realized that nothing was moving from parent to child, we checked > sync_record and sync_server_record tables and noticed that no new records > were being generated by the parent. We verified that all child encounters > were present on the parent, but no parent encounters had been transferred > to the children after the last sync_record. We restarted the OpenMRS Ubuntu > server, and the sync_records started being added again. There has been no > problem since in the last few days, except new records that were dependent > on unsynchronized objects have also caused problems. New sync_record are > now being added again on the parent, starting from just after the system > reboot. > > We would greatly appreciate any further advice you can give us to > understand this problem. > > Thanks, > Jim > > --------------------------------- > > At 11:32 PM 02-04-12, Ben Wolfe wrote: > Also see > https://wiki.openmrs.org/display/docs/Sync+Module+Technical+Overview for > a description of the tables. > > To add to Dave's questions: > Did you set the "do not sync" objects to anything likw * or org.openmrs.* > per chance? > > Ben > > On Mon, Apr 2, 2012 at 4:16 PM, Dave Thomas <[email protected]> wrote: *** > * > > Hi. If you're not seeing any new records in sync_server_record on the > parent, you're in trouble. This is the table that is used to keep track of > what records have propagated to what child servers. Without rows being > inserted into this table, no data will ever propagate to the children. In > your sync overview page on the parent, can you actually see the child > servers, and are they configured correctly? **** > > Did you change the server uuids at all, either on the children or on the > parent? **** > > On the parent, the state field in the sync_record table isn't important. > Its the state field in the sync_server_record that governs propagation.*** > * > > d > > **** > > On Mon, Apr 2, 2012 at 8:05 AM, Erick Mugoma <[email protected] > > wrote: **** > > Hi Ben and Mark **** > > Am just wondering what may or may not be happening with the > sync_server_record and sync_record tables on the central/parent server. ** > ** > > Plse help me understand something, I expect that every time a new record > is created on the parent, the record details are logged to the above tables > and you can even use these tables to track if the record was committed, > sent, new or failed. Why is it that when I do a query to check if there > any records in the sync_server_record table I can't see any record for > example I should be able to see today's records (02-04-12) since the clean > -up was last done on 31 march 2012. Any I deas on what is not Happenning? > **** > > On the parent sync_record table, the state field is not being updated. The > state column value is 'NEW' for all the records. I expect that the state > should change once the record is committed to all the children. Any reason > why this is not so? **** > > When I look at the same table on the child servers, the state column value > is 'COMMITTED' **** > > Looking at the History Changes on the UI What I see are old changes like > last updated is 28-03-2012. That is last week wed when we have new records > being entered even today (02-04-12) and should be able to see them under > history changes. Any Explanation why I can't see the newest/latest updates. > **** > > Thanks**** > > Erick > > **** > ------------------------------ > > Click here to > unsubscribe<[email protected]?body=SIGNOFF%20openmrs-devel-l>from > OpenMRS Developers' mailing list > > **** > ------------------------------ > > Click here to > unsubscribe<[email protected]?body=SIGNOFF%20openmrs-devel-l>from > OpenMRS Developers' mailing list > **** > ------------------------------ > > Click here to > unsubscribe<[email protected]?body=SIGNOFF%20openmrs-devel-l>from > OpenMRS Developers' mailing list > **** > ------------------------------ > Click here to > unsubscribe<[email protected]?body=SIGNOFF%20openmrs-devel-l>from > OpenMRS Developers' mailing list > _________________________________________ To unsubscribe from OpenMRS Developers' mailing list, send an e-mail to [email protected] with "SIGNOFF openmrs-devel-l" in the body (not the subject) of your e-mail. [mailto:[email protected]?body=SIGNOFF%20openmrs-devel-l]
