Hi Jacques, no, I didn't work on fixing the entity sync issues. Instead I did my own sync tool using db triggers and stored procedures.
The entity sync could be fixed, but it would require like two - three weeks of work. I am sorry I didn't post any details, i was kind of busy because of a missed deadline :) The entity sync is kind of useless in this state imho. Reasons: 1) Does not do any fail over - if the sync process is interrupted for any reason - like network outage, or ofbiz restart, or system freeze it will be considered as a success sync and won't continue from the point it has failed. Could be fixed though :) 2) Pretty slow. The entities are transfered over the network using RMI. For transferring 5000 product records ( along with their relations - product_price, product_assoc, etc ) it took over 2 hours using pretty good hard and 50 mbits network connection. 3) The example of entities to sync in the sync howto notes will never work. There will be records with duplicate PKs :) The solution for those issues imho are: 1) Refactor a bit the entity sync code so the client remembers the last successful sync state. This would fix the fail over issues. currently the server ( MCS ) remembers the sync status for each client. 2) Don't use RMI :) It would be much faster if the entities that are going to be transfered in order to get synced are serialized to XML for example and transferred over the network in a text format. RMI is easy to use but has bad performance. Just to give you an example: about 5000 products ( which is probably about 20 000 entities in total to pull ) takes 2 hours on a 50 mbits connection. What if we had a pretty huge supermarket which has 100 000 sales per day ( this means 100 000 sales orders , average 50 order_items per order, 100 000 payments, 100 000 invoices, 100 000 invoice items ). It would require an entire night to sync - if we are lucky :) 3) I can provide the complete list of tables needed to get synced for a POS solution, including fix of the issue with the duplicate PKs :) However for our current project we've used triggers which create a transaction log for push on the POS databases and a transaction log for pull on the MCS instance. Pulling those 5000 products along with all the related records takes about 20 secs on the same hardware. I've configured a crontab job which invoces the sync postgre stored procedures every 5 minutes and it works perfectly. A status table is maintained on the MCS so we can check regularly if a POS hasn't synced for let's say 1 hour - this means either network outage or some other problem. Yes, I know this solution is DB dependent but if the ofbiz sync framework issues are fixed and we use the list of tables to sync i've managed to create something useful could come out of it :) So - i hope my e-mail answers your question, if you want to know more - don't hesitate :) Cheers, Deyan -----Original Message----- From: Jacques Le Roux <[email protected]> Reply-to: "Jacques Le Roux" <[email protected]> To: Deyan Tsvetanov <[email protected]>, [email protected] Subject: Re: EntitySync issues & suggestions ( was EntitySync RMI error ) Date: Mon, 26 Oct 2009 08:34:51 +0100 Hi Deyan, Did you work on this ? I will need to use EntitySync soon and I know you are not the 1st to complain about it (Si Chen did for instance). On the other hand, some seems to have used it successfully... Thanks Jacques ----- Original Message ----- From: Deyan Tsvetanov To: Jacques Le Roux ; [email protected] Sent: Sunday, October 04, 2009 10:57 AM Subject: EntitySync issues & suggestions ( was EntitySync RMI error ) Hi Jacques and list, After playing for almost 2 weeks with the entity sync feature I've come up with the conclusion that it still needs some work to get ready for production environment. Currently it is unreliable, and useless :( - There is a bug in the design. The MCS remembers the sync status and currently reached sync timestamp for each client. This is not a good approach imho. Why ? The client asks the server: have you got something for me ? Next: The MCS checks the last successful sync timestamp for that particular client and selects the entities that should be passed to the client. After successfully transferring the entities to the client over RMI the MCS assumes that the sync is successful and updates the last successful sync timestamp. The problem is that the client does not confirm the success of the operation. If for some reason the client is unable to update the entities ( either shutdown, power failure, DB error or whatever ) the MCS isn't aware of that and the next sync will not include the missed entities. There is no way to fix the skipped entities but re-creating the client's database from zero. That is the main issue actually. It could be easily solved however. The most easy ways are: 1) The client will remember the sync timestamp and will include it as an argument when asking the server for pull sync. This way the client will update the last successful sync timestamp after storing each entity. The server does not have to remember anything regarding the client's sync status. 2) After each sync session the server will wait for confirmation from the client before updating the last successful sync timestamp. If not updated than the sync process is considered to be unsuccessful and the timestamp will not be updated. I personally prefer approach 1) as in case of network error, power failure or whatever error the sync process will continue from the place it has stopped. - There are some swallowed exceptions. If a network problem occurs during the sync process the server's status remains "sync running" and the client is not able to sync anymore until the status is reset by an operator. - The last issue is the performance. Using RMI is an easy approach from developer's perspective but it is very very slow. I had a database with 5000 products. The initial sync took about 2 hours. The hardware is pretty good, the network connection between the MCS and the client ofbiz is about 50 mbits. What if we have 100 000 sales per day ? The sync process will take probably the entire night :) Here I would suggest transferring GenericEntity instances over RMI to be removed. Instead a regular XML could be used. Either over RMI or SOAP - does not matter. So in general the picture would be: 1) Server does not remember any timestamp for any client. 2) Client calls the pull entity sync RMI ( or soap ) method and passes the entity sync ID and the timestamp of the last successful sync as arguments. Including the timezone of course :) 3) The server generates an XML file for all the entities in that entity sync group using the timestamp provided by the client. 4) Client starts updating the entities described in the received XML line by line. After each entity is updated successfully the timestamp is also updated ( on the client side - the server does not remember any client timestamps ). 5) If a failure occurs in the middle of the update process what we'll have is a sync process completed to like 50%. With the correct timestamp. When the next sync job starts the client will use the timestamp and the server will generate a new XML from the place the sync has failed. Of course an optimization could be used - the XML will be stored in a new db table so the client can continue storing the entities without asking the server to generate something he has already done. Some improvements that could be also useful: 1) The client ofbiz instances have sequence-id-prefix. So when the server is accepting push syncs it could also check the prefixes of the PKs and disallow entities without the correct prefix in the primary key. That's just a cheap insurance against errors :) We don't want store A to make sales on behalf of store B just because somebody made an error while configuring the pos ofbiz instance. 2) A time synchronization mechanism could be also implemented. In general when syncing there is a requirement that the server's and client's clocks are in sync ( ntp ). We don't want to set the system clock in ofbiz, but we could implement a check if the clocks are in sync and refuse to start the synchronization in certain cases. 3) A new column could be added to all the entities: isFromEntitySync : flag. This way we could avoid cases in which the client is trying to push records to the server which were actually pulled by the server. Yes, I know the entity's created timestamp should help us avoid such cases, but when the sync process is interrupted things like that could happen. Otherwise for our current project we've implemented a DB level synchronization - using triggers for INSERT, UPDATE and DELETE and a stored procedure which syncs the clients with a MCS. It works much ( tens of times ) faster than ofbiz entity sync . But it is DB dependent. So after getting comments and suggestions to this e-mail I would like to invest some time in fixing all those issues with the ofbiz entity sync and switch to it when it's ready :) Thanks for your time, Deyan -----Original Message----- From: Jacques Le Roux <[email protected]> Reply-to: "Jacques Le Roux" <[email protected]> To: [email protected] Subject: Re: EntitySync RMI error Date: Sun, 27 Sep 2009 17:02:57 +0200 Hi Deyan, It would be very valuable if you could post an article on wiki about your experience with this. Don't worry about where to put it, I will eventually take care to put it under FAQ... TIA Jacques From: "Deyan Tsvetanov" <[email protected]> >I found an issue though: > > If the connection gets dropped during sync : > - the client ( POS ) prints connection reset by peer > - the server ( MCS ) sync status remains to running. > - the next sync does not start because MCS complains there is another > sync running already. > > "Reset run status" from webtools -> entity sync status > helps. > > > I'll investigate further and log a bug, although the solution seems > pretty simple - looks like a swallowed IOException, > which should be handled by re-setting the sync status. > > CHeers, > DEyan > > -----Original Message----- > From: Deyan Tsvetanov <[email protected]> > Reply-to: [email protected] > To: [email protected] > Subject: Re: EntitySync RMI error > Date: Sat, 26 Sep 2009 11:22:55 +0300 > > > Got it ! > > Seems that the previous exception was coming from the MCS ( server ) . > When I stopped the server I started getting the exception I kind of like > more :) > > > Message: Exception calling remote pull and report EntitySync service > with name: remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.Gene > ricServiceException: RMI Error (Connection refused to host: > 192.168.1.100; nested exception is: > java.net.ConnectException: Connection refused: connect) > > So the problem was at the server side - RMIIF env var in startofbiz.sh > wasn't set , so ofbiz was trying to get RMI host IP by resolving the > hostname . > > The misleading things for me was that: > 1) no any sign in the MCS logs that somebody is trying to connect and > the MCS itself can not connect to its own RMI registry. > 2) no any difference in the exceptions text on the POS ( client ) side: > the 2 exceptions ( local one - can not connect to 192.168.1.100 and the > remote one - can not connect to 127.0.0.1 ) > look the same :) > > So issue is solved, > sorry for bothering :) > > -- deyan > > -----Original Message----- > From: Deyan Tsvetanov <[email protected]> > Reply-to: [email protected] > To: [email protected] > Subject: Re: EntitySync RMI error > Date: Sat, 26 Sep 2009 09:52:55 +0300 > > > During startup I get: > > 2009-09-26 08:46:31,905 (default-invoker-Thread-10) [ > AbstractEngine.java:73 :INFO ] Loaded Service Locations : > [main-rmi=rmi://127.0.0. > 1:1099/RMIDispatcher, > main-http=http://127.0.0.1:8080/webtools/control/httpService, > entity-sync-rmi=rmi://192.168.1.100:1099/RMIDispatcher, > entity-sync-http=http://192.168.1.100:8080/webtools/control/httpService, > rita-rmi=rmi://127.0.0.1:1099/RMIDispatcher, eedcc-test=http://127. > 0.0.1:8080/webtools/control/httpService] > > entity-sync-rmi seems to be ok ... > > > > > -----Original Message----- > From: Deyan Tsvetanov <[email protected]> > Reply-to: [email protected] > To: [email protected] > Subject: EntitySync RMI error > Date: Sat, 26 Sep 2009 09:22:23 +0300 > > > Hi guys, > > I'm trying to configure RMI entity sync. I'm following > http://docs.ofbiz.org/display/OFBIZ/Sync+Setup+Notes+and+Example > > What I've done so far: > > 1) entity-sync-rmi to rmi://192.168.1.100:1099/RMIDispatcher > > 2) set RMIIF=-Djava.rmi.server.hostname=127.0.0.1 > ( as per the example ). > > 3) I've imported the entity sync groups, SandJobs, etc. > > However when the sync starts ( on the POS instance ) I get the following > error: > > Exception calling remote pull and report EntitySync service with name: > remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.GenericServic > eException: RMI Invocation Error (Connection refused to host: 127.0.0.1; > nested exception is: > java.net.ConnectException: Connection refused: connect) > Exception: org.ofbiz.service.GenericServiceException > Message: RMI Invocation Error (Connection refused to host: 127.0.0.1; > nested exception is: > java.net.ConnectException: Connection refused: connect) > ---- cause > --------------------------------------------------------------------- > Exception: java.rmi.ConnectException > Message: Connection refused to host: 127.0.0.1; nested exception is: > java.net.ConnectException: Connection refused: connect > ---- cause > --------------------------------------------------------------------- > Exception: java.net.ConnectException > Message: Connection refused: connect > ---- stack trace > --------------------------------------------------------------- > > > It insists connecting to 127.0.0.1 no matter what I type in > serviceengine.xml. > Any help would be appreciated :) > > Thanks in advance, > Deyan >
