Hi Deyan, Pankaj,
Could you share your work done so far with the community ?
I agree that we shouyd try to use and maintain EntitySync.
But for the volume reason Deyan outlined and in the meantime I'm sure his
solution would be appreciated too!
I suggest to create Jira or/and wiki pages. I'm ready to help on this.
Thanks
Jacques
From: "Jacques Le Roux" <jacques.le.r...@les7arts.com>
Thanks for comment Deyan,
I will try Pankaj Jain's tips in a 1st time
Jacques
From: "Deyan Tsvetanov" <deyan.tsveta...@ittconsult.com>
Hi Jacques,
no, I didn't work on fixing the entity sync issues.
Instead I did my own sync tool using db triggers and stored procedures.
The entity sync could be fixed, but it would require like two - three
weeks of work.
I am sorry I didn't post any details, i was kind of busy because of a
missed deadline :)
The entity sync is kind of useless in this state imho.
Reasons:
1) Does not do any fail over - if the sync process is interrupted for
any reason - like network outage, or ofbiz restart, or system freeze
it will be considered as a success sync and won't continue from the
point it has failed. Could be fixed though :)
2) Pretty slow. The entities are transfered over the network using RMI.
For transferring 5000 product records ( along with their relations -
product_price, product_assoc, etc )
it took over 2 hours using pretty good hard and 50 mbits network
connection.
3) The example of entities to sync in the sync howto notes will never
work. There will be records with duplicate PKs :)
The solution for those issues imho are:
1) Refactor a bit the entity sync code so the client remembers the last
successful sync state. This would fix the fail over issues. currently
the server ( MCS ) remembers the
sync status for each client.
2) Don't use RMI :) It would be much faster if the entities that are
going to be transfered in order to get synced are serialized to XML for
example and
transferred over the network in a text format. RMI is easy to use but
has bad performance. Just to give you an example:
about 5000 products ( which is probably about 20 000 entities in total
to pull ) takes 2 hours on a 50 mbits connection. What if we had a
pretty huge supermarket
which has 100 000 sales per day ( this means 100 000 sales orders ,
average 50 order_items per order, 100 000 payments, 100 000 invoices,
100 000 invoice items ).
It would require an entire night to sync - if we are lucky :)
3) I can provide the complete list of tables needed to get synced for a
POS solution, including fix of the issue with the duplicate PKs :)
However for our current project we've used triggers which create a
transaction log for push on the POS databases and a transaction log for
pull on the MCS instance.
Pulling those 5000 products along with all the related records takes
about 20 secs on the same hardware. I've configured a crontab job which
invoces the sync postgre stored procedures
every 5 minutes and it works perfectly. A status table is maintained on
the MCS so we can check regularly if a POS hasn't synced for let's say 1
hour - this means either network outage or some other problem.
Yes, I know this solution is DB dependent but if the ofbiz sync
framework issues are fixed and we use the list of tables to sync i've
managed to create something useful could come out of it :)
So - i hope my e-mail answers your question, if you want to know more -
don't hesitate :)
Cheers,
Deyan
-----Original Message-----
From: Jacques Le Roux <jacques.le.r...@les7arts.com>
Reply-to: "Jacques Le Roux" <jacques.le.r...@les7arts.com>
To: Deyan Tsvetanov <deyan.tsveta...@ittconsult.com>,
user@ofbiz.apache.org
Subject: Re: EntitySync issues & suggestions ( was EntitySync RMI
error )
Date: Mon, 26 Oct 2009 08:34:51 +0100
Hi Deyan,
Did you work on this ? I will need to use EntitySync soon and I know you
are not the 1st to complain about it (Si Chen did for instance). On the
other hand, some seems to have used it successfully...
Thanks
Jacques
----- Original Message -----
From: Deyan Tsvetanov
To: Jacques Le Roux ; user@ofbiz.apache.org
Sent: Sunday, October 04, 2009 10:57 AM
Subject: EntitySync issues & suggestions ( was EntitySync RMI
error )
Hi Jacques and list,
After playing for almost 2 weeks with the entity sync feature
I've come up with the conclusion that it still needs
some work to get ready for production environment. Currently it
is unreliable, and useless :(
- There is a bug in the design. The MCS remembers the sync
status and currently reached sync timestamp for each client.
This is not a good approach imho. Why ?
The client asks the server: have you got something for me ?
Next: The MCS checks the last successful sync timestamp for that
particular client and selects the entities that should be passed
to the client. After successfully transferring the entities to
the client over RMI the MCS assumes that the sync is successful
and updates the last successful sync timestamp.
The problem is that the client does not confirm the success of
the operation. If for some reason the client is unable to update
the entities ( either shutdown,
power failure, DB error or whatever ) the MCS isn't aware of
that and the next sync will not include the missed entities.
There is no way to fix the skipped
entities but re-creating the client's database from zero.
That is the main issue actually. It could be easily solved
however. The most easy ways are:
1) The client will remember the sync timestamp and will include
it as an argument when asking the server for pull sync. This way
the client will
update the last successful sync timestamp after storing each
entity. The server does not have to remember anything regarding
the client's sync status.
2) After each sync session the server will wait for confirmation
from the client before updating the last successful sync
timestamp.
If not updated than the sync process is considered to be
unsuccessful and the timestamp will not be updated.
I personally prefer approach 1) as in case of network error,
power failure or whatever error the sync process will continue
from the place it has stopped.
- There are some swallowed exceptions. If a network problem
occurs during the sync process the server's status remains "sync
running" and the client is
not able to sync anymore until the status is reset by an
operator.
- The last issue is the performance. Using RMI is an easy
approach from developer's perspective but it is very very slow.
I had a database with 5000 products. The initial sync took about
2 hours. The hardware is pretty good, the network connection
between the MCS and the client ofbiz
is about 50 mbits. What if we have 100 000 sales per day ? The
sync process will take probably the entire night :)
Here I would suggest transferring GenericEntity instances over
RMI to be removed. Instead a regular XML could be used. Either
over RMI or SOAP - does not matter.
So in general the picture would be:
1) Server does not remember any timestamp for any client.
2) Client calls the pull entity sync RMI ( or soap ) method and
passes the entity sync ID and the timestamp of the last
successful sync as arguments. Including the timezone of
course :)
3) The server generates an XML file for all the entities in that
entity sync group using the timestamp provided by the client.
4) Client starts updating the entities described in the received
XML line by line. After each entity is updated successfully the
timestamp is also updated ( on the client side - the
server does not remember any client timestamps ).
5) If a failure occurs in the middle of the update process what
we'll have is a sync process completed to like 50%. With the
correct timestamp. When the next sync
job starts the client will use the timestamp and the server will
generate a new XML from the place the sync has failed. Of course
an optimization could be used -
the XML will be stored in a new db table so the client can
continue storing the entities without asking the server to
generate something he has already done.
Some improvements that could be also useful:
1) The client ofbiz instances have sequence-id-prefix. So when
the server is accepting push syncs it could also check the
prefixes of the PKs and disallow entities without the correct
prefix in the primary key. That's just a cheap insurance against
errors :) We don't want store A to make sales on behalf of store
B just because somebody made an error while configuring the
pos ofbiz instance.
2) A time synchronization mechanism could be also implemented.
In general when syncing there is a requirement that the server's
and client's clocks are in sync ( ntp ). We don't want
to set the system clock in ofbiz, but we could implement a check
if the clocks are in sync and refuse to start the
synchronization in certain cases.
3) A new column could be added to all the entities:
isFromEntitySync : flag. This way we could avoid cases in which
the client is trying to push records to the server which were
actually pulled by the server. Yes, I know the entity's created
timestamp should help us avoid such cases, but when the sync
process is interrupted things like that could happen.
Otherwise for our current project we've implemented a DB level
synchronization - using triggers for INSERT, UPDATE and DELETE
and a stored procedure which syncs the clients with a MCS.
It works much ( tens of times ) faster than ofbiz entity sync .
But it is DB dependent. So after getting comments and
suggestions to this e-mail I would like to invest some time in
fixing all those issues with the ofbiz entity sync and switch to
it when it's ready :)
Thanks for your time,
Deyan
-----Original Message-----
From: Jacques Le Roux <jacques.le.r...@les7arts.com>
Reply-to: "Jacques Le Roux" <jacques.le.r...@les7arts.com>
To: user@ofbiz.apache.org
Subject: Re: EntitySync RMI error
Date: Sun, 27 Sep 2009 17:02:57 +0200
Hi Deyan,
It would be very valuable if you could post an article on wiki about your experience with this. Don't worry about where to
put it, I
will eventually take care to put it under FAQ...
TIA
Jacques
From: "Deyan Tsvetanov" <deyan.tsveta...@ittconsult.com>
>I found an issue though:
>
> If the connection gets dropped during sync :
> - the client ( POS ) prints connection reset by peer
> - the server ( MCS ) sync status remains to running.
> - the next sync does not start because MCS complains there is another
> sync running already.
>
> "Reset run status" from webtools -> entity sync status
> helps.
>
>
> I'll investigate further and log a bug, although the solution seems
> pretty simple - looks like a swallowed IOException,
> which should be handled by re-setting the sync status.
>
> CHeers,
> DEyan
>
> -----Original Message-----
> From: Deyan Tsvetanov <deyan.tsveta...@ittconsult.com>
> Reply-to: user@ofbiz.apache.org
> To: user@ofbiz.apache.org
> Subject: Re: EntitySync RMI error
> Date: Sat, 26 Sep 2009 11:22:55 +0300
>
>
> Got it !
>
> Seems that the previous exception was coming from the MCS ( server ) .
> When I stopped the server I started getting the exception I kind of
like
> more :)
>
>
> Message: Exception calling remote pull and report EntitySync service
> with name: remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.Gene
> ricServiceException: RMI Error (Connection refused to host:
> 192.168.1.100; nested exception is:
> java.net.ConnectException: Connection refused: connect)
>
> So the problem was at the server side - RMIIF env var in startofbiz.sh
> wasn't set , so ofbiz was trying to get RMI host IP by resolving the
> hostname .
>
> The misleading things for me was that:
> 1) no any sign in the MCS logs that somebody is trying to connect and
> the MCS itself can not connect to its own RMI registry.
> 2) no any difference in the exceptions text on the POS ( client ) side:
> the 2 exceptions ( local one - can not connect to 192.168.1.100 and
the
> remote one - can not connect to 127.0.0.1 )
> look the same :)
>
> So issue is solved,
> sorry for bothering :)
>
> -- deyan
>
> -----Original Message-----
> From: Deyan Tsvetanov <deyan.tsveta...@ittconsult.com>
> Reply-to: user@ofbiz.apache.org
> To: user@ofbiz.apache.org
> Subject: Re: EntitySync RMI error
> Date: Sat, 26 Sep 2009 09:52:55 +0300
>
>
> During startup I get:
>
> 2009-09-26 08:46:31,905 (default-invoker-Thread-10) [
> AbstractEngine.java:73 :INFO ] Loaded Service Locations :
> [main-rmi=rmi://127.0.0.
> 1:1099/RMIDispatcher,
> main-http=http://127.0.0.1:8080/webtools/control/httpService,
> entity-sync-rmi=rmi://192.168.1.100:1099/RMIDispatcher,
>
entity-sync-http=http://192.168.1.100:8080/webtools/control/httpService,
> rita-rmi=rmi://127.0.0.1:1099/RMIDispatcher, eedcc-test=http://127.
> 0.0.1:8080/webtools/control/httpService]
>
> entity-sync-rmi seems to be ok ...
>
>
>
>
> -----Original Message-----
> From: Deyan Tsvetanov <deyan.tsveta...@ittconsult.com>
> Reply-to: user@ofbiz.apache.org
> To: user@ofbiz.apache.org
> Subject: EntitySync RMI error
> Date: Sat, 26 Sep 2009 09:22:23 +0300
>
>
> Hi guys,
>
> I'm trying to configure RMI entity sync. I'm following
> http://docs.ofbiz.org/display/OFBIZ/Sync+Setup+Notes+and+Example
>
> What I've done so far:
>
> 1) entity-sync-rmi to rmi://192.168.1.100:1099/RMIDispatcher
>
> 2) set RMIIF=-Djava.rmi.server.hostname=127.0.0.1
> ( as per the example ).
>
> 3) I've imported the entity sync groups, SandJobs, etc.
>
> However when the sync starts ( on the POS instance ) I get the
following
> error:
>
> Exception calling remote pull and report EntitySync service with name:
> remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.GenericServic
> eException: RMI Invocation Error (Connection refused to host:
127.0.0.1;
> nested exception is:
> java.net.ConnectException: Connection refused: connect)
> Exception: org.ofbiz.service.GenericServiceException
> Message: RMI Invocation Error (Connection refused to host: 127.0.0.1;
> nested exception is:
> java.net.ConnectException: Connection refused: connect)
> ---- cause
> ---------------------------------------------------------------------
> Exception: java.rmi.ConnectException
> Message: Connection refused to host: 127.0.0.1; nested exception is:
> java.net.ConnectException: Connection refused: connect
> ---- cause
> ---------------------------------------------------------------------
> Exception: java.net.ConnectException
> Message: Connection refused: connect
> ---- stack trace
> ---------------------------------------------------------------
>
>
> It insists connecting to 127.0.0.1 no matter what I type in
> serviceengine.xml.
> Any help would be appreciated :)
>
> Thanks in advance,
> Deyan
>
