On Sun, Jan 06, 2013 at 08:00:51PM +0700, Henrik Sarvell wrote:
> OK so how would you deal with a situation where you have the need to
> quickly increment people's balances like I mentioned previously but at the
> same time you have another process that has to update a lot of objects by
> fetching information from many others?
> This second process will take roughly one minute to complete from start to
> finish and will not update +User in any way.
I would do it the normal, "safe" way, i.e. with 'inc!>'. Note that the
way you proposed doesn't have so much less overhead, I think, because it
still uses the 'upd' argument to 'commit', which triggers communication
with other processes, and 'commit' itself, which does a low-level
locking of the DB.
> If I have understood things correctly simply doing dbSync -> work -> commit
> in the second process won't work here because it will block the balance
Yes, but you can control this, depending on how many updates are done in
the 'work' between dbSync and commit.
We've discussed this in IRC, so for other readers here is what I do
usually in such cases:
(... do one update step ...)
(at (0 . 1000) (commit 'upd) (dbSync)) )
The value of 1000 is an examply, I would try something between 100 and
With that, after every 1000th update step other processes get a chance
to grab the lock in the (dbSync) after the 'commit'.
> Another option would be to do it in a loop and use put!> which will only
> initiate the sync at the time of each update which should not block the
> balance updates for too long.
Right. This would be optimal in terms of giving freedom to other
processes, but it does only one single change in the 'put!>', and thus
the whole update might take too long.
The above sync at every 1000th step allows for a good compromise.
> The question then is how much overhead does this cause when it comes to the
> balance updates in your experience? If significant is it possible to
I would not call this "overhead". It is just so that the "quick"
operation of incrementing the balance may have to wait too long if the
second process does too many changes in a single transaction.
So the problem is not the 'inc!>'. It just sits and waits until it can
do its job, and is then done quite quickly. It is the large update
'work' which may grab the DB for too long periods.
> somehow solve the issue of these two processes creating collateral damage
> to each other so to speak?
If you can isolate the balance (not like in your last example, where two
processes incremented and decremented the balance at the same time), and
make absolutely sure that only once process caches the object at a given
time, you could take the risk and do the incrementing/decrementing
without synchronization, with just (commit).
One way might be to have a single process take care of that,
communicating values to/from other processes with 'tell', so that no one
else needs to access these objects. But that's more complicated than
the straight-forward way.