Re: Randomization in hastd(8) synchronization thread
On 5/21/2011 8:57 AM, Pawel Jakub Dawidek wrote: Hmm, hastd keeps separate bitmap for synchronization. It is stored in am_syncmap field. Blocks that are dirtied during regular writes should not effect on synchronization bitmap and synchronization progress. Possibly, but this policy is not very appropriate for slow links. Please see my other patch, which only dirties blocks on write. Regards, -- Maksym Sobolyev Sippy Software, Inc. Internet Telephony (VoIP) Experts Tel: +1-646-651-1110 Fax: +1-866-857-6942 Web: http://www.sippysoft.com MSN: sa...@sippysoft.com Skype: SippySoft ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Randomization in hastd(8) synchronization thread
On Tue, May 17, 2011 at 12:39:19PM -0700, Maxim Sobolev wrote: > Hi Pawel, > > I am trying to use hastd(8) over slow links and one problem is > apparent right now - current approach with synchronizing content > sequentially is not working in this case. What happens is that hastd > hits the first frequently updated block and cannot make any progress > anymore. In my case I have 30GB of dirty space to be synchronized > over just 1mbps uplink. > > The quick fix that I've applied is randomization in the block > selection code. This way eventually all least used blocks will be > synchronized, leaving only hot ones dirty. More effective approach > would be to use some kind of LRU selection algorithm, but > statistical approach would work just as good in this case. > > Please review the patch below: > > http://sobomax.sippysoft.com/activemap.c.diff Hmm, hastd keeps separate bitmap for synchronization. It is stored in am_syncmap field. Blocks that are dirtied during regular writes should not effect on synchronization bitmap and synchronization progress. -- Pawel Jakub Dawidek http://www.wheelsystems.com FreeBSD committer http://www.FreeBSD.org Am I Evil? Yes, I Am! http://yomoli.com pgp9xz8wcUwuQ.pgp Description: PGP signature
Randomization in hastd(8) synchronization thread
Hi Pawel, I am trying to use hastd(8) over slow links and one problem is apparent right now - current approach with synchronizing content sequentially is not working in this case. What happens is that hastd hits the first frequently updated block and cannot make any progress anymore. In my case I have 30GB of dirty space to be synchronized over just 1mbps uplink. The quick fix that I've applied is randomization in the block selection code. This way eventually all least used blocks will be synchronized, leaving only hot ones dirty. More effective approach would be to use some kind of LRU selection algorithm, but statistical approach would work just as good in this case. Please review the patch below: http://sobomax.sippysoft.com/activemap.c.diff The next thing to make it usable is to make "async" mode working. I think simple support for that mode can be easily implemented by not sending write request to the remote note at all, but instead just doing it locally and kicking the synchronization thread to do it's magic in the background. I hope to follow up with the patch soon. Regards, -- Maksym Sobolyev Sippy Software, Inc. Internet Telephony (VoIP) Experts Tel: +1-646-651-1110 Fax: +1-866-857-6942 Web: http://www.sippysoft.com MSN: sa...@sippysoft.com Skype: SippySoft ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Making hastd working over WAN links (was: Randomization in hastd(8) synchronization thread)
On 5/17/2011 1:28 PM, Maxim Sobolev wrote: The next thing to make it usable is to make "async" mode working. I think simple support for that mode can be easily implemented by not sending write request to the remote note at all, but instead just doing it locally and kicking the synchronization thread to do it's magic in the background. I hope to follow up with the patch soon. Here is a proof of concept path, which simply fails any non-synchronization requests in the send thread when in the async mode. This is non-optimal, as it would cause additional latency in the write path when the send thread is busy with synchronization requests. But it works for me, making it possible to use my virtual machine while synchronizing the disk image. http://sobomax.sippysoft.com/primary.c.diff -Maxim ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Randomization in hastd(8) synchronization thread
Hi Pawel, I am trying to use hastd(8) over slow links and one problem is apparent right now - current approach with synchronizing content sequentially is not working in this case. What happens is that hastd hits the first frequently updated block and cannot make any progress anymore. In my case I have 30GB of dirty space to be synchronized over just 1mbps uplink. The quick fix that I've applied is randomization in the block selection code. This way eventually all least used blocks will be synchronized, leaving only hot ones dirty. More effective approach would be to use some kind of LRU selection algorithm, but statistical approach would work just as good in this case. Please review the patch below: http://sobomax.sippysoft.com/activemap.c.diff The next thing to make it usable is to make "async" mode working. I think simple support for that mode can be easily implemented by not sending write request to the remote note at all, but instead just doing it locally and kicking the synchronization thread to do it's magic in the background. I hope to follow up with the patch soon. -Maxim ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
