Ok, confirmed. Letting dbench work on different subfolders makes it work fine. In terms of log messages: I would love to monitor for problems with gluster. Failing self heals in the logs might not really be a problem per-se, any suggestions of what else I could parse the logs for?
Cheers, Marc On 19.02.2013, at 23:32, Anand Avati <[email protected]> wrote: > They were just self-heal double-checks performed in the lookup path, as we > perform the initial lookup unlocked for performance reasons. Those lines by > themselves do not indicate anything harmful (yet). I admit the log messages > could have been better though! > > Avati > > On Tue, Feb 19, 2013 at 2:20 PM, Marc Seeger <[email protected]> wrote: > I'll give it a try. Isn't it somewhat concerning that this would heal to > missing gfids and failing self-heals though? > > On 19.02.2013, at 21:22, Anand Avati <[email protected]> wrote: > >> We run dbench in the pre-commit script for every patch. Looks like you are >> running dbench on the same directory from multiple clients. Is that even >> supposed to work? Did you instead lunch to run each dbench within their own >> subdirectory? >> >> Avati >> >> On Tue, Feb 19, 2013 at 12:07 PM, Marc Seeger <[email protected]> wrote: >> To test gluster's behavior under heavy load, I'm currently doing this on two >> machines sharing a common /mnt/gfs gluster mount: >> >> ssh bal-6.example.com apt-get install dbench && dbench 6 -t 60 -D /mnt/gfs >> ssh bal-7.example.com apt-get install dbench && dbench 6 -t 60 -D /mnt/gfs >> >> >> One of the processes usually dies pretty quickly like this: >> >> [608] open /mnt/gfs/clients/client5/~dmtmp/PWRPNT/PCBENCHM.PPT failed for >> handle 10003 (No such file or directory) >> (610) ERROR: handle 10003 was not found, >> Child failed with status 1 >> >> >> And the logs are full of things like this (ignore the initial timestamp, >> that's from our logging): >> >> [2013-02-19 14:38:38.714493] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background data missing-entry gfid self-heal failed on >> /clients/client5/~dmtmp/PM/MOVED.DOC, >> [2013-02-19 14:38:38.724494] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background entry self-heal failed on /clients/client3/~dmtmp, >> [2013-02-19 14:38:38.734495] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background data missing-entry gfid self-heal failed on >> /clients/client4/~dmtmp/PM/EVENTS.DOC, >> [2013-02-19 14:38:38.734495] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background data missing-entry gfid self-heal failed on >> /clients/client2/~dmtmp/PM/MOVED.DOC, >> [2013-02-19 14:38:38.734495] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background data missing-entry gfid self-heal failed on >> /clients/client1/~dmtmp/PM/MOVED.DOC, >> [2013-02-19 14:38:38.734495] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background data missing-entry gfid self-heal failed on >> /clients/client0/~dmtmp/PM/MOVED.DOC, >> [2013-02-19 14:38:38.734495] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background entry self-heal failed on /clients/client4/~dmtmp/PM, [build-2 >> system.rb:340], I, >> [2013-02-19T14:39:50.189970 #20802] INFO -- : >> [2013-02-19 14:38:36.041890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background entry self-heal failed on /, >> [2013-02-19 14:38:36.041890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background entry self-heal failed on /, >> [2013-02-19 14:38:36.041890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background entry self-heal failed on /, >> [2013-02-19 14:38:36.041890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background entry self-heal failed on /, >> [2013-02-19 14:38:36.041890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background entry self-heal failed on /, >> [2013-02-19 14:38:36.051890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entry missing-entry gfid self-heal failed on >> /clients, >> [2013-02-19 14:38:36.071890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entry missing-entry gfid self-heal failed on >> /clients/client2, >> [2013-02-19 14:38:36.071890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entry missing-entry gfid self-heal failed on >> /clients/client3, >> [2013-02-19 14:38:36.071890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background entry self-heal failed on /clients/client2, >> [2013-02-19 14:38:36.081890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background entry self-heal failed on /clients/client3, >> [2013-02-19 14:38:36.091890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entry missing-entry gfid self-heal failed on >> /clients/client2/~dmtmp, >> [2013-02-19 14:38:36.091890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entry missing-entry gfid self-heal failed on >> /clients/client3/~dmtmp, >> [2013-02-19 14:38:36.101890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background entry self-heal failed on /clients/client2/~dmtmp, >> [2013-02-19 14:38:36.101890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background entry self-heal failed on /clients/client3/~dmtmp, >> [2013-02-19 14:38:36.111890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entry missing-entry gfid self-heal failed on >> /clients/client2/~dmtmp/WORD, >> [2013-02-19 14:38:36.111890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entry missing-entry gfid self-heal failed on >> /clients/client3/~dmtmp/WORD, >> [2013-02-19 14:38:36.131890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background entry self-heal failed on /clients/client2/~dmtmp/WORD, >> [2013-02-19 14:38:36.141890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background entry self-heal failed on /clients/client3/~dmtmp/WORD, >> [2013-02-19 14:38:36.151890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entry missing-entry gfid self-heal failed on >> /clients/client2/~dmtmp/WORD/CHAP10.DOC, >> [2013-02-19 14:38:36.151890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entry missing-entry gfid self-heal failed on >> /clients/client3/~dmtmp/WORD/CHAP10.DOC, >> [2013-02-19 14:38:36.161890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entry missing-entry gfid self-heal failed on >> /clients/client2/~dmtmp/WORD/BASEMACH.DOC, >> [2013-02-19 14:38:36.161890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entry missing-entry gfid self-heal failed on >> /clients/client3/~dmtmp/WORD/BASEMACH.DOC, >> [2013-02-19 14:38:36.171890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entr [build-2 system.rb:340], I, >> [2013-02-19T14:39:50.189970 #20802] INFO -- : y missing-entry gfid >> self-heal failed on /clients/client2/~dmtmp/WORD/FACTS.DOC, >> [2013-02-19 14:38:36.181890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entry missing-entry gfid self-heal failed on >> /clients/client3/~dmtmp/WORD/FACTS.DOC, >> [2013-02-19 14:38:36.201890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entry missing-entry gfid self-heal failed on >> /clients/client2/~dmtmp/EXCEL, >> [2013-02-19 14:38:36.201890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entry missing-entry gfid self-heal failed on >> /clients/client3/~dmtmp/EXCEL, >> [2013-02-19 14:38:36.201890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background entry self-heal failed on /clients/client2/~dmtmp/EXCEL, >> [2013-02-19 14:38:36.201890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background entry self-heal failed on /clients/client3/~dmtmp/EXCEL, >> [2013-02-19 14:38:36.211890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background entry self-heal failed on /clients/client0/~dmtmp, >> [2013-02-19 14:38:36.211890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entry missing-entry gfid self-heal failed on >> /clients/client2/~dmtmp/EXCEL/PCMAGCD.XLS, >> [2013-02-19 14:38:36.211890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entry missing-entry gfid self-heal failed on >> /clients/client3/~dmtmp/EXCEL/PCMAGCD.XLS, >> [2013-02-19 14:38:36.241890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entry missing-entry gfid self-heal failed on >> /clients/client2/~dmtmp/EXCEL/SALES.XLS, >> [2013-02-19 14:38:36.241890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entry missing-entry gfid self-heal failed on >> /clients/client3/~dmtmp/EXCEL/SALES.XLS, >> [2013-02-19 14:38:36.271890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entry missing-entry gfid self-heal failed on >> /clients/client2/~dmtmp/PWRPNT, >> [2013-02-19 14:38:36.271890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entry missing-entry gfid self-heal failed on >> /clients/client3/~dmtmp/PWRPNT, >> [2013-02-19 14:38:36.281890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background entry self-heal failed on /clients/client2/~dmtmp/PWRPNT, >> [2013-02-19 14:38:36.281890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background entry self-heal failed on /clients/client3/~dmtmp/PWRPNT, >> [2013-02-19 14:38:36.291890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entry missing-entry gfid self-heal failed on >> /clients/client2/~dmtmp/PWRPNT/PCBENCHM.PPT, >> [2013-02-19 14:38:36.311890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entry missing-entry gfid self-heal failed on >> /clients/client3/~dmtmp/PWRPNT/PCBENCHM.PPT, >> [2013-02-19 14:38:36.351890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entry missing-entry gfid self-heal failed on >> /clients/client2/~dmtmp/PWRPNT/ZD16.BMP, >> [2013-02-19 14:38:36.351890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entry missing-entry gfid self-heal failed on >> /clients/client3/~dmtmp/PWRPNT/ZD16.BMP, >> [2013-02-19 14:38:36.381890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entry missing-entry gfid self-heal failed on >> /clients/client2/~dmtmp/PWRPNT/PPTOOLS1.PPA, >> [2013-02-19 14:38:36.391890] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: >> background meta-data data entry missing-entry gfid sel [build-2 >> system.rb:340] >> >> >> >> Any ideas? Can somebody confirm this happens for them too? >> >> The setup is ubuntu lucid machines running 3.3.1 from this PPA: >> https://launchpad.net/~semiosis/+archive/ubuntu-glusterfs-3.3 >> >> >> _______________________________________________ >> Gluster-users mailing list >> [email protected] >> http://supercolony.gluster.org/mailman/listinfo/gluster-users >> > >
_______________________________________________ Gluster-users mailing list [email protected] http://supercolony.gluster.org/mailman/listinfo/gluster-users
