Hi Vlad, The problem is definitely looking like NFS. A Git repository is comprised of a great many files, which must be read and parsed and often point to other files. You want fast filesystem access to work with a Git repository directly.
If you’re seeing that level of slowdown just from a dd test, I bet you’re having more issues than you realize. Directory scans and file seeking are probably slow as well. Every time we fetch a file (which will happen for each file in a diff, at upload and diff viewer time), it’s going to be performing a number of slow steps, resulting in the problems you’re seeing. What I’d suggest is moving away from NFS access to Git. Instead, install our RB Gateway service on the system hosting the repositories, configure it to point to each repository, and then configure those existing repositories in Review Board to access through RB Gateway instead of over NFS. https://www.reviewboard.org/downloads/rbgateway/ This is going to provide deeper integration with Git overall. RB Gateway can take care of some Git management that Review Board doesn’t attempt. It’s the preferred way if you want to use a plain Git repository not backed by a service like GitHub or GitLab. Regarding the database question, yes, you can point multiple Review Board servers to the same database. You must use the same SECRET_KEY on each server. Christian On Fri, Oct 25, 2019 at 06:40 Vlad S. <[email protected]> wrote: > Only new one. The old host is 10x times faster. Ten times. > Generates diffs one-by-one in 200-300ms. > > Old host uses NFS v3, dd reading a large file from repo at ~450MB/s > New host uses NFS v3, dd reading a large file from repo at ~255 MB/s > > SO the old has slower CPU but faster NFS, the new host has faster CPU but > still fast NFS.. > > I want to point both hosts at the same database and compare to find a > problem. The RB manual says it is supported. > > May I use the same SECRET_KEY in /var/www/rb-site1/conf/settings_local.py > across the hosts? > > /v > > пятница, 25 октября 2019 г., 13:39:19 UTC+3 пользователь Christian Hammond > написал: >> >> Hi Vlad, >> >> Is this problem only occurring on the new host and not the old one? >> >> Do you ever see those `git cat-file` errors on the old host? >> >> Any differences in NFS configuration between the hosts? >> >> Christian >> >> On Fri, Oct 25, 2019 at 2:11 AM Vlad S. <[email protected]> wrote: >> > Hi Christian, >>> >>> There are messages like this >>> File >>> "/venvs/rb/lib/python2.7/site-packages/reviewboard/scmtools/git.py", line >>> 702, in _cat_file >>> raise SCMError(errmsg) >>> SCMError: fatal: git cat-file 1ba156144fe754340c79e0f871a52f750ad8f6b7: >>> bad file >>> >>> The repository is on the same network, NFS mount. memcached runs, 2G, >>> cache hit ratio ~95%, 4 cores CPU. >>> Only mysql server is distant with 100ms latency. Tried to turn off >>> syntax highlighting. It does not help much. >>> >>> 163K diff file, 116 files, C text. So it does not look large. >>> >>> # Old fast host - New slow host >>> CPU 4 cores 4 cores >>> both have plenty on RAM >>> memcached v1.5* 2GB v1.4* 2GB >>> memcached memory used ~400MB ~90MB >>> mysql latency 0.2ms 100ms >>> repo host latency both on the same network. >>> >>> Memory usage: >>> >>> 411.7 MB >>> пятница, 25 октября 2019 г., 11:44:25 UTC+3 пользователь Christian >>> Hammond написал: >>>> >>>> Hi Vlad, >>>> >>>> It's often a delay in getting data back from the repository. Are there >>>> any other log messages? Is the repository in the same network? >>>> >>>> How many files are in each diff, and are any excessively large? >>>> >>>> Also, make sure that Review Board is seeing the memcached server. Check >>>> the Admin UI -> Cache information to be sure. >>>> >>>> Christian >>>> >>>> On Fri, Oct 25, 2019 at 1:32 AM Vlad S. <[email protected]> wrote: >>>> >>>>> Hi, >>>>> >>>>> >>>>> Generating a diff file info for interdiffsettakes too much time ~30 >>>>> seconds. How do I know why? >>>>> Is there a command line method to debug this and see all the calls >>>>> made? >>>>> === >>>>> CRITICAL - None - ... - /rb/api/review-requests/27122/diff-context/ - >>>>> .. - Generating diff file info for interdiffset ids 66619-66797 took >>>>> 29.881996 seconds >>>>> === >>>>> Thanks! >>>>> >>>>> /v >>>>> >>>>> -- >>>>> Supercharge your Review Board with Power Pack: >>>>> https://www.reviewboard.org/powerpack/ >>>>> Want us to host Review Board for you? Check out RBCommons: >>>>> https://rbcommons.com/ >>>>> Happy user? Let us know! https://www.reviewboard.org/users/ >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Review Board Community" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/reviewboard/47fd0081-0d5d-4c50-8fd9-5a2fe036806d%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/reviewboard/47fd0081-0d5d-4c50-8fd9-5a2fe036806d%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> >>>> >>>> -- >>>> Christian Hammond >>>> President/CEO of Beanbag <https://www.beanbaginc.com/> >>>> Makers of Review Board <https://www.reviewboard.org/> >>>> >>> -- >>> Supercharge your Review Board with Power Pack: >>> https://www.reviewboard.org/powerpack/ >>> Want us to host Review Board for you? Check out RBCommons: >>> https://rbcommons.com/ >>> Happy user? Let us know! https://www.reviewboard.org/users/ >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Review Board Community" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/reviewboard/a8aa658c-748b-4e7d-99e7-c858220eaf4f%40googlegroups.com >>> <https://groups.google.com/d/msgid/reviewboard/a8aa658c-748b-4e7d-99e7-c858220eaf4f%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> >> -- >> Christian Hammond >> President/CEO of Beanbag <https://www.beanbaginc.com/> >> Makers of Review Board <https://www.reviewboard.org/> >> > -- > Supercharge your Review Board with Power Pack: > https://www.reviewboard.org/powerpack/ > Want us to host Review Board for you? Check out RBCommons: > https://rbcommons.com/ > Happy user? Let us know! https://www.reviewboard.org/users/ > --- > You received this message because you are subscribed to the Google Groups > "Review Board Community" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/reviewboard/c57a4342-d868-4e62-adfc-6e5fffcc9b96%40googlegroups.com > <https://groups.google.com/d/msgid/reviewboard/c57a4342-d868-4e62-adfc-6e5fffcc9b96%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- -- Christian Hammond President/CEO of Beanbag Makers of Review Board -- Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/ Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/ Happy user? Let us know! https://www.reviewboard.org/users/ --- You received this message because you are subscribed to the Google Groups "Review Board Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/reviewboard/CAE7VndkeTc6%2BMvKuyHmz7pjRJLTNtfxQhi9NPk10_Wqy9JRSVQ%40mail.gmail.com.
