Here are the results of the last run:
https://docs.google.com/spreadsheets/d/19JqvuFKZxKifgrhLF-5-bgemYj8XKldUox1QwsmGj2k/edit?usp=sharing
Each test has been run with a rough approximation of the best configuration
I've found (in number of client and brick threads), but I haven't done an
exhaustive search of the best configuration in each case.
The "fio rand write" test seems to have a big regression. An initial check
of the data shows that 2 of the 5 runs have taken > 50% more time. I'll try
to check why.
Many of the tests show a very high disk utilization, so comparisons may not
be accurate. In any case it's clear that we need a method to automatically
adjust the number of worker threads to the given load to make this useful.
Without that it's virtually impossible to find a fixed number of threads
that will work fine in all cases. I'm currently working on this.
Xavi
On Wed, Feb 13, 2019 at 11:34 AM Xavi Hernandez
wrote:
> On Tue, Feb 12, 2019 at 1:30 AM Vijay Bellur wrote:
>
>>
>>
>> On Tue, Feb 5, 2019 at 10:57 PM Xavi Hernandez
>> wrote:
>>
>>> On Wed, Feb 6, 2019 at 7:00 AM Poornima Gurusiddaiah <
>>> pguru...@redhat.com> wrote:
>>>
On Tue, Feb 5, 2019, 10:53 PM Xavi Hernandez >>> wrote:
> On Fri, Feb 1, 2019 at 1:51 PM Xavi Hernandez
> wrote:
>
>> On Fri, Feb 1, 2019 at 1:25 PM Poornima Gurusiddaiah <
>> pguru...@redhat.com> wrote:
>>
>>> Can the threads be categorised to do certain kinds of fops?
>>>
>>
>> Could be, but creating multiple thread groups for different tasks is
>> generally bad because many times you end up with lots of idle threads
>> which
>> waste resources and could increase contention. I think we should only
>> differentiate threads if it's absolutely necessary.
>>
>>
>>> Read/write affinitise to certain set of threads, the other metadata
>>> fops to other set of threads. So we limit the read/write threads and not
>>> the metadata threads? Also if aio is enabled in the backend the threads
>>> will not be blocked on disk IO right?
>>>
>>
>> If we don't block the thread but we don't prevent more requests to go
>> to the disk, then we'll probably have the same problem. Anyway, I'll try
>> to
>> run some tests with AIO to see if anything changes.
>>
>
> I've run some simple tests with AIO enabled and results are not good.
> A simple dd takes >25% more time. Multiple parallel dd take 35% more time
> to complete.
>
Thank you. That is strange! Had few questions, what tests are you
running for measuring the io-threads performance(not particularly aoi)? is
it dd from multiple clients?
>>>
>>> Yes, it's a bit strange. What I see is that many threads from the thread
>>> pool are active but using very little CPU. I also see an AIO thread for
>>> each brick, but its CPU usage is not big either. Wait time is always 0 (I
>>> think this is a side effect of AIO activity). However system load grows
>>> very high. I've seen around 50, while on the normal test without AIO it's
>>> stays around 20-25.
>>>
>>> Right now I'm running the tests on a single machine (no real network
>>> communication) using an NVMe disk as storage. I use a single mount point.
>>> The tests I'm running are these:
>>>
>>>- Single dd, 128 GiB, blocks of 1MiB
>>>- 16 parallel dd, 8 GiB per dd, blocks of 1MiB
>>>- fio in sequential write mode, direct I/O, blocks of 128k, 16
>>>threads, 8GiB per file
>>>- fio in sequential read mode, direct I/O, blocks of 128k, 16
>>>threads, 8GiB per file
>>>- fio in random write mode, direct I/O, blocks of 128k, 16 threads,
>>>8GiB per file
>>>- fio in random read mode, direct I/O, blocks of 128k, 16 threads,
>>>8GiB per file
>>>- smallfile create, 16 threads, 256 files per thread, 32 MiB per
>>>file (with one brick down, for the following test)
>>>- self-heal of an entire brick (from the previous smallfile test)
>>>- pgbench init phase with scale 100
>>>
>>> I run all these tests for a replica 3 volume and a disperse 4+2 volume.
>>>
>>
>>
>> Are these performance results available somewhere? I am quite curious to
>> understand the performance gains on NVMe!
>>
>
> I'm updating test results with the latest build. I'll report it here once
> it's complete.
>
> Xavi
>
>>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel
