Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance

2018-02-21 Thread Matt Benjamin 
Hi Bill,

Let's just talk about the rpc-ping-mt/request minimal latency.

On Wed, Feb 21, 2018 at 8:24 AM, William Allen Simpson
 wrote:

>>
>> It measures minimal request latency all the way to nfs-ganesha's, or
>> the Linux kernel's, rpcnull handler--the "top" of the request handling
>> stack, in the given client/server/network configuration.  Scaled up,
>> it can report the max such calls/s, which is a kind of best-possible
>> value for iops, taking FSAL ops to have 0 latency.
>>
> As I've mentioned elsewhere, this should be entirely dominated by the
> link speed and protocol.  We should see UDP as slowest, TCP in the
> middle, and RDMA as fastest.

I think the point is, the "should" assumes perfect behavior from the
dispatch and (minimal) request execution path within nfs-ganesha, does
it not?  In other words, profiling of rpc null is effectively
profiling the first stage of the request pipeline, and if it's not
fast, more interesting ops won't be.

>
> OTOH, the "max such calls/s" would be reported by using XDR_RAW, which
> is currently not working.
>

My intended meaning, and Daniel's when he articulated the same point
Friday, is that "max such calls/s" -is- meaningful for transports NFS
actually supports, and those are the ones we can compare with other
implementations.  I think the raw transport opens up some interesting
paths, but it seems like they would involve more development.

Matt


-- 

Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel

Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance

2018-02-21 Thread William Allen Simpson 

I really don't have time today to respond to every one-liner
throw-away comment here, so I'll try to stick to the most cogent.

On 2/20/18 8:33 AM, Matt Benjamin wrote:

On Tue, Feb 20, 2018 at 8:12 AM, William Allen Simpson
 wrote:

On 2/18/18 2:47 PM, Matt Benjamin wrote:


On Fri, Feb 16, 2018 at 11:23 AM, William Allen Simpson


But the planned 2.7 improvements are mostly throughput related, not IOPS.


Not at all, though I am trying to ensure that we get async FSAL ops
in.  There are people working on IOPs too.


async FSAL ops are not likely to further improve IOPS.

As I've pointed out many times in the past, async only allows
the same number of threads to handle more concurrent operations.

But it's actually slower.  It basically doubles the number of
system calls.  System calls are one of the reasons that Ganesha is
slower than knfsd.  It also ruins CPU cache coherency.

If we're CPU bound or network bandwidth constrained, it won't help.


Not everything being worked on is a direct answer to this problem.


Anything that isn't about protocol correctness or performance
should be lower priority.

Async FSAL ops could -- in a few cases where network or disk spinning
dominate the thread timing -- improve thread utilization.  But it
will never improve IOPS.



The effort that I've been waiting for *3* years -- add io vectors to
the FSAL interface, so that we can zero-copy buffers -- is more
likely to improve throughput.


Which, as you've noted, also is happening.


Finally.  Again, I've been asking for it now 3 years.  For V2.7, this is
what I anticipate will result in the most performance increase.



Moreover, going through the code and removing locking other such
bottlenecks is more likely to improve IOPS.


No one is disputing this.  It is not a new discovery, however.


And yet we seem to have to keep reminding folks.  I remember that
Malahal did a survey of lock timing/congestion a couple of years ago.

Perhaps he's willing to run it again?



If Ganesha is adding 6 ms to every read operation, we have a serious
problem, and need to profile immediately!



That's kind of what our team is doing.  I look forward to your work
with rpc-ping-mt.


Well, you were able to send me a copy after 6 pm on a Friday, so I'm
now taking a look at it.  Hopefully I'll have something by Friday.


It came up in a meeting on Friday, that's why I sent it Friday.  I got
busy in meetings and issues, that's why after 6 pm.



But I really wish you'd posted it 3 years ago.  It doesn't really test
IOPS, other than whatever bandwidth limit is imposed by the interface,
but it does test the client call interface.


It measures minimal request latency all the way to nfs-ganesha's, or
the Linux kernel's, rpcnull handler--the "top" of the request handling
stack, in the given client/server/network configuration.  Scaled up,
it can report the max such calls/s, which is a kind of best-possible
value for iops, taking FSAL ops to have 0 latency.


As I've mentioned elsewhere, this should be entirely dominated by the
link speed and protocol.  We should see UDP as slowest, TCP in the
middle, and RDMA as fastest.

OTOH, the "max such calls/s" would be reported by using XDR_RAW, which
is currently not working.



It was posted to this list by Tigran, iirc, in 2011 or 2012.


In which case, I really wish Tigran had put it in the tests folder,
so we'd have been continuously using and updating it.  The gtests
are a great improvement.  Now all we need to do is run them every
week for each new -dev release to track improvements/regressions.

I didn't know about this old test, because my archives only go back
to late 2014.  I wasn't involved before then.

Moreover, I didn't appreciate being criticized for not running an
old test that wasn't in the tree and wasn't maintained.



We've been using Jeff Layton's delegation callback work to test, and
this test would have been better and easier.

But a unit test is not what we need.  I wrote "profile".  We need to
know where the CPU bottlenecks are in Ganesha itself.


You also wrote unit test.


Looking back over this thread, I don't see those words.

I wrote "profile".

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel

Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance

2018-02-20 Thread Matt Benjamin 
Hi,

On Tue, Feb 20, 2018 at 8:12 AM, William Allen Simpson
 wrote:
> On 2/18/18 2:47 PM, Matt Benjamin wrote:
>>
>> On Fri, Feb 16, 2018 at 11:23 AM, William Allen Simpson
>>>
>>> But the planned 2.7 improvements are mostly throughput related, not IOPS.
>>
>>
>> Not at all, though I am trying to ensure that we get async FSAL ops
>> in.  There are people working on IOPs too.
>>
> async FSAL ops are not likely to further improve IOPS.
>
> As I've pointed out many times in the past, async only allows
> the same number of threads to handle more concurrent operations.
>
> But it's actually slower.  It basically doubles the number of
> system calls.  System calls are one of the reasons that Ganesha is
> slower than knfsd.  It also ruins CPU cache coherency.
>
> If we're CPU bound or network bandwidth constrained, it won't help.

Not everything being worked on is a direct answer to this problem.

>
> The effort that I've been waiting for *3* years -- add io vectors to
> the FSAL interface, so that we can zero-copy buffers -- is more
> likely to improve throughput.

Which, as you've noted, also is happening.

>
> Moreover, going through the code and removing locking other such
> bottlenecks is more likely to improve IOPS.

No one is disputing this.  It is not a new discovery, however.

>
>
>>> If Ganesha is adding 6 ms to every read operation, we have a serious
>>> problem, and need to profile immediately!
>>>
>>
>> That's kind of what our team is doing.  I look forward to your work
>> with rpc-ping-mt.
>>
> Well, you were able to send me a copy after 6 pm on a Friday, so I'm
> now taking a look at it.  Hopefully I'll have something by Friday.

It came up in a meeting on Friday, that's why I sent it Friday.  I got
busy in meetings and issues, that's why after 6 pm.

>
> But I really wish you'd posted it 3 years ago.  It doesn't really test
> IOPS, other than whatever bandwidth limit is imposed by the interface,
> but it does test the client call interface.

It measures minimal request latency all the way to nfs-ganesha's, or
the Linux kernel's, rpcnull handler--the "top" of the request handling
stack, in the given client/server/network configuration.  Scaled up,
it can report the max such calls/s, which is a kind of best-possible
value for iops, taking FSAL ops to have 0 latency.

It was posted to this list by Tigran, iirc, in 2011 or 2012.

>
> We've been using Jeff Layton's delegation callback work to test, and
> this test would have been better and easier.
>
> But a unit test is not what we need.  I wrote "profile".  We need to
> know where the CPU bottlenecks are in Ganesha itself.

You also wrote unit test.

Matt

-- 

Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel

Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance

2018-02-20 Thread William Allen Simpson 

On 2/18/18 2:47 PM, Matt Benjamin wrote:

On Fri, Feb 16, 2018 at 11:23 AM, William Allen Simpson

But the planned 2.7 improvements are mostly throughput related, not IOPS.


Not at all, though I am trying to ensure that we get async FSAL ops
in.  There are people working on IOPs too.


async FSAL ops are not likely to further improve IOPS.

As I've pointed out many times in the past, async only allows
the same number of threads to handle more concurrent operations.

But it's actually slower.  It basically doubles the number of
system calls.  System calls are one of the reasons that Ganesha is
slower than knfsd.  It also ruins CPU cache coherency.

If we're CPU bound or network bandwidth constrained, it won't help.

The effort that I've been waiting for *3* years -- add io vectors to
the FSAL interface, so that we can zero-copy buffers -- is more
likely to improve throughput.

Moreover, going through the code and removing locking other such
bottlenecks is more likely to improve IOPS.



If Ganesha is adding 6 ms to every read operation, we have a serious
problem, and need to profile immediately!



That's kind of what our team is doing.  I look forward to your work
with rpc-ping-mt.


Well, you were able to send me a copy after 6 pm on a Friday, so I'm
now taking a look at it.  Hopefully I'll have something by Friday.

But I really wish you'd posted it 3 years ago.  It doesn't really test
IOPS, other than whatever bandwidth limit is imposed by the interface,
but it does test the client call interface.

We've been using Jeff Layton's delegation callback work to test, and
this test would have been better and easier.

But a unit test is not what we need.  I wrote "profile".  We need to
know where the CPU bottlenecks are in Ganesha itself.

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel

Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance

2018-02-19 Thread Matt Benjamin 
On Fri, Feb 16, 2018 at 11:23 AM, William Allen Simpson
 wrote:

>>
> Actually, 2.6 should handle as many concurrent client requests as you like.
> (Up to 250 of them.)  That's one of its features.
>
> The client is not sending concurrent requests.

This seems worth digging into.  That's my understanding of 2.6
dispatch too, anyway.

>
>>
> But the planned 2.7 improvements are mostly throughput related, not IOPS.

Not at all, though I am trying to ensure that we get async FSAL ops
in.  There are people working on IOPs too.

>
>
> If Ganesha is adding 6 ms to every read operation, we have a serious
> problem, and need to profile immediately!
>

That's kind of what our team is doing.  I look forward to your work
with rpc-ping-mt.

Matt


-- 

Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel

Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance

2018-02-18 Thread Deepak Jagtap 
>> How many clients are you using?  Each client op can only (currently) be 
>> handled in a single thread, and client's won't send more ops until the 
>> current one is ack'd, so Ganesha can basically only parallelize on a 
>> per-client basis at the moment.
>>


>>Actually, 2.6 should handle as many concurrent client requests as you like.
(Up to 250 of them.)  That's one of its features.

< The client is not sending concurrent requests.
   There is only one client with 2 worker threads each worker thread generating 
64 outstanding requests. Configured both servers (knfs & ganesha) with 128 
worker threads.
   I think client is fine  and is sending the request concurrently.  For knfs 
and nfsganesha client is exactly the same, the only change is server exporting 
the directory.

>> I'm sure there are locking issues; so far we've mostly worked on correctness 
>> rather than performance.  2.6 has changed the threading model a fair amount, 
>> and 2.7 will have more improvements, but it's a slow process.
>>
But the planned 2.7 improvements are mostly throughput related, not IOPS.

> On 02/13/2018 06:38 PM, Deepak Jagtap wrote:
>> Yeah user-kernel context switching is definitely adding up latency, but I 
>> wonder ifrpc or some locking overhead is also in the picture.
>>

< ifrpc?
   Sorry typo: ifrpc => 'if rpc'


>> With 70% read 30% random workload nfs ganesha CPU usage was close to 170% 
>> while remaining 2 cores were pretty much unused (~18K IOPS, latency ~8ms)
>>
>> With 100% read 30% random nfs ganesha CPU usage ~250% ( ~50K IOPS, latency 
>> ~2ms).
>>

< Those latency numbers seem suspect to me.  The dominant latency should be
< the file system.  The system calls shouldn't add more than microseconds.

< If Ganesha is adding 6 ms to every read operation, we have a serious
< problem, and need to profile immediately!

Yeah that's right ideally in the end only physical disk IO latency should 
dominate in this complete stack.



From: William Allen Simpson <william.allen.simp...@gmail.com>
Sent: Friday, February 16, 2018 8:23:16 AM
To: d...@redhat.com; Deepak Jagtap; nfs-ganesha-devel@lists.sourceforge.net
Subject: Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance

On 2/14/18 8:32 AM, Daniel Gryniewicz wrote:
> How many clients are you using?  Each client op can only (currently) be 
> handled in a single thread, and client's won't send more ops until the 
> current one is ack'd, so Ganesha can basically only parallelize on a 
> per-client basis at the moment.
>
Actually, 2.6 should handle as many concurrent client requests as you like.
(Up to 250 of them.)  That's one of its features.

The client is not sending concurrent requests.


> I'm sure there are locking issues; so far we've mostly worked on correctness 
> rather than performance.  2.6 has changed the threading model a fair amount, 
> and 2.7 will have more improvements, but it's a slow process.
>
But the planned 2.7 improvements are mostly throughput related, not IOPS.


> On 02/13/2018 06:38 PM, Deepak Jagtap wrote:
>> Yeah user-kernel context switching is definitely adding up latency, but I 
>> wonder ifrpc or some locking overhead is also in the picture.
>>
ifrpc?


>> With 70% read 30% random workload nfs ganesha CPU usage was close to 170% 
>> while remaining 2 cores were pretty much unused (~18K IOPS, latency ~8ms)
>>
>> With 100% read 30% random nfs ganesha CPU usage ~250% ( ~50K IOPS, latency 
>> ~2ms).
>>
Those latency numbers seem suspect to me.  The dominant latency should be
the file system.  The system calls shouldn't add more than microseconds.

If Ganesha is adding 6 ms to every read operation, we have a serious
problem, and need to profile immediately!
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel

Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance

2018-02-18 Thread Malahal Naineni 
Dan, only the decoder threads are per export and ACKs are sent by the TCP
layer itself. I am pretty sure clients do send multiple requests on the
same socket (we have seen it exceed the default 512 after which we drop
requests) and we use multiple worker threads.

Regards, Malahal.

On Wed, Feb 14, 2018 at 7:02 PM, Daniel Gryniewicz <d...@redhat.com> wrote:

> How many clients are you using?  Each client op can only (currently) be
> handled in a single thread, and client's won't send more ops until the
> current one is ack'd, so Ganesha can basically only parallelize on a
> per-client basis at the moment.
>
> I'm sure there are locking issues; so far we've mostly worked on
> correctness rather than performance.  2.6 has changed the threading model a
> fair amount, and 2.7 will have more improvements, but it's a slow process.
>
> Daniel
>
> On 02/13/2018 06:38 PM, Deepak Jagtap wrote:
>
>> Thanks Daniel!
>>
>> Yeah user-kernel context switching is definitely adding up latency, but I
>> wonder ifrpc or some locking overhead is also in the picture.
>>
>> With 70% read 30% random workload nfs ganesha CPU usage was close to 170%
>> while remaining 2 cores were pretty much unused (~18K IOPS, latency ~8ms)
>>
>> With 100% read 30% random nfs ganesha CPU usage ~250% ( ~50K IOPS,
>> latency ~2ms).
>>
>>
>> -Deepak
>>
>> 
>> *From:* Daniel Gryniewicz <d...@redhat.com>
>> *Sent:* Tuesday, February 13, 2018 6:15:47 AM
>> *To:* nfs-ganesha-devel@lists.sourceforge.net
>> *Subject:* Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance
>> Also keep in mind that FSAL VFS can never, by it's very nature, beat
>> knfsd, since it has to do everything knfsd does, but also has userspace
>> <-> kernespace transitions.  Ganesha's strength is exporting
>> userspace-based cluster filesystems.
>>
>> That said, we're always working to make Ganesha faster, and I'm sure
>> there's gains to be made, even in these circumstances.
>>
>> Daniel
>>
>>
>> On 02/12/2018 07:01 PM, Deepak Jagtap wrote:
>>
>>> Hey Guys,
>>>
>>>
>>> I ran few performance tests to compare nfs gansha and nfs kernel server
>>> and noticed significant difference.
>>>
>>>
>>> Please find my test result:
>>>
>>>
>>> SSD formated with EXT3 exported using nfs ganesha  : ~18K IOPSAvg
>>> latency: ~8ms   Throughput: ~60MBPS
>>>
>>> same directory exported using nfs kernel server: ~75K IOPS
>>> Avg latency: ~0.8ms Throughput: ~300MBPS
>>>
>>>
>>> nfs kernel and nfs ganesha both of them are configured with 128
>>> worker threads. nfs ganesha is configured with VFS FSAL.
>>>
>>>
>>> Am I missing something major in nfs ganesha config or this is expected
>>> behavior.
>>>
>>> Appreciate any inputs as how the performance can be improved for nfs
>>> ganesha.
>>>
>>>
>>>
>>> Please find following ganesha config file that I am using:
>>>
>>>
>>> NFS_Core_Param
>>> {
>>>   Nb_Worker = 128 ;
>>> }
>>>
>>> EXPORT
>>> {
>>>   # Export Id (mandatory, each EXPORT must have a unique Export_Id)
>>>  Export_Id = 77;
>>>  # Exported path (mandatory)
>>>  Path = /host/test;
>>>  Protocols = 3;
>>>  # Pseudo Path (required for NFS v4)
>>>  Pseudo = /host/test;
>>>  # Required for access (default is None)
>>>  # Could use CLIENT blocks instead
>>>  Access_Type = RW;
>>>  # Exporting FSAL
>>>  FSAL {
>>>   Name = VFS;
>>>  }
>>>  CLIENT
>>>  {
>>>   Clients = *;
>>>   Squash = None;
>>>   Access_Type = RW;
>>>  }
>>> }
>>>
>>>
>>>
>>> Thanks & Regards,
>>>
>>> Deepak
>>>
>>>
>>>
>>> 
>>> --
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>
>>>
>>>
>>> ___
>>> Nfs-ganesha-deve

Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance

2018-02-18 Thread J. Bruce Fields 
On Wed, Feb 14, 2018 at 08:32:19AM -0500, Daniel Gryniewicz wrote:
> How many clients are you using?  Each client op can only (currently)
> be handled in a single thread, and client's won't send more ops
> until the current one is ack'd,

What version of NFS was the test run over?

I don't see how the server can limit the number of outstanding requests
for NFS versions less than 4.1.

So if the client's doing NFSv3 or v4.0, then there should still always
be a queue of requests in the server's receive buffer, so it shouldn't
have to wait for a round trip back to the client to process the next
request.

But the reads have to be synchronous assuming the working set isn't in
cache, so you're completely bound by the SSD's read latency there.  If
the writes are asynchronous (so, sent without the "stable" bit set),
there might be a little opportunity for parallelism writing to the drive
as well.

So might be interesting to know what exactly the model of SSD used is.

> so Ganesha can basically only
> parallelize on a per-client basis at the moment.

Still something worth fixing, of course.

--b.

> 
> I'm sure there are locking issues; so far we've mostly worked on
> correctness rather than performance.  2.6 has changed the threading
> model a fair amount, and 2.7 will have more improvements, but it's a
> slow process.
> 
> Daniel
> 
> On 02/13/2018 06:38 PM, Deepak Jagtap wrote:
> >Thanks Daniel!
> >
> >Yeah user-kernel context switching is definitely adding up
> >latency, but I wonder ifrpc or some locking overhead is also in
> >the picture.
> >
> >With 70% read 30% random workload nfs ganesha CPU usage was close
> >to 170% while remaining 2 cores were pretty much unused (~18K
> >IOPS, latency ~8ms)
> >
> >With 100% read 30% random nfs ganesha CPU usage ~250% ( ~50K IOPS,
> >latency ~2ms).
> >
> >
> >-Deepak
> >
> >
> >*From:* Daniel Gryniewicz <d...@redhat.com>
> >*Sent:* Tuesday, February 13, 2018 6:15:47 AM
> >*To:* nfs-ganesha-devel@lists.sourceforge.net
> >*Subject:* Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance
> >Also keep in mind that FSAL VFS can never, by it's very nature, beat
> >knfsd, since it has to do everything knfsd does, but also has userspace
> ><-> kernespace transitions.  Ganesha's strength is exporting
> >userspace-based cluster filesystems.
> >
> >That said, we're always working to make Ganesha faster, and I'm sure
> >there's gains to be made, even in these circumstances.
> >
> >Daniel
> >
> >On 02/12/2018 07:01 PM, Deepak Jagtap wrote:
> >>Hey Guys,
> >>
> >>
> >>I ran few performance tests to compare nfs gansha and nfs kernel
> >>server and noticed significant difference.
> >>
> >>
> >>Please find my test result:
> >>
> >>
> >>SSD formated with EXT3 exported using nfs ganesha  : ~18K IOPS 
> >>  Avg latency: ~8ms       Throughput: ~60MBPS
> >>
> >>same directory exported using nfs kernel server:           
> >> ~75K IOPS      Avg latency: ~0.8ms Throughput: ~300MBPS
> >>
> >>
> >>nfs kernel and nfs ganesha both of them are configured with 128
> >>worker threads. nfs ganesha is configured with VFS FSAL.
> >>
> >>
> >>Am I missing something major in nfs ganesha config or this is
> >>expected behavior.
> >>
> >>Appreciate any inputs as how the performance can be improved for
> >>nfs ganesha.
> >>
> >>
> >>
> >>Please find following ganesha config file that I am using:
> >>
> >>
> >>NFS_Core_Param
> >>{
> >>          Nb_Worker = 128 ;
> >>}
> >>
> >>EXPORT
> >>{
> >>      # Export Id (mandatory, each EXPORT must have a unique Export_Id)
> >>     Export_Id = 77;
> >>     # Exported path (mandatory)
> >>     Path = /host/test;
> >>     Protocols = 3;
> >>     # Pseudo Path (required for NFS v4)
> >>     Pseudo = /host/test;
> >>     # Required for access (default is None)
> >>     # Could use CLIENT blocks instead
> >>     Access_Type = RW;
> >>     # Exporting FSAL
> >>     FSAL {
> >>          Name = VFS;
> >>     }
> >>     CLIENT
> >>     {
> >>          Clients = *;
> >>          Squash = None;
> >>          Access_Type = RW;
> >>     }
> >>}
> >>
> >>
> >>
> >>Thanks

Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance

2018-02-18 Thread William Allen Simpson 

On 2/14/18 8:32 AM, Daniel Gryniewicz wrote:

How many clients are you using?  Each client op can only (currently) be handled 
in a single thread, and client's won't send more ops until the current one is 
ack'd, so Ganesha can basically only parallelize on a per-client basis at the 
moment.


Actually, 2.6 should handle as many concurrent client requests as you like.
(Up to 250 of them.)  That's one of its features.

The client is not sending concurrent requests.



I'm sure there are locking issues; so far we've mostly worked on correctness 
rather than performance.  2.6 has changed the threading model a fair amount, 
and 2.7 will have more improvements, but it's a slow process.


But the planned 2.7 improvements are mostly throughput related, not IOPS.



On 02/13/2018 06:38 PM, Deepak Jagtap wrote:

Yeah user-kernel context switching is definitely adding up latency, but I 
wonder ifrpc or some locking overhead is also in the picture.


ifrpc?



With 70% read 30% random workload nfs ganesha CPU usage was close to 170% while 
remaining 2 cores were pretty much unused (~18K IOPS, latency ~8ms)

With 100% read 30% random nfs ganesha CPU usage ~250% ( ~50K IOPS, latency 
~2ms).


Those latency numbers seem suspect to me.  The dominant latency should be
the file system.  The system calls shouldn't add more than microseconds.

If Ganesha is adding 6 ms to every read operation, we have a serious
problem, and need to profile immediately!

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel

Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance

2018-02-15 Thread J. Bruce Fields 
On Wed, Feb 14, 2018 at 12:00:52AM +, Deepak Jagtap wrote:
> > What is the actual test?
> 
>- nfs server: RHEL based host exporting ext3 formated SSD.
> 
>- nfs client: RHEL based host running windows vm IOmeter with 70% read, 
> 30% random workload, 2 worker threads doing IOs with 64 outstanding IO queue 
> depth each.
> 
> 
> > What are the export options in the knfsd case?
> 
>- I am using this for exporting using knfsd.
> 
>  exportfs -oinsecure,rw,sync,no_root_squash,fsid=77 *:/host/test
> 
> > Is there any reason you're testing ext3 instead of ext4 or xfs?
>- Not in particular. I could have used ext4.  I was evaluating nfs server 
> impact in both scenarios.
> 
> >What's the client?  What are the kernel versions on server and client?
> 
>- Both of them are at 3.10.0-693.17.1.el7.x86_64 kernel version

That all osunds reasonable to me, thanks.--b.

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel

Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance

2018-02-15 Thread Daniel Gryniewicz 
How many clients are you using?  Each client op can only (currently) be 
handled in a single thread, and client's won't send more ops until the 
current one is ack'd, so Ganesha can basically only parallelize on a 
per-client basis at the moment.


I'm sure there are locking issues; so far we've mostly worked on 
correctness rather than performance.  2.6 has changed the threading 
model a fair amount, and 2.7 will have more improvements, but it's a 
slow process.


Daniel

On 02/13/2018 06:38 PM, Deepak Jagtap wrote:

Thanks Daniel!

Yeah user-kernel context switching is definitely adding up latency, but 
I wonder ifrpc or some locking overhead is also in the picture.


With 70% read 30% random workload nfs ganesha CPU usage was close to 
170% while remaining 2 cores were pretty much unused (~18K IOPS, 
latency ~8ms)


With 100% read 30% random nfs ganesha CPU usage ~250% ( ~50K IOPS, 
latency ~2ms).



-Deepak


*From:* Daniel Gryniewicz <d...@redhat.com>
*Sent:* Tuesday, February 13, 2018 6:15:47 AM
*To:* nfs-ganesha-devel@lists.sourceforge.net
*Subject:* Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance
Also keep in mind that FSAL VFS can never, by it's very nature, beat
knfsd, since it has to do everything knfsd does, but also has userspace
<-> kernespace transitions.  Ganesha's strength is exporting
userspace-based cluster filesystems.

That said, we're always working to make Ganesha faster, and I'm sure
there's gains to be made, even in these circumstances.

Daniel

On 02/12/2018 07:01 PM, Deepak Jagtap wrote:

Hey Guys,


I ran few performance tests to compare nfs gansha and nfs kernel server 
and noticed significant difference.



Please find my test result:


SSD formated with EXT3 exported using nfs ganesha  : ~18K IOPS    Avg 
latency: ~8ms       Throughput: ~60MBPS


same directory exported using nfs kernel server:             ~75K IOPS  
    Avg latency: ~0.8ms Throughput: ~300MBPS



nfs kernel and nfs ganesha both of them are configured with 128 
worker threads. nfs ganesha is configured with VFS FSAL.



Am I missing something major in nfs ganesha config or this is expected 
behavior.


Appreciate any inputs as how the performance can be improved for nfs 
ganesha.




Please find following ganesha config file that I am using:


NFS_Core_Param
{
          Nb_Worker = 128 ;
}

EXPORT
{
      # Export Id (mandatory, each EXPORT must have a unique Export_Id)
     Export_Id = 77;
     # Exported path (mandatory)
     Path = /host/test;
     Protocols = 3;
     # Pseudo Path (required for NFS v4)
     Pseudo = /host/test;
     # Required for access (default is None)
     # Could use CLIENT blocks instead
     Access_Type = RW;
     # Exporting FSAL
     FSAL {
          Name = VFS;
     }
     CLIENT
     {
          Clients = *;
          Squash = None;
          Access_Type = RW;
     }
}



Thanks & Regards,

Deepak



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot



___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel




--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel

Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance

2018-02-15 Thread Malahal Naineni 
As Bill said, it is not applicable to V2.6. It is there in V2.5 (yes,
please see src/config_samples/config.txt in that version for details).

On Wed, Feb 14, 2018 at 4:50 AM, Deepak Jagtap <deepak.jag...@maxta.com>
wrote:

> Thanks Malahal, William!
>
>
> Tried both v2.5-stable and 2.6 (next branch).
>
> Noticed marginal improvement, ~19K IOPS with 2.6 compared to ~18K IOPS
> with 2.5.
>
> Couldn't find anything in the 2.6 source with name 'Dispatch_Max_Reqs_Xprt
> '? Is this configurable from config file?
>
>
> Regards,
>
> Deepak
> --
> *From:* William Allen Simpson <william.allen.simp...@gmail.com>
> *Sent:* Tuesday, February 13, 2018 4:38:11 AM
> *To:* Malahal Naineni; Matt Benjamin
> *Cc:* nfs-ganesha-devel@lists.sourceforge.net
> *Subject:* Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance
>
> On 2/13/18 1:21 AM, Malahal Naineni wrote:
> > If your latency is high, then you most likely need to
> change Dispatch_Max_Reqs_Xprt. What your Dispatch_Max_Reqs_Xprt value?
> >
> That shouldn't do anything anymore in V2.6, other than 9P.
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel

Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance

2018-02-15 Thread Deepak Jagtap 
Hi Frank, I tried both 2.6 and 2.5-stable. 2.6  showed marginal improvements 
at-least for my workload (improvement of ~1K for 70% read, 30% random workload).

-Deepak


From: Frank Filz <ffilz...@mindspring.com>
Sent: Tuesday, February 13, 2018 7:26:50 AM
To: Deepak Jagtap; nfs-ganesha-devel@lists.sourceforge.net
Subject: RE: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance


What version are you running? Significant improvements have been made in V2.6 
(which is close to being tagged V2.6.0).



Frank



From: Deepak Jagtap [mailto:deepak.jag...@maxta.com]
Sent: Monday, February 12, 2018 4:01 PM
To: nfs-ganesha-devel@lists.sourceforge.net
Subject: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance



Hey Guys,



I ran few performance tests to compare nfs gansha and nfs kernel server and 
noticed significant difference.



Please find my test result:



SSD formated with EXT3 exported using nfs ganesha  : ~18K IOPSAvg latency: 
~8ms   Throughput: ~60MBPS

same directory exported using nfs kernel server: ~75K IOPSAvg 
latency: ~0.8msThroughput: ~300MBPS



nfs kernel and nfs ganesha both of them are configured with 128 worker threads. 
nfs ganesha is configured with VFS FSAL.



Am I missing something major in nfs ganesha config or this is expected behavior.

Appreciate any inputs as how the performance can be improved for nfs ganesha.





Please find following ganesha config file that I am using:



NFS_Core_Param

{

Nb_Worker = 128 ;

}



EXPORT

{

# Export Id (mandatory, each EXPORT must have a unique Export_Id)

   Export_Id = 77;

   # Exported path (mandatory)

   Path = /host/test;

   Protocols = 3;

   # Pseudo Path (required for NFS v4)

   Pseudo = /host/test;

   # Required for access (default is None)

   # Could use CLIENT blocks instead

   Access_Type = RW;

   # Exporting FSAL

   FSAL {

Name = VFS;

   }

   CLIENT

   {

Clients = *;

Squash = None;

Access_Type = RW;

   }

}







Thanks & Regards,

Deepak

[https://ipmcdn.avast.com/images/icons/icon-envelope-tick-round-orange-animated-no-repeat-v1.gif]<https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=emailclient_term=icon>
Virus-free. 
www.avast.com<https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=emailclient_term=link>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel

Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance

2018-02-15 Thread Deepak Jagtap 
> What is the actual test?

   - nfs server: RHEL based host exporting ext3 formated SSD.

   - nfs client: RHEL based host running windows vm IOmeter with 70% read, 30% 
random workload, 2 worker threads doing IOs with 64 outstanding IO queue depth 
each.


> What are the export options in the knfsd case?

   - I am using this for exporting using knfsd.

 exportfs -oinsecure,rw,sync,no_root_squash,fsid=77 *:/host/test

> Is there any reason you're testing ext3 instead of ext4 or xfs?
   - Not in particular. I could have used ext4.  I was evaluating nfs server 
impact in both scenarios.

>What's the client?  What are the kernel versions on server and client?

   - Both of them are at 3.10.0-693.17.1.el7.x86_64 kernel version


Regards,

Deepak


From: J. Bruce Fields <bfie...@fieldses.org>
Sent: Tuesday, February 13, 2018 11:18:51 AM
To: Deepak Jagtap
Cc: nfs-ganesha-devel@lists.sourceforge.net
Subject: Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance

On Tue, Feb 13, 2018 at 12:01:04AM +, Deepak Jagtap wrote:
> I ran few performance tests to compare nfs gansha and nfs kernel server and 
> noticed significant difference.

What is the actual test?

What are the export options in the knfsd case?

Is there any reason you're testing ext3 instead of ext4 or xfs?

What's the client?  What are the kernel versions on server and client?

--b.

> Please find my test result:
>
>
> SSD formated with EXT3 exported using nfs ganesha  : ~18K IOPSAvg 
> latency: ~8ms   Throughput: ~60MBPS
>
> same directory exported using nfs kernel server: ~75K IOPSAvg 
> latency: ~0.8msThroughput: ~300MBPS
>
>
> nfs kernel and nfs ganesha both of them are configured with 128 worker 
> threads. nfs ganesha is configured with VFS FSAL.
>
>
> Am I missing something major in nfs ganesha config or this is expected 
> behavior.
>
> Appreciate any inputs as how the performance can be improved for nfs ganesha.
>
>
>
> Please find following ganesha config file that I am using:
>
>
> NFS_Core_Param
> {
> Nb_Worker = 128 ;
> }
>
> EXPORT
> {
> # Export Id (mandatory, each EXPORT must have a unique Export_Id)
>Export_Id = 77;
># Exported path (mandatory)
>Path = /host/test;
>Protocols = 3;
># Pseudo Path (required for NFS v4)
>Pseudo = /host/test;
># Required for access (default is None)
># Could use CLIENT blocks instead
>Access_Type = RW;
># Exporting FSAL
>FSAL {
> Name = VFS;
>}
>CLIENT
>{
> Clients = *;
> Squash = None;
> Access_Type = RW;
>}
> }
>
>
>
>
> Thanks & Regards,
>
> Deepak

> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot

> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel

Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance

2018-02-13 Thread Deepak Jagtap 
Hi Matt,


I am using v2.5-stable branch for this evaluation.


Regards,

Deepak



From: Matt Benjamin <mbenj...@redhat.com>
Sent: Monday, February 12, 2018 8:02:12 PM
To: Deepak Jagtap
Cc: nfs-ganesha-devel@lists.sourceforge.net
Subject: Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance

Hi Deepak,

There's known knowns and unknown knowns related to nfs-ganesha
performance, including lots of ongoing work.

One of the first things I think we'd like to know is...what version of
the software you're testing.

Matt

On Mon, Feb 12, 2018 at 7:01 PM, Deepak Jagtap <deepak.jag...@maxta.com> wrote:
> Hey Guys,
>
>
> I ran few performance tests to compare nfs gansha and nfs kernel server and
> noticed significant difference.
>
>
> Please find my test result:
>
>
> SSD formated with EXT3 exported using nfs ganesha  : ~18K IOPSAvg
> latency: ~8ms   Throughput: ~60MBPS
>
> same directory exported using nfs kernel server: ~75K IOPS
> Avg latency: ~0.8msThroughput: ~300MBPS
>
>
> nfs kernel and nfs ganesha both of them are configured with 128 worker
> threads. nfs ganesha is configured with VFS FSAL.
>
>
> Am I missing something major in nfs ganesha config or this is expected
> behavior.
>
> Appreciate any inputs as how the performance can be improved for nfs
> ganesha.
>
>
>
> Please find following ganesha config file that I am using:
>
>
> NFS_Core_Param
> {
> Nb_Worker = 128 ;
> }
>
> EXPORT
> {
> # Export Id (mandatory, each EXPORT must have a unique Export_Id)
>Export_Id = 77;
># Exported path (mandatory)
>Path = /host/test;
>Protocols = 3;
># Pseudo Path (required for NFS v4)
>Pseudo = /host/test;
># Required for access (default is None)
># Could use CLIENT blocks instead
>Access_Type = RW;
># Exporting FSAL
>FSAL {
> Name = VFS;
>}
>CLIENT
>{
> Clients = *;
> Squash = None;
> Access_Type = RW;
>}
> }
>
>
>
> Thanks & Regards,
>
> Deepak
>
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>



--

Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel

Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance

2018-02-13 Thread J. Bruce Fields 
On Tue, Feb 13, 2018 at 12:01:04AM +, Deepak Jagtap wrote:
> I ran few performance tests to compare nfs gansha and nfs kernel server and 
> noticed significant difference.

What is the actual test?

What are the export options in the knfsd case?

Is there any reason you're testing ext3 instead of ext4 or xfs?

What's the client?  What are the kernel versions on server and client?

--b.

> Please find my test result:
> 
> 
> SSD formated with EXT3 exported using nfs ganesha  : ~18K IOPSAvg 
> latency: ~8ms   Throughput: ~60MBPS
> 
> same directory exported using nfs kernel server: ~75K IOPSAvg 
> latency: ~0.8msThroughput: ~300MBPS
> 
> 
> nfs kernel and nfs ganesha both of them are configured with 128 worker 
> threads. nfs ganesha is configured with VFS FSAL.
> 
> 
> Am I missing something major in nfs ganesha config or this is expected 
> behavior.
> 
> Appreciate any inputs as how the performance can be improved for nfs ganesha.
> 
> 
> 
> Please find following ganesha config file that I am using:
> 
> 
> NFS_Core_Param
> {
> Nb_Worker = 128 ;
> }
> 
> EXPORT
> {
> # Export Id (mandatory, each EXPORT must have a unique Export_Id)
>Export_Id = 77;
># Exported path (mandatory)
>Path = /host/test;
>Protocols = 3;
># Pseudo Path (required for NFS v4)
>Pseudo = /host/test;
># Required for access (default is None)
># Could use CLIENT blocks instead
>Access_Type = RW;
># Exporting FSAL
>FSAL {
> Name = VFS;
>}
>CLIENT
>{
> Clients = *;
> Squash = None;
> Access_Type = RW;
>}
> }
> 
> 
> 
> 
> Thanks & Regards,
> 
> Deepak

> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot

> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel

Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance

2018-02-13 Thread Daniel Gryniewicz 
Also keep in mind that FSAL VFS can never, by it's very nature, beat 
knfsd, since it has to do everything knfsd does, but also has userspace 
<-> kernespace transitions.  Ganesha's strength is exporting 
userspace-based cluster filesystems.


That said, we're always working to make Ganesha faster, and I'm sure 
there's gains to be made, even in these circumstances.


Daniel

On 02/12/2018 07:01 PM, Deepak Jagtap wrote:

Hey Guys,


I ran few performance tests to compare nfs gansha and nfs kernel server 
and noticed significant difference.



Please find my test result:


SSD formated with EXT3 exported using nfs ganesha  : ~18K IOPS    Avg 
latency: ~8ms       Throughput: ~60MBPS


same directory exported using nfs kernel server:             ~75K IOPS  
   Avg latency: ~0.8ms Throughput: ~300MBPS



nfs kernel and nfs ganesha both of them are configured with 128 
worker threads. nfs ganesha is configured with VFS FSAL.



Am I missing something major in nfs ganesha config or this is expected 
behavior.


Appreciate any inputs as how the performance can be improved for nfs 
ganesha.




Please find following ganesha config file that I am using:


NFS_Core_Param
{
         Nb_Worker = 128 ;
}

EXPORT
{
     # Export Id (mandatory, each EXPORT must have a unique Export_Id)
    Export_Id = 77;
    # Exported path (mandatory)
    Path = /host/test;
    Protocols = 3;
    # Pseudo Path (required for NFS v4)
    Pseudo = /host/test;
    # Required for access (default is None)
    # Could use CLIENT blocks instead
    Access_Type = RW;
    # Exporting FSAL
    FSAL {
         Name = VFS;
    }
    CLIENT
    {
         Clients = *;
         Squash = None;
         Access_Type = RW;
    }
}



Thanks & Regards,

Deepak



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot



___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel




--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel

Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance

2018-02-13 Thread William Allen Simpson 

On 2/13/18 1:21 AM, Malahal Naineni wrote:

If your latency is high, then you most likely need to change 
Dispatch_Max_Reqs_Xprt. What your Dispatch_Max_Reqs_Xprt value?


That shouldn't do anything anymore in V2.6, other than 9P.

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel

Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance

2018-02-12 Thread Matt Benjamin 
Hi Deepak,

There's known knowns and unknown knowns related to nfs-ganesha
performance, including lots of ongoing work.

One of the first things I think we'd like to know is...what version of
the software you're testing.

Matt

On Mon, Feb 12, 2018 at 7:01 PM, Deepak Jagtap  wrote:
> Hey Guys,
>
>
> I ran few performance tests to compare nfs gansha and nfs kernel server and
> noticed significant difference.
>
>
> Please find my test result:
>
>
> SSD formated with EXT3 exported using nfs ganesha  : ~18K IOPSAvg
> latency: ~8ms   Throughput: ~60MBPS
>
> same directory exported using nfs kernel server: ~75K IOPS
> Avg latency: ~0.8msThroughput: ~300MBPS
>
>
> nfs kernel and nfs ganesha both of them are configured with 128 worker
> threads. nfs ganesha is configured with VFS FSAL.
>
>
> Am I missing something major in nfs ganesha config or this is expected
> behavior.
>
> Appreciate any inputs as how the performance can be improved for nfs
> ganesha.
>
>
>
> Please find following ganesha config file that I am using:
>
>
> NFS_Core_Param
> {
> Nb_Worker = 128 ;
> }
>
> EXPORT
> {
> # Export Id (mandatory, each EXPORT must have a unique Export_Id)
>Export_Id = 77;
># Exported path (mandatory)
>Path = /host/test;
>Protocols = 3;
># Pseudo Path (required for NFS v4)
>Pseudo = /host/test;
># Required for access (default is None)
># Could use CLIENT blocks instead
>Access_Type = RW;
># Exporting FSAL
>FSAL {
> Name = VFS;
>}
>CLIENT
>{
> Clients = *;
> Squash = None;
> Access_Type = RW;
>}
> }
>
>
>
> Thanks & Regards,
>
> Deepak
>
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>



-- 

Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel

[Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance

2018-02-12 Thread Deepak Jagtap 
Hey Guys,


I ran few performance tests to compare nfs gansha and nfs kernel server and 
noticed significant difference.


Please find my test result:


SSD formated with EXT3 exported using nfs ganesha  : ~18K IOPSAvg latency: 
~8ms   Throughput: ~60MBPS

same directory exported using nfs kernel server: ~75K IOPSAvg 
latency: ~0.8msThroughput: ~300MBPS


nfs kernel and nfs ganesha both of them are configured with 128 worker threads. 
nfs ganesha is configured with VFS FSAL.


Am I missing something major in nfs ganesha config or this is expected behavior.

Appreciate any inputs as how the performance can be improved for nfs ganesha.



Please find following ganesha config file that I am using:


NFS_Core_Param
{
Nb_Worker = 128 ;
}

EXPORT
{
# Export Id (mandatory, each EXPORT must have a unique Export_Id)
   Export_Id = 77;
   # Exported path (mandatory)
   Path = /host/test;
   Protocols = 3;
   # Pseudo Path (required for NFS v4)
   Pseudo = /host/test;
   # Required for access (default is None)
   # Could use CLIENT blocks instead
   Access_Type = RW;
   # Exporting FSAL
   FSAL {
Name = VFS;
   }
   CLIENT
   {
Clients = *;
Squash = None;
Access_Type = RW;
   }
}




Thanks & Regards,

Deepak
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel