Hi, inline
----- Original Message ----- > From: "arun kumar" <[email protected]> > To: "Matt Benjamin" <[email protected]> > Cc: [email protected], "William Simpson" > <[email protected]> > Sent: Sunday, April 3, 2016 12:43:52 AM > Subject: Re: [Nfs-ganesha-devel] Return value of consume_ev_sig_nb() not > handled ? > > Hi Matt, > > > On Sat, Apr 2, 2016 at 8:35 PM, Matt Benjamin <[email protected]> wrote: > > > Hi Arun, > > > > ----- Original Message ----- > > > From: "arun kumar" <[email protected]> > > > To: "Matt Benjamin" <[email protected]> > > > Cc: [email protected] > > > Sent: Saturday, April 2, 2016 10:28:45 PM > > > Subject: Re: [Nfs-ganesha-devel] Return value of consume_ev_sig_nb() not > > handled ? > > > > > > Hi Matt, > > > > > > Thanks for your inputs ! Also very sorry for a very late response. > > > > > > so this socket pair is used to wake-up the epoll_wait() whenever there > > are > > > changes in the event handling via epoll_ctl() call. > > > But epoll_ctl() man page doesn't talk about waking up epoll_wait() in > > case > > > of change in event handling via epoll_ctl() call. > > > > > > Assuming that, I will have a busy network and epoll_wait() will not go > > > into sleep for long time, is this socket_pair and waking up the > > > epoll_wait() is really needed? > > > > It's hard to see this leading to any overhead, to be honest. > > > > > > > > > Also for your next question on usage of NFS Ganesha v2.1, I was using NFS > > > Ganesha when v2.1 released. But haven't modified anything into it. > > > Now as I am checking to change the RPC functionality with some user level > > > packet handling, I am checking on RPC code of NFS Ganesha. > > > > Work to do that is part of overall performance work in progress. You're > > welcome to help. You should probably look at the RDMA work that's being > > done on ntirpc's duplex-13 branch (which corresponds to nfs-ganesha's > > "next" branch). A major theme there is the new segmented XDR provider(s) > > and dispatch cleanup. User-space packet handling for TCP is intended to > > use similar primitives. For replacement of the actual sockets processing, > > we're most likely to focus on something that provides a full-featured TCP > > stack--that's probably opendataplane, I think. > > > > > > [Arun]: Thanks for your inputs. I will have a look at the duplex-13 branch > > branch. btw, currently I am trying to use mtcp which is a user level TCP > > stack and trying to use DPDK driver for packet handling from NIC. do you > > have any suggestions on this. I (we) almost certainly have lots of suggestions. :) But seriously, 1. stack -- I know of mtcp, sure (but haven't worked with it, only read a paper and what's on their web site); it sounds like a great package to experiment with 2. api/integration -- revisiting their web page (http://shader.kaist.edu/mtcp/), I see possibly why the epoll interface question came up: which appears to be that mctp emulates Linux epoll along with the sockets api (I'd forgotten that); This is going to be an area (api) where I expect some variation in what user-level stacks provide 3. zero-copy -- I see examples of applications interacting with mtcp using mtcp_read/mtcp_write and mtcp_epoll_* calls; does mtcp not have a zero-copy buffer strategy? if not the integration is simpler, and we can worry about zero-copy later; So mtcp gives you the ability to very quickly swap in a user-space stack with almost no code changes. The pattern we currently follow to handle epoll wakeups in general in Ganesha is definitely not set in stone, but I'm doubtful whether we really want to do the integration in exactly the same way. We're really most interested in providing for eventual (soon) zero-copy support, not just running the stack in user space--but it's fine, modulo some interface churn. The current tcp support is encapsulated in sc_vc and vc_generic modules in ntirpc. The sort-of minimal work to retarget that for your purposes would probably be to implement alternate read and write callouts that hook the xdr providers--they're defined in vc_generic.c. See svc_vc.c and clnt_vc.c to see how they're used. As background, Bill is in the process of a fairly significant refactoring of the Ganesha integration for svc_vc and svc_dg (udp) and ntirpc, which pulls the decoder loop down into ntirpc, giving us reduced coupling of the transport from Ganesha's dispatcher; We think this will lead to Ganesha no longer driving the event abstraction ("event channels" based on epoll) and instead having this entirely hidden within ntirpc. That said, we can deal with that later. Which brings us back to epoll. The first thing you need to think about is overall dispatch. Ganesha request processing currently has 3 stages, when processing TCP or UDP requests. A different, shorter path is followed with RDMA. You may want to look at doing things differently, and I expect you will. Looking at current TCP path, one or a small number of threads act as handlers for events in epoll, but a key decision you/we may want to revisit is the next step; The epoll_wait callout getreq_ng is ganesha code that a) checks throttling, and then if a connection is not throttled, just hands the connection off to a thread pool for decoding. In RDMA, we think we should do decoding inline, and maybe you will also. Probably, that at least that much could just become conditional in the epilogue of getreq_ng. Then the decoder thread pool, when active, decodes requests, obviously, queuing each decoded request on a second, differently dimensioned thread pool for execution. This nfs_rpc_worker_thread thread pool does the real work of executing requests and encoding results (in TCP, a separate ntpric work pool does the work of sending the actual data. I think details of this entire request path could flex depending on what meets your needs. If it's not already obvious, there's a large unstated assumption that encoder/decoder processing, as well as fsAL op processing is done synchronously, in one of the above thread contexts. This is something that is likely to work quite differently in the medium-to-longer term, but there's quite a bit two it at both of the levels (xdr encoding context and fsal op context) before async encode/decode or execution can be supported. Matt > > > > > > > But as you questioned it is better to move to the latest version and I > > will > > > do it :) > > > > Yeah, all of the ongoing work on RPC, cache bypass, and the like is > > happening on next. > > > > Regards, > > > > Matt > > > > > -- Arun > > > > > > On Wed, Mar 30, 2016 at 10:29 AM, Matt Benjamin <[email protected]> > > > wrote: > > > > > > > Also, just noticed this...why 2.1? > > > > > > > > Matt > > > > > > > > ----- Original Message ----- > > > > > From: "Matt Benjamin" <[email protected]> > > > > > To: "arun kumar" <[email protected]> > > > > > Cc: [email protected] > > > > > Sent: Wednesday, March 30, 2016 12:12:41 PM > > > > > Subject: Re: [Nfs-ganesha-devel] Return value of consume_ev_sig_nb() > > not > > > > handled ? > > > > > > > > > > Hi Arun, > > > > > > > > > > The effect is to wake any thread blocking in epoll_wait(). > > > > > > > > > > Matt > > > > > > > > > > ----- Original Message ----- > > > > > > From: "arun kumar" <[email protected]> > > > > > > To: [email protected] > > > > > > Sent: Wednesday, March 30, 2016 3:46:19 AM > > > > > > Subject: [Nfs-ganesha-devel] Return value of consume_ev_sig_nb() > > not > > > > > > handled ? > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > I am currently trying to understand the RPC mechanism of NFS > > Ganesha > > > > (using > > > > > > v2.1). > > > > > > > > > > > > In svc_rqst.c file, A socket pair is created per event channel (for > > > > UDP, > > > > > > TCP > > > > > > etc.) > > > > > > But I can't really find out what is the use of this socket pair. > > > > > > > > > > > > A new event signaled ( via ev_sig() function) whenever there are > > > > events for > > > > > > the socket epoll via epoll_ctl() call. ev_sig() just writes the > > data > > > > into > > > > > > the socket pair (writer end). > > > > > > On the other end, consume_ev_sig_nb() triggered whenever there are > > > > events > > > > > > in > > > > > > the socket pair's receiver end. > > > > > > But the caller don't process returned value from > > consume_ev_sig_nb() > > > > call, > > > > > > It > > > > > > simply ignores it. > > > > > > > > > > > > ** (void)consume_ev_sig_nb(sr_rec->sv[1]); > > > > > > > > > > > > So my question is what is the need for this socket pair in > > svc_rqst.c > > > > file, > > > > > > when the event/signal consumer is not doing any operation based on > > the > > > > > > event. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Arun > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------------ > > > > > > Transform Data into Opportunity. > > > > > > Accelerate data analysis in your applications with > > > > > > Intel Data Analytics Acceleration Library. > > > > > > Click to learn more. > > > > > > http://pubads.g.doubleclick.net/gampad/clk?id=278785471&iu=/4140 > > > > > > _______________________________________________ > > > > > > Nfs-ganesha-devel mailing list > > > > > > [email protected] > > > > > > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > > > > > > > > > > > > > > > > -- > > > > > Matt Benjamin > > > > > Red Hat, Inc. > > > > > 315 West Huron Street, Suite 140A > > > > > Ann Arbor, Michigan 48103 > > > > > > > > > > http://www.redhat.com/en/technologies/storage > > > > > > > > > > tel. 734-707-0660 > > > > > fax. 734-769-8938 > > > > > cel. 734-216-5309 > > > > > > > > > > > > > -- > > > > Matt Benjamin > > > > Red Hat, Inc. > > > > 315 West Huron Street, Suite 140A > > > > Ann Arbor, Michigan 48103 > > > > > > > > http://www.redhat.com/en/technologies/storage > > > > > > > > tel. 734-707-0660 > > > > fax. 734-769-8938 > > > > cel. 734-216-5309 > > > > > > > > > > > -- > > Matt Benjamin > > Red Hat, Inc. > > 315 West Huron Street, Suite 140A > > Ann Arbor, Michigan 48103 > > > > http://www.redhat.com/en/technologies/storage > > > > tel. 734-707-0660 > > fax. 734-769-8938 > > cel. 734-216-5309 > > > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-707-0660 fax. 734-769-8938 cel. 734-216-5309 ------------------------------------------------------------------------------ _______________________________________________ Nfs-ganesha-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
