Re: [Nfs-ganesha-devel] Proposed backports for 2.5.2
Hi Frank, On Wed, Aug 16, 2017 at 1:11 PM, Frank Filz wrote: > Oh, nice. > > > Matt, what about this one? > > > > 814e9cd65 FSAL_RGW: adopt new rgw_mount2 with bucket specified RHCS doesn't officially support this, but I'd say it would be nice to have. Matt > > > > Frank > > > > > > From: Malahal Naineni [mailto:mala...@gmail.com] > Sent: Wednesday, August 16, 2017 9:28 AM > To: Soumya Koduri > Cc: Frank Filz ; d...@redhat.com; Matt Benjamin > ; nfs-ganesha-devel > > Subject: Re: [Nfs-ganesha-devel] Proposed backports for 2.5.2 > > > > I pushed a notes branch "refs/notes/backport" which has a note saying > "backport to V2.5". You should be able to fetch this special branch with > "git fetch origin refs/notes/*:refs/notes/*". After fetching this special > branch, you should do "export GIT_NOTES_REF=refs/notes/backport" in your > SHELL and then run the usual "git log" to see if I missed any commits you > are interested in. > > > > Alternatively, the following are the commits that will NOT be back ported. > Let me know if you need any of these. I will cherry pick things tomorrow and > publish the branch, if there are no comments... > > > > 00b9e0798 Revert "CMake - Have 'make dist' generate the correct tarball > name" > > 1b60d5df2 FSAL_MEM - fix UP thread init/cleanup > > 39119aab0 FSAL_GLUSTER: Use glfs_xreaddirplus_r for readdir > > 4b4e21ed9 Manpage - Fix installing manpages in RPM > > 814e9cd65 FSAL_RGW: adopt new rgw_mount2 with bucket specified > > b862fe360 SAL: extract fs logic from nfs4_recovery > > c29114162 Napalm dispatch plus plus > > c8bc40b69 CMake - Have 'make dist' generate the correct tarball name > > cb787a1cf SAL: introduce new recovery backend based on rados kv store > > eadfc762e New (empty) sample config > > eb4eea134 config: add new config options for rados_kv recovery backend > > fbc905015 cmake: make modulized recovery backends compile as modules > > > > > > On Fri, Aug 11, 2017 at 8:08 AM, Soumya Koduri wrote: > > >> commit 7f2d461277521301a417ca368d3c7656edbfc903 >> FSAL_GLUSTER: Reset caller_garray to NULL upon free >> > > Yes > > On 08/09/2017 08:57 PM, Frank Filz wrote: > > 39119aa Soumya Koduri FSAL_GLUSTER: Use glfs_xreaddirplus_r for > readdir > > Yes? No? It's sort of a new feature, but may be critical for some use cases. > I'd rather it go into stable than end up separately backported for > downstream. > > > Right..as it is more of a new feature, wrt upstream we wanted it to be part > of only 2.6 on wards so as not to break stable branch (in case if there are > nit issues). > > But yes we may end up back-porting to downstream if we do not rebase to 2.6 > by then. > > Thanks, > Soumya > > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > > > > > Virus-free. www.avast.com > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Proposed backports for 2.5.2
I'm confused, napalm is 2.6-dev, not 2.5.x On Thu, Aug 17, 2017 at 5:13 PM, Frank Filz wrote: > Actually, the problem was Bill’s Napalm… > > > > I was able to make sense of the manual merge for > > > > ff98ea64b6d1228443a35b2f7ceb3c61c0a0c1d1 Build libntirpc package when not > using system ntirpc > > > > But not for > > > > 6bd32da613e26a768ac1dc4db1001395bd10c295 CMake - Have 'make dist' > generate the correct tarball name > > > > I pushed my results to > > > > https://github.com/ffilz/nfs-ganesha/commits/V2.5-stable > > > > Frank > > > > *From:* Malahal Naineni [mailto:mala...@gmail.com] > *Sent:* Thursday, August 17, 2017 10:39 AM > *To:* Frank Filz > *Cc:* Matt Benjamin ; Soumya Koduri < > skod...@redhat.com>; nfs-ganesha-devel sourceforge.net> > > *Subject:* Re: [Nfs-ganesha-devel] Proposed backports for 2.5.2 > > > > I did but the failover/failback code re-org looked like contributed, but I > am not positive. > > > > On Thu, Aug 17, 2017 at 7:40 PM, Frank Filz > wrote: > > Hmm, did you cherry pick in the original order? > > > > I’ll take a look at this later today. > > > > Frank > > > > *From:* Malahal Naineni [mailto:mala...@gmail.com] > *Sent:* Wednesday, August 16, 2017 11:34 PM > *To:* Matt Benjamin > *Cc:* Frank Filz ; Soumya Koduri < > skod...@redhat.com>; nfs-ganesha-devel sourceforge.net> > > > *Subject:* Re: [Nfs-ganesha-devel] Proposed backports for 2.5.2 > > > > Dan, I backported everything that was needed except the following 2 as I > don't want to mess with cmake! Can you please quickly send ported patches? > Appreciate your help. The latest V2.5 code is at my personal github branch > V2.5-stable: > > > > https://github.com/malahal/nfs-ganesha/commits/V2.5-stable > > > > The following 2 commits failed to apply: > > > > 6bd32da613e26a768ac1dc4db1001395bd10c295 CMake - Have 'make dist' > generate the correct tarball name > > ff98ea64b6d1228443a35b2f7ceb3c61c0a0c1d1 Build libntirpc package when not > using system ntirpc > > > > > > > > On Wed, Aug 16, 2017 at 10:47 PM, Matt Benjamin > wrote: > > Hi Frank, > > On Wed, Aug 16, 2017 at 1:11 PM, Frank Filz > wrote: > > Oh, nice. > > > > > > > Matt, what about this one? > > > > > > > > 814e9cd65 FSAL_RGW: adopt new rgw_mount2 with bucket specified > > RHCS doesn't officially support this, but I'd say it would be nice to have. > > Matt > > > > > > > > > > Frank > > > > > > > > > > > > From: Malahal Naineni [mailto:mala...@gmail.com] > > Sent: Wednesday, August 16, 2017 9:28 AM > > To: Soumya Koduri > > Cc: Frank Filz ; d...@redhat.com; Matt Benjamin > > ; nfs-ganesha-devel > > > > Subject: Re: [Nfs-ganesha-devel] Proposed backports for 2.5.2 > > > > > > > > I pushed a notes branch "refs/notes/backport" which has a note saying > > "backport to V2.5". You should be able to fetch this special branch with > > "git fetch origin refs/notes/*:refs/notes/*". After fetching this special > > branch, you should do "export GIT_NOTES_REF=refs/notes/backport" in your > > SHELL and then run the usual "git log" to see if I missed any commits you > > are interested in. > > > > > > > > Alternatively, the following are the commits that will NOT be back > ported. > > Let me know if you need any of these. I will cherry pick things tomorrow > and > > publish the branch, if there are no comments... > > > > > > > > 00b9e0798 Revert "CMake - Have 'make dist' generate the correct tarball > > name" > > > > 1b60d5df2 FSAL_MEM - fix UP thread init/cleanup > > > > 39119aab0 FSAL_GLUSTER: Use glfs_xreaddirplus_r for readdir > > > > 4b4e21ed9 Manpage - Fix installing manpages in RPM > > > > 814e9cd65 FSAL_RGW: adopt new rgw_mount2 with bucket specified > > > > b862fe360 SAL: extract fs logic from nfs4_recovery > > > > c29114162 Napalm dispatch plus plus > > > > c8bc40b69 CMake - Have 'make dist' generate the correct tarball name > > > > cb787a1cf SAL: introduce new recovery backend based on rados kv store > > > > eadfc762e New (empty) sample config > > > > eb4eea134 config: add new config options for rados_kv recovery backend > > > > fbc905015 cmake: make mo
Re: [Nfs-ganesha-devel] clnt_call callbacks
The 4.1 call/reply logic definitely did work, but whatever. The call flow is completely changed. As far as I know, there should be no reason to queue anything above ntirpc, after refactoring. Matt On Wed, Sep 20, 2017 at 10:08 AM, William Allen Simpson wrote: > Currently, when clnt_call() is invoked, that thread waits for the > result to come back over the network. There are 250 or so "fridge" > threads during startup for this alone. > > I've already changed the rpc_ctx_xfer_replymsg() to be transport > agnostic (it was clnt_vc only). And added a result XPRT_DISPATCH to > both NTI-RPC and NFS-Ganesha to prepare for distinguishing CALL > from REPLY dispatching. > > There's already a lot of mechanism in struct _rpc_call rpc_call_t > handling a callback. But the nfs_rpc_submit_call() flag > NFS_RPC_CALL_INLINE skipping the nfs_rpc_enqueue_req() doesn't > seem to work well (or at all). > > Ideally, we'd never queue, and an async callback would complete > the transaction. We need to discuss how to tie that together. > > My current expectation is that various fields of the Ganesha > rpc_call_t should be merged/replaced by fields in the NTI_RPC > struct svc_req so that async dispatch can handle the callback. > > But I don't really know the callback code as well as others, so > really would like some discussion and planning. > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Proposal to manage global file descriptors
also propose making the limit for the resource configurable independent >> of >> the ulimit for file descriptors, though if an FSAL is loaded that actually >> uses file descriptors for open files should check that the ulimit is big >> enough, it should also include the limit on state_t also. Of course it >> will >> be impossible to account for file descriptors used for sockets, log files, >> config files, or random libraries that like to open files... > > > Hmmm... I don't think we can do any kind of checking, if we're not going to > use ulimit by default, since it depends on which FSALs are in use at any > given time. I say we either default the limits to ulimit, or just ignore > ulimit entirely and log an appropriate error when EMFILE is returned. > > Daniel > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] The life of tcp drc
Yes, nfs_rpc_free_user_data() is the secret :) Matt On Fri, Oct 13, 2017 at 4:11 AM, Kinglong Mee wrote: > Hi Malahal, > > Thanks for your reply. > > #1. I'd like to post a patch to delete it, because I have some cleanups for > drc. > #2/#3. Sorry for my missing of nfs_rpc_free_user_data(). With it, everything > is okay. > > thanks, > Kinglong Mee > > On 10/13/2017 15:52, Malahal Naineni wrote: >> #1. Looks like a bug! Lines 629 and 630 should be deleted >> #2. See nfs_rpc_free_user_data(). It sets xp_u2 to NULL and drc ref is >> decremented there. >> #3. Life time of drc should start when it is allocated in >> nfs_dupreq_get_drc() using alloc_tcp_drc(). >> It can live beyond xprt's xp_u2 setting to NULL. It will live until we >> decide to free in drc_free_expired() using free_tcp_drc(). >> >> Regards, Malahal. >> PS: The comment "drc cache maintains a ref count." seems to imply that it >> will have a refcount for keeping it in the hash table itself. I may have >> kept those two lines because of that but It doesn't make sense as refcnt >> will never go to zero this way. >> >> On Thu, Oct 12, 2017 at 3:48 PM, Kinglong Mee > <mailto:kinglong...@gmail.com>> wrote: >> >> Describes in src/RPCAL/nfs_dupreq.c, >> >> * The life of tcp drc: it gets allocated when we process the first >> * request on the connection. It is put into rbtree (tcp_drc_recycle_t). >> * drc cache maintains a ref count. Every request as well as the xprt >> * holds a ref count. Its ref count should go to zero when the >> * connection's xprt gets freed (all requests should be completed on the >> * xprt by this time). When the ref count goes to zero, it is also put >> * into a recycle queue (tcp_drc_recycle_q). When a reconnection >> * happens, we hope to find the same drc that was used before, and the >> * ref count goes up again. At the same time, the drc will be removed >> * from the recycle queue. Only drc's with ref count zero end up in the >> * recycle queue. If a reconnection doesn't happen in time, the drc gets >> * freed by drc_free_expired() after some period of inactivety. >> >> Some questions about the life time of tcp drc, >> 1. The are two references of drc for xprt in nfs_dupreq_get_drc(). >>629 /* xprt ref */ >>630 drc->refcnt = 1; >>... >>638 (void)nfs_dupreq_ref_drc(drc); /* xprt >> ref */ >>... >>653 req->rq_xprt->xp_u2 = (void *)drc; >> >>I think it's a bug. The first one needs remove. Right? >> >> 2. The is no place to decrease the reference of drc for xprt. >>The xprt argument in nfs_dupreq_put_drc() is unused. >>Should it be used to decrease the ref? >>I think it's the right place to decrease the ref in >> nfs_dupreq_put_drc(). >> >> 3. My doubts is that, the life time of drc stored in req->rq_xprt->xp_u2 >> ? >>Start at #1, end at #2 (req->rq_xprt->xp_u2 = NULL) ? >>If that, the bad case is always lookup drc from tcp_drc_recycle_t. >> >>Otherwise, don't put the reference at #2, when to put it? >>the bad case is the drc ref always be 1 forever, am I right? >> >> thanks, >> Kinglong Mee >> >> >> -- >> Check out the vibrant tech community on one of the world's most >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >> ___ >> Nfs-ganesha-devel mailing list >> Nfs-ganesha-devel@lists.sourceforge.net >> <mailto:Nfs-ganesha-devel@lists.sourceforge.net> >> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel >> <https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel> >> >> > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] nlm_async retries
A few people know this code. One is Frank. I don't immediately see the reason for strong concern, feel free to improve. Matt On Wed, Nov 1, 2017 at 5:20 AM, William Allen Simpson wrote: > I'm flummoxed. Who knows this code? > > Problem 1: the timeout is set to 10 microseconds. Holy heck? And > historically, that's the maximum total wait time, so it would try at > least three (3) times within 10 *MICRO*seconds? > > Probably should be milliseconds. > > Problem 2: there's a retry loop that sets up and tears down a TCP > (or UDP) connection 3 times. On top of the 3 tries in RPC itself? > > This looks like a lot of self-flagellation, maybe because the > timeout above was set too short? > > Problem 3: this isn't really async -- nlm_send_async() actually > runs a pthread_cond_timedwait() before returning. That's sync! > > But we already have a timedwait in RPC. And a signal. So this > completely duplicates the underlying RPC library code. Why? > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Change in ffilz/nfs-ganesha[next]: CLNT_CALL with clnt_req
oh, come on. not sure what needs to be done to reduce log noise, but I'm sure we can make a dent. Matt On Sat, Nov 4, 2017 at 1:38 AM, William Allen Simpson wrote: > We already discussed this on Tuesday, October 24th. Malahal agreed > that a half second was good, 3 seconds was OK, 5 seconds was long. > And Matt agreed we'd log more than 10 seconds. > > Obviously, you have vastly more Internet experience than I, and > therefore are much better able to decide Internet timing parameters. > > Also, your time is so much more valuable than mine, and you need to > post the weekly maintenance updates late (eastern time) on Friday to > avoid disrupting your thought processes -- so that I have to work > weekends, and my commits are stalled by yet another week. > > > > Note that I'm currently taking a lot of half-week holiday over the > next few months, so hopefully this will make it in *early* next week. > Then maybe, just maybe, my next patch will be ready the week before > Thanksgiving > > On 11/3/17 2:04 PM, GerritHub wrote: >> >> Frank Filz *posted comments* on this change. >> >> View Change <https://review.gerrithub.io/385451> >> >> Patch set 1: >> >> I'd like a review by malahal before merging this one >> >> I'm really not sure about the 3 second timeouts >> >> Can anyone test this in something resembling a real customer environment? >> >> To view, visit change 385451 <https://review.gerrithub.io/385451>. To >> unsubscribe, visit settings <https://review.gerrithub.io/settings>. >> >> Gerrit-Project: ffilz/nfs-ganesha >> Gerrit-Branch: next >> Gerrit-MessageType: comment >> Gerrit-Change-Id: I92b02eca435f4b1f6104b740c6c5b3747c380840 >> Gerrit-Change-Number: 385451 >> Gerrit-PatchSet: 1 >> Gerrit-Owner: william.allen.simp...@gmail.com >> Gerrit-Reviewer: CEA-HPC >> Gerrit-Reviewer: Daniel Gryniewicz >> Gerrit-Reviewer: Frank Filz >> Gerrit-Reviewer: Gluster Community Jenkins >> Gerrit-Reviewer: Malahal >> Gerrit-Reviewer: openstack-ci-service+rdo-ci-cen...@redhat.com >> Gerrit-Comment-Date: Fri, 03 Nov 2017 18:04:27 + >> Gerrit-HasComments: No > > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Ganesha 2.5 - mdc_readdir_chunk_object :INODE :CRIT :Collision while adding dirent for .nfsFD8E
cookie == 7fff == 2^31 looks maybe just a bit suspicious? Matt On Tue, Nov 7, 2017 at 10:14 AM, Sachin Punadikar wrote: > Hello, > During tests on Ganesha 2.5, we are getting below logs with the critical > message: > 2017-11-03 05:30:05 : epoch 000100d3 : c40abc1pn13.gpfs.net : > ganesha.nfsd-36297[work-226] mdcache_avl_insert_ck :INODE :WARN :Already > existent when inserting dirent 0x3ffbe8015a60 for .nfsFD8E on > entry=0x3ffb08019ed0 FSAL cookie=7fff, duplicated directory cookies make > READDIR unreliable. > 2017-11-03 05:30:05 : epoch 000100d3 : c40abc1pn13.gpfs.net : > ganesha.nfsd-36297[work-226] mdc_readdir_chunk_object :INODE :CRIT > :Collision while adding dirent for .nfsFD8E > > Would like to understand what exactly mean by FSAL cookie collision ? Does > it mean same operation has been done by UPCALL thread ? Is the message > really CRIT ? > If I compare with 2.3 code (I know there is lot of change related to > caching), there we are not throwing any CRIT message. > > -- > with regards, > Sachin Punadikar > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Stacked FSALs and fsal_export parameters and op_ctx
I'd like to see this use of TLS as a "hidden parameter" replaced regardless. It has been a source of bugs, and locks us into a pthreads execution model I think needlessly. Matt On Fri, Dec 8, 2017 at 10:07 AM, Frank Filz wrote: >> On 12/7/17 7:54 PM, Frank Filz wrote: >> > Stacked FSALs often depend on op_ctx->fsal_export being set. >> > >> > We also have lots of FSAL methods that take the fsal_export as a >> parameter. >> > >> The latter sounds better. >> >> Now that we know every single thread local storage access involves a hidden >> lock/unlock sequence in glibc "magically" invoked by the linker, it would be >> better to remove as many TLS references as possible! >> >> After all, too many lock/unlock are a real performance issue. >> >> Perhaps we should pass op_ctx as the parameter instead. > > I thought the lock was only to create the TLS variable, and not on every > reference. > > Frank > > > --- > This email has been checked for viruses by Avast antivirus software. > https://www.avast.com/antivirus > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] libntirpc thread local storage
I've already proposed we remove this. No one is invested in it, I don't think. Matt On Dec 10, 2017 2:38 AM, "William Allen Simpson" < william.allen.simp...@gmail.com> wrote: > I've run into another TLS problem. It's been there since tirpc. > > Apparently, once upon a time, rpc_createerr was a static global. > It still says that in the man pages. > > When a client create function fails, they stash the error there, > and return NULL for the CLIENT. Basically, you check for NULL, > and then check rpc_createerr > > This is also used extensively by the RPC bind code. > > Then, they made it a keyed thread local to try and salvage it > without a major code re-write. > > With async threads, that's not working. We've got memory leaks. > AFAICT, only on errors. But accessing them on a different > thread gives the wrong error code (or none at all). Not good. > > All the functions that use it are still not MT-safe, usually > because they stash a string in global memory without locking. > They need re-definition to alloc/free the string. > > Worse, it's not a good definition. > > rpc_createerr has both clnt_stat and rpc_err, but struct rpc_err > also has a clnt_stat (since original tirpc). clnt_stat is not > consistently set properly, as it is in two places. So the error > checking code is often wrong. > > I'd like to get rid of the whole mess, but that means every client > create would have new semantics. Fortunately, there aren't many > (in Ganesha). Plus we already have new definitions -- all named > *_ncreate with a tirpc_compat.h to munge them back. > > But should we do it now? Or in 2.7? We've been living with it for > years, although recent code changes have made it worse. Again, it > only happens on errors. Especially for RPC bind. > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] XID missing in error path for RPC AUTH failure.
That sounds right, I'm uncertain whether this has regressed in the
text, or maybe in the likelihood of inlining in the new dispatch
model. Bill?
Matt
On Wed, Dec 13, 2017 at 9:38 AM, Pradeep wrote:
> Hello,
>
> When using krb5 exports, I noticed that TIRPC does not send XID in response
> - see xdr_reply_encode() for MSG_DENIED case. Looks like Linux clients can't
> decode the message and go in to an infinite loop retrying the same NFS
> operation. I tried adding XID back (like it is done for normal case) and it
> seems to have fixed the problem. Is this the right thing to do?
>
> diff --git a/src/rpc_dplx_msg.c b/src/rpc_dplx_msg.c
> index 01e5a5c..a585e8a 100644
> --- a/src/rpc_dplx_msg.c
> +++ b/src/rpc_dplx_msg.c
> @@ -194,9 +194,12 @@ xdr_reply_encode(XDR *xdrs, struct rpc_msg *dmsg)
> __warnx(TIRPC_DEBUG_FLAG_RPC_MSG,
> "%s:%u DENIED AUTH",
> __func__, __LINE__);
> - buf = XDR_INLINE(xdrs, 2 * BYTES_PER_XDR_UNIT);
> + buf = XDR_INLINE(xdrs, 5 * BYTES_PER_XDR_UNIT);
>
> if (buf != NULL) {
> + IXDR_PUT_INT32(buf, dmsg->rm_xid);
> + IXDR_PUT_ENUM(buf, dmsg->rm_direction);
> + IXDR_PUT_ENUM(buf, dmsg->rm_reply.rp_stat);
> IXDR_PUT_ENUM(buf, rr->rj_stat);
> IXDR_PUT_ENUM(buf, rr->rj_why);
> } else if (!xdr_putenum(xdrs, rr->rj_stat)) {
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>
--
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101
fax. 734-769-8938
cel. 734-216-5309
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] NFS-Ganesha v2.5.2 Ceph and RGW Peformance
Hi Supriti, On Thu, Dec 14, 2017 at 4:26 PM, Supriti Singh wrote: > Hello all, > > At SUSE, we did a performance benchmarking for nfs-ganesha v2.5.2 for FSAL > Ceph and RGW. For FSAL CEPH, we did a comparison for kernel cephfs client > and nfs-ganesha. Please find report attached. > > Key points > 1. Earlier, I shared results on ceph-devel mailing list as well. I have > added comparison between those results and current results on page 9. In > earlier test, I did not tune the parameters "Dispatch_Max_Reqs" and cephfs > default pool size. That could be one reason, why ganesha performs better in > the current test. Thanks to Malahal for pointing out the parameters. > > 2. For multiple clients and multiple jobs, single nfs-ganesha server > performance degrades significantly. As we go ahead, active-active > nfs-ganesha server or pnfs may improve performance. Any thoughts on this? I > did stumble upon Jeff's blog: > https://jtlayton.wordpress.com/2017/11/07/active-active-nfs-over-cephfs/ Is > this something already in loop for 2.7? I have reason to believe that the request queuing and throttling behavior in versions before 2.6 is responsible for a lot of the degradation, even with good tuning. I'm very interested to see your results with 2.6. Looking forwards, yes, I think pNFS hsa the potential to help out a lot. Also, I believe there are available speedups in libcephfs, but I haven't worked with it in a long time. > > 3. For NFS-RGW, as it supports only sync write, I was not able to use fio > for testing. Is there any other tool that someone else has used? The semantics of FSAL_RGW were designed as those of upload, not full NFS semantics, so the usual benchmarks dont really work. Im not sure what to suggest, beyond less general benchmarking, like copying up some large files. I'm working on removing the restricted upload semantics in Mimic. > > 4. For 2.6, I am aware that "Dispatch_Max_Reqs" goes away. But I have not > looked into code, so please excuse if question is not to the point. Are > those changes expected to improve performance? Yes. Fairness between clients should be improved. You may want to experiment with different (larger) values of nb_worker in 2.6. post 2.6 we're moving to async dispatch. > > 5. Also, I think it would be nice to have event messages for tunable > parameters. So that if nfs-ganesha is slowing down, because some parameters > have reached threshold values, users can understand for the log messages. I > am only aware of health messages, but it does not explain what could be > wrong. > > Please let me know your feedback, if I missed out on some optimization or > analysis. > > Thanks, > Supriti > > -- > Supriti Singh > SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, > HRB 21284 (AG Nürnberg) > > Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] NTIRPC ENOMEM
The established plan is to finish async and evaluate. Matt On Dec 20, 2017 5:02 AM, "William Allen Simpson" < william.allen.simp...@gmail.com> wrote: > DanG has raised an interesting issue about recovery from low memory. > In Ganesha, we've been assiduously changing NULL checks to assert or > segfault on alloc failures. Just had a few more patches by Kaleb. > > Since 2013 or 2014, we've been doing the same to NTIRPC. There are > currently 105 mem_.*alloc locations, and almost all of them > deliberately segfault. > > DanG argues that we should report the ENOMEM in an error return, or > in the alternative return NULL in those cases, and let the caller > decide what to do, to make the library more general. > > The current TI-RPC does return NULL in many cases. Rarely reports > ENOMEM. And often segfaults. > > This would be a major reworking. Should we do this? If so, what is > the target date? > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Implement a FSAL for S3-compatible storage
That would work. The logic that the RGW FSAL uses to do all of Frank describes is in the Ceph source rather than FSAL_RGW for the same reason. The strategy I take is to represent the RGW file handle as concatenated hashes of S3/swift container name and full object name, which yields stable handles. For directories, cookies are also the hash of the entry name. Frank's whence-is-name and compute-readdir-cookie apis were invented to support the RGW FSAL. Using them, you avoid the need to keep an indexed representation of the S3 namespace in the FSAL (or in my case, librgw). Matt On Thu, Jan 4, 2018 at 7:18 AM, DENIEL Philippe wrote: > Hi Aurélien, > > I can provide you an alternate solution, still nfs-ganesha based. For the > need of a project, I developed an open-source library that emulate a POSIX > namespace using a KVS (for metadata) and an object store (for data). For > example, you can use REDIS and RADOS. I have written a FSAL for it (it is > not pushed in the official branch) but with no compliancy to support_ex, > it's still using the former FSAL semantics (so it should be ported to > support_ex). If you are interested, I can give you some pointers (the code > is on github). You could use S3 as data storage for example. In particular, > I had to solve the same "inode" issue that you met. This solution as very > few impact on nfs-ganesha code (it just adds a new FSAL). > > Regards > > Philippe > > On 01/03/18 19:58, Aurelien RAINONE wrote: > > To follow up on the development on an FSAL for S3, I have some doubts and > questions I'd like to share. > > Apart from its full path, S3 doesn't have the concept of file descriptor, I > mean, there's nothing else > > than the full path that I can provide to S3 in order to get attribute of > content of a specific object. > > I have some doubts regarding the implementation of the S3 fsal object handle > (s3_fsal_obj_handle). > > > > Should s3_fsal_obj_handle be very simple, for example should it only contain > a key that maps to the full S3 filename, in an key-value store. > > Or on the contrary, should the handle implement a tree like structure, like > I saw in FSAL_MEM? > > Or something in between, but what? > > Having a very simple handle has some advantages but may require some more > frequent network calls, > > for example readdir won't have any kind of information about the content of > the directory. > > Having a whole tree-like structure in the handle would allow to have direct > access to directory content, > > but isn't that the role of ganesha cache to do that? > > My questions probably shows that I have problems to understand the > responsability of my FSAL implementation > > regarding the cache. Who does what, what doesn't do what? > > Good evening, > > Aurélien > > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > > > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Patches not backported to V2.5-stable
I9ef314125c5eb86fd21c449787349b78f32da1df Rework delegations to >> use new >> lease_op2 interface >> > Ia1f36cfaa8bd49474d7c3cf103af2515edea3fce nfs: fix error handling >> in >> nfs_rpc_v41_single >> > Ie9ff2d2181e184b2dfef7b6b2b793d1f40e7bb82 V2.6-dev.8 >> > I808ec1051b2ef58309b14f3b1fdc42d866d5c196 nfs: use >> OPEN_DELEGATE_NONE_EXT >> when not granting a delegation on v4.1+ >> > Ic0c0eca52182009289634a985c2e4d1e34dcb895 Take a reference to the >> session >> over a v4.1+ callback >> > I4c6407aacb4f119fa856d0857df60b7d19791e55 nfs: fix cleanup after >> delegrecall >> > Ic6fac7c66d802b25dece9aa0c3d34ac2671b8f74 nfs: >> nfs41_complete_single -> >> nfs41_release_single >> > Ic774170db363f3782754672d5ae5675603ee1a41 FSAL_PROXY : code >> cleaning, >> remove useless comments >> > I80adb7c583c2c28e57d0f75686220cb474d65367 FSAL_PROXY : storing >> stateid >> from background NFS server >> > If675f156b253247d2d0ac1f5cb82aa38e9eb83bb FSAL_RGW: Remove obsolete >> (non-support_ex) create method >> > I6b049361f205e662551d382750766dddfc090652 V2.6-dev.7 >> > I9f24cc76405ba8ddede2395abadb04660b57840b Fix RPM Release tag when >> no >> GANESHA_EXTRA_VERSION >> > I8c6da75bb3f3266fe1871143bb97f01ba383066e nfs: make delegation >> recalls use >> new callback infrastructure >> > I517ff36a9276bbda75d550a96be0574a630bc783 nfs: make the single-op >> callback >> helper be minorversion neutral >> > Ie3341c897d4b9141c91b20f634fb8aa8d58202f3 nfs: remove "free_op" >> argument >> from nfs_rpc_v41_single >> > Ib7f6b8ec0a66e753cb40744123c136e7d8153fdb Remove non-support_ex >> FSAL open, >> reopen, and status methods >> > If40959b04cfb39ef56fabf082a8aa8367bb9c7ba Remove non-support_ex >> FSAL >> commit method >> > I042a4b3352926fc5484f2aac9fcc0c8b2ff0cf14 Remove non-support_ex >> FSAL >> share_op method >> > Ic1f43f41f970e48633006d02f3f7fdf48fa80717 Remove non-support_ex >> lock_op >> and fso_lock_support_owner >> > I01405fe6b788e16f7b24f89b9ddd72ed037e623a Remove non-support_ex >> create >> FSAL method >> > I7c986c444f17c83aecd730569f105f6e649b1ca3 Remove non-support_ex >> setattrs >> FSAL method >> > I24b8c9d292d2869540df0c22d420541283f16662 Remove the >> non-support_ex read >> and write methods >> > I09e17c305ba4c90e2699251940649a91882e8c9f Remove support_ex FSAL >> method >> > I069b41f0d497bbbc79fbb2b9dea1149bb3a58475 Assume support_ex in >> FSAL/fsal_helper.c and include/fsal.h >> > Id87f54ffe86911604bbcf270ae095c385a04fc25 Remove share counters >> from SAL - >> WARNING - delegations have been broken >> > I54f5a2ad56ce6feb0bcbd52950c741174c1c4b93 Replace calls to >> state_share_anonymous_io_start with state_deleg_conflict >> > I3a6f021d344ddafd95f1309c8189e91a2faf9aa1 Assume support_ex in >> NLM_SHARE/NLM_UNSHARE handling >> > Iab3ec0848624757c0f93944aac2a781f7e1ca601 Strip legacy state_share >> functions used by NFS4 >> > I580d8615ba33e71488956960c9f4bd4f553d511e Assume support_ex in >> SAL/state_lock.c >> > I97528bcf148c25d4fb7509c1cd02943e6f1dcc99 Disable do_lease_op - >> FSAL >> lock_op method is not implemented by anyone >> > Ic3cf431ccb02f30774ce7d402b50a3ce642f05da Assume support_ex in >> SAL/nfs4_state.c >> > I0b6b8136f47ac47b03ab1f12436a5d4a428c5f02 Assume support_ex in NFS4 >> protocol functions >> > Id9fc3e0c6ce76b377a55ba96d086e825c3312685 Always assume >> support_ex in NFS3 >> protocol functions >> > I1505e5e606d2a360e3c58833d692aa70883fe00f Always assume >> support_ex in 9p >> > I45ed9dd89fab9dcad902340021b11ee31f17dbe6 (libntirpc) update pull >> request >> #70 & #71 >> > I51517b281ffca7411a6a229e07088028ca336c44 V2.6-dev.6 >> > Ib5ce7e184a2029ff36830e8b0d59d96df3f717fa Napalm nfs_worker_thread >> NFS_REQUEST queue >> > Ie6a6d625cf114091bec2e6785602154b9f2df6dc DBUS interface for >> purging >> idmapper cache. >> > Ia53c8eec07a840425877e03ec58682c42d512b34 FSAL_PROXY : add >> verbosity in >> EXCHANGE_ID >> > Icb67b2df86a6060c68a8b2c16389d9b44c8aafe5 Remove libzfswrap >> > Id020de3fb0d91e31d3eb81c86a2ef9f3b9097ce7 Strip FSAL_ZFS out >> > I2611365be9f2f342760c35c6023ee4ab02766fa9 V2.6-dev.5 >> > Ie2aec841612e3270f2a7904fa88eda39db93c190 V2.6-dev.4a >> > Ibc31cf3745a5a20a4236a9a37712572d1f0f87f4 V2.6-dev-4 >> > I6f2dfd9dc4431df6372d30e3c6510b290ec9e8de CMake - Have 'make dist' >> generate the correct tarball name >> > I0428d3c316a12fc1cab750f745640a50c03a34cc FSAL_MEM - fix UP thread >> init/cleanup >> > I9c0810bfb211dd133b3f33e04036b57f69ef0c4f V2.6-dev-3 >> > I81a5e85ad5eb9935a09937d1262b959fdc7cabb9 Napalm dispatch plus plus >> > Icc0c7d806e2e8dcaa715d72f26a98d5f7f71c77d Revert "CMake - Have >> 'make dist' >> generate the correct tarball name" >> > I6925dc73cf930bb8cfe747baab1642164045 V2.6-dev-2 >> > I10dc7925db271eab2bcd3f9a035ffbdfb21e2450 CMake - Have 'make dist' >> generate the correct tarball name >> > I195759fd0c1394651d9bf188eb19229ebcf46f68 V2.6-dev-1 >> >> >> --- >> This email has been checked for viruses by Avast antivirus software. >> https://www.avast.com/antivirus <https://www.avast.com/antivirus> >> >> >> >> -- >> Check out the vibrant tech community on one of the world's most >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >> ___ >> Nfs-ganesha-devel mailing list >> Nfs-ganesha-devel@lists.sourceforge.net >> <mailto:Nfs-ganesha-devel@lists.sourceforge.net> >> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel >> <https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel> >> >> >> >> >> >> -- >> Check out the vibrant tech community on one of the world's most >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >> >> >> >> ___ >> Nfs-ganesha-devel mailing list >> Nfs-ganesha-devel@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel >> > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] LizardFS NFS Ganesha integration with no shared library available
I don't know for certain if GPLv3 is an acceptable license for the files in an in-tree FSAL. Probably? Any NFS-Ganesha common files or changes to same need to be under our prevailing license (or a successor license, but I don't believe we have that clause), of course. Matt On Thu, Jan 18, 2018 at 8:58 AM, Jonathan Dieter wrote: > On Thu, 2018-01-18 at 07:34 -0500, Kaleb S. KEITHLEY wrote: >> On 01/18/2018 03:23 AM, Jonathan Dieter wrote: >> > What is the best way for me to include NFS Ganesha support in LizardFS? >> >1. Include the latest Fedora NFS Ganesha source and add a hard requires >> > to that version in the LizardFS NFS subpackage. >> >2. Convince the LizardFS developers to try to move their FSAL into NFS >> > Ganesha? (LizardFS has just added a client library and doesn't have >> > a server library, so this will probably take some work) >> >> That would be the path of least resistance. And the one I would suggest >> and highly recommend. If they've already written a FSAL it should be >> easy to add it to the nfs-ganesha source. >> >> Depending on the license of course. > > If this is both the easiest and the recommended path, then lets see if > the LizardFS developers will go for it. The license for this code is > GPLv3. > >> >3. Convince the NFS Ganesha developers to create a library that you can >> > compile FSAL's against? (The last thread I saw in the NFS Ganesha >> > devel list on this subject was six years old) >> >> That would be a non-trivial amount of work with little benefit to the >> current FSAL developers. >> >> It's not an unreasonable request, per se, but we have other, higher >> priorities. > > I totally understand that. > >> >4. ??? >> > >> > Any advice would be great, as I'd love to get this feature into >> > Fedora's release of LizardFS. >> >> Do you have a contact for someone at LizardFS? I looked at their web >> site and nothing popped out at me. > > The main developer that I've communicated with is: > Piotr Sarna > > Jonathan > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] 'missed' recv with 2.6-rc2?
Hmm. We used to handle that ;) Matt On Thu, Jan 25, 2018 at 9:11 PM, Pradeep wrote: > If recv() returns EAGAIN, then svc_vc_recv() returns without rearming the > epoll_fd. How does it get back to svc_vc_recv() again? > > On Wed, Jan 24, 2018 at 9:26 PM, Pradeep wrote: >> >> Hello, >> >> I seem to be hitting a corner case where ganesha (2.6-rc2) does not >> respond to a RENEW request from 4.0 client. Enabled the debug logs and >> noticed that NFS layer has not seen the RENEW request (I can see it in >> tcpdump). >> >> I collected netstat output periodically and found that there is a time >> window of ~60 sec where the receive buffer size remains the same. This means >> the RPC layer somehow missed a 'recv' call. Now if I enable debug on TIRPC, >> I can't reproduce the issue. Any pointers to potential races where I could >> enable selective prints would be helpful. >> >> svc_rqst_epoll_event() resets SVC_XPRT_FLAG_ADDED. Is it possible for >> another thread to svc_rqst_rearm_events()? In that case if >> svc_rqst_epoll_event() could reset the flag set by svc_rqst_rearm_events and >> complete the current receive before the other thread could call epoll_ctl(), >> right? >> >> Thanks, >> Pradeep > > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] 'missed' recv with 2.6-rc2?
Yes, I wasn't claiming there is anything missing. Before 2.6, there was a rearm method being called. Matt On Fri, Jan 26, 2018 at 9:20 AM, Daniel Gryniewicz wrote: > I don't think you re-arm a FD in epoll. You arm it once, and it fires until > you disarm it, as far as I know. You just call epoll_wait() to get new > events. > > The thread model is a bit odd; When the epoll fires, all the events are > found, and a thread is submitted for each one except one. That one is > handled in the local thread (since it's expected that most epoll triggers > will have one event on them, thus using the current hot thread). In > addition, a new thread is submitted to go back and wait for events, so > there's no delay handling new events. So EAGAIN is handled by just > indicating this thread is done, and returning it to the thread pool. When > the socket is ready again, it will trigger a new event on the thread waiting > on the epoll. > > Bill, please correct me if I'm wrong. > > Daniel > > > On 01/25/2018 09:13 PM, Matt Benjamin wrote: >> >> Hmm. We used to handle that ;) >> >> Matt >> >> On Thu, Jan 25, 2018 at 9:11 PM, Pradeep wrote: >>> >>> If recv() returns EAGAIN, then svc_vc_recv() returns without rearming the >>> epoll_fd. How does it get back to svc_vc_recv() again? >>> >>> On Wed, Jan 24, 2018 at 9:26 PM, Pradeep wrote: >>>> >>>> >>>> Hello, >>>> >>>> I seem to be hitting a corner case where ganesha (2.6-rc2) does not >>>> respond to a RENEW request from 4.0 client. Enabled the debug logs and >>>> noticed that NFS layer has not seen the RENEW request (I can see it in >>>> tcpdump). >>>> >>>> I collected netstat output periodically and found that there is a time >>>> window of ~60 sec where the receive buffer size remains the same. This >>>> means >>>> the RPC layer somehow missed a 'recv' call. Now if I enable debug on >>>> TIRPC, >>>> I can't reproduce the issue. Any pointers to potential races where I >>>> could >>>> enable selective prints would be helpful. >>>> >>>> svc_rqst_epoll_event() resets SVC_XPRT_FLAG_ADDED. Is it possible for >>>> another thread to svc_rqst_rearm_events()? In that case if >>>> svc_rqst_epoll_event() could reset the flag set by svc_rqst_rearm_events >>>> and >>>> complete the current receive before the other thread could call >>>> epoll_ctl(), >>>> right? >>>> >>>> Thanks, >>>> Pradeep >>> >>> >>> >>> >>> >>> -- >>> Check out the vibrant tech community on one of the world's most >>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >>> _______ >>> Nfs-ganesha-devel mailing list >>> Nfs-ganesha-devel@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel >>> >> >> >> > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Is there a field in the SVCXPRT Ganesha can use
That sounds right, I figured we could reclaim xp_u1. Matt On Tue, Jan 30, 2018 at 9:22 AM, William Allen Simpson wrote: > On 1/29/18 3:32 PM, Frank Filz wrote: >> >> I haven't looked at how the SVCXPRT structure has changed, but if there's >> a >> field in there we can attach a Ganesha structure to that would be cool, or >> if not, if we could add one. >> > There are two: xp_u1, and xp_u2. > > Right now, Ganesha is using xp_u2 for dup request cache pointers. > > But I've eliminated all old usage of xp_u1 in V2.6. > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Is there a field in the SVCXPRT Ganesha can use
I don't think that's the case. the behavior is distinct for v4 and v3 (which does not have clientids). DRC is bypassed for v4.1+ Matt On Tue, Jan 30, 2018 at 9:36 AM, William Allen Simpson wrote: > On 1/30/18 9:22 AM, William Allen Simpson wrote: >> >> On 1/29/18 3:32 PM, Frank Filz wrote: >>> >>> I haven't looked at how the SVCXPRT structure has changed, but if there's >>> a >>> field in there we can attach a Ganesha structure to that would be cool, >>> or >>> if not, if we could add one. >>> >> There are two: xp_u1, and xp_u2. >> >> Right now, Ganesha is using xp_u2 for dup request cache pointers. >> >> But I've eliminated all old usage of xp_u1 in V2.6. > > > Looking at src/RPCAL/nfs_dupreq.c, I'm not sure why that doesn't already > have a client or export reference there. It seems we'll return the > duplicate data to any client that happens to use the same xid. Seems > like a bug > > But the code is obscure, so I could be missing something. > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Is there a field in the SVCXPRT Ganesha can use
innovations? ok. there's other innovation needed in the gss impl, that would be welcome as well Matt On Tue, Jan 30, 2018 at 9:50 AM, William Allen Simpson wrote: > On 1/30/18 9:36 AM, William Allen Simpson wrote: >> >> But the code is obscure, so I could be missing something. > > > Also, it bears repeating that the dupreq cache wasn't working for > secure connections. Pre-V2.6 checksummed the ciphertext, which is by > definition different on every request. We'd never see duplicates. > > One of the innovations in V2.6 (ntirpc 1.6) is the checksum is of the > plaintext. So duplicate requests will be detected. > > I'm not sure how often we have duplicate requests, but it should be > working now. > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Is there a field in the SVCXPRT Ganesha can use
Restating this, the code is unified for tcp v4 and v3. It consults xid and the checksum, which I think is a correct intent. The slot-reply cache in 4.1+ is, we hope, the improved mechanism. Matt On Tue, Jan 30, 2018 at 9:45 AM, Matt Benjamin wrote: > I don't think that's the case. the behavior is distinct for v4 and v3 > (which does not have clientids). DRC is bypassed for v4.1+ > > Matt > > On Tue, Jan 30, 2018 at 9:36 AM, William Allen Simpson > wrote: >> On 1/30/18 9:22 AM, William Allen Simpson wrote: >>> >>> On 1/29/18 3:32 PM, Frank Filz wrote: >>>> >>>> I haven't looked at how the SVCXPRT structure has changed, but if there's >>>> a >>>> field in there we can attach a Ganesha structure to that would be cool, >>>> or >>>> if not, if we could add one. >>>> >>> There are two: xp_u1, and xp_u2. >>> >>> Right now, Ganesha is using xp_u2 for dup request cache pointers. >>> >>> But I've eliminated all old usage of xp_u1 in V2.6. >> >> >> Looking at src/RPCAL/nfs_dupreq.c, I'm not sure why that doesn't already >> have a client or export reference there. It seems we'll return the >> duplicate data to any client that happens to use the same xid. Seems >> like a bug >> >> But the code is obscure, so I could be missing something. >> >> >> -- >> Check out the vibrant tech community on one of the world's most >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >> ___ >> Nfs-ganesha-devel mailing list >> Nfs-ganesha-devel@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > > > > -- > > Matt Benjamin > Red Hat, Inc. > 315 West Huron Street, Suite 140A > Ann Arbor, Michigan 48103 > > http://www.redhat.com/en/technologies/storage > > tel. 734-821-5101 > fax. 734-769-8938 > cel. 734-216-5309 -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Correct initialization sequence
reordering, I hope Matt On Tue, Jan 30, 2018 at 1:40 PM, Pradeep wrote: > Hello, > > It is possible to receive requests anytime after nfs_Init_svc() is > completed. We initialize several things in nfs_Init() after this. This could > lead to processing of incoming requests racing with the rest of > initialization (ex: dupreq2_pkginit()). Is it possible to re-order > nfs_Init_svc() so that rest of ganesha is ready to process requests as soon > as we start listing on the NFS port? Another way is to return NFS4ERR_DELAY > until 'nfs_init.init_complete' is true. Any thoughts? > > > Thanks, > Pradeep > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance
Hi Deepak,
There's known knowns and unknown knowns related to nfs-ganesha
performance, including lots of ongoing work.
One of the first things I think we'd like to know is...what version of
the software you're testing.
Matt
On Mon, Feb 12, 2018 at 7:01 PM, Deepak Jagtap wrote:
> Hey Guys,
>
>
> I ran few performance tests to compare nfs gansha and nfs kernel server and
> noticed significant difference.
>
>
> Please find my test result:
>
>
> SSD formated with EXT3 exported using nfs ganesha : ~18K IOPSAvg
> latency: ~8ms Throughput: ~60MBPS
>
> same directory exported using nfs kernel server: ~75K IOPS
> Avg latency: ~0.8msThroughput: ~300MBPS
>
>
> nfs kernel and nfs ganesha both of them are configured with 128 worker
> threads. nfs ganesha is configured with VFS FSAL.
>
>
> Am I missing something major in nfs ganesha config or this is expected
> behavior.
>
> Appreciate any inputs as how the performance can be improved for nfs
> ganesha.
>
>
>
> Please find following ganesha config file that I am using:
>
>
> NFS_Core_Param
> {
> Nb_Worker = 128 ;
> }
>
> EXPORT
> {
> # Export Id (mandatory, each EXPORT must have a unique Export_Id)
>Export_Id = 77;
># Exported path (mandatory)
>Path = /host/test;
>Protocols = 3;
># Pseudo Path (required for NFS v4)
>Pseudo = /host/test;
># Required for access (default is None)
># Could use CLIENT blocks instead
>Access_Type = RW;
># Exporting FSAL
>FSAL {
> Name = VFS;
>}
>CLIENT
>{
> Clients = *;
> Squash = None;
> Access_Type = RW;
>}
> }
>
>
>
> Thanks & Regards,
>
> Deepak
>
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>
--
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101
fax. 734-769-8938
cel. 734-216-5309
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance
On Fri, Feb 16, 2018 at 11:23 AM, William Allen Simpson wrote: >> > Actually, 2.6 should handle as many concurrent client requests as you like. > (Up to 250 of them.) That's one of its features. > > The client is not sending concurrent requests. This seems worth digging into. That's my understanding of 2.6 dispatch too, anyway. > >> > But the planned 2.7 improvements are mostly throughput related, not IOPS. Not at all, though I am trying to ensure that we get async FSAL ops in. There are people working on IOPs too. > > > If Ganesha is adding 6 ms to every read operation, we have a serious > problem, and need to profile immediately! > That's kind of what our team is doing. I look forward to your work with rpc-ping-mt. Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance
Hi, On Tue, Feb 20, 2018 at 8:12 AM, William Allen Simpson wrote: > On 2/18/18 2:47 PM, Matt Benjamin wrote: >> >> On Fri, Feb 16, 2018 at 11:23 AM, William Allen Simpson >>> >>> But the planned 2.7 improvements are mostly throughput related, not IOPS. >> >> >> Not at all, though I am trying to ensure that we get async FSAL ops >> in. There are people working on IOPs too. >> > async FSAL ops are not likely to further improve IOPS. > > As I've pointed out many times in the past, async only allows > the same number of threads to handle more concurrent operations. > > But it's actually slower. It basically doubles the number of > system calls. System calls are one of the reasons that Ganesha is > slower than knfsd. It also ruins CPU cache coherency. > > If we're CPU bound or network bandwidth constrained, it won't help. Not everything being worked on is a direct answer to this problem. > > The effort that I've been waiting for *3* years -- add io vectors to > the FSAL interface, so that we can zero-copy buffers -- is more > likely to improve throughput. Which, as you've noted, also is happening. > > Moreover, going through the code and removing locking other such > bottlenecks is more likely to improve IOPS. No one is disputing this. It is not a new discovery, however. > > >>> If Ganesha is adding 6 ms to every read operation, we have a serious >>> problem, and need to profile immediately! >>> >> >> That's kind of what our team is doing. I look forward to your work >> with rpc-ping-mt. >> > Well, you were able to send me a copy after 6 pm on a Friday, so I'm > now taking a look at it. Hopefully I'll have something by Friday. It came up in a meeting on Friday, that's why I sent it Friday. I got busy in meetings and issues, that's why after 6 pm. > > But I really wish you'd posted it 3 years ago. It doesn't really test > IOPS, other than whatever bandwidth limit is imposed by the interface, > but it does test the client call interface. It measures minimal request latency all the way to nfs-ganesha's, or the Linux kernel's, rpcnull handler--the "top" of the request handling stack, in the given client/server/network configuration. Scaled up, it can report the max such calls/s, which is a kind of best-possible value for iops, taking FSAL ops to have 0 latency. It was posted to this list by Tigran, iirc, in 2011 or 2012. > > We've been using Jeff Layton's delegation callback work to test, and > this test would have been better and easier. > > But a unit test is not what we need. I wrote "profile". We need to > know where the CPU bottlenecks are in Ganesha itself. You also wrote unit test. Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance
Hi Bill, Let's just talk about the rpc-ping-mt/request minimal latency. On Wed, Feb 21, 2018 at 8:24 AM, William Allen Simpson wrote: >> >> It measures minimal request latency all the way to nfs-ganesha's, or >> the Linux kernel's, rpcnull handler--the "top" of the request handling >> stack, in the given client/server/network configuration. Scaled up, >> it can report the max such calls/s, which is a kind of best-possible >> value for iops, taking FSAL ops to have 0 latency. >> > As I've mentioned elsewhere, this should be entirely dominated by the > link speed and protocol. We should see UDP as slowest, TCP in the > middle, and RDMA as fastest. I think the point is, the "should" assumes perfect behavior from the dispatch and (minimal) request execution path within nfs-ganesha, does it not? In other words, profiling of rpc null is effectively profiling the first stage of the request pipeline, and if it's not fast, more interesting ops won't be. > > OTOH, the "max such calls/s" would be reported by using XDR_RAW, which > is currently not working. > My intended meaning, and Daniel's when he articulated the same point Friday, is that "max such calls/s" -is- meaningful for transports NFS actually supports, and those are the ones we can compare with other implementations. I think the raw transport opens up some interesting paths, but it seems like they would involve more development. Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Multiprotocol support in ganesha
Basically, this. For the sake of discussion, is Daniel's take on SMB integration directly in nfs-ganesha what you're thinking of, or were you ireferring to co-export of Linux filesystems with Samba or some other Linux-integrated SMB stack? Matt On Tue, Mar 6, 2018 at 12:29 PM, Daniel Gryniewicz wrote: > Ganesha has multi-protocol (NFS3, NFS4, and 9P). There are no plans to add > CIFS, since that is an insanely complicated protocol, and has a userspace > daemon implementation already (in the form of Samba). I personally wouldn't > reject such support if it was offered, but as far as I know, no one is even > thinking about working on it. > > Daniel > > > On 03/06/2018 12:20 PM, Pradeep wrote: >> >> Hello, >> >> Is there plans to implement multiprotocol (NFS and CIFS accessing same >> export/share) in ganesha? I believe current FD cache will need changes to >> support that. >> >> Thanks, >> Pradeep >> >> >> >> -- >> Check out the vibrant tech community on one of the world's most >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >> >> >> >> ___ >> Nfs-ganesha-devel mailing list >> Nfs-ganesha-devel@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel >> > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] review request https://review.gerrithub.io/#/c/390652/
Hi Satya, To reply, to a reply on the top level (can even be blank), all your inline comments will publish then. Matt On Fri, Mar 9, 2018 at 11:21 AM, Satya Prakash GS wrote: > I had replied to the comments on the same day Matt posted. My replies show > as drafts, looks like I have to publish them. I don't see a publish button > either. Can you guys help me out. > > Thanks, > Satya. > > On 9 Mar 2018 20:48, "Frank Filz" wrote: >> >> Matt had called for additional discussion on this, so let's get that >> discussion going. >> >> Could you address Matt's questions? >> >> Frank >> >> > -Original Message- >> > From: Satya Prakash GS [mailto:g.satyaprak...@gmail.com] >> > Sent: Friday, March 9, 2018 4:17 AM >> > To: nfs-ganesha-devel@lists.sourceforge.net >> > Cc: Malahal Naineni ; Frank Filz >> > >> > Subject: review request https://review.gerrithub.io/#/c/390652/ >> > >> > Can somebody please review this change : >> > https://review.gerrithub.io/#/c/390652/ >> > >> > It addresses this issue : >> > >> > Leak in DRC when client disconnects nfs_dupreq_finish doesn't call >> > put_drc >> > always. It does only if it meets certain criteria (drc_should_retire). >> > This can leak >> > the drc and the dupreq entries within it when the client disconnects. >> > More >> > information can be found here : https://sourceforge.net/p/nfs- >> > ganesha/mailman/message/35815930/ >> > >> > >> > >> > Main idea behind the change. >> > >> > Introduced a new drc queue which holds all the active drc objects >> > (tcp_drc_q in >> > drc_st). >> > Every new drc is added to tcp_drc_q initially. Eventually it is moved to >> > tcp_drc_recycle_q. Drcs are freed from tcp_drc_recycle_q. Every drc is >> > either in >> > the active drc queue or in the recycle queue. >> > >> > DRC Refcount and transition from active drc to recycle queue : >> > >> > Drc refcnt is initialized to 2. In dupreq_start, increment the drc >> > refcount. In >> > dupreq_rele, decrement the drc refcnt. Drc refcnt is also decremented in >> > nfs_rpc_free_user_data. When drc refcnt goes to 0 and drc is found not >> > in use >> > for 10 minutes, pick it up and free the entries in iterations of 32 >> > items at at time. >> > Once the dupreq entries goes to 0, remove the drc from tcp_drc_q and add >> > it to >> > tcp_drc_recycle_q. Today, entries added to tcp_drc_recycle_q are cleaned >> > up >> > periodically. Same logic should clean up these entries too. >> > >> > Thanks, >> > Satya. >> > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] zero-copy read
Hi Bill, On Sat, Mar 10, 2018 at 10:24 AM, William Allen Simpson wrote: > Now that DanG has a workable vector i-o for read and write, I'm > trying again to make reading zero-copy. Man-oh-man, do we have > our work cut out for us > > It seems that currently we provide a buffer to read. Then XDR > makes a new object, puts headers into it, makes another data_val > and copies data into that, then it is all eventually passed to > ntirpc where a buffer is created and all copied into that. > > If GSS, another copy is made. (This one cannot be avoided.) > > So we're copying large amounts of data 4-5 times. Not counting > whatever the FSAL library call does internally. Sounds like you have lots to work on. > > Then there is NFS_LOOKAHEAD_READ, and a nfs_request_lookahead. > Could somebody explain what that is doing? These are just tracking the ops being decoded, currently to help decide if reqs should be cached. It copies no data. > > AFAICT, the only test is in dup_req, and it won't keep the dup_req > "because large, though idempotent". Isn't a large read exactly > where we'd benefit from holding onto a dup_req? Basically, no. The purpose of the DRC is to identify requests that change the state of files/objects (are non-idempotent), and to satisfy a client's retry of such a request, without re-executing it. The goal is not to be a data cache. > > NFS_LOOKAHEAD_HIGH_LATENCY is never used. It's a constant. > > There are a lot of XDR tests for x_public having the pointer to > nfs_request_lookahead, yet setting that pointer is one of the > early things in nfs_worker_thread.c nfs_rpc_process_request(). It sets the structure in x_public so that it can be accessed from within XDR decoders. > > Finally, and what I'll do this weekend, my attempt to edit > xdr_nfs23.c won't pass checkpatch commit, because all the headers > are still pre-1989 pre-ANSI K&R-style. > > Unfortunately, Red Hat Linux doesn't seem to have cproto built-in, > even though it's on the usual https://linux.die.net/man/1/cproto. > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] zero-copy read
Marcus has code that prototypes using gss_iov from mit-krb5 1.1.12. I recall describing this to you in 2013. On Sat, Mar 10, 2018 at 11:12 AM, William Allen Simpson wrote: > On 3/10/18 10:24 AM, William Allen Simpson wrote: > > But as I delved deeper, I'll have to make GSS work on vector i-o, as > it currently requires one big buffer input. This will be awhile. > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] zero-copy read
No, you won't. Matt On Sun, Mar 11, 2018 at 7:15 AM, William Allen Simpson wrote: > On 3/10/18 11:18 AM, Matt Benjamin wrote: >> >> Marcus has code that prototypes using gss_iov from mit-krb5 1.1.12. I >> recall describing this to you in 2013. >> > That would be surprising, as I didn't start working on this project > until a year or so later than that > > Anyway, last year Marcus sent me a link to his prototype. It's > hopelessly out of date by now. I'll need to start over. -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] rpcping
That's certainly suggestive. I found it hard to believe the red-black tree performance could be that bad, at a loading of 10K items--even inserting, searching, and removing in-order. Then again, I never benchmarked the opr_rbtree code. If I understand correctly, we always insert records in xid order, and xid is monotonically increasing by 1. I guess pings might come back in any order, but if we assume xids retire in xid order also, and keep a window of 1 records in-tree, that seems maybe like a reasonable starting point for measuring this? I wrote a gtest program (gerrit) that I think does the above in a single thread, no locks, for 1M cycles (search, remove, insert). On lemon, compiled at O2, the gtest profiling says the test finishes in less than 150ms (I saw as low as 124). That's over 6M cycles/s, I think. Matt Matt On Mon, Mar 12, 2018 at 4:06 PM, William Allen Simpson wrote: > [These are with a Ganesha that doesn't dupreq cache the null operation.] > > Just how slow is this RB tree? > > Here's a comparison of 1000 entries versus 100 entries in ops per second: > > rpcping tcp localhost threads=5 count=1000 (port=2049 program=13 > version=3 procedure=0): average 2963.2517, total 14816.2587 > rpcping tcp localhost threads=5 count=1000 (port=2049 program=13 > version=3 procedure=0): average 3999.0897, total 19995.4486 > > rpcping tcp localhost threads=5 count=100 (port=2049 program=13 > version=3 procedure=0): average 39738.1842, total 198690.9208 > rpcping tcp localhost threads=5 count=100 (port=2049 program=13 > version=3 procedure=0): average 39913.1032, total 199565.5161 -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] rpcping
On Tue, Mar 13, 2018 at 2:38 AM, William Allen Simpson wrote: > On 3/12/18 6:25 PM, Matt Benjamin wrote: >> >> If I understand correctly, we always insert records in xid order, and >> xid is monotonically increasing by 1. I guess pings might come back >> in any order, > > > No, they always come back in order. This is TCP. I've gone to some > lengths to fix the problem that operations were being executed in > arbitrary order. (As was reported in the past.) We're aware of the issues with former req queuing. It was one of my top priorities to fix in napalm, and we did it. > > For UDP, there is always the possibility of loss or re-ordering of > datagrams, one of the reasons for switching to TCP in NFSv3 (and > eliminating UDP in NFSv4). > > Threads can still block in apparently random order, because of > timing variances inside FSAL calls. Should not be an issue here. > > >> but if we assume xids retire in xid order also, > > > They do. Should be no variance. Eliminating the dupreq caching -- > also using the rbtree -- significantly improved the timing. It's certainly correct not to cache, but it's also a special case that arises from...benchmarking with rpcping, not NFS. Same goes for retire order. Who said, let's assume the rpcping requests retire in order? Oh yes, me above. Do you think NFS requests in general are required to retire in arrival order? No, of course not. What workload is the general case for the DRC? NFS. > > Apparently picked the worst tree choice for this data, according to > computer science. If all you have is a hammer What motivates you to write this stuff? Here are two facts you may have overlooked: 1. The DRC has a constant insert-delete workload, and for this application, IIRC, I put the last inserted entries directly into the cache. This both applies standard art on trees (rbtree vs avl perfomance on insert/delete heavy workloads, and ostensibly avoids searching the tree in the common case; I measured hitrate informally, looked to be working). 2. the key in the DRC caches is hk,not xid. > > >> and keep >> a window of 1 records in-tree, that seems maybe like a reasonable >> starting point for measuring this? >> I've not tried 10,000 or 100,000 recently. (The original code > > default sent 100,000.) > > I've not recorded how many remain in-tree during the run. > > In my measurements, using the new CLNT_CALL_BACK(), the client thread > starts sending a stream of pings. In every case, it peaks at a > relatively stable rate. > > For 1,000, <4,000/s. For 100, 40,000/s. Fairly linear relationship. > > By running multiple threads, I showed that each individual thread ran > roughly the same (on average). But there is some variance per run. > > I only posted the 5 thread results, lowest and highest achieved. > > My original message had up to 200 threads and 4 results, but I decided > such a long series was overkill, so removed them before sending. > > That 4,000 and 40,000 per client thread was stable across all runs. > > >> I wrote a gtest program (gerrit) that I think does the above in a >> single thread, no locks, for 1M cycles (search, remove, insert). On >> lemon, compiled at O2, the gtest profiling says the test finishes in >> less than 150ms (I saw as low as 124). That's over 6M cycles/s, I >> think. >> > What have you compared it to? Need a gtest of avl and tailq with the > same data. That's what the papers I looked at do The point is, that is very low latency, a lot less than I expected. It's probably minimized from CPU caching and so forth, but it tries to address the more basic question, is expected or unexpected latency from searching the rb tree a likely contributor to overall latency? If we get 2M retires per sec (let alone 6-7), is that a likely supposition? The rb tree either is, or isn't a major contributor to latency. We'll ditch it if it is. Substituting a tailq (linear search) seems an unlikely choice, but if you can prove your case with the numbers, no one's going to object. Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] rpcping
Daniel doesn't think you've measured much accurately yet, but at least the effort (if not the discussion) aims to. On Wed, Mar 14, 2018 at 2:54 AM, William Allen Simpson wrote: Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] rpcping
Hi Bill, I was not (not intentionally, and, I think, not at all) imputing sentiment to Daniel nor myself. I was paraphrasing statements Daniel made not only informally to me, but publically in this week's nfs-ganesha call, in which you were present. I'm not denigrating your work, but I did dispute your conclusion on the performance of rbtree in the client in a specific scenario, for which I provided detailed measurement and a program for reproducing them. I did also challenge the unambiguous claim by you that something about my use of rbtree in ntirpc and DRC was an embarrassment to computing science, but did not use inappropriate language or imputations (fighting words) in doing so. THAT appears to be a denigration of my work by you, not the other way around. There were other examples in that email, but it was the most glaring. I could go into that pattern further, but I don't think I can or should on a public mailing list. I won't post further on this or similar threads. Matt On Wed, Mar 14, 2018 at 4:27 PM, William Allen Simpson wrote: > On 3/14/18 7:27 AM, Matt Benjamin wrote: >> >> Daniel doesn't think you've measured much accurately yet, but at least >> the effort (if not the discussion) aims to. >> > I'm sure Daniel can speak for himself. At your time of writing, > Daniel had not yet arrived in the office after my post this am. > > So I'm assuming you're speculating. Or denigrating my work and > attributing that sentiment to Daniel. I'd appreciate you cease > doing that. > > I've done my best with the Tigran's code design that you held onto > for 6 years without putting it into the tree or keeping it up-to-date. > > At this time, there's no indication any numbers are in error. > > If you have quantitative information, please provide it. -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] rpcping profile
1 What is the peak outstanding size of outstanding calls 1.1 if e.g. > 100k is that correct: as last week, why would a sensible client issue more than e.g. 1000 calls without seeing replies? 1.3 if outstanding calls is <= 1, why can test_rbt retire millions of duty cycles / s in that scenario? 2 what does the search workload look like when replies are mixed with calls? Ie bidirectional rpc this is intended for? 2.2 Hint: xid dist is not generally sorted; client defines only its own issue order, not reply order nor peer xids; why is it safe to base reply matching around xids being in sorted order? Matt On Sun, Mar 25, 2018, 1:40 PM William Allen Simpson < william.allen.simp...@gmail.com> wrote: > On 3/24/18 7:50 AM, William Allen Simpson wrote: > > Noting that the top problem is exactly my prediction by knowledge of > > the code: > >clnt_req_callback() opr_rbtree_insert() > > > > The second is also exactly as expected: > > > >svc_rqst_expire_insert() opr_rbtree_insert() svc_rqst_expire_cmpf() > > > > These are both inserted in ascending order, sorted in ascending order, > > and removed in ascending order > > > > QED: rb_tree is a poor data structure for this purpose. > > I've replaced those 2 rbtrees with TAILQ, so that we are not > spending 49% of the time there anymore, and am now seeing: > > rpcping tcp localhost count=1000 threads=1 workers=5 (port=2049 > program=13 version=3 procedure=0): mean 151800.6287, total 151800.6287 > rpcping tcp localhost count=1000 threads=1 workers=5 (port=2049 > program=13 version=3 procedure=0): mean 167828.8817, total 167828.8817 > > This is probably good enough for now. Time to move on to > more interesting things. > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] rpcping profile
With N=10 and num_calls=100, on Lemon, test_rbt averages 2.8M reqs/s. That's about half the rate when N=1, which I think is expected. If this is really an available rbt in-order search-remove-insert retire rate when N is 10, my intuition would be it's sufficiently fast not to be the bottleneck your result claims, and I think it's necessary to understand why. Matt On Sun, Mar 25, 2018 at 6:17 PM, Matt Benjamin wrote: > 1 What is the peak outstanding size of outstanding calls > > 1.1 if e.g. > 100k is that correct: as last week, why would a sensible > client issue more than e.g. 1000 calls without seeing replies? > > 1.3 if outstanding calls is <= 1, why can test_rbt retire millions of > duty cycles / s in that scenario? > > 2 what does the search workload look like when replies are mixed with calls? > Ie bidirectional rpc this is intended for? > > 2.2 Hint: xid dist is not generally sorted; client defines only its own > issue order, not reply order nor peer xids; why is it safe to base reply > matching around xids being in sorted order? > > Matt > > On Sun, Mar 25, 2018, 1:40 PM William Allen Simpson > wrote: >> >> On 3/24/18 7:50 AM, William Allen Simpson wrote: >> > Noting that the top problem is exactly my prediction by knowledge of >> > the code: >> >clnt_req_callback() opr_rbtree_insert() >> > >> > The second is also exactly as expected: >> > >> >svc_rqst_expire_insert() opr_rbtree_insert() svc_rqst_expire_cmpf() >> > >> > These are both inserted in ascending order, sorted in ascending order, >> > and removed in ascending order >> > >> > QED: rb_tree is a poor data structure for this purpose. >> >> I've replaced those 2 rbtrees with TAILQ, so that we are not >> spending 49% of the time there anymore, and am now seeing: >> >> rpcping tcp localhost count=1000 threads=1 workers=5 (port=2049 >> program=13 version=3 procedure=0): mean 151800.6287, total 151800.6287 >> rpcping tcp localhost count=1000 threads=1 workers=5 (port=2049 >> program=13 version=3 procedure=0): mean 167828.8817, total 167828.8817 >> >> This is probably good enough for now. Time to move on to >> more interesting things. >> >> >> -- >> Check out the vibrant tech community on one of the world's most >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >> ___ >> Nfs-ganesha-devel mailing list >> Nfs-ganesha-devel@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] nfstest_delegation
In order to test delegations, Ganesha needs to be in a FSAL configuration that supports them. To the best of my understanding, delegations are enabled by default for GPFS, which until recently was the only FSAL which supported delegations. Jeff Layton recently added delegation support for CEPH. You need a recent Ceph to use them, due to a libcephfs API dependency, iiuc. Matt On Wed, Mar 28, 2018 at 7:26 AM, Malahal Naineni wrote: > Yes, it should be default if the code is stable! > > regards, malahal. > > On Wed, Mar 28, 2018 at 3:49 PM, William Allen Simpson > wrote: >> >> I see that Patrice hasn't posted here about this problem yet. >> >> Linux client folks say our V2.7-dev delegations aren't working. >> >> At this week's bake-a-thon, Patrice has tried turning it on a >> couple of different ways. Shouldn't delegations be on by default? >> >> Could we get the nfstest suite added to CI? >> >> >> -- >> Check out the vibrant tech community on one of the world's most >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >> ___ >> Nfs-ganesha-devel mailing list >> Nfs-ganesha-devel@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] mdcache growing beyond limits.
Hi Frank, On Tue, Apr 3, 2018 at 11:33 AM, Frank Filz wrote: > Thanks for the explanation. You are observing in practice something I > considered in theory... > > I like the idea of demoting entries when entries > entries_hiwat, Matt, > Daniel, do you see any negative side effects to that? It sounds reasonable to me. I don't remember anything that would make it problematic. > > About the open fds, my thinking is shifting to not keeping an fd open for > getattr/settattr. If the global fd is already open use it, otherwise, just > open a temp fd for the operation. With NFS v4 clients, that will virtually > eliminate global fd usage and for V3 clients will mean the global fd is only > open for files the client is doing I/O on. That sounds like a great idea. Matt > > Frank > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Ganesha2.3 failure on ppc arch identified
Hi, It looks like some call sites (e.g., GPFS) just continue. I don't know whether that's a safe default? That's a pretty large default stack, but I see it's configurable as THREAD_STACK_SIZE and WORK_POOL_STACK_SIZE (but hard coded in GPFS, LUSTRE, and GLUSTER). Matt - Original Message - From: "Malahal Naineni" To: Nfs-ganesha-devel@lists.sourceforge.net Sent: Thursday, July 30, 2015 11:22:24 PM Subject: [Nfs-ganesha-devel] Ganesha2.3 failure on ppc arch identified Hi Frank and Matt, For some odd reason, pthread_attr_setstacksize() fails on ppc with EAGAIN. It should only fail with EINVAL if passed in value WORK_POOL_STACK_SIZE doesn't pass some restrictions. Oddly, the macro value is OFF by one for 16 byte alignment, but correcting it didn't seem to help either. Completely removing that call itself worked fine. The error code is wrong on pppc and we have a proof that 2116488 works based on already existing code in GPFS_FSAL. I will investigate this more, but I wish the code just crashed or continued to create the thread instead of bailing out prematurely. Patch will follow for ntirpc. Thank you all for your time for looking at possible endian issues with me. Regards, Malahal. -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Ganesha2.3 failure on ppc arch identified
pthread return: sounds like a glibc bug (sorry). proposal: Sounds reasonable to me. Matt - Original Message - From: "Malahal Naineni" To: "Matt Benjamin" Cc: Nfs-ganesha-devel@lists.sourceforge.net Sent: Friday, July 31, 2015 12:17:07 PM Subject: Re: [Nfs-ganesha-devel] Ganesha2.3 failure on ppc arch identified pthread_attr_setstacksize() should return EINVAL if the passed in size is less than PTHREAD_STACK_MIN. PTHREAD_STACK_MIN is 128K our systems (2 pages), so it should have returned EINVAL but looks like there is a bug causing it to return EAGAIN. :-( In any case, current WORK_POOL_STACK_SIZE is one byte less than 64K. I will correct it unless there is a valid reason for that lone byte. Will fix the actual problem that the passed in value is at least PTHREAD_STACK_MIN as well. Regards, Malahal. Matt Benjamin [mbenja...@redhat.com] wrote: > Hi, > > It looks like some call sites (e.g., GPFS) just continue. I don't know > whether that's > a safe default? > > That's a pretty large default stack, but I see it's configurable as > THREAD_STACK_SIZE > and WORK_POOL_STACK_SIZE (but hard coded in GPFS, LUSTRE, and GLUSTER). > > Matt > > - Original Message - > From: "Malahal Naineni" > To: Nfs-ganesha-devel@lists.sourceforge.net > Sent: Thursday, July 30, 2015 11:22:24 PM > Subject: [Nfs-ganesha-devel] Ganesha2.3 failure on ppc arch identified > > Hi Frank and Matt, > > For some odd reason, pthread_attr_setstacksize() fails on ppc with > EAGAIN. It should only fail with EINVAL if passed in value > WORK_POOL_STACK_SIZE doesn't pass some restrictions. Oddly, the macro > value is OFF by one for 16 byte alignment, but correcting it didn't seem > to help either. Completely removing that call itself worked fine. > > The error code is wrong on pppc and we have a proof that 2116488 works > based on already existing code in GPFS_FSAL. > > I will investigate this more, but I wish the code just crashed or > continued to create the thread instead of bailing out prematurely. Patch > will follow for ntirpc. Thank you all for your time for looking at > possible endian issues with me. > > Regards, Malahal. > > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] thr_decode_rpc_requests exiting too soon
Hi Malahal, The code in nfs_rpc_continue_decoding calls poll(), and was intended to provide an extra chance to keep a decoder thread around, rather than re-arming epoll and re-running all the prologue. I also see that the code in question has been replaced with #if 0. I'm not clear how that went upstream. I think it should likely be restored. It might be plausible to introduce a short delay and counter that keeps decoder threads alive slightly longer, when idling out. Past some small limit, it's better to exit. Matt - Original Message - From: "Malahal Naineni" To: Nfs-ganesha-devel@lists.sourceforge.net Cc: mbenja...@redhat.com, "william allen simpson" Sent: Monday, August 10, 2015 12:16:34 AM Subject: thr_decode_rpc_requests exiting too soon Hi Matt and Bill, With our recent testing, looks like thr_decode_rpc_requests is existing too soon, most likely due to SVC_RECV() returning false and SVC_STAT returning XPRT_IDLE. This is probably the reason for heavy contention on the decoder fridge mutex. I will get some stats on these soon, but is there a way for SVC_RECV to wait for the data (with a possible timeout) instead of quickly returning false? Regards, Malahal. -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] svc_ioq_callback() issues
Hi Malahal, - Original Message - > From: "Malahal Naineni" > To: "Matt Benjamin" > Cc: Nfs-ganesha-devel@lists.sourceforge.net, "william allen simpson" > , "Daniel > Gryniewicz" > Sent: Tuesday, August 11, 2015 3:26:23 AM > Subject: svc_ioq_callback() issues > > Thank you Matt. > > "git blame" does indicate that you experimented with it but it was > disabled due to not making much difference with throughput. Makes sense. > I didn't go deeper into the git history. Let's restore it in general. > > I enabled it and the decoder fridge mutex has gone out from the > contention list. We see work pool mutex (svc_work_pool.pqh.qmutex) > at the top now, so I am trying to fix this. I made a small patch so > that a work pool thread waits for 1 full second before working on a > different task or waiting in the global pool. Those seem to be different cases. I'm more comfortable with delaying on return to global pool, than I am with just biasing towards in favor of whatever transport we were most recently sending for. > > I used xd->shared.ioq.qcount as a condition. It is not used in the > current code but seems to be appropriately updated. My patch normally > works in my unit testing but crashes with the real workload. The crash > indicated that xd->shared.ioq.qcount was 8 but there was nothing in the > actual list. So the call TAILQ_REMOVE in svc_ioq_callback() seg faulted! > I could use TAILQ_EMPTY as a condition variable but wondering why this > isn't working. Bill and Matt, it is a small patch and appreciate your > input on why this may fail. > > Another coredump happened while making XDR_DESTROY() in > svc_ioq_callback(). I must be not honoring proper refcounts > some where, causing memory corruption??? Evidently you're doing something illegal here, but perhaps you're just exposing a bug that was already present. We certainly want to rule that out. I have some concerns about the idea as a whole. You're moving locks in the mutrace report--how much has performance changed in your workload? Is throughput going up, and by how much?I'm not sold on the change as a whole, really, as above. Matt > > Here is the patch (top commit here): > https://github.com/malahal/ntirpc/commits/adp3 > > Regards, Malahal. > > Matt Benjamin [mbenja...@redhat.com] wrote: > > Hi Malahal, > > > > The code in nfs_rpc_continue_decoding calls poll(), and was intended to > > provide an extra chance to keep a decoder thread around, rather than > > re-arming epoll and re-running all the prologue. I also see that the code > > in question has been replaced with #if 0. I'm not clear how that went > > upstream. I think it should likely be restored. It might be plausible to > > introduce a short delay and counter that keeps decoder threads alive > > slightly longer, when idling out. Past some small limit, it's better to > > exit. > > > > Matt > > > > - Original Message - > > From: "Malahal Naineni" > > To: Nfs-ganesha-devel@lists.sourceforge.net > > Cc: mbenja...@redhat.com, "william allen simpson" > > > > Sent: Monday, August 10, 2015 12:16:34 AM > > Subject: thr_decode_rpc_requests exiting too soon > > > > Hi Matt and Bill, > > > > With our recent testing, looks like thr_decode_rpc_requests is existing > > too soon, most likely due to SVC_RECV() returning false and SVC_STAT > > returning XPRT_IDLE. This is probably the reason for heavy contention on > > the decoder fridge mutex. I will get some stats on these soon, but is > > there a way for SVC_RECV to wait for the data (with a possible timeout) > > instead of quickly returning false? > > > > Regards, Malahal. > > > > -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] svc_ioq_callback() issues
Hi Malahal, - Original Message - > From: "Malahal Naineni" > To: "Matt Benjamin" > Cc: Nfs-ganesha-devel@lists.sourceforge.net, "william allen simpson" > , "Daniel > Gryniewicz" > Sent: Tuesday, August 11, 2015 11:04:04 AM > Subject: Re: svc_ioq_callback() issues > > > I have some concerns about the idea as a whole. You're moving locks in > > the mutrace report--how much has performance changed in your workload? > > Is throughput going up, and by how much?I'm not sold on the > > change as a whole, really, as above. > > I am not sold either! We did various things in the config and small > patches that brought about 80% improvement in ops/sec. That's a good result--when you get time, I look forward to some breakdown on what changes had what effect. Matt > > Regards, Malahal. > > -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] AVL Code
Hi, I sourced the AVL code, and I made those additions. Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "Frank Filz" > To: "DENIEL Philippe" , > nfs-ganesha-devel@lists.sourceforge.net > Sent: Wednesday, August 12, 2015 3:21:34 PM > Subject: [Nfs-ganesha-devel] AVL Code > > Philippe, > > It looks like maybe you pulled the AVL code from here: > > https://github.com/fbuihuu/libtree/commit/513f463141dc36c5c05197274fd2658fc3 > 697707 > > And then added avltree_size and avltree_inf. > > Unfortunately at that point, the code is licensed LGPLv2 which can not just > be converted to LGPLv3. > > I'd like to move to the current HEAD of that tree to pick up LGPLv2.1+ > licensing (which we can then convert to LGPLv3), but I need to figure out > the provenance of the two above additions before I do so (and then maybe > resolve some other issues due to other changes in the AVL code). > > Thanks > > Frank > > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] AVL Code
excellent, thank you -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "Frank Filz" > To: "Matt Benjamin" > Cc: "DENIEL Philippe" , > nfs-ganesha-devel@lists.sourceforge.net > Sent: Wednesday, August 12, 2015 5:22:38 PM > Subject: RE: [Nfs-ganesha-devel] AVL Code > > > I sourced the AVL code, and I made those additions. > > Ok, thanks Matt. I will make those changes to the current version. > > Thanks > > Frank > > > Matt Benjamin > > Red Hat, Inc. > > 315 West Huron Street, Suite 140A > > Ann Arbor, Michigan 48103 > > > > http://www.redhat.com/en/technologies/storage > > > > tel. 734-761-4689 > > fax. 734-769-8938 > > cel. 734-216-5309 > > > > - Original Message - > > > From: "Frank Filz" > > > To: "DENIEL Philippe" , > > > nfs-ganesha-devel@lists.sourceforge.net > > > Sent: Wednesday, August 12, 2015 3:21:34 PM > > > Subject: [Nfs-ganesha-devel] AVL Code > > > > > > Philippe, > > > > > > It looks like maybe you pulled the AVL code from here: > > > > > > > > https://github.com/fbuihuu/libtree/commit/513f463141dc36c5c05197274fd2 > > > 658fc3 > > > 697707 > > > > > > And then added avltree_size and avltree_inf. > > > > > > Unfortunately at that point, the code is licensed LGPLv2 which can not > > > just be converted to LGPLv3. > > > > > > I'd like to move to the current HEAD of that tree to pick up LGPLv2.1+ > > > licensing (which we can then convert to LGPLv3), but I need to figure > > > out the provenance of the two above additions before I do so (and then > > > maybe resolve some other issues due to other changes in the AVL code). > > > > > > Thanks > > > > > > Frank > > > > > > > > > -- > > > ___ > > > Nfs-ganesha-devel mailing list > > > Nfs-ganesha-devel@lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > > > > > -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] FSAL_VFS and ESXi
Hi Frank, Your reasoning in the change seems plausible. I would say remove the special case code. I don't have an ESXi vm set up to test with, but I can try to verify in the next week or so. Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "Frank Filz" > To: "Matt Benjamin" , > nfs-ganesha-devel@lists.sourceforge.net > Sent: Friday, August 14, 2015 1:40:38 PM > Subject: FSAL_VFS and ESXi > > Matt, > > You originally put in code to retry truncate for ESXi. Do you think that > code is still necessary? > > See the changes in this patch: > > https://github.com/ffilz/nfs-ganesha/commit/82c3323a7027c6912f5e8ab75e3a05cf > 56a3fbfb > > Thanks > > Frank > > -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] nfs4_op_write_same_Free doesn't exist
Huh, odd. I might still be on the prior dev? Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "William Allen Simpson" > To: "NFS Ganesha Developers" > Sent: Wednesday, August 19, 2015 4:28:59 AM > Subject: [Nfs-ganesha-devel] nfs4_op_write_same_Free doesn't exist > > How have folks been testing these past 2 weeks? > > dev 14 doesn't load or run on my machine. > > commit 726482a7159ef3d9443089cda7d0942643925a0c > Author: Frank S. Filz > Date: Fri Jul 24 13:46:31 2015 -0400 > > Eliminate separate cache_inode_rdwr_plus > ... > > > - .funct = nfs4_op_write_plus, > - .free_res = nfs4_op_write_Free, > + .funct = nfs4_op_write_same, > + .free_res = nfs4_op_write_same_Free, > > > src/include/nfs_proto_functions.h had nfs4_op_write_plus_Free(), > which doesn't exist either. > > /* NFSv4.2 */ > -int nfs4_op_write_plus(struct nfs_argop4 *, compound_data_t *, > +int nfs4_op_write_same(struct nfs_argop4 *, compound_data_t *, >struct nfs_resop4 *); > > -void nfs4_op_write_plus_Free(nfs_resop4 *resp); > +void nfs4_op_write_same_Free(nfs_resop4 *resp); > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Ganesha2.3 failure on ppc arch identified
Hi Malahal, I don't see a problem with these offhand. I can coordinate that happening now on ntirpc duplex-12, but consider this decoupled from Frank's ability to ensure anything is in an official dev for this week. regards, Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "Malahal Naineni" > To: "Matt Benjamin" , > Nfs-ganesha-devel@lists.sourceforge.net > Sent: Friday, August 21, 2015 10:20:54 AM > Subject: Re: [Nfs-ganesha-devel] Ganesha2.3 failure on ppc arch identified > > Matt, the following commits were tested and they work fine on powerpc64. > Without the patches, ganesha on powerpc64 just hangs. Can you please take > the libntirpc patch into upstream. > > I will co-ordinate with Frank on pushing the corresponding ganesha > commit once the libntirpc patches go in. > > Regards, Malahal. > PS: they are based on a bit old code, but rebase should work just fine. > Let me know if you need me to rebase. Thank you. > > Malahal Naineni [mala...@us.ibm.com] wrote: > > Matt, here is a fix for the stack size as well as propagating the error > > back to ganesha. Top 2 commits: > > > > https://github.com/malahal/ntirpc/commits/adp23 > > > > Corresponding ganesha patch: > > > > https://github.com/malahal/nfs-ganesha/commits/adp23 > > > > Regards, Malahal. > > -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Couple of ntirpc compilation warning/errors on clang
thanks, ok -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "Dominique Martinet" > To: "Matt Benjamin" , "nfs-ganesha-devel" > > Sent: Thursday, August 27, 2015 12:32:43 PM > Subject: Couple of ntirpc compilation warning/errors on clang > > I've got these when messing around (clang has some more warnings than > gcc, some being useful): > > nfs-ganesha/src/libntirpc/src/clnt_vc.c:81:17: warning: > field 'cmsg' with variable sized type 'struct cmsghdr' not at the end of > a > struct or class is a GNU extension > [-Wgnu-variable-sized-type-not-at-end] > struct cmsghdr cmsg; > > -> this look like it's not used at all, can we just remove the struct > cmessage definition? > > nfs-ganesha/src/libntirpc/src/authgss_hash.c:274:11: > warning: taking the absolute value of unsigned type 'unsigned int' has > no effect > [-Wabsolute-value] > ((abs(axp->gen - gd->gen) > > > -> probably needs some macro to do the compare then difference > > > I'll push a few changes for next week (rc2) for the main ganesha code > (got a few more there), but it's a bit more cumbersome to push to ntirpc > so can I leave these to you? > > Thanks, > -- > Dominique > -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Announce Push of V2.3-rc1
Good point. I'll try to push a change for next week. Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "Kaleb KEITHLEY" > To: nfs-ganesha-devel@lists.sourceforge.net > Sent: Friday, August 28, 2015 1:01:06 PM > Subject: Re: [Nfs-ganesha-devel] Announce Push of V2.3-rc1 > > On 08/27/2015 07:47 PM, Frank Filz wrote: > > Branch next > > > > Tag:V2.3-rc1 > > > > NOTE: This tag includes a libntirpc update, please make sure your > > submodule is updated before pushing additional submissions. > > > > Highlights > > > > ... > > > > * libntirpc update > > > > We, i.e. me, for Fedora packaging, need this to be tagged with a version. > > What's the expected landing date for symbol versions? > > Thanks, > > -- > > Kaleb > > > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] RFE - disable [R|S][UN]LOCK
Hi, I have deja vu again (i.e., we've gone over these interfaces at least 3 times in the last couple of months). What is required to be serialized was (for TCP) operations which read or write the recv, send channel, respectively. There was a bit more legacy complexity carried forward from older tirpc, where there had been a lot of serialization (out-of-order calls were not even possible). UDP send/recv interfaces claim atomicity, hence the bypass for UDP. RDMA is new. The change to encode TCP entirely out-of-band avoids even the need to serialize around *_reply() and *_error(), but that's been true since 2.2. Likewise, since 2.0, all decoding for a given transport is in a single thread context at any given time, so it doesn't in general need to lock the recv side--but, for the same reason, there wasn't any contention. I'm actually pretty sure that "entire bottom half" is, in fact, not serialized per transport, and don't think anyone thinks they have observed that. (Much as it would be nice to be able to just switch off the "suck flag" and gad-zongle performance.) I think we've gone over that a couple of times here at the office... Which is not so say that I don't mind improving or removing these. The macros exist to excempt UDP, and as we've discussed, I assumed you would avoid serializing RDMA unnecessarily (in the macros, or just by making the lock ops no-ops). The place to be somewhat careful is around authenticate operations. Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "William Allen Simpson" > To: "NFS Ganesha Developers" > Sent: Wednesday, September 2, 2015 10:10:51 AM > Subject: [Nfs-ganesha-devel] RFE - disable [R|S][UN]LOCK > > https://review.gerrithub.io/245072 RFE - disable [R|S][UN]LOCK > > This is a test for eliminating these 2 locks. They are the last > remaining uses of SVC_[UN]LOCK(). I've been testing this since > Aug 3 without issue. (I've now updated the patch date.) > > Their unfortunate side effect is that only 1 request per > transport can be running in each top/bottom half at a time. > This may be a reason we plateau during a lot of traffic. > > I'm hoping the nearby *kin timing will give a good indication of > the speedup. > > The only reason for these locks is to prevent a conflict with > config. There has to be a better way, especially since these > loops shouldn't be running until config has completed > > Perhaps these were intended for dbus configs? I'd asked Matt, > and he didn't remember the exact problem they are addressing. > > One way would be to make atomic accesses for the very few config > variables used here. > > Another would be to use read/write locks instead. Many readers, > only 1 very rare writer. And limit them to a short section of > code, instead of the entire dispatch loop! > > -- > Monitor Your Dynamic Infrastructure at Any Scale With Datadog! > Get real-time metrics from all of your servers, apps and tools > in one place. > SourceForge users - Click here to start your Free Trial of Datadog now! > http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140 > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- Monitor Your Dynamic Infrastructure at Any Scale With Datadog! Get real-time metrics from all of your servers, apps and tools in one place. SourceForge users - Click here to start your Free Trial of Datadog now! http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140 ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Fridge worker pool to ntirpc?
Hi Bill, There has not been griping. Malahal has done performance measurement. IIUC (per IRC) Malahal: 1. has empirical evidence that moving the current Ganesha -dispatch queue- bands into lanes measurably improves throughput, when the number of worker threads is large (apparently, 64 and (!) 256) That is expected, so I'm looking to Malahal to send a change (and some other tuning changes) for review. 2. Malahal indicated he found a bug of some kind in the thread fridge, provoked by the shutdown check in the current dispatch queue code, which he says he fixed, so if he hasn't already sent a change, I'm expecting to see one soon which addresses this. 3. Malahal described in IRC an additional change he made to split the ntirpc output ioq into lanes, and believed he saw improvement (as of ~2 weeks ago), but was still benchmarking in order to split out the impact of this change relative to others. I'm surprised this is helpful (but, it would empirically disprove bottom-half serialization by xprt, if true--oh, the measured improvement from reduced xprt->lock contention also seems to disprove it). 4. As I've indicated in discussion (2-3 occasions) with you here, it's been in our medium-term plan to move all of the current dispatch operations back into a new ntirpc run-loop, replacing legacy svc_run(). I consider that completely orthogonal to Ganesha having it's own thread pool. As well, it's very likely that the full change we should be looking at would be using platform async i/o for (at least) all recv and likely send paths. That will require work on decoders and is big enough that I don't see it coming in 2.3, unless RDMA starts working great first. Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "William Allen Simpson" > To: "NFS Ganesha Developers" > Sent: Wednesday, September 2, 2015 10:35:38 AM > Subject: [Nfs-ganesha-devel] Fridge worker pool to ntirpc? > > There has been some griping (that hasn't yet appeared on > this mailing list) that the ntirpc worker pool is too slow > processing TCP output. But no specific proof. > > There are several possible reasons for that, including that > the work model there would be better per interface, rather > than per connection. I've mentioned fair queuing before. > > I've made a lock-free variant for RDMA, but for TCP that > requires combining the fridge thread pool with the ntirpc > pool. I've had code for that sitting around since May, > although it's rather bit-rotten in the meantime. > > I've been assuming you don't want it until the new RDMA? > > -- > Monitor Your Dynamic Infrastructure at Any Scale With Datadog! > Get real-time metrics from all of your servers, apps and tools > in one place. > SourceForge users - Click here to start your Free Trial of Datadog now! > http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140 > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- Monitor Your Dynamic Infrastructure at Any Scale With Datadog! Get real-time metrics from all of your servers, apps and tools in one place. SourceForge users - Click here to start your Free Trial of Datadog now! http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140 ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
[Nfs-ganesha-devel] ntirpc: two patches on duplex-12
Hi, I've pushed 2 changes to the head of ntirpc duplex-12: 1. malahal's fix for Coverity CID 130080 2. a candidate change to implement shared-object versioning Kaleb, in particular, could you look over the versioning change and see how it fits with your other packaging work? Thanks! Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- Monitor Your Dynamic Infrastructure at Any Scale With Datadog! Get real-time metrics from all of your servers, apps and tools in one place. SourceForge users - Click here to start your Free Trial of Datadog now! http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140 ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] libntirpc duplex-12
I.e., this must be on linuxbox2 ntirpc.git? Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "William Allen Simpson" > To: "Matt Benjamin" > Cc: "NFS Ganesha Developers" > Sent: Thursday, September 3, 2015 8:31:57 AM > Subject: libntirpc duplex-12 > > As requested, I've updated duplex-12 with the current state of RDMA. > This means duplex-12 has the history of copying the old rdma files, > integrating Mooshika library, and then removing the old rdma files > (Feb 9 to today). > > I believe this is now ready to make into a versioned library. > > Note that the corresponding Ganesha changes have not been made yet. > But this libntirpc compiles with or without RDMA. > -- Monitor Your Dynamic Infrastructure at Any Scale With Datadog! Get real-time metrics from all of your servers, apps and tools in one place. SourceForge users - Click here to start your Free Trial of Datadog now! http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140 ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Fridge worker pool to ntirpc?
inline -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > > >> 3. Malahal described in IRC an additional change he made to split the > >> ntirpc output ioq into > >> lanes, and believed he saw improvement (as of ~2 weeks ago), but was still > >> benchmarking in order to > >> split out the impact of this change relative to others. > > > > I did this as well, but the gains seemed marginal at best! > > > As I'd suspected. My patch there will eventually eliminate the > lock entirely. > Who knows, bring on the measurement (later). Matt -- Monitor Your Dynamic Infrastructure at Any Scale With Datadog! Get real-time metrics from all of your servers, apps and tools in one place. SourceForge users - Click here to start your Free Trial of Datadog now! http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140 ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Lost wakeup with nfs_rpc_dequeue_req() and nfs_rpc_enqueue_req()
(ie, you're welcome to work with our code here) Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "Matt W. Benjamin" > To: "Malahal Naineni" > Cc: nfs-ganesha-devel@lists.sourceforge.net > Sent: Wednesday, September 9, 2015 9:40:11 AM > Subject: Re: [Nfs-ganesha-devel] Lost wakeup with nfs_rpc_dequeue_req() and > nfs_rpc_enqueue_req() > > I'm not sure I believe that there is hang case, but if there is, the > simple fix for it is to bound the offending wait. > > I have a branch that uses a lock-free bounded mpmc queue based work we > did last year in the Ceph codebase. I have something working but not > efficiently, and I haven't have time to explore the issues/tuning. > > Any dramatic change here should probably not be for 2.3. > > Matt > > - "Malahal Naineni" wrote: > > > William Allen Simpson [william.allen.simp...@gmail.com] wrote: > > > On 9/8/15 9:03 PM, Malahal Naineni wrote: > > > > The nfs_rpc_enqueue_req is called by the producer and > > > > nfs_rpc_dequeue_req is called by the consumer. Here is the high > > level > > > > view of those functions. > > > > > > > > In particular, if a consumer finds no request in the queue, and > > then > > > > attempts to go to sleep, but before he adds himself to the wait > > list > > > > (before step 2b), if producer adds another request at that > > instant, the > > > > producer would find the wait list empty and couldn't wake up any > > > > consumer. > > > > > > > I've not paid much attention to this, as my goal is to eliminate > > it, > > > but isn't there a lock involved? That would make all the > > difference. > > > > Locks are there but they fail to ensure the race! How are you going > > to > > eliminate it? Wondering if having a lane for each worker thread is a > > good idea here. This would make it SPSC (single producer single > > consumer) which should be easy for a lockless code. > > > > Regards, Malahal. > > > > > > -- > > Monitor Your Dynamic Infrastructure at Any Scale With Datadog! > > Get real-time metrics from all of your servers, apps and tools > > in one place. > > SourceForge users - Click here to start your Free Trial of Datadog > > now! > > http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140 > > ___ > > Nfs-ganesha-devel mailing list > > Nfs-ganesha-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > > -- > Matt Benjamin > CohortFS, LLC. > 315 West Huron Street, Suite 140A > Ann Arbor, Michigan 48103 > > http://cohortfs.com > > tel. 734-761-4689 > fax. 734-769-8938 > cel. 734-216-5309 > > -- > Monitor Your Dynamic Infrastructure at Any Scale With Datadog! > Get real-time metrics from all of your servers, apps and tools > in one place. > SourceForge users - Click here to start your Free Trial of Datadog now! > http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140 > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- Monitor Your Dynamic Infrastructure at Any Scale With Datadog! Get real-time metrics from all of your servers, apps and tools in one place. SourceForge users - Click here to start your Free Trial of Datadog now! http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140 ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] rc2 staged
My first "attempt" at versioning is still the one we're using, but Dan: a) put the ntirpc version on the output file (which I forgot to do) b) put the libntirpc.so in a better location c) moved Ganesha CMakeLists.txt into a subdir so the standalone build could have a CMakeLists.txt The install doesn't need to accommodate anything. These changes to the library packaging change are purely iterations on one change--if you want to be isolated from this kind of thing, just work on a stable dev! Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "William Allen Simpson" > To: "Daniel Gryniewicz" > Cc: "nfs-ganesha-devel" > Sent: Friday, September 11, 2015 1:42:53 PM > Subject: Re: [Nfs-ganesha-devel] rc2 staged > > On 9/11/15 1:34 PM, William Allen Simpson wrote: > > The date on the latter libntirpc.so seems slightly off. Is this an > > old place for storing it? Or confusing the loader? > > > Yes, my next older log shows: > > -- Installing: /home/bill/rdma/install/var/run/ganesha > -- Installing: /home/bill/rdma/install/lib64/ganesha/libntirpc.so > -- Installing: /home/bill/rdma/install/lib64/ganesha/libfsalnull.so.4.2.0 > -- Up-to-date: /home/bill/rdma/install/lib64/ganesha/libfsalnull.so.4 > -- Up-to-date: /home/bill/rdma/install/lib64/ganesha/libfsalnull.so > > So, we either need to get the install to: > * not put it there, or > * remove old ones, or > * always put it there. > > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] rc2 staged
I think this patch is as ready as it can be. What Dan sent to gerrit will behave correctly for anyone not updating ntirpc branches by hand. Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "Frank Filz" > To: "nfs-ganesha-devel" > Sent: Friday, September 11, 2015 1:55:40 PM > Subject: Re: [Nfs-ganesha-devel] rc2 staged > > > I have +2 code reviewed all the patches I am considering staged for rc2. I > will > > start the merge Friday morning. > > > > I figured out the problem with cthon04, turns out we actually had a bug in > > RELEASE_LOCKOWNER that has been there for quite some time... > > > > I was also wrong about lock owners, it turns out I was misreading the > > tcpdump trace. It's not easy to follow cthon04 lock tests (which > unfortunately > > really don't test all that much). > > > > Dan - no one has reviewed your patch yet, so I did not stage that. If it > gets > > positive review by morning I will add it to the staging. If anyone wants > to > > review: > > > > https://review.gerrithub.io/246297 > > I'm not clear that this patch is quite ready to go, so I will proceed on the > merge with the patches I have +2 code reviewed already. > > More fun next week... > > Thanks > > Frank > > > --- > This email has been checked for viruses by Avast antivirus software. > https://www.avast.com/antivirus > > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] rc2 staged
Hi Bill, If it installed libntirpc.so.1.3.0 (*) in 2 places, it's broken. If it installed FSALs in lib64/ganesha and ntirpc in lib64, then it did what Dan intended, iirc. In order to be -looking for them- there, the end user will need to update LD_LIBRARY_PATH and/or edit /etc/ld.so.conf (Linux), if your Ganesha build isn't putting them in a path already being searched. Or, so I believe. Users of a packaged Ganesha and libntirpc.so on (e.g.) Fedora will presumably not need to do any such thing. Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "William Allen Simpson" > To: "Matt Benjamin" , "Frank Filz" > > Cc: "nfs-ganesha-devel" > Sent: Friday, September 11, 2015 2:23:00 PM > Subject: Re: [Nfs-ganesha-devel] rc2 staged > > On 9/11/15 2:01 PM, Matt Benjamin wrote: > > I think this patch is as ready as it can be. What Dan sent to gerrit will > > behave correctly for anyone > > not updating ntirpc branches by hand. > > > Sorry, Matt, but this was a fresh install with no updating ntirpc. > > I compiled and installed before Dan's patch, followed by > adding Dan's patch -- rm -rf * my build directory, and then > new cmake && make install > > I've tried to remove the lonely: > > -rw-r--r-- 1 bill bill 1005416 Sep 10 22:33 libntirpc.so > bill@simpson:~/rdma/install/lib64/ganesha$ rm libntirpc.so > bill@simpson:~/rdma/install/lib64/ganesha$ cd ../.. > bill@simpson:~/rdma/install$ sudo bash > [sudo] password for bill: > root@simpson:~/rdma/install# ~/setup.sh > :/home/bill/rdma/install/lib:/home/bill/rdma/install/lib64 > root@simpson:~/rdma/install# ./bin/ganesha.nfsd > ./bin/ganesha.nfsd: error while loading shared libraries: libntirpc.so.1.3: > cannot open shared object file: No such file or directory > root@simpson:~/rdma/install# > > It's installing libntirpc in lib64, and one lonely part in > lib64/ganesha. Where is it looking for them? > > I really want this in today, too, and stayed up last night to > try to get it going. > > -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] rc2 staged
There is probably no need to pullup ntirpc twice. Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "William Allen Simpson" > To: "Daniel Gryniewicz" , "nfs-ganesha-devel" > , "Frank > Filz" > Sent: Friday, September 11, 2015 3:39:28 PM > Subject: Re: [Nfs-ganesha-devel] rc2 staged > > On 9/11/15 3:07 PM, William Allen Simpson wrote: > > I've redone the entire sequence for the umpteenth time and sent the > > entire log to Dan. Maybe he can see the issue with every bit of > > data I've got, unexpurgated. > > > All praise to Dan! It was in the script that I've been using to > setup and run ganesha for the past 9 months or so > > So I'll +1 Dan's patch. > > And Frank, don't forget my https://review.gerrithub.io/246391, > which should actually go before Dan's patch (based on its > libntirpc commit pointer). > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] thread cleanup hook for FSAL
Hi Dirk, That sounds well motivated. You might start discussion on IRC about interfaces. Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "Dirk Jagdmann" > To: "NFS Ganesha Developers (Nfs-ganesha-devel@lists.sourceforge.net)" > > Sent: Monday, September 21, 2015 4:00:58 PM > Subject: [Nfs-ganesha-devel] thread cleanup hook for FSAL > > hello developers, > > I'm writing my FSAL which uses IPC to communicate with another process which > is > providing the storage. This works well for me. I am currently using one IPC > "channel" for each NFS-Ganesha worker thread. With a thread local variable > these > channels are created on demand. However I currently can not close these > channels, as my FSAL code doesn't know when a worker thread is stopped. > > Can I add a hook mechanism to the NFS-Ganesha thread maintaining code? This > would allow a FSAL (or other code) to register a callback function that will > be > called, just prior to the worker thread exit. Which functions would be best > suited to call those hooks? Or does it already exist, and I didn't find it? > > -- > ---> Dirk Jagdmann > > http://cubic.org/~doj > -> http://llg.cubic.org > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] thread cleanup hook for FSAL
(as to what already exists, please consult the "thread fridge" code) Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "Matt Benjamin" > To: "Dirk Jagdmann" > Cc: "NFS Ganesha Developers (Nfs-ganesha-devel@lists.sourceforge.net)" > > Sent: Monday, September 21, 2015 4:03:11 PM > Subject: Re: [Nfs-ganesha-devel] thread cleanup hook for FSAL > > Hi Dirk, > > That sounds well motivated. You might start discussion on IRC about > interfaces. > > Matt > > -- > Matt Benjamin > Red Hat, Inc. > 315 West Huron Street, Suite 140A > Ann Arbor, Michigan 48103 > > http://www.redhat.com/en/technologies/storage > > tel. 734-761-4689 > fax. 734-769-8938 > cel. 734-216-5309 > > - Original Message - > > From: "Dirk Jagdmann" > > To: "NFS Ganesha Developers (Nfs-ganesha-devel@lists.sourceforge.net)" > > > > Sent: Monday, September 21, 2015 4:00:58 PM > > Subject: [Nfs-ganesha-devel] thread cleanup hook for FSAL > > > > hello developers, > > > > I'm writing my FSAL which uses IPC to communicate with another process > > which > > is > > providing the storage. This works well for me. I am currently using one IPC > > "channel" for each NFS-Ganesha worker thread. With a thread local variable > > these > > channels are created on demand. However I currently can not close these > > channels, as my FSAL code doesn't know when a worker thread is stopped. > > > > Can I add a hook mechanism to the NFS-Ganesha thread maintaining code? This > > would allow a FSAL (or other code) to register a callback function that > > will > > be > > called, just prior to the worker thread exit. Which functions would be best > > suited to call those hooks? Or does it already exist, and I didn't find it? > > > > -- > > ---> Dirk Jagdmann > > > http://cubic.org/~doj > > -> http://llg.cubic.org > > > > -- > > ___ > > Nfs-ganesha-devel mailing list > > Nfs-ganesha-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > > > -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] thread cleanup hook for FSAL
Hi Dirk, The design sounds fine to me. Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "Dirk Jagdmann" > To: "NFS Ganesha Developers" > Sent: Tuesday, September 22, 2015 7:13:09 PM > Subject: Re: [Nfs-ganesha-devel] thread cleanup hook for FSAL > > So I've implemented a minimal solution that works for my use case. First this > is > the new public API, which can be used to register a thread cleanup callback > function: > > /** > * type of a callback function that will be called when a thread exits. > * Use the fridgethr_register_thread_cleanup_cb() function to register > * the callback function *for each thread* you want it called at > * thread exit. > */ > typedef void (*thread_cleanup_cb) (void*); > > /** > * register a callback function that will be called when the current thread > exits. > * > * You can only register a callback function once per thread for each > * unique param value. Subsequent calls of this function with the > * same value for cb and param will not register additional callbacks. > * > * @param cb pointer to callback function. > * @param param parameter that will be passed to cb. > */ > void fridgethr_register_thread_cleanup_cb(thread_cleanup_cb cb, void *param); > > > This feature is implemented with a linked list with the nodes containing the > thread_cleanup_cb pointer and the param pointer. Each thread maintains its > own > linked list with a thread local glist_head. A new function: > > /** > * call every registered thread cleanup callback function. > */ > static void fridgethr_call_cleanup_cb(); > > will traverse the linked list and call every registered cleanup function. > This > new fridgethr_call_cleanup_cb function is called in fridgethr_start_routine() > before the thread_finalize function is called. > > I'll do some more testing during this week, to see that my implementation is > working. If you like the general idea, I'll post a review request later with > my > changes. > > -- > ---> Dirk Jagdmann > > http://cubic.org/~doj > -> http://llg.cubic.org > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- Monitor Your Dynamic Infrastructure at Any Scale With Datadog! Get real-time metrics from all of your servers, apps and tools in one place. SourceForge users - Click here to start your Free Trial of Datadog now! http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140 ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Export_id larger than 16 bits
That would be one option, but it should also be possible to use the full range of FSALs over NFSv4.x. (I don't know if that works today, but.) Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "Gary Ma" > To: "Frank Filz" > Cc: "NFS Ganesha Developers" > Sent: Wednesday, September 23, 2015 9:21:15 AM > Subject: Re: [Nfs-ganesha-devel] Export_id larger than 16 bits > > What are the options if we need 32-bit export id? Not enabling the FSALs that > have larger handle? > > Thanks, > Gary > > On Tue, Sep 22, 2015 at 9:50 PM, Frank Filz < ffilz...@mindspring.com > > wrote: > > > The problem with expanding export id is making everything fit in a 64 byte > nfs v3 handle. We have several FSALs that already wind up with a 62 byte > handle. > > Frank > > Sent from my iPhone > > On Sep 22, 2015, at 5:01 PM, Dirk Jagdmann < d...@cubic.org > wrote: > > > Currently gsh_export::export_id is a 16 bit unsigned. This also corresponds > > to a > > 16 unsigned defined for the EXPORT configuration. Is there any reason why > > this > > Export ID is limited to 16 bit? Could this be increased to 64 bits? > > > > -- > > ---> Dirk Jagdmann > > > http://cubic.org/~doj > > -> http://llg.cubic.org > > > > -- > > Monitor Your Dynamic Infrastructure at Any Scale With Datadog! > > Get real-time metrics from all of your servers, apps and tools > > in one place. > > SourceForge users - Click here to start your Free Trial of Datadog now! > > http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140 > > ___ > > Nfs-ganesha-devel mailing list > > Nfs-ganesha-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > > -- > Monitor Your Dynamic Infrastructure at Any Scale With Datadog! > Get real-time metrics from all of your servers, apps and tools > in one place. > SourceForge users - Click here to start your Free Trial of Datadog now! > http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140 > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > > > -- > Monitor Your Dynamic Infrastructure at Any Scale With Datadog! > Get real-time metrics from all of your servers, apps and tools > in one place. > SourceForge users - Click here to start your Free Trial of Datadog now! > http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140 > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- Monitor Your Dynamic Infrastructure at Any Scale With Datadog! Get real-time metrics from all of your servers, apps and tools in one place. SourceForge users - Click here to start your Free Trial of Datadog now! http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140 ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] [PATCH] ntirpc: remove redundant assignment of ->warnx
Hi Wei, If it's already initialized, it doesn't need to be initialized (by default) again here, no. But let's not create the precedent that we can call tirpc_control before calling svc_init(). Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "Wei Fang" > To: m...@cohortfs.com > Cc: nfs-ganesha-devel@lists.sourceforge.net > Sent: Thursday, September 24, 2015 3:52:16 AM > Subject: Re: [Nfs-ganesha-devel] [PATCH] ntirpc: remove redundant assignment > of ->warnx > > Hi, Matt, > > On 2015/9/22 21:32, Matt W. Benjamin wrote: > > Hi, > > > > The svc_init() call is intended to precede any call to tirpc_control, and > > we do want this call to default-initialize. > > ->warnx has been initialized when __pkg_params was declared, is it necessary > to initialise it again here? > > thanks, > Wei > > > Regards, > > > > Matt > > > > - fangw...@huawei.com wrote: > > > >> From: Wei Fang > >> > >> ->warnx has been assigned defaultly. If we assigned here when > >> SVC_INIT_WARNX unsetted, ->warnx assigned by tirpc_control would be > >> overwrited to default, which we don't want to happen. > >> > >> Signed-off-by: Wei Fang > >> --- > >> src/svc.c |2 -- > >> 1 files changed, 0 insertions(+), 2 deletions(-) > >> > >> diff --git a/src/svc.c b/src/svc.c > >> index 1ff1c84..c91386c 100644 > >> --- a/src/svc.c > >> +++ b/src/svc.c > >> @@ -147,8 +147,6 @@ svc_init(svc_init_params *params) > >> > >>if (params->flags & SVC_INIT_WARNX) > >>__ntirpc_pkg_params.warnx = params->warnx; > >> - else > >> - __ntirpc_pkg_params.warnx = warnx; > >> > >>/* svc_vc */ > >>__svc_params->xprt_u.vc.nconns = 0; > >> -- > >> 1.7.1 > > > > > -- > Monitor Your Dynamic Infrastructure at Any Scale With Datadog! > Get real-time metrics from all of your servers, apps and tools > in one place. > SourceForge users - Click here to start your Free Trial of Datadog now! > http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140 > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- Monitor Your Dynamic Infrastructure at Any Scale With Datadog! Get real-time metrics from all of your servers, apps and tools in one place. SourceForge users - Click here to start your Free Trial of Datadog now! http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140 ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] passing extra data via ganesha.conf
Hi Nilesh,
1. the most straightforward thing you can do is add a YOURFSAL block to the
config file--at the top-level most commonly (see GPFS, Lustre, etc), but you
could probably extend other blocks with some discussion on list and/or irc
2. you know the export it used, and the protocol-defined details of the client,
that's it
Matt
--
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
- Original Message -
> From: "Nilesh Chate"
> To: nfs-ganesha-devel@lists.sourceforge.net
> Sent: Thursday, October 1, 2015 3:56:52 AM
> Subject: [Nfs-ganesha-devel] passing extra data via ganesha.conf
>
> Hi,
> I am using the following ganesha.conf
>
> EXPORT
> {
> Export_Id = 77;
> Path = /home;
> Pseudo = /home;
> Access_Type = RW;
> FSAL {
> Name = VFS;
> }
> }
>
> Q1. I want to also pass some extra details via this config file. How do i do
> that?. Ex. I want to pass a string(some extra mounpoints) via this config
> file?.
>
> Q2. Can i know the path where the client have mounted our exported path?. If
> yes how?
>
>
> Regards,
> Nilesh
>
> --
>
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>
--
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] WARNING to all ntirpc developers
Bill, Well, something broke. The defined model for fetching the ntirpc src did not depend on the default branch of the upstream repository, nor on its top commit, we made the commit explicit. I tend to think that's preferable to freezing a branch. There hasn't been official use of linuxbox2/ntirpc.git by nfs-ganesha checkouts for, iirc, 3 years. I agree it's duplex-12 branch should, if it is updated, only contain commits seen on upstream duplex-12. But please do not assume we are keeping it up to date--because no one expected to be relying on it. This is you inventing a rule, and complaining about me breaking it. Please tone down the rhetoric a bit. These are changes on a development branch. If you want to vent, use the provided IRC channel ;). Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "William Allen Simpson" > To: "Matt W. Benjamin" > Cc: nfs-ganesha-devel@lists.sourceforge.net > Sent: Sunday, October 4, 2015 7:37:52 AM > Subject: Re: [Nfs-ganesha-devel] WARNING to all ntirpc developers > > Dan's cmake completely overwrites src/libntirpc *IN PLACE* the > next time you make ganesha. > > If you have any changes pending, save/push your commits > somewhere before your next compile after updating. > > > On 10/3/15 8:32 AM, Matt W. Benjamin wrote: > > - "William Allen Simpson" wrote: > >> Frank, could you please change nfs-ganesha/ntirpc to default > >> branch duplex-12? Matt forgot, and I've reminded him twice. > > > > Ok, fine, I've done this. I don't think this was a blocker. > > > No submodule to checkout a specific commit, so defaulted to > duplex-11. duplex-11 has missing/moved SVCXPRT symbols, etc. > > Complete showstopper. Glad it's fixed now. > > > >> Dan, you forgot to push your dang/ntirpc external to both > >> linuxbox2/ntirpc > > > > Not assured by us to be kept at parity with the upstream repo, it's a > > 3rd-party repo, wrt nfs-ganesha. > > > > and nfs-ganesha/ntirpc duplex-12 > > > I've been treating it as the official ntirpc development site. > > Should it be abandoned, then? > > > > I merged Dan's PR Thursday AM eastern. What exactly is not merged? > > > commit f9ec4b8e37ac59f81b5feaac9f4151686acd61ac > Author: Daniel Gryniewicz > Date: Thu Oct 1 11:32:51 2015 -0400 > > Fix RDMA build > > Signed-off-by: Daniel Gryniewicz > > commit 93fc447716624bd6c92b3b11b4c6b574d9b2aaa7 > Merge: c4e264a 669383d > Author: Matt Benjamin > Date: Thu Oct 1 08:35:45 2015 -0400 > > Merge pull request #4 from dang/external > > Allow ntirpc to be used as a CMake external > > > > -- RDMA > >> cmake and compile fail. > > > > Um... Will revisit next week, I guess. > > > Goody. > > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] versioned libibverbs issue
Hi Bill, - Original Message - > > > > Sure looks to me like this is the mismatch (with librdmacm above)! > > Should lib64/libntirpc.so not be linked against libibverbs.so too? The > ldd command above does not show the dynamic linkage. Still the symbol is > used, but with an unknown version (likely because it was not explicitly > checked during linking). Yes, it should. This is the issue I suggested we check on 3 weeks ago, and again last week, when we looked over the ntirpc link line together, on your laptop. Matt > > > Does anybody out there know how to tell a library to use a > > specific library version? > > When building libntirpc, make sure that -libverbs is passed in the > LD_FLAGS. I expect that libntirpc.so will have a versioned dependency on > the symbol after that. > > I might be able to try this out later this week. You are also more > familiar with the libntirpc build process, and might be able to test > this much quicker than me. > > Niels > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] versioned libibverbs issue
Thanks, Neils. Thought I was responding to Bill. :) Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "Matt Benjamin" > To: "Niels de Vos" > Cc: "NFS Ganesha Developers" > Sent: Sunday, October 4, 2015 10:40:24 AM > Subject: Re: [Nfs-ganesha-devel] versioned libibverbs issue > > Hi Bill, > > - Original Message - > > > > > > > Sure looks to me like this is the mismatch (with librdmacm above)! > > > > Should lib64/libntirpc.so not be linked against libibverbs.so too? The > > ldd command above does not show the dynamic linkage. Still the symbol is > > used, but with an unknown version (likely because it was not explicitly > > checked during linking). > > Yes, it should. This is the issue I suggested we check on 3 weeks ago, and > again last week, when we looked over the ntirpc link line together, on your > laptop. > > Matt > > > > > > Does anybody out there know how to tell a library to use a > > > specific library version? > > > > When building libntirpc, make sure that -libverbs is passed in the > > LD_FLAGS. I expect that libntirpc.so will have a versioned dependency on > > the symbol after that. > > > > I might be able to try this out later this week. You are also more > > familiar with the libntirpc build process, and might be able to test > > this much quicker than me. > > > > Niels > > > > ------ > > ___ > > Nfs-ganesha-devel mailing list > > Nfs-ganesha-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > > > > -- > Matt Benjamin > Red Hat, Inc. > 315 West Huron Street, Suite 140A > Ann Arbor, Michigan 48103 > > http://www.redhat.com/en/technologies/storage > > tel. 734-761-4689 > fax. 734-769-8938 > cel. 734-216-5309 > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] versioned libibverbs issue
Your dislike of cross-posting isn't really on topic.
--
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
- Original Message -
> From: "William Allen Simpson"
> To: "Niels de Vos"
> Cc: "Matt Benjamin" , "NFS Ganesha Developers"
>
> Sent: Sunday, October 4, 2015 7:41:10 PM
> Subject: Re: [Nfs-ganesha-devel] versioned libibverbs issue
>
> On 10/4/15 10:40 AM, Matt Benjamin wrote:
> >>> Sure looks to me like this is the mismatch (with librdmacm above)!
> >>
> >> Should lib64/libntirpc.so not be linked against libibverbs.so too? The
> >> ldd command above does not show the dynamic linkage. Still the symbol is
> >> used, but with an unknown version (likely because it was not explicitly
> >> checked during linking).
> >
> > Yes, it should. This is the issue I suggested we check on 3 weeks ago, and
> > again last week, when we looked over the ntirpc link line together, on
> > your laptop.
> >
> Although I brought this to your attention 3 weeks ago,
> bringing my laptop to your office, I don't see the fix in
> your patch, nor Dan's later patches.
>
> The link line we looked at again last week was the Ganesha line.
> (with the VERBOSE=1 flag)
>
> Since neither I nor you nor Dan appear to know how to set a link
> version on a library in cmake, I'm hoping somebody on this list does!
I do know how to select from a desired range of library files matching the
pattern, because I google'd it. I don't see any documentation on selecting for
a version namespace at link time, but what I DO know is:
1. if the GNU toolchain allows that, it's not commonly done
2. as I think I and Dan (and most GNU developers know), there is a convention
in wide use to encode the library version in the filename, and using it the
packager would supply a libibverbs.so.1.0 and later a libibverbs.so.1.1.0 and
libibverbs.so would be a symlink to the latest supported
3. I reviewed what Mellanox Accelio does, because it uses these APIs
successfully, and, as I expected and Niels suggests, what you do to make use of
it in an intermediate shared library seems to be just dd -libverbs and -lrdmacm
to the library link line; I don't see them doing anything interesting to
select a library version, they have dynamic linkage, and this just works
>
>
> >>> Does anybody out there know how to tell a library to use a
> >>> specific library version?
> >>
> >> When building libntirpc, make sure that -libverbs is passed in the
> >> LD_FLAGS. I expect that libntirpc.so will have a versioned dependency on
> >> the symbol after that.
> >>
> >> I might be able to try this out later this week. You are also more
> >> familiar with the libntirpc build process, and might be able to test
> >> this much quicker than me.
> >>
> Thanks, Niels. How do we do that?
There are a lot of examples, including the CMakeLists.txt file in nfs-ganesha.
I think there is more than one way, but I believe that the Ganesha
CMakeLists.txt file in ntirpc should just add:
set(CMAKE_SHARED_LINKER_FLAGS, "${CMAKE_SHARED_LINKER_FLAGS} -libverbs
-lrdmacm")
I would have described this in prior weeks, if I realized it was a blocker.
Maybe it isn't, but let's try it shall we?
Matt
>
>
>
--
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] versioned libibverbs issue
sorry, top-posting; example: this also, forgot to top post: the top-level ganesha CMake -is- the relevant one; The way the process still works in this case is that Ganesha's cmake step finds the subordinate src/libntirpc CMakeList.txt just as it would any other, when Ganesha builds libntirpc.so. Dan's change a week ago cleaned up the standalone cmake build for libntirpc.so. IIRC, the linker flags editing should be done in src/libntirpc/src/CMakeLists.txt, so those should propagate to both internal and external builds. Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "William Allen Simpson" > To: "Niels de Vos" > Cc: "Matt Benjamin" , "NFS Ganesha Developers" > > Sent: Sunday, October 4, 2015 7:41:10 PM > Subject: Re: [Nfs-ganesha-devel] versioned libibverbs issue > > On 10/4/15 10:40 AM, Matt Benjamin wrote: > >>> Sure looks to me like this is the mismatch (with librdmacm above)! > >> > >> Should lib64/libntirpc.so not be linked against libibverbs.so too? The > >> ldd command above does not show the dynamic linkage. Still the symbol is > >> used, but with an unknown version (likely because it was not explicitly > >> checked during linking). > > > > Yes, it should. This is the issue I suggested we check on 3 weeks ago, and > > again last week, when we looked over the ntirpc link line together, on > > your laptop. > > > Although I brought this to your attention 3 weeks ago, > bringing my laptop to your office, I don't see the fix in > your patch, nor Dan's later patches. > > The link line we looked at again last week was the Ganesha line. > (with the VERBOSE=1 flag) > > Since neither I nor you nor Dan appear to know how to set a link > version on a library in cmake, I'm hoping somebody on this list does! > > > >>> Does anybody out there know how to tell a library to use a > >>> specific library version? > >> > >> When building libntirpc, make sure that -libverbs is passed in the > >> LD_FLAGS. I expect that libntirpc.so will have a versioned dependency on > >> the symbol after that. > >> > >> I might be able to try this out later this week. You are also more > >> familiar with the libntirpc build process, and might be able to test > >> this much quicker than me. > >> > Thanks, Niels. How do we do that? > > > -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Bad news... pynfs tests suddenly failing
I don't see how... We'll jump on it. Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "Frank Filz" > To: nfs-ganesha-devel@lists.sourceforge.net > Sent: Tuesday, October 6, 2015 7:46:07 PM > Subject: [Nfs-ganesha-devel] Bad news... pynfs tests suddenly failing > > I'm not sure what happened between tagging V2.3-rc5 and today, but I am now > getting the following pynfs failures: > > INIT st_setclientid.testValid : PASS > LOCK14 st_lock.testGrabLock1: FAILURE >Getting read lock when another owner has write lock >should return NFS4ERR_DENIED, instead got NFS4_OK > LOCK15 st_lock.testGrabLock2: FAILURE >Getting read lock when another owner has write lock >should return NFS4ERR_DENIED, instead got NFS4_OK > LOCK16 st_lock.testReadLocks1 : FAILURE >Getting write lock when another owner has read lock >should return NFS4ERR_DENIED, instead got NFS4_OK > LOCK17 st_lock.testReadLocks2 : FAILURE >Getting write lock when another owner has read lock >should return NFS4ERR_DENIED, instead got NFS4_OK > LOCK20 st_lock.testBlockTimeout : FAILURE >Conflicting lock on LOCK20 should return >NFS4ERR_DENIED, instead got NFS4_OK > MKFILE st_open.testOpen : PASS > RPLY6st_replay.testLockDenied : FAILURE >Call to be replayed should return NFS4ERR_DENIED, >instead got NFS4_OK > ** > Command line asked for 8 of 655 tests > Of those: 0 Skipped, 6 Failed, 0 Warned, 2 Passed > > Doing a bisect, they start failing with this patch: > > commit 80b12852391a8b9b7ec0abab9d3110d0e2ebf181 > Author: Daniel Gryniewicz > Date: Wed Sep 30 10:28:15 2015 -0400 > > Move ntirpc to a CMake external > > Has something somehow changed with the version of libntirpc picked up? > > At this point, this will be priority to fix before taking any other patches > this week, this is a major regression. > > Thanks > > Frank > > > --- > This email has been checked for viruses by Avast antivirus software. > https://www.avast.com/antivirus > > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- Full-scale, agent-less Infrastructure Monitoring from a single dashboard Integrate with 40+ ManageEngine ITSM Solutions for complete visibility Physical-Virtual-Cloud Infrastructure monitoring from one console Real user monitoring with APM Insights and performance trend reports Learn More http://pubads.g.doubleclick.net/gampad/clk?id=247754911&iu=/4140 ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] V2.3-dev-10 broke kerberos mounts, regression!
Hi Malahal, Sure, we'll track it down. The most likely situation is use of segmented i/O in any case where, currently, due to our use of GSSAPI's gss_wrap and gss_getmic APIs, we can't tolerate it. Until we can support newer APIs (such as the one introduced in MIT kerberos 12) with vector support, we need to substitute contiguous output buffers whenever RPCSEC_GSS is in use. Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "Malahal Naineni" > To: nfs-ganesha-devel@lists.sourceforge.net > Cc: "william allen simpson" , > mbenja...@redhat.com > Sent: Wednesday, October 7, 2015 8:50:17 PM > Subject: V2.3-dev-10 broke kerberos mounts, regression! > > Hi All, > > The latest ganesha2.3 doesn't work with kerberos mounts. It > appears to work with krb5 mounts but definitely fails with krb5i or > krb5p. I primarily tested with krb5p mounts, and git bisect pointed to > ganesha commit: 2de276416a5d2034fcd765434ed51794d622e916 > > That is just a submodule commit, so bisecting libntirpc pointed to this > commit: 378832326fea58945c88963cef7ef3fe8500fd47 > > I don't see much there but some new flags were introduced in that commit > related to gss. > > Appreciate if Matt or Bill can help figure this out. I am not familiar > with this area at all. The failure is very easy to recreate, the mount > itself fails. tcpdump shows we are sending something but the client > fails to decode the response. This results in mount() syscall failing > with EIO at the client. > > Regards, Malahal. > > -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] [PATCH] Fix the lack of mutex initialization
Hi,
Ouch. I'll verify this and if necessary apply this for next week.
Thanks!
Matt
--
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
- Original Message -
> From: "Yijing Wang"
> To: m...@linuxbox.com
> Cc: nfs-ganesha-devel@lists.sourceforge.net
> Sent: Thursday, October 8, 2015 9:41:30 PM
> Subject: [Nfs-ganesha-devel] [PATCH] Fix the lack of mutex initialization
>
> The x_vc_data->shared.ioq.qmutex doesn't be
> initialized before using it, fix it in
> alloc_x_vc_data().
>
> Signed-off-by: Yijing Wang
> ---
> src/clnt_internal.h |3 ++-
> 1 files changed, 2 insertions(+), 1 deletions(-)
>
> diff --git a/src/clnt_internal.h b/src/clnt_internal.h
> index f18650a..118aad7 100644
> --- a/src/clnt_internal.h
> +++ b/src/clnt_internal.h
> @@ -252,13 +252,14 @@ static inline struct x_vc_data *
> alloc_x_vc_data(void)
> {
> struct x_vc_data *xd = mem_zalloc(sizeof(struct x_vc_data));
> - TAILQ_INIT(&xd->shared.ioq.qh);
> + poolq_head_setup(&xd->shared.ioq);
> return (xd);
> }
>
> static inline void
> free_x_vc_data(struct x_vc_data *xd)
> {
> + poolq_head_destroy(&xd->shared.ioq);
> mem_free(xd, sizeof(struct x_vc_data));
> }
>
> --
> 1.7.1
>
>
> --
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>
--
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] V2.3-dev-10 broke kerberos mounts, regression!
Marcus, Could you retest from this baseline? Thanks, Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-707-0660 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "Malahal Naineni" > To: nfs-ganesha-devel@lists.sourceforge.net, "william allen simpson" > , > mbenja...@redhat.com > Sent: Thursday, October 15, 2015 1:26:55 PM > Subject: Re: [Nfs-ganesha-devel] V2.3-dev-10 broke kerberos mounts, > regression! > > Bill, I did test with your d12-pri branch. The kerberos mount does work > with it but every other mount fails! I don't think this is something to > do with your patch. I did make sure that all file creates and reads do > work. > > The mount failure (every other!) is: > > # mount -overs=4,sec=krb5p regcluster:/ibm/gpfs0/regfset2 /mnt1 > mount.nfs: Operation not permitted > > I will take tcpdump and update if I have any more details. > > Regards, Malahal. > > > Malahal Naineni [mala...@us.ibm.com] wrote: > > Hi All, > > > > The latest ganesha2.3 doesn't work with kerberos mounts. It > > appears to work with krb5 mounts but definitely fails with krb5i or > > krb5p. I primarily tested with krb5p mounts, and git bisect pointed to > > ganesha commit: 2de276416a5d2034fcd765434ed51794d622e916 > > > > That is just a submodule commit, so bisecting libntirpc pointed to this > > commit: 378832326fea58945c88963cef7ef3fe8500fd47 > > > > I don't see much there but some new flags were introduced in that commit > > related to gss. > > > > Appreciate if Matt or Bill can help figure this out. I am not familiar > > with this area at all. The failure is very easy to recreate, the mount > > itself fails. tcpdump shows we are sending something but the client > > fails to decode the response. This results in mount() syscall failing > > with EIO at the client. > > > > Regards, Malahal. > > > > > > -- > > ___ > > Nfs-ganesha-devel mailing list > > Nfs-ganesha-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > > > > -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] V2.3-dev-10 broke kerberos mounts, regression!
malahal is using a (up-to-date?) rhel 7.0. Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-707-0660 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "Malahal Naineni" > To: nfs-ganesha-devel@lists.sourceforge.net, "william allen simpson" > , > mbenja...@redhat.com > Sent: Thursday, October 15, 2015 1:26:55 PM > Subject: Re: [Nfs-ganesha-devel] V2.3-dev-10 broke kerberos mounts, > regression! > > Bill, I did test with your d12-pri branch. The kerberos mount does work > with it but every other mount fails! I don't think this is something to > do with your patch. I did make sure that all file creates and reads do > work. > > The mount failure (every other!) is: > > # mount -overs=4,sec=krb5p regcluster:/ibm/gpfs0/regfset2 /mnt1 > mount.nfs: Operation not permitted > > I will take tcpdump and update if I have any more details. > > Regards, Malahal. > > > Malahal Naineni [mala...@us.ibm.com] wrote: > > Hi All, > > > > The latest ganesha2.3 doesn't work with kerberos mounts. It > > appears to work with krb5 mounts but definitely fails with krb5i or > > krb5p. I primarily tested with krb5p mounts, and git bisect pointed to > > ganesha commit: 2de276416a5d2034fcd765434ed51794d622e916 > > > > That is just a submodule commit, so bisecting libntirpc pointed to this > > commit: 378832326fea58945c88963cef7ef3fe8500fd47 > > > > I don't see much there but some new flags were introduced in that commit > > related to gss. > > > > Appreciate if Matt or Bill can help figure this out. I am not familiar > > with this area at all. The failure is very easy to recreate, the mount > > itself fails. tcpdump shows we are sending something but the client > > fails to decode the response. This results in mount() syscall failing > > with EIO at the client. > > > > Regards, Malahal. > > > > > > -- > > ___ > > Nfs-ganesha-devel mailing list > > Nfs-ganesha-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > > > > -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] [PATCH] Avoid reinitializing svc_rqst_set.mtx
This looks suspicious--if we are reinitializing mtx--are we also the rbtx? Is there a problem instead with the double-lock init here? Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-707-0660 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "Malahal Naineni" > To: nfs-ganesha-devel@lists.sourceforge.net > Cc: mbenja...@redhat.com > Sent: Saturday, October 17, 2015 9:11:17 AM > Subject: [PATCH] Avoid reinitializing svc_rqst_set.mtx > > From e087e9337fa34592c05376bd947f0b430f175e4c Mon Sep 17 00:00:00 2001 > From: Malahal Naineni > Date: Sat, 17 Oct 2015 08:06:47 -0500 > Subject: [PATCH] Avoid reinitializing svc_rqst_set.mtx > > Signed-off-by: Malahal Naineni > --- > src/svc_rqst.c | 1 - > 1 file changed, 1 deletion(-) > > diff --git a/src/svc_rqst.c b/src/svc_rqst.c > index 6d83fae..8a8de25 100644 > --- a/src/svc_rqst.c > +++ b/src/svc_rqst.c > @@ -167,7 +167,6 @@ svc_rqst_init() > if (initialized) > goto unlock; > > - mutex_init(&svc_rqst_set.mtx, NULL); > code = rbtx_init(&svc_rqst_set.xt, rqst_thrd_cmpf, SVC_RQST_PARTITIONS, >RBT_X_FLAG_ALLOC | RBT_X_FLAG_CACHE_RT); > if (code) > -- > 1.8.3.1 > > -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Subject: [PATCH] Call destroy on mutexes and condition variables
ok
(PR?)
--
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-707-0660
fax. 734-769-8938
cel. 734-216-5309
- Original Message -
> From: "Malahal Naineni"
> To: nfs-ganesha-devel@lists.sourceforge.net
> Cc: mbenja...@redhat.com
> Sent: Saturday, October 17, 2015 9:38:47 AM
> Subject: Subject: [PATCH] Call destroy on mutexes and condition variables
>
> From 510492a32b6427d265b812b3923de5910d96d310 Mon Sep 17 00:00:00 2001
> From: Malahal Naineni
> Date: Sat, 17 Oct 2015 08:29:26 -0500
> Subject: [PATCH] Call destroy on mutexes and condition variables
>
> We should destroy them before freeing the memory. Supresses valgrind
> errors.
> ---
> src/clnt_vc.c | 3 +++
> src/rpc_ctx.c | 4
> 2 files changed, 7 insertions(+)
>
> diff --git a/src/clnt_vc.c b/src/clnt_vc.c
> index 4c614d1..c893605 100644
> --- a/src/clnt_vc.c
> +++ b/src/clnt_vc.c
> @@ -316,6 +316,7 @@ clnt_vc_ncreate2(int fd, /* open file descriptor */
> if (cs) {
> mem_free(cs, sizeof(struct ct_serialized));
> }
> + mutex_destroy(&clnt->cl_lock);
> mem_free(clnt, sizeof(CLIENT));
> }
> if (rec) {
> @@ -800,6 +801,7 @@ clnt_vc_release(CLIENT *clnt, u_int flags)
> mem_free(clnt->cl_netid, strlen(clnt->cl_netid) + 1);
> if (clnt->cl_tp && clnt->cl_tp[0])
> mem_free(clnt->cl_tp, strlen(clnt->cl_tp) + 1);
> + mutex_destroy(&clnt->cl_lock);
> mem_free(clnt, sizeof(CLIENT));
>
> REC_LOCK(rec);
> @@ -861,6 +863,7 @@ clnt_vc_destroy(CLIENT *clnt)
> mem_free(clnt->cl_netid, strlen(clnt->cl_netid) + 1);
> if (clnt->cl_tp && clnt->cl_tp[0])
> mem_free(clnt->cl_tp, strlen(clnt->cl_tp) + 1);
> + mutex_destroy(&clnt->cl_lock);
> mem_free(clnt, sizeof(CLIENT));
>
> if (xd_refcnt == 0) {
> diff --git a/src/rpc_ctx.c b/src/rpc_ctx.c
> index 3590422..6c99927 100644
> --- a/src/rpc_ctx.c
> +++ b/src/rpc_ctx.c
> @@ -85,6 +85,8 @@ alloc_rpc_call_ctx(CLIENT *clnt, rpcproc_t proc, xdrproc_t
> xdr_args,
> "%s: call ctx insert failed (xid %d client %p)",
> __func__, ctx->xid, clnt);
> REC_UNLOCK(rec);
> + mutex_destroy(&ctx->we.mtx);
> + cond_destroy(&ctx->we.cv);
> mem_free(ctx, sizeof(rpc_ctx_t));
> ctx = NULL;
> goto out;
> @@ -248,5 +250,7 @@ free_rpc_call_ctx(rpc_ctx_t *ctx, uint32_t flags)
>
> if (ctx->msg)
> free_rpc_msg(ctx->msg);
> + mutex_destroy(&ctx->we.mtx);
> + cond_destroy(&ctx->we.cv);
> mem_free(ctx, sizeof(rpc_ctx_t));
> }
> --
> 1.8.3.1
>
>
--
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Subject: [PATCH] Call destroy on mutexes and condition variables
sorry, i meant pull request
--
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-707-0660
fax. 734-769-8938
cel. 734-216-5309
- Original Message -
> From: "Malahal Naineni"
> To: "Matt Benjamin"
> Cc: nfs-ganesha-devel@lists.sourceforge.net
> Sent: Monday, October 19, 2015 10:30:04 AM
> Subject: Re: Subject: [PATCH] Call destroy on mutexes and condition variables
>
> Matt Benjamin [mbenja...@redhat.com] wrote:
> > ok
> >
> > (PR?)
>
> You mean problem report? valgrind drd tool!
>
> > --
> > Matt Benjamin
> > Red Hat, Inc.
> > 315 West Huron Street, Suite 140A
> > Ann Arbor, Michigan 48103
> >
> > http://www.redhat.com/en/technologies/storage
> >
> > tel. 734-707-0660
> > fax. 734-769-8938
> > cel. 734-216-5309
> >
> > - Original Message -
> > > From: "Malahal Naineni"
> > > To: nfs-ganesha-devel@lists.sourceforge.net
> > > Cc: mbenja...@redhat.com
> > > Sent: Saturday, October 17, 2015 9:38:47 AM
> > > Subject: Subject: [PATCH] Call destroy on mutexes and condition variables
> > >
> > > From 510492a32b6427d265b812b3923de5910d96d310 Mon Sep 17 00:00:00 2001
> > > From: Malahal Naineni
> > > Date: Sat, 17 Oct 2015 08:29:26 -0500
> > > Subject: [PATCH] Call destroy on mutexes and condition variables
> > >
> > > We should destroy them before freeing the memory. Supresses valgrind
> > > errors.
> > > ---
> > > src/clnt_vc.c | 3 +++
> > > src/rpc_ctx.c | 4
> > > 2 files changed, 7 insertions(+)
> > >
> > > diff --git a/src/clnt_vc.c b/src/clnt_vc.c
> > > index 4c614d1..c893605 100644
> > > --- a/src/clnt_vc.c
> > > +++ b/src/clnt_vc.c
> > > @@ -316,6 +316,7 @@ clnt_vc_ncreate2(int fd, /* open file descriptor
> > > */
> > > if (cs) {
> > > mem_free(cs, sizeof(struct ct_serialized));
> > > }
> > > + mutex_destroy(&clnt->cl_lock);
> > > mem_free(clnt, sizeof(CLIENT));
> > > }
> > > if (rec) {
> > > @@ -800,6 +801,7 @@ clnt_vc_release(CLIENT *clnt, u_int flags)
> > > mem_free(clnt->cl_netid, strlen(clnt->cl_netid) + 1);
> > > if (clnt->cl_tp && clnt->cl_tp[0])
> > > mem_free(clnt->cl_tp, strlen(clnt->cl_tp) + 1);
> > > + mutex_destroy(&clnt->cl_lock);
> > > mem_free(clnt, sizeof(CLIENT));
> > >
> > > REC_LOCK(rec);
> > > @@ -861,6 +863,7 @@ clnt_vc_destroy(CLIENT *clnt)
> > > mem_free(clnt->cl_netid, strlen(clnt->cl_netid) + 1);
> > > if (clnt->cl_tp && clnt->cl_tp[0])
> > > mem_free(clnt->cl_tp, strlen(clnt->cl_tp) + 1);
> > > + mutex_destroy(&clnt->cl_lock);
> > > mem_free(clnt, sizeof(CLIENT));
> > >
> > > if (xd_refcnt == 0) {
> > > diff --git a/src/rpc_ctx.c b/src/rpc_ctx.c
> > > index 3590422..6c99927 100644
> > > --- a/src/rpc_ctx.c
> > > +++ b/src/rpc_ctx.c
> > > @@ -85,6 +85,8 @@ alloc_rpc_call_ctx(CLIENT *clnt, rpcproc_t proc,
> > > xdrproc_t
> > > xdr_args,
> > > "%s: call ctx insert failed (xid %d client %p)",
> > > __func__, ctx->xid, clnt);
> > > REC_UNLOCK(rec);
> > > + mutex_destroy(&ctx->we.mtx);
> > > + cond_destroy(&ctx->we.cv);
> > > mem_free(ctx, sizeof(rpc_ctx_t));
> > > ctx = NULL;
> > > goto out;
> > > @@ -248,5 +250,7 @@ free_rpc_call_ctx(rpc_ctx_t *ctx, uint32_t flags)
> > >
> > > if (ctx->msg)
> > > free_rpc_msg(ctx->msg);
> > > + mutex_destroy(&ctx->we.mtx);
> > > + cond_destroy(&ctx->we.cv);
> > > mem_free(ctx, sizeof(rpc_ctx_t));
> > > }
> > > --
> > > 1.8.3.1
> > >
> > >
> >
>
>
--
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] NFS-Ganesha Proxy
This looks like to me as if you need to export LD_LIBRARY_PATH so as to include the path to libntirpc.so? Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-707-0660 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "Teddy Wong" > To: "Daniel Gryniewicz" > Cc: nfs-ganesha-supp...@lists.sourceforge.net, "NFS Ganesha Developers" > > Sent: Monday, October 26, 2015 6:44:24 PM > Subject: Re: [Nfs-ganesha-devel] NFS-Ganesha Proxy > > I removed the parameters and the errors no longer appears. > > Yet, I am still seeing an issue loading the module file: > > 26/10/2015 18:41:08 : epoch 562eac04 : mtl-centos7 : ganesha.nfsd-13248[main] > show_tree :FSAL :INFO : /dev/hugepages > 26/10/2015 18:41:08 : epoch 562eac04 : mtl-centos7 : ganesha.nfsd-13248[main] > show_tree :FSAL :INFO : /run > 26/10/2015 18:41:08 : epoch 562eac04 : mtl-centos7 : ganesha.nfsd-13248[main] > show_tree :FSAL :INFO : /boot > 26/10/2015 18:41:08 : epoch 562eac04 : mtl-centos7 : ganesha.nfsd-13248[main] > show_tree :FSAL :INFO : /test > 26/10/2015 18:41:08 : epoch 562eac04 : mtl-centos7 : ganesha.nfsd-13248[main] > re_index_fs_fsid :FSAL :DEBUG :Reindex /test from > 0x134090f283633754.0xd5f08ba760ea4a95 to > 0x954af290.0x54376383 > 26/10/2015 18:41:08 : epoch 562eac04 : mtl-centos7 : ganesha.nfsd-13248[main] > process_claim :FSAL :DEBUG :FSAL XFS Claiming /test > 26/10/2015 18:41:08 : epoch 562eac04 : mtl-centos7 : ganesha.nfsd-13248[main] > claim_posix_filesystems :FSAL :INFO :Root fs for export /bricks/demo/scratch > is /test > 26/10/2015 18:41:08 : epoch 562eac04 : mtl-centos7 : ganesha.nfsd-13248[main] > export_commit_common :CONFIG :INFO :Export 1 created at pseudo > (/bricks/demo/scratch) with path (/bricks/demo/scratch) and tag ((null)) > perms (no_root_squash, RWrw, , , , , , ) > 26/10/2015 18:41:08 : epoch 562eac04 : mtl-centos7 : ganesha.nfsd-13248[main] > export_commit_common :CONFIG :INFO :Export 1 has 0 defined clients > 26/10/2015 18:41:08 : epoch 562eac04 : mtl-centos7 : ganesha.nfsd-13248[main] > load_fsal :NFS STARTUP :DEBUG :Loading FSAL PROXY with > /usr/lib64/ganesha/libfsalproxy.so > 26/10/2015 18:41:08 : epoch 562eac04 : mtl-centos7 : ganesha.nfsd-13248[main] > load_fsal :NFS STARTUP :CRIT :Could not dlopen > module:/usr/lib64/ganesha/libfsalproxy.so > Error:/usr/lib64/ganesha/libfsalproxy.so: undefined symbol: > xdr_free_null_stream > 26/10/2015 18:41:08 : epoch 562eac04 : mtl-centos7 : ganesha.nfsd-13248[main] > load_fsal :NFS STARTUP :MAJ :Failed to load module > (/usr/lib64/ganesha/libfsalproxy.so) because: Can not access a needed shared > library > 26/10/2015 18:41:08 : epoch 562eac04 : mtl-centos7 : ganesha.nfsd-13248[main] > build_default_root :CONFIG :DEBUG :Allocating Pseudo root export > 26/10/2015 18:41:08 : epoch 562eac04 : mtl-centos7 : ganesha.nfsd-13248[main] > pseudofs_create_export :FSAL :DEBUG :Created exp 0x7faf00ade640 - / > 26/10/2015 18:41:08 : epoch 562eac04 : mtl-centos7 : ganesha.nfsd-13248[main] > build_default_root :CONFIG :INFO :Export 0 (/) successfully created > 26/10/2015 18:41:08 : epoch 562eac04 : mtl-centos7 : ganesha.nfsd-13248[main] > config_errs_to_log :CONFIG :CRIT :Config File > (/etc/ganesha/ganesha.conf:41): Failed to load FSAL (PROXY) because: Can not > access a needed shared library > 26/10/2015 18:41:08 : epoch 562eac04 : mtl-centos7 : ganesha.nfsd-13248[main] > config_errs_to_log :CONFIG :CRIT :Config File > (/etc/ganesha/ganesha.conf:41): 1 validation errors in block FSAL > 26/10/2015 18:41:08 : epoch 562eac04 : mtl-centos7 : ganesha.nfsd-13248[main] > config_errs_to_log :CONFIG :CRIT :Config File > (/etc/ganesha/ganesha.conf:41): Errors processing block (FSAL) > 26/10/2015 18:41:08 : epoch 562eac04 : mtl-centos7 : ganesha.nfsd-13248[main] > config_errs_to_log :CONFIG :CRIT :Config File > (/etc/ganesha/ganesha.conf:24): 1 validation errors in block EXPORT > 26/10/2015 18:41:08 : epoch 562eac04 : mtl-centos7 : ganesha.nfsd-13248[main] > config_errs_to_log :CONFIG :CRIT :Config File > (/etc/ganesha/ganesha.conf:24): Errors processing block (EXPORT) > 26/10/2015 18:41:08 : epoch 562eac04 : mtl-centos7 : ganesha.nfsd-13248[main] > lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed > for proper quota management in FSAL > 26/10/2015 18:41:08 : epoch 562eac04 : mtl-centos7 : ganesha.nfsd-13248[main] > lower_my_caps :NFS STARTUP :EVENT :currenty set capabilities are: = > cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_
Re: [Nfs-ganesha-devel] Conference Call move to Tuesday
ok -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-707-0660 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "Kaleb S. KEITHLEY" > To: nfs-ganesha-devel@lists.sourceforge.net > Sent: Thursday, October 29, 2015 3:16:28 PM > Subject: Re: [Nfs-ganesha-devel] Conference Call move to Tuesday > > fine with me too > > > On 10/29/2015 01:38 PM, Frank Filz wrote: > > I'd like to see if we can move the conference call to Tuesday for the rest > > of the year. > > > > Thursday Nov 16 and Thursday Dec 2 I will likely be with my wife at the > > hospital with new baby related stuff. Thursday Nov 26 is Thanksgiving. > > > > With Tuesday calls, we should be able to avoid a three week outage. > > > > We probably should move the merge to Tuesday or Wednesday also (otherwise > > we > > could wind up with three weeks of no merge, or worse, once we come home > > with > > the baby, I'll be out for at least one more week). > > > > Oh, and no dev-1 this week. Let's give folks a chance to get their wheels > > turning again on 2.4. > > > > Thanks > > > > Frank > > > > > > --- > > This email has been checked for viruses by Avast antivirus software. > > https://www.avast.com/antivirus > > > > > > -- > > ___ > > Nfs-ganesha-devel mailing list > > Nfs-ganesha-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > > > > -- > > Kaleb > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] [nfs-ganesha] why forking libntirpc? (#81)
Hi Dmitry, Let me fill in background. The linuxbox2 submodule is a friendly fork of the Linux libtirpc, which for agreed-on reasons, need to diverge. (The Linux library has strict ABI compatibility with userland on Solaris, for example, and will never support bi-directional RPC.) The linuxbox2 repo, though, was never an autonomous project--the real project is nfs-ganesha, and has been since inception. If you don't like the fact that nfs-ganesha appears to be a fork, we can re-create it? Regards, Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-707-0660 fax. 734-769-8938 cel. 734-216-5309 - Original Message - > From: "Frank Filz" > To: Nfs-ganesha-devel@lists.sourceforge.net > Cc: mbenja...@redhat.com, d...@redhat.com > Sent: Monday, November 16, 2015 7:27:54 PM > Subject: FW: [nfs-ganesha] why forking libntirpc? (#81) > > Dan, Matt, > > > > Can we address this? I can’t do anything about it to resolve the issue… > > > > Frank > > > > From: Dmitry Smirnov [mailto:notificati...@github.com] > Sent: Monday, November 16, 2015 2:00 PM > To: nfs-ganesha/nfs-ganesha > Cc: Frank Filz > Subject: Re: [nfs-ganesha] why forking libntirpc? (#81) > > > > Directly sub-moduling libntirpc from linuxbox2 would cause less confusion. > Other projects manage to sub-module without forking so can you. > > — > Reply to this email directly or view it on GitHub > <https://github.com/nfs-ganesha/nfs-ganesha/issues/81#issuecomment-157184074> > . > <https://github.com/notifications/beacon/AAYYUzq7z4_pyh0MueAkevfRUGz6s1iPks5pGklqgaJpZM4Givk_.gif> > > > > --- > This email has been checked for viruses by Avast antivirus software. > https://www.avast.com/antivirus > -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] memory leak related to same client using v3 and v4 mounts and fixes
Hi,
inline
--
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-707-0660
fax. 734-769-8938
cel. 734-216-5309
- Original Message -
> From: "Jeremy Bongio"
> To: nfs-ganesha-devel@lists.sourceforge.net
> Sent: Tuesday, November 17, 2015 11:46:06 AM
> Subject: [Nfs-ganesha-devel] memory leak related to same client using v3 and
> v4 mounts and fixes
>
> There is a memory leak that is caused by V3 and V4 duplicate request
> caches being shared.
>
> We don't keep track of whether a DRC was used for NFSv4 or NFSv3 in the
> hashkey, only the address ... but each cache _does_ have a protocol
> type. This type is used later to decide which request/replies should be
> cached and which shouldn't. This results in large operations like READs
> being cached when there is a mismatch between the request protocol-type
> and the DRC protocol-type. This can quickly (in a few minutes) eat up
> all memory (and trigger the OOM killer) in targeted testing.
>
> 1. Either we can include the DRC type when creating the hashkey for the DRC.
you could include it in the sort, but not the hash key, because is pre-computed
by the XDR layer (and we want to continue doing that)
>
> 2. Or we could stop relying on the type of the DRC and rely instead on
> the type of the current request. This would involve in
> nfs_dupreq_start() using get_drc_type(req); instead of drc->type.
that would be fine
Matt
>
>
> What do you think is the best fix?
>
> Here is one quick fix I tested that worked. However, is it safe to
> simply add to the hashkey? I think so, but maybe I'm not thinking of all
> scenarios.
> @@ -574,6 +574,12 @@ nfs_dupreq_get_drc(struct svc_req *req)
> "get drc for addr: %s", str);
> }
>
> + /* Now include the nfsv3 or nfsv4 type in hashkey.
> +* Otherwise we confuse V4 and V3 caches which will
> +* later mess up process for deciding if a
> request is
> +* is cacheable or not. */
> + drc_k.d_u.tcp.hk += dtype;
> +
> t =
> rbtx_partition_of_scalar(&drc_st->tcp_drc_recycle_t,
> drc_k.d_u.tcp.hk);
> DRC_ST_LOCK();
>
>
> Here is a simple script I use to reproduce the defect:
> #!/usr/bin/perl
>
> my $server_ip = "10.10.0.11";
>
> while(1) {
> my $output = `mount -t nfs -o vers=4 $server_ip:/ibma /mnt/cthon;
> cat /mnt/cthon/a; umount /mnt/cthon; `;
> my $output = `mount -t nfs -o vers=3 $server_ip:/ibm/gpfs0/a
> /mnt/cthon; cat /mnt/cthon/a; umount /mnt/cthon; `;
> }
>
> --
> Jeremy Bongio
>
> jbon...@us.ibm.com
> IBM Linux Technology Center - Linux Filesystems Team
> Linux Development Engineer
>
>
> --
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>
--
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] [Nfs-ganesha-support] NFS-Ganesha Proxy
Hi,
Must not be exported. Fix will require building the library with update map
file.
Matt
- Original Message -
> From: "Kenneth Waegeman"
> To: nfs-ganesha-devel@lists.sourceforge.net
> Sent: Thursday, November 26, 2015 7:49:53 AM
> Subject: Re: [Nfs-ganesha-devel] [Nfs-ganesha-support] NFS-Ganesha Proxy
>
> See message http://sourceforge.net/p/nfs-ganesha/mailman/message/34571513/
>
> We are having the same problem.
> It looks like building nfs-ganesha 2.3 does not include the
> xdr_free_null_stream symbol:
> #` nm -D /usr/lib64/libntirpc.so.1.3.1 |grep xdr_free `gives no result
>
> It is in the code base:
>
> libntirpc]# git grep xdr_free_null_stream
> ntirpc/rpc/xdr.h:extern XDR xdr_free_null_stream;
> ntirpc/rpc/xdr.h: return (*proc) (&xdr_free_null_stream, objp);
> src/xdr.c:XDR xdr_free_null_stream = {
>
> What can be the problem?
> Thanks!
>
> Kenneth
>
>
>
>
>
> --
> Go from Idea to Many App Stores Faster with Intel(R) XDK
> Give your users amazing mobile app experiences with Intel(R) XDK.
> Use one codebase in this all-in-one HTML5 development environment.
> Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs.
> http://pubads.g.doubleclick.net/gampad/clk?id=254741551&iu=/4140
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>
--
--
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-707-0660
fax. 734-769-8938
cel. 734-216-5309
--
Go from Idea to Many App Stores Faster with Intel(R) XDK
Give your users amazing mobile app experiences with Intel(R) XDK.
Use one codebase in this all-in-one HTML5 development environment.
Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs.
http://pubads.g.doubleclick.net/gampad/clk?id=254741911&iu=/4140
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] [Nfs-ganesha-support] NFS-Ganesha Proxy
Hi,
I've pushed the mapfile change onto ntirpc duplex-13. A pullup for it should
happen in the next window.
Thanks,
Matt
- Original Message -
> From: "Matt Benjamin"
> To: "Kenneth Waegeman"
> Cc: nfs-ganesha-devel@lists.sourceforge.net
> Sent: Monday, November 30, 2015 3:30:58 PM
> Subject: Re: [Nfs-ganesha-devel] [Nfs-ganesha-support] NFS-Ganesha Proxy
>
> Hi,
>
> Must not be exported. Fix will require building the library with update map
> file.
>
> Matt
>
> - Original Message -
> > From: "Kenneth Waegeman"
> > To: nfs-ganesha-devel@lists.sourceforge.net
> > Sent: Thursday, November 26, 2015 7:49:53 AM
> > Subject: Re: [Nfs-ganesha-devel] [Nfs-ganesha-support] NFS-Ganesha Proxy
> >
> > See message http://sourceforge.net/p/nfs-ganesha/mailman/message/34571513/
> >
> > We are having the same problem.
> > It looks like building nfs-ganesha 2.3 does not include the
> > xdr_free_null_stream symbol:
> > #` nm -D /usr/lib64/libntirpc.so.1.3.1 |grep xdr_free `gives no result
> >
> > It is in the code base:
> >
> > libntirpc]# git grep xdr_free_null_stream
> > ntirpc/rpc/xdr.h:extern XDR xdr_free_null_stream;
> > ntirpc/rpc/xdr.h: return (*proc) (&xdr_free_null_stream, objp);
> > src/xdr.c:XDR xdr_free_null_stream = {
> >
> > What can be the problem?
> > Thanks!
> >
> > Kenneth
> >
> >
> >
> >
> >
> > --
> > Go from Idea to Many App Stores Faster with Intel(R) XDK
> > Give your users amazing mobile app experiences with Intel(R) XDK.
> > Use one codebase in this all-in-one HTML5 development environment.
> > Design, debug & build mobile apps & 2D/3D high-impact games for multiple
> > OSs.
> > http://pubads.g.doubleclick.net/gampad/clk?id=254741551&iu=/4140
> > ___
> > Nfs-ganesha-devel mailing list
> > Nfs-ganesha-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
> >
>
> --
> --
> Matt Benjamin
> Red Hat, Inc.
> 315 West Huron Street, Suite 140A
> Ann Arbor, Michigan 48103
>
> http://www.redhat.com/en/technologies/storage
>
> tel. 734-707-0660
> fax. 734-769-8938
> cel. 734-216-5309
>
--
--
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-707-0660
fax. 734-769-8938
cel. 734-216-5309
--
Go from Idea to Many App Stores Faster with Intel(R) XDK
Give your users amazing mobile app experiences with Intel(R) XDK.
Use one codebase in this all-in-one HTML5 development environment.
Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs.
http://pubads.g.doubleclick.net/gampad/clk?id=254741911&iu=/4140
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] switch warnings as errors
Hi Bill, I don't think there is a "maintainer mode" issue per se, rather, you're building ntirpc -ahead- of ganesha's submodule, where I had staged my vsock change. I've force-updated that, which may fix your issue. Maybe it would be safest to build a topic branch from the commit ganesha is pulled up to use, though? Then your work would be isolated from what's happening on the head. Matt - Original Message - > From: "William Allen Simpson" > To: Nfs-ganesha-devel@lists.sourceforge.net > Sent: Tuesday, December 1, 2015 7:05:35 AM > Subject: [Nfs-ganesha-devel] switch warnings as errors > > With NTI-RPC as a separate library, the switch warnings > as errors has got to be turned off. The current tree is > again in a non-compilable state in Maintainer Mode. > > The separate library can have new enum cases (or have > some removed). The latest is Matt's XPRT_VSOCK which is > supported in ntirpc, but not (yet) in Ganesha. > > The tree has to keep compiling even with defaults. > Unknown defaults have to be logged, not result in > compile errors. > > Warnings as errors is a completely unworkable environment. > > -- > Go from Idea to Many App Stores Faster with Intel(R) XDK > Give your users amazing mobile app experiences with Intel(R) XDK. > Use one codebase in this all-in-one HTML5 development environment. > Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs. > http://pubads.g.doubleclick.net/gampad/clk?id=254741911&iu=/4140 > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-707-0660 fax. 734-769-8938 cel. 734-216-5309 -- Go from Idea to Many App Stores Faster with Intel(R) XDK Give your users amazing mobile app experiences with Intel(R) XDK. Use one codebase in this all-in-one HTML5 development environment. Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs. http://pubads.g.doubleclick.net/gampad/clk?id=254741911&iu=/4140 ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Change in ffilz/nfs-ganesha[next]: [CLEANUP] Initial cleanup of file nfs_rpc_dispatcher_thread.
Hi, Look folks, we need to all take a step back and ask, whenever we send email to (at least) the nfs-ganesha mailing lists: *** is the email I am about to send, thoughtful, collegial, and constructive *** I know from experience with you, Bill, that you don't mean to be personally hurtful in anything you said in this thread, but I think you are not fully aware of how your words read here, and I think we need to restart this conversation 1. from a place of quiet 2. from a place of pure technical evaluation 3. on gerrit (Dan's suggestion), where we can just think about lines of code Let's not be the kind of community where people have unisql-style flame wars and code reviews are more heat than light. Let's stay focused on how to make the most of everyone's effort, creativity, and attention, not on netiquette. Thanks, Matt - Original Message - > From: "William Allen Simpson" > To: "Swen Schillig" > Cc: nfs-ganesha-devel@lists.sourceforge.net > Sent: Monday, December 7, 2015 9:11:30 AM > Subject: Re: [Nfs-ganesha-devel] Change in ffilz/nfs-ganesha[next]: [CLEANUP] > Initial cleanup of file > nfs_rpc_dispatcher_thread. > > On 12/7/15 7:05 AM, Swen Schillig wrote: > > I understood that you dislike the changes I proposed. > > Maybe you can tell us what your changes would be to improve > > readability, maintainability and "style". > > > If it already works, don't "fix" it. > > I don't make changes to improve "style" unless I'm already > doing work to actually fix something in the code. > > Heck, as a long-time consultant, I usually simply conform to the > same style currently used in the code. If the style changes > from module to module, I conform to each individually. > > My anticipated future changes to this code have already been > demonstrated in nfs_rpc_rdma.c > > Note the nice clean simple RDMA dispatcher. Likewise, we are > planning on moving the rest of the TCP and UDP dispatchers back > into ntirpc (where they were originally). > > Eventually, the whole kit-and-kaboodle will call a userland TCP. > > But first, there are a lot of more urgent things to do, such as > determine how we need to setup memory management to parallel > Frank's recent changes to Ganesha. I'm working on that > > > > Besides, this patch is NOT a rewrite but a cleanup. > > > I beg to differ. > > In any case, should you want to help, I've already requested > that you IBM folks help determine bottlenecks -- something that > really needs to be done. > > Let's make changes based on data-driven testing. > > In September, I gave a patch to be applied for testing. We > need a before and after performance test. And especially > need to learn its impact on the complaint by Malahal that it > peaks at 2 clients. Only you have the tests Malahal used. > > https://review.gerrithub.io/245072 RFE - disable [R|S][UN]LOCK > > Note, this isn't a final patch. It's just a testing patch. > > > -- > Go from Idea to Many App Stores Faster with Intel(R) XDK > Give your users amazing mobile app experiences with Intel(R) XDK. > Use one codebase in this all-in-one HTML5 development environment. > Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs. > http://pubads.g.doubleclick.net/gampad/clk?id=254741911&iu=/4140 > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-707-0660 fax. 734-769-8938 cel. 734-216-5309 -- Go from Idea to Many App Stores Faster with Intel(R) XDK Give your users amazing mobile app experiences with Intel(R) XDK. Use one codebase in this all-in-one HTML5 development environment. Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs. http://pubads.g.doubleclick.net/gampad/clk?id=254741911&iu=/4140 ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] [RFC] Replace fsal_status_t by base type
Hi Swen, I don't really think that would be preferable, no. I think the current style is more in the direction C++, at least, is moving, and I'm happy with it. I would like to see -less- use of TLS in the call path, as a side note, but that is a detail. I welcome other feedback. Matt - Original Message - > From: "Swen Schillig" > To: "nfs-ganesha-devel" > Sent: Tuesday, December 15, 2015 7:52:24 AM > Subject: [Nfs-ganesha-devel] [RFC] Replace fsal_status_t by base type > > Quiet a few functions do have a return value of the type fsal_status_t > which is basically a struct of two integers. > > In a non-error situation the second value is not of any interest and even > if an error occured, the second value is often set to zero. > > Therefore, I suggest to turn the status variable (back) to a base-type > (e.g. int) > and, if required, use a thread-local variable to indicate additional > information. > > Pretty much like errno, in fact, why not use errno. > The current second value of the struct (minor) seems to be used for > POSIX error codes already, which is pretty much for what errno is used. > > This change would simplify the code as it would spare the effort of > always building > the struct fsal_status_t including the handling to figure out if an > error occured (FSAL_IS_ERROR). > > Comments ? > > Cheers Swen > > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-707-0660 fax. 734-769-8938 cel. 734-216-5309 -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] pNFS documentation
Hi Bishoy, Broadly, the FSAL you're using gets to decide how it implements layout operations, and also how it maintains consistency. You could be observing a generic bug of some kind, but in the best case, Glusterfs is responsible for keeping the distributed namespace consistent. (Which means that Glusterfs folks may be of most help at least initially.) Matt - Original Message - > From: "Gmail" > To: nfs-ganesha-devel@lists.sourceforge.net > Sent: Monday, December 21, 2015 8:18:56 PM > Subject: [Nfs-ganesha-devel] pNFS documentation > > Hello All, > > Is there any documentation describing how does pNFS works in NFS-Ganesha? > > I’ve done a simple setup with 3 Gluster storage nodes and two clients, > NFS-Ganesha is running on the three storage nodes, the Gluster volume is > configured in distributed configuration. > > I was able to get pNFS to work and I’ve traced the RPCs and I found that the > client is able to send data directly to the storage nodes, so the > functionality is working. > > But when I tried to mount the Gluster volume from two different NFS-Ganesha > nodes on two different clients, I was able to mount successfully, and I was > able to write from both clients, but what I’ve noticed when I use two > different storage nodes for writes to the same file, I see the modified file > from the other client immediately, but the file disappears from the client > where I’ve wrote it then shows up after about 30 seconds! > > Is there anyone who knows how the MDS is working in pNFS Ganesha?! and how > does the MDS in the whole clusters keep their metadata consistent? > > > > — Bishoy > > > > > > > > -- > > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-707-0660 fax. 734-769-8938 cel. 734-216-5309 -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] No merge this week
Happy holidays. Matt - Original Message - > From: "Frank Filz" > To: nfs-ganesha-devel@lists.sourceforge.net > Sent: Wednesday, December 23, 2015 7:38:43 PM > Subject: [Nfs-ganesha-devel] No merge this week > > I am doing some additional testing of multiple-fd and I'm finding some > issues. No sense pushing code that doesn't test as working... > > With that, of course I have been spending all my time debugging, so no time > to merge anything else either... > > We will have a conference call Tuesday January 5th 8:30 AM PST where we will > discuss where we should go. > > Thanks and happy holidays and a happy new year to all. > > Frank > > > --- > This email has been checked for viruses by Avast antivirus software. > https://www.avast.com/antivirus > > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-707-0660 fax. 734-769-8938 cel. 734-216-5309 -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Patches that need back port from V2.4 to V2.3
Yes, should be backported. Matt - Original Message - > > commit 3054a3b41600a9f3e7bfed52aaa0f2f0a3d89cde > Author: Matt Benjamin > > FSAL_CEPH: NUL-terminate symlink buffers > -- -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-707-0660 fax. 734-769-8938 cel. 734-216-5309 -- Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140 ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Cache inode entries marked 'LRU_ENTRY_CLEANUP'
Hi Soumya,
Originally, cleanup was merely moving the erstwhile behavior of "kill-entry"
out-of-line.
All entries everywhere should have a refcount thta is positive, starting at 1
for "not in use but resident in the cache entry lookup table"--or 0 when all
refs are returned, and only then briefly before being destroyed.
We should probably review all the workflows the involve the cleanup queue, and
discuss whether they are fully coherent. It seems possible that entries which
don't clean up are "special" in some way that evolved more recently, like being
at the root of an export?
Matt
- Original Message -
> From: "Soumya Koduri"
> To: nfs-ganesha-devel@lists.sourceforge.net
> Sent: Monday, February 1, 2016 4:15:32 AM
> Subject: [Nfs-ganesha-devel] Cache inode entries marked 'LRU_ENTRY_CLEANUP'
>
> Hi,
>
> In cache_inode_unexport(), if we encounter an entry marked
> 'LRU_ENTRY_CLEANUP', we ignore it and continue.
>
>status = cache_inode_lru_ref(entry, LRU_FLAG_NONE);
>
>
>
>if (status != CACHE_INODE_SUCCESS) {
>/* This entry was going stale, skip it. */
>
> PTHREAD_RWLOCK_unlock(&export->lock);
> continue;
> }
>
> Who is responsible for cleaning up such entries? Shouldn't we do force
> cleanup here?
> We ran into a case where dbus thread is looping indefinitely on one such
> entry and there were no threads involved in cleaning it up [1]. That
> particular entry even had a positive refcnt.
>
> Please provide your inputs.
>
> Thanks,
> Soumya
>
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1301450#c3
>
> --
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>
--
--
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-707-0660
fax. 734-769-8938
cel. 734-216-5309
--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] guide about fsal driver
Hi, Dan means FSAL_RGW, which exposes RGW, Ceph's S3-style object storage service. So this, too, is similar but different. FSAL_RGW is similar to the Ceph, GPFS, Gluster, ... FSALs, in that it dispatches into a Ceph-provided library that does all the operations. I haven't worked with S3fs, and I don't know how it emulates filesystem I/O operations and semantics. Putatively, though, there should be nothing terribly substantial that S3fs already handles already for VFS (renames, writes at offsets, not to mention opens), that FSAL_VFS cannot in principle delegate. That said, FSAL_VFS may be making some wrong assumptions or need some behavioral tweaks unique to S3fs+fuse (or even just fuse--has anyone explored running -other- fuse-mounted filesystems under FSAL_VFS recently? I have not!). My intuition is that if you want to be namespace and I/O compatible with with S3fs, a quick path to get there would be to 1) debug and experiment with FSAL_VFS to find required adjustments, and then 2) move those adaptatations into a new FSAL stacked on FSAL_VFS. To Max's point, I always get confused by what FSAL_FUSELIKE did. If FSAL_FUSELIKE allowed Ganesha to be a FUSE filesystem provider, it might still make sense to talk about a FSAL_FUSE that allowed a Ganesha to generally export existing FUSE drivers? This would give you complete re-use of the FUSE behavior, without the kernel round-trips. Matt - Original Message - > From: "Daniel Gryniewicz" > To: nfs-ganesha-devel@lists.sourceforge.net > Sent: Thursday, February 11, 2016 8:01:27 AM > Subject: Re: [Nfs-ganesha-devel] guide about fsal driver > > On 02/10/2016 09:11 PM, Bryan Zhu wrote: > > Hi, > > > > I want to integrate S3FS which uses fuse with ganesha, is there any > > guide about how to do? > > > > Thanks, > > Bryan > > > > The best (although not the simplest) way to integrate is to import the > s3fs code as a library and wrap it in its own FSAL. This is what RedHat > is doing with FSAL_GWS for Ceph (which will be released soon). This > avoids the transition to and from Posix semantics, which can cause > problems, especially when both ends of that transition are not quite Posix. > > Dan > > > -- > Site24x7 APM Insight: Get Deep Visibility into Application Performance > APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month > Monitor end-to-end web transactions and take corrective actions now > Troubleshoot faster and improve end-user experience. Signup Now! > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-707-0660 fax. 734-769-8938 cel. 734-216-5309 -- Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
[Nfs-ganesha-devel] rebased, bisectable ntirpc params and vsock
are on g...@github.com:linuxbox2/nfs-ganesha.git next-vsock regards, Matt -- -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-707-0660 fax. 734-769-8938 cel. 734-216-5309 -- Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] SIGSEGV in "xdr_nfs_fh3()"
If I read the code correctly (xdr_inline.h), nodesize==0 connotes "read 0
bytes." So the surrounding code should not index into any such bytes in that
case, if it can in fact arise. I also am not sure what that case would mean.
Matt
- Original Message -
> From: "Soumya Koduri"
> To: "Matt W. Benjamin" , "Daniel Gryniewicz"
>
> Cc: nfs-ganesha-devel@lists.sourceforge.net
> Sent: Monday, February 15, 2016 4:40:15 AM
> Subject: [Nfs-ganesha-devel] SIGSEGV in "xdr_nfs_fh3()"
>
> Hi Matt/Daniel,
>
> There is a crash reported due to SIGSEGV in "xdr_nfs_fh3()" [1]. As
> mentioned in the comments [2], we couldn't analyze the core, but when
> inspected the code, found a potential issue.
>
>
> In 'xdr_nfs_fh3(..)' {
>
> .
> 458 if (!xdr_bytes
> 459 (xdrs, (char **)&objp->data.data_val,
> 460 (u_int *) & objp->data.data_len, 64))
> 461 return (false);
> 462
> 463 if (xdrs->x_op == XDR_DECODE) {
> 464 fh = (file_handle_v3_t *)objp->data.data_val;
> 465 fh->exportid = ntohs(fh->exportid);
> 466 }
>
> ...
> }
>
> In xdr_bytes(..) {
>
> 522 case XDR_DECODE:
> 523 if (nodesize == 0)
> 524 return (true);
> 525 if (sp == NULL)
> 526 *cpp = sp = mem_alloc(nodesize);
> 527 if (sp == NULL) {
> 528 __warnx(TIRPC_DEBUG_FLAG_XDR,
> 529 "xdr_bytes: out of memory");
> 530 return (false);
> 531 }
>
>
>
> ...
> }
>
> In the above routine (xdr_bytes), if nodesize is '0' (line:522) we
> return true without allocating any memory. And in 'xdr_nfs_fh3()', we
> shall end up de-referencing this unallocated memory at line 465.
>
> I am not sure in what cases nodesize could be '0' and how we can extract
> filehandle from the xdr object then. Could you please confirm the same.
>
>
> Thanks,
> Soumya
>
>
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1306691
> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1306691#c5
>
>
>
> --
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>
--
--
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-707-0660
fax. 734-769-8938
cel. 734-216-5309
--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Mounting between linux server and client with krb5 enabled
s > Squash = No_root_squash; # To enable/disable root squashing > Disable_ACL = TRUE; # To enable/disable ACL > #Protocols = 3,4 ; # NFS protocols supported > Protocols = 4; > Transports = "UDP,TCP" ; # Transport protocols supported > SecType = "krb5"; # Security flavors supported > } > > Thanks! > > -- > Site24x7 APM Insight: Get Deep Visibility into Application Performance > APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month > Monitor end-to-end web transactions and take corrective actions now > Troubleshoot faster and improve end-user experience. Signup Now! > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-707-0660 fax. 734-769-8938 cel. 734-216-5309 -- Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] [PROPOSAL] Using the CentOS CI for nfs-ganesha/next testing and patch verification
I definitely support. I need to add that the Ceph team needs us to have ceph and later rgw ganesha testing running under ceph's teuthology regime, but there's probably in the end a lot of overlap which should save rework. Matt - Original Message - > From: "Niels de Vos" > To: nfs-ganesha-devel@lists.sourceforge.net > Cc: "Karanbir Singh" , "Brian Stinson" > Sent: Sunday, February 21, 2016 8:38:47 AM > Subject: [Nfs-ganesha-devel] [PROPOSAL] Using the CentOS CI for > nfs-ganesha/next testing and patch verification > > Hi all, > > I'd like to propose to use the CentOS CI (at http://ci.centos.org) for > patch verification and eventually additional automated testing. At the > moment we use at least two Jenkins infrastructures for testing, neither > of which are accessible by 'external' NFS-Ganesha contributors. > > The CentOS team offers upstream projects the ability to run tests on > their infrastructure (physical machines cleanly installed for each test > run). I would like to move the "build FSAL_GLUSTER" job that is > currently running on a Red Hat internal environment to the CentOS CI. > > Other tests, like the CEA checker, and additional ones like pynfs could > be added to the NFS-Ganesha project in the CentOS CI later on. The > infrastructure makes it possible to do complex client-server testing on > CentOS-6 and 7. Things like pNFS functionality or other clustering > things can be done. GPFS might not be possible, but full open source > stacks should not be any problem. > > I would like to know who is interested in this so that I can file a > request with the CentOS team for this. > > Thanks, > Niels > > -- > Site24x7 APM Insight: Get Deep Visibility into Application Performance > APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month > Monitor end-to-end web transactions and take corrective actions now > Troubleshoot faster and improve end-user experience. Signup Now! > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-707-0660 fax. 734-769-8938 cel. 734-216-5309 -- Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Unbounded memory allocations (based on client data)
yes, thanks for this effort
matt
- Original Message -
> From: "Dominique Martinet"
> To: "nfs-ganesha-devel"
> Sent: Monday, February 29, 2016 6:12:49 PM
> Subject: [Nfs-ganesha-devel] Unbounded memory allocations (based on client
> data)
>
> Hi,
>
> Synopsis (coverity and codenomicon) folks have been fuzzing us a bit and
> found that we blindly trust the client and allocate huge buffers,
> e.g. send us some small string but pretending it's 0x length.
>
> I've found alot of these in include/nfsv41.h, what I've done so far is:
> - every xdr_array (allocates the whole array beforehand, we can
> probably fix that and handle elements one at a time eventually) is now
> bound to 1k elements
> - every inline_xdr_bytes that was bound to ~0 is now bound to 1MB
> I'm sure we'll want read/writes to go up to say 8MB or even configurable
> eventually but that's just some initial filter to pass more fuzzing
>
> Also found some comment that says there's nothing to free, but there is
> (the xdr_array data) so I'll remove that as well in a separate commit...
>
>
> Initial ballpark patch coming tomorrow when it's been tested a bit more,
> but thoughts/comments welcome. It's basically alot of these:
> @@ -4070,7 +4069,7 @@ extern "C" {
> return false;
> if (!xdr_array
> (xdrs, (char **)&objp->na41_aces.na41_aces_val,
> -(u_int *) &objp->na41_aces.na41_aces_len, ~0,
> +(u_int *) &objp->na41_aces.na41_aces_len, 1024,
> sizeof(nfsace4), (xdrproc_t) xdr_nfsace4))
> return false;
> return true;
>
>
> (I just need more time to make a constant macro and fix line lengths
> etc)
>
>
> Productive first day!
> --
> Dominique
>
> --
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>
--
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-707-0660
fax. 734-769-8938
cel. 734-216-5309
--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
