Re: Propose to link fuse-encfs against pkg libperfuse [Was Re: mmap errors related to using encfs]
Greg Troxel writes: > If I flip fuse-encfs's Makefile to use filesystems/fuse instead of > fuse.buildlink3.mk, it builds, and works. But git fails with a > complaint about mmap. Sorry, too many trial builds. With filesystems/fuse, it fails to build, probably because of mount API issues.
Re: Propose to link fuse-encfs against pkg libperfuse [Was Re: mmap errors related to using encfs]
Things are seeming very fuzzy to me and we need clarity before we can agree to change anything. First, I should be clear that we are only talking about how to use the high-level FUSE API. The low-level API and /dev/fuse are only available via libfuse/perfuse and perfuse, respectively. So no one has ever suggested, nor made it work, to use a filesystem that uses the low-level or kernel API on NetBSD, other than via perfuse. Indeed, that's why manu@ wrote perfuse. I am only aware of glusterfs which is in this category (but that's very important). Some comments: I don't know what "pkg fuse" means. If you mean pkgsrc/filesystems/fuse, then it doesn't make sense to use librefuse, as filesystems/fuse is the standard implementation of the FUSE API, and expects to talk to /dev/fuse. I am calling this libfuse as that is what is installed by that package. You talk about fuse-* "linking with". But, the way we do things is to include mk/fuse.buildlink3.mk. And the way those work is that we have logic to select defaults and variables to list which options are acceptable to the package and which are preferred to the user. I see no reason to depart from our practices in this area. Perhaps that's what you mean. Some people prefer librefuse because it does not involve a daemon and has fewer data copies. It is meant as an compatible implementation of the high-level API. Some people prefer libfuse/perfuse because of a belief in greater compatibility. It is clear that we do not have consensus and that this is a situation where preferences differ depending on how one gives weight to various issues (which is ok). In dealing with mk/fuse.buildlink3.mk, there are two separable issues. One is, on NetBSD, choosing to use librefuse (/usr/include/fuse.h and a faked-up pc), vs the fuse package. This does not appear to be implemented and I think most would think that adding it is reasonable. The second is changing the default away from librefuse. Comments have been mixed, and I think that should not happen soon, at the very least because this new way doesn't exist yet in pkgsrc and thus does not have wide experience. You asserted or at least implies that encfs mmap does not work when linked with librefuse but does with libfuse/perfuse. But that (that it works on NetBSD with libfuse/perfuse) sounds like news, not something I've seen on the list, and we lack an analysis of what is missing and why. It could be useful to add mmap to librefuse, or at least to clarify some documentation. It is normal to have a bl3 for a package and to select the base system version if adequate and present, and the package if not. That already exists for filesystems/perfuse. So we talk about depending on the perfuse package or not, with the understanding that this may resolve to an actual package or to the base system. pkgsrc/filesystems/fuse does link against perfuse and everything about this seems correct. There is very little clarity about multiple FUSE API versions and the extent to which both libfuse and librefuse do or don't support this, and what various fuse-foo programs do. To make progress, someone is going to have to actually analyze this by reading header files and documentation. (It seems obvious that librefuse should have at least the level of compat that FUSE does.) Related, I've been hearing about the FUSE 3.0 API, but fuse in pkgsrc is at 2.9.3. It appears to have API level 26. So open questions requiring careful and precise analysis are: What's going on with mmap on NetBSD with FUSE, in both the libfuse/perfuse case, and the librefuse case? What is the situation with (upstream) libfuse API versions? How do librefuse versions related to libfuse versions? What is the situation with various filesystems in terms of which API version they use? What does upstream libfuse say about this? Some notes: If I flip fuse-encfs's Makefile to use filesystems/fuse instead of fuse.buildlink3.mk, it builds, and works. But git fails with a complaint about mmap.
Re: Propose to link fuse-encfs against pkg libperfuse [Was Re: mmap errors related to using encfs]
On Sun, Apr 14, 2019 at 12:30:24PM -0400, Greg Troxel wrote: > Opinions differ about whether libfuse/perfuse and librefuse should be > the standard approach. For a very long time, librefuse has been the > standard approach. So obviously just changing to use libfuse is out of > the question. > > Adding some kind of preference variable so that people who wish to have > all packages that need the FUSE high-level API use libfuse/perfuse > instead (leaving the default) seems fine; that's similar to how we let > alternative implementations of other things be selected. I suppose you meant perfuse is for low-level APIs, or did I miss something? So if I understand things, the options are these: fuse-* to by default link with librefuse. [I do not know technical reasons behind this preference, but that apart. Users have to be aware that they won't get some features of the filesystem e.g. mmap, instead of them discovering this the hard way]. fuse-* to provide an option to link with pkg fuse, for those who want low level APIs. Most fuse-* patches would need to become conditional. Going by encfs as an example, most patches may not be needed if you choose pkg fuse. [I think since pkg fuse layer exists, where Maya has got all patch upstreaming done already, probably it's not productive to chase upstream for fuse-* patches that are nb specific. That's just my 2 cents.] pkg fuse need not link with librefuse as fuse-* can link with it directly. [This I am not 100% sure about. Just seeking views.] pkg fuse may have an option to link with pkg libperfuse or base libperfuse. Or may be instead of user choosing this, it can be decided by the following logic: As confirmed by Emmanuel, up to NetBSD 8.0 it needs to link with pkg libperfuse only. For NetBSD>8.0 base libperfuse should be usable. Please let me know if above analysis is correct. Mayuresh
Re: Propose to link fuse-encfs against pkg libperfuse [Was Re: mmap errors related to using encfs]
Mayuresh writes: > 1. in mk/fuse.buildlink3.mk we should include > filesystems/fuse/buildlink3.mk for NetBSD as well. Alternatively provide a > choice to do so. > > [I have changed the code in an ad hoc manner right now. If the proposal is > ok I'll try to do it neater and submit a patch. But I am not so conversant > with changing mk/ files, do's and dont's of doing so etc.] Opinions differ about whether libfuse/perfuse and librefuse should be the standard approach. For a very long time, librefuse has been the standard approach. So obviously just changing to use libfuse is out of the question. Adding some kind of preference variable so that people who wish to have all packages that need the FUSE high-level API use libfuse/perfuse instead (leaving the default) seems fine; that's similar to how we let alternative implementations of other things be selected. > 2. Two of the patches in fuse-encfs (1 added last week and 1 older) seem > to be no more required if we link with filesystems/fuse: > patch-encfs_FileUtils.cpp patch-encfs_main.cpp > > Not sure about the third one: patches/patch-encfs_DirNode.cpp. (Will > test without it and confirm.) These patches should perhaps be addressed upstream, and perhaps librefuse should be updated to be consistent with upstream FUSE. That's going to take some careful reading of code and specs and patches.
Re: Propose to link fuse-encfs against pkg libperfuse [Was Re: mmap errors related to using encfs]
Mayuresh writes: > 1. in mk/fuse.buildlink3.mk we should include > filesystems/fuse/buildlink3.mk for NetBSD as well. Alternatively provide a > choice to do so. > > [I have changed the code in an ad hoc manner right now. If the proposal is > ok I'll try to do it neater and submit a patch. But I am not so conversant > with changing mk/ files, do's and dont's of doing so etc.] Opinions differ about whether libfuse/perfuse and librefuse should be the standard approach. For a very long time, librefuse has been the standard approach. So obviously just changing to use libfuse is out of the question. Adding some kind of preference variable so that people who wish to have all packages that need the FUSE high-level API use libfuse/perfuse instead (leaving the default) seems fine; that's similar to how we let alternative implementations of other things be selected. > 2. Two of the patches in fuse-encfs (1 added last week and 1 older) seem > to be no more required if we link with filesystems/fuse: > patch-encfs_FileUtils.cpp patch-encfs_main.cpp > > Not sure about the third one: patches/patch-encfs_DirNode.cpp. (Will > test without it and confirm.) These patches should perhaps be addressed upstream, and perhaps librefuse should be updated to be consistent with upstream FUSE. That's going to take some careful reading of code and specs and patches.
Re: Propose to link fuse-encfs against pkg libperfuse [Was Re: mmap errors related to using encfs]
On Sun, Apr 14, 2019 at 10:48:51AM +0530, Mayuresh wrote: > 3. We might as well take this opportunity to encfs-1.9.5. I have already > confirmed the results on the list. Missing words: upgrade to. Mayuresh
Propose to link fuse-encfs against pkg libperfuse [Was Re: mmap errors related to using encfs]
On Sat, Apr 13, 2019 at 12:31:26AM +0530, Mayuresh wrote: > Is it possible that the issue / limitation is with librefuse? (And in that > case shall we be trying using filesystems/fuse? I pursued above point further. I took offline help from Emmanuel Dreyfus. Thanks Emmanuel for your inputs. For reference, this old post gives the genesis of libperfuse: https://mail-index.netbsd.org/tech-userlevel/2010/08/22/msg003843.html Indeed the low level interface mentioned in the post is needed by some filesystems. E.g. git wasn't working on encfs mounted dir due to unsupported mmap. But if we link with libperfuse it is found working now. Would also mention that linking to base libperfuse isn't helping as (at least as of 8.0) it is not up to date. pkg's libperfuse works fine though. (current should work fine, though I think better to link with pkg). I propose the following: 1. in mk/fuse.buildlink3.mk we should include filesystems/fuse/buildlink3.mk for NetBSD as well. Alternatively provide a choice to do so. [I have changed the code in an ad hoc manner right now. If the proposal is ok I'll try to do it neater and submit a patch. But I am not so conversant with changing mk/ files, do's and dont's of doing so etc.] 2. Two of the patches in fuse-encfs (1 added last week and 1 older) seem to be no more required if we link with filesystems/fuse: patch-encfs_FileUtils.cpp patch-encfs_main.cpp Not sure about the third one: patches/patch-encfs_DirNode.cpp. (Will test without it and confirm.) 3. We might as well take this opportunity to encfs-1.9.5. I have already confirmed the results on the list. Request your comments. Mayuresh
Re: mmap errors related to using encfs
On Sat, Apr 13, 2019 at 01:29:26PM -, Michael van Elst wrote: > Yes. You need to increase kern.sbmax. Thanks. Yes, that solved this issue. But there is io error occurring repeatedly. E.g. if I do ls in mounted dir, it works once, but second time around just shows io error. /var/log/messages shows following kinds of errors repeatedly when io error mentioned above occurs: encfs[1434]: ERROR withFileNode: error caught in fgetattr: fh=139135124959488 not found in fuseFhMap Mayuresh
Re: mmap errors related to using encfs
mayur...@acm.org (Mayuresh) writes: >Env var, something like, PERFUSE_BUFSIZE=131072 solves above error. >(Larger than that doesn't work. May be some sysctl setting would make it >work.) Yes. You need to increase kern.sbmax. -- -- Michael van Elst Internet: mlel...@serpens.de "A potential Snark may lurk in every tree."
Re: mmap errors related to using encfs
On Sat, Apr 13, 2019 at 09:24:17AM +0530, Mayuresh wrote: > encfs: perfuse_open: setsockopt SO_SNDBUF to 2162688 failed: No buffer space > available Env var, something like, PERFUSE_BUFSIZE=131072 solves above error. (Larger than that doesn't work. May be some sysctl setting would make it work.) However now I get input/output error on the mounted encfs. E.g. # ls ls: .: Input/output error Usually first time after mounting, ls works, 2nd command onwards it stops working. Mayuresh
Re: mmap errors related to using encfs
On Sat, Apr 13, 2019 at 12:31:26AM +0530, Mayuresh wrote: > Is it possible that the issue / limitation is with librefuse? (And in that > case shall we be trying using filesystems/fuse? Just giving it a try. Was able to build filesystems/fuse-encfs against filesystems/fuse. # ldd /usr/pkg/bin/encfs /usr/pkg/bin/encfs: -lfuse.2 => /usr/pkg/lib/libfuse.so.2 -lperfuse.0 => /usr/lib/libperfuse.so.0 -lpuffs.2 => /usr/lib/libpuffs.so.2 -lc.12 => /usr/lib/libc.so.12 -lpthread.1 => /usr/lib/libpthread.so.1 -lssl.12 => /usr/lib/libssl.so.12 -lcrypto.12 => /usr/lib/libcrypto.so.12 -lcrypt.1 => /lib/libcrypt.so.1 -lintl.1 => /usr/lib/libintl.so.1 -lstdc++.8 => /usr/lib/libstdc++.so.8 -lm.0 => /usr/lib/libm.so.0 -lgcc_s.1 => /usr/lib/libgcc_s.so.1 When I run it I get these errors: encfs: perfuse_open: setsockopt SO_SNDBUF to 2162688 failed: No buffer space available encfs: perfuse_open: setsockopt SO_RCVBUF to 2162688 failed: No buffer space available encfs: perfuse_open: setsockopt SO_SNDBUF to 2162688 failed: No buffer space available encfs: perfuse_open: setsockopt SO_RCVBUF to 2162688 failed: No buffer space available encfs: perfuse_open: setsockopt SO_SNDBUF to 2162688 failed: No buffer space available encfs: perfuse_open: setsockopt SO_RCVBUF to 2162688 failed: No buffer space available Will investigate by looking at code, but I have no other background. Sharing so that I get help/inputs as well. Mayuresh
Re: mmap errors related to using encfs
On Fri, Apr 12, 2019 at 12:14:08PM -0400, Greg Troxel wrote: > The following hints may seem obvious, but because you didn't report on > your investigations into the source code of encfs and git, I'll send > them anyway :-) Thanks. These are useful. In encfs I did not yet find any clue, but it might be alright for it to not support mmap, as you suggest. I'll try it out on Linux to see what happens. In git code I have not found it backtracking if mmap fails, but I do not know whether that's a conscious choice or not. I'll log a bug upstream. BTW it limits utility of encfs for me, as the issue is not just git. I was planning to use encfs for various kinds of files, applications of some of which might run into mmap issue. Mayuresh
mmap errors related to using encfs
Trying out filesystems/fuse-encfs on NetBSD 8.0 amd64. # mount | grep puff /dev/puffs on /mnt/foo type puffs|refuse:encfs (nodev, nosuid) # cd /mnt/foo/ # git init --bare error: unable to mmap '/mnt/foo/config': Operation not supported fatal: could not set 'core.filemode' to 'true' This problem occurs with encfs mounted directory only. From the error message it does not look like a git problem, rather some interaction between mmap and encfs. Would be interesting to know what's happening and is it possible to deal with this error. Mayuresh
