Re: CTF: wpa_supplicant/hostapd 2.4 import
On 4/26/15, Oliver Pinter oliver.pin...@hardenedbsd.org wrote: On 4/23/15, Oliver Pinter oliver.pin...@hardenedbsd.org wrote: Rui, FYI: http://w1.fi/security/2015-1/wpa_supplicant-p2p-ssid-overflow.txt On Sun, Apr 19, 2015 at 9:41 PM, Rui Paulo rpa...@me.com wrote: Hi, Please test the new wpa_supplicant/hostapd. Here's the patch against FreeBSD HEAD: https://people.freebsd.org/~rpaulo/wpa-2.4.diff Thanks, Hi Rui! Could you please cherry-pick / merge this commit from DragonflyBSD: https://github.com/DragonFlyBSD/DragonFlyBSD/commit/584c4a9f0c9071cb62abe9c870a2b08afe746a88 ? This fixes the CVE-2015-1863 . And this is the original commit: http://w1.fi/cgit/hostap/commit/src/p2p/p2p.c?id=9ed4eee345f85e3025c33c6e20aa25696e341ccd -- Rui Paulo ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Call For Testers: Synaptics touchpads
Woops. I missed the thread hierarchy. :-( If you're using new-enough synaptics, MFS of r281708 and r281709 in stable/10 would be sufficient. If you're using semi-MT one as mentioned by Jan Kokem〓ller (like me), you should additionally need applying his patch. Clarifying a bit more about my case, without Jan's patch, TrackPad (synaptics variant) works except two finger scrolling. Enabling/disabling TrackPad via hw.psm.synaptics.touchpad_off works OK. With Jan's patch, two finger scrolling start working for me, but vertical only. (Maybe r281708 and r281709 would be same.) Mouse cursor issue I mentioned below occurs with- and without- Jan's patch. On Sun, 26 Apr 2015 08:31:34 +0900 Tomoaki AOKI junch...@dec.sakura.ne.jp wrote: On Sat, 25 Apr 2015 13:26:09 -0700 (MST) r00ster srid...@outlook.com wrote: Jan, Thanks for your work! Running 'mostly' OK for me. (See below) Without your patch, two finger scrolling didn't work for me. r00ster, Applicable to stable/10 amd64 at r281732 and r281981, and running for me, but I haven't tried release/10.* and releng/10.* branch. You should need applying diffs for r281708 and r281709 in stable/10 manually, as these are prerequisite for Jan's patch. https://lists.freebsd.org/pipermail/svn-src-stable-10/2015-April/005163.html https://lists.freebsd.org/pipermail/svn-src-stable-10/2015-April/005164.html Working mostly good for me on my ThinkPad T420, but mouse cursor runs fast to right if I touch anywhere on right edge (about 5 - 10% of the pad). Other edges are OK. Unfortunately, I haven't determined why (hardware? software?) and how to fix it. Do you know if this patch can be applied to 10.1 RELEASE? -- View this message in context: http://freebsd.1045724.n5.nabble.com/Call-For-Testers-Synaptics-touchpads-tp6003555p6007628.html Sent from the freebsd-current mailing list archive at Nabble.com. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org -- Tomoaki AOKIjunch...@dec.sakura.ne.jp ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: readdir/telldir/seekdir problem (i think)
Julian Elischer wrote: On 4/25/15 5:52 AM, Jilles Tjoelker wrote: On Fri, Apr 24, 2015 at 04:28:12PM -0400, John Baldwin wrote: Yes, this isn't at all safe. There's no guarantee whatsoever that the offset on the directory fd that isn't something returned by getdirentries has any meaning. In particular, the size of the directory entry in a random filesystem might be a different size than the structure returned by getdirentries (since it converts things into a FS-independent format). This might work for UFS by accident, but this is probably why ZFS doesn't work. I took a quick look at UFS, ZFS and NFS: - UFS appears to use the offset as a block + offset within the block, so it is understandable that it works for the simple seek to byte offset case. - One oddity here is that the loop that copies out directory entries (about line#2188 in ufs_vnops.c) skips over d_ino == 0. Then uiomove() copies the data out and offset is updated. -- Seems to me that uio_offset skips entries with d_ino == 0, so it isn't the actual byte offset in the on-disk directory. *** Somehow, this seems it would break seekdir/telldir at some point? (This handling of uio_offset seems just plain broken to me?) - ZFS appears to number directory entries 1, 2, 3, 4, ... But, except for 1, 2, 3 (which are somewhat special, indicating ., .. and one other I've already forgotten, maybe .zfs), the rest are serialized, which seems to mean that the byte offset is kept in a list along with some hash function used to find them. -- I didn't really follow the code, but it appears to offset 28, so I don't think the low order 28bits are used to get the directory block. As such, I'm not too surprised that adding in the byte position within the block when doing lseek() doesn't break it. (I didn't really follow the stuff in zap_micro.c, so I could be all wrong here.) - I don't know if removal of an entry changes these indices 3, 4, ...? - NFS gets the block and mostly ignores the byte position within the block. (ie. It does uio_offset / NFS_DIRBLKSIZ and then uses that to get a buffer cache block. This will, in turn, look up this value to find a cached cookie.) -- The only effect of uio_offset % NFS_DIRBLKSIZ != 0 is that it will return data from this logical position within a block for the readdir. -- If a Readdir RPC is done for the cached cookie after a Remove, it is up to the NFS server's implementation w.r.t. what you'll get back. Not sure if the above is actually useful, at least until all file systems are evaluated, rick. However, this might be properly fixed by the thing that ino64 is doing where each directory entry returned by getdirentries gives you a seek offset that you _can_ directly seek to (as opposed to seeking to the start of the block and then walking forward N entries until you get an inter-block entry that is the same). The ino64 branch only reserves space for d_off and does not use it in any way. This is appropriate since actually using d_off is a major feature addition. A proper d_off would still be useful even if UFS's readdir keeps masking off the offset so a directory read always starts at the beginning of a 512-byte directory block, since this allows more distinct offset values than safely using getdirentries()'s *basep. With d_off, one outer loop must read at least one directory block to avoid spinning indefinitely, while using getdirentries()'s *basep requires reading the whole getdirentries() buffer. Some Linux filesystems go further and provide a unique d_off for each entry. Another idea would be to store the last d_ino instead of dd_loc into the struct ddloc. On seekdir(), this would seek to loc_seek as before and skip entries until that d_ino is found, or to the start of the buffer if not found (and possibly return some entries again that should not be returned, but Samba copes with that). yes.. though maybe a hash of hte inode number and the name may be a better thing to search on.. I need to check on your statement about samba to see if its true for 3.6.. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Call For Testers: Synaptics touchpads
By the way, just to clarify my question, what I'm asking is if it's possible for me to put the relevant additions from this patch for two-finger scrolling into 10.1. This is a feature I've really been looking forward to, so thank you very much for the work you put into it. :) -- View this message in context: http://freebsd.1045724.n5.nabble.com/Call-For-Testers-Synaptics-touchpads-tp6003555p6007638.html Sent from the freebsd-current mailing list archive at Nabble.com. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: readdir/telldir/seekdir problem (i think)
Julian Elischer wrote: On 4/25/15 9:39 AM, Rick Macklem wrote: Jilles Tjoelker wrote: On Fri, Apr 24, 2015 at 04:28:12PM -0400, John Baldwin wrote: Yes, this isn't at all safe. There's no guarantee whatsoever that the offset on the directory fd that isn't something returned by getdirentries has any meaning. In particular, the size of the directory entry in a random filesystem might be a different size than the structure returned by getdirentries (since it converts things into a FS-independent format). This might work for UFS by accident, but this is probably why ZFS doesn't work. However, this might be properly fixed by the thing that ino64 is doing where each directory entry returned by getdirentries gives you a seek offset that you _can_ directly seek to (as opposed to seeking to the start of the block and then walking forward N entries until you get an inter-block entry that is the same). The ino64 branch only reserves space for d_off and does not use it in any way. This is appropriate since actually using d_off is a major feature addition. Well, at some point ino64 will need to define a new getdirentries(2) syscall and I believe this new syscall can have different/additional arguments. yes, posix only specifies 2 mandatory fields (d_ino and d_name) and everything else is implementation dependent. I'd suggest that the new gtedirentries(2) syscall should return a flag to indicate that the underlying file system is filling in d_off. Then the libc functions can use d_off if it it available. (They will still need to work at least as well as they do now if the file system doesn't support d_off. The old getdirentries(2) syscall will be returning the old/current struct dirent which doesn't have the field anyhow.) Another bit of fun is that the argument for seekdir()/telldir() is a long and ends up 32bits for some arches. d_off is 64bits, since that is what some file systems require. Now that I've taken a look at the libc code, I can see that loc_XX structures handle the telldir/seekdir stuff (mapping the seek position plus offset within a block to an index). I think d_off could be saved there as well (if the file system provides it) and the new getdirentries() could take a physical offset argument where this d_off could be passed down to the file system. I think this would allow the problem to be handled properly in general. It still doesn't quite solve the NFS case. - For NFS, a server is supposed to have a directory_cookie_verifier, which allows the server to decide if a directory offset cookie is still valid. -- This has never been implemented for the FreeBSD server and it isn't easy. In part, real NFS servers don't do what the RFC describes, at least not in an obvious way. They understand if offset cookies still work after removal of an entry and also must store this directory_offset_verifier in the file's attributes. FreeBSD just never checks this verifier. rick what does linux use? -- In glibc up to version 2.1.1, the return type of telldir() was off_t. POSIX.1-2001 specifies long, and this is the type used since glibc 2.1.2. also from the linux man page: this is interesting.. In early filesystems, the value returned by telldir() was a simple file offset within a directory. Modern filesystems use tree or hash structures, rather than flat tables, to represent directories. On such filesystems, the value returned by telldir() (and used internally by readdir(3)) is a cookie that is used by the implementation to derive a position within a directory. Application programs should treat this strictly as an opaque value, making no assumptions about its contents. -- but glibc uses the contents in a nonopaque (and possibly wrong) way itself in seekdir. . (not following their own advice.) Maybe the library code can only use d_off if it is a 64bit arch and the file system is filling it in. (Or maybe the library can keep track of 32-64bit mappings for the offsets. I haven't looked at the libc functions for a while, so I can't remember what they keep track of.) one supposes a 32 bit system would not have such large file systems on it.. (maybe?) rick A proper d_off would still be useful even if UFS's readdir keeps masking off the offset so a directory read always starts at the beginning of a 512-byte directory block, since this allows more distinct offset values than safely using getdirentries()'s *basep. With d_off, one outer loop must read at least one directory block to avoid spinning indefinitely, while using getdirentries()'s *basep requires reading the whole getdirentries() buffer. Some Linux filesystems go further and provide a unique d_off for each entry.
Re: Call For Testers: Synaptics touchpads
Do you know if this patch can be applied to 10.1 RELEASE? -- View this message in context: http://freebsd.1045724.n5.nabble.com/Call-For-Testers-Synaptics-touchpads-tp6003555p6007628.html Sent from the freebsd-current mailing list archive at Nabble.com. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: readdir/telldir/seekdir problem (i think)
Julian Elischer wrote: On 4/25/15 9:39 AM, Rick Macklem wrote: Jilles Tjoelker wrote: On Fri, Apr 24, 2015 at 04:28:12PM -0400, John Baldwin wrote: Yes, this isn't at all safe. There's no guarantee whatsoever that the offset on the directory fd that isn't something returned by getdirentries has any meaning. In particular, the size of the directory entry in a random filesystem might be a different size than the structure returned by getdirentries (since it converts things into a FS-independent format). This might work for UFS by accident, but this is probably why ZFS doesn't work. However, this might be properly fixed by the thing that ino64 is doing where each directory entry returned by getdirentries gives you a seek offset that you _can_ directly seek to (as opposed to seeking to the start of the block and then walking forward N entries until you get an inter-block entry that is the same). The ino64 branch only reserves space for d_off and does not use it in any way. This is appropriate since actually using d_off is a major feature addition. Well, at some point ino64 will need to define a new getdirentries(2) syscall and I believe this new syscall can have different/additional arguments. yes, posix only specifies 2 mandatory fields (d_ino and d_name) and everything else is implementation dependent. I'd suggest that the new gtedirentries(2) syscall should return a flag to indicate that the underlying file system is filling in d_off. Then the libc functions can use d_off if it it available. (They will still need to work at least as well as they do now if the file system doesn't support d_off. The old getdirentries(2) syscall will be returning the old/current struct dirent which doesn't have the field anyhow.) Another bit of fun is that the argument for seekdir()/telldir() is a long and ends up 32bits for some arches. d_off is 64bits, since that is what some file systems require. what does linux use? Btw, I found this: https://bugs.centos.org/view.php?id=5496 It appears that Linux has been having fun with this too, at least for NFS. I still think that reading the whole directory is the only way to fix NFS. (Unfortunately, they don't say how the Linux distros fixed it.;-) Have fun with it, rick -- In glibc up to version 2.1.1, the return type of telldir() was off_t. POSIX.1-2001 specifies long, and this is the type used since glibc 2.1.2. also from the linux man page: this is interesting.. In early filesystems, the value returned by telldir() was a simple file offset within a directory. Modern filesystems use tree or hash structures, rather than flat tables, to represent directories. On such filesystems, the value returned by telldir() (and used internally by readdir(3)) is a cookie that is used by the implementation to derive a position within a directory. Application programs should treat this strictly as an opaque value, making no assumptions about its contents. -- but glibc uses the contents in a nonopaque (and possibly wrong) way itself in seekdir. . (not following their own advice.) Maybe the library code can only use d_off if it is a 64bit arch and the file system is filling it in. (Or maybe the library can keep track of 32-64bit mappings for the offsets. I haven't looked at the libc functions for a while, so I can't remember what they keep track of.) one supposes a 32 bit system would not have such large file systems on it.. (maybe?) rick A proper d_off would still be useful even if UFS's readdir keeps masking off the offset so a directory read always starts at the beginning of a 512-byte directory block, since this allows more distinct offset values than safely using getdirentries()'s *basep. With d_off, one outer loop must read at least one directory block to avoid spinning indefinitely, while using getdirentries()'s *basep requires reading the whole getdirentries() buffer. Some Linux filesystems go further and provide a unique d_off for each entry. Another idea would be to store the last d_ino instead of dd_loc into the struct ddloc. On seekdir(), this would seek to loc_seek as before and skip entries until that d_ino is found, or to the start of the buffer if not found (and possibly return some entries again that should not be returned, but Samba copes with that). -- Jilles Tjoelker ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___ freebsd-current@freebsd.org mailing list
Re: Call For Testers: Synaptics touchpads
On Sat, 25 Apr 2015 13:26:09 -0700 (MST) r00ster srid...@outlook.com wrote: Jan, Thanks for your work! Running 'mostly' OK for me. (See below) Without your patch, two finger scrolling didn't work for me. r00ster, Applicable to stable/10 amd64 at r281732 and r281981, and running for me, but I haven't tried release/10.* and releng/10.* branch. You should need applying diffs for r281708 and r281709 in stable/10 manually, as these are prerequisite for Jan's patch. https://lists.freebsd.org/pipermail/svn-src-stable-10/2015-April/005163.html https://lists.freebsd.org/pipermail/svn-src-stable-10/2015-April/005164.html Working mostly good for me on my ThinkPad T420, but mouse cursor runs fast to right if I touch anywhere on right edge (about 5 - 10% of the pad). Other edges are OK. Unfortunately, I haven't determined why (hardware? software?) and how to fix it. Do you know if this patch can be applied to 10.1 RELEASE? -- View this message in context: http://freebsd.1045724.n5.nabble.com/Call-For-Testers-Synaptics-touchpads-tp6003555p6007628.html Sent from the freebsd-current mailing list archive at Nabble.com. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org -- 青木 知明 [Tomoaki AOKI] junch...@dec.sakura.ne.jp mxe02...@nifty.com ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: CTF: wpa_supplicant/hostapd 2.4 import
On 4/23/15, Oliver Pinter oliver.pin...@hardenedbsd.org wrote: Rui, FYI: http://w1.fi/security/2015-1/wpa_supplicant-p2p-ssid-overflow.txt On Sun, Apr 19, 2015 at 9:41 PM, Rui Paulo rpa...@me.com wrote: Hi, Please test the new wpa_supplicant/hostapd. Here's the patch against FreeBSD HEAD: https://people.freebsd.org/~rpaulo/wpa-2.4.diff Thanks, Hi Rui! Could you please cherry-pick / merge this commit from DragonflyBSD: https://github.com/DragonFlyBSD/DragonFlyBSD/commit/584c4a9f0c9071cb62abe9c870a2b08afe746a88 ? This fixes the CVE-2015-1863 . -- Rui Paulo ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [patch] fix add_bounce_page() for BUS_DMA_KEEP_PG_OFFSET
On 04/25/15 07:59, Hans Petter Selasky wrote: On 04/24/15 15:29, Ian Lepore wrote: On Fri, 2015-04-24 at 13:34 +0200, Svatopluk Kraus wrote: The add_bounce_page() can be called from _bus_dmamap_load_phys(). Client virtual address is zero (not valid) in that case. So, client physical address must be used to get an offset when BUS_DMA_KEEP_PG_OFFSET flag is set. Note that client physical address is always valid. Svatopluk Kraus The only user of BUS_DMA_KEEP_PG_OFFSET in the system is USB. If the flag didn't exist, we could do bouncing with a whole lot more efficiency, at least in the ARM world. I wonder if it would be possible to just eliminate the flag (and of course, USB's need for it, which I fully admit I don't understand even a little bit). Hi, USB (EHCI/UHCI/OHCI) needs this flag when computing scatter/gather lists. Please ensure it works like expected. The good news is XHCI needs it less than the EHCI. Ian: The patch looks good - will you commit it? The reason is that the EHCI/UHCI/OHCI doesn't have separate length fields for each data chunk and somtimes use the 0x1000 - (addr 0xFFF) formula to compute how many bytes are left of a transfer when a transfer wraps around 0x1000. Thank you for you understanding this is not a USB problem in general. --HPS ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: geli partition accepts passphrase but won't mount [SOLVED]
On Fri, Apr 24, 2015 at 11:52:49PM +0100, John wrote: Hello, I'm running 11.0-CURRENT #0 r281867. I've followed the instructions given at https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/disks-encrypting.html section 18.12.2. I was able to create the encrypted slice and mount it. I transferred some documents to that drive for safekeeping and unmounted it. Updated the machine and added these lines to the kernel: options GEOM_ELI device crypto but before I could rebuild the system later, it (some hours after this) went into a hard lock. Powering off then on again rebooted the system and from there I was able to run buildworld and friends. Rebooted again, now trying to mount the disk: root@:~ # geli attach -k /root/da0p1.key /dev/da0p1 Enter passphrase: root@:~ # root@:~ geli status Name Status Components da0p1.eli ACTIVE da0p1 Trying to mount it, this happens: root@:~ # mount /dev/da0p1.eli /mnt mount: /dev/da0p1.eli: Invalid argument I think this needs fsck but I get root@:~ # fsck -p -t ffs /dev/da0p1.eli Cannot find file system superblock What can I do? In the grand tradition of replying to my own question, thought I'd post what fixed this. For some reason, I had both a /dev/da0p1.key and a /dev/da0.key. Both were the same. Firstly tried restoring the metadata for /dev/da0p1.eli. Was able to attach but failed mount. Then had the idea of restoring from /var/backups/da0.eli. Then tried to attach da0.key but this time it gave a better error message that said unable to mount r/w because the filesystem was dirty. So I ran fsck_ffs -y /dev/da0.eli and from there (after fixing lots of incorrect block count errors) I was able to mount da0.eli and read/write to it. da0p1.eli I think was made because the initial format of the disk gave da0p1. But looking back at the time before this that the disk was successfully mounted (in the system log emails) the disk appeared as /dev/da0.eli. *whew!* -- John ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Merging GitHub Pull Requests into Subversion using git-svn
On 23 Apr 2015, at 00:12, Craig Rodrigues rodr...@freebsd.org wrote: While not as smooth as clicking a merge button in GitHub, this is a valid way to accept patches submitted via GitHub pull requests, and integrate them in our FreeBSD Subversion repo. The merge button on GitHub does the wrong thing anyway (merges without fast-forward, so you end up with a tangled history), so (after the initial setup) the steps that I use for merging pull requests from GitHub projects are very similar (locally pull the branch with fast-fordward, test, push). David ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: readdir/telldir/seekdir problem (i think)
Julian Elischer wrote: On 4/25/15 9:39 AM, Rick Macklem wrote: Jilles Tjoelker wrote: On Fri, Apr 24, 2015 at 04:28:12PM -0400, John Baldwin wrote: Yes, this isn't at all safe. There's no guarantee whatsoever that the offset on the directory fd that isn't something returned by getdirentries has any meaning. In particular, the size of the directory entry in a random filesystem might be a different size than the structure returned by getdirentries (since it converts things into a FS-independent format). This might work for UFS by accident, but this is probably why ZFS doesn't work. However, this might be properly fixed by the thing that ino64 is doing where each directory entry returned by getdirentries gives you a seek offset that you _can_ directly seek to (as opposed to seeking to the start of the block and then walking forward N entries until you get an inter-block entry that is the same). The ino64 branch only reserves space for d_off and does not use it in any way. This is appropriate since actually using d_off is a major feature addition. Well, at some point ino64 will need to define a new getdirentries(2) syscall and I believe this new syscall can have different/additional arguments. yes, posix only specifies 2 mandatory fields (d_ino and d_name) and everything else is implementation dependent. I'd suggest that the new gtedirentries(2) syscall should return a flag to indicate that the underlying file system is filling in d_off. Then the libc functions can use d_off if it it available. (They will still need to work at least as well as they do now if the file system doesn't support d_off. The old getdirentries(2) syscall will be returning the old/current struct dirent which doesn't have the field anyhow.) Another bit of fun is that the argument for seekdir()/telldir() is a long and ends up 32bits for some arches. d_off is 64bits, since that is what some file systems require. what does linux use? -- In glibc up to version 2.1.1, the return type of telldir() was off_t. POSIX.1-2001 specifies long, and this is the type used since glibc 2.1.2. also from the linux man page: this is interesting.. In early filesystems, the value returned by telldir() was a simple file offset within a directory. Modern filesystems use tree or hash structures, rather than flat tables, to represent directories. On such filesystems, the value returned by telldir() (and used internally by readdir(3)) is a cookie that is used by the implementation to derive a position within a directory. Application programs should treat this strictly as an opaque value, making no assumptions about its contents. -- but glibc uses the contents in a nonopaque (and possibly wrong) way itself in seekdir. . (not following their own advice.) I believe that most of the FreeBSD file systems except UFS and ZFS just copy the fields of their internal directory structure to fields in struct dirent, filling blocks sequentially. (Actually, I only took a quick look, but ZFS might also be this way.) As such, the offsets for FreeBSD are byte offsets into these logical directory blocks. The problem is (as already discussed) that there is no way to predict how these will change for a given file system when entries are removed/added. (I think the only way to know what the modified logical directory looks like is to read it again from the beginning, so that all the directory entries go through the conversion to logical again.) UFS and the NFS client ensure that no struct dirent crosses a 512byte block boundary. I don't think the other file systems do this. I mention this, since the libc functions can't assume the UFS behaviour for this. (At one time, UFS just consumed removed entries into the preceding struct direct or set it invalid, if it was the first entry in a 512byte block. This implied that the byte offsets (logical == physical) didn't change for subsequent entries upon a removal. It sounds like UFS is no longer doing this, from one of your posts?) I am curious to see what glibc does, since I had assumed it just read the entire directory at opendir/first-readdir. rick ps: This is what I recall from fooling with struct dirent a few months ago and I'm getting old, so it may all be wrong;-) Maybe the library code can only use d_off if it is a 64bit arch and the file system is filling it in. (Or maybe the library can keep track of 32-64bit mappings for the offsets. I haven't looked at the libc functions for a while, so I can't remember what they keep track of.) one supposes a 32 bit system would not have such large file systems on it.. (maybe?) rick A proper d_off would still be useful even if UFS's readdir keeps masking off the offset so a directory
Re: readdir/telldir/seekdir problem (i think)
Jilles Toelker wrote: On Fri, Apr 24, 2015 at 04:28:12PM -0400, John Baldwin wrote: Yes, this isn't at all safe. There's no guarantee whatsoever that the offset on the directory fd that isn't something returned by getdirentries has any meaning. In particular, the size of the directory entry in a random filesystem might be a different size than the structure returned by getdirentries (since it converts things into a FS-independent format). This might work for UFS by accident, but this is probably why ZFS doesn't work. However, this might be properly fixed by the thing that ino64 is doing where each directory entry returned by getdirentries gives you a seek offset that you _can_ directly seek to (as opposed to seeking to the start of the block and then walking forward N entries until you get an inter-block entry that is the same). The ino64 branch only reserves space for d_off and does not use it in any way. This is appropriate since actually using d_off is a major feature addition. Just fyi, I did a patch: http://people.freebsd.org/~rmacklem/64bitfileno.patch which does fill this field in for the various file systems and copies the new struct dirent to struct dirent32, so it works with an unchanged userland. Since the ino64 folks felt this should come after the ino64 change, I didn't pursue committing it. To commit it to head, it would have to be modified slightly, so that the new structure is called struct dirent64, so it doesn't screw up userland builds using struct dirent, but I think that is the only change required for it to go into head/current if people felt that was appropriate. (Oh, and it called d_off d_cookie, although I have no strong preference w.r.t. what it is called.) If you look at it, you'll see most of the changes are for NFS, but the rest were pretty trivial. rick A proper d_off would still be useful even if UFS's readdir keeps masking off the offset so a directory read always starts at the beginning of a 512-byte directory block, since this allows more distinct offset values than safely using getdirentries()'s *basep. With d_off, one outer loop must read at least one directory block to avoid spinning indefinitely, while using getdirentries()'s *basep requires reading the whole getdirentries() buffer. Some Linux filesystems go further and provide a unique d_off for each entry. Another idea would be to store the last d_ino instead of dd_loc into the struct ddloc. On seekdir(), this would seek to loc_seek as before and skip entries until that d_ino is found, or to the start of the buffer if not found (and possibly return some entries again that should not be returned, but Samba copes with that). -- Jilles Tjoelker ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: readdir/telldir/seekdir problem (i think)
Julian Elischer wrote:
On 4/25/15 4:28 AM, John Baldwin wrote:
On Saturday, April 25, 2015 02:36:24 AM Julian Elischer wrote:
On 4/25/15 1:30 AM, Julian Elischer wrote:
On 4/24/15 10:59 PM, John Baldwin wrote:
On Friday, April 24, 2015 01:02:39 PM Julian Elischer wrote:
On 4/23/15 9:54 PM, John Baldwin wrote:
On Thursday, April 23, 2015 05:02:08 PM Julian Elischer wrote:
On 4/23/15 11:20 AM, Julian Elischer wrote:
I'm debugging a problem being seen with samba 3.6.
basically telldir/seekdir/readdir don't seem to work as
advertised..
ok so it looks like readdir() (and friends) is totally broken
in
the face
of deletes unless you read the entire directory at once or
reset
to the
the first file before the deletes, or earlier.
I'm not sure that Samba isn't assuming non-portable behavior.
For example:
From
http://pubs.opengroup.org/onlinepubs/009695399/functions/readdir_r.html
If a file is removed from or added to the directory after the
most recent call
to opendir() or rewinddir(), whether a subsequent call to
readdir() returns an
entry for that file is unspecified.
While this doesn't speak directly to your case, it does note
that
you will
get inconsistencies if you scan a directory concurrent with
add
and remove.
UFS might kind of work actually since deletes do not compact
the
backing
directory, but I suspect NFS and ZFS would not work. In
addition, our
current NFS support for seekdir is pretty flaky and can't be
fixed without
changes to return the seek offset for each directory entry (I
believe that
the projects/ino64 patches include this since they are
breaking
the ABI of
the relevant structures already). The ABI breakage makes this
a
very
non-trivial task. However, even if you have that per-item
cookie, it is
likely meaningless in the face of filesystems that use any
sort
of more
advanced structure than an array (such as trees, etc.) to
store
directory
entries. POSIX specifically mentions this in the rationale
for
seekdir:
One of the perceived problems of implementation is that
returning
to a given point in a directory is quite difficult to describe
formally, in spite of its intuitive appeal, when systems that
use
B-trees, hashing functions, or other similar mechanisms to
order
their directories are considered. The definition of seekdir()
and
telldir() does not specify whether, when using these
interfaces,
a given directory entry will be seen at all, or more than
once.
In fact, given that quote, I would argue that what Samba is
doing is
non-portable. This would seem to indicate that a conforming
seekdir could
just change readdir to immediately return EOF until you call
rewinddir.
how does readdir know that the directory has been changed?
fstat?
Undefined. FreeBSD's libc doesn't cache the entire directory
(unless you
are using a union mount), instead it just caches the most recent
call to
getdirentries(). It then serves up entries from that block
until
you hit
the end. When you hit the end it reads the next block, etc.
When you
call telldir(), the kernel saves the seek offset from the start
of the
current block and the offset within that block and returns a
cookie to
you. seekdir() looks up the cookie to find the (seek offset,
block
offset)
tuple. If the location matches the directory's current location
it
doesn't
do anything, otherwise it seeks to the seek offset and reads a
new
block via
getdirentries(). There is no check for seeing if a directory is
changed. Instead, you can only be stale by one block. When
you
read
a new block it is relative to the directory's state at that
time.
Rick's suggestion of reusing the block for a seek within a block
would be
fairly easy to implement, I think it would just be:
Index: head/lib/libc/gen/telldir.c
===
--- head/lib/libc/gen/telldir.c (revision 281929)
+++ head/lib/libc/gen/telldir.c (working copy)
@@ -101,8 +101,10 @@
return;
if (lp-loc_loc == dirp-dd_loc lp-loc_seek ==
dirp-dd_seek)
return;
- (void) lseek(dirp-dd_fd, (off_t)lp-loc_seek,
SEEK_SET);
- dirp-dd_seek = lp-loc_seek;
+ if (lp-loc_seek != dirp-dd_seek) {
+ (void) lseek(dirp-dd_fd, (off_t)lp-loc_seek,
SEEK_SET);
+ dirp-dd_seek = lp-loc_seek;
+ }
yes I did that yesterday but it still fails when you transition
blocks.. (badly).
I also tried bigger blocks.. also fails (eventually)
I did find a way to make it work... you had to seek back
to the first block you deleted on each set..
then work forward from there again.. unfortunately since
I'm trying to make a microsoft program not fail (via samba)
I have no control over how it does things and
Re: Newer yacc needed for building world
On 24/04/2015 12:05, Garrett Cooper wrote:
On Apr 24, 2015, at 3:03, Garrett Cooper yaneurab...@gmail.com
wrote:
On Apr 24, 2015, at 2:59, Garrett Cooper yaneurab...@gmail.com
wrote:
On Apr 23, 2015, at 2:05, Willem Jan Withagen w...@digiware.nl
wrote:
On 22/04/2015 23:37, Ed Maste wrote:
On 22 April 2015 at 15:55, Willem Jan Withagen
w...@digiware.nl wrote:
Yes:
https://lists.freebsd.org/pipermail/freebsd-current/2015-February/054740.html
But this is not enough to make yacc part of the build tools??
yacc is unconditionally built during bootstrap-tools as of
r281615. What SVN rev is your tree?
# svn info Path: . Working Copy Root Path: /usr/src URL:
svn://svn.freebsd.org/base/stable/10 Relative URL: ^/stable/10
Repository Root: svn://svn.freebsd.org/base Repository UUID:
ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f Revision: 281853 Node
Kind: directory Schedule: normal Last Changed Author: kib Last
Changed Rev: 281849 Last Changed Date: 2015-04-22 12:59:05
+0200 (Wed, 22 Apr 2015)
Then I removed /usr/obj/* to get a fresh start. Removing yacc
just gets me into trouble even earlier:
# make -j 32 buildworld . . . ---
_bootstrap-tools-usr.bin/compile_et --- --- parse.c --- yacc -d
-o parse.c
/usr/src/usr.bin/compile_et/../../contrib/com_err/parse.y yacc:
not found --- _bootstrap-tools-usr.sbin/bsnmpd/gensnmptree ---
/usr/obj/usr/src/tmp/usr/src/usr.sbin/bsnmpd/gensnmptree
created for /usr/src/usr.sbin/bsnmpd/gensnmptree ---
_bootstrap-tools-usr.bin/compile_et --- *** [parse.c] Error
code 127
So I have relatively little further to test. Perhaps the '-j
32' was a bit aggressive, but it gets fast where the error is.
Well, that’s amusing :(. You found a new race that wasn’t present
before my changes to parallelize bootstrap-tools (kerberos comes
before yacc in bootstrap-tools). Do you have yacc installed on
your machine? Please try out this patch. Thanks! -NGie
$ svn diff Makefile.inc1 Index: Makefile.inc1
===
--- Makefile.inc1 (revision 281823)
+++ Makefile.inc1 (working copy) @@ -1358,6 +1358,8 @@
usr.bin/compile_et
.ORDER: ${_kerberos5_bootstrap_tools:C/^/${_bt}-/g} +
+${_bt}-usr.bin/compile_et: ${_bt}-usr.bin/yacc .endif
bootstrap-tools: .PHONY
It’ll also need lex too. This should be a bit more comprehensive:
Index: Makefile.inc1
===
--- Makefile.inc1 (revision 281823)
+++ Makefile.inc1 (working copy) @@ -1358,6 +1358,8 @@
usr.bin/compile_et
.ORDER: ${_kerberos5_bootstrap_tools:C/^/${_bt}-/g} +
+${_bt}-usr.bin/compile_et: ${_bt}$-usr.bin/lex
${_bt}-usr.bin/yacc .endif
bootstrap-tools: .PHONY
I’ll work out the finally kinks with how to spell lex and yacc…
This is part of the reason why I think BOOTSTRAPPING needs to be
kicked to the curb and everything needs to be built in parallel, but
enough people haven’t complained about built failures, so the
optimization remains..
Remember I moved /usr/bin/out of the path. Which would be different for
using an outdated yacc. Which is where the bug originally started.
This fixes the problem for a non-parallel build.
But once parallel builds (with -j 5 ) are done, the
.for _tool in \
loop runs into trouble when _yacc is not finished/started before config
is being build.
Normally this would not be a problem as long as config is not dependant
on any of the new features/size in the yacc to be build.
--WjW
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: readdir/telldir/seekdir problem (i think)
Julian Elischer wrote: On 4/25/15 9:39 AM, Rick Macklem wrote: Jilles Tjoelker wrote: On Fri, Apr 24, 2015 at 04:28:12PM -0400, John Baldwin wrote: Yes, this isn't at all safe. There's no guarantee whatsoever that the offset on the directory fd that isn't something returned by getdirentries has any meaning. In particular, the size of the directory entry in a random filesystem might be a different size than the structure returned by getdirentries (since it converts things into a FS-independent format). This might work for UFS by accident, but this is probably why ZFS doesn't work. However, this might be properly fixed by the thing that ino64 is doing where each directory entry returned by getdirentries gives you a seek offset that you _can_ directly seek to (as opposed to seeking to the start of the block and then walking forward N entries until you get an inter-block entry that is the same). The ino64 branch only reserves space for d_off and does not use it in any way. This is appropriate since actually using d_off is a major feature addition. Well, at some point ino64 will need to define a new getdirentries(2) syscall and I believe this new syscall can have different/additional arguments. yes, posix only specifies 2 mandatory fields (d_ino and d_name) and everything else is implementation dependent. I'd suggest that the new gtedirentries(2) syscall should return a flag to indicate that the underlying file system is filling in d_off. Then the libc functions can use d_off if it it available. (They will still need to work at least as well as they do now if the file system doesn't support d_off. The old getdirentries(2) syscall will be returning the old/current struct dirent which doesn't have the field anyhow.) Another bit of fun is that the argument for seekdir()/telldir() is a long and ends up 32bits for some arches. d_off is 64bits, since that is what some file systems require. what does linux use? -- In glibc up to version 2.1.1, the return type of telldir() was off_t. POSIX.1-2001 specifies long, and this is the type used since glibc 2.1.2. also from the linux man page: this is interesting.. In early filesystems, the value returned by telldir() was a simple file offset within a directory. Modern filesystems use tree or hash structures, rather than flat tables, to represent directories. On such filesystems, the value returned by telldir() (and used internally by readdir(3)) is a cookie that is used by the implementation to derive a position within a directory. Application programs should treat this strictly as an opaque value, making no assumptions about its contents. -- but glibc uses the contents in a nonopaque (and possibly wrong) way itself in seekdir. . (not following their own advice.) Maybe the library code can only use d_off if it is a 64bit arch and the file system is filling it in. (Or maybe the library can keep track of 32-64bit mappings for the offsets. I haven't looked at the libc functions for a while, so I can't remember what they keep track of.) one supposes a 32 bit system would not have such large file systems on it.. (maybe?) For NFS, the cookie is always an opaque 64bits. These cookies are cached in the kernel by the client, with one for each logical UFS-like directory block generated for getdirentries(2). As such, the NFS case does a 64bit-32bit mapping. (Because of endianness etc, there is no guarantee that most of these cookies are 0 for the high order 32bits.) I need to look at the library code (both glibc and ours) before I understand this better and can say more. Have fun with it, rick ps: The ino64 stuff will never be MFC'd, so it would be nice to improve what the libc functions do without use of d_off. rick A proper d_off would still be useful even if UFS's readdir keeps masking off the offset so a directory read always starts at the beginning of a 512-byte directory block, since this allows more distinct offset values than safely using getdirentries()'s *basep. With d_off, one outer loop must read at least one directory block to avoid spinning indefinitely, while using getdirentries()'s *basep requires reading the whole getdirentries() buffer. Some Linux filesystems go further and provide a unique d_off for each entry. Another idea would be to store the last d_ino instead of dd_loc into the struct ddloc. On seekdir(), this would seek to loc_seek as before and skip entries until that d_ino is found, or to the start of the buffer if not found (and possibly return some entries again that should not be returned, but Samba copes with that). -- Jilles Tjoelker ___
