Re: reiser4 plugins
Gregory Maxwell wrote: On 6/26/05, Lincoln Dale [EMAIL PROTECTED] wrote: the l-k community have asked YOU may times. any lack of response isn't because of the kernel cabal .. its because YOU are refusing to entertain any notion that what Reiser4 has implemented is unpalatable to the kernel community. A lot of this is based on misconceptions, for example in recent times reiser4 is faulted for layering violations.. But it doesn't have them, it neither duplicates nor modifies the VFS. It has also been requested that reiser4 be changed to move some of it's operations above the VFS... not only would that not make sense for the currently provided functions, but merging was put off previously because of changes to the VFS now that it doesn't change the VFS we are asking hans to push it off until it does?? sigh it has NEVER been a case of Reiser4 not being merged because it required changes to VFS. the whole point of VFS is to provide a standard API for data to/from individual filesystems. over the course of history, VFS itself hasn't been a static thing - it has had to adapt and change as a result of the needs of filesystems. but it hasn't ever been a case of individual filesystems doing 'proprietary' things (i.e. there isn't a sys_ext3() system call) - where it has made sense to do things at a VFS layer, VFS itself has been adapted to handle those things. a semi-recent example of this is extended attributes. it is with some irony that on my desktop i make use of the excellent open-source desktop search tool 'beagle'. it, however, uses extended attributes for storing things - and Reiser4's EAs are incompatible with the standard EAs such that Reiser4 is incompatible with beagle. this is the WHOLE point of standardization .. i don't think its that Reiser4's EAs offer any more or less capabilities than standard EAs - BUT they haven't used the standard mechanisms available for implementing them, such for Beagle to work on Reiser4, there now needs to be logic added to Beagle to do so. lets take this a step further. what about compression? do we accept that each filesystem can implement its own proprietary compression via its own API - and now we need individual user-space tools to understand each of these APIs? how about encryption? ... and so-on. suddenly every user app out there needs to have specialized knowledge of each type of filesystem. Hans should be applauded for the 'plug-in' concept and showing how it can be used. however, from an implementation stand-point, it really shouldn't come as any great surprise that numerous kernel developers are pushing back saying 'layering violation' and why can't this be done at the VFS layer. none of this is rocket-science. its just plain common sense. It's a filesysem for gods sake. Hans and his team have worked hard to minimize its impact and they are still willing to accept more guidance, i don't see any acceptance at this point. simply lots of hot air that smells like marketing PR. cheers, lincoln.
Re: reiser4 plugins
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Lincoln Dale wrote: [...] this is the WHOLE point of standardization .. i don't think its that Reiser4's EAs offer any more or less capabilities than standard EAs - They do. Reiser4's EAs can look like any other object -- files, folders, symlinks, whatever. This is important, especially for transparency. For one thing, can I access a Beagle search as a folder? BUT they haven't used the standard mechanisms available for implementing them, such for Beagle to work on Reiser4, there now needs to be logic added to Beagle to do so. Well, ideally, I'd like to see people stop bickering, come up with something better than sys_reiser4, add an emulation layer for xattrs and mark them obsolete. But I don't speak for Namesys. lets take this a step further. what about compression? do we accept that each filesystem can implement its own proprietary compression via its own API - and now we need individual user-space tools to understand No, that's the beauty of these EAs in Reiser4. The API is standard write(2) commands. sys_reiser4 supposedly implements an interface to make this scale better, but otherwise have the same semantics. And who said anything about proprietary compression? I think we were planning on the kernel's zlib, though we might have been planning to make it a bit more seekable... each of these APIs? So, the API becomes something like: cat crypto/inflated/foo # transparently decompressed cat crypto/raw/foo.gz # raw, gzip-compressed Another possibility, if you like file-as-a-directory: cat foo.gz # raw cat foo.gz/inflated # decompressed One could easily imagine things like these two potentially equivalent commands: cp foo bar.zip/ zip bar foo The whole point is to have less userland tools, not more. I'm not saying we move zip into the kernel, just that the user now has one less command to remember. But, back to reality. file-as-directory probably won't happen, at least not for awhile, so imagine more along the lines of my first example. how about encryption? About the same, only now you have a key file that you write to in order to unlock the decrypted files. ... and so-on. suddenly every user app out there needs to have specialized knowledge of each type of filesystem. Not really. More like, every app that cares to has generalized knowledge, if that. none of this is rocket-science. its just plain common sense. I could say the same of my stuff, but lots of people seem to disagree with me, or at least fail to see it. I guess I can say the same of your comments. It's a filesysem for gods sake. Hans and his team have worked hard to minimize its impact and they are still willing to accept more guidance, i don't see any acceptance at this point. simply lots of hot air that smells like marketing PR. They do keep asking for specifically what they need to do to put this stuff in VFS. Or am I wrong? Or maybe it should be obvious to them? -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQr5dtngHNmZLgCUhAQIBGxAAiLK6EyHnLRhEA+rUIDCwacM4K89wlE7X +dcw3xv3Pc9tZZqVVAd7Y27whEzjmNOwfGkvPkzPk/ATQditnyt+7xHcuXpqORNU j7zHc5zS8MGDRU8Re4MXTO6jCXDgtTwQHjcdg4i8KYWLMPT7LpO+DHY/mZyQEgpD kZGE4WJePA+aNlHAzySW9u/atnwp5hSRvmfuF4zzN8ng5tf8SSMvbfoCyjYSue8l N6jvcGnt+yItmbHVaij0IdHUw1/9/u6b3Q0Ut39NBk8fUKXJcASHmKtjwLTAoWW+ hiYVhLdZQGkWo2d6XdzdNY2OgE3kWVnLBqrOuTo7zCjMojvWIrEGE/x3Yh/E6Hs8 cAPVRebG5yUQBxJk1lcDeJOBozutIpCVyTzwBnKU1nz3KArqanU51oT++3cTjVha 1tBnaLS4RLcdy8UD1ewS+VHj61VSlcBjv2abCrYw4DC0anUFrYUSciNjx3tdYJRx o7l/pEn7UYPpGaXgHyBdVDIRlRNdOoTRZp3aIY2Z2v6/jyi3TeufMUjGtpuQHl2k BuYm7tV4l1Ec/QZLM+PbAyVU9qqlz9BuHlI1U7z1p3gYeAzz0guAeDHfi1l95sUn l+bFCOfmXi3qRAxVZyidczqeOtGtCed3nIUH+1z+siuFzH3jecjzUpGWKcGxzVlc jkUS+tlihfg= =C4UP -END PGP SIGNATURE-
Re: reiser4 plugins
David Masover wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Lincoln Dale wrote: [...] this is the WHOLE point of standardization .. i don't think its that Reiser4's EAs offer any more or less capabilities than standard EAs - They do. Reiser4's EAs can look like any other object -- files, folders, symlinks, whatever. This is important, especially for transparency. it was accepted not so long ago that 'file-as-directory' and 'EA' are two different things, predominantly because existing tools and apps won't necessarily do the right thingtm. this has been discussed to death previously. oh what a surprise. http://lwn.net/Articles/100271/ cheers, lincoln. cheers, lincoln.
Re: reiser4 plugins
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Lincoln Dale wrote: David Masover wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Lincoln Dale wrote: [...] this is the WHOLE point of standardization .. i don't think its that Reiser4's EAs offer any more or less capabilities than standard EAs - They do. Reiser4's EAs can look like any other object -- files, folders, symlinks, whatever. This is important, especially for transparency. it was accepted not so long ago that 'file-as-directory' and 'EA' are two different things, predominantly because existing tools and apps won't necessarily do the right thingtm. They are different, but not quite -- what's the word -- discrete? Rather, file-as-directory is one logical conclusion of EA, and one possible representation of EA. Anyway, Reiser's EAs currently (I think) could appear as files, but not necessarily the file-as-directory madness we were talking about then. They could also be accessed by openat. And they can also be hierarchical, which I would think makes them cleaner and faster. this has been discussed to death previously. oh what a surprise. http://lwn.net/Articles/100271/ I was there. That was fun to watch, actually, once it got over my head. The three points he lists are only relevent if you try to do file-as-directory. Right now, I like the idea of file attributes as showing up in separate magic directories, or just creating magic directories first and filling them with files. Symlinks could help clean up messiness, and allow you to do things like encrypt /etc/passwd - -- symlink to /etc/crypto/passwd/decrypted or something. Yes, I know about shadow, and there are still some paranoid reasons to encrypt passwd. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQr521XgHNmZLgCUhAQKSSA/9HK8JiG6wW40jQDUXsyKPz+l9UvoaNuLs hqIXo3sdQj1CWAuwa1BKY16w91tJBN3uX75jLgwR36ix4A1jXQ5v1sRjvfkLKR/7 3a51v9UsRAhAisiqWFGWYTrbXgAh+S5B51xHuXv3qTs/wqC8sxuJtAHwuCldPaBr cR1WzPtGvgd+ESqx5avllZIfNCy9tcH3P5fUsaIFCCm30E6u9PVO6xBOHylFWZKS Pywv+wGUTxbgCFSmLG07/zhwVF94fAHPIWXQXQGmhPrGf3Wtt7VTMiRpkpyONyso eoY1hFiwyh2YMrIPxdL0Uo+mcDvErWFFZyTcCGqIMp6x+QSe0Ww9Ie90afZPvvS0 Q4DmdST2JEHEinal5MCiqr3S83wanv3+h9ywTCEkTcl3mEK1iwtc4dwmvRNVHfkx f34CAxM1rBfu++kd3xgL+Kb/ao4LOCDhVL3XY6pNDglX6Y+Kk5dRSJjOm4kAU2fB j66uGOUZOiCCzSMLvVrCAPnNXmUavLKUXDKXKL2gTCYM6TL+7RkAXEMrqu5YxdZM ihIbfmGc7FzItbQCDZhhozG51IMkLCotU9U9CaotnhfUaosibmwEnmeY8FWY75ba MOs+VH3UJUZObRBwmBHSX4pwg5sDhSjyqPR05M26A5xz8nGh4kukD/E6QZYCB9EP lVdDus4iJUM= =hsOJ -END PGP SIGNATURE-
kernel BUG at fs/reiserfs/journal.c:1313!
This is a report of a bug filed with gentoo a few weeks ago.
http://bugs.gentoo.org/show_bug.cgi?id=95043
Below is the contents of this bug. The kernel that was in place is no more, so no further testing
can be done on this particular machine.
Rob Baxter
Description: [reply]Opened: 2005-06-04 11:03 PDT
Load shot up on the server, file system was useless (could not read or write to any partition) until
reboot. I'm upgrading the kernel now to r9, but I thought this may be of use. Let me know if you
need more info.
[ cut here ]
kernel BUG at fs/reiserfs/journal.c:1313!
invalid operand: [#1]
PREEMPT SMP
Modules linked in: iptable_filter
CPU:0
EIP:0060:[c01b7ae0]Not tainted VLI
EFLAGS: 00010202 (2.6.11-gentoo-r6)
EIP is at flush_journal_list+0x580/0x640
eax: 0001 ebx: f8bac1a0 ecx: 0001 edx:
esi: ceb851bc edi: ebp: c48be880 esp: d6607db0
ds: 007b es: 007b ss: 0068
Process pdflush (pid: 32240, threadinfo=d6606000 task=d846b020)
Stack: f8bac1a0 c48be880 0001 c27f7ce0 c015c07a d6befefc f8816000
004c c48be880 01ec 0100 e390c080 c01b7f48
f7ca3000 c48be880 0001 d6607e10 01ec 0003 f88160f4 c48bea00
Call Trace:
[c015c07a] bh_lru_install+0xba/0xf0
[c01b7f48] flush_used_journal_lists+0xd8/0xf0
[c01bb5a4] flush_old_journal_lists+0x44/0x60
[c01bbba2] do_journal_end+0x5e2/0xa20
[c01ba085] do_journal_begin_r+0x25/0x2d0
[c01407d0] pdflush+0x0/0x30
[c01babd5] journal_end_sync+0x55/0xa0
[c01a7c69] reiserfs_sync_fs+0x69/0x80
[c011733c] finish_task_switch+0x3c/0x90
[c01a7c98] reiserfs_write_super+0x18/0x20
[c0160163] sync_supers+0x93/0xa0
[c013fbc8] wb_kupdate+0x28/0x110
[c01406e0] __pdflush+0xc0/0x1b0
[c01407f6] pdflush+0x26/0x30
[c013fba0] wb_kupdate+0x0/0x110
[c01407d0] pdflush+0x0/0x30
[c013105a] kthread+0xba/0xc0
[c0130fa0] kthread+0x0/0xc0
[c0101375] kernel_thread_helper+0x5/0x10
Code: 04 89 0c 24 e8 d2 39 ff ff e9 bd fb ff ff 0f 0b 1c 05 cd ee 36 c0 e9 ff fe ff ff 89 1c 24 e8
e8 2f 00 00 85 c0 0f 84 01 ff ff ff 0f 0b 21 05 cd ee 36 c0 e9 f4 fe ff ff b8 60 51 37 c0 89 44 24
--- Additional Comment #1 From Robert Baxter 2005-06-04 11:07 PDT [reply]
---
before I upgrade the kernel, I figured it would be pertent to grab some more
info. This server IS rsync4.ca.gentoo.org I can't have it down for too too
long. ;)
Portage 2.0.51.19 (default-linux/x86/2005.0, gcc-3.3.5-20050130,
glibc-2.3.4.20041102-r1, 2.6.11-gentoo-r6 i686)
=
System uname: 2.6.11-gentoo-r6 i686 Intel(R) Pentium(R) 4 CPU 3.00GHz
Gentoo Base System version 1.4.16
Python: dev-lang/python-2.3.5 [2.3.5 (#1, Apr 28 2005, 03:07:48)]
dev-lang/python: 2.3.5
sys-apps/sandbox:[Not Present]
sys-devel/autoconf: 2.59-r6, 2.13
sys-devel/automake: 1.7.9-r1, 1.8.5-r3, 1.5, 1.4_p6, 1.6.3, 1.9.5
sys-devel/binutils: 2.15.92.0.2-r7
sys-devel/libtool: 1.5.16
virtual/os-headers: 2.6.8.1-r1, 2.6.8.1-r2
ACCEPT_KEYWORDS=x86
AUTOCLEAN=yes
CFLAGS=-O3 -march=pentium4 -pipe -fomit-frame-pointer
CHOST=i686-pc-linux-gnu
CONFIG_PROTECT=/etc /usr/kde/2/share/config /usr/kde/3/share/config
/usr/share/config /usr/share/texmf/dvipdfm/config/
/usr/share/texmf/dvips/config/ /usr/share/texmf/tex/generic/config/
/usr/share/texmf/tex/platex/config/ /usr/share/texmf/xdvi/ /var/bind
/var/qmail/alias /var/qmail/control /var/vpopmail/domains /var/vpopmail/etc
CONFIG_PROTECT_MASK=/etc/gconf /etc/terminfo /etc/env.d
CXXFLAGS=-O3 -march=pentium4 -pipe -fomit-frame-pointer
DISTDIR=/usr/portage/distfiles
FEATURES=autoaddcvs autoconfig ccache distlocks sandbox sfperms strict
GENTOO_MIRRORS=ftp://mirrors.tera-byte.com/pub/gentoo;
MAKEOPTS=-j2
PKGDIR=/usr/portage/packages
PORTAGE_TMPDIR=/var/tmp
PORTDIR=/usr/portage
SYNC=rsync://rsync4.ca.gentoo.org/gentoo-portage
USE=x86 acpi apache2 avi bitmap-fonts bzip2 crypt curl emboss encode exif fam
flash foomaticdb gd gdbm gif gpm gtk2 imagemagick imap imlib jpeg libg++ libwww
mad maildir mikmod motif mp3 mpeg mysql ncurses nls nptl nptlonly ogg oggvorbis
pam pdflib perl png python quicktime readline samba sdl slang snmp spell sse ssl
tcpd tetex tiff truetype truetype-fonts type1-fonts userlocales vorbis xml2 xv
zlib userland_GNU kernel_linux elibc_glibc
Unset: ASFLAGS, CBUILD, CTARGET, LANG, LC_ALL, LDFLAGS, LINGUAS,
PORTDIR_OVERLAY
--- Additional Comment #2 From Daniel Drake 2005-06-13 15:45 PDT [reply]
---
Is this reproducable or just a one-off?
--- Additional Comment #3 From Robert Baxter 2005-06-13 17:28 PDT [reply]
---
I haven't had it happen again, and can't seem to make it happen again either. It
was too long ago to remember what exactly the server was doing.
--- Additional Comment #4 From Daniel Drake 2005-06-26 02:40 PDT [reply]
---
OK, it is probably going to be hard to fix then, and might just have been a disk
hiccup or something.
Re: -mm - 2.6.13 merge status
Hubert Chan [EMAIL PROTECTED] writes: On Sat, 25 Jun 2005 12:23:41 -0700, Hans Reiser [EMAIL PROTECTED] said: assert(trace_hash-89, is_hashed(foo) != 0); Lots of people like corporate anonymity. Some don't. I don't. I like knowing who wrote what. ... For what it's worth (I know: not much), I like the named asserts in Reiser4/Reiserfs. Although I haven't been bitten by any BUGs yet (maybe I'm just lucky), whenever I see these on the mailing list, it gives the impression that the users are interacting more directly with the developers, and it helps us to get to know the developers better. If people really want more standard-looking identifiers, I think Namesys should keep the names and make a hybrid identifier, like nikita-123(file:line) This already happens: together with uid-serial, reiser4 outputs __FILE__, __LINE__, __FUNCTION__, and a bunch of other stuff: -- reiser4[xine(11262)]: txn_end (fs/reiser4/txnmgr.c:504)[nominaodiosasunt-2967]: code: -2 at fs/reiser4/search.c:1285 reiser4 panicked cowardly: assertion failed: lock_stack_isclean(get_current_lock_stack()) -- Nikita.
Re: reiser4 plugins
On Wed, Jun 22, 2005 at 08:39:01PM -0700, Hans Reiser wrote: Correct me if I am wrong: What exists currently in VFS are vector instances, not classes. Plugins, selected by pluginids, are vector classes, with each pluginid selecting a vector class. You propose to have the vector class layer (aka plugin layer) in reiser4 export the vector instance to VFS for VFS to handle for each object, rather than having VFS select reiser4 and reiser4 selecting a vector class (aka plugin) which selects a method. If we agree on the above, then I will comment further. I'm a bit confused about what you're saying here. What does the 'vector' in all these expressions mean? In OO terminology our *_operation structures are interfaces. Each particular instance of such a struct that is filled with function pointers is a class. Each instance of an inode/file/dentry/superblock/... that uses these operation vectors is an object. What I propose (or rather demand ;-)) is that you don't duplicate this infrastructure, and add another dispath layer below these objects but instead re-use it for what it is done and only handle things specific to reiser4 in your own code.
Re: reiser4 plugins
On Wed, Jun 22, 2005 at 02:46:44AM -0500, David Masover wrote: There's been sloppy code in the kernel before. I remember one bit in particular which was commented Fuck me gently with a chainsaw. If I remember correctly, this had all of the PCI ids and the names and manufacturers of the corresponding devices -- in a data structure -- in C source code. It was something like one massive array definition, or maybe it was a structure. I don't remember exactly, but... Every device driver has a big array of corresponing device ids as an array in C code - oh my god we're doomed .. not. I could throw the same sarcasm back at you. We must be doomed because Reiser does some stuff that VFS already does! Or am I misunderstanding the complaint? I rather wanted to say I absolutely don't see any correlation of your PCI driver example to what we're discussing here. PCI driver hardcode ID tables because they are supposed to do that. And if a PCI driver works around hardware bugs for a specific subset of hardware it needs to use an ID table for that one aswell. And adding a strong comment about broken hardware is considered to be just fine in Linux kernel land aswell. Now to your reply. We're not doomed if a driver re-implements something we already have in common code. We would be doomed if every driver reimplements lots of things, that's why we push hard to avoid drivers doing that. It gets even more important if that something duplicated is not some simple piece code but complex abstractions. How does it get proven if you won't give it a chance as a *separate* unproven mess, with a big fat EXPERIMENTAL flag, for users to play with? I know, it exists as a separate patch. But it works now, and I think the best way to prove it would be to package it with the kernel. You're free to test it as much outside the kernel tree. This might make it more proven one day, but not automatically less of a mess. The real point here is that we already have a useful abstraction in that area, and we spent a lot of work to make it proven and not a mess (or just a little bit of a mess..), and thus we'd rather see people extending it reasonably for their needs instead of duplicating lots of infastructure in less than ideal ways.
Re: reiser4 plugins
On Wed, Jun 22, 2005 at 04:08:49AM -0500, David Masover wrote: I've been reading a bit of history, and the reason Linux got so popular in the first place was the tendency to include stuff that worked and provided a feature people wanted, even if it was ugly. The philosophy would be: choose a good implementation over an ugly one, but choose an ugly one over nothing at all. And things change over time. Back in those days the linux codebase was small and it was easy to change things all over the place. These times our codebase is huge, and people that know enough parts of the kernel to do big changes are overloaded with work. Thus we have to set our acceptance criteria a lot higher now - else we'd be totally lost with the current size of the project already. We have to maintain said ugly code for decades. Maintainability is a big deal when you deal with the timeframes we deal with. Maintainability is like optimization. The maintainability of a non-working program is irrelevant. You'd be right if we already had plugins-in-the-VFS. We don't. The most maintainable solution for plugins-in-the-FS that actually exists is Reiser4, exactly as it is now - -- because it is the _only_ one that actually exists right now. We do have plugins in the VFS, every filesystem is a set of a few of them and some gluecode. skipping a lot stuff David and Hans, I've read through my backlog a lot now, and I must say it's pretty pointless - you're discussing lots of highlevel what if and don't actually care about something as boring as actual technical details. Hans has lots of very skillfull technical people like zam and vs, and maybe he should give them some freedom to sort out technical issues for a basic reiser4 merge, and one that is done we can turn back to discussion of advanced features and their implementation, hopefully with a few more arguments on both sides and a real technical discussion.
Re: reiser4 plugins
Christoph Hellwig wrote: I'm a bit confused about what you're saying here. What does the 'vector' in all these expressions mean? In OO terminology our *_operation structures are interfaces. Each particular instance of such a struct that is filled with function pointers is a class. Each instance of an inode/file/dentry/superblock/... that uses these operation vectors is an object. What I propose (or rather demand ;-)) is that you don't duplicate this infrastructure, and add another dispath layer below these objects but instead re-use it for what it is done and only handle things specific to reiser4 in your own code. Just out of curiosity, could you please specify few exact examples with specific file/function names which duplicate the existing infrastructure. What do they duplicate and why? How should these functions be implemented on VFS? Ho should the the other FSes implement/ignore them? Why are you shure they will fit VFS well? etc. Thanks. -- Best regards, Artem B. Bityuckiy Oktet Labs (St. Petersburg), Software Engineer. +78124286709 (office) +79112449030 (mobile) E-mail: [EMAIL PROTECTED], web: http://www.oktetlabs.ru
Re: reiser4 plugins
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Christoph Hellwig wrote: On Wed, Jun 22, 2005 at 02:46:44AM -0500, David Masover wrote: There's been sloppy code in the kernel before. I remember one bit in particular which was commented Fuck me gently with a chainsaw. If I remember correctly, this had all of the PCI ids and the names and manufacturers of the corresponding devices -- in a data structure -- in C source code. It was something like one massive array definition, or maybe it was a structure. I don't remember exactly, but... Every device driver has a big array of corresponing device ids as an array in C code - oh my god we're doomed .. not. I could throw the same sarcasm back at you. We must be doomed because Reiser does some stuff that VFS already does! Or am I misunderstanding the complaint? I rather wanted to say I absolutely don't see any correlation of your PCI driver example to what we're discussing here. PCI driver hardcode ID tables because they are supposed to do that. And if a PCI driver works around hardware bugs for a specific subset of hardware it needs to use an ID table for that one aswell. And adding a strong comment about broken hardware is considered to be just fine in Linux kernel land aswell. I seem to remember the comment was more about hardcoded ID tables. And this was the generic ID code database, which is now maintained out of the kernel: /usr/share/misc/pci.ids A list of all known PCI ID's (vendors, devices, classes and sub-classes). That is what I was referring to, that used to be in the kernel. What this has to do with is whether you believe that it's better to keep code out until it's perfectly clean, or to let in code that has some things about it that you don't like, but works, is useful, and the maintainers seem very willing to fix it later on. I believe it's better to let stuff in so long as it works and doesn't break anything, so long as it is being improved to the *real* standards along the way. But, this opinion obviously isn't shared by most people here, so I'm going to stop arguing it now. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQr7kMHgHNmZLgCUhAQIHdg/+M0ExRndz9Wym0rTcnSjlg4dc3cM5ZW7N LZ1lcLM+4Vtwsj16dc9ezNj7VLAx7Vmj/3afW3TxmjQKn50J53ZlTfd6HgpBvkAi i3V7syjYJg/WaqOlEWCDwO3i/HzYdd9QAgJBbL30g56/ZtJj6SNFlKvb6UizYjIf dHgY/ZG3BuUKLBTQFUaFuBmkb/eHlhZAq7qbwn6om3bR9UmjDr41l6nP/Ry9k438 gKvwNUQZX9EkQK/F34uDM8S1bbqt8YBcULbUBYp6A12kTL8dLS3Ax8TMbHK0Zxk1 OAMpel7e9kVUbC+NAGv2PRgJQEL4jDzwWS8kP12oBbeacxRnwj1WAKixBM+3v+uq ThGwcN8CXCfPW8DBokzqYw32vVA+PVpz3tCmnVyobPPPQZdZ5wlKTgZ+Yg4NCp+M WstSf/LE6OjVIH7xjVLBeZGuynmXHshuEubwRwaiHJKp9kUli+WEpJconvR0W3ph 4WQ6/7px64XxOPnxfR7jjSCNVKjzEZjeXOf1LbJF0a8yh6eVd5g1lsxYS3Ht04w3 yEup3PKWkhvIlGZw8w7IxR4fDI80/t4F9EugILJMLDwMwvJ1k5rRWrEpzF1/I6OM wdq99cj380B600peLWZ1PIQpqb0iDj+FBqio/MIQSz4Pqlg7qaME21kI8PHxIPib kJUIRwlarKA= =TNGY -END PGP SIGNATURE-
Re: reiser4 plugins
On Gwe, 2005-06-24 at 20:21, Hans Reiser wrote: Alan, this is FUD. Our V3 fsck was written after everything else was, for lack of staffing reasons (why write an fsck before you have an FS worth using). As a result, there was a long period where the fsck code was unstable. It is reliable now. People often think that our tree makes fsck less robust. Actually fsck can throw the entire internal tree away and rebuild from leaf nodes, and frankly that makes things pretty robust. I did a series of tests well after resier3 had fsck that consisted of modelling the behaviour of systems under error state. I modelled random bit errors, bit errors at a fixed offset (class ram failure), sector 4 byte slip (known IDE fail case) and sectors going away. Reiserfs didn't handle it anything like as gracefully as ext2. Its a pretty easy experiment to write the code for and the results are interesting.
Re: reiser4 plugins
On Sad, 2005-06-25 at 04:14, David Masover wrote: Right, but even if I was a door geek, changing hinges costs more than changing code. Now, if I were building a new house and I happened to Probably not, programmers are expensive 8) DVDs are cheap nowdays: http://www.newegg.com/Product/Product.asp?Item=N82E16817502002 Ok, you might need two of those. Still, it's not much, unless that's At least four because the media decay patterns are not well understood. So you need raid DVD too 8) 1. It must work 2. It must be clean code that follows the kernel style 3. It must not break other stuff 4. It needs a maintainer who won't get bored 12 months later and only support reiser5 It's #2 and #3 that I'm worried about. #4 is a little unfair, and I can verify that #1 is satisfied. Please review the 2.5 history of reiserfs3 if you believe so
Re: reiser4 plugins
On Sun, Jun 26, 2005 at 12:21:52PM -0500, David Masover wrote: I seem to remember the comment was more about hardcoded ID tables. And this was the generic ID code database, which is now maintained out of the kernel: /usr/share/misc/pci.ids A list of all known PCI ID's (vendors, devices, classes and sub-classes). That is what I was referring to, that used to be in the kernel. And you once again showed that you don't understand what you're talking about. Said database is a pci id to name mapping, which is completely irrelevant for any driver. For things like your example there's very little thing you can do but hardcoding a set of pci ids in one way or another. What this has to do with is whether you believe that it's better to keep code out until it's perfectly clean, or to let in code that has some I doesn't need to be perfect, we just need it in a reasonable state and have a buying that it's going to continue to evolve in the rig?t direction. And we're are very far from both of them in this ccase.
Re: reiser4 plugins
On Sun, 26 Jun 2005, Alan Cox wrote: On Gwe, 2005-06-24 at 20:21, Hans Reiser wrote: Alan, this is FUD. Our V3 fsck was written after everything else was, for lack of staffing reasons (why write an fsck before you have an FS worth using). As a result, there was a long period where the fsck code was unstable. It is reliable now. People often think that our tree makes fsck less robust. Actually fsck can throw the entire internal tree away and rebuild from leaf nodes, and frankly that makes things pretty robust. I did a series of tests well after resier3 had fsck that consisted of modelling the behaviour of systems under error state. I modelled random bit errors, bit errors at a fixed offset (class ram failure), sector 4 byte slip (known IDE fail case) and sectors going away. Reiserfs didn't handle it anything like as gracefully as ext2. Its a pretty easy experiment to write the code for and the results are interesting. Maybe but I once checked some other error scenario. I generated (by mistake of course) dm table that lineary connected 3 times the same partition (instead of 3 different partitions). Both Reiser4 and Reiserfs3 gave a lot of errors while trying to use such device. Ext3 did not give single error and was hapily droping my data, I agree that this is not very useful test case for disk problems but it shows that, at least, checks for trouble in Reiser4 are miles before those in Ext2/3. If only Reiser4 could print a note what I done wrong... ;-) Grzegorz Kulewski
Re: reiser4 plugins
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Christoph Hellwig wrote: On Sun, Jun 26, 2005 at 12:21:52PM -0500, David Masover wrote: I seem to remember the comment was more about hardcoded ID tables. And this was the generic ID code database, which is now maintained out of the kernel: /usr/share/misc/pci.ids A list of all known PCI ID's (vendors, devices, classes and sub-classes). That is what I was referring to, that used to be in the kernel. And you once again showed that you don't understand what you're talking about. Said database is a pci id to name mapping, which is completely irrelevant for any driver. For things like your example there's very little thing you can do but hardcoding a set of pci ids in one way or another. I hope I admitted ahead of time that I don't understand what I'm talking about. For the record -- in this particular example, I don't. But, the distinction is irrelevant. PCI id to name mapping used to be (I think?) in the kernel, it was ugly. It may not have been where the chainsaw comment was, though. What this has to do with is whether you believe that it's better to keep code out until it's perfectly clean, or to let in code that has some I doesn't need to be perfect, we just need it in a reasonable state and have a buying that it's going to continue to evolve in the rigýt direction. And we're are very far from both of them in this ccase. I believe that we are closer to both of those than much of what's already in the kernel. Sorry I picked such a bad example, though. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQr7rLngHNmZLgCUhAQIl7A//QTNclAuPe0Q+qzZjT7uMXeWXVgxlKGbR cif4g55qtafpCA0m4/SkLYUmXpAL9Il384tCSK3vvnK7w8bQjGMCGk+xBeWQrDOC qvuyu+GOaZh+jCeo25IMm5rQRyxrsFb9d+0r+mclN2MDBmzn3l/lMiAIFq6NnlxZ gP0dOrCGpG1LXohfRhxLTphcmG1UMX/q7WbSSOCNAIMOxoqH2ez0ahdkJ44K+L45 hyMDtDugKCMeJw5r0No8RFl37h6bES/Q3DRv6Q1OQjbk4NYUKo9xCt69ypIvYRLJ SSNE63CRIjOiOn2sgii1q5kwNj+nShnMrl7bBTjtslMLoariWRPJwswwxMPJh6Nv xlovTxJKQnmg++KkJSZ6eDc7oCQapcGjeVxRCryHTIphuJ0OgRpw7xM4fsmOSj5R ruE1XrJZADage92NaozNvCASDh9wLdnkJG9cepJVMsZTb/emDQ54UbOv3yqwtHEm IbKnDNdSUVs0sgnLErWjRiQjfTh2xn0jof60mquVLufcJ0Ev9n7vWDTUBgKsFLVh xU4n9y+GWDqT31nql+si+pEJlBeYQCt4Sz9aci7Sni+sVDv929HnRbf4T9GTi99J 5RumBvrBRdfbo/SJos4ttxEQDxUFbIc92UJtYESyLzwH1SIl8lihqfN+fP/oPDdf unJVGXcgnMQ= =BJ6I -END PGP SIGNATURE-
Re: reiser4 plugins
On Sun, 26 Jun 2005 02:48:06 CDT, David Masover said:
Lincoln Dale wrote:
this is the WHOLE point of standardization .. i don't think its that
Reiser4's EAs offer any more or less capabilities than standard EAs -
They do. Reiser4's EAs can look like any other object -- files,
folders, symlinks, whatever. This is important, especially for
transparency.
No, you want them to look like the same objects that {get|set}xattr() manage
currently. You don't want programs to have to guess what an EA looks like
this week, with this user's combination of plugins that's different from
everybody else's.
lets take this a step further. what about compression? do we accept
that each filesystem can implement its own proprietary compression via
its own API - and now we need individual user-space tools to understand
No, that's the beauty of these EAs in Reiser4. The API is standard
write(2) commands. sys_reiser4 supposedly implements an interface to
make this scale better, but otherwise have the same semantics. And who
said anything about proprietary compression? I think we were planning
on the kernel's zlib, though we might have been planning to make it a
bit more seekable...
each of these APIs?
So, the API becomes something like:
cat crypto/inflated/foo # transparently decompressed
cat crypto/raw/foo.gz # raw, gzip-compressed
And 'cat crypto/raw/foo' or 'crypto/inflated/foo.gz' gets you what, exactly?
Now throw some .bz2 and .zip files into the mix... ;)
Another possibility, if you like file-as-a-directory:
cat foo.gz# raw
cat foo.gz/inflated # decompressed
One could easily imagine things like these two potentially equivalent
commands:
cp foo bar.zip/
zip bar foo
Unless of course the user had done 'mkdir sorted.by.city.zip' to make
a directory of files containing data sorted by USPS Zip code.
And what happens if the user has a file 'bar' that's not a ZIP file,
and a directory 'bar.zip' isn't a view into 'bar'?
Most of the time, if I have a file 'linux-2.6.12.tar.bz2' and a
directory 'linux-2.6.12', what is under the directory is *NOT* the same
data as what's in the .bz2 - I've done 'make oldconfig' and a few builds
and some variable amount of patching, usually with rejects, and I *don't*
want that .bz2 being updated during all this (hint - what's my next command
after 'rm -rf linux-2.6.12' likely to be, and why, and what expectations
do I have when I do it?)
You want to think this sort of thing through *really* thoroughly, because
there's a *lot* of things, both users and programs, that have expectations
about The Way Things Work.
The whole point is to have less userland tools, not more. I'm not
saying we move zip into the kernel, just that the user now has one less
command to remember.
But now instead of having to remember the one meme I can manage any
compressed-archive format that's stored in a file, and put other files in it,
and all I need is the appropriate userspace tool, they have to remember the
cp trick works for .zip and .tar, but I'll get a not a directory error if I
try it with a .hqx file, and that other file format may or may not work,
because I can't remember if this kernel has a working out-of-tree module for
this kernel
pgp996ifajjEW.pgp
Description: PGP signature
Re: reiser4 plugins
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
[EMAIL PROTECTED] wrote:
On Sun, 26 Jun 2005 02:48:06 CDT, David Masover said:
Lincoln Dale wrote:
this is the WHOLE point of standardization .. i don't think its that
Reiser4's EAs offer any more or less capabilities than standard EAs -
They do. Reiser4's EAs can look like any other object -- files,
folders, symlinks, whatever. This is important, especially for
transparency.
No, you want them to look like the same objects that {get|set}xattr() manage
currently. You don't want programs to have to guess what an EA looks like
this week, with this user's combination of plugins that's different from
everybody else's.
Plugins is a bad word. This user's combination of plugins is most
likely identical to other users', it's just which ones are enabled, and
which aren't? If they are all included, I assume they play nice.
And just because they are called plugins doesn't mean the EA looks
different every week.
I cannot stress this enough. In Reiser4, EVERYTHING is some sort of
plugin. We now have a standard disk format plugin, and a standard set
of plugins providing UNIX filesystem-like things (vanilla files,
directories, symlinks, device nodes, named pipes...) which has been
stable for some time.
I see no problems with providing at least as consistent an EA interface
as any other FS, only more extensible because it's not flat.
lets take this a step further. what about compression? do we accept
that each filesystem can implement its own proprietary compression via
its own API - and now we need individual user-space tools to understand
No, that's the beauty of these EAs in Reiser4. The API is standard
write(2) commands. sys_reiser4 supposedly implements an interface to
make this scale better, but otherwise have the same semantics. And who
said anything about proprietary compression? I think we were planning
on the kernel's zlib, though we might have been planning to make it a
bit more seekable...
each of these APIs?
So, the API becomes something like:
cat crypto/inflated/foo # transparently decompressed
cat crypto/raw/foo.gz # raw, gzip-compressed
And 'cat crypto/raw/foo' or 'crypto/inflated/foo.gz' gets you what, exactly?
Now throw some .bz2 and .zip files into the mix... ;)
Interface is the same. Only, zip files aren't just compression, so
maybe the interface changes a little there.
Point is, now you have a standard interface for any program to access
any simple lossless compression, transparently.
Another possibility, if you like file-as-a-directory:
cat foo.gz# raw
cat foo.gz/inflated # decompressed
One could easily imagine things like these two potentially equivalent
commands:
cp foo bar.zip/
zip bar foo
Unless of course the user had done 'mkdir sorted.by.city.zip' to make
a directory of files containing data sorted by USPS Zip code.
What's this got to do with anything?
And what happens if the user has a file 'bar' that's not a ZIP file,
and a directory 'bar.zip' isn't a view into 'bar'?
In file-as-a-directory (which is probably NOT happening soon), bar.zip
is both the actual zipfile and the view inside, depending on whether you
try to open() it directly or peek inside it as a directory.
I've used that interface for some simpler things like file permissions.
I could do things like echo 0 some_file/metas/uid. The interface
is gone for now, because it broke some things, I'm told, but nothing
serious for me while I was using it. I don't think the problem is
insurmountable.
However, let's not discuss this now. I do NOT want to start another
silent semantic changes with reiser4 thread. File-as-directory is not
happening this time, so don't worry about it -- this time.
Most of the time, if I have a file 'linux-2.6.12.tar.bz2' and a
directory 'linux-2.6.12', what is under the directory is *NOT* the same
data as what's in the .bz2 - I've done 'make oldconfig' and a few builds
and some variable amount of patching, usually with rejects, and I *don't*
want that .bz2 being updated during all this (hint - what's my next command
after 'rm -rf linux-2.6.12' likely to be, and why, and what expectations
do I have when I do it?)
You're misunderstanding. man zip.
$ zip bar foo
creates/modifies a file named bar.zip, not bar, which contains the
file foo.
You want to think this sort of thing through *really* thoroughly, because
there's a *lot* of things, both users and programs, that have expectations
about The Way Things Work.
Or, I can avoid those issues altogether, and simply delegate this kind
of stuff to user-created-but-magic directories. For instance, I could
have a directory called /foo which contains encrypted files, and
/foo/decrypted which has transparently decrypted representations of them.
The whole point is to have less userland tools, not more. I'm not
saying we move zip into the kernel, just that the user now has one less
Re: reiser4 plugins
On Sun, 26 Jun 2005 14:58:07 CDT, David Masover said: Plugins is a bad word. This user's combination of plugins is most likely identical to other users', it's just which ones are enabled, and which aren't? If they are all included, I assume they play nice. Which ones are enabled. Exactly. And just because they are called plugins doesn't mean the EA looks different every week. They do if the one enabled this week is make EAs look like symlinks, and last week's was make EAs look like folders. (Don't blame me, *you're* the one that said EAs can look like any other object..) And 'cat crypto/raw/foo' or 'crypto/inflated/foo.gz' gets you what, exactly ? Now throw some .bz2 and .zip files into the mix... ;) Interface is the same. Only, zip files aren't just compression, so maybe the interface changes a little there. Right. So please explain what crypto/raw/foo and crypto/inflated/foo.gz give you. Point is, now you have a standard interface for any program to access any simple lossless compression, transparently. Another possibility, if you like file-as-a-directory: cat foo.gz # raw cat foo.gz/inflated # decompressed One could easily imagine things like these two potentially equivalent commands: cp foo bar.zip/ zip bar foo Unless of course the user had done 'mkdir sorted.by.city.zip' to make a directory of files containing data sorted by USPS Zip code. What's this got to do with anything? It's got a *LOT* to do with it if I created a *DIRECTORY*, to use *AS A DIRECTORY*, the way Unix-style systems have done for 3 decades, and suddenly my system is running like a pig because the kernel decided that it's a .zip file. And what happens if the user has a file 'bar' that's not a ZIP file, and a directory 'bar.zip' isn't a view into 'bar'? In file-as-a-directory (which is probably NOT happening soon), bar.zip is both the actual zipfile and the view inside, depending on whether you try to open() it directly or peek inside it as a directory. Ahem. bar.zip' is a *DIRECTORY*. I said 'mkdir bar.zip' - why is it not acting like a directory? However, let's not discuss this now. I do NOT want to start another silent semantic changes with reiser4 thread. File-as-directory is not happening this time, so don't worry about it -- this time. Fish or cut bait. You are the one who started handwaving the 'file-as-directory'. If you don't want it discussed, don't mention it. Most of the time, if I have a file 'linux-2.6.12.tar.bz2' and a directory 'linux-2.6.12', what is under the directory is *NOT* the same data as what's in the .bz2 - I've done 'make oldconfig' and a few builds and some variable amount of patching, usually with rejects, and I *don't* want that .bz2 being updated during all this (hint - what's my next command after 'rm -rf linux-2.6.12' likely to be, and why, and what expectations do I have when I do it?) You're misunderstanding. man zip. $ zip bar foo creates/modifies a file named bar.zip, not bar, which contains the file foo. No. *YOU* are misunderstanding. I have a directory 'linux-2.6.12', and I have a file 'linux-2.6.12.tar.bz2', and I do *NOT* want directory operations to be silently converted into let's scribble into the middle of this tar file and then compress it. (Hint - work out how long a kernel 'make' would take if you were doing it inside a .tar.bz2). You want to think this sort of thing through *really* thoroughly, because there's a *lot* of things, both users and programs, that have expectations about The Way Things Work. Or, I can avoid those issues altogether, and simply delegate this kind of stuff to user-created-but-magic directories. For instance, I could have a directory called /foo which contains encrypted files, and /foo/decrypted which has transparently decrypted representations of them. So rather than everything working in a funky manner, a program gets to guess how funky, and in what direction, a given magical directory is pgp8ioinE1xvZ.pgp Description: PGP signature
Re: reiser4 plugins
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 [EMAIL PROTECTED] wrote: On Sun, 26 Jun 2005 14:58:07 CDT, David Masover said: Plugins is a bad word. This user's combination of plugins is most likely identical to other users', it's just which ones are enabled, and which aren't? If they are all included, I assume they play nice. Which ones are enabled. Exactly. I doubt there will be duplicate plugins, once things settle down. So you just have the program demand certain ones to be enabled. And just because they are called plugins doesn't mean the EA looks different every week. They do if the one enabled this week is make EAs look like symlinks, and last week's was make EAs look like folders. We could just as easily do that with other extended attributes. Sure, the details would be different -- maybe we just randomly add a XA onto the beginning of every string. (Don't blame me, *you're* the one that said EAs can look like any other object..) Doesn't mean we can't guarentee a certain kind in a certain configuration will always be available for a certain plugin, once it's been accepted. And 'cat crypto/raw/foo' or 'crypto/inflated/foo.gz' gets you what, exactly ? Now throw some .bz2 and .zip files into the mix... ;) Interface is the same. Only, zip files aren't just compression, so maybe the interface changes a little there. Right. So please explain what crypto/raw/foo and crypto/inflated/foo.gz give you. In that example (shouldn't have used the name crypto, but oh well), it should be crypto/raw/foo.gz and crypto/inflated/foo -- where foo.gz is the gzip'ed file and foo is the transparently compressed/decompressed file. Basically, these are equivalent: $ zcat crypto/raw/foo.gz $ cat crypto/inflated/foo Point is, now you have a standard interface for any program to access any simple lossless compression, transparently. Another possibility, if you like file-as-a-directory: cat foo.gz # raw cat foo.gz/inflated # decompressed One could easily imagine things like these two potentially equivalent commands: cp foo bar.zip/ zip bar foo Unless of course the user had done 'mkdir sorted.by.city.zip' to make a directory of files containing data sorted by USPS Zip code. What's this got to do with anything? It's got a *LOT* to do with it if I created a *DIRECTORY*, to use *AS A DIRECTORY*, the way Unix-style systems have done for 3 decades, and suddenly my system is running like a pig because the kernel decided that it's a .zip file. The kernel does not decide that. You do. And it doesn't automatically decide that every time you create a file. You have to use some interface to trigger the plugins. Originally, this was file-as-a-directory. Now? I'm not sure, I guess we could use a separate delimiter. Something like: foo # file ...foo # directory containing xattrs of file, list of plugins used. foo/# directory foo/... # directory containing xattrs of file, list of plugins used. I guess I need a new name for this approach. That's three possible ways of doing this? And what happens if the user has a file 'bar' that's not a ZIP file, and a directory 'bar.zip' isn't a view into 'bar'? In file-as-a-directory (which is probably NOT happening soon), bar.zip is both the actual zipfile and the view inside, depending on whether you try to open() it directly or peek inside it as a directory. Ahem. bar.zip' is a *DIRECTORY*. I said 'mkdir bar.zip' - why is it not acting like a directory? If you said mkdir, it would act like a directory. More likely than foo.zip/bar would be foo.zip/.../contents/bar. Which would also work for tarballs. But would not automatically compress anything you didn't want it to. However, let's not discuss this now. I do NOT want to start another silent semantic changes with reiser4 thread. File-as-directory is not happening this time, so don't worry about it -- this time. Fish or cut bait. You are the one who started handwaving the 'file-as-directory'. If you don't want it discussed, don't mention it. I do want it discussed. I'm not sure it's a good idea now. But looks like you got me... Most of the time, if I have a file 'linux-2.6.12.tar.bz2' and a directory 'linux-2.6.12', what is under the directory is *NOT* the same data as what's in the .bz2 - I've done 'make oldconfig' and a few builds and some variable amount of patching, usually with rejects, and I *don't* want that .bz2 being updated during all this (hint - what's my next command after 'rm -rf linux-2.6.12' likely to be, and why, and what expectations do I have when I do it?) You're misunderstanding. man zip. $ zip bar foo creates/modifies a file named bar.zip, not bar, which contains the file foo. No. *YOU* are misunderstanding. I have a directory 'linux-2.6.12', and I have a file 'linux-2.6.12.tar.bz2', and I do *NOT* want directory operations to be silently converted into let's scribble into the
Re: reiser4 plugins
[EMAIL PROTECTED] wrote: (Hint - work out how long a kernel 'make' would take if you were doing it inside a .tar.bz2). After the first time, not very long, if you had enough ram the plugin would keep the data uncompressed until it flushed it to disk. Performance might even improve since less would be written to disk.
Re: reiser4 plugins
Christoph Hellwig wrote: On Wed, Jun 22, 2005 at 08:39:01PM -0700, Hans Reiser wrote: Correct me if I am wrong: What exists currently in VFS are vector instances, not classes. Plugins, selected by pluginids, are vector classes, with each pluginid selecting a vector class. You propose to have the vector class layer (aka plugin layer) in reiser4 export the vector instance to VFS for VFS to handle for each object, rather than having VFS select reiser4 and reiser4 selecting a vector class (aka plugin) which selects a method. If we agree on the above, then I will comment further. I'm a bit confused about what you're saying here. What does the 'vector' in all these expressions mean? Was your word, I thought, for the *_operation structures. In OO terminology our *_operation structures are interfaces. Each particular instance of such a struct that is filled with function pointers is a class. Each instance of an inode/file/dentry/superblock/... that uses these operation vectors is an object. What I propose (or rather demand ;-)) is that you don't duplicate this infrastructure, and add another dispath layer below these objects but instead re-use it for what it is done and only handle things specific to reiser4 in your own code. Well, that is very different from saying that we get rid of the plugin layer. So, rather than having a hierarchy, in which we first select filesystem, and then select plugin, VFS should treat each plugin as though it was a filesystem, if I understand you. Plugins and pluginids continue to exist with the exact same functionality, we just eliminate a function dereference for the *_operation structures. Let's see how it codes up in its details. Zam, can you work on this? Hans
Re: reiser4 plugins
David Masover wrote: [EMAIL PROTECTED] wrote: On Sun, 26 Jun 2005 14:58:07 CDT, David Masover said: Plugins is a bad word. This user's combination of plugins is most likely identical to other users', it's just which ones are enabled, and which aren't? If they are all included, I assume they play nice. Which ones are enabled. Exactly. Reiser4 plugins are not per user, but per kernel. They are compiled in. The model is intended to ease the development process, nothing more. Apologies if the naming suggests more. Hans
Re: reiser4 plugins
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hans Reiser wrote: David Masover wrote: [EMAIL PROTECTED] wrote: On Sun, 26 Jun 2005 14:58:07 CDT, David Masover said: Plugins is a bad word. This user's combination of plugins is most likely identical to other users', it's just which ones are enabled, and which aren't? If they are all included, I assume they play nice. Which ones are enabled. Exactly. Reiser4 plugins are not per user, but per kernel. They are compiled in. The model is intended to ease the development process, nothing more. Apologies if the naming suggests more. But, to avoid confusion, the inclusion of a crytocompress plugin in a given kernel doesn't mean that all files accessed from that kernel are encrypted and compressed. It just means that you can pick an individual file and set it to be transparently encrypted/compressed. That is what I meant by enabled. Not per-user, but per-file. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQr9FcHgHNmZLgCUhAQJ2+BAAgnqAcEg41VXTUmpUUZWMduWeJKKYDE16 QvWore6MK/FWoU8hqFMddTzJVpzKNH8NoZ1Jfm2nEC51nndckwtBDsuIlMb1CGC+ I+wCrpe1kjXTz8sTs++CwDg29YojHP1/Wa5MwnUgTEBkZxacPI7r0VXyt1oO+f/N Eu6Jpgl4IN3ba52/PS+dXCQrmOJDhLeHAb+l4rtRsK+LASJitqmGmpf4PRwEejM8 dWZyjEYdiAcMx6mkeB/lho5IIcp1Xi5QBACf6S8SHXvHxRW50cKeouAISR25Ttk2 Oa1vKhWPGo4IBREjK7fzgj6GwfpIBnUfE25ZRjrBRupWumwekaCAY91JOoRkdRI+ Lw70OZhqqIKQ/O46xqhyCrroP/6is5ZxLyIRe1q3qsQfsWUfHBOONRUBdtaTQlaa 3OWzAU49cxn4Jv9S9UEYEyKx9ggqJ5hd94wZpXPq4GogWt4S8cYK3d4ZtRIyXEBr qOrLoJoTF9WeT254u+uq5gLP5kxq7Z7J51aMXbGF+4luuGJPq/50Y4hbapAMQWFA v0Z8YNWoOnOJgYji/+u8qJhsfzBdi/dmWlhhy9Te1e1b/1+zHcsbCslgsIrG2spk 3uF1GOv5NorO2n7RhierhkUrkUsLLlFqf0vPmgMqJ9DG4h6wl3bOJkYShBKTYQB+ ldYcxERKMZ4= =jv9S -END PGP SIGNATURE-
Re: reiser4 plugins
On Sun, 26 Jun 2005 19:16:48 CDT, David Masover said: But, to avoid confusion, the inclusion of a crytocompress plugin in a given kernel doesn't mean that all files accessed from that kernel are encrypted and compressed. It just means that you can pick an individual file and set it to be transparently encrypted/compressed. That is what I meant by enabled. Not per-user, but per-file. Doing key management in a secure manner is going to be *fun*. :) pgpOGJMOJGgcF.pgp Description: PGP signature
Re: reiser4 plugins
On Sun, 26 Jun 2005 17:35:48 CDT, David Masover said: Right. So please explain what crypto/raw/foo and crypto/inflated/foo.gz give you. In that example (shouldn't have used the name crypto, but oh well), it should be crypto/raw/foo.gz and crypto/inflated/foo -- where foo.gz is the gzip'ed file and foo is the transparently compressed/decompressed file. Basically, these are equivalent: $ zcat crypto/raw/foo.gz $ cat crypto/inflated/foo I'm *quite* aware of what your preconceived notions think it *should* be. Maybe the two examples I asked for have *real-world* meanings that you should be allowing for. Like, for instance, on a mail server, where the A/V software may need to unzip a file 5 or 6 times to find out if there's malicious content. Or seeing if it's a .zip bomb, where a small .zip will decompress to gigabytes. Or I'm testing a new compression algorithm, to see if multiple compressions help (yes, I know that it *shouldn't* help - but I've seen real-world cases where the algorithm could only look at a 4K or 8K window at a time, and if you hit a *very* long run of duplicate 4K segments, a second compression would compress all the identical or near-identical this is a 4K chunk tokens...) It's got a *LOT* to do with it if I created a *DIRECTORY*, to use *AS A DIRECTORY*, the way Unix-style systems have done for 3 decades, and suddenly my system is running like a pig because the kernel decided that it's a .zip file. The kernel does not decide that. You do. And it doesn't automatically decide that every time you create a file. You have to use some interface to trigger the plugins. Oh, I'm waiting for the fun the first time somebody deploys a plugin that has similar semantics to 'chmod g+s dirname/' ;) I guess I need a new name for this approach. That's three possible ways of doing this? I *said* you need to think this through in detail, didn't I? ;) I remember discussing that, actually. It wouldn't automatically do this if you didn't want it to, but it would be nice if, say, it was something truly seekable like linux-2.6.12.zip, and linux-2.6.12 was a user-created symlink to linux-2.6.12.zip/.../contents, and we had a nice caching system... I think you're highly deluded as to just how much or little performance gain this will get you. Model what happens with a kernel 'make' on a 256M machine with and without all that zipping and compressing, and assume that a constant 48M is available for caching of the linux-2.6.12/ tree. This is nice because then you get exactly the same performance during make as you would with unzip make, only better, because files you don't ever use (lots of arch, for instance) are not unpacked. Go read http://www.tux.org/lkml/#s7-7 and ponder until enlightenment arrives. pgpIiBhIz7zum.pgp Description: PGP signature
Re: reiser4 plugins
On Sun, 26 Jun 2005 15:54:25 PDT, Hans Reiser said: [EMAIL PROTECTED] wrote: (Hint - work out how long a kernel 'make' would take if you were doing it inside a .tar.bz2). After the first time, not very long, if you had enough ram the plugin would keep the data uncompressed until it flushed it to disk. You're not allowed to use current existing stuff like the disk buffer cache to weasel your way out on this one. if you had enough ram has been true for decades. The trouble is that quite often you *don't* have enough ram Performance might even improve since less would be written to disk. I've worked with filesystems where performance improves due to compression (AIX's JFS). It's a lot harder to provide an improvement if you're writing 37 more bytes in between bytes 399457 and 399458 (I suppose by aligning byte 399458 so it actually is on the start of a 4K block you can do that, but then you're losing the advantages of the compression.. ;) pgpt7ncoA2bcx.pgp Description: PGP signature
Re: reiser4 plugins
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 [EMAIL PROTECTED] wrote: On Sun, 26 Jun 2005 17:35:48 CDT, David Masover said: Right. So please explain what crypto/raw/foo and crypto/inflated/foo.gz give you. In that example (shouldn't have used the name crypto, but oh well), it should be crypto/raw/foo.gz and crypto/inflated/foo -- where foo.gz is the gzip'ed file and foo is the transparently compressed/decompressed file. Basically, these are equivalent: $ zcat crypto/raw/foo.gz $ cat crypto/inflated/foo I'm *quite* aware of what your preconceived notions think it *should* be. Maybe the two examples I asked for have *real-world* meanings that you should be allowing for. Like, for instance, on a mail server, where the A/V software may need to unzip a file 5 or 6 times to find out if there's malicious content. Or seeing if it's a .zip bomb, where a small .zip will decompress to gigabytes. Or I'm testing a new compression algorithm, to see if multiple compressions help (yes, I know that it *shouldn't* help - but I've seen real-world cases where the algorithm could only look at a 4K or 8K window at a time, and if you hit a *very* long run of duplicate 4K segments, a second compression would compress all the identical or near-identical this is a 4K chunk tokens...) It's got a *LOT* to do with it if I created a *DIRECTORY*, to use *AS A DIRECTORY*, the way Unix-style systems have done for 3 decades, and suddenly my system is running like a pig because the kernel decided that it's a .zip file. The kernel does not decide that. You do. And it doesn't automatically decide that every time you create a file. You have to use some interface to trigger the plugins. Oh, I'm waiting for the fun the first time somebody deploys a plugin that has similar semantics to 'chmod g+s dirname/' ;) I guess I need a new name for this approach. That's three possible ways of doing this? I *said* you need to think this through in detail, didn't I? ;) I remember discussing that, actually. It wouldn't automatically do this if you didn't want it to, but it would be nice if, say, it was something truly seekable like linux-2.6.12.zip, and linux-2.6.12 was a user-created symlink to linux-2.6.12.zip/.../contents, and we had a nice caching system... I think you're highly deluded as to just how much or little performance gain this will get you. Model what happens with a kernel 'make' on a 256M machine with and without all that zipping and compressing, and assume that a constant 48M is available for caching of the linux-2.6.12/ tree. Ignoring Hans' point, there is still a performance gain. Assume we can do on-disk caching, similar to fscache/cachefs for nfs. Now, benchmark: $ unzip linux-2.6.12.zip make -C linux-2.6.12 versus the hypothetical $ make -C linux-2.6.12.zip/.../contents This is an automatic performance gain, in theory, because the second command is identical to unzipping just the parts you need into linux-2.6.12, then running make. The one disadvantage is that because the unzipping is done on demand, it only really performs well if you can keep the zip binary cached. Even on most embedded systems, 54K of RAM is really not much to ask these days. Also, once you've run make once, you get to run it as many times as you like, and so long as the on-disk cache of the zipfile is still there and valid, you never have the overhead of unzipping again. Of course, this probably saves only a minute or two at most per kernel compile. But that doesn't mean there aren't real-world situations where this kind of architecture would be much more beneficial. This is nice because then you get exactly the same performance during make as you would with unzip make, only better, because files you don't ever use (lots of arch, for instance) are not unpacked. Go read http://www.tux.org/lkml/#s7-7 and ponder until enlightenment arrives. So what? I don't intend to convince anyone based on how much slower/faster their kernel compiles. It's meant to illustrate the principle of the thing. Besides, your point was that you could not run make inside of a kernel tarball/zipfile. Nobody ever suggested that you would actually want to. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQr9mfHgHNmZLgCUhAQJi0g//RGxFXWi8Om4EnKsZHcI0J7X3G6T9pj2a nwkWwjLdyg6jykdt3a5MTELBgOM2uT87k7CeO7TasA/T1ZkZ/y2Yw7x50YIgrb7j W1u5N/vfDLw3C2Nd6O2fe/b4ygReyBB6HQtNTw/k+XxDswxtEp0mcZHpNxt+W9B4 s0naezawRjF51P4ISqa6HoRo7vZUkXv+3CwuZinC5m8KnW2Us8Ww5uDjtNBLpJGR zs79w24zaj6VSHjF8lO6CuMKQLjSelMDXKDEkFHaybJgz7AckkcZg5c6VDTrc3/t m8HM5oyHWfRqVeQt9cMdLdcBZnhdbspSwQaNQkdkrZx1IX96mPoMNDUwk1B/TIi7 ++iqpkDYeOdg+DWzGPVpwQ+5LQC+7m8vRHv5dROIM6T89TnlUck2clBiPovzSP+8 KMR1R6F7qBPxJMkPcy2MNOVMkjN+VLSOCzOeOXVEUkNYdXjrJB5h3XIN5MyRR7C/ pRmjB2crzPTUz2yBatP+QUFNUMadikMqj44sEc/ie8iHbo9pfxQC/wY4M4VkJcf2
Re: reiser4 plugins
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 [EMAIL PROTECTED] wrote: On Sun, 26 Jun 2005 17:35:48 CDT, David Masover said: Right. So please explain what crypto/raw/foo and crypto/inflated/foo.gz give you. In that example (shouldn't have used the name crypto, but oh well), it should be crypto/raw/foo.gz and crypto/inflated/foo -- where foo.gz is the gzip'ed file and foo is the transparently compressed/decompressed file. Basically, these are equivalent: $ zcat crypto/raw/foo.gz $ cat crypto/inflated/foo I'm *quite* aware of what your preconceived notions think it *should* be. So, by what would ... give you, you meant what the benefits are? I was just clarifying how they work, I *thought* that's what you meant... Maybe the two examples I asked for have *real-world* meanings that you should be allowing for. Like, for instance, on a mail server, where the A/V software may need to unzip a file 5 or 6 times to find out if there's malicious content. Why would it want to unzip the *same* file that many times? If you're talking about nested zipfiles, we can do nested plugins just as easily. Or seeing if it's a .zip bomb, where a small .zip will decompress to gigabytes. Or I'm testing a new compression algorithm, to see if multiple compressions help (yes, I know that it *shouldn't* help - but I've seen real-world cases where the algorithm could only look at a 4K or 8K window at a time, and if you hit a *very* long run of duplicate 4K segments, a second compression would compress all the identical or near-identical this is a 4K chunk tokens...) Tune the plugin, or use zip directly. I'm not proposing necessarily to embed zip in the kernel and drop it from userland, just to have a kernel interface so that apps/people don't have to all know about the zip program and how to use it. Besides, in the zip example, I made very sure that unless you point a program specifically at where the zip plugin unpacks stuff, you can easily get the zipfile. Of course, that could be turned on its head if it was more low-level transparent compression, the kind where you just compress everything in /etc because it's all tiny text files. In that case, you would want a setup where you have to work to get the raw data. But in that case, it's a different kind of plugin anyway -- a plugin for storing data, rather than just accessing it. Stop worrying about automagic stuff, though. It won't trigger unless you go looking for it. It will just be easier to find if you do. I guess I need a new name for this approach. That's three possible ways of doing this? I *said* you need to think this through in detail, didn't I? ;) I said we already did, which is true. I don't think I'm making these up, I just don't remember where they were in Silent Semantic. (oops, I fired the last one off before I was done...) -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQr9pUXgHNmZLgCUhAQKQ8w/+OQXWdt1pEF6t/0D+x3mPYWq2lhlUbchZ BYywcGN6Z/yFZQrZovJABZVnB+CXlUlx8DGqQN+Lj9+8HLko/P75ErTWHfuiscpS wRJIN5dVeZ4f1RImEa8PjQf3c+n+B/ft9hq28yaR58C9vBQwAWOPVL0/4n4unNAV 49odg/IKJM9sdSF+6sVwgWNfuacRcZ5/jXtkDFXIIyJzKl6r45r3HmTkbgRdxqpU p1aOIsXa7fciN+UK+eiw5jruzJsKHaJtBdISsMWbPdVBFjsDVZmhsdOMv2RZMPo6 2V78OQ4g7pJHSMZsaWQ/vvD3PuHxZm9qglJcdGnL6jNk8OXkKzxXaGDn/SHjYFTu c8keR1KYu+U5r5RmTiihavFPpN3zefS5W5o5IyLvEZkApRCngDFitq4t2tRfDbfT zV1JZWz7/bqtoY0zdaZ3gFwxXxh8FMw/LCnsjKkBQ7etlvnvcW2IPssd9rTIV0+4 9CmA2EaYIiXAwGoFzLbbPoKf0a+6tB38oHanBrRBZuTzu9mKHZvyefQa0j2+1KOp iiCmrK49/APXfp4IUu284r2dPUqAu33LtomxwFbNLaPDZFWgQ9qPjafJBO3L8Y/9 HlFe46Jo5tL7YAEq/UyKpDYWxVcUiDZ40z2w/WKq+v9JjmUZMm+jonXHa9Nq0cQn YlE9bxtEac8= =5eSh -END PGP SIGNATURE-
Re: reiser4 plugins
On Sun, 26 Jun 2005 20:40:29 -0400, [EMAIL PROTECTED] said: On Sun, 26 Jun 2005 17:35:48 CDT, David Masover said: The kernel does not decide that. You do. And it doesn't automatically decide that every time you create a file. You have to use some interface to trigger the plugins. Oh, I'm waiting for the fun the first time somebody deploys a plugin that has similar semantics to 'chmod g+s dirname/' ;) Reiser4 plugins have to be compiled into the kernel. (They're not plugins in the sense that most people use that word.) And any admin who would compile that kind of plugin into the kernel needs to have his head examined. Not to mention that plugins must first go through Hans and/or Linus before they can get included into the kernel. The kernel defines the set of plugins available to the user. The user selects (to a certain degree) which plugins to use. -- Hubert Chan [EMAIL PROTECTED] - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred.
Re: reiser4 plugins
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Kyle Moffett wrote: On Jun 26, 2005, at 22:37:48, David Masover wrote: $ make -C linux-2.6.12.zip/.../contents I've yet to see how such automatic unzipping does not inherently introduce system insecurity into _existing_ applications by changing POSIX and UNIX semantics. When I do this in my suid perl script: open my $fh, '', $ARGV[0]; I do _not_ mean this: system '/bin/zip', 'x', $path1, $path2; Neither do I want the kernel to unzip it, because that just introduces the kernel to a whole series of normally application-level vulnerabilities. That just means the zip plugin has to be more carefully written than I thought. It would have to be sandboxed and limited to prevent buffer overruns and zip bombs. I think that you could create a situation where an untrusted zip could be explored, and the worst side effect would be that the files you see inside the untrusted zip wouldn't be what you expect. But that's no surprise -- if the zip is really malicious, there's no amount of sandboxing that would give you valid files anyway. I was probably taking this too lightly, though. Remember that zip, at least, would not be in the kernel. I think what is going in the kernel is gzip and lzo, and it's being done so transparently that you never actually see the compressed version. Can you illustrate for me with precise, clear, and unambiguous arguments That will take some time. And some thinking. how this can avoid all possible pitfalls, Especially if you want perfection. including the automatic encryption/decryption and most other non- standard filesystem features (Basically the whole '...' directory), should probably be left out of any patch submitted for inclusion until they can be _proven_ (or at least thoroughly checked) not to have undesirable results. I am doing that out of habit. Actually, it probably ends up more as: .../foo.zip/ instead of foo.zip/.../ But, it is left out. At least that interface. Now, the cryptocompress as it currently stands does not involve ... and does not introduce any new security holes in the way that you are describing. There might be some issues with key management (someone hinted vaguely at that), but nothing insurmountable. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQr9xZ3gHNmZLgCUhAQKmOA/9EGus4fGXKnBPoK4Rpd9K/6tgy0A7QFIO EeF50BSgl7M5sot9Vp28Dg1DA9X8gXHf/BxWIUse2doEdrbYKy3HFNd4ZChPFXCS j4ZtJo/eGYQntIFZ+exNJG2gDeprSBUH9AnMxF9xBfG8CxBdl6WL1dx8d7kc7ip/ UYGiu+9WmC9LanEb0ezhsO8I0KBvjx73Q3FTbD9N0cMIzK1Drv7p9r9rhswsoIzg eZnKrT2Z0u8BASbd0GfCAT3DH3Zn6zHf6Zk9FOaPaqcLwWlXbELaTbFvBp+2rpnH 9j3NwKHTtCKX5Z2BOQqtiEDEE8QInuDlcEV2K/y4g9YM1mMw3TNoXswaZrbPWWeF RD/YyzknaDhfgQdz9kQ2bfHJM7//Y9IUJZp/3NmWhvhc2+QBhXBBoTb8UEnRl7tK D9UIgsDFDVHlLcO9KPKokjyf3fRL57dd0ictHFORvicVIK6NFSfFP9LY/ZsmUXF9 Fbqwwuu8/5n9z5j9IyIqf5m7XJkHcZCeLFDkn89VS4ZM8W1aB0g3yRIlhSEDDkVt 0nZZRzvIlRHCPoPMZuoFhWSngi50ZAYXlRicKrHQERP8f7ECkB1JvY2rLSFqM+FX BGUZpXxgHe/zgg806ACNbfdnny5WgZ7qXl/IXD9svQa3lIgxY6We8ACj8Oi6c7eu 5ooBcT7AmRE= =pBA8 -END PGP SIGNATURE-
Re: reiser4 plugins
David Masover wrote: Hans Reiser wrote: David Masover wrote: [EMAIL PROTECTED] wrote: On Sun, 26 Jun 2005 14:58:07 CDT, David Masover said: Plugins is a bad word. This user's combination of plugins is most likely identical to other users', it's just which ones are enabled, and which aren't? If they are all included, I assume they play nice. Which ones are enabled. Exactly. Reiser4 plugins are not per user, but per kernel. They are compiled in. The model is intended to ease the development process, nothing more. Apologies if the naming suggests more. But, to avoid confusion, the inclusion of a crytocompress plugin in a given kernel doesn't mean that all files accessed from that kernel are encrypted and compressed. It just means that you can pick an individual file and set it to be transparently encrypted/compressed. That is what I meant by enabled. Not per-user, but per-file. You are correct.
Re: reiser4 plugins
On Sun, 26 Jun 2005 21:37:48 CDT, David Masover said: Assume we can do on-disk caching, similar to fscache/cachefs for nfs. Now, benchmark: $ unzip linux-2.6.12.zip make -C linux-2.6.12 versus the hypothetical $ make -C linux-2.6.12.zip/.../contents This is an automatic performance gain, in theory, because the second command is identical to unzipping just the parts you need into linux-2.6.12, then running make. Nope, they're not identical. The first specifically unzips it into the file system, leaving the zip file intact. The second, you're having to take all those .o files and other stuff that the 'make' generates and put them back into the .zip file *on the fly* - when the 'make' is half done, the .zip should reflect a directory tree that has had half the make execute (Think - after that hyptothetical 'make' completes, where is 'vmlinux'? ;) pgpMW7gmAGlYr.pgp Description: PGP signature
Re: reiser4 plugins
On Sun, 26 Jun 2005 21:37:48 CDT, David Masover said: Go read http://www.tux.org/lkml/#s7-7 and ponder until enlightenment arrives. So what? I don't intend to convince anyone based on how much slower/faster their kernel compiles. It's meant to illustrate the principle of the thing. No, you seemed convinced that you'd have a big win based on the fact that big chunks don't get unpacked - when in fact it's not as much of a win as you might think. And at least in the real world, performance *does* matter - if doing it the traditional way is 3 times faster, nobody's going to be interested. Besides, your point was that you could not run make inside of a kernel tarball/zipfile. Nobody ever suggested that you would actually want to. Here's a new facility. Don't bother trying to actually use it. Is that the message you're trying to send? pgpkXNloSX147.pgp Description: PGP signature
Re: reiser4 plugins
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 [EMAIL PROTECTED] wrote: On Sun, 26 Jun 2005 21:37:48 CDT, David Masover said: Go read http://www.tux.org/lkml/#s7-7 and ponder until enlightenment arrives. So what? I don't intend to convince anyone based on how much slower/faster their kernel compiles. It's meant to illustrate the principle of the thing. No, you seemed convinced that you'd have a big win based on the fact that big chunks don't get unpacked - when in fact it's not as much of a win as you might think. Not in this case. And at least in the real world, performance *does* matter - if doing it the traditional way is 3 times faster, nobody's going to be interested. Not true. I've said it before, and I'll say it again: http://www.apple.com/macosx/features/spotlight/ Users *are* willing to trade speed for features, under the right circumstances. Tiger *is* noticeably slower. People *do* make webservers out of it anyway. Or, for an extreme example: http://www.doom3.com/ Doom 3 can be made to run on much older hardware (the Voodoo3, I think), by disabling things like the dynamic lighting. The old way of doing lighting is probably more than 3 times faster, I know it's more than twice as fast. The new way looks much, much better, and many people upgraded their hardware -- at least a couple hundred dollars upgrade, sometimes an entirely new box -- in order to play this game. For that matter, the old way of doing this kind of discussion is email, the new way seems to be a PHP Wiki, which has got to be more of a load on the server, but people use it anyway. In the real world, killer features trump performance. All of this is irrelevent. The performance isn't that bad. It may even be better, even if it's not *much* better. Besides, your point was that you could not run make inside of a kernel tarball/zipfile. Nobody ever suggested that you would actually want to. Here's a new facility. Don't bother trying to actually use it. Not the facility. The specific example. Is that the message you're trying to send? I'm lacking a little imagination right now. It's Sunday night, after all. Actually, I think it might be a big win for Debian-like package management systems. That's also somewhere in the Silent Semantic archives. Here's another, somewhat better example: modifying a zisofs CD image. I'm going to go reply to your other mail now; I think the details belong there. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQr+F3XgHNmZLgCUhAQJWDA/+NmbL7kCANpEEeV+4zKx3MUmABuoRRo6Y hbRPDlBk9PCPM8GWQx+XwRh4XXb17FQklP7oIJHnnGtQelwDtcvbX8bSsDUTPd3s Gt8a11GMAJkGcPAHh2oBtlGjslJj7O/eJRlj3eeYv5LIRxq/KEfvBt0wvx2W73uN d8Mm65baeq7ipWDqKf0yQTcJT8M8kQQczO5gb9E5Q68fL+8rO7diQEOCm2A+Sp2G rlKpvcQuhQ/P7IBOnX8ABUG4UphQfMqtICIwMmkLimLfjO+ID9pnO516K7O4erMI Q2deFh6+KQ62Ngmokkh6GOu7VEmUD4XZubJpJknGWWBaKdCz12LsvHt2DeucKD5e 5mWi4oVwSInlxIRboso0CmRkoJZdEmqBJWFPtjXEtGGdufqJFySIHxPXpsqwRsa4 SuIIa2SK5vDXoc9URqKtPZtGkrGjG4VL1fxLNp9hD91hp+T5bLhsO+/8v2ZqfyRC UvTKb9u3mqdu//UxJhhj0LTTjNv3eNc+2UQFhAg7uSpa7FZjV+nEvD5ZwlpimQL1 InDt+nd5wQWWQ+SfjbZNvdvE/XGfmMArT/Ik7tYv9LkSLp0ohwOVQw3fThPGY+BD 1SW0E5FLruAMh/cRl0mwIhDQ9CZO/jrKjRbueyC7CYFMjmyBLUxbt0vUIJqsB8rJ P58IcgUaWFs= =R57s -END PGP SIGNATURE-
Re: reiser4 plugins
On Sun, 26 Jun 2005 23:10:43 EDT, Hubert Chan said:
On Sun, 26 Jun 2005 20:40:29 -0400, [EMAIL PROTECTED] said:
Oh, I'm waiting for the fun the first time somebody deploys a plugin
that has similar semantics to 'chmod g+s dirname/' ;)
(You *did* notice it was set-GID of a *directory* not an executable file,
right?)
Reiser4 plugins have to be compiled into the kernel. (They're not
plugins in the sense that most people use that word.) And any admin who
would compile that kind of plugin into the kernel needs to have his head
Oh? You saying that it *wont* be permitted for a user to say:
mkdir $HOME/zipped
chattr files under here are ZIP files $HOME/zipped
and instead you have to do that chattr by hand for *every* *single* zip file?
Or files on this filesystem are encrypted by default?
I suspect that this sort of thing is going to be one of the *first* things
that will get created, and any admin who tries to sell this idea to the users
*without* that sort of functionality will be handed their head.
Or, if that type of plugin.. needs to have their head examimed, I suggest
that you go to your kernel source tree, find fs/ext3/ialloc.c, and this code
in ext3_new_inode():
if (test_opt (sb, GRPID))
inode-i_gid = dir-i_gid;
else if (dir-i_mode S_ISGID) {
inode-i_gid = dir-i_gid;
if (S_ISDIR(mode))
mode |= S_ISGID;
} else
inode-i_gid = current-fsgid;
and #ifdef out all but the last line, and see if anything breaks. ;)
examined. Not to mention that plugins must first go through Hans and/or
Linus before they can get included into the kernel.
The kernel defines the set of plugins available to the user. The user
selects (to a certain degree) which plugins to use.
The point you missed was that plugins *will* have interactions, and as
the guys who are working on a stacker for LSM modules have found out the
hard way, trying to deal with the composition of functions is fiendishly
difficult.
And notice that it doesn't *have* to be quite so obvious - how about if a
user creates a directory $HOME/zipped/ and flags it as anything under here
is a zipped file.
Now throw in multiple users and CPU limits. User A enters that directory and
references everything, causing the buffer cache to get filled up. While there,
A makes changes, so the pages are dirty - for i in */*; do echo$i;
done
would do the job... User B now does something that causes a writeback of one
of those buffer cache pages.
A) What process currently gets ticked for the CPU and I/O for the writeback?
B) In your model, who will get ticked for the resources?
C) Will the users riot? (Note that you can't win here - currently, the price
of writing back A's and B's pages are about equal. However, if A gets dinked
for an expensive writeback due to B's process, A will get miffed. If B gets
charge for an expensive writeback of A's, B will get miffed. If you say screw
it
and bill it to a kernel thread, the bean counters will get miffed... ;)
pgpeCE1Y8XJs7.pgp
Description: PGP signature
Re: reiser4 plugins
[EMAIL PROTECTED] wrote: tarball/zipfile. Nobody ever suggested that you would actually want to. Besides, your point was that you could not run make inside of a kernel Umm, try it when we have it working, on a 1-4GB RAM machine it might not be so bad. We have the compression (albeit still with bugs) but not the tar plugin.
Re: reiser4 plugins
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 [EMAIL PROTECTED] wrote: On Sun, 26 Jun 2005 21:37:48 CDT, David Masover said: Assume we can do on-disk caching, similar to fscache/cachefs for nfs. Now, benchmark: $ unzip linux-2.6.12.zip make -C linux-2.6.12 versus the hypothetical $ make -C linux-2.6.12.zip/.../contents This is an automatic performance gain, in theory, because the second command is identical to unzipping just the parts you need into linux-2.6.12, then running make. Nope, they're not identical. The first specifically unzips it into the file system, leaving the zip file intact. The second, you're having to take all those .o files and other stuff that the 'make' generates and put them back into the .zip file *on the fly* - when the 'make' is half done, the .zip should reflect a directory tree that has had half the make execute This assumes we limit ourselves that way. I doubt something I think of in five minutes is the final API. (Think - after that hyptothetical 'make' completes, where is 'vmlinux'? ;) In the cache where the stuff was unpacked to. Anyone who wants to discuss things that will actually be done soon, or anything relevant to merging reiser4 in the near future, can stop reading now. *If* we decide that this must go both ways, *then* we either turn off write support inside the zipfile and do make with a symlink farm (cp - -as isn't hard), or (better) we can set things up so that only on access (most likely a read) of the original zipfile do we re-add all the changes. I think copy-on-write files are planned sometime, too. So one could imagine doing a COW of the original zipfile, so that you have: linux-2.6.12.zip# original, for redistributing and patching linux-2.6.12-mine.zip # your own patchset, ready for redistributing And, of course, there's always an option to treat the cache as its own COW of the original zipfile. This system (which I did discuss on Silent Semantic) is also useful for various other ways one would want to package something. Imagine someone's made a bootable iso which uses zisofs for transparent compression. You can set things up so that the iso-files plugin works as zip does above, grabbing the files you need on the fly and putting them in an on-disk cache, and files-iso takes only the files which have changed and runs them through mkzftree, then takes the whole tree and runs it through mkisofs. This actually isn't a performance win at all, if you know what you're doing. You could always set up a script (as I did when playing with this) which uses mkzftree's --crib-path option. But, transparency is nice. This way, all that's needed to build a new image is to chroot into some_file.iso/.../contents and change whatever you wanted, even use a package manager from the image, and when you're done, cdrecord some_file.iso will build the image and burn it. It also doesn't have to be lazy. Files could be compressed as you go, either in a blocking way or at a low priority, or they could be put off until you actually want to burn the image. I can imagine all kinds of packaging getting easier this way. A universal interface to peek inside of some sort of package, image, or software distribution and change its contents, then build a modified image. I guess that makes it a sort of make plugin. OK, here's an example where the zip-like behavior makes sense: Suppose you buy a computer with a desktop Linux pre-installed. But, on your first boot, the system is entirely composed of one giant copy-on-write tree, which refers to all the packages needed to build the entire system. All the packages in the standard distribution are pre-loaded onto the hard drive as (say) .deb files. In this situation, every time the user runs a program for the first time, it unpacks from a (presumably compressed) package. When a package is entirely unpacked, it may be removed. Since most users won't use a majority of the software, there's a huge amount of disk space saved. Yet when a user does want to use some random piece of software, they simply run it, and it behaves as if it was already installed. Download a package, and it can be run almost immediately after installation. A bit of a slow startup, but maybe not as much as trying to unpack absolutely everything in the package all at once and get it set up. For example, on Debian, the Foomatic PPDs all come in one package, making it easy to find the right one from the CUPS web interface -- but you only need a maximum of one PPD file per printer you want to use, and there are hundreds of PPD files in the foomatic-ppds package. There are plenty of other programs (Mozilla, OpenOffice) which install lots of components which you may never use, and certainly won't use immediately on your very first run. This way you save space and installation time. I'm sure this was also discussed in Silent Semantic, and I think it is mostly not my idea. -BEGIN PGP SIGNATURE- Version:
Re: reiser4 plugins
On Mon, 27 Jun 2005 00:31:46 CDT, David Masover said: *If* we decide that this must go both ways, *then* we either turn off write support inside the zipfile Oh, *that* will do wonders for command symmetry. And you just shot down the whole 'mv foo bar' being equivalent to 'zip bar foo' concept. ;) and do make with a symlink farm (cp - -as isn't hard), or (better) we can set things up so that only on access (most likely a read) of the original zipfile do we re-add all the changes. Those chuckleheads who have filled up a disk by saying 'tar cvf foo.tar .' just got a whole new way to fill the disk... ;) pgp8cnDLIoZE0.pgp Description: PGP signature
Re: reiser4 plugins
David Masover [EMAIL PROTECTED] wrote: Hans Reiser wrote: [...] Reiser4 plugins are not per user, but per kernel. They are compiled in. The model is intended to ease the development process, nothing more. Apologies if the naming suggests more. What do you gain by this? It is just a kernel configuration option of sorts? Just name mangling of existing mechanisms for no good reason at all? But, to avoid confusion, the inclusion of a crytocompress plugin in a given kernel doesn't mean that all files accessed from that kernel are encrypted and compressed. It just means that you can pick an individual file and set it to be transparently encrypted/compressed. That is what I meant by enabled. Not per-user, but per-file. Wonderful! I carefully transparently encrypt my secret files, so /everybody/ can read them! Now /that/ is progress! -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, ChileFax: +56 32 797513
Re: reiser4 plugins
On 6/27/05, Horst von Brand [EMAIL PROTECTED] wrote: Wonderful! I carefully transparently encrypt my secret files, so /everybody/ can read them! Now /that/ is progress! All of this side feature argument is completely offtopic for the inclusion of reiser4, but oh well. In any case, the real use for encrypted files (vs encrypted partitions) would be for doing things like tying keying into the login process so that your files are only accessible while you are logged in. This would be a very nice feature on a multiuser system.
Re: reiser4 plugins
David Masover [EMAIL PROTECTED] wrote: Kyle Moffett wrote: On Jun 26, 2005, at 22:37:48, David Masover wrote: [...] That just means the zip plugin has to be more carefully written than I thought. It would have to be sandboxed and limited to prevent buffer overruns and zip bombs. [...] Remember that zip, at least, would not be in the kernel. I think what is going in the kernel is gzip and lzo, and it's being done so transparently that you never actually see the compressed version. Doesn't matter if it is zip of some other compression, the problem is exactly the same. Can you illustrate for me with precise, clear, and unambiguous arguments That will take some time. And some thinking. See? Exactly what has been demanded by all the unfair, ReiserFS-racist, shove-any-junk-but-do-not-accept-perfect-filesystems-into-the-kernel etc people here on LKML from the very start. how this can avoid all possible pitfalls, Especially if you want perfection. Perfection would be nice, but (IMHO) not required. [...] Now, the cryptocompress as it currently stands does not involve ... and does not introduce any new security holes in the way that you are describing. There might be some issues with key management (someone hinted vaguely at that), but nothing insurmountable. OK, I see a week of flamefest going by until you admit it has the same problemas as compression (Hint: Most encryption compresses first, in order to give would-be cryptoanalysts less of a toehold via non-uniform distributions, and having less work to do), and then a whole lot of its /own/ problems (that somebody can peek at a cached uncompressed copy of my files is not so bad, if they can peek at my decrypted files I'd be rather less pleased... and here you have to include a malicious root (Yes, I'm paranoid. Doesn't mean root is not out to get me.).). Perhaps this can't be all solved, but the exact boundaries of what /is/ provided, and what /isn't/ must be made clear. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, ChileFax: +56 32 797513
Re: reiser4 plugins
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 [EMAIL PROTECTED] wrote: On Sun, 26 Jun 2005 23:10:43 EDT, Hubert Chan said: On Sun, 26 Jun 2005 20:40:29 -0400, [EMAIL PROTECTED] said: Reiser4 plugins have to be compiled into the kernel. (They're not plugins in the sense that most people use that word.) And any admin who would compile that kind of plugin into the kernel needs to have his head Oh? You saying that it *wont* be permitted for a user to say: mkdir $HOME/zipped chattr files under here are ZIP files $HOME/zipped and instead you have to do that chattr by hand for *every* *single* zip file? chown works this way. At least for username, there's no default user possible per directory. But scroll down a bit... Or files on this filesystem are encrypted by default? I suspect that this sort of thing is going to be one of the *first* things that will get created, and any admin who tries to sell this idea to the users *without* that sort of functionality will be handed their head. I think it may be possible to set defaults to a certain extent, but I think that it's still actually a per-file setting whether the file is encrypted. Obviously, the keys have to be shared somehow, so there's probably more grouping than just defaults... There has been some mention of inheritance, but I've forgotten how that's supposed to work. If there's some sort of inheritance where children inherit properties of their parent directory, and also inherit changes to those properties, than Hans probably wants that to be the prefered way of doing things? I don't think there's that kind of inheritance, though. And notice that it doesn't *have* to be quite so obvious - how about if a user creates a directory $HOME/zipped/ and flags it as anything under here is a zipped file. Now throw in multiple users and CPU limits. User A enters that directory and references everything, causing the buffer cache to get filled up. While there, A makes changes, so the pages are dirty - for i in */*; do echo$i; done would do the job... User B now does something that causes a writeback of one of those buffer cache pages. A) What process currently gets ticked for the CPU and I/O for the writeback? B) In your model, who will get ticked for the resources? C) Will the users riot? (Note that you can't win here - currently, the price of writing back A's and B's pages are about equal. However, if A gets dinked for an expensive writeback due to B's process, A will get miffed. If B gets charge for an expensive writeback of A's, B will get miffed. If you say screw it and bill it to a kernel thread, the bean counters will get miffed... ;) If I understand this correctly, this is somewhat like if user A creates a 50 meg file on a system with 100 megs free RAM, and user B runs sync. Also similar to if B were to suddenly fill up 75 megs of RAM, forcing A's file to be flushed -- last I checked, in Reiser4, only a sync or memory pressure causes writes to flush. Right? This is tempting to comment on, but I want to make sure I grok it first... -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQr+UiHgHNmZLgCUhAQK+KBAAm6BSQuoFNeY5+cPQQaVK2BACqwJZabMH Ze440Hjf9Pgn6Is/xWbCKKv6Mx4Vfx5P4+E4dkZJOyFBaVym6v5wy7ovPhFZVD8f oW8IOUnCngnQ/Ea7fhxwr+hst2L4jEbMF0deG3zg328zYKwHoY0NA7hQZg2xLhF6 +J5jQWNR+CWyhFCBMD/NG+XwtSd4pxzOjb42e7zlEuKoCgGiTB92qPGDcOYMw/Te 3ez1Z+iJ6gIMUgwrzO4J6TzsgR9d9W0rJKMpm6ulto0AQtg1Joln3Vj5pxBX6ULe 5hkV6zeNOW8Uisz6+tdUX6PC+hjfSPvJhzkPMccTm8cJGyiF+j+PI5nUj37hyAz6 iL8kBPMrsulrGphuJzeb81yLzmgknX4+tc6WrVqMPcCpP4iIYOi0RMpWxSxQuRH9 ooSUnN/JRuhHaz4JeJ6VOvkwwl4nw1dcChBnBiws4IlFQS+mgKTAzZHhHm/+/E6q nHY/uiFo1Tr95wyxrWyNcdGA/axriSIAaCXc1bEpQpzpOOcXr5d437TwEhBTIeXr UlXwM6CdSrA3XGy7ksHovRghdm/7MOmNxL4SagVvY98Dlc+UlWi755rpin7JRget 9tFGtH0IESmPwSXfsBJYNIZCk4bG8X9d2W1/6e96yVtvKdn16MgX6n3Hdt3aVgzv Ec9kEA42Ixs= =qALY -END PGP SIGNATURE-
Re: reiser4 plugins
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 [EMAIL PROTECTED] wrote: On Mon, 27 Jun 2005 00:31:46 CDT, David Masover said: *If* we decide that this must go both ways, *then* we either turn off write support inside the zipfile Oh, *that* will do wonders for command symmetry. And you just shot down the whole 'mv foo bar' being equivalent to 'zip bar foo' concept. ;) In one of three possible settings for the imaginary zipfile plugin, yes. But if we're talking about a kernel source tree, how many of us actually build zipfiles/tarballs of their kernel source trees, rather than unpack existing ones? and do make with a symlink farm (cp - -as isn't hard), or (better) we can set things up so that only on access (most likely a read) of the original zipfile do we re-add all the changes. Those chuckleheads who have filled up a disk by saying 'tar cvf foo.tar .' just got a whole new way to fill the disk... ;) Just a new interface :P -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQr+VYngHNmZLgCUhAQK/bQ/+MbWZrE+sg3te+YWCajaZhsZLyM4CZrGe BrvUb3dLSKLLg+2CzsKXQp9u3bavJg7OEEmnzgY4dK3FaOHZSATzbwmRgi1hYtfe TfVWKQb4z93zzS6MXafRkgL98qEIUV4n0aFKIEzDgeXAwMYx+Gv5Q7YMky7DEBNq HV5ZiKxhsqCEJdsUKo80G2R12XBm9reunJM/xRt+iA7WfsevBi+/Fl7qbAIDG+sd t83kZHiHQROTEmyyMj4zsJ0j7gRkJsXWKng3HEMl08AkRBIZxQRx0xmzWyZEs+wW AaPPFvsFo04tfwplasvwZ7EcLoCgFP2rfnjx190wov6ZvEWwIbKepsyLqusEuRcR hY0ohrWzYmQkHDXxs4FW+/QMsDF0L7YojHJgHOI1xSeCMOEgGPmHG9Jmf9FeK97Q 4xBXLDGXjlcDBZvDZciLzJCloH3pb+3TrzSkbIkTGnGbgtsS46bWRUAFOY1d2ynM C1FTebc5LPGoDetsu9/cLLuJrCJbbmpN/AWR33WUOnu9/O+H0lWNhiYyOg0x2G/z p6toqr95RPKO6tC5OliU7GfEpqjDhr+RQkjSUbrHqoNjZp3P07nCwGCfwMTCdsBx AkpQyuyfn/RfZvaFOnpUSOTKCWfZKYB2/J8ij+FH8tcPpYl5uzb6H6mHNyEPlvAa DOlFJixKQ7o= =6LFS -END PGP SIGNATURE-
