Re: Is there a "blackhole" /dev/null directory?
Jan Engelhardt wrote: On Feb 14 2008 10:46, Andi Kleen wrote: Jasper Bryant-Greene <[EMAIL PROTECTED]> writes: This could be done fairly trivially with FUSE, and IMHO is a good use for FUSE because since you're just throwing most data away, performance is not a concern. There is a much more interesting 'problem' with a "/dev/null directory". Q: Why would you need such a directory? A: To temporarily fool a program into believing it wrote something. Also: to let a program believe it was creating files which are used to hold the written data. Otherwise /dev/null would probably be the solution. Q: Should all files disappear? (e.g. "unlink after open") A: Maybe not, programs may stat() the file right afterwards and get confused by the "inexistence". I think what is going to happen is that files created behave as if they are the result of a mknod resulting in a /dev/null clone. Q: What if a program attempts to mkdir /dev/nullmnt/foo to just create a file /dev/nullmnt/foo/barfile? A: /dev/nullmnt/foo must continue to exist or be accepted for a while, or perhaps for eternity. The directory structure can persist, it's the writing of data which can be avoided. Real example: A program which reads log files and prepares a whole raft of reports in a directory specified. If you just want to see the summary (stdout) and exception notices (stderr) having a nulldir would avoid the disk space and i/o load if you were just looking at the critical output rather than the analysis. Yes, if this was an original program requirement it would or should have been a feature. Real world cases sometimes use tools in creative ways. -- Bill Davidsen <[EMAIL PROTECTED]> "We have more to fear from the bungling of the incompetent than from the machinations of the wicked." - from Slashdot -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Is there a blackhole /dev/null directory?
Jan Engelhardt wrote: On Feb 14 2008 10:46, Andi Kleen wrote: Jasper Bryant-Greene [EMAIL PROTECTED] writes: This could be done fairly trivially with FUSE, and IMHO is a good use for FUSE because since you're just throwing most data away, performance is not a concern. There is a much more interesting 'problem' with a /dev/null directory. Q: Why would you need such a directory? A: To temporarily fool a program into believing it wrote something. Also: to let a program believe it was creating files which are used to hold the written data. Otherwise /dev/null would probably be the solution. Q: Should all files disappear? (e.g. unlink after open) A: Maybe not, programs may stat() the file right afterwards and get confused by the inexistence. I think what is going to happen is that files created behave as if they are the result of a mknod resulting in a /dev/null clone. Q: What if a program attempts to mkdir /dev/nullmnt/foo to just create a file /dev/nullmnt/foo/barfile? A: /dev/nullmnt/foo must continue to exist or be accepted for a while, or perhaps for eternity. The directory structure can persist, it's the writing of data which can be avoided. Real example: A program which reads log files and prepares a whole raft of reports in a directory specified. If you just want to see the summary (stdout) and exception notices (stderr) having a nulldir would avoid the disk space and i/o load if you were just looking at the critical output rather than the analysis. Yes, if this was an original program requirement it would or should have been a feature. Real world cases sometimes use tools in creative ways. -- Bill Davidsen [EMAIL PROTECTED] We have more to fear from the bungling of the incompetent than from the machinations of the wicked. - from Slashdot -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Is there a "blackhole" /dev/null directory?
Hans-Jürgen Koch <[EMAIL PROTECTED]> wrote: > schrieb Jan Engelhardt <[EMAIL PROTECTED]>: >> There is a much more interesting 'problem' with a "/dev/null >> directory". >> >> Q: Why would you need such a directory? >> A: To temporarily fool a program into believing it wrote something. >> >> Q: Should all files disappear? (e.g. "unlink after open") >> A: Maybe not, programs may stat() the file right afterwards and >>get confused by the "inexistence". >> >> Q: What if a program attempts to mkdir /dev/nullmnt/foo to just >>create a file /dev/nullmnt/foo/barfile? >> A: /dev/nullmnt/foo must continue to exist or be accepted for a while, >>or perhaps for eternity. > > Well, the problem seems to be that a "directory" is not just data but > also contains metadata. While it's easy to write data to /dev/null, you > cannot simply discard metadata associated with a directory. So, such a > "/dev/null-directory" would have to remember metadata (at least all > created filenames including subdirectories) in the same way as other > filesystems do. Only file _content_ can be discarded. > To be honest, I still cannot see many sensible usecases for that... Since both of you seem to know about the (possible) problems, maybe you can take a look at my patch. http://7eggert.dyndns.org:8080/tmp/autounlink.patch (Not inline, because it would be duplicate in this thread.) (Yes, this patch needs some cleanup. I just did checkpatch.) First I thought I'd modify tmpfs to delete the file on O_CREAT, but it turned out tmpfs will increase the dentry count in order to prevent the delete-on-close effect. Skipping this step was allmost enough, I had to prevent link() from pinning these files, too. Some loops creating files or linking them did not show any decrease in available memory, and to the best of my knowlege, I did not introduce new memory leaks. And the best thing is: It's only 150 bytes of code. Not bad for an additional mount flag, is it? -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Is there a "blackhole" /dev/null directory?
rzryyvzy <[EMAIL PROTECTED]> wrote: > /dev/null is often very useful, specially if programs force to save data in > some file. But some programs like to creates different temporary file names, > so /dev/null could no more work. > > What is with a "/dev/null"-directory? > I mean a "blackhole pseudo directory" which eats every write to null. > > Here is how it could work: > mount -t nulldir nulldir /dev/nulldir > > Now if a program does a create(2), > it creates in the memory the file with its fd. > Then if a program does a write(2) to the fd, it eats the writes and give out > fakely it has written the number of bytes. When the program calls does a > close(2) of the fd, then the complete inode is deleted in the memory. > > The directory should be permanently empty except for the inodes with open > file descriptors. So only inode information would be temporary saved in this > "nulldir tmpfs" directory. > > Is there already existing a possibility to create a null directory? Please try the patch below. It will add an autounlink option to tmpfs, which should automatically get rid of non-referenced files. diff -X dontdiff -dpruN linux-2.6.24.pure/include/linux/shmem_fs.h linux-2.6.24.autounlink/include/linux/shmem_fs.h --- linux-2.6.24.pure/include/linux/shmem_fs.h 2006-11-29 22:57:37.0 +0100 +++ linux-2.6.24.autounlink/include/linux/shmem_fs.h2008-02-14 15:35:01.0 +0100 @@ -30,11 +30,14 @@ struct shmem_sb_info { unsigned long free_blocks; /* How many are left for allocation */ unsigned long max_inodes; /* How many inodes are allowed */ unsigned long free_inodes; /* How many are left for allocation */ - int policy; /* Default NUMA memory alloc policy */ - nodemask_t policy_nodes;/* nodemask for preferred and bind */ + unsigned int flags; + int policy; /* Default NUMA memory alloc policy */ + nodemask_tpolicy_nodes; /* nodemask for preferred and bind */ spinlock_tstat_lock; }; +#define TMPFS_FL_AUTOREMOVE 1 + static inline struct shmem_inode_info *SHMEM_I(struct inode *inode) { return container_of(inode, struct shmem_inode_info, vfs_inode); diff -X dontdiff -dpruN linux-2.6.24.pure/mm/shmem.c linux-2.6.24.autounlink/mm/shmem.c --- linux-2.6.24.pure/mm/shmem.c2008-01-25 15:09:39.0 +0100 +++ linux-2.6.24.autounlink/mm/shmem.c 2008-02-14 18:00:54.0 +0100 @@ -1747,31 +1747,41 @@ static int shmem_mknod(struct inode *dir, struct dentry *dentry, int mode, dev_t dev) { struct inode *inode = shmem_get_inode(dir->i_sb, mode, dev); + struct shmem_sb_info *sbinfo = SHMEM_SB(dir->i_sb); int error = -ENOSPC; - if (inode) { - error = security_inode_init_security(inode, dir, NULL, NULL, -NULL); - if (error) { - if (error != -EOPNOTSUPP) { - iput(inode); - return error; - } - } - error = shmem_acl_init(inode, dir); - if (error) { + if (!inode) + return error; + + error = security_inode_init_security(inode, dir, NULL, NULL, +NULL); + if (error) { + if (error != -EOPNOTSUPP) { iput(inode); return error; } - if (dir->i_mode & S_ISGID) { - inode->i_gid = dir->i_gid; - if (S_ISDIR(mode)) - inode->i_mode |= S_ISGID; - } - dir->i_size += BOGO_DIRENT_SIZE; - dir->i_ctime = dir->i_mtime = CURRENT_TIME; - d_instantiate(dentry, inode); + } + error = shmem_acl_init(inode, dir); + if (error) { + iput(inode); + return error; + } + if (dir->i_mode & S_ISGID) { + inode->i_gid = dir->i_gid; + if (S_ISDIR(mode)) + inode->i_mode |= S_ISGID; + } + + dir->i_size += BOGO_DIRENT_SIZE; + dir->i_ctime = dir->i_mtime = CURRENT_TIME; + d_instantiate(dentry, inode); + if ( S_ISDIR(mode) + || !(sbinfo->flags & TMPFS_FL_AUTOREMOVE)) + { dget(dentry); /* Extra count - pin the dentry in core */ + } else { + dir->i_size -= BOGO_DIRENT_SIZE; + drop_nlink(inode); } return error; } @@ -1800,6 +1810,11 @@ static int shmem_link(struct dentry *old struct inode *inode = old_dentry->d_inode; struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb); + /* In auto-unlink mode, the newly created link would be unlinked + immediately. We don't need to do anything here. */ + if (sbinfo->flags &
Re: Is there a "blackhole" /dev/null directory?
Am Thu, 14 Feb 2008 16:23:37 +0100 (CET) schrieb Jan Engelhardt <[EMAIL PROTECTED]>: > > On Feb 14 2008 16:19, Hans-Jürgen Koch wrote: > >> > >> Q: What if a program attempts to mkdir /dev/nullmnt/foo to just > >>create a file /dev/nullmnt/foo/barfile? > >> A: /dev/nullmnt/foo must continue to exist or be accepted for a > >> while, or perhaps for eternity. > > > >Well, the problem seems to be that a "directory" is not just data but > >also contains metadata. While it's easy to write data to /dev/null, > >you cannot simply discard metadata associated with a directory. So, > >such a "/dev/null-directory" would have to remember metadata (at > >least all created filenames including subdirectories) in the same > >way as other filesystems do. Only file _content_ can be discarded. > > Not even that. Suppose a userspace program (whose output you'd like > to discard) does: [...] Well, if an application wants to read back written data, you can never use such a thing, not even in simple cases where the existing /dev/null would be enough. > } > > >To be honest, I still cannot see many sensible usecases for that... > > I agree. Good :-) Hans -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Is there a "blackhole" /dev/null directory?
On Feb 14 2008 16:19, Hans-Jürgen Koch wrote: >> >> Q: What if a program attempts to mkdir /dev/nullmnt/foo to just >>create a file /dev/nullmnt/foo/barfile? >> A: /dev/nullmnt/foo must continue to exist or be accepted for a while, >>or perhaps for eternity. > >Well, the problem seems to be that a "directory" is not just data but >also contains metadata. While it's easy to write data to /dev/null, you >cannot simply discard metadata associated with a directory. So, such a >"/dev/null-directory" would have to remember metadata (at least all >created filenames including subdirectories) in the same way as other >filesystems do. Only file _content_ can be discarded. Not even that. Suppose a userspace program (whose output you'd like to discard) does: int main(void) { int fd = open("/nullmnt/foo.txt", O_WRONLY | O_CREAT | O_EXCL); /* write lots of nonsensical data that we don't need anyway */ write(fd, "Hello Wor(l)d", 13); if (lseek(fd, 0, SEEK_SET) < 0) { /* should not happen */ fprintf(stderr, "Huh, did we write to a pipe or cdev?\n"); abort(); } /* verify */ char buf[13]; read(fd, buf, 13); if (memcmp(buf, "Hello Wor(l)d", 13) != 0) fprintf(stderr, "Aïe, disk corruption!\n"); } >To be honest, I still cannot see many sensible usecases for that... I agree. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Is there a "blackhole" /dev/null directory?
Am Thu, 14 Feb 2008 16:00:06 +0100 (CET) schrieb Jan Engelhardt <[EMAIL PROTECTED]>: > > On Feb 14 2008 10:46, Andi Kleen wrote: > >Jasper Bryant-Greene <[EMAIL PROTECTED]> writes: > >> > >> This could be done fairly trivially with FUSE, and IMHO is a good > >> use for FUSE because since you're just throwing most data away, > >> performance is not a concern. > > There is a much more interesting 'problem' with a "/dev/null > directory". > > Q: Why would you need such a directory? > A: To temporarily fool a program into believing it wrote something. > > Q: Should all files disappear? (e.g. "unlink after open") > A: Maybe not, programs may stat() the file right afterwards and >get confused by the "inexistence". > > Q: What if a program attempts to mkdir /dev/nullmnt/foo to just >create a file /dev/nullmnt/foo/barfile? > A: /dev/nullmnt/foo must continue to exist or be accepted for a while, >or perhaps for eternity. Well, the problem seems to be that a "directory" is not just data but also contains metadata. While it's easy to write data to /dev/null, you cannot simply discard metadata associated with a directory. So, such a "/dev/null-directory" would have to remember metadata (at least all created filenames including subdirectories) in the same way as other filesystems do. Only file _content_ can be discarded. To be honest, I still cannot see many sensible usecases for that... Thanks, Hans -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Is there a "blackhole" /dev/null directory?
On Thu, 14 Feb 2008, Mika Lawando wrote: > Jasper Bryant-Greene schrieb: >> On Thu, 2008-02-14 at 10:30 +0100, rzryyvzy wrote: >> >>> /dev/null is often very useful, specially if programs force to save data in >>> some file. But some programs like to creates different temporary file >>> names, so /dev/null could no more work. >>> >>> What is with a "/dev/null"-directory? >>> I mean a "blackhole pseudo directory" which eats every write to null. >>> >>> Here is how it could work: >>> mount -t nulldir nulldir /dev/nulldir >>> >>> Now if a program does a create(2), >>> it creates in the memory the file with its fd. >>> Then if a program does a write(2) to the fd, it eats the writes and give >>> out fakely it has written the number of bytes. >>> When the program calls does a close(2) of the fd, then the complete inode >>> is deleted in the memory. >>> >>> The directory should be permanently empty except for the inodes with open >>> file descriptors. So only inode information would be temporary saved in >>> this "nulldir tmpfs" directory. >>> >>> Is there already existing a possibility to create a null directory? >>> >> >> This could be done fairly trivially with FUSE, and IMHO is a good use >> for FUSE because since you're just throwing most data away, performance >> is not a concern. >> > Unfortunately performance is a concern because if not I would write on > the hard disk the files, and then remove them with a cronjob. > But from the point of view of the time of developpment, FUSE is a good > idea, because its possible to write a filesystem quickly in Perl. > > -- > Best regards, > Mika > -- Creating a null directory wouldn't work because a directory is just a link to find a file. The actual file gets written using the file-descriptor, without any reference whosoever to the path. If you have root privileges, you can use `mknod tempfile c 1 3` to create a null file with any name you want. Unfortunately, somebody decided that you need root privileges to execute mknod so ordinary users can not do this. Cheers, Dick Johnson Penguin : Linux version 2.6.22.1 on an i686 machine (5588.28 BogoMips). My book : http://www.AbominableFirebug.com/ _ The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [EMAIL PROTECTED] - and destroy all copies of this information, including any attachments, without reading or disclosing them. Thank you. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Is there a "blackhole" /dev/null directory?
On Feb 14 2008 10:46, Andi Kleen wrote: >Jasper Bryant-Greene <[EMAIL PROTECTED]> writes: >> >> This could be done fairly trivially with FUSE, and IMHO is a good use >> for FUSE because since you're just throwing most data away, performance >> is not a concern. There is a much more interesting 'problem' with a "/dev/null directory". Q: Why would you need such a directory? A: To temporarily fool a program into believing it wrote something. Q: Should all files disappear? (e.g. "unlink after open") A: Maybe not, programs may stat() the file right afterwards and get confused by the "inexistence". Q: What if a program attempts to mkdir /dev/nullmnt/foo to just create a file /dev/nullmnt/foo/barfile? A: /dev/nullmnt/foo must continue to exist or be accepted for a while, or perhaps for eternity. Been there, done that, - http://dev.computergmbh.de/wsvn/misc_kernel/nullfs/trunk/nullfs.c - and hit that wall of unanswerable questions. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Is there a "blackhole" /dev/null directory?
Jasper Bryant-Greene schrieb: On Thu, 2008-02-14 at 10:30 +0100, rzryyvzy wrote: /dev/null is often very useful, specially if programs force to save data in some file. But some programs like to creates different temporary file names, so /dev/null could no more work. What is with a "/dev/null"-directory? I mean a "blackhole pseudo directory" which eats every write to null. Here is how it could work: mount -t nulldir nulldir /dev/nulldir Now if a program does a create(2), it creates in the memory the file with its fd. Then if a program does a write(2) to the fd, it eats the writes and give out fakely it has written the number of bytes. When the program calls does a close(2) of the fd, then the complete inode is deleted in the memory. The directory should be permanently empty except for the inodes with open file descriptors. So only inode information would be temporary saved in this "nulldir tmpfs" directory. Is there already existing a possibility to create a null directory? This could be done fairly trivially with FUSE, and IMHO is a good use for FUSE because since you're just throwing most data away, performance is not a concern. Unfortunately performance is a concern because if not I would write on the hard disk the files, and then remove them with a cronjob. But from the point of view of the time of developpment, FUSE is a good idea, because its possible to write a filesystem quickly in Perl. -- Best regards, Mika -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Is there a "blackhole" /dev/null directory?
Jasper Bryant-Greene <[EMAIL PROTECTED]> writes: > > This could be done fairly trivially with FUSE, and IMHO is a good use > for FUSE because since you're just throwing most data away, performance > is not a concern. Q.: how much work would fuse have to do until the user file system server could decide to ignore the data? A.: pretty much all of a cached write including all the copies and context switches. That is because FUSE has to first hand all the data to the server until it can decide to do nothing and that's pretty much all (and then some more) of the cost of a cached write. So if you want any performance benefit from this (I'm a little sceptical) you should exactly not use FUSE. The basic problem with the idea is that programs who create temporary files usually want to read them back at some point too. So if you throw everything away things break. If you just don't want the write to not (usually) hit disk you can use tmpfs, although I believe at least ext2 (not ext3 unfortunately) is also reasonably good at not writing out very short lived files. -Andi -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Is there a "blackhole" /dev/null directory?
On Thu, 2008-02-14 at 10:30 +0100, rzryyvzy wrote: > /dev/null is often very useful, specially if programs force to save data in > some file. But some programs like to creates different temporary file names, > so /dev/null could no more work. > > What is with a "/dev/null"-directory? > I mean a "blackhole pseudo directory" which eats every write to null. > > Here is how it could work: > mount -t nulldir nulldir /dev/nulldir > > Now if a program does a create(2), > it creates in the memory the file with its fd. > Then if a program does a write(2) to the fd, it eats the writes and give out > fakely it has written the number of bytes. > When the program calls does a close(2) of the fd, then the complete inode is > deleted in the memory. > > The directory should be permanently empty except for the inodes with open > file descriptors. So only inode information would be temporary saved in this > "nulldir tmpfs" directory. > > Is there already existing a possibility to create a null directory? This could be done fairly trivially with FUSE, and IMHO is a good use for FUSE because since you're just throwing most data away, performance is not a concern. -j signature.asc Description: This is a digitally signed message part
Is there a "blackhole" /dev/null directory?
Hello Linux Kernel Hackers, /dev/null is often very useful, specially if programs force to save data in some file. But some programs like to creates different temporary file names, so /dev/null could no more work. What is with a "/dev/null"-directory? I mean a "blackhole pseudo directory" which eats every write to null. Here is how it could work: mount -t nulldir nulldir /dev/nulldir Now if a program does a create(2), it creates in the memory the file with its fd. Then if a program does a write(2) to the fd, it eats the writes and give out fakely it has written the number of bytes. When the program calls does a close(2) of the fd, then the complete inode is deleted in the memory. The directory should be permanently empty except for the inodes with open file descriptors. So only inode information would be temporary saved in this "nulldir tmpfs" directory. Is there already existing a possibility to create a null directory? -- Best regards, Mika Lawando -- E-Mail sent with anti-spam site TrashMail.net! Free disposable email addresses: http://www.trashmail.net/ -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Is there a blackhole /dev/null directory?
Hello Linux Kernel Hackers, /dev/null is often very useful, specially if programs force to save data in some file. But some programs like to creates different temporary file names, so /dev/null could no more work. What is with a /dev/null-directory? I mean a blackhole pseudo directory which eats every write to null. Here is how it could work: mount -t nulldir nulldir /dev/nulldir Now if a program does a create(2), it creates in the memory the file with its fd. Then if a program does a write(2) to the fd, it eats the writes and give out fakely it has written the number of bytes. When the program calls does a close(2) of the fd, then the complete inode is deleted in the memory. The directory should be permanently empty except for the inodes with open file descriptors. So only inode information would be temporary saved in this nulldir tmpfs directory. Is there already existing a possibility to create a null directory? -- Best regards, Mika Lawando -- E-Mail sent with anti-spam site TrashMail.net! Free disposable email addresses: http://www.trashmail.net/ -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Is there a blackhole /dev/null directory?
On Thu, 2008-02-14 at 10:30 +0100, rzryyvzy wrote: /dev/null is often very useful, specially if programs force to save data in some file. But some programs like to creates different temporary file names, so /dev/null could no more work. What is with a /dev/null-directory? I mean a blackhole pseudo directory which eats every write to null. Here is how it could work: mount -t nulldir nulldir /dev/nulldir Now if a program does a create(2), it creates in the memory the file with its fd. Then if a program does a write(2) to the fd, it eats the writes and give out fakely it has written the number of bytes. When the program calls does a close(2) of the fd, then the complete inode is deleted in the memory. The directory should be permanently empty except for the inodes with open file descriptors. So only inode information would be temporary saved in this nulldir tmpfs directory. Is there already existing a possibility to create a null directory? This could be done fairly trivially with FUSE, and IMHO is a good use for FUSE because since you're just throwing most data away, performance is not a concern. -j signature.asc Description: This is a digitally signed message part
Re: Is there a blackhole /dev/null directory?
Jasper Bryant-Greene [EMAIL PROTECTED] writes: This could be done fairly trivially with FUSE, and IMHO is a good use for FUSE because since you're just throwing most data away, performance is not a concern. Q.: how much work would fuse have to do until the user file system server could decide to ignore the data? A.: pretty much all of a cached write including all the copies and context switches. That is because FUSE has to first hand all the data to the server until it can decide to do nothing and that's pretty much all (and then some more) of the cost of a cached write. So if you want any performance benefit from this (I'm a little sceptical) you should exactly not use FUSE. The basic problem with the idea is that programs who create temporary files usually want to read them back at some point too. So if you throw everything away things break. If you just don't want the write to not (usually) hit disk you can use tmpfs, although I believe at least ext2 (not ext3 unfortunately) is also reasonably good at not writing out very short lived files. -Andi -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Is there a blackhole /dev/null directory?
Jasper Bryant-Greene schrieb: On Thu, 2008-02-14 at 10:30 +0100, rzryyvzy wrote: /dev/null is often very useful, specially if programs force to save data in some file. But some programs like to creates different temporary file names, so /dev/null could no more work. What is with a /dev/null-directory? I mean a blackhole pseudo directory which eats every write to null. Here is how it could work: mount -t nulldir nulldir /dev/nulldir Now if a program does a create(2), it creates in the memory the file with its fd. Then if a program does a write(2) to the fd, it eats the writes and give out fakely it has written the number of bytes. When the program calls does a close(2) of the fd, then the complete inode is deleted in the memory. The directory should be permanently empty except for the inodes with open file descriptors. So only inode information would be temporary saved in this nulldir tmpfs directory. Is there already existing a possibility to create a null directory? This could be done fairly trivially with FUSE, and IMHO is a good use for FUSE because since you're just throwing most data away, performance is not a concern. Unfortunately performance is a concern because if not I would write on the hard disk the files, and then remove them with a cronjob. But from the point of view of the time of developpment, FUSE is a good idea, because its possible to write a filesystem quickly in Perl. -- Best regards, Mika -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Is there a blackhole /dev/null directory?
On Feb 14 2008 10:46, Andi Kleen wrote: Jasper Bryant-Greene [EMAIL PROTECTED] writes: This could be done fairly trivially with FUSE, and IMHO is a good use for FUSE because since you're just throwing most data away, performance is not a concern. There is a much more interesting 'problem' with a /dev/null directory. Q: Why would you need such a directory? A: To temporarily fool a program into believing it wrote something. Q: Should all files disappear? (e.g. unlink after open) A: Maybe not, programs may stat() the file right afterwards and get confused by the inexistence. Q: What if a program attempts to mkdir /dev/nullmnt/foo to just create a file /dev/nullmnt/foo/barfile? A: /dev/nullmnt/foo must continue to exist or be accepted for a while, or perhaps for eternity. Been there, done that, - http://dev.computergmbh.de/wsvn/misc_kernel/nullfs/trunk/nullfs.c - and hit that wall of unanswerable questions. -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Is there a blackhole /dev/null directory?
On Thu, 14 Feb 2008, Mika Lawando wrote: Jasper Bryant-Greene schrieb: On Thu, 2008-02-14 at 10:30 +0100, rzryyvzy wrote: /dev/null is often very useful, specially if programs force to save data in some file. But some programs like to creates different temporary file names, so /dev/null could no more work. What is with a /dev/null-directory? I mean a blackhole pseudo directory which eats every write to null. Here is how it could work: mount -t nulldir nulldir /dev/nulldir Now if a program does a create(2), it creates in the memory the file with its fd. Then if a program does a write(2) to the fd, it eats the writes and give out fakely it has written the number of bytes. When the program calls does a close(2) of the fd, then the complete inode is deleted in the memory. The directory should be permanently empty except for the inodes with open file descriptors. So only inode information would be temporary saved in this nulldir tmpfs directory. Is there already existing a possibility to create a null directory? This could be done fairly trivially with FUSE, and IMHO is a good use for FUSE because since you're just throwing most data away, performance is not a concern. Unfortunately performance is a concern because if not I would write on the hard disk the files, and then remove them with a cronjob. But from the point of view of the time of developpment, FUSE is a good idea, because its possible to write a filesystem quickly in Perl. -- Best regards, Mika -- Creating a null directory wouldn't work because a directory is just a link to find a file. The actual file gets written using the file-descriptor, without any reference whosoever to the path. If you have root privileges, you can use `mknod tempfile c 1 3` to create a null file with any name you want. Unfortunately, somebody decided that you need root privileges to execute mknod so ordinary users can not do this. Cheers, Dick Johnson Penguin : Linux version 2.6.22.1 on an i686 machine (5588.28 BogoMips). My book : http://www.AbominableFirebug.com/ _ The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [EMAIL PROTECTED] - and destroy all copies of this information, including any attachments, without reading or disclosing them. Thank you. -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Is there a blackhole /dev/null directory?
Am Thu, 14 Feb 2008 16:00:06 +0100 (CET) schrieb Jan Engelhardt [EMAIL PROTECTED]: On Feb 14 2008 10:46, Andi Kleen wrote: Jasper Bryant-Greene [EMAIL PROTECTED] writes: This could be done fairly trivially with FUSE, and IMHO is a good use for FUSE because since you're just throwing most data away, performance is not a concern. There is a much more interesting 'problem' with a /dev/null directory. Q: Why would you need such a directory? A: To temporarily fool a program into believing it wrote something. Q: Should all files disappear? (e.g. unlink after open) A: Maybe not, programs may stat() the file right afterwards and get confused by the inexistence. Q: What if a program attempts to mkdir /dev/nullmnt/foo to just create a file /dev/nullmnt/foo/barfile? A: /dev/nullmnt/foo must continue to exist or be accepted for a while, or perhaps for eternity. Well, the problem seems to be that a directory is not just data but also contains metadata. While it's easy to write data to /dev/null, you cannot simply discard metadata associated with a directory. So, such a /dev/null-directory would have to remember metadata (at least all created filenames including subdirectories) in the same way as other filesystems do. Only file _content_ can be discarded. To be honest, I still cannot see many sensible usecases for that... Thanks, Hans -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Is there a blackhole /dev/null directory?
On Feb 14 2008 16:19, Hans-Jürgen Koch wrote: Q: What if a program attempts to mkdir /dev/nullmnt/foo to just create a file /dev/nullmnt/foo/barfile? A: /dev/nullmnt/foo must continue to exist or be accepted for a while, or perhaps for eternity. Well, the problem seems to be that a directory is not just data but also contains metadata. While it's easy to write data to /dev/null, you cannot simply discard metadata associated with a directory. So, such a /dev/null-directory would have to remember metadata (at least all created filenames including subdirectories) in the same way as other filesystems do. Only file _content_ can be discarded. Not even that. Suppose a userspace program (whose output you'd like to discard) does: int main(void) { int fd = open(/nullmnt/foo.txt, O_WRONLY | O_CREAT | O_EXCL); /* write lots of nonsensical data that we don't need anyway */ write(fd, Hello Wor(l)d, 13); if (lseek(fd, 0, SEEK_SET) 0) { /* should not happen */ fprintf(stderr, Huh, did we write to a pipe or cdev?\n); abort(); } /* verify */ char buf[13]; read(fd, buf, 13); if (memcmp(buf, Hello Wor(l)d, 13) != 0) fprintf(stderr, Aïe, disk corruption!\n); } To be honest, I still cannot see many sensible usecases for that... I agree. -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Is there a blackhole /dev/null directory?
Am Thu, 14 Feb 2008 16:23:37 +0100 (CET) schrieb Jan Engelhardt [EMAIL PROTECTED]: On Feb 14 2008 16:19, Hans-Jürgen Koch wrote: Q: What if a program attempts to mkdir /dev/nullmnt/foo to just create a file /dev/nullmnt/foo/barfile? A: /dev/nullmnt/foo must continue to exist or be accepted for a while, or perhaps for eternity. Well, the problem seems to be that a directory is not just data but also contains metadata. While it's easy to write data to /dev/null, you cannot simply discard metadata associated with a directory. So, such a /dev/null-directory would have to remember metadata (at least all created filenames including subdirectories) in the same way as other filesystems do. Only file _content_ can be discarded. Not even that. Suppose a userspace program (whose output you'd like to discard) does: [...] Well, if an application wants to read back written data, you can never use such a thing, not even in simple cases where the existing /dev/null would be enough. } To be honest, I still cannot see many sensible usecases for that... I agree. Good :-) Hans -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Is there a blackhole /dev/null directory?
rzryyvzy [EMAIL PROTECTED] wrote: /dev/null is often very useful, specially if programs force to save data in some file. But some programs like to creates different temporary file names, so /dev/null could no more work. What is with a /dev/null-directory? I mean a blackhole pseudo directory which eats every write to null. Here is how it could work: mount -t nulldir nulldir /dev/nulldir Now if a program does a create(2), it creates in the memory the file with its fd. Then if a program does a write(2) to the fd, it eats the writes and give out fakely it has written the number of bytes. When the program calls does a close(2) of the fd, then the complete inode is deleted in the memory. The directory should be permanently empty except for the inodes with open file descriptors. So only inode information would be temporary saved in this nulldir tmpfs directory. Is there already existing a possibility to create a null directory? Please try the patch below. It will add an autounlink option to tmpfs, which should automatically get rid of non-referenced files. diff -X dontdiff -dpruN linux-2.6.24.pure/include/linux/shmem_fs.h linux-2.6.24.autounlink/include/linux/shmem_fs.h --- linux-2.6.24.pure/include/linux/shmem_fs.h 2006-11-29 22:57:37.0 +0100 +++ linux-2.6.24.autounlink/include/linux/shmem_fs.h2008-02-14 15:35:01.0 +0100 @@ -30,11 +30,14 @@ struct shmem_sb_info { unsigned long free_blocks; /* How many are left for allocation */ unsigned long max_inodes; /* How many inodes are allowed */ unsigned long free_inodes; /* How many are left for allocation */ - int policy; /* Default NUMA memory alloc policy */ - nodemask_t policy_nodes;/* nodemask for preferred and bind */ + unsigned int flags; + int policy; /* Default NUMA memory alloc policy */ + nodemask_tpolicy_nodes; /* nodemask for preferred and bind */ spinlock_tstat_lock; }; +#define TMPFS_FL_AUTOREMOVE 1 + static inline struct shmem_inode_info *SHMEM_I(struct inode *inode) { return container_of(inode, struct shmem_inode_info, vfs_inode); diff -X dontdiff -dpruN linux-2.6.24.pure/mm/shmem.c linux-2.6.24.autounlink/mm/shmem.c --- linux-2.6.24.pure/mm/shmem.c2008-01-25 15:09:39.0 +0100 +++ linux-2.6.24.autounlink/mm/shmem.c 2008-02-14 18:00:54.0 +0100 @@ -1747,31 +1747,41 @@ static int shmem_mknod(struct inode *dir, struct dentry *dentry, int mode, dev_t dev) { struct inode *inode = shmem_get_inode(dir-i_sb, mode, dev); + struct shmem_sb_info *sbinfo = SHMEM_SB(dir-i_sb); int error = -ENOSPC; - if (inode) { - error = security_inode_init_security(inode, dir, NULL, NULL, -NULL); - if (error) { - if (error != -EOPNOTSUPP) { - iput(inode); - return error; - } - } - error = shmem_acl_init(inode, dir); - if (error) { + if (!inode) + return error; + + error = security_inode_init_security(inode, dir, NULL, NULL, +NULL); + if (error) { + if (error != -EOPNOTSUPP) { iput(inode); return error; } - if (dir-i_mode S_ISGID) { - inode-i_gid = dir-i_gid; - if (S_ISDIR(mode)) - inode-i_mode |= S_ISGID; - } - dir-i_size += BOGO_DIRENT_SIZE; - dir-i_ctime = dir-i_mtime = CURRENT_TIME; - d_instantiate(dentry, inode); + } + error = shmem_acl_init(inode, dir); + if (error) { + iput(inode); + return error; + } + if (dir-i_mode S_ISGID) { + inode-i_gid = dir-i_gid; + if (S_ISDIR(mode)) + inode-i_mode |= S_ISGID; + } + + dir-i_size += BOGO_DIRENT_SIZE; + dir-i_ctime = dir-i_mtime = CURRENT_TIME; + d_instantiate(dentry, inode); + if ( S_ISDIR(mode) + || !(sbinfo-flags TMPFS_FL_AUTOREMOVE)) + { dget(dentry); /* Extra count - pin the dentry in core */ + } else { + dir-i_size -= BOGO_DIRENT_SIZE; + drop_nlink(inode); } return error; } @@ -1800,6 +1810,11 @@ static int shmem_link(struct dentry *old struct inode *inode = old_dentry-d_inode; struct shmem_sb_info *sbinfo = SHMEM_SB(inode-i_sb); + /* In auto-unlink mode, the newly created link would be unlinked + immediately. We don't need to do anything here. */ + if (sbinfo-flags TMPFS_FL_AUTOREMOVE) + return 0; + /*
Re: Is there a blackhole /dev/null directory?
Hans-Jürgen Koch [EMAIL PROTECTED] wrote: schrieb Jan Engelhardt [EMAIL PROTECTED]: There is a much more interesting 'problem' with a /dev/null directory. Q: Why would you need such a directory? A: To temporarily fool a program into believing it wrote something. Q: Should all files disappear? (e.g. unlink after open) A: Maybe not, programs may stat() the file right afterwards and get confused by the inexistence. Q: What if a program attempts to mkdir /dev/nullmnt/foo to just create a file /dev/nullmnt/foo/barfile? A: /dev/nullmnt/foo must continue to exist or be accepted for a while, or perhaps for eternity. Well, the problem seems to be that a directory is not just data but also contains metadata. While it's easy to write data to /dev/null, you cannot simply discard metadata associated with a directory. So, such a /dev/null-directory would have to remember metadata (at least all created filenames including subdirectories) in the same way as other filesystems do. Only file _content_ can be discarded. To be honest, I still cannot see many sensible usecases for that... Since both of you seem to know about the (possible) problems, maybe you can take a look at my patch. http://7eggert.dyndns.org:8080/tmp/autounlink.patch (Not inline, because it would be duplicate in this thread.) (Yes, this patch needs some cleanup. I just did checkpatch.) First I thought I'd modify tmpfs to delete the file on O_CREAT, but it turned out tmpfs will increase the dentry count in order to prevent the delete-on-close effect. Skipping this step was allmost enough, I had to prevent link() from pinning these files, too. Some loops creating files or linking them did not show any decrease in available memory, and to the best of my knowlege, I did not introduce new memory leaks. And the best thing is: It's only 150 bytes of code. Not bad for an additional mount flag, is it? -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/