Re: [zfs-discuss] Intrusion Detection - powered by ZFS Checksumming ?
Richard Elling richard.ell...@gmail.com writes: On Feb 8, 2010, at 9:10 PM, Damon Atkins wrote: I would have thought that if I write 1k then ZFS txg times out in 30secs, then the 1k will be written to disk in a 1k record block, and then if I write 4k then 30secs latter txg happen another 4k record size block will be written, and then if I write 130k a 128k and 2k record block will be written. Making the file have record sizes of 1k+4k+128k+2k Close. Once the max record size is achieved, it is not reduced. So the allocation is: 1KB + 4KB + 128KB + 128KB I think the above is easily misunderstood. I assume the OP means append, not rewrites, and in that case (with recordsize=128k): * after the first write, the file will consist of a single 1 KiB record. * after the first append, the file will consist of a single 5 KiB record. * after the second append, one 128 KiB record and one 7 KiB record. in each of these operations, the *whole* file will be rewritten to a new location, but after a third append, only the tail record will be rewritten. -- Kjetil T. Homme Redpill Linpro AS - Changing the game ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Intrusion Detection - powered by ZFS Checksumming ?
On 02/09/10 08:18, Kjetil Torgrim Homme wrote: Richard Elling richard.ell...@gmail.com writes: On Feb 8, 2010, at 9:10 PM, Damon Atkins wrote: I would have thought that if I write 1k then ZFS txg times out in 30secs, then the 1k will be written to disk in a 1k record block, and then if I write 4k then 30secs latter txg happen another 4k record size block will be written, and then if I write 130k a 128k and 2k record block will be written. Making the file have record sizes of 1k+4k+128k+2k Close. Once the max record size is achieved, it is not reduced. So the allocation is: 1KB + 4KB + 128KB + 128KB I think the above is easily misunderstood. I assume the OP means append, not rewrites, and in that case (with recordsize=128k): * after the first write, the file will consist of a single 1 KiB record. * after the first append, the file will consist of a single 5 KiB record. Good so far. * after the second append, one 128 KiB record and one 7 KiB record. A long time ago we used to write short tail blocks, but not any more. So after the 2nd append we actually have 2 128KB blocks. in each of these operations, the *whole* file will be rewritten to a new location, but after a third append, only the tail record will be rewritten. So after the third append we'd actually have 3 128KB blocks. The first doesn't need to be re-written. Neil. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Intrusion Detection - powered by ZFS Checksumming ?
Neil Perrin neil.per...@sun.com writes: On 02/09/10 08:18, Kjetil Torgrim Homme wrote: I think the above is easily misunderstood. I assume the OP means append, not rewrites, and in that case (with recordsize=128k): * after the first write, the file will consist of a single 1 KiB record. * after the first append, the file will consist of a single 5 KiB record. Good so far. * after the second append, one 128 KiB record and one 7 KiB record. A long time ago we used to write short tail blocks, but not any more. So after the 2nd append we actually have 2 128KB blocks. thanks a lot for the correction! -- Kjetil T. Homme Redpill Linpro AS - Changing the game ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Intrusion Detection - powered by ZFS Checksumming ?
Hello, an idea popped into my mind while talking about security and intrusion detection. Host based ID may use Checksumming for file change tracking. It works like this: Once installed and knowning the software is OK, a baseline is created. Then in every check - verify the current status of the data with the baseline and report changes. An example for this is AIDE. The difficult part is the checksumming - this takes time. My idea would be to use ZFS snapshots for this. baseline creation = create snapshot baseline verification = verify the checksums of the objects and report objects diffent This could work for non-zvol environments. Is it possible to extract the checksums of ZFS objects with a command line tool ? Regards, Robert -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Intrusion Detection - powered by ZFS Checksumming ?
On 08/02/2010 12:55, Lutz Schumann wrote: Hello, an idea popped into my mind while talking about security and intrusion detection. Host based ID may use Checksumming for file change tracking. It works like this: Once installed and knowning the software is OK, a baseline is created. Then in every check - verify the current status of the data with the baseline and report changes. An example for this is AIDE. The difficult part is the checksumming - this takes time. My idea would be to use ZFS snapshots for this. baseline creation = create snapshot baseline verification = verify the checksums of the objects and report objects diffent This could work for non-zvol environments. Is it possible to extract the checksums of ZFS objects with a command line tool ? Only with the zdb(1M) tool but note that the checksums are NOT of files but of the ZFS blocks. -- Darren J Moffat ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Intrusion Detection - powered by ZFS Checksumming ?
Only with the zdb(1M) tool but note that the checksums are NOT of files but of the ZFS blocks. Thanks - bocks, right (doh) - thats what I was missing. Damn it would be so nice :( -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Intrusion Detection - powered by ZFS Checksumming ?
On Mon, Feb 08, 2010 at 11:24:56AM -0800, Lutz Schumann wrote: Only with the zdb(1M) tool but note that the checksums are NOT of files but of the ZFS blocks. Thanks - bocks, right (doh) - thats what I was missing. Damn it would be so nice :( If you're comparing the current data to a snapshot baseline on the same pool, it just means you need to compare more checksums (several per file), it doesn't invalidate the idea. There may also be other ways of checking quickly that the file data is unmodified since snapshot X, but again it will require looking at zfs internals. This is far from the first use case for an official interface to get at this kind of data. It's quite similar to the question of how to verify send|recv integrity from yesterday, for example. As yet I don't know of a concrete proposal of what such an interface should look like (since there's nothing to borrow from POSIX), let alone an implementation. It more complicated if you're comparing checksums against an external baseline reference (such as from a build) because block sizes and checksum algorithms may vary between pools. However, as you note, that's already catered for by existing tools, so they could work together. -- Dan. pgp3k63UrsurG.pgp Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Intrusion Detection - powered by ZFS Checksumming ?
May be look at rsync and rsync lib (http://librsync.sourceforge.net/) code to see if a ZFS API could be design to help rsync/librsync in the future as well as diff. It might be a good idea for POSIX to have a single checksum and a multi-checksum interface. One problem could be block sizes, if a file is re-written and is the same size it may have different ZFS record sizes within, if it was written over a long period of time (txg's)(ignoring compression), and therefore you could not use ZFS checksum to compare two files. Side Note: It would be nice if ZFS on every txg only wrote full record sizes unless it was short on memory, or a file was closed. Maybe the txg could happen more often if it just scanned for full recordsize's writes and closed files. Or block which had not be altered for three scan's. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Intrusion Detection - powered by ZFS Checksumming ?
Damon Atkins damon_atk...@yahoo.com.au writes: One problem could be block sizes, if a file is re-written and is the same size it may have different ZFS record sizes within, if it was written over a long period of time (txg's)(ignoring compression), and therefore you could not use ZFS checksum to compare two files. the record size used for a file is chosen when that file is created. it can't change. when the default record size for the dataset changes, only new files will be affected. ZFS *must* write a complete record even if you change just one byte (unless it's the tail record of course), since there isn't any better granularity for the block pointers. -- Kjetil T. Homme Redpill Linpro AS - Changing the game ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Intrusion Detection - powered by ZFS Checksumming ?
I would have thought that if I write 1k then ZFS txg times out in 30secs, then the 1k will be written to disk in a 1k record block, and then if I write 4k then 30secs latter txg happen another 4k record size block will be written, and then if I write 130k a 128k and 2k record block will be written. Making the file have record sizes of 1k+4k+128k+2k -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Intrusion Detection - powered by ZFS Checksumming ?
On Feb 8, 2010, at 9:10 PM, Damon Atkins wrote: I would have thought that if I write 1k then ZFS txg times out in 30secs, then the 1k will be written to disk in a 1k record block, and then if I write 4k then 30secs latter txg happen another 4k record size block will be written, and then if I write 130k a 128k and 2k record block will be written. Making the file have record sizes of 1k+4k+128k+2k Close. Once the max record size is achieved, it is not reduced. So the allocation is: 1KB + 4KB + 128KB + 128KB Physical writes tend to be coalesced, which is one reason why a transactional system performs well. -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
