Re: Hammer FS: imposing a size limit on PFS?
He, * Matthew Dillon wrote: > > The idea with HAMMER is you just create one big filesystem and use the > PFS functionality to break it up into separate management domains. > Currently a size limit may not be placed on a PFS. This raises an important question: what to do if one uses HAMMER on a multi user box? It should be possible for a malicious user to fill up his $HOME on the /home PFS and thus blow the fs up, or? Correct me if I'm wrong. If I'm right what is the policy for admins using HAMMER on a machine with a lot of user shells? Regards Matthias PS: Someone interested to hack quota support for HAMMER? :)
Re: BFBI OTT bug or limitation? UPDATE2
Responded to off-list, as it is:
A) tedious the way I did it, and not certain to have been free of
hardware glitches..
B) a lead-pipe cinch that Matt will come up wth a better test methodology.
;-)
Bill
Matthew Dillon wrote:
:Bill Hacker wrote:
:> Top-posting to my own post ...
:
:Again.
:
:Reproduced the original verbose crash. Only the last line is the same as
:below.
:
:Failed to set up the HP-200LX serial, so will run it again...
:
:Bill
:
:> :-(
:>
:> du -h > dulist
:>
:> Two more runs, one OK with hammer mirror-stream over ssh NOT running,
:> second run with it mirroring a nearly empty dirtree (five static,
:> one-line text files only), runs for several minutes, then drops into
:> debugger with a mere three lines, rather than the original
:> scrolled-off-the screen;
:>
:> CRC DATA @ 900a3b15b280/128 FAILED
:> Debuger ("CRCFAILED: DATA")
:> Stopped atDebugger+0x34:movb$0, in_Debugger.3970
:>
:> But this does not seem to relate. Could be an NATACONTROL + old HHD I/O
:> error artifact.
:>
:> First looong err message had to do with vnodes..
:>
:> More testing to do, then will swap-in a newer 500 GB SATA.
:>
:> Bill
Bill, could you be more precise on exactly which tests you are running?
If I understand it you are doing some combination of bonnie++ and
mirroring but you are explicitly filling up the disk and allowing it
to hit full-disk error conditions.
It sounds like a test I should be running here, particularly if you
managed to get a CRCFAILED panic out of it. I want to try to reproduce
that.
It is possible that the CRCFAILED is due to an actual I/O screwup,
since we're talking PATA drives here. This could be particularly
true if they are master/slave on the same controller. However, my
first assumption is that it is a software bug related to pounding
it while it is full.
Note that HAMMER is not designed to run on an 8GB partition. The
minimum is 50GB. That said, HAMMER should never get a CRC error
even if running on an 8GB partition. So I want to track it down.
-Matt
Re: Hammer FS: imposing a size limit on PFS?
Matthew Dillon wrote: Our installer support for HAMMER isn't advanced enough yet. What we really want is a UFS /boot, swap, and then a HAMMER root that covers everything else. I would (and have) taken that a step farther. 'once upon a time' '/usr' was not part of the core essential. It really was 'userland' Long since, far too many of the *system's* 'needful things' in the way of binary and libs migrated there. Recall my push to get a decent static-compiled editor into the root partition so we could at least edit a FUBAR'ed /etc/rc.conf w/o having to manually mount a (potentially damaged) /usr. 'These days' one gains a bit of respect for NetBSD / OpenBSD plutting things into /usr/pkg rather than /usr/local, if only to keep them out of the way of 'real userland' - and even looks yearningly at Linux use of '/opt' Reality is that a 'healthy' system needs '/usr' (libs and binaries) and '/var' (pidfiles, logs, and spools) to be mounted more or less 'at all costs'. Ergo, one wants to push anything that really IS userland and user-app, or 'production use' specific out into bespoke mounts. True whether the box is to be used for familiarization, learning, experimenting, OR 'production'. And regardless of fs chosen The idea with HAMMER is you just create one big filesystem and use the PFS functionality to break it up into separate management domains. Currently a size limit may not be placed on a PFS. -Matt I want my 'core' to be as damage-resistant as can be. So long as it IS such, and can boot and mount rapidly and respond to console - better yet ssh from afar - I have the wherewithal to manage, repair, or even nuke and reinstall - all the rest. Ergo, absent a 'netboot' or flash/USB boot - I submit: 'Best Current Practice'; Minimum with one device: - A modest 'slice' for the OS install, partioned and UFS, OR 'shared' with hammer. - One or more SEPARATE partons if not SLICES for hammer-as-bulk storage, application support, etc. IOW - not entangled in any way with '/usr', '/var'. You can wipe it and start over over-the-wire, as the 'core' is isolated. Better yet - multiple devices, where second and subsequent devices where hammer owns the entire device. If we cannot isolate and protect the 'core' within a hammer PFS, then we should not put it into the same PFS 'family' and open it to overflow or damage. JM-scar-tissue's-2CW - but we have found this 'safe' from CP/M 1.X onward. Logs and spool aside, 'core' has slow or no rate of change. Bill
Re: mirror-stream over ssh w/o 'root' privs
On Wed, 25 Feb 2009 10:06:14 -0800 (PST) Matthew Dillon wrote: > Well, you can always write the stream out to a file I guess, > but the basic problem here is that the mirroring stream is a > B-Tree layer stream. If we can't trust the source there's no > point running the stream onto an actual filesystem without some > major auditing of its contents. I don't think that's the real problem, at least it's not the bit that makes me nervous. The bit that makes me nervous is opening root access by ssh at all, it's possible to lock down root ssh access but it's fiddly and a mistake in doing so leaves a gaping security hole. I think what I'd like to be able to do is open a tunnel between the machines using an unprivileged user and attach a hammer process to each end. This can of course be done using tools like hose and faucet but I found them clumsy to use with jscan. Perhaps something like a -p argument for mirror-read, mirror-read-stream and mirror-write - for mirror-write it would establish a listening socket on localhost that would only accept one connection at a time, for the read operations it would connect to the specified port on localhost. If you think it's a good idea I'm pretty sure I can implement it. -- Steve O'Hara-Smith | Directable Mirror Arrays C:>WIN | A better way to focus the sun The computer obeys and wins.|licences available see You lose and Bill collects. |http://www.sohara.org/
Re: Hammer FS: imposing a size limit on PFS?
Thank you very much! Kind regards, Jurij Kovacic Matthew Dillon wrote: Our installer support for HAMMER isn't advanced enough yet. What we really want is a UFS /boot, swap, and then a HAMMER root that covers everything else. The idea with HAMMER is you just create one big filesystem and use the PFS functionality to break it up into separate management domains. Currently a size limit may not be placed on a PFS. -Matt
Re: mirror-stream over ssh w/o 'root' privs
Well, you can always write the stream out to a file I guess, but the basic problem here is that the mirroring stream is a B-Tree layer stream. If we can't trust the source there's no point running the stream onto an actual filesystem without some major auditing of its contents. Auditing the stream is certainly possible, maybe a GSOC project? Or perhaps the solution here is to integrate the auditing with the (upcoming) fsck functionality. Hmm. -Matt
Re: BFBI OTT bug or limitation? UPDATE2
:Bill Hacker wrote:
:> Top-posting to my own post ...
:
:Again.
:
:Reproduced the original verbose crash. Only the last line is the same as
:below.
:
:Failed to set up the HP-200LX serial, so will run it again...
:
:Bill
:
:> :-(
:>
:> du -h > dulist
:>
:> Two more runs, one OK with hammer mirror-stream over ssh NOT running,
:> second run with it mirroring a nearly empty dirtree (five static,
:> one-line text files only), runs for several minutes, then drops into
:> debugger with a mere three lines, rather than the original
:> scrolled-off-the screen;
:>
:> CRC DATA @ 900a3b15b280/128 FAILED
:> Debuger ("CRCFAILED: DATA")
:> Stopped atDebugger+0x34:movb$0, in_Debugger.3970
:>
:> But this does not seem to relate. Could be an NATACONTROL + old HHD I/O
:> error artifact.
:>
:> First looong err message had to do with vnodes..
:>
:> More testing to do, then will swap-in a newer 500 GB SATA.
:>
:> Bill
Bill, could you be more precise on exactly which tests you are running?
If I understand it you are doing some combination of bonnie++ and
mirroring but you are explicitly filling up the disk and allowing it
to hit full-disk error conditions.
It sounds like a test I should be running here, particularly if you
managed to get a CRCFAILED panic out of it. I want to try to reproduce
that.
It is possible that the CRCFAILED is due to an actual I/O screwup,
since we're talking PATA drives here. This could be particularly
true if they are master/slave on the same controller. However, my
first assumption is that it is a software bug related to pounding
it while it is full.
Note that HAMMER is not designed to run on an 8GB partition. The
minimum is 50GB. That said, HAMMER should never get a CRC error
even if running on an 8GB partition. So I want to track it down.
-Matt
Re: OT - was Hammer or ZFS based backup, encryption
Generally speaking the idea with HAMMER's snapshotting and mirroring is that everything is based on transaction-ids stored in the B-Tree. The mirroring functionality does not require snapshotting per-say, because EVERY sync HAMMER does to the media (including the automatic filesystem syncs done by the kernel every 30-60 seconds) is effectively a snapshot. There is a downside to the way HAMMER manages its historical data store and it is unclear how much of burden this will wind up being without some specific tests. The downside is that the historical information is stored in the HAMMER B-Tree side-by-side with current information. If you make 50,000 modifications to the same offset within a file, for example, with a fsync() inbetween each one, and assuming you don't prune the filesystem, then you will have 50,000 records for that HAMMER data block in the B-Tree. This can be optimized... HAMMER doesn't have to scan 50,000 B-Tree elements. It can seek to the last (most current) one when it traverses the tree. I may not be doing that yet but there is no data structure limitation that would prevent it. Even with the optimization there will certainly be some overhead. The mitigating factor is, of course, that the HAMMER B-Tree is pruned every night to match the requested snapshot policy. -- It would be cool if someone familiar with both ZFS's mirroring and HAMMER's mirroring could test the feature and performance set. What I like most about HAMMER's mirroring is that the mirroring target can have a different history retention policy then the master. HAMMER's current mirror streaming feature is also pretty cool if I do say so myself. Since incremental mirroring is so fast, the hammer utility can poll for changes every few seconds and since the stream isn't queued it can be killed and restarted at any time. Network outages don't really effect it. I also added a very cool feature to the hammer mirror-stream directive which allows you to limit the bandwidth, preventing the mirroring operation from interfering with production performance. -Matt
Re: Hammer FS: imposing a size limit on PFS?
Our installer support for HAMMER isn't advanced enough yet. What we really want is a UFS /boot, swap, and then a HAMMER root that covers everything else. The idea with HAMMER is you just create one big filesystem and use the PFS functionality to break it up into separate management domains. Currently a size limit may not be placed on a PFS. -Matt
Re: New mirror in Russia
On 25/02/2009, Matthias Schmidt wrote: > Hi, > > * Constantine A. Murenin wrote: > > On 22/02/2009, Justin C. Sherrill wrote: > > > > > Speaking of mirrors in Russia, there is no internet in the Asian part > > of Russia, let alone any mirrors, so it's a bit strange that Russian > > mirrors are still listed in the Asian section of the mirrors page. :) > > > I was the one who listed Russia in the Asian section. The reason was > pretty simple, more parts of Russia are part of the Asian continent so > the majority won :) See http://en.wikipedia.org/wiki/Ural_Mountains See http://en.wikipedia.org/wiki/European_Russia. The majority of active internet users in Russia reside in Moscow alone. I suggest we rename Russia to Moscow, to be more informative, and list it under Europe. Speaking of which, in practical terms, Moscow isn't even Russia (well, this is an inside joke :), so technically, this mere suggestion is not even meant to be taken as a joke. Seriously, though, all the known DragonFly mirrors in Russia are located specifically in Moscow for the reasons specified above, so please do move them from Asia to Europe to avoid the confusion. :-) C.
Re: New mirror in Russia
Hi, * Constantine A. Murenin wrote: > On 22/02/2009, Justin C. Sherrill wrote: > > Speaking of mirrors in Russia, there is no internet in the Asian part > of Russia, let alone any mirrors, so it's a bit strange that Russian > mirrors are still listed in the Asian section of the mirrors page. :) I was the one who listed Russia in the Asian section. The reason was pretty simple, more parts of Russia are part of the Asian continent so the majority won :) See http://en.wikipedia.org/wiki/Ural_Mountains I'm pretty sure everybody will find a Russian mirror if (s)he wants, so please do not start a bikeshed about the order of the mirror page. I'm very happy that we now have a working and ordered mirror page at all. Cheers and thanks Matthias
