Re: Freebsd filesystem ( hard reboot )
> Exactly, which is why I thought that just bypassing all those > interventions with -y was 'brushing under the carpet'. No? Ah I see. Yes. Given that all bets are off, it's hoping for the best ;) > I realise it would normally be excessively cautious to go for > synchronous mounting, but what about for environments where power supply > is such a major problem? If write caching is disabled (and confirmed to truly be disabled), it should not be needed. So as an added step beyond disabling write caching, it doesn't feel particularly useful. If write caching is still enabled, synchronous writes won't help except perhaps to lower the statistical probability of running into problems (that's just a guess). -- / Peter Schuller PGP userID: 0xE9758B7D or 'Peter Schuller <[EMAIL PROTECTED]>' Key retrieval: Send an E-Mail to [EMAIL PROTECTED] E-Mail: [EMAIL PROTECTED] Web: http://www.scode.org signature.asc Description: This is a digitally signed message part.
Re: Freebsd filesystem ( hard reboot )
Peter Schuller wrote: My understanding from the reading I have done is that in a situation like this where power outages are a danger (and presuably having the UPS signal the server to shut down gracefully is not practical), you need to make the file system as robust as possible in the first place, rather than rely on fsck -y after the event. Doesn't fsck -y rather sweep potential problems under the carpet? fsck is not sweeping potential problems under the carpet, as long as nothing unexpected goes wrong (software bug, hardware problem). The reason fsck works to begin with, is that it is designed to fix specific inconsistencies in the file system that are expected. The file system (takling about UFS here, and other non-journaled file systems that care about this stuff) is designed very carefully such that certain correctable inconsistencies happen, while preventing those that are not correctable. That is, under fully expected circumstances, UFS is intended to require fsck on reboot. But it is NOT intended that fsck find unexpected inconcistencies and ask for operator intervention. Exactly, which is why I thought that just bypassing all those interventions with -y was 'brushing under the carpet'. No? What happens in the event of write caching + power failure, software bug or hardware bugs, is that you end up with semi-random inconsistencies. fsck *may* be able to patch the situation enough for the file system to be usable, but fundamentally all bets are off. First step surely is to *disable* write caching if you have drives that are doing it? For UFS/reiserfs/xfs/jfs/ext3fs/ext2fs, yes. Then consider mounting the file system synchronously. Mind you, I don't know what the scale of the performance loss would be, and whether anyone does this nowadays! Synchronous mounting is not required for consistency (except perhaps for ext2fs; not sure). It is enough that the system does not break the file system's ability to guarantee ordering of certain critical operations, which is why write caching causes a problem (the drive re-orders writes for performance and you end up with B happening before A, but consistency depended on B happening AFTER A). I realise it would normally be excessively cautious to go for synchronous mounting, but what about for environments where power supply is such a major problem? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Freebsd filesystem ( hard reboot )
> My understanding from the reading I have done is that in a situation like > this where power outages are a danger (and presuably having the UPS signal > the server to shut down gracefully is not practical), you need to make the > file system as robust as possible in the first place, rather than rely on > fsck -y after the event. Doesn't fsck -y rather sweep potential problems > under the carpet? fsck is not sweeping potential problems under the carpet, as long as nothing unexpected goes wrong (software bug, hardware problem). The reason fsck works to begin with, is that it is designed to fix specific inconsistencies in the file system that are expected. The file system (takling about UFS here, and other non-journaled file systems that care about this stuff) is designed very carefully such that certain correctable inconsistencies happen, while preventing those that are not correctable. That is, under fully expected circumstances, UFS is intended to require fsck on reboot. But it is NOT intended that fsck find unexpected inconcistencies and ask for operator intervention. What happens in the event of write caching + power failure, software bug or hardware bugs, is that you end up with semi-random inconsistencies. fsck *may* be able to patch the situation enough for the file system to be usable, but fundamentally all bets are off. > First step surely is to *disable* write caching if you have drives that > are doing it? For UFS/reiserfs/xfs/jfs/ext3fs/ext2fs, yes. > Then consider mounting the file system synchronously. Mind you, I don't > know what the scale of the performance loss would be, and whether anyone > does this nowadays! Synchronous mounting is not required for consistency (except perhaps for ext2fs; not sure). It is enough that the system does not break the file system's ability to guarantee ordering of certain critical operations, which is why write caching causes a problem (the drive re-orders writes for performance and you end up with B happening before A, but consistency depended on B happening AFTER A). -- / Peter Schuller PGP userID: 0xE9758B7D or 'Peter Schuller <[EMAIL PROTECTED]>' Key retrieval: Send an E-Mail to [EMAIL PROTECTED] E-Mail: [EMAIL PROTECTED] Web: http://www.scode.org signature.asc Description: This is a digitally signed message part.
Re: Freebsd filesystem ( hard reboot )
> If you are running without write caching turned on (which is the default), That should be, "if you are running WITH write caching turned on". -- / Peter Schuller PGP userID: 0xE9758B7D or 'Peter Schuller <[EMAIL PROTECTED]>' Key retrieval: Send an E-Mail to [EMAIL PROTECTED] E-Mail: [EMAIL PROTECTED] Web: http://www.scode.org signature.asc Description: This is a digitally signed message part.
Re: Freebsd filesystem ( hard reboot )
Randy Ramsdell wrote: > We started using FreeBSD for some network monitoring, but have found > that a hard reboot forces us to answer filesytem questions on boot. Is > there a way to mount each filesystem without this? Or how can we use > FreeBSD in a remote location without needing to intervene in > situatutions like this? Usually, FreeBSD can boot just fine without user interaction when power fails or there's some other cause for a "hard" OS failure (e.g. a kernel panic). That it isn't doing so in your case is significant, because it may point to serious problems. Firstly, what is the server used for? Do you have unusually high load on the file system - a busy file server or a database with lots of writes? - The default is for the root partition to be mounted "noasync" and Soft-updates to be used for all other file systems. Both settings can handle "hard" OS failures pretty good. Did you alter the default settings when creating the file systems? If you don't know, send the output of "mount" command. - How are your drives set up? Are they on a "dumb" disk controller or on a RAID controller? (which one?) Is the RAID controller perhaps doing write caching without a battery unit? - Can you connect the UPS devices with the server machines so they can shutdown gracefully when the power is about to fail? signature.asc Description: OpenPGP digital signature
Re: Freebsd filesystem ( hard reboot )
Randy Ramsdell wrote: Vince wrote: Randy Ramsdell wrote: We started using FreeBSD for some network monitoring, but have found that a hard reboot forces us to answer filesytem questions on boot. Is there a way to mount each filesystem without this? Or how can we use FreeBSD in a remote location without needing to intervene in situatutions like this? This is unusual in my experience, part of the charm of FreeBSD for me is how rarely I have had to interact with fsck thanks to the whole background fsck thing. What version of FreeBSD are you using? Assuming a 5.x or later since you say you've started to use FreeBSD. I am fairly sure it is v6.2 What is the value of background_fsck in /etc/rc.conf ? This isn't set. Was is supposed to be? So far, I have only installed applications we need. And everything seem fine except the reboot issue. This will be an offsite system so I do not want human intervention on boot for power outages or hard reboots. You can also try setting fsck_y_enable="YES" in rc.conf (this will do fsck -y if the initial preen fails.) I will use this. Do you mean by try, that this will work? I assume so. Thanks Vince! Oh, Is there a way to not receive 2 messages for every reply to this thread? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]" Questions on a FreeBSD reboot? Any boxes I have ever built can reboot by themselves with no human intervention. BUT SEE BELOW: This sounds like a BIOS settings issue on a box that has powered off. ~Al Plant - Honolulu, Hawaii - Phone: 808-284-2740 + http://hawaiidakine.com + http://freebsdinfo.org + [EMAIL PROTECTED] + + http://aloha50.net - Supporting - FreeBSD 6.* - 7.* + "All that's really worth doing is what we do for others."- Lewis Carrol ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Freebsd filesystem ( hard reboot )
Randy Ramsdell wrote: Robert Huff wrote: Randy Ramsdell writes: > What is the value of background_fsck in /etc/rc.conf ? >This isn't set. Was is supposed to be? So far, I have only installed applications we need. And everything seem fine except the reboot issue. I'm going to jump in here. Based on what you've said, it sounds like: the system was running there was power outage, the system did not have a UPS when the system rebooted, fsck complained but nothing was done about the errors Let me clarify. All our servers are on UPSes, however the power outage outlasted the UPSes. That is why I stated prolonged power outage. Fsck did complain, but everything was fixed as I sat there and dealt with it. We do not want to deal with it in an offsite location and that is why this thread. IF THAT'S TRUE ... ... then the "seems" in your description is applicable, and should be a red flag. Get someone to the system, reboot it into single user mode, and run (and re-run) fsck until it runs without error. (Answer 'y' to all prompts.) This should be done whenever the file system is not shutdown cleanly. The filesystem is fine. Our set up just does not recover gracefully on hard reboots. (And consider a UPS, even it it only keeps the system alive for the few minutes necessary for a clean shutdown.) Is a dirty file system causing the reboots? No. a power prolonged power failure caused the last shutdown. Possibly; wiser heads than mine would have to lay out scearios. But it may also be responsible for other damage, more subtle but equally unpleasant. A fix is available. Use it. I understand this will not be easy, and I sympathize. Balance that pain against the small but non-trivial chance to catastrophic data loss, and choose wisely. Robert Huff Thanks for the reply. I think I will just set the rc.conf variable to answer "Y" to fsck questions unless there is a better way. A side note, this system has been hard shutdown two times and each time required intervention. We also use several Linux system ( reiserfs and ext3 ) and raely do I have to interact with the systems on reboot. There is a differnce and I am in the fisrt stages trying to understand this. PS. I am confused about why so many people are replying to the list and my personal e-mail. This one was sent to me only. Others were sent to me and the list. Actually, every other reply. Is this normal for the list as I am new as of today? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]" Aloha, If the power outages are prolonged and outlast the UPS you can parallel the small battery in the UPS with a 100 amp hour stationary battery if you have the room. We can get 9 hours on a stationery battery for our servers if we lose power here in Hawaii. (which just happened this week during a winter storm.) ~Al Plant - Honolulu, Hawaii - Phone: 808-284-2740 + http://hawaiidakine.com + http://freebsdinfo.org + [EMAIL PROTECTED] + + http://aloha50.net - Supporting - FreeBSD 6.* - 7.* + "All that's really worth doing is what we do for others."- Lewis Carrol ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Freebsd filesystem ( hard reboot )
On Thu, December 6, 2007 4:42 pm, Randy Ramsdell wrote: > Vince wrote: > >> Randy Ramsdell wrote: >> >> >>> We started using FreeBSD for some network monitoring, but have found >>> that a hard reboot forces us to answer filesytem questions on boot. Is >>> there a way to mount each filesystem without this? Or how can we use >>> FreeBSD in a remote location without needing to intervene in >>> situatutions like this? >>> >>> >> >> This is unusual in my experience, part of the charm of FreeBSD for me >> is how rarely I have had to interact with fsck thanks to the whole >> background fsck thing. What version of FreeBSD are you using? >> >> Assuming a 5.x or later since you say you've started to use FreeBSD. >> >> >> > I am fairly sure it is v6.2 > >> What is the value of background_fsck in /etc/rc.conf ? >> >> >> > This isn't set. Was is supposed to be? So far, I have only installed > applications we need. And everything seem fine except the reboot issue. > This will be an offsite system so I do not want human intervention on > boot for power outages or hard reboots. >> You can also try setting fsck_y_enable="YES" in rc.conf (this will do >> fsck -y if the initial preen fails.) >> >> > I will use this. Do you mean by try, that this will work? I assume so. > > > Thanks Vince! > I should first say that I am pretty new to all this, so my response is intended as a question as much as an answer! My understanding from the reading I have done is that in a situation like this where power outages are a danger (and presuably having the UPS signal the server to shut down gracefully is not practical), you need to make the file system as robust as possible in the first place, rather than rely on fsck -y after the event. Doesn't fsck -y rather sweep potential problems under the carpet? First step surely is to *disable* write caching if you have drives that are doing it? Then consider mounting the file system synchronously. Mind you, I don't know what the scale of the performance loss would be, and whether anyone does this nowadays! As I say, don't rely on my knowledge, but I was prompted to write by your latching on to a suggestion that was probably not intended to be the whole solution. Barnaby Scott ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Freebsd filesystem ( hard reboot )
On Thu, 06 Dec 2007 11:42:22 -0500 Randy Ramsdell <[EMAIL PROTECTED]> wrote: > Vince wrote: > > What is the value of background_fsck in /etc/rc.conf ? > > > > > This isn't set. Was is supposed to be? No, it's on by default, but there's an early check to determine if the background check can be run. That depends on whether the partition has soft-updates turned-on (the default in sysinstall), and whether soft-updates managed to protect the partition during the shut-down. If fsck is asking for user input, those checks would have failed. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Freebsd filesystem ( hard reboot )
> Well any number of things, but the most recent was a prolonged power > outage. It is important to differentiate between expected fsck activity and unexpected. If you are running without write caching turned on (which is the default), a power outtage will constitute a crash from which a file system cannot guarantee to recover unless it takes measure to punch through the write cache at appropriate moments. If you machine shutdown softly as a result of the UPS communicating that power was running out, you "should" not have to fsck (barring other issues in the past). fsck need in these cases would indicate a software bug, or a hardware problem. If on the other hand your machine just lost power when the UPS finally died, you are relying on luck for recovery if you're on ufs/reiserfs/xfs/etc. Some environments will correctly handle this (e.g., ZFS), but most won't. The problem being that drive write caching will prevent the file system from guaranteeing ordering of certain critical operations that must be ordered in order to guarantee successfull recovery to a consistent state. That said, you are not supposed to need to answer interactive questions on boot for all cases of expected inconsistencies. If you are getting prompts as a result of unexpected inconsistencies, that indicates *something* is wrong. -- / Peter Schuller PGP userID: 0xE9758B7D or 'Peter Schuller <[EMAIL PROTECTED]>' Key retrieval: Send an E-Mail to [EMAIL PROTECTED] E-Mail: [EMAIL PROTECTED] Web: http://www.scode.org signature.asc Description: This is a digitally signed message part.
Re: Freebsd filesystem ( hard reboot )
Randy Ramsdell wrote: I think I will just set the rc.conf variable to answer "Y" to fsck questions unless there is a better way. A side note, this system has been hard shutdown two times and each time required intervention. We also use several Linux system ( reiserfs and ext3 ) and raely do I have to interact with the systems on reboot. There is a differnce and I am in the fisrt stages trying to understand this. Might want to be careful with the always answering "Y" thing, though...or at least make sure you have periodic backups ready to go in case the repair further loses data in the process. Is there an area that seems to be needing repairs in particular? Maybe you could find a way to move it to another drive so the system will come up more reliably and then other processes can remotely check the unmountable drive and remount it and start your process monitoring. PS. I am confused about why so many people are replying to the list and my personal e-mail. This one was sent to me only. Others were sent to me and the list. Actually, every other reply. Is this normal for the list as I am new as of today? If looks like if you hit reply all, both addresses are inserted (the list isn't stripping the personal one out). Unless the replier removes the personal address or the reply function on their mail client is only using a primary address (of the list) to send to, you'll get replies on both...depending on the issue some people prefer getting a response personally faster than waiting for the list to send it if for some reason it's bogging down :-) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Freebsd filesystem ( hard reboot )
On Thu, Dec 06, 2007 at 12:50:55PM -0500, Randy Ramsdell wrote: > > > PS. I am confused about why so many people are replying to the list and my > personal e-mail. This one was sent to me only. Others were sent to me and > the list. Actually, every other reply. Is this normal for the list as I am > new as of today? It is standard procedure on this list to reply both to the list and directly to the person you are replying to. Since this is the "general help-line" for FreeBSD, there are many people posting here who are not subscribed to the list, and thus won't see the answers if they are only sent to the list. -- Erik Trulsson [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Freebsd filesystem ( hard reboot )
On Thu, Dec 06, 2007 at 11:42:22AM -0500, Randy Ramsdell wrote: > Oh, Is there a way to not receive 2 messages for every reply to this thread? Something like this in ~/.procmailrc :0 Wh: msgid.lock | $FORMAIL -D 8192 msgid.cache or like this in ~/.mailfilter `reformail -D 8000 duplicate.cache` if ( $RETURNCODE == 0 ) exit I'm sure other delivery agents will have similar functionality. Dan -- Daniel Bye _ ASCII ribbon campaign ( ) - against HTML, vCards and X - proprietary attachments in e-mail / \ pgprdY468w5SZ.pgp Description: PGP signature
Re: Freebsd filesystem ( hard reboot )
Robert Huff wrote: Randy Ramsdell writes: > What is the value of background_fsck in /etc/rc.conf ? > This isn't set. Was is supposed to be? So far, I have only installed applications we need. And everything seem fine except the reboot issue. I'm going to jump in here. Based on what you've said, it sounds like: the system was running there was power outage, the system did not have a UPS when the system rebooted, fsck complained but nothing was done about the errors Let me clarify. All our servers are on UPSes, however the power outage outlasted the UPSes. That is why I stated prolonged power outage. Fsck did complain, but everything was fixed as I sat there and dealt with it. We do not want to deal with it in an offsite location and that is why this thread. IF THAT'S TRUE ... ... then the "seems" in your description is applicable, and should be a red flag. Get someone to the system, reboot it into single user mode, and run (and re-run) fsck until it runs without error. (Answer 'y' to all prompts.) This should be done whenever the file system is not shutdown cleanly. The filesystem is fine. Our set up just does not recover gracefully on hard reboots. (And consider a UPS, even it it only keeps the system alive for the few minutes necessary for a clean shutdown.) Is a dirty file system causing the reboots? No. a power prolonged power failure caused the last shutdown. Possibly; wiser heads than mine would have to lay out scearios. But it may also be responsible for other damage, more subtle but equally unpleasant. A fix is available. Use it. I understand this will not be easy, and I sympathize. Balance that pain against the small but non-trivial chance to catastrophic data loss, and choose wisely. Robert Huff Thanks for the reply. I think I will just set the rc.conf variable to answer "Y" to fsck questions unless there is a better way. A side note, this system has been hard shutdown two times and each time required intervention. We also use several Linux system ( reiserfs and ext3 ) and raely do I have to interact with the systems on reboot. There is a differnce and I am in the fisrt stages trying to understand this. PS. I am confused about why so many people are replying to the list and my personal e-mail. This one was sent to me only. Others were sent to me and the list. Actually, every other reply. Is this normal for the list as I am new as of today? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Freebsd filesystem ( hard reboot )
Vince wrote: Randy Ramsdell wrote: We started using FreeBSD for some network monitoring, but have found that a hard reboot forces us to answer filesytem questions on boot. Is there a way to mount each filesystem without this? Or how can we use FreeBSD in a remote location without needing to intervene in situatutions like this? This is unusual in my experience, part of the charm of FreeBSD for me is how rarely I have had to interact with fsck thanks to the whole background fsck thing. What version of FreeBSD are you using? Assuming a 5.x or later since you say you've started to use FreeBSD. I am fairly sure it is v6.2 What is the value of background_fsck in /etc/rc.conf ? This isn't set. Was is supposed to be? So far, I have only installed applications we need. And everything seem fine except the reboot issue. This will be an offsite system so I do not want human intervention on boot for power outages or hard reboots. You can also try setting fsck_y_enable="YES" in rc.conf (this will do fsck -y if the initial preen fails.) I will use this. Do you mean by try, that this will work? I assume so. Thanks Vince! Oh, Is there a way to not receive 2 messages for every reply to this thread? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Freebsd filesystem ( hard reboot )
Randy Ramsdell wrote: > We started using FreeBSD for some network monitoring, but have found > that a hard reboot forces us to answer filesytem questions on boot. Is > there a way to mount each filesystem without this? Or how can we use > FreeBSD in a remote location without needing to intervene in > situatutions like this? > This is unusual in my experience, part of the charm of FreeBSD for me is how rarely I have had to interact with fsck thanks to the whole background fsck thing. What version of FreeBSD are you using? Assuming a 5.x or later since you say you've started to use FreeBSD. What is the value of background_fsck in /etc/rc.conf ? You can also try setting fsck_y_enable="YES" in rc.conf (this will do fsck -y if the initial preen fails.) Vince > Thanks, > Randy Ramsdell > Unix Systems Administrator > ___ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to > "[EMAIL PROTECTED]" ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Freebsd filesystem ( hard reboot )
Bart Silverstrim wrote: Randy Ramsdell wrote: We started using FreeBSD for some network monitoring, but have found that a hard reboot forces us to answer filesytem questions on boot. Is there a way to mount each filesystem without this? Or how can we use FreeBSD in a remote location without needing to intervene in situatutions like this? What's causing the hard reboots? Is it on a UPS? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]" Well any number of things, but the most recent was a prolonged power outage. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Freebsd filesystem ( hard reboot )
Randy Ramsdell wrote: We started using FreeBSD for some network monitoring, but have found that a hard reboot forces us to answer filesytem questions on boot. Is there a way to mount each filesystem without this? Or how can we use FreeBSD in a remote location without needing to intervene in situatutions like this? What's causing the hard reboots? Is it on a UPS? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
