Re: ufs recovery

[Polytropon]

On Tue, 10 Sep 2013 22:38:46 +0200, Laszlo Danielisz wrote:
> Dear All,
> 
> It looks like I'm able to recover all of the deleted files.
> I'm using UFS Explorer Professional Recovery, I'm working on it
> for more than 30 hours, its a long time but it works!

If recovery works, time does not matter. Success does. :-)




-- 
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: ufs recovery

[Laszlo Danielisz]

Dear All,

It looks like I'm able to recover all of the deleted files.
I'm using UFS Explorer Professional Recovery, I'm working on it for more than 
30 hours, its a long time but it works!

Yaaay!
Laci

Sent from my mobile. 

On 2013.09.09., at 0:36, kpn...@pobox.com wrote:

> On Mon, Sep 09, 2013 at 12:03:52AM +0200, Roland Smith wrote:
>> On Sun, Sep 08, 2013 at 10:46:35AM +0200, Laszlo Danielisz wrote:
>>> Hi,
>>> 
>>> By mistake I forgot to edit my crontab on my FreeBSD 8.3 after I took out
>>> one of the hard drives.  I had a little rsync script which I used to
>>> synchronise a directory between those two hard drives, because one of the
>>> hard drives were not present anymore and rsync had the --delete parameter I
>>> end up deleting the whole directory, of course with precious informations.
>> 
>> Ouch. I have a similar procedure going. But I put it in a shell-script that
>> mounts the destination _and_ checks if the destination is properly mounted 
>> _before_
>> starting the rsync. I would suggest you do something similar in the future.
>> 
>> Just to be clear, was the information deleted from _both_ harddisks?
> 
> How about using UFS labels and putting the label device into the fstab
> instead of the raw block device? Then if the situation happens again
> the changed block device names will not matter.
> 
> I believe "tunefs -L" will do the trick.
> -- 
> "A method for inducing cats to exercise consists of directing a beam of
> invisible light produced by a hand-held laser apparatus onto the floor ...
> in the vicinity of the cat, then moving the laser ... in an irregular way
> fascinating to cats,..." -- US patent 5443036, "Method of exercising a cat"
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: ufs recovery

[Roland Smith]

On Sun, Sep 08, 2013 at 10:46:35AM +0200, Laszlo Danielisz wrote:
> Hi,
> 
> By mistake I forgot to edit my crontab on my FreeBSD 8.3 after I took out
> one of the hard drives.  I had a little rsync script which I used to
> synchronise a directory between those two hard drives, because one of the
> hard drives were not present anymore and rsync had the --delete parameter I
> end up deleting the whole directory, of course with precious informations.

Ouch. I have a similar procedure going. But I put it in a shell-script that
mounts the destination _and_ checks if the destination is properly mounted 
_before_
starting the rsync. I would suggest you do something similar in the future.

Just to be clear, was the information deleted from _both_ harddisks?

> I have ufs on the hdd, after the "accident" I've turned off the computer to
> avoid any writings on the disk.  Do you have any idea how can I recover the
> lost directory?

Do you perhaps have a snapshot of the filesystem in question available? If so,
you can mount that and restore the files from it. See e.g.:
http://www.freebsd.org/doc/handbook/snapshots.html

If all else fails, have a look at sysutils/sleuthkit. Restoring deleted files
on UFS is very difficult, but you can find some pointers here:

http://wiki.sleuthkit.org/index.php?title=FS_Analysis#Manual_Deleted_File_Recovery

It helps if you know what kind of data is contained in the deleted file.

To prevent this from happening again, make regular backups e.g. to an external
harddisk that you use for that purpose alone.


Roland
-- 
R.F.Smith   http://rsmith.home.xs4all.nl/
[plain text _non-HTML_ PGP/GnuPG encrypted/signed email much appreciated]
pgp: 1A2B 477F 9970 BA3C 2914  B7CE 1277 EFB0 C321 A725 (KeyID: C321A725)


pgp56wP0pv4BG.pgp
Description: PGP signature

Re: ufs recovery

[Polytropon]

On Sun, 8 Sep 2013 11:39:08 +0200, Laszlo Danielisz wrote:
> Hi Frank,
> 
> Thank you very much for the information!
> Meanwhile I've found this software: http://www.ufsexplorer.com/, I'm going to 
> give a try.

That program was on my "famous list of recovery tools for
futile attempts". :-)

I may say that I have the same problem (of unclear origin).
Files have been removed, but the assumption that the data
could still be somewhere is alive. In such situations, you
would usually have two choices:

1. money

Get as much money as you can. You'll need it. Several 1000
euro / dollar / local currency will buy you service at a
company specialized in recovery. There is no guarantee they
will be successful.

2. time

You invest time in learning how UFS works. There are many
excellent articles (especially the authoritative one by
M. K. McKusick). You try out different tools (with different
scope). If you are lucky, you get your data back. (I was
lucky once, got my data back!)

There are _many_ good tools around. Most of them are free,
so you don't need to invest massive amounts of money in a
repeating "trial & error" process.



Allow me to repeat my list (which gets a little bit modified
each time I post it to this list):

OS tools:

fetch -rR 
recoverdisk

Ports collection:

ddrescue
dd_rescue   <- use this to create images to work with
magicrescue
testdisk<- restores content
recoverjpeg
foremost
photorec
ffs2recov
scan_ffs
tsk <- The Sleuth Kit
fls
dls
ils
    autopsy

There are some commercial tools worth mentioning: "UFS Explorer"
can be run in wine. It probably won't restore your data, but it
can be used to determine if there is something to restore. Also
consider "R-Studio" and "R-Studio Emergency" (live CD). Those
offer free versions that can be used for testing.

Finally, I'd like to mention The Sleuth Kit. It's one of the
most powerful toolsets, also used in forensics and investigation.

As I said, I ran into a similar problem (files deleted). Maybe
you can find this discussion thread in the archives and gain
some more inspiration from it. A massive data loss (meanwhile
cured!) brought me to this list, so I continue to spread my
experience about recovery when needed. :-)



Good luck!



-- 
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: ufs recovery

[Laszlo Danielisz]

Thanks Graeme, 

Also my vga card is broken, probably tomorrow I'm getting a new one and I can 
give a try. 




On 2013 September 8 Sunday at 6:16 PM, Graeme Dargie wrote:

> Assuming the disk has not been written to, then making a full DD image of the 
> drive is your 1st step, then make a copy of that DD image and store it 
> somewhere safe in case something goes wrong with the one you are working on. 
> You can try Foremost which can recover data even deleted stuff from a DD 
> image, there was another package that works on the command line but I cannot 
> recall the name of it just now. Your success rate will depend on the type of 
> data you are trying to recover, from experience foremost works better on 
> certain types of files. 
> 
> 
> Regards
> Graeme Dargie
> -Original Message-
> From: owner-freebsd-questi...@freebsd.org 
> [mailto:owner-freebsd-questi...@freebsd.org] On Behalf Of Laszlo Danielisz
> Sent: 08 September 2013 09:47
> To: FreeBSD Questions
> Subject: ufs recovery
> 
> Hi,
> 
> By mistake I forgot to edit my crontab on my FreeBSD 8.3 after I took out one 
> of the hard drives.
> I had a little rsync script which I used to synchronise a directory between 
> those two hard drives, because one of the hard drives were not present 
> anymore and rsync had the --delete parameter I end up deleting the whole 
> directory, of course with precious informations. 
> 
> I have ufs on the hdd, after the "accident" I've turned off the computer to 
> avoid any writings on the disk.
> Do you have any idea how can I recover the lost directory? 
> 
> Thank you!
> Laci
> ___
> freebsd-questions@freebsd.org mailing list 
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org 
> (mailto:freebsd-questions-unsubscr...@freebsd.org)"
> ___
> freebsd-questions@freebsd.org (mailto:freebsd-questions@freebsd.org) mailing 
> list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org 
> (mailto:freebsd-questions-unsubscr...@freebsd.org)"
> 
> 


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

RE: ufs recovery

[Graeme Dargie]

Assuming the disk has not been written to, then making a full DD image of the 
drive is your 1st step, then make a copy of that DD image and store it 
somewhere safe in case something goes wrong with the one you are working on. 
You can try Foremost which can recover data even deleted stuff from a DD image, 
there was another package that works on the command line but I cannot recall 
the name of it just now. Your success rate will depend on the type of data you 
are trying to recover, from experience foremost works better on certain types 
of files. 


Regards
Graeme Dargie
-Original Message-
From: owner-freebsd-questi...@freebsd.org 
[mailto:owner-freebsd-questi...@freebsd.org] On Behalf Of Laszlo Danielisz
Sent: 08 September 2013 09:47
To: FreeBSD Questions
Subject: ufs recovery

Hi,

By mistake I forgot to edit my crontab on my FreeBSD 8.3 after I took out one 
of the hard drives.
I had a little rsync script which I used to synchronise a directory between 
those two hard drives, because one of the hard drives were not present anymore 
and rsync had the --delete parameter I end up deleting the whole directory, of 
course with precious informations. 

I have ufs on the hdd, after the "accident" I've turned off the computer to 
avoid any writings on the disk.
Do you have any idea how can I recover the lost directory? 

Thank you!
Laci
___
freebsd-questions@freebsd.org mailing list 
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: ufs recovery

[Frank Leonhardt]

On 08/09/2013 10:39, Laszlo Danielisz wrote:
On 2013.09.08., at 11:07, Frank Leonhardt <mailto:freebsd-...@fjl.co.uk>> wrote:



On 08/09/2013 09:46, Laszlo Danielisz wrote:

Hi,

By mistake I forgot to edit my crontab on my FreeBSD 8.3 after I 
took out one of the hard drives.
I had a little rsync script which I used to synchronise a directory 
between those two hard drives, because one of the hard drives were 
not present anymore and rsync had the --delete parameter I end up 
deleting the whole directory, of course with precious informations.


I have ufs on the hdd, after the "accident" I've turned off the 
computer to avoid any writings on the disk.

Do you have any idea how can I recover the lost directory?

Thank you!
Laci



Hi Laci,

I'm sorry to have to tell you that recovering UFS is not easy. It's 
not like MS-DOS or NFTS at all in that respect. When you delete from 
UFS it removes inode data and adds the space released to the free 
block list. It's a one-way process; there is no journalling and no 
way to undo any of it.


I don't know of any public domain utilities that will do what you 
need. EnCase can do something with UFS, and a utility called "Raise 
Data Recovery" will get stuff from damaged disks. This isn't the same 
as getting back deleted files.


The only option I've ever found to work is to scan the disk's free 
blocks (all of them in your case) with a utility that recognises 
specific file formats and pieces the file together using the contents 
it reads from each block, using "best guess" and manual choice to 
decide which the next block is. This is no joke if you've lost a lot 
of files, but worth it if you have one or two vital ones amongst them.


Sorry I can't be of any more comfort. As I'm sure someone will chip 
in, there are things you can do before the event.


Regards, Frank.



Hi Frank,

Thank you very much for the information!
Meanwhile I've found this software: http://www.ufsexplorer.com/, I'm 
going to give a try.



Regards,
Laci



That's the company that produces the "Raise Data Recovery" product I 
mentioned. However, I believe it's better for recovering data from a 
broken FS in the case of UFS2, not for undeleteing a whole 
directory/disk full of "deliberately" deleted files. I just checked, and 
it has a try-before-buy feature so you have nothing to lose. Good luck, 
and please keep us informed!


FWIW I use Pandora for jobs similar to this, although it doesn't 
specifically support UFS. Piriform's Recuva also has its uses. But where 
UFS is involved I've failed to find a magic solution - just recovery 
from a backup unless it's one or two odd files. About the only thing you 
have going for you with UFS is the directory retains the file name after 
deletion if you haven't created any new files over it. But the inode 
(where it is on the disk) is another matter.


Regards, Frank.

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: ufs recovery

[Laszlo Danielisz]

Hi Frank,

Thank you very much for the information!
Meanwhile I've found this software: http://www.ufsexplorer.com/, I'm going to 
give a try.


Regards,
Laci

Sent from my mobile. 

On 2013.09.08., at 11:07, Frank Leonhardt  wrote:

> On 08/09/2013 09:46, Laszlo Danielisz wrote:
>> Hi,
>> 
>> By mistake I forgot to edit my crontab on my FreeBSD 8.3 after I took out 
>> one of the hard drives.
>> I had a little rsync script which I used to synchronise a directory between 
>> those two hard drives, because one of the hard drives were not present 
>> anymore and rsync had the --delete parameter I end up deleting the whole 
>> directory, of course with precious informations.
>> 
>> I have ufs on the hdd, after the "accident" I've turned off the computer to 
>> avoid any writings on the disk.
>> Do you have any idea how can I recover the lost directory?
>> 
>> Thank you!
>> Laci
> 
> Hi Laci,
> 
> I'm sorry to have to tell you that recovering UFS is not easy. It's not like 
> MS-DOS or NFTS at all in that respect. When you delete from UFS it removes 
> inode data and adds the space released to the free block list. It's a one-way 
> process; there is no journalling and no way to undo any of it.
> 
> I don't know of any public domain utilities that will do what you need. 
> EnCase can do something with UFS, and a utility called "Raise Data Recovery" 
> will get stuff from damaged disks. This isn't the same as getting back 
> deleted files.
> 
> The only option I've ever found to work is to scan the disk's free blocks 
> (all of them in your case) with a utility that recognises specific file 
> formats and pieces the file together using the contents it reads from each 
> block, using "best guess" and manual choice to decide which the next block 
> is. This is no joke if you've lost a lot of files, but worth it if you have 
> one or two vital ones amongst them.
> 
> Sorry I can't be of any more comfort. As I'm sure someone will chip in, there 
> are things you can do before the event.
> 
> Regards, Frank.
> 
> 
> 
> ___
> freebsd-questions@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: ufs recovery

[Frank Leonhardt]

On 08/09/2013 09:46, Laszlo Danielisz wrote:

Hi,

By mistake I forgot to edit my crontab on my FreeBSD 8.3 after I took out one 
of the hard drives.
I had a little rsync script which I used to synchronise a directory between 
those two hard drives, because one of the hard drives were not present anymore 
and rsync had the --delete parameter I end up deleting the whole directory, of 
course with precious informations.

I have ufs on the hdd, after the "accident" I've turned off the computer to 
avoid any writings on the disk.
Do you have any idea how can I recover the lost directory?

Thank you!
Laci



Hi Laci,

I'm sorry to have to tell you that recovering UFS is not easy. It's not 
like MS-DOS or NFTS at all in that respect. When you delete from UFS it 
removes inode data and adds the space released to the free block list. 
It's a one-way process; there is no journalling and no way to undo any 
of it.


I don't know of any public domain utilities that will do what you need. 
EnCase can do something with UFS, and a utility called "Raise Data 
Recovery" will get stuff from damaged disks. This isn't the same as 
getting back deleted files.


The only option I've ever found to work is to scan the disk's free 
blocks (all of them in your case) with a utility that recognises 
specific file formats and pieces the file together using the contents it 
reads from each block, using "best guess" and manual choice to decide 
which the next block is. This is no joke if you've lost a lot of files, 
but worth it if you have one or two vital ones amongst them.


Sorry I can't be of any more comfort. As I'm sure someone will chip in, 
there are things you can do before the event.


Regards, Frank.



___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

ufs recovery

[Laszlo Danielisz]

Hi,

By mistake I forgot to edit my crontab on my FreeBSD 8.3 after I took out one 
of the hard drives.
I had a little rsync script which I used to synchronise a directory between 
those two hard drives, because one of the hard drives were not present anymore 
and rsync had the --delete parameter I end up deleting the whole directory, of 
course with precious informations. 

I have ufs on the hdd, after the "accident" I've turned off the computer to 
avoid any writings on the disk.
Do you have any idea how can I recover the lost directory? 

Thank you!
Laci
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: Removing UFS label from root

[Kevin Oberman]

On Sun, Sep 9, 2012 at 8:00 PM, Warren Block  wrote:
> On Sun, 9 Sep 2012, Kevin Oberman wrote:
>
>> On Sun, Sep 9, 2012 at 6:49 PM, Warren Block  wrote:
>>>
>>> On Sun, 9 Sep 2012, Warren Block wrote:
>>>
>>>> On Sun, 9 Sep 2012, Kevin Oberman wrote:
>>>>
>>>>> I added a label to my root fs some time ago. I really prefer the gpt
>>>>> label and I have added it, but I can't figure out how to remove the
>>>>> ufs label.
>>>>>
>>>>> I boot single user and run 'tunefs -L "" /dev/ada1p2' but I get an
>>>>> "unable to write superblock" error. This is before mounting the drive
>>>>> RW, but it is, of course, mounted RO.
>>>>
>>>>
>>>>
>>>> This worked in a VM when I tested it just now.  Maybe running fsck on
>>>> that
>>>> filesystem first will help.
>>
>>
>> Already tried that.
>>
>> I know I was able to label it originally, so it does seem like
>> something must have changed, but I don't know what. I did use gpart to
>> expand my /usr partition and then used growfs to expand the FS, but I
>> did nothing to root and I was able to remove the labels on all of the
>> other partitions.
>
>
> Might there be a duplicate filesystem label of "root" on ada0 or another
> disk?

No joy. I checked all ufs partitions and none are "root" except for
/dev/ada1p2. All partitions on ada0 are ntfs. (It's my Windows disk.)

But, thanks for trying, Warren. I appreciate it.
-- 
R. Kevin Oberman, Network Engineer
E-mail: kob6...@gmail.com
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: Removing UFS label from root

[Warren Block]

On Sun, 9 Sep 2012, Kevin Oberman wrote:


On Sun, Sep 9, 2012 at 6:49 PM, Warren Block  wrote:

On Sun, 9 Sep 2012, Warren Block wrote:


On Sun, 9 Sep 2012, Kevin Oberman wrote:


I added a label to my root fs some time ago. I really prefer the gpt
label and I have added it, but I can't figure out how to remove the
ufs label.

I boot single user and run 'tunefs -L "" /dev/ada1p2' but I get an
"unable to write superblock" error. This is before mounting the drive
RW, but it is, of course, mounted RO.



This worked in a VM when I tested it just now.  Maybe running fsck on that
filesystem first will help.


Already tried that.

I know I was able to label it originally, so it does seem like
something must have changed, but I don't know what. I did use gpart to
expand my /usr partition and then used growfs to expand the FS, but I
did nothing to root and I was able to remove the labels on all of the
other partitions.


Might there be a duplicate filesystem label of "root" on ada0 or another 
disk?

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: Removing UFS label from root

[Kevin Oberman]

On Sun, Sep 9, 2012 at 6:49 PM, Warren Block  wrote:
> On Sun, 9 Sep 2012, Warren Block wrote:
>
>> On Sun, 9 Sep 2012, Kevin Oberman wrote:
>>
>>> I added a label to my root fs some time ago. I really prefer the gpt
>>> label and I have added it, but I can't figure out how to remove the
>>> ufs label.
>>>
>>> I boot single user and run 'tunefs -L "" /dev/ada1p2' but I get an
>>> "unable to write superblock" error. This is before mounting the drive
>>> RW, but it is, of course, mounted RO.
>>
>>
>> This worked in a VM when I tested it just now.  Maybe running fsck on that
>> filesystem first will help.

Already tried that.

I know I was able to label it originally, so it does seem like
something must have changed, but I don't know what. I did use gpart to
expand my /usr partition and then used growfs to expand the FS, but I
did nothing to root and I was able to remove the labels on all of the
other partitions.

> Alternative thought: is ada1p2 actually a UFS filesystem?

Well, the fact that it has a ufs label, fstab show ufs and fsck runs
fsck_ufs are strong hints.
2. Name: ada1p2
   Mediasize: 1073741824 (1.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   rawuuid: 43f0eafd-ba3a-11e0-b70a-f0def166a11e
   rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b
   label: root
   length: 1073741824
   offset: 86016
   type: freebsd-ufs
   index: 2
   end: 2097319
   start: 168
-- 
R. Kevin Oberman, Network Engineer
E-mail: kob6...@gmail.com
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: Removing UFS label from root

[Warren Block]

On Sun, 9 Sep 2012, Warren Block wrote:


On Sun, 9 Sep 2012, Kevin Oberman wrote:


I added a label to my root fs some time ago. I really prefer the gpt
label and I have added it, but I can't figure out how to remove the
ufs label.

I boot single user and run 'tunefs -L "" /dev/ada1p2' but I get an
"unable to write superblock" error. This is before mounting the drive
RW, but it is, of course, mounted RO.


This worked in a VM when I tested it just now.  Maybe running fsck on that 
filesystem first will help.


Alternative thought: is ada1p2 actually a UFS filesystem?
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: Removing UFS label from root

[Warren Block]

On Sun, 9 Sep 2012, Kevin Oberman wrote:


I added a label to my root fs some time ago. I really prefer the gpt
label and I have added it, but I can't figure out how to remove the
ufs label.

I boot single user and run 'tunefs -L "" /dev/ada1p2' but I get an
"unable to write superblock" error. This is before mounting the drive
RW, but it is, of course, mounted RO.


This worked in a VM when I tested it just now.  Maybe running fsck on 
that filesystem first will help.

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: Removing UFS label from root

[Erich Dollansky]

Hi,

On Sun, 9 Sep 2012 17:47:43 -0700
Kevin Oberman  wrote:

> I added a label to my root fs some time ago. I really prefer the gpt
> label and I have added it, but I can't figure out how to remove the
> ufs label.
> 
> I boot single user and run 'tunefs -L "" /dev/ada1p2' but I get an
> "unable to write superblock" error. This is before mounting the drive
> RW, but it is, of course, mounted RO. I tried setting the "allow
> foot-shooting" debug flag, but it did not help.
> 
> Is my only hope to boot a live system? (I have no CD, so I guess I
> could try a a thumb drive.)

as media size is growing, I started to make now every backup copy
bootable. Big devices get then the full FreeBSD system.

Erich
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Removing UFS label from root

[Kevin Oberman]

I added a label to my root fs some time ago. I really prefer the gpt
label and I have added it, but I can't figure out how to remove the
ufs label.

I boot single user and run 'tunefs -L "" /dev/ada1p2' but I get an
"unable to write superblock" error. This is before mounting the drive
RW, but it is, of course, mounted RO. I tried setting the "allow
foot-shooting" debug flag, but it did not help.

Is my only hope to boot a live system? (I have no CD, so I guess I
could try a a thumb drive.)
-- 
R. Kevin Oberman, Network Engineer
E-mail: kob6...@gmail.com
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Edward M]

On 05/11/2012 05:18 PM, Chad Perrin wrote:

I appreciate the time you put into this.

   It was no problem at all:-)
   had fun comparing.
Now that I'm free and have more time I went over the 3rd edition 
table of contents and found a few instances
that mentions FreeBSD. In chapter "Adding a Disk"  describes the 
FFS, shows a freebsd fstab example file and
teaches how to add a disk in FreeBSD,etc. I  Continued  glancing  
at the contents and it appears the rest of the book is pretty much

on subjects that apply to all UNIX OS.

the fourth edition text has for some reason
basically traded FreeBSD for AIX -- which makes little sense to me.


   I found a site that it kinda shows that this is was happened, AIX 
replaced FreeBSD:-(
   mid way  through the site shows the 4th edition only focuses on  
redhat, opensuse, rhel, solaris, HPUX and IBM AIX.


http://www.admin.com/




  I

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Chad Perrin]

On Fri, May 11, 2012 at 12:11:48PM -0700, Edward M wrote:
> 
>So far I think I found a few that may make a difference.
> According to  the "table of contents" in the 4th edition in the
> chapter called "Booting and shuting down it
>only shows entries for: red hat, HP-UX, AIX, SUSE,Ubuntu. However
> in the third edition, show entries for FreeBSD's Booting and shuting
> down process.
>And another  example is in the 4th edition the chapter called
> "Adding new users", only mentions how to add users for:
> SUSE, Redhat Solaris HP-UX and AIX. However in the 3rd edition,
> explains how to add users on  FreeBSD  and
> how FreeBSD's master.passwd file, login.conf. work,etc  The
> third edition's chapter called
> "Drivers and the kernel" shows how to build a freebsd kernel,
> create a BSD config file, tuning the freebsd kernel, add freebsd
> device  drivers,etc.
> I  was not able  to  find those entries in the the 4th editions
> "Drivers and the kernel. chapter.the 3rd editions  TCP/IP
> chapter shows network config for freebsd.
>However in the table of contents of the 4th edition does not.
> I'm searching for a website that contains  the 3rd edition table of
> contents so one can compare between
> the  two editions for better judgement.
>  unfortunate,  those were a few examples i have time to point
> out. I think may make a great difference.

Okay, thanks.  You've provided a pretty good representative selection, I
think.  I guess there are two problems: the third edition index is
woefully incomplete, and the fourth edition text has for some reason
basically traded FreeBSD for AIX -- which makes little sense to me.

I appreciate the time you put into this.

-- 
Chad Perrin [ original content licensed OWL: http://owl.apotheon.org ]
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Edward M]

On 05/11/2012 12:11 PM, Edward M wrote:
So far I think I found a few that may make a difference.   According 
to  the "table of contents" in the 4th edition in the  chapter called 
"Booting and shuting down it
   only shows entries for: red hat, HP-UX, AIX, SUSE,Ubuntu. However 
in the third edition, show entries for FreeBSD's Booting and shuting 
down process.
   And another  example is in the 4th edition the chapter called 
"Adding new users", only mentions how to add users for:
SUSE, Redhat Solaris HP-UX and AIX. However in the 3rd edition, 
explains how to add users on  FreeBSD  and
how FreeBSD's master.passwd file, login.conf. work,etc  The third 
edition's chapter called
"Drivers and the kernel" shows how to build a freebsd kernel, 
create a BSD config file, tuning the freebsd kernel, add freebsd 
device  drivers,etc.
I  was not able  to  find those entries in the the 4th editions 
"Drivers and the kernel. chapter.the 3rd editions  TCP/IP chapter 
shows network config for freebsd.
   However in the table of contents of the 4th edition does not.  I'm 
searching for a website that contains  the 3rd edition table of 
contents so one can compare between

the  two editions for better judgement.
 unfortunate,  those were a few examples i have time to point out. 
I think may make a great difference. 



   I apologized,  if my email came out looking strange with chopped up/ 
uneven sentences,etc.   I have to check into that:-(

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Edward M]

On 05/11/2012 10:47 AM, Chad Perrin wrote:

Is there something else I should try to find in the index or table of
contents that would be in the third edition but not the fourth?  Can you
give me some examples of the sorts of things you'd expect to find in the
table of contents that is lacking in the fourth edition but present in
the third?

Hi,:-)


   So far I think I found a few that may make a difference.   According 
to  the "table of contents" in the 4th edition in the  chapter called 
"Booting and shuting down it
   only shows entries for: red hat, HP-UX, AIX, SUSE,Ubuntu. However in 
the third edition, show entries for FreeBSD's Booting and shuting down 
process.
   And another  example is in the 4th edition the chapter called 
"Adding new users", only mentions how to add users for:
SUSE, Redhat Solaris HP-UX and AIX. However in the 3rd edition, 
explains how to add users on  FreeBSD  and
how FreeBSD's master.passwd file, login.conf. work,etc  The third 
edition's chapter called
"Drivers and the kernel" shows how to build a freebsd kernel, 
create a BSD config file, tuning the freebsd kernel, add freebsd device  
drivers,etc.
I  was not able  to  find those entries in the the 4th editions 
"Drivers and the kernel. chapter.the 3rd editions  TCP/IP chapter 
shows network config for freebsd.
   However in the table of contents of the 4th edition does not.  I'm 
searching for a website that contains  the 3rd edition table of contents 
so one can compare between

the  two editions for better judgement.
 unfortunate,  those were a few examples i have time to point out. 
I think may make a great difference.











___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Chad Perrin]

On Thu, May 10, 2012 at 06:29:05PM -0700, Edward M wrote:
> On 05/10/2012 03:45 PM, Alejandro Imass wrote:
> >Regarding Nemeth's I am undecided between the 4th (Unix&  Linux) or
> >the 3rd. Please advise.
> 
> i purchased the third edition because I took a look  in the 4th
> the table of contents
>  and it appears  anything   FreeBSD related   was remove and it
> only focuses on: Solaris
> Linux( red hat ubuntu) and AIX. However third edition mentions BSDs

>From the index of my copy of the third edition, I see these entries:

4.4BSD 2
. . .
BSD (Berkeley UNIX) 2
. . .
FreeBSD 4

>From the index of my copy of the fourth edition, I see these entries:

BSD Printing 1054-1065
see also printing
architecture 1054-1055
configuration 1059-1065
lpc command 1057-1059
lpd daemon 1056
lpq command 1056-1057
lpr command 1056
lprm command 1057
printcap file 1059-1065
PRINTER environment variable 1054
BSD UNIX 8, 12, 1268-1273
. . .
FreeBSD 8
. . .
NetBSD 8
. . .
OpenBSD 8

Page 8 of the fourth edition mentions various BSD Unix systems in the
section "Friction Between UNIX and Linux".

Page 12's mention of BSD Unix in the fourth edition appears to correspond
to page 3's un-indexed mention of FreeBSD in the third edition
(specifically FreeBSD 3.4), in reference to the example Unix OSes they
chose to use when discussing various OSes, though FreeBSD is not
mentioned specifically in the fourth edition on that page and BSD Unix is
largely referred to in a historical context.  This appears to be a
legitimate case of BSD Unix being phased out of part of the text as a
relevant OS, but it is not a section that actually says anything of
specific technical value.

Pages 1268-1273 in the fourth edition correspond to the bulk of the
section "A Brief History of System Administration" in the back of the
book.  The third edition's equivalent is the end of page 2 and a little
over half of page 3, "The Sordid History of UNIX".

The fourth edition's index mentions "jail, chroot" which, when
investigated in the text, has nothing at all to do with FreeBSD jails;
it's just about chroot.  The third edition also contains information
about chroot, but does not mention it under the J section of the index.

It looks to me like the fourth edition probably presents quite a bit more
historical information particular to BSD Unix systems than the third
edition, judging by the index.  In the table of contents, I see that the
third edition has a section set aside for BSD printing, despite lack of
mention in the index.  It looks like the table of contents section for
"BSD and AIX printing" in the fourth edition (the first edition to
include coverage of AIX, apparently) goes into a fair bit more detail
about what's in the equivalent section.

It looks to me, at a glance, like the fourth edition probably kept all of
the BSD Unix related stuff from the third, probably updated slightly but
not expanded outside of historical information.  While a failure to
expand technical information on BSD Unix systems would result in a
reduction of the percentage of the book that covers BSD Unix technical
matters, given the growth in size between third and fourth editions, the
quantity of technical information about BSD Unix systems does not appear
to have shrunk at all, from what I've seen.  Of course, I might easily
have overlooked something.

Is there something else I should try to find in the index or table of
contents that would be in the third edition but not the fourth?  Can you
give me some examples of the sorts of things you'd expect to find in the
table of contents that is lacking in the fourth edition but present in
the third?

-- 
Chad Perrin [ original content licensed OWL: http://owl.apotheon.org ]
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Wed, May 9, 2012 at 9:48 AM, Polytropon  wrote:
> On Wed, 9 May 2012 09:30:37 -0400, Alejandro Imass wrote:
>> On Wed, May 9, 2012 at 8:53 AM, Erich Dollansky
>>  wrote:
>> >> For your recommendation above, what are the advantages or differences
>> >> of slicing the disk versus partitioning on a single slice?
>> >>
>> > it could be a misunderstanding. What is a partition? What is a slice. I 
>> > have to look always into the handbook. Anyway, as long the OS see 
>> > different units which have to be mounted independent of each other, it all 
>> > does not matter what is what.
>> >
>>
>> I meant in Unix terms of course. Slice is slice (partition in other
>> OS) and partition a thru h
>>
>> The question is if it has any advantage of using a slice to mount the
>> basejail in RO as opposed to doing the same thing on a partition.
>
> The answer is: It it not possible. :-)
>
> You cannot mount a slice.
>
> Given the BSD terminology: A slice _has_ to contain partitions.
> You cannot format a slice, you can only format partitions. A
> formatted partition carries a UFS file system. (However, it's
> possible to omit the slice, and partition the whole disk instead,
> this is called "dedicated mode"). A third method is formatting
> the whole disk ("the 'c' device"), in that case the 'c' is omitted.
>
> The _only_ time you can mount a slice is when it is used in its
> common meaning, being a "DOS primary partition"; in this case,
> a FAT or NTFS file system will be placed directly into a slice,
> as those do not support any (BSD-style) partitioning.
>
> /dev/ad0        -> the disk
> /dev/ad0s1      -> 1st slice
> /dev/ad0s1a     -> 1st partition on 1st slice
>                   THIS is something you can mount.
> -or-
> /dev/ad0a       -> 1st partition on disk ("dedicated")
>                   THIS can also be mounted.
> -or-
> /dev/ad0        -> the whole disk (equals /dev/ad0c)
>                   Even THIS can be mounted.
>
> In case I'm misunderstanding your question, could you alter the
> expression?
>

Thanks. The question was more advantages of a single slice + single
partition versus a slice and multiple partitions, for mounting the
EzJail basejail in RO mode.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Thu, May 10, 2012 at 9:29 PM, Edward M  wrote:
> On 05/10/2012 03:45 PM, Alejandro Imass wrote:
>>
>> Regarding Nemeth's I am undecided between the 4th (Unix&  Linux) or
>> the 3rd. Please advise.
>
>
>    i purchased the third edition because I took a look  in the 4th the table
> of contents
>     and it appears  anything   FreeBSD related   was remove and it only
> focuses on: Solaris
>    Linux( red hat ubuntu) and AIX. However third edition mentions BSDs
>

Yep, agreed. 3rd edition it is.

Thanks,

-- 
Alejandro Imass
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Edward M]

On 05/10/2012 03:45 PM, Alejandro Imass wrote:

Regarding Nemeth's I am undecided between the 4th (Unix&  Linux) or
the 3rd. Please advise.


i purchased the third edition because I took a look  in the 4th the 
table of contents
 and it appears  anything   FreeBSD related   was remove and it 
only focuses on: Solaris

Linux( red hat ubuntu) and AIX. However third edition mentions BSDs


table of contents of 4th edition.

http://www.amazon.com/Linux-System-Administration-Handbook-Edition/dp/0131480057/ref=sr_1_1?s=books&ie=UTF8&qid=1336698969&; 
sr=1-1#reader_0131480057

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Tue, May 1, 2012 at 1:58 AM, Robert Bonomi  wrote:
>

[...]

> Reading _both_ of McKusick's  "Design of .." books, and the 'Unix System
> Admininstration Handbook', by Nemeth, et al.  is a good _start_.
>

I just bought the FreeBSD one only unless there is a reason I should
read the older 4.4BSD ?

Regarding Nemeth's I am undecided between the 4th (Unix & Linux) or
the 3rd. Please advise.

Thanks,

-- 
Alejandro Imass

> Having a bunch of the books from O'Reilley & Assoc. (),
> especially for 'standard' tools that you need to get the most out of, is
> also highly recommended.
>
> Disclaimer:  I know a lot of the authors of those books, persoally.
> ___
> freebsd-questions@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Panic in FreeBSD 9.0-RELEASE after reload mountd to export for NFS an UFS snapshot

[Jose Garcia Juanino]

Hi,

Today I have got a panic under the following scenario:

* FreeBSD 9.0-RELEASE in VMWare ESXi virtualized host
* Very busy host (java compilers, NFS server, lot of UFS snapshots)
* apache web server
* pgsql and mysql databases
* GENERIC kernel

The panic happened after:

1- to umount a UFS snapshot mount point
2- to mount other similar UFS snapshot
   (with snapshot mount /fs:tag /mountpoint)
3- Update /etc/exports to include the previous /mountpoint
4- service mountd reload

The crash dump is here:

http://www.mipaginapersonal.movistar.es/web3/jjuanino/core.txt.0

Please keep me in CC:

Best regards


pgpiVcXYsiM8R.pgp
Description: PGP signature

Re: UFS Crash and directories now missing

[Erich Dollansky]

Hi,

On Wednesday 09 May 2012 20:30:37 Alejandro Imass wrote:
> On Wed, May 9, 2012 at 8:53 AM, Erich Dollansky
>  wrote:
> > Hi,
> >
> > On Wednesday 09 May 2012 18:57:06 Alejandro Imass wrote:
> >> On Thu, May 3, 2012 at 1:14 PM, Alejandro Imass  wrote:
> >> > On Thu, May 3, 2012 at 9:35 AM, Robert Bonomi  
> >> > wrote:
> >> >>
> >>
> >> [...]
> >>
> >> >> One comment: for 'defensive' purposes it would be useful to break ad6 up
> >> >> into two slices, putting 'basejail' in it's own slice.  Then, for 
> >> >> production
> >> >> use, that slice can be mounted RO, and with the 'system immutable' flag
> >> >> set on everything in that filesystem.
> >> >>
> >> >
> >> > Yes. From one of your posts that became somewhat clear to me: Having
> >> > all the jails on a single 150GB slice seems like a bad idea.
> >> >
> >>
> >> For your recommendation above, what are the advantages or differences
> >> of slicing the disk versus partitioning on a single slice?
> >>
> > it could be a misunderstanding. What is a partition? What is a slice. I 
> > have to look always into the handbook. Anyway, as long the OS see different 
> > units which have to be mounted independent of each other, it all does not 
> > matter what is what.
> >
> 
> I meant in Unix terms of course. Slice is slice (partition in other
> OS) and partition a thru h
> 
> The question is if it has any advantage of using a slice to mount the
> basejail in RO as opposed to doing the same thing on a partition.
> 
I do not think so. As long it is mounted as a separated unit, FreeBSD has to 
keep everything separated.

Erich
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Polytropon]

On Wed, 9 May 2012 09:30:37 -0400, Alejandro Imass wrote:
> On Wed, May 9, 2012 at 8:53 AM, Erich Dollansky
>  wrote:
> >> For your recommendation above, what are the advantages or differences
> >> of slicing the disk versus partitioning on a single slice?
> >>
> > it could be a misunderstanding. What is a partition? What is a slice. I 
> > have to look always into the handbook. Anyway, as long the OS see different 
> > units which have to be mounted independent of each other, it all does not 
> > matter what is what.
> >
> 
> I meant in Unix terms of course. Slice is slice (partition in other
> OS) and partition a thru h
> 
> The question is if it has any advantage of using a slice to mount the
> basejail in RO as opposed to doing the same thing on a partition.

The answer is: It it not possible. :-)

You cannot mount a slice.

Given the BSD terminology: A slice _has_ to contain partitions.
You cannot format a slice, you can only format partitions. A
formatted partition carries a UFS file system. (However, it's
possible to omit the slice, and partition the whole disk instead,
this is called "dedicated mode"). A third method is formatting
the whole disk ("the 'c' device"), in that case the 'c' is omitted.

The _only_ time you can mount a slice is when it is used in its
common meaning, being a "DOS primary partition"; in this case,
a FAT or NTFS file system will be placed directly into a slice,
as those do not support any (BSD-style) partitioning.

/dev/ad0-> the disk
/dev/ad0s1  -> 1st slice
/dev/ad0s1a -> 1st partition on 1st slice
   THIS is something you can mount.
-or-
/dev/ad0a   -> 1st partition on disk ("dedicated")
   THIS can also be mounted.
-or-
/dev/ad0-> the whole disk (equals /dev/ad0c)
   Even THIS can be mounted.

In case I'm misunderstanding your question, could you alter the
expression?



-- 
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Wed, May 9, 2012 at 8:53 AM, Erich Dollansky
 wrote:
> Hi,
>
> On Wednesday 09 May 2012 18:57:06 Alejandro Imass wrote:
>> On Thu, May 3, 2012 at 1:14 PM, Alejandro Imass  wrote:
>> > On Thu, May 3, 2012 at 9:35 AM, Robert Bonomi  
>> > wrote:
>> >>
>>
>> [...]
>>
>> >> One comment: for 'defensive' purposes it would be useful to break ad6 up
>> >> into two slices, putting 'basejail' in it's own slice.  Then, for 
>> >> production
>> >> use, that slice can be mounted RO, and with the 'system immutable' flag
>> >> set on everything in that filesystem.
>> >>
>> >
>> > Yes. From one of your posts that became somewhat clear to me: Having
>> > all the jails on a single 150GB slice seems like a bad idea.
>> >
>>
>> For your recommendation above, what are the advantages or differences
>> of slicing the disk versus partitioning on a single slice?
>>
> it could be a misunderstanding. What is a partition? What is a slice. I have 
> to look always into the handbook. Anyway, as long the OS see different units 
> which have to be mounted independent of each other, it all does not matter 
> what is what.
>

I meant in Unix terms of course. Slice is slice (partition in other
OS) and partition a thru h

The question is if it has any advantage of using a slice to mount the
basejail in RO as opposed to doing the same thing on a partition.

Thanks,

-- 
Alejandro Imass
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Erich Dollansky]

Hi,

On Wednesday 09 May 2012 18:57:06 Alejandro Imass wrote:
> On Thu, May 3, 2012 at 1:14 PM, Alejandro Imass  wrote:
> > On Thu, May 3, 2012 at 9:35 AM, Robert Bonomi  
> > wrote:
> >>
> 
> [...]
> 
> >> One comment: for 'defensive' purposes it would be useful to break ad6 up
> >> into two slices, putting 'basejail' in it's own slice.  Then, for 
> >> production
> >> use, that slice can be mounted RO, and with the 'system immutable' flag
> >> set on everything in that filesystem.
> >>
> >
> > Yes. From one of your posts that became somewhat clear to me: Having
> > all the jails on a single 150GB slice seems like a bad idea.
> >
> 
> For your recommendation above, what are the advantages or differences
> of slicing the disk versus partitioning on a single slice?
> 
it could be a misunderstanding. What is a partition? What is a slice. I have to 
look always into the handbook. Anyway, as long the OS see different units which 
have to be mounted independent of each other, it all does not matter what is 
what.

Erich
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Thu, May 3, 2012 at 1:14 PM, Alejandro Imass  wrote:
> On Thu, May 3, 2012 at 9:35 AM, Robert Bonomi  
> wrote:
>>

[...]

>> One comment: for 'defensive' purposes it would be useful to break ad6 up
>> into two slices, putting 'basejail' in it's own slice.  Then, for production
>> use, that slice can be mounted RO, and with the 'system immutable' flag
>> set on everything in that filesystem.
>>
>
> Yes. From one of your posts that became somewhat clear to me: Having
> all the jails on a single 150GB slice seems like a bad idea.
>

For your recommendation above, what are the advantages or differences
of slicing the disk versus partitioning on a single slice?

Thanks,

-- 
Alejandro Imass
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[jb]

Alejandro Imass  p2ee.org> writes:

> ...
OK. There is this anomaly in your troubled jail.
Because I have not been familiar with ezjail so I am learning it now as we go.
I will suggest to you some steps by intuition, and you have to judge for both
of us how to do it and what's appropriate. Feel free to give feedback anytime.

I understand that this troubled jail can be stopped any time without any
problems to your clients.
Just in case do a backup of it with ezjail admin entry. Use ezjail admin
entries by the way (no 'mv', 'cp', etc).

I look at an ezjail tutorial right now and see the following: 
ezjail Default File Locations
/usr/jails/ : Default location to store base jail system template.
/usr/jails/flavours/ : Customization for each jail can be done via flavours.
   For e.g. adding default /etc/resolv.conf file or updating existing 
   /etc/make.conf can be done here.
/usr/jails/basejail/ : Base jail will be exported and mounted as read only for
   each jail. This will save disk space.
/usr/local/etc/rc.d/ezjail.sh : Stop / Start / Restart jails script.
/usr/local/etc/ezjail.conf : Configuration file for ezjail script. contains 
  settings that control the operation of the ezjail rc script. It is also read 
  by the ezjail-admin utility to figure out where it should perform its actions.
/usr/local/etc/ezjail/ : All your jail configuration files are stored here.

Now, your task is to figure out where your mounts are configured for that jail,
those that are present in the 'mount' output.
After that you would stop that jail and delete it (assuming no problem to your
customer service) to get rid of those lines in mount output.
The reconfigure that jail properly anew (do not use backup for this !) and
create it again, check the 'mount' output to see proper sequence of mounts.
Then restart the jail and see how it goes.
Give us feedback as you go so we can follow you.
jb




  


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Thu, May 3, 2012 at 1:40 PM, jb  wrote:
> Alejandro Imass  p2ee.org> writes:
>
>> ...
>> I have no idea, but cmm-php52-1 is in fact the problematic jail with
>> the MySQL problem.
>
> Could you please include displays of
> 1. your troubled machine's
>   $ cat /etc/fstab
>   Note: you already showed us 'mount' output.
> 2. your other trouble-free server's
>   $ cat /etc/fstab
>   $ mount
>

The fstab was in a previous mail but here it is again...

>From the troubled server:
# DeviceMountpoint  FStype  Options DumpPass#
/dev/ad4s1b none    swapsw  0   0
/dev/ad4s1a /   ufs rw  1   1
/dev/ad4s1d /tmp    ufs rw  2   2
/dev/ad4s1f /usr    ufs rw  2   2
/dev/ad4s1e /var    ufs rw  2   2
/dev/ad6s1.journal  /usr/jails  ufs rw,async2   2
/dev/cd0/cdrom  cd9660  ro,noauto   0   0

>From a good server (single disk machine):
/dev/ad4s1b none    swapsw  0   0
/dev/ad4s1a /       ufs rw  1   1
/dev/ad4s1d /tmpufs     rw  2   2
/dev/ad4s1f.journal /usr    ufs rw,async
 2   2
/dev/ad4s1e /varufs rw  2   2

/dev/ad4s1a on / (ufs, local)
devfs on /dev (devfs, local, multilabel)
/dev/ad4s1d on /tmp (ufs, local, soft-updates)
/dev/ad4s1f.journal on /usr (ufs, asynchronous, local, gjournal)
/dev/ad4s1e on /var (ufs, local, soft-updates)
/usr/jails/basejail on /usr/jails/httpProxy/basejail (nullfs, local, read-only)
devfs on /usr/jails/httpProxy/dev (devfs, local, multilabel)
fdescfs on /usr/jails/httpProxy/dev/fd (fdescfs)
procfs on /usr/jails/httpProxy/proc (procfs, local)
/usr/jails/basejail on /usr/jails/cat58base/basejail (nullfs, local, read-only)
devfs on /usr/jails/cat58base/dev (devfs, local, multilabel)
fdescfs on /usr/jails/cat58base/dev/fd (fdescfs)
procfs on /usr/jails/cat58base/proc (procfs, local)
/usr/jails/basejail on /usr/jails/watwkyTesting/basejail (nullfs,
local, read-only)
devfs on /usr/jails/watwkyTesting/dev (devfs, local, multilabel)
fdescfs on /usr/jails/watwkyTesting/dev/fd (fdescfs)
procfs on /usr/jails/watwkyTesting/proc (procfs, local)
/usr/jails/basejail on /usr/jails/mta1/basejail (nullfs, local, read-only)
devfs on /usr/jails/mta1/dev (devfs, local, multilabel)
fdescfs on /usr/jails/mta1/dev/fd (fdescfs)
procfs on /usr/jails/mta1/proc (procfs, local)
/usr/jails/basejail on /usr/jails/migdev/basejail (nullfs, local, read-only)
devfs on /usr/jails/migdev/dev (devfs, local, multilabel)
fdescfs on /usr/jails/migdev/dev/fd (fdescfs)
procfs on /usr/jails/migdev/proc (procfs, local)








> jb
>
>
>
>
> ___
> freebsd-questions@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[jb]

Alejandro Imass  p2ee.org> writes:

> ...
> I have no idea, but cmm-php52-1 is in fact the problematic jail with
> the MySQL problem.

Could you please include displays of
1. your troubled machine's 
   $ cat /etc/fstab
   Note: you already showed us 'mount' output.
2. your other trouble-free server's
   $ cat /etc/fstab
   $ mount

jb




___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Thu, May 3, 2012 at 12:05 PM, jb  wrote:
> Alejandro Imass  p2ee.org> writes:
>
>> ...
>> devfs on /usr/jails/cmm-php52-1/dev (devfs, local, multilabel)
>> ...
>> /usr/jails/basejail on /usr/jails/cmm-php52-1/basejail (nullfs, local,
>> read-only)
>> fdescfs on /usr/jails/cmm-php52-1/dev/fd (fdescfs)
>> procfs on /usr/jails/cmm-php52-1/proc (procfs, local)
>
> There is one thing that looks like an anomaly.
> For each jail, should the master template basejail be mounted into it first,
> followed by /dev and anything else in there ?
>
> /usr/jails/basejail on /usr/jails/cmm-php52-1/basejail (nullfs, local,
> read-only)
> devfs on /usr/jails/cmm-php52-1/dev (devfs, local, multilabel)
> fdescfs on /usr/jails/cmm-php52-1/dev/fd (fdescfs)
> procfs on /usr/jails/cmm-php52-1/proc (procfs, local)
>
> Does it matter ?

I have no idea, but cmm-php52-1 is in fact the problematic jail with
the MySQL problem.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Thu, May 3, 2012 at 9:35 AM, Robert Bonomi  wrote:
>
> Alejandro Imass  wrote:
>
> [ megasnip ]
>
>> > Things to investigate :
>> > - When was the last time this box was rebooted normally ? Did it went fine 
>> > ?
>>
>> After I moved the jails to the right place I archived the jails with
>> ezjail-admin and rebooted the server several times, and everything
>> worked as expected.
>
> Rephrasing -- when was the last time _before_the_problem_was_discovered_
> that the machine was re-booted?
>

The jails moved Friday 27th so the last reboot before that was Apr 4
and before Feb 29

Feb 29 10:18:46 nune reboot: rebooted by aimass
Apr  4 19:45:03 nune reboot: rebooted by aimass
Apr 27 19:47:06 nune reboot: rebooted by aimass
Apr 28 02:03:57 nune reboot: rebooted by aimass

>> > Were the jails created at this time ?
>>
>> No. Most of these jails have been operational for over a year on this
>> server without any incidents.
>
> Clarifying the question -- were the jails created at the time of the last
> _prior_ reboot?  i.e., had the machine been re-booted successfully _after_
> the jails were installed, or was this the _first_ such reboot?
>

No not at all. Most of these jails were created last year, but here is
the detail. cmm_php52_1 is the problematic jail with the MySQL, you
will see a recent date in the config file because I recently added
some cpuset as a band-aid to limit the jail's ability to bring down
the whole system, leaving at least a couple of CPUs free to be able to
ssh and shut it down. There is however a new jail corcaribe_php53 and
was the reason we rebboted the server on Apr 4th, just to make sure
that eveything would boot OK after reboot.

-rw-r--r--  1 root  wheel   917 Feb 16  2011 cat58base
-rw-r--r--  1 root  wheel   917 Apr 29  2011 cm_idvida
-rw-r--r--  1 root  wheel   937 Apr  3  2011 cm_website
-rw-r--r--  1 root  wheel   960 May  2 09:48 cmm_php52_1
-rw-r--r--  1 root  wheel  1037 Apr  4 20:00 corcaribe_php53
-rw-r--r--  1 root  wheel   950 Feb 16  2011 http_proxy
-rw-r--r--  1 root  wheel   917 Aug  3  2011 mcs_cat58
-rw-r--r--  1 root  wheel   917 Feb 10  2011 php52base
-rw-r--r--  1 root  wheel   917 Feb 12  2011 php53base
-rw-r--r--  1 root  wheel   877 Dec 27 20:33 pyugmao
-rw-r--r--  1 root  wheel   877 Mar 21 22:03 testbed
-rw-r--r--  1 root  wheel  1017 May 13  2011 yabarana_cat58
-rw-r--r--  1 root  wheel  1017 Feb 13  2011 yabarana_php52
-rw-r--r--  1 root  wheel  1017 Feb 13  2011 yabarana_php53


> It appears you misunderstood the 'at this time' reference -- it did ot
> mean 'at the time of the incident', but  'at the time of the last prior
> reboot'.  If English is not your primary language, it is an understandable
> misread.
>
>> As I told you earlier, this server has been running for over a year
>> and we have rebooted many times.
>
> I don't believe you ever mentioed that particular point (multiple
> successful reboots after istallation) before.  Repeating a prior
> question, _how_long_ before the problem showed up was the most recent
> re-boot?  (Doesn't have to be exact -- an 'order of magnitude' estimate
> [a day, a week, a month, several months] is sufficient.)
>

Apr 4th

>>                                  If there are such problems they exist
>> by using the EzJail commands and I find this unlikely.
>
> What you 'find unlikely' is irrelevant.  The entire situation is 'unlikely',
> yet it happened.  So one -has- to look at unlikely things.  
>

funny

>> here is the mount output is that's of any help:
>
> [ first disk, and 'fdescfs', and 'procfs' references removed, for clarity ]
>
>> /dev/ad6s1.journal on /usr/jails (ufs, asynchronous, local, gjournal)
>> /usr/jails/basejail on /usr/jails/yabarana-php53/basejail (nullfs,
[...]

>
> Yes, that is a good start at useful detail.  It is, presumably, _after_
> the problem, and _after_ you had restored things to their proper places.
>

Yes.

> Is it safe to  assume that you do -not- have such a 'mount' output from
> some time 'before' the problem?  ( There's no rational reason why you
> -would- have such, but _if_ it existed, and there were any differences
> between 'then' and 'now', it could be very informative.)
>

No, but from what I remember it's mostly very similar. I can pull off
similar mount statement from other server(s) where we run similar
set-ups and that have never failed if needed.

> Aother critical piece of information is what diretories -- by full path
> name -- disappeared from 'where they were', and where -- by full path name,
> again -- did you find them, and _with_what_names_?   If every

Re: UFS Crash and directories now missing

[jb]

Alejandro Imass  p2ee.org> writes:

> ...
> devfs on /usr/jails/cmm-php52-1/dev (devfs, local, multilabel)
> ...
> /usr/jails/basejail on /usr/jails/cmm-php52-1/basejail (nullfs, local,
> read-only)
> fdescfs on /usr/jails/cmm-php52-1/dev/fd (fdescfs)
> procfs on /usr/jails/cmm-php52-1/proc (procfs, local)

There is one thing that looks like an anomaly.
For each jail, should the master template basejail be mounted into it first, 
followed by /dev and anything else in there ?

/usr/jails/basejail on /usr/jails/cmm-php52-1/basejail (nullfs, local,
read-only)
devfs on /usr/jails/cmm-php52-1/dev (devfs, local, multilabel)
fdescfs on /usr/jails/cmm-php52-1/dev/fd (fdescfs)
procfs on /usr/jails/cmm-php52-1/proc (procfs, local)

Does it matter ?
jb


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Robert Bonomi]

Alejandro Imass  wrote:

[ megasnip ]

> > Things to investigate :
> > - When was the last time this box was rebooted normally ? Did it went fine ?
>
> After I moved the jails to the right place I archived the jails with
> ezjail-admin and rebooted the server several times, and everything
> worked as expected.

Rephrasing -- when was the last time _before_the_problem_was_discovered_
that the machine was re-booted?

> > Were the jails created at this time ?
>
> No. Most of these jails have been operational for over a year on this
> server without any incidents.

Clarifying the question -- were the jails created at the time of the last
_prior_ reboot?  i.e., had the machine been re-booted successfully _after_
the jails were installed, or was this the _first_ such reboot?  

It appears you misunderstood the 'at this time' reference -- it did ot
mean 'at the time of the incident', but  'at the time of the last prior
reboot'.  If English is not your primary language, it is an understandable
misread.

> As I told you earlier, this server has been running for over a year
> and we have rebooted many times.

I don't believe you ever mentioed that particular point (multiple 
successful reboots after istallation) before.  Repeating a prior
question, _how_long_ before the problem showed up was the most recent
re-boot?  (Doesn't have to be exact -- an 'order of magnitude' estimate
[a day, a week, a month, several months] is sufficient.) 

>  If there are such problems they exist
> by using the EzJail commands and I find this unlikely.

What you 'find unlikely' is irrelevant.  The entire situation is 'unlikely',
yet it happened.  So one -has- to look at unlikely things.  

> here is the mount output is that's of any help:

[ first disk, and 'fdescfs', and 'procfs' references removed, for clarity ]

> /dev/ad6s1.journal on /usr/jails (ufs, asynchronous, local, gjournal)
> /usr/jails/basejail on /usr/jails/yabarana-php53/basejail (nullfs,
> local, read-only)
> /usr/jails/basejail on /usr/jails/yabarana-php52/basejail (nullfs,
> local, read-only)
> /usr/jails/basejail on /usr/jails/yabarana-cat58/basejail (nullfs,
> local, read-only)
> /usr/jails/basejail on /usr/jails/testbed/basejail (nullfs, local, read-only)
> /usr/jails/basejail on /usr/jails/pyugmao/basejail (nullfs, local, read-only)
> /usr/jails/basejail on /usr/jails/php53base/basejail (nullfs, local, 
> read-only)
> /usr/jails/basejail on /usr/jails/php52base/basejail (nullfs, local, 
> read-only)
> /usr/jails/basejail on /usr/jails/mcs-cat58/basejail (nullfs, local, 
> read-only)
> /usr/jails/basejail on /usr/jails/http-proxy/basejail (nullfs, local, 
> read-only)
> /usr/jails/basejail on /usr/jails/corcaribe-php53/basejail (nullfs,
> local, read-only)
> /usr/jails/basejail on /usr/jails/cm-website/basejail (nullfs, local, 
> read-only)
> /usr/jails/basejail on /usr/jails/cm-idvida/basejail (nullfs, local, 
> read-only)
> /usr/jails/basejail on /usr/jails/cat58base/basejail (nullfs, local, 
> read-only)
> /usr/jails/basejail on /usr/jails/cmm-php52-1/basejail (nullfs, local,
> read-only)

Yes, that is a good start at useful detail.  It is, presumably, _after_
the problem, and _after_ you had restored things to their proper places.

Is it safe to  assume that you do -not- have such a 'mount' output from
some time 'before' the problem?  ( There's no rational reason why you
-would- have such, but _if_ it existed, and there were any differences
between 'then' and 'now', it could be very informative.)

Aother critical piece of information is what diretories -- by full path
name -- disappeared from 'where they were', and where -- by full path name,
again -- did you find them, and _with_what_names_?   If everything was
moved from the same source point to the same destination, it's not necessary
to itemize each one, but the details of _one_ 'typicaal' migration is needed.
It is also significant if there was 'anything else' in the 'where they 
belonged' directory that was -not- moved.  *OR* if there was anything else
(something other than the '/' of a jail) there, that was _also_ moved.

"Narrative" descriptions, as previously provided, and while clear to someone 
familiar with the machcine in question, are not sufficiently precise to allow 
an 'outsider' to follow the events without 'logically' replicating the setup, 
and then guessing at the meaning of any shorthands employed.



One comment: for 'defensive' purposes it would be useful to break ad6 up
into two slices, putting 'basejail' in it's own slice.  Then, for production
use, that slice can be mounted RO, and with the 'system immutable' flag
set on everything in that filesystem.


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Alejandro Imass]

-used "ln" had been discarded as
> "stupid", but the information came a lot later in answer to another post. Of

Yes, I must apologize for having ignored your post, but I found your a
priori *assumption* of human error almost as insulting as Robert
Bonomi's posts.  If I had done something that I think could have
contributed I would have said so. Do you think I would come here and
post something blaming the system, when in fact I would have thought
that human error was involved? I find that insulting. Do you think I
am afraid to say "I screwed up please help"? These a priori
assumptions about people is what pissed me off, especially because
it's not the first time I post on this list.

> course in the mean time I learned that he was using ezjail, which, if I had
> known earlier, would have made me wonder if he had not overused nullfs or
> ln. He furthermore discarded the possibility saying that he did not think
> that ezjail was using links, just nullfs. Well too bad ezjail is massively
> using links, at least for basejail, and sometime for port trees or perl
> setup depending which guide you are using as your reference.

The expectation that as a user I am required to know if EzJail uses
nullfs or links or the "source of fsck" is wrong. Some of this
information is the manual pages, some is scattered around, and yes,
there are several guides, but ultimately it all comes down to using
the EzJail commands, so when I say using it by the book, is because I
haven't done anything outside the EzJail commands, nor have I abused
nullfs, links or whatever.

I find it incredible that this is "the exception". I think the
majority of people that use FBSD *should* be using it by the book.

> During the thread he pretty much bashed anyone who tried to tell him that no
> amount of jail/ezjail/nullfs/journal screw up could have resulted in the
> entire content of the jails being moved into another completely unrelated
> directory node.  If one jail had moved it would already have been
> extraordinary, with a probability of it happening so cleanly that fsck would
> find nothing already magnitude of order above the chances of winning the
> national lottery. But all of them ? Not a chance. He finally admitted that
> he had very little knowledge about UFS and fsck, but still managed to do it
> in a quite offensive way.
>

This is false. I have only been offensive, actually defensive, against
Robert Bonobi.

> That was basically the point were I decided to stop to try to help him. I
> think others felt the same. This problem is quite interesting  in itself,
> and I think a lot of the most talented people on this list would have been
> on it but were repelled by the attitude.
>

Sorry, but this is false. At this point what I see is just you
justifying after the fact that a possible problem with nullfs has
resurfaced. Your prior assumptions like:
- "Nothing even remotely rings a bell.".
- "most of us will be inclined to think that you did something wrong."
-  "Extremely unlikely."

Were wrong to begin with and your attitude was wrong from the start,
so now you have to come here and turn this on me, when in fact it was
not only after several insulting threads that I ranted away, and not
even against you.

> On the other hand Alejandro Imass pretty much jumped on anything that would
> be a third party interaction. From someone hacking into his box to a
> potential nullfs bug that might result in a PR.
>

The problem with living in these little worlds is not being able to
picture oneself in the other person's shoes. If you have suddenly lost
more than a dozen jails without having done anything more than reboot
the system, I am pretty sure you would see this from another
perspective.

> Now the thing is that EZJail make use of the "system immutable flag" quite a
> lot for its config file, resulting in quite a lot of file being impossible
> to delete or move unless the box is running at kern_secure_level 0. This
> renders the whole "jails moved on their own" theory even more improbable.
>

Believe what you want and you are entitled to that; it's your right
and your opinion. But regardless of what you beleive, this is
something that actually happened. You don't hold the truth, only an
opinion.

> After so much ranting, I would feel bad not to try to help a little :
> Here are the facts :
> - In a jail, MySQL was grabbing all the CPU and making the box non
> responsive. This is due to TigerCRM making requests to a too huge database.
>        -> The jail was working
>        -> Unless all the data were in memory at this time (unprobable), it
> means that access path/nullfs/EZJail were OK at this time.
>
> - After a force reboot all the jails were gone, or more exactly moved inside
> a

Re: UFS Crash and directories now missing

[Erich Dollansky]

Hi,

On Tuesday 01 May 2012 20:43:43 Polytropon wrote:
> On Tue, 01 May 2012 00:37:51 -0700, Edward M wrote:
> > On 04/30/2012 10:58 PM, Robert Bonomi wrote:
> > > Reading_both_  of McKusick's  "Design of .." books, and the 'Unix System
> > > Admininstration Handbook', by Nemeth, et al.  is a good_start_.
> > >
> > > Having a bunch of the books from O'Reilley&  Assoc. 
> > > (),
> > > especially for 'standard' tools that you need to get the most out of, is
> > > also highly recommended.
> > >   
> > 
> > After realising  I lack ton of  knowledge, especially how the 
> > internals work. I'm using this advice:-) .
> 
> Except buying (good) books, you can also search for
> articles on the web. For example, "A Fast File System
> for UNIX" by M. K. McKusick is very interesting (at
> least it was for me when I lost all my important data).
> 
you wanted to say 'real man do not need a backup'?

> Some fs-related articles here:
> http://www.mckusick.com/articles.html
> 
This is one advantage of systems like FreeBSD. If the need arises, you can do 
it yourself.

>   The docs that used to live in this directory now exist on the wiki:
>   http://wiki.sleuthkit.org/
> 
It must be a disease.

Erich
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Erich Dollansky]

Hi,

On Tuesday 01 May 2012 20:43:43 Polytropon wrote:
> On Tue, 01 May 2012 00:37:51 -0700, Edward M wrote:
> > On 04/30/2012 10:58 PM, Robert Bonomi wrote:
> > > Reading_both_  of McKusick's  "Design of .." books, and the 'Unix System
> > > Admininstration Handbook', by Nemeth, et al.  is a good_start_.
> > >
> > > Having a bunch of the books from O'Reilley&  Assoc. 
> > > (),
> > > especially for 'standard' tools that you need to get the most out of, is
> > > also highly recommended.
> > >   
> > 
> > After realising  I lack ton of  knowledge, especially how the 
> > internals work. I'm using this advice:-) .
> 
> Except buying (good) books, you can also search for
> articles on the web. For example, "A Fast File System
> for UNIX" by M. K. McKusick is very interesting (at
> least it was for me when I lost all my important data).
> 
you wanted to say 'real man do not need a backup'?

> Some fs-related articles here:
> http://www.mckusick.com/articles.html
> 
This is one advantage of systems like FreeBSD. If the need arises, you can do 
it yourself.

>   The docs that used to live in this directory now exist on the wiki:
>   http://wiki.sleuthkit.org/
> 
It must be a disease.

Erich
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Polytropon]

On Wed, 2 May 2012 06:07:23 +0700, Erich Dollansky wrote:
> Hi,
> 
> On Tuesday 01 May 2012 20:43:43 Polytropon wrote:
> > On Tue, 01 May 2012 00:37:51 -0700, Edward M wrote:
> > > On 04/30/2012 10:58 PM, Robert Bonomi wrote:
> > > > Reading_both_  of McKusick's  "Design of .." books, and the 'Unix System
> > > > Admininstration Handbook', by Nemeth, et al.  is a good_start_.
> > > >
> > > > Having a bunch of the books from O'Reilley&  Assoc. 
> > > > (),
> > > > especially for 'standard' tools that you need to get the most out of, is
> > > > also highly recommended.
> > > >   
> > > 
> > > After realising  I lack ton of  knowledge, especially how the 
> > > internals work. I'm using this advice:-) .
> > 
> > Except buying (good) books, you can also search for
> > articles on the web. For example, "A Fast File System
> > for UNIX" by M. K. McKusick is very interesting (at
> > least it was for me when I lost all my important data).
> > 
> you wanted to say 'real man do not need a backup'?

No. Real men don't eat quiche. And real programmers don't
use Pascal. Also, stupidity must be punished (even if it's
me who is stupid), and it _will_ be done. Always and
repeatedly. You only learn the hard way. :-)



> > Some fs-related articles here:
> > http://www.mckusick.com/articles.html
> > 
> This is one advantage of systems like FreeBSD. If the need
> arises, you can do it yourself.

Exactly, and you're not depending on buying expensive software
that might not cover your particular problem case. The documentation
of all the "inner elements" of FreeBSD are present, and you can
learn (in worst case) everything yourself to solve the problem.
As other skilled persons have estimated and experienced the
need for professional tools, they've created them. Many of the
free tools can cope with the expensive ones designed for proprietary
platforms. The "default action" of UNIX ("if in doubt, do nothing
and let the master decide") can save your data, whereas the
habit of "repairing" things can make things worse (which includes
automounting r/w, fiddling with the FS or other nonsense that
destroys data).



> > The docs that used to live in this directory now exist on the wiki:
> > http://wiki.sleuthkit.org/
> > 
> It must be a disease.

TSK had _good_ documentation locally installed. I don't really
understand what's the idea behind moving it to a location that
can only be accessed via Internet connection. Really, it's not
_that_ much (hundreds of MB) that you couldn't leave it in the
install... sad, just sad...

Again, programs like portdowngrade help a lot. :-)




-- 
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Chad Perrin]

On Tue, May 01, 2012 at 12:58:10AM -0500, Robert Bonomi wrote:
> 
> Reading _both_ of McKusick's  "Design of .." books, and the 'Unix System 
> Admininstration Handbook', by Nemeth, et al.  is a good _start_.

"Both"?  I'm aware of at least three (FreeBSD, 4.3BSD, and 4.4BSD) that
are probably within the realm of what you're talking about (learning
about the workings of a BSD Unix system), all of which seem a little
redundant -- just different editions of the same book, from the look of
it.  What do you mean by "both" of McKusick's books?

I think there's an answer book for at least one of those, too.  Do you
perhaps mean the main book and the answer book?  Do you mean to include
the general-purpose "open source" book as one of the books (Open Sources:
Voices from the Open Source Revolution)?


> 
> Having a bunch of the books from O'Reilley & Assoc. (),
> especially for 'standard' tools that you need to get the most out of, is
> also highly recommended.  
> 
> Disclaimer:  I know a lot of the authors of those books, persoally.

If you have a decent ebook reader, I recommend just getting on the
O'Reilly mailing list for its periodic announcements of ebook discount
deals and picking up an occasional good book from those deals.  It's easy
to get far more excellent books than you have time to read that way, for
really good prices.  In fact, O'Reilly has a 50% off deal for a few
ebooks about C programming right now:

http://shop.oreilly.com/category/deals/c-programming.do

O'Reilly's ebook deals are about the only way I've found to get good
technical books from a major publisher in digital formats at a reasonable
price, considering most of the publishing world still thinks it's okay to
charge more for ebooks than for hardcopy books for some asinine reason.

O'Reilly is, in fact, pretty far ahead of competitors on its handling of
ebooks.  For instance, if you have a hardcopy O'Reilly book, you can
register it by ISBN with O'Reilly, then get an ebook copy of it for about
five bucks.  By contrast, The Pragmatic Bookshelf (which produces very
high quality books as well) at *best* gives you the opportunity to get a
hardcopy book plus a PDF book at the same time for about 150% of the
cover price of the hardcopy alone, *only* if you buy them together from
the Pragmatic website itself, and if you only have the ebook or the
hardcopy book you have no way to get a discount on the other; you have to
pay full price.  Pragmatic does offer ebooks at slightly lower price than
hardcopy, which is at least better than the "standard" industry practice
for science fiction, but it's a ridiculous price for a bundle of bits in
a digital file.

O'Reilly offers some kind of discount on hardcopies for people who have
the ebooks, too, I think.  I'm not sure -- I've never taken advantage of
that discount, because I only started collecting ebook copies of O'Reilly
books after getting an e-ink reader, which I find every bit as good for
many (though not all) reading purposes as a physical dead tree format
book.  Your mileage may vary, I suppose.

-- 
Chad Perrin [ original content licensed OWL: http://owl.apotheon.org ]
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Edward M]

On 05/01/2012 06:43 AM, Polytropon wrote:

Except buying (good) books, you can also search for
articles on the web. For example, "A Fast File System
for UNIX" by M. K. McKusick is very interesting (at
least it was for me when I lost all my important data).

Some fs-related articles here:
http://www.mckusick.com/articles.html

They help you to understand how things work or what
maybe makes them stop working.:-)

Also the documentation of tools like TSK (ports/sleuthkit),
ex TCT, is very helpful in understanding all the low-level
details that_really_  matter when you_need_  to get your
hands dirty in order to perform a forensic analysis or to
recover important data. Sadly, that documentation has moved
from local storage in/usr/local/share/doc/sleuthkit/  (where
I've seen it the last time) to some on-line place or Wiki,
something_I_  consider "a bad idea" especially in worst case
considerations (i. e. no internet connection); the only
content in README.txt,

The docs that used to live in this directory now exist on the wiki:
http://wiki.sleuthkit.org/

doesn't make it any better, sorry.

   Thanks for the help...I will definitely check McKusick site and the docs
   I'm self learning UNIX/programming. so I need all the info and help 
I can get.:-)


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Polytropon]

On Tue, 01 May 2012 00:37:51 -0700, Edward M wrote:
> On 04/30/2012 10:58 PM, Robert Bonomi wrote:
> > Reading_both_  of McKusick's  "Design of .." books, and the 'Unix System
> > Admininstration Handbook', by Nemeth, et al.  is a good_start_.
> >
> > Having a bunch of the books from O'Reilley&  Assoc. (),
> > especially for 'standard' tools that you need to get the most out of, is
> > also highly recommended.
> >   
> 
> After realising  I lack ton of  knowledge, especially how the 
> internals work. I'm using this advice:-) .

Except buying (good) books, you can also search for
articles on the web. For example, "A Fast File System
for UNIX" by M. K. McKusick is very interesting (at
least it was for me when I lost all my important data).

Some fs-related articles here:
http://www.mckusick.com/articles.html

They help you to understand how things work or what
maybe makes them stop working. :-)

Also the documentation of tools like TSK (ports/sleuthkit),
ex TCT, is very helpful in understanding all the low-level
details that _really_ matter when you _need_ to get your
hands dirty in order to perform a forensic analysis or to
recover important data. Sadly, that documentation has moved
from local storage in /usr/local/share/doc/sleuthkit/ (where
I've seen it the last time) to some on-line place or Wiki,
something _I_ consider "a bad idea" especially in worst case
considerations (i. e. no internet connection); the only
content in README.txt,

The docs that used to live in this directory now exist on the wiki:
http://wiki.sleuthkit.org/

doesn't make it any better, sorry.



-- 
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Edward M]

On 04/30/2012 10:58 PM, Robert Bonomi wrote:

Reading_both_  of McKusick's  "Design of .." books, and the 'Unix System
Admininstration Handbook', by Nemeth, et al.  is a good_start_.

Having a bunch of the books from O'Reilley&  Assoc. (),
especially for 'standard' tools that you need to get the most out of, is
also highly recommended.
  


   After realising  I lack ton of  knowledge, especially how the 
internals work. I'm using this advice:-) .

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Robert Bonomi]

Eitan Adler  wrote:
> On 30 April 2012 07:36, Robert Bonomi  wrote:
> > A competennt, "not stupid", sysadmin would know these things.  And not
> > 'remove all doubt' (in the words of Abraham Lincoln), by raising such
> > nonsense questions.
>
> A competent sysadmin would ask questions when they don't know the
> answer bringing up possibilities they thought about.
> A stupid sysadmin would yell at someone asking a question claiming
> they should have known the answer.

An informed critic would have recognized that the 'lack of knowledge' issue,
and the 'nonsense questions' were two -entirely- different matters. 

One who lacks knowledge of system fundamentals and asks questions _about_
_the_fundammentals_ that they do not understand is not subject to 
criticizm -- they are educatable.

Those who make grossly false-to-fact assumptions about the behavior of those 
fundamentals, and extrapolate wildly from those erroneous assumptions
cannot be engaged in rational conversation -without- hauling them back
to the initial erroneous assumptions, and correcting those errors.  And,
when that is done, it invaliates everything extrapolated from the false
premise.

Those who continue to extrapolate wildly in such manner cannot be helped.

It was also established that the OP's descriptions were woefully incomplete
and unreliable.  A second disk was involved.  'dangerously dedicated' or
otherwise?  partitioning?  slices? label type?  There is indirect indication
'everything of interest' was on a single slice, but that is only an inference.
There's no indication of where _in_the_filesystem_ on the slice that the 
jails '/' directories were located, or by what names they were known to the
system outside the jail.  The 'pattern' of the names, and placement in the 
hierarchy _is_ likely of some significance. As is (a) ownership, (b) 
permissions, and (c) 'flags', of (1) the original 'containing' directory,
(b) the external view of the jail '/' directories in that directory, and
(c) 'where they ended up'.  It is likely that that 'external view' (pre-
problem) of the jail '/'s does not exist -- unless one had historical data 
from before the problem.  "Everything" was running in jails.  Except for
things that weren't.

For any constructive analysis of "what happened", one needed to capture *all*
the bits in the directory (itself) where the jails ended up -- a directory
'listing', e.g. 'ls' (regardless of options), is not sufficient -- and the 
same for the directory where they 'should have been', plus a copy of the 
slice's complete inode table -- i.e., from _all_ the cylinder groups.  Then 
one would examine the 'last modified' timestamp on the directory where the 
jails were found, and -then- the timestamps on the jail directories 
themselves. 

Among other things, this data allows one to establish whether or not the
jail directories were ever _really_ where one thought they were, or whether
they just 'appeared' to be there, e.g. due to nullfs, or a 'link'.  And an 
'initial estimate' of -when- it may have happened.  (if 'malice' is involved,
or certain kinds of backup/restore activities, the timestamps _may_ not be 
accurate, but they are a 'best available' guess.)

Capturing -all- the data from the 'where they were' directory, allows one
to examine the 'deleted' entries -- where one _should_ find entries for
the jails, and 'last accessed' timestamps which put a lower bound on when
the 'move' occured.

When the 'apparently impossible' happens, it is *VERY*OFTEN* the case that
'reality' is *NOT* what someone 'knows' it is.  No matter how 'obvious' it
is, one has to =verify=.  

It is also _FAR_ 'easier to believe' that (especially) a nullfs mount (or,
less likely, a hard link) disappeared, than directories actually got moved.
The move may well have happened, but one must 'positively' eliminate the 
'more plausible' alternatives first.  Things that would 'give the appearance'
of what was reported, but from -very- different causations.

Of course, to capture this kind of information, one have to know "what's 
where" in the filesystem metadata, and have means to capture it _without_ 
changing any of that data.  And _that_ means that you have to have a fair
understanding of the mechanics of how the filesystem works.  Which rapidly
leads into gory details of how the O/S does disk I/O, and the various
performance optimizations (and trade-offs) employed.

Reading _both_ of McKusick's  "Design of .." books, and the 'Unix System 
Admininstration Handbook', by Nemeth, et al.  is a good _start_.

Having a bunch of the books from O'Reilley & Assoc. (),
especially for 'standard' tools that you need to get the most out of, is
also highly recommended.  

Disclaimer:  I know a lot of the authors of those books, persoally.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions

Re: UFS Crash and directories now missing

[Erich Dollansky]

Hi,

On Monday 30 April 2012 22:38:13 Alejandro Imass wrote:
> On Mon, Apr 30, 2012 at 7:36 AM, Robert Bonomi  
> wrote:
> 
> > A competennt, "not stupid", sysadmin would know these things.  And not
> > 'remove all doubt' (in the words of Abraham Lincoln), by raising such
> > nonsense questions.
> >
> > Postulating the "right" combination of _unrelated_ failures, virtually
> > *anything* can happen.   cf. "Nasal Monnkeys".
> >
> 
> the more than 14 years I have been participating in ANY mailing list.

oh, you have not been in many then. Or you missed the fine prints there.

Just ignore it.

Erich
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Jerome Herman]

On 30/04/2012 19:23, Eitan Adler wrote:

On 30 April 2012 07:36, Robert Bonomi  wrote:

A competennt, "not stupid", sysadmin would know these things.  And not
'remove all doubt' (in the words of Abraham Lincoln), by raising such
nonsense questions.

A competent sysadmin would ask questions when they don't know the
answer bringing up possibilities they thought about.
A stupid sysadmin would yell at someone asking a question claiming
they should have known the answer.

I must admit that Robert Bonomi tone was highly insulting for this list, 
and though I completely condemn the form of his post, I cannot say I 
disagree with the content.


There are quite a lot of things that are wrong with Alejandro Imass' 
post and analysis.
The fist thing is that he did not give is setup in one go. It took quite 
a while to figure what happened, what system he was using and how he was 
using it.
At first he had to hard reboot an unresponsive system, then at reboot he 
would have lost all of his jail.
Then it appeared that all the jails where inside another jail and that 
the unresponsiveness came from MySQL.

Then we learn that all his daemons are inside jails.
Then we learn that ftp-proxy is not.
Then we learned that jail are not handled manually but through EZJail.
Then we are told that the problem with MySQL is known and comes from a 
client using TigerCRM with a too much data.
There are litterally dozens of little pieces of important knowledge all 
over the thread. And you have to read it all to make sure you have the 
global view. Not really a good start.
It is OK to forget to mention a thing or two, discarding what you think 
is irrelevant to the problem at hand, but it is not OK to force people 
who are trying to help you to read 50+ posts to learn about the basics 
of your installation.


What is even more irritating is the fact that Alejandro Imass ignores 
pretty much anything that would leads toward a human mistake. Most posts 
implying a possible bad use of jails/nullfs/ezjail are ignored or 
answered by a simple "I have done everything by the book".  Now from my 
experience someone with 6 servers, each containing multiple jails will 
not do everything by the book every time. It might be that Alejandro is 
exceptional, but it is more likely that at least one if not more of 
these jails were not made "by the book". Nothing to blame anyone in 
here, we all get tired/bored/overconfident sometime - but refusing to 
admit the very possibility of a human mistake won't help at all in 
finding a solution. Reading the thread I realized that my suggestion 
that he might have over-used "ln" had been discarded as "stupid", but 
the information came a lot later in answer to another post. Of course in 
the mean time I learned that he was using ezjail, which, if I had known 
earlier, would have made me wonder if he had not overused nullfs or ln. 
He furthermore discarded the possibility saying that he did not think 
that ezjail was using links, just nullfs. Well too bad ezjail is 
massively using links, at least for basejail, and sometime for port 
trees or perl setup depending which guide you are using as your reference.
During the thread he pretty much bashed anyone who tried to tell him 
that no amount of jail/ezjail/nullfs/journal screw up could have 
resulted in the entire content of the jails being moved into another 
completely unrelated directory node.  If one jail had moved it would 
already have been extraordinary, with a probability of it happening so 
cleanly that fsck would find nothing already magnitude of order above 
the chances of winning the national lottery. But all of them ? Not a 
chance. He finally admitted that he had very little knowledge about UFS 
and fsck, but still managed to do it in a quite offensive way.


That was basically the point were I decided to stop to try to help him. 
I think others felt the same. This problem is quite interesting  in 
itself, and I think a lot of the most talented people on this list would 
have been on it but were repelled by the attitude.


On the other hand Alejandro Imass pretty much jumped on anything that 
would be a third party interaction. From someone hacking into his box to 
a potential nullfs bug that might result in a PR.


Now the thing is that EZJail make use of the "system immutable flag" 
quite a lot for its config file, resulting in quite a lot of file being 
impossible to delete or move unless the box is running at 
kern_secure_level 0. This renders the whole "jails moved on their own" 
theory even more improbable.


After so much ranting, I would feel bad not to try to help a little :
Here are the facts :
- In a jail, MySQL was grabbing all the CPU and making the box non 
responsive. This is due to TigerCRM making requests to a too huge database.

-> The jail was working
-> Unless all the data were in memory at this time 
(unprob

Re: UFS Crash and directories now missing

[Polytropon]

On Mon, 30 Apr 2012 13:23:40 -0400, Eitan Adler wrote:
> On 30 April 2012 07:36, Robert Bonomi  wrote:
> > A competennt, "not stupid", sysadmin would know these things.  And not
> > 'remove all doubt' (in the words of Abraham Lincoln), by raising such
> > nonsense questions.
> 
> A competent sysadmin would ask questions when they don't know the
> answer bringing up possibilities they thought about.
> A stupid sysadmin would yell at someone asking a question claiming
> they should have known the answer.

I know I don't add anything substantial by the following
statement, but allow me to post it anyway in addition to
your statement:

There is no problem in mentioning thoughts, possibilities
and options. It's also not a problem to admit a lack of
knowledge in certain fields (e. g. how UFS, journaling,
nullfs and fsck do "interact" with each other).

Things start to be problematic when conclusions are made
out of untrue assumptions or expectations. "It must be
a system error, as I don't see a human error here."
The problem is: "don't see" != "doesn't exist", and of
course != "can't be proven". Such kinds of conclusion
often lead into wrong directions.

Of course it's hard to narrow down possibilities. A "test
bed" with limited variables is neccessary to have. Also
the proper tools and procedures of testing are important.
That's the ONLY way to be sure - by eliminating one
possibility after the other. What's being found in the
end - and even if it's regarded unprobable from the
beginning - must be the reason.

Robert mentioned important things to consider. If you
(unintendedly) destroy evidence for a forensic analysis
of what happened (whatever it may be), you'll have a
hard time finding out _what_ happened - except you can
get it to happen again. In case of security breaches
this is something you _don't_ want to risk IN PUBLIC
just to be able to observe it.

At this point, one could argue politeness vs. importance
of arguments. From what I've seen on other lists, Robert's
statements are "still polite" and full of things you can
take as a start to what to additionally learn. You should
concentrate on that essence. If you take the time to "do
your homework", you'll be better prepared _if_ such thing
should ever happen again. Finding out _what_ has happened
is very hard (which I admit), and maybe it's even impossible.
You would have needed a more verbose auditing facility to
find out what program (user) caused a "mv-like syscall".
Command logs can be altered, logged syscalls... yes, it's
not impossible, but magnitudes _harder_ to remove trails.

By the way, I can understand the frustration when something
"impossible" happened and you never can _really_ say what
it was, hoping it would not happen again. I've experienced
such kinds of trouble myself. (That's why I'm on this list.)



-- 
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[jb]

Alejandro Imass  p2ee.org> writes:

> ...
> Thanks for pointing a plausible cause. What I have done so far is
> limit the offending jail to a specific cpuset and I wanted to add
> another disk to avoid contention with other jails. MySQL not only
> consumes the whole CPUs but also limits the whole drive, while it's
> doing some crazy full scan query on a very large database.
> ...

I would strongly suggest that you file a PR for nullfs, with included
reference to that PR#147420 I mentioned.
There is enough of circumstancial evidence.
I think there is a chance that a jail panic (corrupted fs or "nullfs path
translation") may be the cause of MySQL going bananas.
Anyway, the more similar reports they receive the better a chance they will
discover any problems, if any.

It would be good if you read these sections:

JAIL(8)
Setting up the Host Environment - a potential for mixing up traffic of same
  services in host env and jail env.
Jails and File Systems - multiple jails sharing the same file system can
  influence each other.
Hierarchical Jails - jail names reflect the hierarchy, but jids on the other
  hand exist in a single space, and each jail must have a unique jid.
BUGS
NOTES

JEXEC(8)
BUGS (ref to JAIL(8), Hierarchical Jails)

jb

 




___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Mon, Apr 30, 2012 at 1:57 PM, jb  wrote:
> Alejandro Imass  p2ee.org> writes:
>
>>...
>> If you have really followed the thread, all I have done is try to find
>> some explanation for a strange behavior of the system under normal
>> use. It hung, and some directories were moved, period. I have posted
>> some ideas to share with other people expecting some insight and maybe
>> similar experience from other users, which there probably are many,
>> but many times afraid to speak up and avoid getting insulted.
>> ...
>
>
> I looked at problem reports for nullfs and there are quite few.
> Hierarchical Jails
> NOTES
>
> You said you have your jail env on a separate disk.
>

Yes.

> I looked at problem reports for nullfs and there are quite few.
> http://www.freebsd.org/cgi/query-pr-summary.cgi?category=&severity=&priority=&cl
> ass=&state=&sort=none&text=nullfs&responsible=&multitext=&originator=&release=
>
> As a matter of fact I just mounted a nullfs but was not able to unmount it
> (device busy) - a Google search shows it as a problem reported for many many
> years.
> Nullfs does not seem to be stable.
>

Dirk Engling guessed that somehow nullfs was involved.

> Anyway, I found one PR
> http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/147420
>
> that is about troubles with jails, nullfs, UFS, and NFS.
> Synopsis:       [ufs] [panic] ufs_dirbad, nullfs, jail panic (corrupt inode)
>
> Take a look at this paragraphs:
> "...
> After two more failures, I now found the offending inode ..."
> "...
> As one point, I found the inode in a directory which usually is mounted for
> an (ez-) jail via nullfs."
>
> This proves that problems with jails, nullfs, and fs corruption are possible.
> So, they can not be excluded up front in your case too because nullfs is just
> a simple "path translation".
>

Up until yesterday (and Dirk's answer) I didn't look for specific
references to nullfs, and today I was busy getting vicious myself ;)

Thanks for pointing a plausible cause. What I have done so far is
limit the offending jail to a specific cpuset and I wanted to add
another disk to avoid contention with other jails. MySQL not only
consumes the whole CPUs but also limits the whole drive, while it's
doing some crazy full scan query on a very large database.

I don't have any control of the code or the MySQL myself and the
client said it's known problem with VTiger CRM and the way it
implements some reports that I guess were not designed for the amount
of data they are handling. I have already recommended they move to a
dedicated server altogether because their system simply outgrew what
we sold them.

I really appreciate the time you dedicated to search for a possible
explanation and at the very least it helps in taking some immediate
steps to avoid it from happening again. Hopefully, someone with deep
knowledge will find the root cause and a long-term fix. What is true,
that if it happened to me, it can happen to anyone, so maybe your
findings will help someone pin-point the problem and fix it.

Thanks,

-- 
Alejandro
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Chip Camden]

Quoth Eitan Adler on Monday, 30 April 2012:
> On 30 April 2012 07:36, Robert Bonomi  wrote:
> > A competennt, "not stupid", sysadmin would know these things.  And not
> > 'remove all doubt' (in the words of Abraham Lincoln), by raising such
> > nonsense questions.
> 
> A competent sysadmin would ask questions when they don't know the
> answer bringing up possibilities they thought about.
> A stupid sysadmin would yell at someone asking a question claiming
> they should have known the answer.
> 
> -- 
> Eitan Adler
> ___
> freebsd-questions@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Best response in this thread so far.

-- 
.O. | Sterling (Chip) Camden  | http://camdensoftware.com
..O | sterl...@camdensoftware.com | http://chipsquips.com
OOO | 2048R/D6DBAF91  | http://chipstips.com


pgpPEnYYipOaV.pgp
Description: PGP signature

Re: UFS Crash and directories now missing

[jb]

Alejandro Imass  p2ee.org> writes:

>... 
> If you have really followed the thread, all I have done is try to find
> some explanation for a strange behavior of the system under normal
> use. It hung, and some directories were moved, period. I have posted
> some ideas to share with other people expecting some insight and maybe
> similar experience from other users, which there probably are many,
> but many times afraid to speak up and avoid getting insulted.
> ...

Well, I try to help only, so I hope I do not get insulted ... I could become
vicious :-) 

Here it is.

You said you have your jail env on a separate disk.

I looked at problem reports for nullfs and there are quite few.
Hierarchical Jails
NOTES

You said you have your jail env on a separate disk.

I looked at problem reports for nullfs and there are quite few.
http://www.freebsd.org/cgi/query-pr-summary.cgi?category=&severity=&priority=&cl
ass=&state=&sort=none&text=nullfs&responsible=&multitext=&originator=&release=

As a matter of fact I just mounted a nullfs but was not able to unmount it
(device busy) - a Google search shows it as a problem reported for many many
years.
Nullfs does not seem to be stable.

Anyway, I found one PR
http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/147420

that is about troubles with jails, nullfs, UFS, and NFS.
Synopsis:   [ufs] [panic] ufs_dirbad, nullfs, jail panic (corrupt inode)

Take a look at this paragraphs:
"...
After two more failures, I now found the offending inode ..."
"...
As one point, I found the inode in a directory which usually is mounted for
an (ez-) jail via nullfs."

This proves that problems with jails, nullfs, and fs corruption are possible.
So, they can not be excluded up front in your case too because nullfs is just
a simple "path translation".

jb


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Edward M]

On 04/30/2012 10:22 AM, Alejandro Imass wrote:

Oh, please! He's not helping anyone. He's just being an obnoxious
prick that thinks that by pointing out a lot of technical blabber and
some cheap philosophical posé


   I guess i was going according to the fact that i have followed his 
suggestions
   on a problem i was having and i was able to find the cause and 
solved the problem. :-)


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Mon, Apr 30, 2012 at 1:23 PM, Eitan Adler  wrote:
> On 30 April 2012 07:36, Robert Bonomi  wrote:
>> A competennt, "not stupid", sysadmin would know these things.  And not
>> 'remove all doubt' (in the words of Abraham Lincoln), by raising such
>> nonsense questions.
>
> A competent sysadmin would ask questions when they don't know the
> answer bringing up possibilities they thought about.
> A stupid sysadmin would yell at someone asking a question claiming
> they should have known the answer.
>

Thank you Eitan!

I am admittedly limited in the use of the English language and many
times frustrated not to be able to redact such beautifully and to the
point.

-- 
Alejandro Imass
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Eitan Adler]

On 30 April 2012 07:36, Robert Bonomi  wrote:
> A competennt, "not stupid", sysadmin would know these things.  And not
> 'remove all doubt' (in the words of Abraham Lincoln), by raising such
> nonsense questions.

A competent sysadmin would ask questions when they don't know the
answer bringing up possibilities they thought about.
A stupid sysadmin would yell at someone asking a question claiming
they should have known the answer.

-- 
Eitan Adler
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Mon, Apr 30, 2012 at 1:03 PM, Edward M  wrote:
> On 04/30/2012 08:38 AM, Alejandro Imass wrote:
>>
>>  just not very helpful or fun. This attitude will get
>
>
>    He is helping,you need to  learn how UFS, jails, nullfs, journaling, disk
> I/O  and other stuff work.
>    I have been following this thread and i must admit I also need to learn
> more on those subjects.:-)
>

Oh, please! He's not helping anyone. He's just being an obnoxious
prick that thinks that by pointing out a lot of technical blabber and
some cheap philosophical posé, he's going to gain some sort of place
amongst his peers, and you are just trying to do the same by
sucking-up, siding with him and seconding an simply unacceptable
attitude in a community of real peers.

If you truly know your stuff you don't have to go putting people down
and patronizing them to show off. It is only when you go over the top,
trying to prove something that your are actually just showing your
insecurities and just plain ignorance.

Why don't you google and read my posts over the years when I help
other people in things they don't know, and tell me if it's remotely
close, or if I patronize people. I might go tell someone to RTFM but I
would never go and try to put someone down just to show off that I
know a lot.

Furthermore, this is a user's list, not a deeply technical one. I
don't have to read the fsck source code to use FreeBSD or participate
on this list. If you are indeed an expert you try to help other
people, or at least give the other person the benefit of the doubt.

If you have really followed the thread, all I have done is try to find
some explanation for a strange behavior of the system under normal
use. It hung, and some directories were moved, period. I have posted
some ideas to share with other people expecting some insight and maybe
similar experience from other users, which there probably are many,
but many times afraid to speak up and avoid getting insulted.

I don't take that crap from anyone and much less in a community that I
have come to love and respect.

And it's all about that: RESPECT and you can either learn it the easy
way or the hard way, but I will tech respect one way or another.

-- 
Alejandro Imass
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Mon, Apr 30, 2012 at 8:22 AM, Erich Dollansky
 wrote:
> Hi,
>
> On Monday 30 April 2012 18:36:08 Robert Bonomi wrote:
>>
>> Alejandro Imass  wrote:
>> That simply *ISN'T* going to happen -- not without a -lot- more evidence
>> than any individual can provide from a single =unrepeadable= incident.
>>
> ok, I am not the original poster but let me tell me of an experience here I 
> have had. I reported also something extremely strange. Of course, the 
> comments I have gotten have been the same as here. But what happened then?
>
> I do not know why but somebody found a race condition in the affected system. 
> There is no fix available yet.
>
> With other words, no matter how strange things are, I encourage people to 
> report it. Not with the real hope to get a solution at the spot. But with the 
> chance that somebody who knows the code well and has some strange feelings 
> takes a look.
>
> I also encourage my clients to do the same for our products and services.
>

Thanks Erich for pionting this out. This is the FreeBSD USERS LIST,
not the elite exchange. I I was posting this on some expert list like
the kernel list or some other more technical list I could understand
the attitude, but this is the user's list. We are NOT required to know
the details, just share experiences and try to help one another, not
put other people down for trying to solve our issues as users.

What is really frustrating is that it actually happened and I try to
do everything by the book. I don't do any fancy or strange things, so
something caused these directories to be moved through NORMAL use of
the system, regardless if some people believe it or not, I could care
less. It happened, period, and if someone wants to help fine, if not
they should just shut up.

Thanks again for pointing this out. We are the users, we are the
people that keep this project alive and share the good.

-- 
Alejandro Imass
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Edward M]

On 04/30/2012 08:38 AM, Alejandro Imass wrote:

  just not very helpful or fun. This attitude will get


He is helping,you need to  learn how UFS, jails, nullfs, 
journaling, disk I/O  and other stuff work.
I have been following this thread and i must admit I also need to 
learn more on those subjects.:-)


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Mon, Apr 30, 2012 at 7:36 AM, Robert Bonomi  wrote:
>
> Alejandro Imass  wrote:
>> On Sun, Apr 29, 2012 at 11:49 PM, Erich Dollansky wrote:
>> > On Monday 30 April 2012 02:02:41 jb wrote:
>> >> Alejandro Imass  p2ee.org> writes:
>> >> > ...
>>

[...]

> A competennt, "not stupid", sysadmin would know these things.  And not
> 'remove all doubt' (in the words of Abraham Lincoln), by raising such
> nonsense questions.
>
>> Whatever the cause, it actually happened and I have already ruled out
>> just about anything. It doesn't seem to have been an attack, it surely
>> wasn't me, and EzJail author agrees it was not the EzJail scripts. So
>> maybe nullfs and journaling, or crash + nullfs + journaling, could
>> cause something like this to happen?
>
> Postulating the "right" combination of _unrelated_ failures, virtually
> *anything* can happen.   cf. "Nasal Monnkeys".
>

OK, I tried to be patient with you and tried to keep my composure and
nettiquete against your insistence to insult me and by doing so,
damaging the good spirited nature of this mailing list, FreeBSD and
Open Source in general.

And sorry beforehand to my fellow co-listers, and other nice people
here,  that I have to do this publicly but there is a limit and I am
sure many of you have been just waiting for this to happen.

I mean, I have had a couple of altercations here and there with a few
smart asses, but I have *NEVER* seen such an obnoxious little shit in
the more than 14 years I have been participating in ANY mailing list.
This used to be one of the most enjoyable and helpful lists and it is
people like you who draw people away from sharing and trying to help
one another.

What is your problem man? Who do you think you are? Who gives you the
right to go patronizing and insulting people, and by the way using
these ridiculous quotes, like some stupid little jerk, relying on
other people's wisdom quotes instead of your own words. Instead of
being frontal,  you are probably frustrated with your own little techy
life that you have to take out your frustration on other people.

I find you intoxicating to this list and to this community, no matter
how smart you are, if half the stuff you say is even accurate or true.
You don't contribute anything except to the degradation of the FreeBSD
ambiance and to drive people away, and from sharing. You don't have
the right to do that.

I honestly love FreeBSD and this community and I am not going to let
you ruin that for me or anyone else here. Why don't you take the time
to read your posts and see that you propose nothing, so why even
bother to participate? What are you trying to prove?  If you were so
smart as *you believe* you are, you would be helping instead of trying
to prove something by your condescending attitude. The very fact that
you need to use this attitude is proof of your insecurities, and your
need to bully other people, but not me, Sir. I have been in this too
long.

I am surely not going to take this shit from you man so if you don't
have anything positive to say, just shut up and let other people help
each other, without being scared of being insulted or patronized. I am
surely not afraid of you and I am sick and tired of your attitude, so
if no one else here has the balls to tell you off, I will.

This is the kind of shit that drives people away and refrains people
from participating and sharing experiences and knowledge, and I'm not
going to let you do that, to me or any one else here. This is not
*your list* nor do you have any special right here, you are just like
everybody else, just not very helpful or fun. This attitude will get
you nowhere but deeper into your creepy little world.

-- 
Alejandro Imass
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Erich Dollansky]

Hi,

On Monday 30 April 2012 18:36:08 Robert Bonomi wrote:
> 
> Alejandro Imass  wrote:
> That simply *ISN'T* going to happen -- not without a -lot- more evidence 
> than any individual can provide from a single =unrepeadable= incident.
> 
ok, I am not the original poster but let me tell me of an experience here I 
have had. I reported also something extremely strange. Of course, the comments 
I have gotten have been the same as here. But what happened then?

I do not know why but somebody found a race condition in the affected system. 
There is no fix available yet.

With other words, no matter how strange things are, I encourage people to 
report it. Not with the real hope to get a solution at the spot. But with the 
chance that somebody who knows the code well and has some strange feelings 
takes a look.

I also encourage my clients to do the same for our products and services.

Erich
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Robert Bonomi]

Alejandro Imass  wrote:
> On Sun, Apr 29, 2012 at 11:49 PM, Erich Dollansky wrote:
> > On Monday 30 April 2012 02:02:41 jb wrote:
> >> Alejandro Imass  p2ee.org> writes:
> >> > ...
>
> Back to theory on how the http-proxy jail 'swallowed' all the other
> jails including the basejail. 

A "theory" that contains assumptions which are, unfortunately, unsupported
by any factual evidence. 

Just like _every_other_ "theory" you have advanced to date.

FACT: It is a virtual certainty that something operating -outside- any jail
environment is what did the deed.

Available evidence to date is that you 'fixate' on a particular _remote_
possibility -- *without* knowledge of what it would take for that scenario
to come to pass -- making a sh*tload of 'assumptions' along the way (many
of which are contrary to reality), and offer that as 'the explanation' for
events.



> Given that EzJail uses a single basejail and links/mounts stuff in the
> child jails it would seem plausible (regression?) that somehow any
> jail could access other jails' files,

Demonstrating, yet again, that you do not understand how jails work. :((

>   or that _maybe_ in an event of
> crash the nullsfs mounts confuse the system somehow when fsck restores
> or the journal is recovered.

Demonstrating, yet again, that you do not understand what nullfs is, how
it works, or that it is totally -irrelevant- to fsck and/or journaling.

Hint: nullfs is merely a 'path translation' mechanism -- it affects _only_ 
'file open' syscalls.  fsck doesn't _touch_ nullfs.

Hint; journaling is an add-on to the UFS filesystem.  nullfs doesn't know 
what journaling is.   "Journal recovery" doesn't _touch_ a nullfs.

A competennt, "not stupid", sysadmin would know these things.  And not 
'remove all doubt' (in the words of Abraham Lincoln), by raising such
nonsense questions.

> Whatever the cause, it actually happened and I have already ruled out
> just about anything. It doesn't seem to have been an attack, it surely
> wasn't me, and EzJail author agrees it was not the EzJail scripts. So
> maybe nullfs and journaling, or crash + nullfs + journaling, could
> cause something like this to happen?

Postulating the "right" combination of _unrelated_ failures, virtually
*anything* can happen.   cf. "Nasal Monnkeys".

It has already been demonstrated how the (im-)probability of such an event
relates to the age of the universe.

>         Maybe journal has some confusion
> on restoring the nullfs view of the directories or something after bad
> crash like this one??

Short answer: "No chance."  Again, if you had any understandinng of how 
UFS, and nullfs for that matter, works -- not to mention how disk I/O works
inside the kernel, you wouldn't be embarassing yourself by your _continued_
raising of what are, to put it charitiably, such 'patently ridiculous' 
questions.

You can engage in all the 'unfounded speculation' you want to, but you are
simply -not- going to determine "what happened".  

IF there was a systemic fault, you have already destroyed the forensic 
evidence trail that _might_ have allowed a qualified expert to run it down,
*if* you could afford to have such an analysis done.  (middle five figures
is a starting point for such an analysis.)

Absent _multiple_ reports of like events, *WITH* enough detail in the reports
to have a reasonable chance of identifying a 'pattern' of events leading to 
the failure, *OR* the existance of a -reliable-, =repeadable=, method of 
inducing the failure, this simply isn't going to go anywere.  Absent any 
of those things, it is a 'freak' event, *PROBABLY* (read 'virtually certain')
caused by human error (despite your claim of the 'impossibility' of that 
factor) in some form.

If you insist on 'knowing' what happened in any future instance of single
putatively 'abnormal' events, you will need to change to a MIL-SPEC 'B2' 
(or higher) rated O/S, with active mandatory access controls, 'security 
labels' with multi-level, non-hierarchical,  security enabled, audit 
logging of -every- system call, etc.  This also requires a staff position 
of 'security officer', which is _separate_and_distinct_ from 'system 
administrtor'.   I strongly suspect that you cannot afford the required 
hardware and software for this type of 'solution'.

The 'underlying cause' almost certainly falls into the class known as PEBKAC.
(The current admin has demonstrated an inability to accurately report the 
 state of his system -- that at least one thing he previously asserted to be
 

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Sun, Apr 29, 2012 at 11:49 PM, Erich Dollansky
 wrote:
> Hi,
>
> On Monday 30 April 2012 02:02:41 jb wrote:
>> Alejandro Imass  p2ee.org> writes:
>>
>> > ...
>> > > What you should do right now is to get some recent general or security 
>> > > cd/dvd
>> > > with chkrootkit and rkhunter and run them from that external read-only 
>> > > media.
>> > > I would also suggest that you look over config files of all packages
>> > > involved.
>> > > jb
>> > >
>> >
>> > Thanks! Will do, but I don't know of any FreeBSD and/or derived
>> > distros for security. Or can I use any Linux security distro? I
>> > remember reading about some trouble of Linux chkrootkit on FBSD
>>
>> It looks like you have only one choice with prebuilt rkhunter package only:
>> http://www.freebsd.org/releases/9.0R/announce.html
>>
>> dvd1
>> This contains everything necessary to install the base FreeBSD operating 
>> system,
>> a collection of pre-built packages aimed at getting a graphical workstation 
>> up
>> and running. It also supports booting into a "livefs" based rescue mode. This
>> should be all you need if you can burn and use DVD-sized media.
>>
>> ftp://ftp.freebsd.org/pub/FreeBSD/ports/packages/security/
>> rkhunter-1.3.8_1.tbz          04/18/12        18:56:00
>>
>> With regard to verification of config  files - you said you got backups 
>> (those
>> pre-incident would be best) and you have the incident-time files, so do a 
>> diff
>> on dirs (in particular /etc and /usr/local/etc)
>>
> I would burn the backup of these files to an optical disk, start the system 
> and do a diff as the first step. The system can be started from an USB drive 
> (take the 9.0 installation image) or DVD.
>
> Of course, rkhunter can be started in the second step.

ran both, found nothing

Back to theory on how the http-proxy jail 'swallowed' all the other
jails including the basejail. I noticed that jail had a not so old bug
in 2010 FBSD 8.0 which


The jail(8) utility does not change the current working directory while
imprisoning.  The current working directory can be accessed by its
descendants.


Reference: http://security.freebsd.org/advisories/FreeBSD-SA-10:04.jail.asc

Given that EzJail uses a single basejail and links/mounts stuff in the
child jails it would seem plausible (regression?) that somehow any
jail could access other jails' files, or that _maybe_ in an event of
crash the nullsfs mounts confuse the system somehow when fsck restores
or the journal is recovered.

Whatever the cause, it actually happened and I have already ruled out
just about anything. It doesn't seem to have been an attack, it surely
wasn't me, and EzJail author agrees it was not the EzJail scripts. So
maybe nullfs and journaling, or crash + nullfs + journaling, could
cause something like this to happen? Maybe journal has some confusion
on restoring the nullfs view of the directories or something after bad
crash like this one??
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Erich Dollansky]

Hi,

On Monday 30 April 2012 02:02:41 jb wrote:
> Alejandro Imass  p2ee.org> writes:
> 
> > ... 
> > > What you should do right now is to get some recent general or security 
> > > cd/dvd
> > > with chkrootkit and rkhunter and run them from that external read-only 
> > > media.
> > > I would also suggest that you look over config files of all packages 
> > > involved.
> > > jb
> > >
> > 
> > Thanks! Will do, but I don't know of any FreeBSD and/or derived
> > distros for security. Or can I use any Linux security distro? I
> > remember reading about some trouble of Linux chkrootkit on FBSD
> 
> It looks like you have only one choice with prebuilt rkhunter package only:
> http://www.freebsd.org/releases/9.0R/announce.html  
> 
> dvd1
> This contains everything necessary to install the base FreeBSD operating 
> system,
> a collection of pre-built packages aimed at getting a graphical workstation up
> and running. It also supports booting into a "livefs" based rescue mode. This
> should be all you need if you can burn and use DVD-sized media.
> 
> ftp://ftp.freebsd.org/pub/FreeBSD/ports/packages/security/
> rkhunter-1.3.8_1.tbz  04/18/1218:56:00
> 
> With regard to verification of config  files - you said you got backups (those
> pre-incident would be best) and you have the incident-time files, so do a diff
> on dirs (in particular /etc and /usr/local/etc) 
> 
I would burn the backup of these files to an optical disk, start the system and 
do a diff as the first step. The system can be started from an USB drive (take 
the 9.0 installation image) or DVD.

Of course, rkhunter can be started in the second step.

Erich
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[jb]

Alejandro Imass  p2ee.org> writes:

> ... 
> > What you should do right now is to get some recent general or security 
> > cd/dvd
> > with chkrootkit and rkhunter and run them from that external read-only 
> > media.
> > I would also suggest that you look over config files of all packages 
> > involved.
> > jb
> >
> 
> Thanks! Will do, but I don't know of any FreeBSD and/or derived
> distros for security. Or can I use any Linux security distro? I
> remember reading about some trouble of Linux chkrootkit on FBSD

It looks like you have only one choice with prebuilt rkhunter package only:
http://www.freebsd.org/releases/9.0R/announce.html  

dvd1
This contains everything necessary to install the base FreeBSD operating system,
a collection of pre-built packages aimed at getting a graphical workstation up
and running. It also supports booting into a "livefs" based rescue mode. This
should be all you need if you can burn and use DVD-sized media.

ftp://ftp.freebsd.org/pub/FreeBSD/ports/packages/security/
rkhunter-1.3.8_1.tbz04/18/1218:56:00

With regard to verification of config  files - you said you got backups (those
pre-incident would be best) and you have the incident-time files, so do a diff
on dirs (in particular /etc and /usr/local/etc) 

jb


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Sun, Apr 29, 2012 at 1:15 PM, jb  wrote:
> Alejandro Imass  p2ee.org> writes:
>
>> ...
>> And there was a log of a couple of ftp connections the same day this
>> happened, the ONLY 3 messages before the reboot at about 6 pm and they
>> were NOT from any of our customers. Here are the log entries:
>>
>> Apr 27 05:54:37 nune ftp.proxy[2726]: connected to client:
>> host-46-50-183-5.bbcustomer.zsttk.net, interface= 207.158.52.74:21
>> Apr 27 05:54:37 nune ftp.proxy[2726]: info: monitor mode: off, ccp: 
>> Apr 27 05:54:38 nune ftp.proxy[2726]: -ERR: missing hostname
>> Apr 27 18:55:42 nune syslogd: kernel boot file is /boot/kernel/kernel
>> ...
>
> What you should do right now is to get some recent general or security cd/dvd
> with chkrootkit and rkhunter and run them from that external read-only media.
> I would also suggest that you look over config files of all packages involved.
> jb
>

Thanks! Will do, but I don't know of any FreeBSD and/or derived
distros for security. Or can I use any Linux security distro? I
remember reading about some trouble of Linux chkrootkit on FBSD

Thanks,

-- 
Alejandro

>
> ___
> freebsd-questions@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[jb]

Alejandro Imass  p2ee.org> writes:

> ... 
> And there was a log of a couple of ftp connections the same day this
> happened, the ONLY 3 messages before the reboot at about 6 pm and they
> were NOT from any of our customers. Here are the log entries:
> 
> Apr 27 05:54:37 nune ftp.proxy[2726]: connected to client:
> host-46-50-183-5.bbcustomer.zsttk.net, interface= 207.158.52.74:21
> Apr 27 05:54:37 nune ftp.proxy[2726]: info: monitor mode: off, ccp: 
> Apr 27 05:54:38 nune ftp.proxy[2726]: -ERR: missing hostname
> Apr 27 18:55:42 nune syslogd: kernel boot file is /boot/kernel/kernel
> ...

What you should do right now is to get some recent general or security cd/dvd
with chkrootkit and rkhunter and run them from that external read-only media.  
I would also suggest that you look over config files of all packages involved.
jb


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Sun, Apr 29, 2012 at 4:37 AM, Polytropon  wrote:
> On Sun, 29 Apr 2012 00:26:50 -0700, per...@pluto.rain.com wrote:
>> Alejandro Imass  wrote:
>>
>> > 3) the directories were moved at reboot by journal recovery,
>> > fsck or something else
>>
>> I think it's *extremely* unlikely that fsck was involved, because
>> it just doesn't do things like that.
>
> The point is: fsck moving directories "looks different". In
> case inodes get "de-connected" (their reference entries on
> level n-1 are gone, their data on level n is still present),
> fsck will access the lost+found/ directory in the corresponding
> partition's root directory (or create it, if not present) and
> write _new_ directory entries with the inode as their name,
> because that's the only naming information possible (as the
> original names on n-1 aren't accessible anymore). So those
> directories will have names like #177628676/ and they _can_
> contain subtrees full of data, _including_ names from levels
> n+1 and onward. Files also are named #4767667892 and their
> names can _maybe_ identified from their content (the "file"
> command is helpful, and if they are textfiles containing
> a CVS or other revision control system data tag, it's possible
> to find out what they've been in their previous life).
>
> However, as it has been explained, fsck will _not_ do so
> unless being _allowed explicitely_ to do that kind of
> MODIFICATION to the file system. Flags like -yf can do
> that, but they are _not_ the default. This is due to the
> fact that _any_ critical modification of file systems
> requires the _responsible administrator_ to give permission.
>

OK, so fsck couldn't have done this. Besides fsck reported the fs as
clean so I have to conclude as others have commented that it must have
been a mv

I've been looking at the logs very carefully and trying to make sense
of this. There is a possibility that it could have been an attack
because we enabled ftp.proxy so that some clients could upload stuff
to their jails using ftp. So I was initially wrong in my assessment
because on this particular server we are running a service outside of
jails and it's this ftp.proxy that was suppose to be a temporary
solution but I guess we never got around to fixing this.

The ftp.proxy is started via inetd like so:
ftpstream tcp  nowait nobody /usr/local/sbin/ftp.proxy ftp.proxy -e

And there was a log of a couple of ftp connections the same day this
happened, the ONLY 3 messages before the reboot at about 6 pm and they
were NOT from any of our customers. Here are the log entries:

Apr 27 05:54:37 nune ftp.proxy[2726]: connected to client:
host-46-50-183-5.bbcustomer.zsttk.net, interface= 207.158.52.74:21
Apr 27 05:54:37 nune ftp.proxy[2726]: info: monitor mode: off, ccp: 
Apr 27 05:54:38 nune ftp.proxy[2726]: -ERR: missing hostname
Apr 27 18:55:42 nune syslogd: kernel boot file is /boot/kernel/kernel

OK. So let's suppose ftp.proxy is the culprit is there any way the
could have done the mv by cracking ftp and ftp.proxy ??

I have of course disabled the ftp and I am now thinking that another
possibility or combination by also using the ftp proxy on the
http-proxy jail, that is, the jail that swallowed the other jails. The
http-proxy jails was also running apache ftp proxy.

So the question now becomes: could a break in ftp.proxy coupled with
Apache ftp proxy have caused the http-proxy jails to have swallowed
all the other jails into it's configuration directory??

-- 
Alejandro Imass
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Polytropon]

On Sun, 29 Apr 2012 00:26:50 -0700, per...@pluto.rain.com wrote:
> Alejandro Imass  wrote:
> 
> > 3) the directories were moved at reboot by journal recovery,
> > fsck or something else
> 
> I think it's *extremely* unlikely that fsck was involved, because
> it just doesn't do things like that. 

The point is: fsck moving directories "looks different". In
case inodes get "de-connected" (their reference entries on
level n-1 are gone, their data on level n is still present),
fsck will access the lost+found/ directory in the corresponding
partition's root directory (or create it, if not present) and
write _new_ directory entries with the inode as their name,
because that's the only naming information possible (as the
original names on n-1 aren't accessible anymore). So those
directories will have names like #177628676/ and they _can_
contain subtrees full of data, _including_ names from levels
n+1 and onward. Files also are named #4767667892 and their
names can _maybe_ identified from their content (the "file"
command is helpful, and if they are textfiles containing
a CVS or other revision control system data tag, it's possible
to find out what they've been in their previous life).

However, as it has been explained, fsck will _not_ do so
unless being _allowed explicitely_ to do that kind of
MODIFICATION to the file system. Flags like -yf can do
that, but they are _not_ the default. This is due to the
fact that _any_ critical modification of file systems
requires the _responsible administrator_ to give permission.



-- 
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Sat, Apr 28, 2012 at 10:20 PM, Erich Dollansky
 wrote:
> Hi,
>
> On Sunday 29 April 2012 08:58:17 Alejandro Imass wrote:
>> On Sat, Apr 28, 2012 at 5:03 PM, Erich Dollansky
>>  wrote:

[...]

>>
>> Hi Erich, thanks for your reply.
>>
>> I don't know what links you are referring to, but please point me in
>
> man link
>
> They are practical in jails when things are read only. Mark everything 
> read-only and nothing should go wrong.
>

I though you were referring to something else entirely. No, I don't
use soft or hard links with jails, unless EzJail uses them but I doubt
it, I think everything like that in EzJail is done by mounting via
nullfs.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Erich Dollansky]

Hi,

On Sunday 29 April 2012 08:58:17 Alejandro Imass wrote:
> On Sat, Apr 28, 2012 at 5:03 PM, Erich Dollansky
>  wrote:
> > On Saturday 28 April 2012 20:15:25 Alejandro Imass wrote:
> >> On Sat, Apr 28, 2012 at 3:22 AM, Wojciech Puchar
> >>  wrote:
> >> >> I somewhat agree, but it wasn't a person. I am the only administrator,
> >> >> the only one with root access. The jails were effectively moved to the
> >> >> /usr/local/etc/apache22 of the single that survived at the top level.
> >> >> I'm thinking something between mount, EzJail, the journal and the way
> >> >> MySQL created a great deal of head contention, so something must have
> >> >> gotten corrupted at the directory level like you state, but the
> >> >> strange part is no _data_ corruption as such, because I was able to
> >> >> physically archive the jails, move them to the correct directory and
> >> >
> >> >
> >> > no matter what you do FreeBSD DOES NOT ramdomly move directories. if you 
> >> > are
> >> > sure you didn't move it yourself then it must be machine hardware problem
> >> > but still unlikely.
> >>
> >> After a little more research, ___it it NOT unlikely at all___ that
> >> under high distress and a hard boot, UFS could have somehow corrupted
> >> the directory structure, whilst maintaining the data intact. From what
> >> I've learned so far, UFS is actually divided into 2 layers: one that
> >> controls the directory structure and metadata and a lower layer
> >> containing the data, so the directories being screwed up and the data
> >> intact it is actually quite possible.
> >>
> >> What I'm trying to do is figure out is how it happened, and try
> >> prevent it from happening again, so instead of dismissing it as
> >> impossibility, I think we all should spend a little time figuring out
> >> how these things can happen and determine how it can be prevented or
> >> reduced.
> >
> > somebody mentioned the links. Did you use links in the jails to access the 
> > data? If then the directories of the jails got screwed, the links are gone 
> > but the original data is still there. The damaged directory might got fixed 
> > during the first reboot after the crash and you never noticed the fix.
> >
> 
> Hi Erich, thanks for your reply.
> 
> I don't know what links you are referring to, but please point me in

man link

They are practical in jails when things are read only. Mark everything 
read-only and nothing should go wrong.

> that direction. I initially suspected that it could have been the
> journal recovery and/or fsck but as you can see, a couple of people

I only installed journals on a new machine without any experience there. UFS 
does normally the job for me.

> have said this is impossible, but have to admit my ignorance on some
> specifics of the UFS filesystem, yet out of logic seems like the most
> plausible explanation.

This is not a good reasoning. I have had clients using my own software for 
years before it crashed with an error which was in there since the start.

> 
> I've been running FBSD since 6.2 and jails since then as well.  Today
> I run 6 public servers in 8.2 with between 15 to 20 jails each and we
> switched to ezjail last year and use strictly by the book. I do use
> flavours though, and I may archive and re-create jails with a specific
> archive but always using ezjail-admin. Since all our servers are 8.2
> and all updated the same, I may port jails from one server to the
> other using the ezjail archive method, but nothing as stupid as
> someone was suggesting that I was using cp or soft links.
> 
I never used ezjail in real life.

> I've never had any problems except in _this particular server_ where I
> have client that has a problem with MySQL and under some conditions it
> drains the whole server. I suspected corruption of the fs because of
> all the contention generated by MySQL to the point where it simply
> hung and had to hard-reboot. I doubt it's hardware because these are
> relatively new servers Xeon X3370, 8GB RAM, 2 x 150GB 10,000rpm
> Velociraptor disks. We have the pristine OS in one disk and jails in
> the other. Nothing runs outside of jails, not even the MTA which runs
> postfix inside one of the jails.
> 
> This is the first crash when anything like this has happened in over 6
> years running FBSD, and I am surprised as anyone here because of the
> weirdness of the jail directories moving like that. We had backups of
> the previous night, but I didn't even u

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Sun, Apr 29, 2012 at 3:26 AM,   wrote:
> Alejandro Imass  wrote:
>

[...]

>
> Any chance that your base system -- rather than one of the jails --
> has somehow been cracked; maybe even that the cracker precipitated
> the crash?  It might be wise to restore the whole system from backup,
> the base from a moderately old one since it doesn't change anyway,
> rather than trying to recover.

There is always that possibility but I strive to keep these servers
updated, I block most ap, nigeria and russia ip blocks using updated
Wizcrafts' lists, run fail2ban and other security measures. We have a
policy of only one password and there are no users or services in the
base system other than mine. As I said in another mail I run 6 servers
and been runing FBSD for almost 7 years and this is the first time
I've seen this happen.

-- 
Alejandro
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Sat, Apr 28, 2012 at 5:03 PM, Erich Dollansky
 wrote:
> Hi,
>
> On Saturday 28 April 2012 20:15:25 Alejandro Imass wrote:
>> On Sat, Apr 28, 2012 at 3:22 AM, Wojciech Puchar
>>  wrote:
>> >> I somewhat agree, but it wasn't a person. I am the only administrator,
>> >> the only one with root access. The jails were effectively moved to the
>> >> /usr/local/etc/apache22 of the single that survived at the top level.
>> >> I'm thinking something between mount, EzJail, the journal and the way
>> >> MySQL created a great deal of head contention, so something must have
>> >> gotten corrupted at the directory level like you state, but the
>> >> strange part is no _data_ corruption as such, because I was able to
>> >> physically archive the jails, move them to the correct directory and
>> >
>> >
>> > no matter what you do FreeBSD DOES NOT ramdomly move directories. if you 
>> > are
>> > sure you didn't move it yourself then it must be machine hardware problem
>> > but still unlikely.
>>
>> After a little more research, ___it it NOT unlikely at all___ that
>> under high distress and a hard boot, UFS could have somehow corrupted
>> the directory structure, whilst maintaining the data intact. From what
>> I've learned so far, UFS is actually divided into 2 layers: one that
>> controls the directory structure and metadata and a lower layer
>> containing the data, so the directories being screwed up and the data
>> intact it is actually quite possible.
>>
>> What I'm trying to do is figure out is how it happened, and try
>> prevent it from happening again, so instead of dismissing it as
>> impossibility, I think we all should spend a little time figuring out
>> how these things can happen and determine how it can be prevented or
>> reduced.
>
> somebody mentioned the links. Did you use links in the jails to access the 
> data? If then the directories of the jails got screwed, the links are gone 
> but the original data is still there. The damaged directory might got fixed 
> during the first reboot after the crash and you never noticed the fix.
>

Hi Erich, thanks for your reply.

I don't know what links you are referring to, but please point me in
that direction. I initially suspected that it could have been the
journal recovery and/or fsck but as you can see, a couple of people
have said this is impossible, but have to admit my ignorance on some
specifics of the UFS filesystem, yet out of logic seems like the most
plausible explanation.

I've been running FBSD since 6.2 and jails since then as well.  Today
I run 6 public servers in 8.2 with between 15 to 20 jails each and we
switched to ezjail last year and use strictly by the book. I do use
flavours though, and I may archive and re-create jails with a specific
archive but always using ezjail-admin. Since all our servers are 8.2
and all updated the same, I may port jails from one server to the
other using the ezjail archive method, but nothing as stupid as
someone was suggesting that I was using cp or soft links.

I've never had any problems except in _this particular server_ where I
have client that has a problem with MySQL and under some conditions it
drains the whole server. I suspected corruption of the fs because of
all the contention generated by MySQL to the point where it simply
hung and had to hard-reboot. I doubt it's hardware because these are
relatively new servers Xeon X3370, 8GB RAM, 2 x 150GB 10,000rpm
Velociraptor disks. We have the pristine OS in one disk and jails in
the other. Nothing runs outside of jails, not even the MTA which runs
postfix inside one of the jails.

This is the first crash when anything like this has happened in over 6
years running FBSD, and I am surprised as anyone here because of the
weirdness of the jail directories moving like that. We had backups of
the previous night, but I didn't even use them. The data was all
there, intact, just moved inside the only surviving jail, which
happens to be the http reverse proxy of all the other jails.

If you have any leads as to how this can happen other than cosmic rays
I would greatly appreciate it.

Thanks!

-- 
Alejandro

> Erich
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[perryh]

Alejandro Imass  wrote:

> 3) the directories were moved at reboot by journal recovery,
> fsck or something else

I think it's *extremely* unlikely that fsck was involved, because
it just doesn't do things like that.  It might move an orphaned
directory (or file) to lost+found, but nowhere else.  That's in
addition to the fact that, as someone already mentioned, it asks
before doing anything.  I don't know enough about the details of
journal recovery to comment on it as a suspect.

> That is what worries me, is that it wasn't just some random bit
> or cosmic ray, but the potential of happening again ...

Any chance that your base system -- rather than one of the jails --
has somehow been cracked; maybe even that the cracker precipitated
the crash?  It might be wise to restore the whole system from backup,
the base from a moderately old one since it doesn't change anyway,
rather than trying to recover.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Erich Dollansky]

Hi,

On Saturday 28 April 2012 20:15:25 Alejandro Imass wrote:
> On Sat, Apr 28, 2012 at 3:22 AM, Wojciech Puchar
>  wrote:
> >> I somewhat agree, but it wasn't a person. I am the only administrator,
> >> the only one with root access. The jails were effectively moved to the
> >> /usr/local/etc/apache22 of the single that survived at the top level.
> >> I'm thinking something between mount, EzJail, the journal and the way
> >> MySQL created a great deal of head contention, so something must have
> >> gotten corrupted at the directory level like you state, but the
> >> strange part is no _data_ corruption as such, because I was able to
> >> physically archive the jails, move them to the correct directory and
> >
> >
> > no matter what you do FreeBSD DOES NOT ramdomly move directories. if you are
> > sure you didn't move it yourself then it must be machine hardware problem
> > but still unlikely.
> 
> After a little more research, ___it it NOT unlikely at all___ that
> under high distress and a hard boot, UFS could have somehow corrupted
> the directory structure, whilst maintaining the data intact. From what
> I've learned so far, UFS is actually divided into 2 layers: one that
> controls the directory structure and metadata and a lower layer
> containing the data, so the directories being screwed up and the data
> intact it is actually quite possible.
> 
> What I'm trying to do is figure out is how it happened, and try
> prevent it from happening again, so instead of dismissing it as
> impossibility, I think we all should spend a little time figuring out
> how these things can happen and determine how it can be prevented or
> reduced.

somebody mentioned the links. Did you use links in the jails to access the 
data? If then the directories of the jails got screwed, the links are gone but 
the original data is still there. The damaged directory might got fixed during 
the first reboot after the crash and you never noticed the fix.

Erich
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Edward M]

On 04/28/2012 11:16 AM, Alejandro Imass wrote:

That is what worries me, is that it wasn't just some random bit or
cosmic ray, but the potential of happening again. I am not so sure
that it is*impossible*  that a jail could affect other jails with
EzJail.


  Sorry I'm late to the party. How about contacting EZjail and 
explaining what has happen:-)

  it may be a bug?

  http://erdgeist.org/arts/software/ezjail/#Author
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Jerome Herman]

On 28/04/2012 19:52, Alejandro Imass wrote:

On Sat, Apr 28, 2012 at 1:31 PM, Robert Bonomi  wrote:

Alejandro Imass  wrote:

On Sat, Apr 28, 2012 at 11:39 AM, Robert Bonomi
  wrote:

  Alejandro Imass  wrote:

After a little more research, ___it it NOT unlikely at all___ that
under high distress and a hard boot, UFS could have somehow corrupted
the directory structure, whilst maintaining the data intact.

This is techically accurate, *BUT* the specifics of the quote "corruption"
unquote in the case under discussion make it *EXTREMELY* unlikely that this
is what happened.

99.99+++% of all UFS filesystem "corruption' issues are the result of a
system crash _between_ the time cached 'meta-data' is updated in memory
and that data is flushed to disk (a deferred write).

The second most common (and vanishingly rare) failure mode is a powerfail
_as_ a sector of disk is being written -- resulting in 'garbage data'
being written to disk.

The next possibility is 'cosmic rays'.  If running on 'cheap' hardware
(i.e., without 'ECC' memory), this can cause a *SINGLE-BIT* error in
data being output.

The fact that the 'corrupted' filesystem passed fsck -without- any reported
errors shows that everything in the filesystem meta-data was consistent


[...]


I think it is safe to conclude that the probabilities -greatly- favor
alternative #1.


OK. So after your comments and further research I concur with you on
the mv but if it wasn't a human, then this might be exposing a serious
security flaw in the jail system or the way EzJail implements it.

BOGON ALERT!!!


I admit my ignorance on how the filesystem works but I don't think
your condescending remarks add a lot of value. The issue here is this
actually happened and there is a flaw somewhere other than "the stupid
administrator did it".

Ok,

Not wanting to take any side in what could end up in personal attacks 
and nasty things being said about any poster genitors but :


- Jails are very widely used, in fact it is probably one of the most 
used functionnality of FreeBSD. Far beyond ZFS, MAC or any of the other 
nice thingies FreeBSD has.
- Jails are very often misused. Though not overly complex, creating a 
proper jail and upgrading it can sometime be a bit tricky.
- Though not entirely devoid of bug and perfect, FreeBSD 8.2 is probably 
the best thing there is out there when it comes to system stability. It 
might be lacking some little nooks and cranies when it comes to perfect 
compliance with obscure standards, it might not behave as expected in 
some very few situation, but these are extremely rare. FreeBSD 8.2 is 
very widely used and this is one of the first time I heard of such a 
problem in jails. Nothing even remotely rings a bell.


Take all these information into account and put yourself in our shoes. 
When reading your problem description, most of us will be inclined to 
think that you did something wrong.


My personnal guess would be that you probably abused  "ln" a bit too 
much when creating the jails (total shot in the dark here, but it could 
explain what happened).  I don't see how journaling could impact your 
jails in anyway except if your jails were all extremely new when the 
crash happened or that the I/O was such that FreeBSD could never sync 
and commit journal from the time you created your jails to the time 
where the system crashed.

Extremely unlikely.

So my question is : where all the jail created properly ? Did you cpdup 
each and every one of them or were you lazy at some point ? Are all the 
jails properly declared in rc.conf ? My guess would be that the first 
jail was created in the right way, but that others were created using cp 
and ln, resulting in unexpected behaviour in the end. If I am right then 
the "surviving" jail would be either the first or the last you created.


Jerome Herman


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Sat, Apr 28, 2012 at 2:01 PM, Polytropon  wrote:
> On Sat, 28 Apr 2012 13:52:02 -0400, Alejandro Imass wrote:
>> On Sat, Apr 28, 2012 at 1:31 PM, Robert Bonomi  
>> wrote:
>> >
>> > Alejandro Imass  wrote:
>> >> On Sat, Apr 28, 2012 at 11:39 AM, Robert Bonomi
>> >>  wrote:
>> >> >  Alejandro Imass  wrote:
>> >> >> After a little more research, ___it it NOT unlikely at all___ that
>> >> >> under high distress and a hard boot, UFS could have somehow corrupted
>> >> >> the directory structure, whilst maintaining the data intact.
>> >> >
>> >> > This is techically accurate, *BUT* the specifics of the quote 
>> >> > "corruption"
>> >> > unquote in the case under discussion make it *EXTREMELY* unlikely that 
>> >> > this
>> >> > is what happened.
>> >> >
>> >> > 99.99+++% of all UFS filesystem "corruption' issues are the result of a
>> >> > system crash _between_ the time cached 'meta-data' is updated in memory
>> >> > and that data is flushed to disk (a deferred write).
>> >> >
>> >> > The second most common (and vanishingly rare) failure mode is a 
>> >> > powerfail
>> >> > _as_ a sector of disk is being written -- resulting in 'garbage data'
>> >> > being written to disk.
>> >> >
>> >> > The next possibility is 'cosmic rays'.  If running on 'cheap' hardware
>> >> > (i.e., without 'ECC' memory), this can cause a *SINGLE-BIT* error in
>> >> > data being output.
>> >> >
>> >> > The fact that the 'corrupted' filesystem passed fsck -without- any 
>> >> > reported
>> >> > errors shows that everything in the filesystem meta-data was consistent
>> >> >
>> >> [...]
>> >>
>> >> > I think it is safe to conclude that the probabilities -greatly- favor
>> >> > alternative #1.
>> >> >
>> >>
>> >> OK. So after your comments and further research I concur with you on
>> >> the mv but if it wasn't a human, then this might be exposing a serious
>> >> security flaw in the jail system or the way EzJail implements it.
>> >
>> > BOGON ALERT!!!
>> >
>>
>> I admit my ignorance on how the filesystem works but I don't think
>> your condescending remarks add a lot of value. The issue here is this
>> actually happened and there is a flaw somewhere other than "the stupid
>> administrator did it".
>
> If you search the archives of this list, you'll find my _first_
> post to that list: I've had a similar problem, df shows data
> must be there after crash (panic -> reboot -> fsck trouble), but
> files aren't there (even _not_ in lost+found). It's quite possible
> that in _exceptional_ moments this can happen. The fsck program
> is intended to repair the most typical file system faults, but
> nothing "complicated" will be done without interaction: Altering
> data on disk will _always_ involve the responsible (!) admin to
> check if it is really intended "to do so".
>
> There can be many reasons. I've never found out what was the
[...]

> that might help locate "lost" data (quotes intended as long as
> the data is still on the disk). The more complex your setting
> is (e. g. striped disks, or ZFS), this can be nearly impossible.
> "Plain old UFS" can sometimes be your saviour (but BACKUP should
> be your real friend).
>

Thanks for your reply.

I can't figure out how there was no data loss and yet the directories
moved just like that. We have nightly backups and it's one of the
features we love about EzJail and it's archive feature. The base
system sits on another disk entirely and it's pristine, we don't
install anything except the basic system on the system disk and the
other disk is exclusively divided in jails, so the possibility of an
outside process doing the mv is unlikely.

Everything point to that something or someone executed a mv but how
was this done? or if there is a potential problem and could happen
again. And contrary to other comments here, and my admitted ignorance,
I believe there are actually 3 possibilities:

1) something inside a jail was able to move the other jails into itself
2) something outside the jails moved the jails
3) the directories were moved at reboot by journal recovery, fsck or
something else

That is what worries me, is that it wasn't just some random bit or
cosmic ray, but the potential of happening again. I am not so sure
that it is *impossible* that a jail could affect other jails with
EzJail.

-- 
Alejandro
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Polytropon]

On Sat, 28 Apr 2012 13:52:02 -0400, Alejandro Imass wrote:
> On Sat, Apr 28, 2012 at 1:31 PM, Robert Bonomi  
> wrote:
> >
> > Alejandro Imass  wrote:
> >> On Sat, Apr 28, 2012 at 11:39 AM, Robert Bonomi
> >>  wrote:
> >> >  Alejandro Imass  wrote:
> >> >> After a little more research, ___it it NOT unlikely at all___ that
> >> >> under high distress and a hard boot, UFS could have somehow corrupted
> >> >> the directory structure, whilst maintaining the data intact.
> >> >
> >> > This is techically accurate, *BUT* the specifics of the quote 
> >> > "corruption"
> >> > unquote in the case under discussion make it *EXTREMELY* unlikely that 
> >> > this
> >> > is what happened.
> >> >
> >> > 99.99+++% of all UFS filesystem "corruption' issues are the result of a
> >> > system crash _between_ the time cached 'meta-data' is updated in memory
> >> > and that data is flushed to disk (a deferred write).
> >> >
> >> > The second most common (and vanishingly rare) failure mode is a powerfail
> >> > _as_ a sector of disk is being written -- resulting in 'garbage data'
> >> > being written to disk.
> >> >
> >> > The next possibility is 'cosmic rays'.  If running on 'cheap' hardware
> >> > (i.e., without 'ECC' memory), this can cause a *SINGLE-BIT* error in
> >> > data being output.
> >> >
> >> > The fact that the 'corrupted' filesystem passed fsck -without- any 
> >> > reported
> >> > errors shows that everything in the filesystem meta-data was consistent
> >> >
> >> [...]
> >>
> >> > I think it is safe to conclude that the probabilities -greatly- favor
> >> > alternative #1.
> >> >
> >>
> >> OK. So after your comments and further research I concur with you on
> >> the mv but if it wasn't a human, then this might be exposing a serious
> >> security flaw in the jail system or the way EzJail implements it.
> >
> > BOGON ALERT!!!
> >
> 
> I admit my ignorance on how the filesystem works but I don't think
> your condescending remarks add a lot of value. The issue here is this
> actually happened and there is a flaw somewhere other than "the stupid
> administrator did it".

If you search the archives of this list, you'll find my _first_
post to that list: I've had a similar problem, df shows data
must be there after crash (panic -> reboot -> fsck trouble), but
files aren't there (even _not_ in lost+found). It's quite possible
that in _exceptional_ moments this can happen. The fsck program
is intended to repair the most typical file system faults, but
nothing "complicated" will be done without interaction: Altering
data on disk will _always_ involve the responsible (!) admin to
check if it is really intended "to do so".

There can be many reasons. I've never found out what was the
reason for the trouble I've had. Some years ago, I found a "make"
failing because "/uss/src/blah... something not found", and
a quick memtest revealed the secret: defective RAM module that
caused a "bit error", and the difference between "r" and "s"
is just one bit. Replaced the module - everything worked.
Mean soldering rays from outer space. :-)

You'll find many useful forensic tools in the ports collection
that might help locate "lost" data (quotes intended as long as
the data is still on the disk). The more complex your setting
is (e. g. striped disks, or ZFS), this can be nearly impossible.
"Plain old UFS" can sometimes be your saviour (but BACKUP should
be your real friend).





-- 
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Sat, Apr 28, 2012 at 1:31 PM, Robert Bonomi  wrote:
>
> Alejandro Imass  wrote:
>> On Sat, Apr 28, 2012 at 11:39 AM, Robert Bonomi
>>  wrote:
>> >  Alejandro Imass  wrote:
>> >> After a little more research, ___it it NOT unlikely at all___ that
>> >> under high distress and a hard boot, UFS could have somehow corrupted
>> >> the directory structure, whilst maintaining the data intact.
>> >
>> > This is techically accurate, *BUT* the specifics of the quote "corruption"
>> > unquote in the case under discussion make it *EXTREMELY* unlikely that this
>> > is what happened.
>> >
>> > 99.99+++% of all UFS filesystem "corruption' issues are the result of a
>> > system crash _between_ the time cached 'meta-data' is updated in memory
>> > and that data is flushed to disk (a deferred write).
>> >
>> > The second most common (and vanishingly rare) failure mode is a powerfail
>> > _as_ a sector of disk is being written -- resulting in 'garbage data'
>> > being written to disk.
>> >
>> > The next possibility is 'cosmic rays'.  If running on 'cheap' hardware
>> > (i.e., without 'ECC' memory), this can cause a *SINGLE-BIT* error in
>> > data being output.
>> >
>> > The fact that the 'corrupted' filesystem passed fsck -without- any reported
>> > errors shows that everything in the filesystem meta-data was consistent
>> >
>> [...]
>>
>> > I think it is safe to conclude that the probabilities -greatly- favor
>> > alternative #1.
>> >
>>
>> OK. So after your comments and further research I concur with you on
>> the mv but if it wasn't a human, then this might be exposing a serious
>> security flaw in the jail system or the way EzJail implements it.
>
> BOGON ALERT!!!
>

I admit my ignorance on how the filesystem works but I don't think
your condescending remarks add a lot of value. The issue here is this
actually happened and there is a flaw somewhere other than "the stupid
administrator did it".
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Robert Bonomi]

Alejandro Imass  wrote:
> On Sat, Apr 28, 2012 at 11:39 AM, Robert Bonomi
>  wrote:
> >  Alejandro Imass  wrote:
> >> After a little more research, ___it it NOT unlikely at all___ that
> >> under high distress and a hard boot, UFS could have somehow corrupted
> >> the directory structure, whilst maintaining the data intact.
> >
> > This is techically accurate, *BUT* the specifics of the quote "corruption"
> > unquote in the case under discussion make it *EXTREMELY* unlikely that this
> > is what happened.
> >
> > 99.99+++% of all UFS filesystem "corruption' issues are the result of a
> > system crash _between_ the time cached 'meta-data' is updated in memory
> > and that data is flushed to disk (a deferred write).
> >
> > The second most common (and vanishingly rare) failure mode is a powerfail
> > _as_ a sector of disk is being written -- resulting in 'garbage data'
> > being written to disk.
> >
> > The next possibility is 'cosmic rays'.  If running on 'cheap' hardware 
> > (i.e., without 'ECC' memory), this can cause a *SINGLE-BIT* error in 
> > data being output.
> >
> > The fact that the 'corrupted' filesystem passed fsck -without- any reported
> > errors shows that everything in the filesystem meta-data was consistent
> >
> [...]
>
> > I think it is safe to conclude that the probabilities -greatly- favor
> > alternative #1.
> >
>
> OK. So after your comments and further research I concur with you on
> the mv but if it wasn't a human, then this might be exposing a serious
> security flaw in the jail system or the way EzJail implements it.

BOGON ALERT!!!   

Jails only prevent stuff -inside- the jail from affecting stuff outside the
jail.   They do *NOT* prevent stuff 'oustide the jail' from affecting stuff
INSIDE the jail.


"For any fool-proof system, there exists a *sufficiently*determined* fool
 capable of breaking it."

>   The
> whole point of using jails is to protect things like this from
> happening. 

FALSE TO FACT.

>Given that the only jail that survived was the front-end
> Apache Web server/reverse proxy, then it is also safe to suspect the
> apache (or other) process running on it was able to perform a mv of
> the rest of the jails to it's own /usr/local/etc/apache22 directory.

FALSE TO FACT.

> Is there no possibility is that after the system crash, the journal
> recocery process and/or fsck could have moved this directories ?

"Anything is 'possible'" -- c.f. 'nasal monkeys'.

HOWEVER, if, for example, you would bother to examine the source code for 
fsck you would discover that it doesn't do -anything- 'significant' without
ASKING FIRST.   You reported it didn't find any problems -- not even anay
of the 'petty' ones it will correct w/o asking -if- the '-p' option is
specified.

"Journal revovery" _could_, 'theoretically' have done it -- *IF* "something 
else" did the 'mv' just before the crash, and that operation was journaled,
but not yet committed to disk at the time of the crash.  However, on a 
standard UFS filesystem, filesystem metadata updates are written 
'synchronously', which should eliminate _that_ wild, unfounded, speculaction.


 "You sir, don't know what you don't know, and much of what you "think" 
 you know is incorrect."





___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Sat, Apr 28, 2012 at 12:36 PM, Alejandro Imass  wrote:
> On Sat, Apr 28, 2012 at 11:39 AM, Robert Bonomi
>  wrote:
>>
>>  Alejandro Imass  wrote:
>>> On Sat, Apr 28, 2012 at 3:22 AM, Wojciech Puchar
>>>  wrote:
>>> >> I somewhat agree, but it wasn't a person. I am the only administrator,
>>> >> the only one with root access. The jails were effectively moved to the
>>> >> /usr/local/etc/apache22 of the single that survived at the top level.
>>> >> I'm thinking something between mount, EzJail, the journal and the way
>>> >> MySQL created a great deal of head contention, so something must have
>>> >> gotten corrupted at the directory level like you state, but the
>>> >> strange part is no _data_ corruption as such, because I was able to
>>> >> physically archive the jails, move them to the correct directory and
>>> >
>>> >
>>> > no matter what you do FreeBSD DOES NOT ramdomly move directories. if you 
>>> > are
>>> > sure you didn't move it yourself then it must be machine hardware problem
>>> > but still unlikely.
>>>
>>> After a little more research, ___it it NOT unlikely at all___ that
>>> under high distress and a hard boot, UFS could have somehow corrupted
>>> the directory structure, whilst maintaining the data intact.
>>
>> This is techically accurate, *BUT* the specifics of the quote "corruption"
>> unquote in the case under discussion make it *EXTREMELY* unlikely that this
>> is what happened.
>>
>> 99.99+++% of all UFS filesystem "corruption' issues are the result of a
>> system crash _between_ the time cached 'meta-data' is updated in memory
>> and that data is flushed to disk (a deferred write).
>>
>> The second most common (and vanishingly rare) failure mode is a powerfail
>> _as_ a sector of disk is being written -- resulting in 'garbage data'
>> being written to disk.
>>
>> The next possibility is 'cosmic rays'.  If running on 'cheap' hardware (i.e.,
>> without 'ECC' memory), this can cause a *SINGLE-BIT* error in data being
>> output.
>>
>> The fact that the 'corrupted' filesystem passed fsck -without- any reported
>> errors shows that everything in the filesystem meta-data was consistent
>>
>> Given *that*, there are precisely *TWO* ways that the 'results' that have
>> been reported could have happened.
>>
>>  1) "Something" did a mv(2) of the various jail directories 'from' their
>>     original location to the 'apache' diretory.  This involves simply
>>     *copying* the diretory entry from the jail's 'parent directory' to
>>     the apache directory, and then marking the entry in the original
>>     parent as 'unused'.  Nothing other than the  directory whre the jail
>>     'used to live', and the directory 'where it was found' are touched.
>>     This occured _through_ the system 'mv' function, so all the normal
>>     'housekeeping' was done properly.
>>
>>  2) it was -not- done though mv(2) -- but that requires that a whole
>>     *series* of "corruptions" of the filesystem, _ALL_ of which had to
>>     occur in 'exactly' the right way.  They are:
>
> [...]
>
>> I think it is safe to conclude that the probabilities -greatly- favor
>> alternative #1.
>>
>
> OK. So after your comments and further research I concur with you on
> the mv but if it wasn't a human, then this might be exposing a serious
> security flaw in the jail system or the way EzJail implements it. The
> whole point of using jails is to protect things like this from
> happening. Given that the only jail that survived was the front-end
> Apache Web server/reverse proxy, then it is also safe to suspect the
> apache (or other) process running on it was able to perform a mv of
> the rest of the jails to it's own /usr/local/etc/apache22 directory.
>
> Is there no possibility is that after the system crash, the journal
> recocery process and/or fsck could have moved this directories ?
>

Also note that even the EzJail basejail was moved also, so it could be
a security hole in the way nullfs is used or in nullfs itself. but the
curious thing is that the basejail is supposed to be mounted read-only
so how did that get moved to the http-proxy jail??

That is why I suspect it could have been something in the boot process
like the journal recovery, fsck or something else with that kind of
privilege and when the EzJail filesystems were unmounted.

-- 
Alejandro
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Sat, Apr 28, 2012 at 11:39 AM, Robert Bonomi
 wrote:
>
>  Alejandro Imass  wrote:
>> On Sat, Apr 28, 2012 at 3:22 AM, Wojciech Puchar
>>  wrote:
>> >> I somewhat agree, but it wasn't a person. I am the only administrator,
>> >> the only one with root access. The jails were effectively moved to the
>> >> /usr/local/etc/apache22 of the single that survived at the top level.
>> >> I'm thinking something between mount, EzJail, the journal and the way
>> >> MySQL created a great deal of head contention, so something must have
>> >> gotten corrupted at the directory level like you state, but the
>> >> strange part is no _data_ corruption as such, because I was able to
>> >> physically archive the jails, move them to the correct directory and
>> >
>> >
>> > no matter what you do FreeBSD DOES NOT ramdomly move directories. if you 
>> > are
>> > sure you didn't move it yourself then it must be machine hardware problem
>> > but still unlikely.
>>
>> After a little more research, ___it it NOT unlikely at all___ that
>> under high distress and a hard boot, UFS could have somehow corrupted
>> the directory structure, whilst maintaining the data intact.
>
> This is techically accurate, *BUT* the specifics of the quote "corruption"
> unquote in the case under discussion make it *EXTREMELY* unlikely that this
> is what happened.
>
> 99.99+++% of all UFS filesystem "corruption' issues are the result of a
> system crash _between_ the time cached 'meta-data' is updated in memory
> and that data is flushed to disk (a deferred write).
>
> The second most common (and vanishingly rare) failure mode is a powerfail
> _as_ a sector of disk is being written -- resulting in 'garbage data'
> being written to disk.
>
> The next possibility is 'cosmic rays'.  If running on 'cheap' hardware (i.e.,
> without 'ECC' memory), this can cause a *SINGLE-BIT* error in data being
> output.
>
> The fact that the 'corrupted' filesystem passed fsck -without- any reported
> errors shows that everything in the filesystem meta-data was consistent
>
> Given *that*, there are precisely *TWO* ways that the 'results' that have
> been reported could have happened.
>
>  1) "Something" did a mv(2) of the various jail directories 'from' their
>     original location to the 'apache' diretory.  This involves simply
>     *copying* the diretory entry from the jail's 'parent directory' to
>     the apache directory, and then marking the entry in the original
>     parent as 'unused'.  Nothing other than the  directory whre the jail
>     'used to live', and the directory 'where it was found' are touched.
>     This occured _through_ the system 'mv' function, so all the normal
>     'housekeeping' was done properly.
>
>  2) it was -not- done though mv(2) -- but that requires that a whole
>     *series* of "corruptions" of the filesystem, _ALL_ of which had to
>     occur in 'exactly' the right way.  They are:

[...]

> I think it is safe to conclude that the probabilities -greatly- favor
> alternative #1.
>

OK. So after your comments and further research I concur with you on
the mv but if it wasn't a human, then this might be exposing a serious
security flaw in the jail system or the way EzJail implements it. The
whole point of using jails is to protect things like this from
happening. Given that the only jail that survived was the front-end
Apache Web server/reverse proxy, then it is also safe to suspect the
apache (or other) process running on it was able to perform a mv of
the rest of the jails to it's own /usr/local/etc/apache22 directory.

Is there no possibility is that after the system crash, the journal
recocery process and/or fsck could have moved this directories ?

Thanks,

-- 
Alejandro
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Robert Bonomi]

 Alejandro Imass  wrote:
> On Sat, Apr 28, 2012 at 3:22 AM, Wojciech Puchar
>  wrote:
> >> I somewhat agree, but it wasn't a person. I am the only administrator,
> >> the only one with root access. The jails were effectively moved to the
> >> /usr/local/etc/apache22 of the single that survived at the top level.
> >> I'm thinking something between mount, EzJail, the journal and the way
> >> MySQL created a great deal of head contention, so something must have
> >> gotten corrupted at the directory level like you state, but the
> >> strange part is no _data_ corruption as such, because I was able to
> >> physically archive the jails, move them to the correct directory and
> >
> >
> > no matter what you do FreeBSD DOES NOT ramdomly move directories. if you are
> > sure you didn't move it yourself then it must be machine hardware problem
> > but still unlikely.
>
> After a little more research, ___it it NOT unlikely at all___ that
> under high distress and a hard boot, UFS could have somehow corrupted
> the directory structure, whilst maintaining the data intact.

This is techically accurate, *BUT* the specifics of the quote "corruption"
unquote in the case under discussion make it *EXTREMELY* unlikely that this
is what happened.

99.99+++% of all UFS filesystem "corruption' issues are the result of a 
system crash _between_ the time cached 'meta-data' is updated in memory
and that data is flushed to disk (a deferred write).

The second most common (and vanishingly rare) failure mode is a powerfail
_as_ a sector of disk is being written -- resulting in 'garbage data' 
being written to disk.

The next possibility is 'cosmic rays'.  If running on 'cheap' hardware (i.e.,
without 'ECC' memory), this can cause a *SINGLE-BIT* error in data being
output.

The fact that the 'corrupted' filesystem passed fsck -without- any reported
errors shows that everything in the filesystem meta-data was consistent

Given *that*, there are precisely *TWO* ways that the 'results' that have 
been reported could have happened.

  1) "Something" did a mv(2) of the various jail directories 'from' their
 original location to the 'apache' diretory.  This involves simply
 *copying* the diretory entry from the jail's 'parent directory' to
 the apache directory, and then marking the entry in the original 
 parent as 'unused'.  Nothing other than the  directory whre the jail
 'used to live', and the directory 'where it was found' are touched.
 This occured _through_ the system 'mv' function, so all the normal
 'housekeeping' was done properly.

  2) it was -not- done though mv(2) -- but that requires that a whole 
 *series* of "corruptions" of the filesystem, _ALL_ of which had to 
 occur in 'exactly' the right way.  They are:
   1) The -size- (filesystem metadata) of the orignal parent directory 
  had to be changed to reflect the smaller size.
   2) the 'indirect block' info for the original parent directory had to
  be changed to reflect the absense of the block(s) that are no
  longer part of that file.
   3) the _size_ of the Apache directory had to be increased to reflect
  the additional block(s) that are now part o that directory.
   4) the 'indirect block' info for the apache directory has to be
  changed to reflect the presense of the new block(s) that are now
  part of that file.

This requires multiple -hundreds- of bits 'in error', in a minimum of
FOUR separate disk locations. A -single- failure simply *CANNOT* cause
all of this.

The probability of a random single-bit error in a gigabyte of RAM is on the
order of one such occurance in six months.  The odds of having multiple 
*simultaneous* errors is the probability of a single-bit error raised to
the power of the number of bits in error.  e.g. the probability of a
simultaneous 10-bit radom error is roughly 1 in 30 million years.  The odds
of it being a -specific- ten bits out of that gigabyte is preposterously
small.  The odds of the required specific _multiple-hundreds_ of bits in 
error occuringis (conservatively) 1 in
  (30 million years)**50 * ((2**30)!) / ((2^9)!)

The first factor, alone, is over 7.1E373 years.

I think it is safe to conclude that the probabilities -greatly- favor
alternative #1.

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Sat, Apr 28, 2012 at 3:22 AM, Wojciech Puchar
 wrote:
>> I somewhat agree, but it wasn't a person. I am the only administrator,
>> the only one with root access. The jails were effectively moved to the
>> /usr/local/etc/apache22 of the single that survived at the top level.
>> I'm thinking something between mount, EzJail, the journal and the way
>> MySQL created a great deal of head contention, so something must have
>> gotten corrupted at the directory level like you state, but the
>> strange part is no _data_ corruption as such, because I was able to
>> physically archive the jails, move them to the correct directory and
>
>
> no matter what you do FreeBSD DOES NOT ramdomly move directories. if you are
> sure you didn't move it yourself then it must be machine hardware problem
> but still unlikely.

After a little more research, ___it it NOT unlikely at all___ that
under high distress and a hard boot, UFS could have somehow corrupted
the directory structure, whilst maintaining the data intact. From what
I've learned so far, UFS is actually divided into 2 layers: one that
controls the directory structure and metadata and a lower layer
containing the data, so the directories being screwed up and the data
intact it is actually quite possible.

What I'm trying to do is figure out is how it happened, and try
prevent it from happening again, so instead of dismissing it as
impossibility, I think we all should spend a little time figuring out
how these things can happen and determine how it can be prevented or
reduced.

"Should you find your neighbor's beard catch fire, it's wise to soak one's own"

-- 
Alejandro
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Wojciech Puchar]

I somewhat agree, but it wasn't a person. I am the only administrator,
the only one with root access. The jails were effectively moved to the
/usr/local/etc/apache22 of the single that survived at the top level.
I'm thinking something between mount, EzJail, the journal and the way
MySQL created a great deal of head contention, so something must have
gotten corrupted at the directory level like you state, but the
strange part is no _data_ corruption as such, because I was able to
physically archive the jails, move them to the correct directory and


no matter what you do FreeBSD DOES NOT ramdomly move directories. if you 
are sure you didn't move it yourself then it must be machine 
hardware problem but still unlikely.

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Sat, Apr 28, 2012 at 1:43 AM, Wojciech Puchar
 wrote:
>>
>> All the jails wound up in the /usr/local/etc/apache22 of the only
>> surviving jail which is the http proxy to all the other jails.
>>
>> Right before the server crashed I noticed MySQL at 100% o several CPUs
>> and the server was on it's knees, so I'm wondering was this an
>> attack? is it possible that Apache or MySQL moved the files??
>>
>> I mean the jails are there, I'm even backing them up right now but
>> how did these directories move here?
>>
>> Anybody has ANY logical explanation???
>>
> 99% - someone did moved them.
> 1% - hardware problem possibly memory. without this there is no way for
> directory to be "accidentally" moved

I somewhat agree, but it wasn't a person. I am the only administrator,
the only one with root access. The jails were effectively moved to the
/usr/local/etc/apache22 of the single that survived at the top level.
I'm thinking something between mount, EzJail, the journal and the way
MySQL created a great deal of head contention, so something must have
gotten corrupted at the directory level like you state, but the
strange part is no _data_ corruption as such, because I was able to
physically archive the jails, move them to the correct directory and
archived them all with ezjail-admin to a different disk. I was
thinking of formatting the jails drive, but after all this disk
activity and no errors, and everything booted up correctly, I am not
so sure now that it's needed it.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Wojciech Puchar]

All the jails wound up in the /usr/local/etc/apache22 of the only
surviving jail which is the http proxy to all the other jails.

Right before the server crashed I noticed MySQL at 100% o several CPUs
and the server was on it's knees, so I'm wondering was this an
attack? is it possible that Apache or MySQL moved the files??

I mean the jails are there, I'm even backing them up right now but
how did these directories move here?

Anybody has ANY logical explanation???


99% - someone did moved them.
1% - hardware problem possibly memory. without this there is no way for 
directory to be "accidentally" moved

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Wojciech Puchar]

something. I unmounted the drive and ran fsck and reported no
problems. df shows the data being use so where is the data??


your data is here as df shown usage and fsck see no errors. most probably 
root directory of that volume got corrupted and subdirs were found and put 
in lost+found

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Fri, Apr 27, 2012 at 11:00 PM, Erich Dollansky
 wrote:
> Hi,
>
> On Saturday 28 April 2012 09:33:47 Alejandro Imass wrote:
>> On Fri, Apr 27, 2012 at 7:52 PM, Alejandro Imass  wrote:
>> >
>> > We had a server crash and required a hard reboot. The system is on one
>> > disk and another disc mounts /usr/jails and everything runs in jails,
>> > pristine base system, and the base system is working perfectly.
>> >
>> > The second volume, the one with the jails mounted but every jail
>> > directory disappeared except one. df still shows the data being used
>> > so I'm guessing it's a logical error in the directory structure or
>> > something. I unmounted the drive and ran fsck and reported no
>> > problems. df shows the data being use so where is the data??
>> >
>
> what is du saying?
>>
>> OK, so here is an update, maybe someone has some clue here
>>
>> All the jails wound up in the /usr/local/etc/apache22 of the only
>> surviving jail which is the http proxy to all the other jails.
>
> You want to say that all the data you were looking for have been moved to 
> this directory?
>>

EXACTLY THAT. In fact the data is intact and I have already backed-up
everything to another disk.

>> Right before the server crashed I noticed MySQL at 100% o several CPUs
>> and the server was on it's knees, so I'm wondering was this an
>> attack? is it possible that Apache or MySQL moved the files??
>>
>> I mean the jails are there, I'm even backing them up right now but
>> how did these directories move here?
>>
>> Anybody has ANY logical explanation???
>
> Journaling is new to me. Could this be the cause?
>

Maybe so, I have no idea.

Maybe it's because EzJail mount volumes with each jail or some other
wild explanation. I honestly have never seen this before. I am just
glad that UFS was nice enough to keep my data somewhere at least, and
after my bad experiences with ZFS I can now say with a lot more
certainty that UFS rocks. I mean something got screwed up but the data
was not lost.

Hope someone can shed some light here..

-- 
Alejandro
> Erich
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Erich Dollansky]

Hi,

On Saturday 28 April 2012 09:33:47 Alejandro Imass wrote:
> On Fri, Apr 27, 2012 at 7:52 PM, Alejandro Imass  wrote:
> >
> > We had a server crash and required a hard reboot. The system is on one
> > disk and another disc mounts /usr/jails and everything runs in jails,
> > pristine base system, and the base system is working perfectly.
> >
> > The second volume, the one with the jails mounted but every jail
> > directory disappeared except one. df still shows the data being used
> > so I'm guessing it's a logical error in the directory structure or
> > something. I unmounted the drive and ran fsck and reported no
> > problems. df shows the data being use so where is the data??
> >

what is du saying?
> 
> OK, so here is an update, maybe someone has some clue here
> 
> All the jails wound up in the /usr/local/etc/apache22 of the only
> surviving jail which is the http proxy to all the other jails.

You want to say that all the data you were looking for have been moved to this 
directory?
> 
> Right before the server crashed I noticed MySQL at 100% o several CPUs
> and the server was on it's knees, so I'm wondering was this an
> attack? is it possible that Apache or MySQL moved the files??
> 
> I mean the jails are there, I'm even backing them up right now but
> how did these directories move here?
> 
> Anybody has ANY logical explanation???

Journaling is new to me. Could this be the cause?

Erich
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS Crash and directories now missing

[Alejandro Imass]

On Fri, Apr 27, 2012 at 7:52 PM, Alejandro Imass  wrote:
> Hi folks,
>
> We had a server crash and required a hard reboot. The system is on one
> disk and another disc mounts /usr/jails and everything runs in jails,
> pristine base system, and the base system is working perfectly.
>
> The second volume, the one with the jails mounted but every jail
> directory disappeared except one. df still shows the data being used
> so I'm guessing it's a logical error in the directory structure or
> something. I unmounted the drive and ran fsck and reported no
> problems. df shows the data being use so where is the data??
>

OK, so here is an update, maybe someone has some clue here

All the jails wound up in the /usr/local/etc/apache22 of the only
surviving jail which is the http proxy to all the other jails.

Right before the server crashed I noticed MySQL at 100% o several CPUs
and the server was on it's knees, so I'm wondering was this an
attack? is it possible that Apache or MySQL moved the files??

I mean the jails are there, I'm even backing them up right now but
how did these directories move here?

Anybody has ANY logical explanation???

Thanks,

-- 
Alejandro Imass

> This is FreeBSD 8.2 updated, patched etc. The volume was UFS + Journal
>
> Any help is GREATLY appreciated!
>
> Thanks!
>
> --
> Alejandro Imass
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

UFS Crash and directories now missing

[Alejandro Imass]

Hi folks,

We had a server crash and required a hard reboot. The system is on one
disk and another disc mounts /usr/jails and everything runs in jails,
pristine base system, and the base system is working perfectly.

The second volume, the one with the jails mounted but every jail
directory disappeared except one. df still shows the data being used
so I'm guessing it's a logical error in the directory structure or
something. I unmounted the drive and ran fsck and reported no
problems. df shows the data being use so where is the data??

This is FreeBSD 8.2 updated, patched etc. The volume was UFS + Journal

Any help is GREATLY appreciated!

Thanks!

-- 
Alejandro Imass
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: UFS+SU+J and still background fs check?

[RW]

On Mon, 30 Jan 2012 13:17:16 +0100 (CET)
Marco Beishuizen wrote:

> Hi,
> 
> When booting my computer today I noticed the message at the end:
> starting background filesystem check in 60 seconds. This seems
> strange to me since SU+J is enabled on all filesystems. How is this
> possible? NB running FreeBSD 9-STABLE

It just means that the fsck process that would perform background fsck
for any filesystem that supports it and isn't clean will run in 60
second. You can turn it off with background_fsck="NO" in rc.conf,  which
will disable background fsck support.  

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

UFS+SU+J and still background fs check?

[Marco Beishuizen]

Hi,

When booting my computer today I noticed the message at the end: starting 
background filesystem check in 60 seconds. This seems strange to me since 
SU+J is enabled on all filesystems. How is this possible? NB running 
FreeBSD 9-STABLE


Regards,
Marco
--
In most instances, all an argument
proves is that two people are present.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Old UFS label complains on 9.0

[David Demelier]

Hello,

I've just updated my laptop from 8.2-RELEASE to 9.0-RELEASE, everything 
worked but the kernel complains about labels see :


g_dev_taste: make_dev_p() failed (gp->name=vol/root, error=17)
g_dev_taste: make_dev_p() failed (gp->name=vol/root, error=17)
g_dev_taste: make_dev_p() failed (gp->name=vol/var, error=17)
g_dev_taste: make_dev_p() failed (gp->name=vol/var, error=17)
g_dev_taste: make_dev_p() failed (gp->name=vol/tmp, error=17)
g_dev_taste: make_dev_p() failed (gp->name=vol/tmp, error=17)

The labels works because my fstab relies on them.

How can I fix that?

Cheers,

--
David Demelier
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: restore(8) to UFS on USB key: terrible slow

[Michael Sierchio]

Cheap USB drives, and even many CF drives, aren't much good as random
read-write devices.  On my Soekris boxen I run FreeBSD, and mount the
root filesystem rw,noatime.  And I don't write to it. ;-)  /var is a
memory filesystem, there /var/db/... contain symbolic links to
/usr/local/db/.. because the package database can grow quite large.
/tmp is a symlink to /var/tmp.

Configured this way, these machines are trouble-free.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: restore(8) to UFS on USB key: terrible slow

[Julian H. Stacey]

Matthias Apitz wrote:
> El día Thursday, December 08, 2011 a las 09:57:13AM -0600, Dan Nelson 
> escribió:
> 
> > Cheap USB thumb drives aren't really optimized for small random-I/O writes. 
> > Can you try mounting the filesystem async?  that might help a little.  A
> > workaround would be to use mdconfig to create a block device (backed by
> > either swap or a file on your hard drive) the same size as your flash drive,
> > newfs and restore to that, then umount the filesystem and dd the raw image
> > directly to your flash drive.
> 
> Hello Dan,
> 
> Thanks for your hints. I tend to add that those USB thum drives aren't
> good for anything. I have a certain number of them containing each a
> complete bootable FreeBSD (including 'src', 'obj' and binary packages)
> to install my laptops and netbooks from them;
> 
> after some time these USB keys tend to loose data:
> files are corrupt a bit, dirs are missing and so on; that's
> why I wanted to make dump(8) nackups of them, to restore them from time
> to time; I will drop the idea and will just make dd(1) backups of the
> full /dev/da0;

Additional to all the other good points others wrote earlier,
may I mention: ...

I've found some sticks are slower than others.
Sometimes I do a performance & integrity test with my
http://berklix.com/~jhs/src/bsd/jhs/bin/public/testblock

One (free promo) stick I found lies, see this comment in my
http://berklix.com/~jhs/src/bsd/fixes/FreeBSD/src/jhs/etc/devd/jhs.conf
#   The end of it is write only memory !
#   It lets one write the last chunck,
#   It even lets one read that last chunk,
#   but the content of the last chunck is all zeroes.

Another 2G stick was particularly slow: (marked as) Sony,
bought at a computer Sunday 'flea' market in Croydon England, 
in retrospect I wonder if manufacturer Sony might have withdrawn them
from sale, marked the batch for destruction, & possibly some criminal
'liberated' them for resale again ?

(Such things do happen, eg In Germany years back, pre USB
era, CT Mag reported reported Betruger/Placebo cache chips.
They were just ceramic with no silicon in, it was reported
importers (in Munich I think) were afraid to sue chinese
exporters, fear of Triads! maybe last bit was speculation,
but wan't My speculation, I read it, whatever, can't remember
more now)

Block Sizes:
Maybe USB sticks may have different size/ speed front end
cache chips on USB sticks ? Hans would know I suppose. ?

Apart from soft updates, one can also choose the block sizes
newfs creates, I recall FFS is larger than UFS ?.  

Maybe we should send-pr some suggested size for man newfs
if targeting images for USB sticks.
(is that a question to consider jointly with fs@ list ? )

Voltages:
I've  recently been bitten by appalling problems on a bunch
of 2 of my externals discs, using 2 different laptops, 2/3
hubs, & 3 power supplies.  Various combinations come back
to bad voltage regulation, usually too low, some too high.

But I assume discs will be more susceptible than sticks.

However next time a motherboard fails for any of you, I
suggest don't discard, first hacksaw off the double USB socket,
solder wires across, add extra wires for a meter, so you
can monitor voltage & current.

Mastering first on hard disc (per Dan's suggestion, mdconfig etc)
is a good idea, I was considering this earlier when building
a new stick/ extended Live-FS. .. using mdconfig etc,
but it's heavy & slow after the initial image create, to keep
rewriting, even if at large dd bs=

So I use incremental writes 
I keep personal backups & bins & Live FS & mp3 to play etc
all on USB sticks.  Still usable though 'cos I rarely change
too much at one time. & rdist6 updates what's changed.
(would also correct odd corruption Matthias)

I even sue gbde encrypted FS (ie more performance degradation)
.. and updates still happens acceptably if not exactly fast.
Others could use rsync if they dont fancy rdist6.

Cheers,
Julian
-- 
Julian Stacey, BSD Unix Linux C Sys Eng Consultants Munich http://berklix.com
 Reply below not above, cumulative like a play script, & indent with "> ".
 Format: Plain text. Not HTML, multipart/alternative, base64, quoted-printable.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"