Re: Causes, cures and prevention of orphaned inodes?

[Stephen P. Molnar]

On Tue, 2018-02-06 at 09:56 +1300, Ben Caradoc-Davies wrote:
> On 06/02/18 04:52, Stephen P. Molnar wrote:
> > I installed memtest86+ and ran it with all of the defaults.  It
> > took
> > over an hour, but no errors were reported.
> 
> Please try parallel memtester and stress. These found memory errors
> for 
> me that were not found by memtest86+.
> 
> In *eight* terminal windows as root run (noting that you have 8GB
> RAM):
> 
> memtester 768M
> 
> In one terminal window run (nothing that the FX-8320 has only 4 FPUs
> and 
> stress -c spins on sqrt):
> 
> stress -c 4
> 
> You can also tell memtester to use multiple iterations:
> 
> memtester 768M 10
> 
> > I am rather hesitant about updating the BIOS/UFEI.  In fact I can't
> > seem to find an upgrade for the FX-8320 on the AMD web site.
> 
> The BIOS/UEFI upgrade is an upgrade for your motherboard and can be 
> obtained from your motherboard vendor. For the Asus M5A97 R2.0:
> https://www.asus.com/us/Motherboards/M5A97_R20/HelpDesk_BIOS/
> 
> You are currently running "Version 2006 2013/10/16".
> Please try upgrading to "Version 2603 2015/07/24".
> 
> Note the multiple entries entitled "Improve system stability".
> 
> > Also, I'm rather hesitant about installing the amd64-package.
> 
> wat? Why? Modern CPUs are released full of microcode bugs. AMD
> release 
> new microcode that they wrote to fix these bugs in their microcode
> and 
> that they recommend. Why would you not use it?
> 
> Kind regards,
> 
-- 

Thank you for your fine and detailed email.  It has set my mind at ease.

Stephen P. Molnar, Ph.D.
Consultant
www.molecular-modeling.net
(614)312-7528 (c)
Skype: smolnar1

Re: Causes, cures and prevention of orphaned inodes?

[Ben Caradoc-Davies]

On 06/02/18 04:52, Stephen P. Molnar wrote:

I installed memtest86+ and ran it with all of the defaults.  It took
over an hour, but no errors were reported.


Please try parallel memtester and stress. These found memory errors for 
me that were not found by memtest86+.


In *eight* terminal windows as root run (noting that you have 8GB RAM):

memtester 768M

In one terminal window run (nothing that the FX-8320 has only 4 FPUs and 
stress -c spins on sqrt):


stress -c 4

You can also tell memtester to use multiple iterations:

memtester 768M 10


I am rather hesitant about updating the BIOS/UFEI.  In fact I can't
seem to find an upgrade for the FX-8320 on the AMD web site.


The BIOS/UEFI upgrade is an upgrade for your motherboard and can be 
obtained from your motherboard vendor. For the Asus M5A97 R2.0:

https://www.asus.com/us/Motherboards/M5A97_R20/HelpDesk_BIOS/

You are currently running "Version 2006 2013/10/16".
Please try upgrading to "Version 2603 2015/07/24".

Note the multiple entries entitled "Improve system stability".


Also, I'm rather hesitant about installing the amd64-package.


wat? Why? Modern CPUs are released full of microcode bugs. AMD release 
new microcode that they wrote to fix these bugs in their microcode and 
that they recommend. Why would you not use it?


Kind regards,

--
Ben Caradoc-Davies 
Director
Transient Software Limited 
New Zealand

Re: Causes, cures and prevention of orphaned inodes?

[Ben Caradoc-Davies]

On 06/02/18 07:32, David Wright wrote:

I'm not in the habit of upgrading BIOS/UEFI on my computers.
(I do have {amd64,intel}-microcode installed.) What old or
buggy code would I be running when booting a linux installation?
(I accept that Grub has to run, and the kernel and initramfs be
found, initially.)


Your BIOS/UEFI may perform boot-time memory training to adjust memory 
frequency and voltages. I think it might also be responsible for setting 
CPU and other chipset voltages during operation. These have a profound 
impact on system stability. I always upgrade my BIOS/UEFI to take 
advantage of any improvements recommended by the manufacturer.


Kind regards,

--
Ben Caradoc-Davies 
Director
Transient Software Limited 
New Zealand

Re: Causes, cures and prevention of orphaned inodes?

[Stephen P. Molnar]

On Mon, 2018-02-05 at 19:40 +0100, Felipe Salvador wrote:
> On Sun, Feb 04, 2018 at 11:06:21PM -0200, Henrique de Moraes Holschuh
> wrote:
> > On Sun, 04 Feb 2018, Stephen P. Molnar wrote:
> > > I am running Debian Stretch on am eight thread AMD GPU platform.
> > > Lately, it seems if I have been plagued by surfeit of orphaned
> > > nodes.
> > 
> > That means:
> > 
> > 1. "unlinked" files or directories were still open when the
> > filesystem
> >    had to be shutdown/made read-only.  Uncommon, unless you did
> > something
> >    like breaking the system by having /run on something other than
> >    tmpfs, etc.
> > 
> > 2. Filesystem corruption, most likely due to bad RAM.
> > 
> > 3. Really unusual workload and usage (or *mis*usage ?) pattern that
> >    results in (1), perhaps involving VMs.
> 
> Could it be due to an undersized(or faulty) power supply?
> 
> Could a barely sized power supply, on a heavy load scenario,
> lead to file system corruption?
> 
> Thank you
> 
> > This assumes XFS or ext3/ext4, which seems to be your case.
> > 
> > > At first I though it might be impending hard drive failure as I
> > > started
> > > receiving warning messages from the OS, but installing a brand
> > > new
> > > drive has not solved the problem,  They seem to happen when I am
> > > running  four or more apps at the same time. I have 8GB of RAM in
> > > the
> > > system.
> > 
> > Test that RAM.
> > 
> > > Model name:AMD FX(tm)-8320 Eight-Core Processor
> > 
> > Ensure you are running the latest AMD microcode on that processor:
> > Update the motherboard firmware (BIOS/UEFI) to the latest version
> > available from your vendor, and *also* install package amd64-
> > microcode
> > from non-free.  Unless you're on the latest AMD microcode on the
> > FX-8300, you cannot even run VMs safely.  Who knows what else could
> > be
> > broken...
> > 
> > -- 
> >   Henrique Holschuh
> 
> Regards


That's an excellent question.  I really don't know.

-- 
Stephen P. Molnar, Ph.D.
Consultant
www.molecular-modeling.net
(614)312-7528 (c)
Skype: smolnar1

Re: Causes, cures and prevention of orphaned inodes?

[Felipe Salvador]

On Sun, Feb 04, 2018 at 11:06:21PM -0200, Henrique de Moraes Holschuh wrote:
> On Sun, 04 Feb 2018, Stephen P. Molnar wrote:
> > I am running Debian Stretch on am eight thread AMD GPU platform.
> > Lately, it seems if I have been plagued by surfeit of orphaned nodes.
> 
> That means:
> 
> 1. "unlinked" files or directories were still open when the filesystem
>had to be shutdown/made read-only.  Uncommon, unless you did something
>like breaking the system by having /run on something other than
>tmpfs, etc.
> 
> 2. Filesystem corruption, most likely due to bad RAM.
> 
> 3. Really unusual workload and usage (or *mis*usage ?) pattern that
>results in (1), perhaps involving VMs.

Could it be due to an undersized(or faulty) power supply?

Could a barely sized power supply, on a heavy load scenario,
lead to file system corruption?

Thank you

> This assumes XFS or ext3/ext4, which seems to be your case.
> 
> > At first I though it might be impending hard drive failure as I started
> > receiving warning messages from the OS, but installing a brand new
> > drive has not solved the problem,  They seem to happen when I am
> > running  four or more apps at the same time. I have 8GB of RAM in the
> > system.
> 
> Test that RAM.
> 
> > Model name:AMD FX(tm)-8320 Eight-Core Processor
> 
> Ensure you are running the latest AMD microcode on that processor:
> Update the motherboard firmware (BIOS/UEFI) to the latest version
> available from your vendor, and *also* install package amd64-microcode
> from non-free.  Unless you're on the latest AMD microcode on the
> FX-8300, you cannot even run VMs safely.  Who knows what else could be
> broken...
> 
> -- 
>   Henrique Holschuh

Regards
-- 
Felipe Salvador

Re: Causes, cures and prevention of orphaned inodes?

[David Wright]

On Mon 05 Feb 2018 at 14:45:17 (-0200), Henrique de Moraes Holschuh wrote:
> On Mon, 05 Feb 2018, Stephen P. Molnar wrote:

> > I am rather hesitant about updating the BIOS/UFEI.  In fact I can't
> > seem to find an upgrade for the FX-8320 on the AMD web site.
> 
> An update, if any are available, would be found in the ASUSTEK support
> page for your motherboard.
> 
> > Also, I'm rather hesitant about installing the amd64-package.
> 
> If you never updated that BIOS/UEFI, chances are you are running very
> old microcode, which is known to be buggy, and unsupported.  There is
> one extremely nasty VM security issue in that processor, as well as some
> isses that could result in data corruption in rare cases.  They must be
> patched by the most recent AMD microcode update for correct system
> operation.

I'm not in the habit of upgrading BIOS/UEFI on my computers.
(I do have {amd64,intel}-microcode installed.) What old or
buggy code would I be running when booting a linux installation?
(I accept that Grub has to run, and the kernel and initramfs be
found, initially.)

Cheers,
David.

Re: Causes, cures and prevention of orphaned inodes?

[Cindy-Sue Causey]

On 2/4/18, Henrique de Moraes Holschuh  wrote:
> On Sun, 04 Feb 2018, Stephen P. Molnar wrote:
>> I am running Debian Stretch on am eight thread AMD GPU platform.
>> Lately, it seems if I have been plagued by surfeit of orphaned nodes.
>
> That means:
>
> 1. "unlinked" files or directories were still open when the filesystem
>had to be shutdown/made read-only.  Uncommon, unless you did something
>like breaking the system by having /run on something other than
>tmpfs, etc.


As soon as I grasped the context, similar was my kneejerk thought. I
see that inode message related to either power failure or my having to
hardcore shut the power off rather than shut down a safer, more
logical way that protects data. So my thinking is that it could be
about something [buggy] not shutting down properly or simply just in
time even if a "proper" reboot/shutdown is being performed..

For some reason, I'm tying it most often to browsers in my case, but I
can't remember why I always think that. Maybe it's most often about
browsers seeming to cause the situation or something ? :)

Cindy :)
-- 
Cindy-Sue Causey
Talking Rock, Pickens County, Georgia, USA

* runs with duct tape *

Re: Causes, cures and prevention of orphaned inodes?

[Jochen Spieker]

Stephen P. Molnar:
> 
> I installed memtest86+ and ran it with all of the defaults.  It took
> over an hour, but no errors were reported.

That's not long enough. From what I have read you should let it run for
a day or so and even then you cannot be sure that there are no memory
errors.

> I am rather hesitant about updating the BIOS/UFEI.  In fact I can't
> seem to find an upgrade for the FX-8320 on the AMD web site.

You need a UEFI update for your mainboard, not the CPU.

> Also, I'm rather hesitant about installing the amd64-package.

Do it (installing the amd64-microcode package). Henrique knows about
this stuff better than most people. Also, there is comparatively little
risk involved since the microcode is loaded during boot. You can always
boot from another medium and remove the package if it doesn't work.

J.
-- 
In this bunker there are women and children. There are no weapons.
[Agree]   [Disagree]
 


signature.asc
Description: PGP signature

Re: Causes, cures and prevention of orphaned inodes?

[Henrique de Moraes Holschuh]

On Mon, 05 Feb 2018, Stephen P. Molnar wrote:
> I installed memtest86+ and ran it with all of the defaults.  It took
> over an hour, but no errors were reported.

Memtest86 and memtest86+ are not always that good at finding memory
errors for some reason...  Try a 24h/48h test over the weekend or
something like that if you really want to be sure.

> I am rather hesitant about updating the BIOS/UFEI.  In fact I can't
> seem to find an upgrade for the FX-8320 on the AMD web site.

An update, if any are available, would be found in the ASUSTEK support
page for your motherboard.

> Also, I'm rather hesitant about installing the amd64-package.

If you never updated that BIOS/UEFI, chances are you are running very
old microcode, which is known to be buggy, and unsupported.  There is
one extremely nasty VM security issue in that processor, as well as some
isses that could result in data corruption in rare cases.  They must be
patched by the most recent AMD microcode update for correct system
operation.

The amd64-microcode packages do not make any permanent changes to the
system, you can just remove them (and re-create the initramfs) if you
want to.

But it is up to you, really.  Your system is clearly misbehaving (unless
you like to power it off improperly or something, in which case the
orphan inodes are *expected* and you should have told us that first
thing :p).

-- 
  Henrique Holschuh

Re: Causes, cures and prevention of orphaned inodes?

[Stephen P. Molnar]

On Mon, 2018-02-05 at 10:15 +1300, Ben Caradoc-Davies wrote:
> On 05/02/18 09:49, Stephen P. Molnar wrote:
> > They seem to happen when I am
> > running  four or more apps at the same time.
> 
> I would never expect to see orphaned inodes except after a system
> crash 
> or kernel memory corruption. How did you test your CPU and RAM? Do
> you 
> see any other symptoms such as segfaults that could suggest memory 
> problems under concurrent load? How long have you seen this problem?
> I 
> see you are using ext4; are the inodes on these filesystems? ext4 is 
> very well tested and robust.
> 
> My preferred memory test for my 4-core (8-thread) Kaby Lake i7 is to
> run 
> concurrent "memtester" instances equal to the number of cores (4 in
> my 
> case), concurrent with "stress" equal to the number of cores ("stress
> -c 
> 4" in my case). This workout detected memory problems not found by
> other 
> tools such as "memtest86+" or "mprime -t".
> 
> Other hardware issues to consider are overheating (addressed with
> better 
> cooling and thermald) and power supply problems which may only be 
> evident at load. Is your system prime stable (i.e. runs with "mprime
> -t" 
> (with AVX disabled) for many hours)? How do you monitor system
> temperature?
> 
> Kind regards,
> 
Thanks for your reply.

As a matter of fact I did get an overheating warning from the OS when
running a rather large organic molecule on the Orca package with 8
threads.  I upgraded the CPU cooler to a Hyper 212 EVO and the problem
went away.

-- 
Stephen P. Molnar, Ph.D.
Consultant
www.molecular-modeling.net
(614)312-7528 (c)
Skype: smolnar1

Re: Causes, cures and prevention of orphaned inodes?

[Stephen P. Molnar]

On Sun, 2018-02-04 at 23:06 -0200, Henrique de Moraes Holschuh wrote:
> On Sun, 04 Feb 2018, Stephen P. Molnar wrote:
> > I am running Debian Stretch on am eight thread AMD GPU platform.
> > Lately, it seems if I have been plagued by surfeit of orphaned
> > nodes.
> 
> That means:
> 
> 1. "unlinked" files or directories were still open when the
> filesystem
>    had to be shutdown/made read-only.  Uncommon, unless you did
> something
>    like breaking the system by having /run on something other than
>    tmpfs, etc.
> 
> 2. Filesystem corruption, most likely due to bad RAM.
> 
> 3. Really unusual workload and usage (or *mis*usage ?) pattern that
>    results in (1), perhaps involving VMs.
> 
> This assumes XFS or ext3/ext4, which seems to be your case.
> 
> > At first I though it might be impending hard drive failure as I
> > started
> > receiving warning messages from the OS, but installing a brand new
> > drive has not solved the problem,  They seem to happen when I am
> > running  four or more apps at the same time. I have 8GB of RAM in
> > the
> > system.
> 
> Test that RAM.
> 
> > Model name:AMD FX(tm)-8320 Eight-Core Processor
> 
> Ensure you are running the latest AMD microcode on that processor:
> Update the motherboard firmware (BIOS/UEFI) to the latest version
> available from your vendor, and *also* install package amd64-
> microcode
> from non-free.  Unless you're on the latest AMD microcode on the
> FX-8300, you cannot even run VMs safely.  Who knows what else could
> be
> broken...
> 

Thanks for your reply.

I installed memtest86+ and ran it with all of the defaults.  It took
over an hour, but no errors were reported.

I am rather hesitant about updating the BIOS/UFEI.  In fact I can't
seem to find an upgrade for the FX-8320 on the AMD web site.

Also, I'm rather hesitant about installing the amd64-package.

-- 
Stephen P. Molnar, Ph.D.
Consultant
www.molecular-modeling.net
(614)312-7528 (c)
Skype: smolnar1

Re: Causes, cures and prevention of orphaned inodes?

[Stephen P. Molnar]

On Mon, 2018-02-05 at 08:03 +0900, Mark Fletcher wrote:
> On Sun, Feb 04, 2018 at 03:49:36PM -0500, Stephen P. Molnar wrote:
> > I am running Debian Stretch on am eight thread AMD GPU platform.
> > Lately, it seems if I have been plagued by surfeit of orphaned
> > nodes.
> > 
> > I have goggled the causes. cures and prevention, but have gotten no
> > results that make any sense to me. I've been using computer since
> > the
> > early 1960's but am an organic chemist by training and experience,
> > not
> > a hardware expert.
> > 
> 
> The problem may not be hardware. From reading the details you
> provided, 
> it looks like you are using ext4 filesystems on your disks. Is that 
> correct? We occasionally get people on here reporting problems with
> more 
> esoteric / exotic file systems (cue the cries of protest from
> various 
> corners that super-duper-dijeridoo-fs isn't exotic, and that I'm a 
> dinosaur) but ext4 is in very wide use and as far as I know, stable.
> 
> Anyway worth confirming what filesystem(s) is/are actually on the
> disks 
> where orphaned inodes are occurring. If it is something more
> unusual, 
> you might have found a bug in the filesystem. Also, do you use 
> encryption on your disks eg LUKS?
> 
> Just a couple of thoughts
> 
> Mark.


I appreciate your suggestion.

Here are the results of blkid:

Installed drives:
root@AbNormal:/home/comp# blkid
/dev/sda1: UUID="8fa0b985-70ca-4d3e-a448-1a419d8b078b" TYPE="ext4" 
PARTUUID="b5226506-01"
/dev/sda5: UUID="3d0d7ebe-26f4-4f0e-be01-dca7ce9c9132" TYPE="swap" 
PARTUUID="b5226506-05"
/dev/sdb1: UUID="d65867da-c658-4e35-928c-9dd2d6dd5742" TYPE="ext4" 
PARTUUID="0003d403-01"
/dev/sdb2: UUID="007c1f16-34a4-438c-9d15-e3df601649ba" TYPE="ext4" 
PARTUUID="0003d403-02"

External USB Thumb drive:
root@AbNormal:/home/comp# blkid
/dev/sda1: UUID="8fa0b985-70ca-4d3e-a448-1a419d8b078b" TYPE="ext4" 
PARTUUID="b5226506-01"
/dev/sda5: UUID="3d0d7ebe-26f4-4f0e-be01-dca7ce9c9132" TYPE="swap" 
PARTUUID="b5226506-05"
/dev/sdb1: UUID="d65867da-c658-4e35-928c-9dd2d6dd5742" TYPE="ext4" 
PARTUUID="0003d403-01"
/dev/sdb2: UUID="007c1f16-34a4-438c-9d15-e3df601649ba" TYPE="ext4" 
PARTUUID="0003d403-02"
/dev/sdc1: UUID="28C1-0F73" TYPE="vfat"

External Backup drive:
root@AbNormal:/home/comp# blkid
/dev/sda1: UUID="8fa0b985-70ca-4d3e-a448-1a419d8b078b" TYPE="ext4" 
PARTUUID="b5226506-01"
/dev/sda5: UUID="3d0d7ebe-26f4-4f0e-be01-dca7ce9c9132" TYPE="swap" 
PARTUUID="b5226506-05"
/dev/sdb1: UUID="d65867da-c658-4e35-928c-9dd2d6dd5742" TYPE="ext4" 
PARTUUID="0003d403-01"
/dev/sdb2: UUID="007c1f16-34a4-438c-9d15-e3df601649ba" TYPE="ext4" 
PARTUUID="0003d403-02"
/dev/sdc1: LABEL="Seagate Expansion Drive" UUID="F0DAF608DAF5CABC" TYPE="ntfs" 
PARTUUID="6cccaf93-01"

However, I usually gt et the orphaned inodes where there are no exteranl drives 
mounted.

-- 
Stephen P. Molnar, Ph.D.
Consultant
www.molecular-modeling.net
(614)312-7528 (c)
Skype: smolnar1

Re: Causes, cures and prevention of orphaned inodes?

[Stephen P. Molnar]

On Sun, 2018-02-04 at 20:45 -0500, Gene Heskett wrote:
> On Sunday 04 February 2018 15:49:36 Stephen P. Molnar wrote:
> 
> > I am running Debian Stretch on am eight thread AMD GPU platform.
> > Lately, it seems if I have been plagued by surfeit of orphaned
> nodes.
> >
> > I have goggled the causes. cures and prevention, but have gotten no
> > results that make any sense to me. I've been using computer since
> the
> > early 1960's but am an organic chemist by training and experience,
> not
> > a hardware expert.
> >
> > At first I though it might be impending hard drive failure as I
> > started receiving warning messages from the OS, but installing a
> brand
> > new drive has not solved the problem,  They seem to happen when I
> am
> > running  four or more apps at the same time. I have 8GB of RAM in
> the
> > system.
> >
> > I've installed several tools, lscup and lsh, but in all candor I
> don't
> > have the faintest idea what to make of the results. I'm not even
> sure
> > if the results have any application to solutions for the problem.
> >
> > Can anyone give me some guidance in what I should be looking
> for?  It
> > would be much appreciated.
> >
> > Thanks in advance.
> 
> Are you perchance running clamav? version 99.3's first release about
> 2 
> weeks back used up the available unix sockets, bricking the machine
> till 
> a reboot, wash, rinse, repeat. A new 99.3 has fixed that, but until
> that 
> filters thru the distro channels, you should shut clamav down. So I 
> have, and have commented out, the calls to inspect incoming emails
> out 
> of my .procmailrc.
> 
> Cheers, Gene Heskett
> --
> "There are four boxes to be used in defense of liberty:
>  soap, ballot, jury, and ammo. Please use in that order."
> -Ed Howdershelt (Author)
> Genes Web page 
> 

Thanks for the suggestion.

No. I'm not using clamav.
-- 
Stephen P. Molnar, Ph.D.
Consultant
www.molecular-modeling.net
(614)312-7528 (c)
Skype: smolnar1

Re: Causes, cures and prevention of orphaned inodes?

[Stephen P. Molnar]

On Mon, 2018-02-05 at 07:37 +0100, deloptes wrote:
> Stephen P. Molnar wrote:
> 
> > Can anyone give me some guidance in what I should be looking
> > for?  It
> > would be much appreciated.
> > From my experience most probably inappropriate shutdown (no unmount
> > when
> 
> shutdown).
> How do you shutdown your machine?
> Can you try without systemd (just install sysvinit and add to the
> kernel
> cmdline at boot: init=/lib/sysvinit/init)
> 
> regards
> 
> 
I am always greatful for the great advice, freely given by the folks
reading the Debian-users list. 

I always shut down the system with the Shutdown command.  The only
exception would be a locked system, which is the only recourse I know
of for orphaned inodes?

-- 
Stephen P. Molnar, Ph.D.
Consultant
www.molecular-modeling.net
(614)312-7528 (c)
Skype: smolnar1

Re: Causes, cures and prevention of orphaned inodes?

[deloptes]

Stephen P. Molnar wrote:

> Can anyone give me some guidance in what I should be looking for?  It
> would be much appreciated.

>From my experience most probably inappropriate shutdown (no unmount when
shutdown).
How do you shutdown your machine?
Can you try without systemd (just install sysvinit and add to the kernel
cmdline at boot: init=/lib/sysvinit/init)

regards

Re: Causes, cures and prevention of orphaned inodes?

[David Christensen]

On 2/4/18 12:49 PM, Stephen P. Molnar wrote:

receiving warning messages from the OS,


Please post the warning messages from the OS, and identify where they 
are coming from.



Please run 'mount' and post the prompt, the command, and the relevant 
portions of the output.  Please do the same for 'df' and 'dmesg'.



David

Re: Causes, cures and prevention of orphaned inodes?

[The Wanderer]

On 2018-02-04 at 20:06, Henrique de Moraes Holschuh wrote:

> On Sun, 04 Feb 2018, Stephen P. Molnar wrote:
>
>> I am running Debian Stretch on am eight thread AMD GPU platform.
>> Lately, it seems if I have been plagued by surfeit of orphaned nodes.
> 
> That means:
> 
> 1. "unlinked" files or directories were still open when the filesystem
>had to be shutdown/made read-only.  Uncommon, unless you did something
>like breaking the system by having /run on something other than
>tmpfs, etc.

I see a handful of this type of notification not uncommonly on boot-up,
and I suspect it's from this cause, for the simple reason that most of
my shutdowns are unclean; specifically, most of my shutdowns are due to
power loss. I don't tend to shut the machine down on purpose very often.

I don't find it surprising that orphaned inodes happen when the
filesystem didn't get a chance to clean up in the normal course of
operation.

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man. -- George Bernard Shaw



signature.asc
Description: OpenPGP digital signature

Re: Causes, cures and prevention of orphaned inodes?

[Gene Heskett]

On Sunday 04 February 2018 15:49:36 Stephen P. Molnar wrote:

> I am running Debian Stretch on am eight thread AMD GPU platform.
> Lately, it seems if I have been plagued by surfeit of orphaned nodes.
>
> I have goggled the causes. cures and prevention, but have gotten no
> results that make any sense to me. I've been using computer since the
> early 1960's but am an organic chemist by training and experience, not
> a hardware expert.
>
> At first I though it might be impending hard drive failure as I
> started receiving warning messages from the OS, but installing a brand
> new drive has not solved the problem,  They seem to happen when I am
> running  four or more apps at the same time. I have 8GB of RAM in the
> system.
>
> I've installed several tools, lscup and lsh, but in all candor I don't
> have the faintest idea what to make of the results. I'm not even sure
> if the results have any application to solutions for the problem.
>
> Can anyone give me some guidance in what I should be looking for?  It
> would be much appreciated.
>
> Thanks in advance.

Are you perchance running clamav? version 99.3's first release about 2 
weeks back used up the available unix sockets, bricking the machine till 
a reboot, wash, rinse, repeat. A new 99.3 has fixed that, but until that 
filters thru the distro channels, you should shut clamav down. So I 
have, and have commented out, the calls to inspect incoming emails out 
of my .procmailrc.

Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 

Re: Causes, cures and prevention of orphaned inodes?

[Henrique de Moraes Holschuh]

On Sun, 04 Feb 2018, Stephen P. Molnar wrote:
> I am running Debian Stretch on am eight thread AMD GPU platform.
> Lately, it seems if I have been plagued by surfeit of orphaned nodes.

That means:

1. "unlinked" files or directories were still open when the filesystem
   had to be shutdown/made read-only.  Uncommon, unless you did something
   like breaking the system by having /run on something other than
   tmpfs, etc.

2. Filesystem corruption, most likely due to bad RAM.

3. Really unusual workload and usage (or *mis*usage ?) pattern that
   results in (1), perhaps involving VMs.

This assumes XFS or ext3/ext4, which seems to be your case.

> At first I though it might be impending hard drive failure as I started
> receiving warning messages from the OS, but installing a brand new
> drive has not solved the problem,  They seem to happen when I am
> running  four or more apps at the same time. I have 8GB of RAM in the
> system.

Test that RAM.

> Model name:AMD FX(tm)-8320 Eight-Core Processor

Ensure you are running the latest AMD microcode on that processor:
Update the motherboard firmware (BIOS/UEFI) to the latest version
available from your vendor, and *also* install package amd64-microcode
from non-free.  Unless you're on the latest AMD microcode on the
FX-8300, you cannot even run VMs safely.  Who knows what else could be
broken...

-- 
  Henrique Holschuh

Re: Causes, cures and prevention of orphaned inodes?

[Mark Fletcher]

On Sun, Feb 04, 2018 at 03:49:36PM -0500, Stephen P. Molnar wrote:
> I am running Debian Stretch on am eight thread AMD GPU platform.
> Lately, it seems if I have been plagued by surfeit of orphaned nodes.
> 
> I have goggled the causes. cures and prevention, but have gotten no
> results that make any sense to me. I've been using computer since the
> early 1960's but am an organic chemist by training and experience, not
> a hardware expert.
> 

The problem may not be hardware. From reading the details you provided, 
it looks like you are using ext4 filesystems on your disks. Is that 
correct? We occasionally get people on here reporting problems with more 
esoteric / exotic file systems (cue the cries of protest from various 
corners that super-duper-dijeridoo-fs isn't exotic, and that I'm a 
dinosaur) but ext4 is in very wide use and as far as I know, stable.

Anyway worth confirming what filesystem(s) is/are actually on the disks 
where orphaned inodes are occurring. If it is something more unusual, 
you might have found a bug in the filesystem. Also, do you use 
encryption on your disks eg LUKS?

Just a couple of thoughts

Mark

Re: Causes, cures and prevention of orphaned inodes?

[Ben Caradoc-Davies]

On 05/02/18 09:49, Stephen P. Molnar wrote:

They seem to happen when I am
running  four or more apps at the same time.


I would never expect to see orphaned inodes except after a system crash 
or kernel memory corruption. How did you test your CPU and RAM? Do you 
see any other symptoms such as segfaults that could suggest memory 
problems under concurrent load? How long have you seen this problem? I 
see you are using ext4; are the inodes on these filesystems? ext4 is 
very well tested and robust.


My preferred memory test for my 4-core (8-thread) Kaby Lake i7 is to run 
concurrent "memtester" instances equal to the number of cores (4 in my 
case), concurrent with "stress" equal to the number of cores ("stress -c 
4" in my case). This workout detected memory problems not found by other 
tools such as "memtest86+" or "mprime -t".


Other hardware issues to consider are overheating (addressed with better 
cooling and thermald) and power supply problems which may only be 
evident at load. Is your system prime stable (i.e. runs with "mprime -t" 
(with AVX disabled) for many hours)? How do you monitor system temperature?


Kind regards,

--
Ben Caradoc-Davies 
Director
Transient Software Limited 
New Zealand

Causes, cures and prevention of orphaned inodes?

[Stephen P. Molnar]

I am running Debian Stretch on am eight thread AMD GPU platform.
Lately, it seems if I have been plagued by surfeit of orphaned nodes.

I have goggled the causes. cures and prevention, but have gotten no
results that make any sense to me. I've been using computer since the
early 1960's but am an organic chemist by training and experience, not
a hardware expert.

At first I though it might be impending hard drive failure as I started
receiving warning messages from the OS, but installing a brand new
drive has not solved the problem,  They seem to happen when I am
running  four or more apps at the same time. I have 8GB of RAM in the
system.

I've installed several tools, lscup and lsh, but in all candor I don't
have the faintest idea what to make of the results. I'm not even sure
if the results have any application to solutions for the problem.

Can anyone give me some guidance in what I should be looking for?  It
would be much appreciated.

Thanks in advance.
Architecture:  x86_64
CPU op-mode(s):32-bit, 64-bit
Byte Order:Little Endian
CPU(s):8
On-line CPU(s) list:   0-7
Thread(s) per core:2
Core(s) per socket:4
Socket(s): 1
NUMA node(s):  1
Vendor ID: AuthenticAMD
CPU family:21
Model: 2
Model name:AMD FX(tm)-8320 Eight-Core Processor
Stepping:  0
CPU MHz:   1400.000
CPU max MHz:   3500.
CPU min MHz:   1400.
BogoMIPS:  7023.93
Virtualization:AMD-V
L1d cache: 16K
L1i cache: 64K
L2 cache:  2048K
L3 cache:  8192K
NUMA node0 CPU(s): 0-7
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb 
rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf 
eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave 
avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 
3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext 
perfctr_core perfctr_nb cpb hw_pstate vmmcall bmi1 arat npt lbrv svm_lock 
nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
abnormal
description: Desktop Computer
product: To be filled by O.E.M. (SKU)
vendor: To be filled by O.E.M.
version: To be filled by O.E.M.
serial: To be filled by O.E.M.
width: 64 bits
capabilities: smbios-2.7 dmi-2.7 smp vsyscall32
configuration: boot=normal chassis=desktop family=To be filled by O.E.M. 
sku=SKU uuid=A002987B-3FB8-DC11-9897-BCEE7B5E8336
  *-core
   description: Motherboard
   product: M5A97 R2.0
   vendor: ASUSTeK COMPUTER INC.
   physical id: 0
   version: Rev 1.xx
   serial: 13101770621
   slot: To be filled by O.E.M.
 *-firmware
  description: BIOS
  vendor: American Megatrends Inc.
  physical id: 0
  version: 2006
  date: 10/01/2013
  size: 64KiB
  capacity: 8128KiB
  capabilities: pci upgrade shadowing cdboot bootselect socketedrom edd 
int13floppy1200 int13floppy720 int13floppy2880 int5printscreen int9keyboard 
int14serial int17printer acpi usb biosbootspecification uefi
 *-cpu
  description: CPU
  product: AMD FX(tm)-8320 Eight-Core Processor
  vendor: Advanced Micro Devices [AMD]
  physical id: 4
  bus info: cpu@0
  version: AMD FX(tm)-8320 Eight-Core Processor
  serial: To Be Filled By O.E.M.
  slot: Socket 942
  size: 1400MHz
  capacity: 3500MHz
  width: 64 bits
  clock: 200MHz
  capabilities: x86-64 fpu fpu_exception wp vme de pse tsc msr pae mce 
cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall 
nx mmxext fxsr_opt pdpe1gb rdtscp constant_tsc rep_good nopl nonstop_tsc 
extd_apicid aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 
sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm 
sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr 
tbm topoext perfctr_core perfctr_nb cpb hw_pstate vmmcall bmi1 arat npt lbrv 
svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter 
pfthreshold cpufreq
  configuration: cores=8 enabledcores=8 threads=8
*-cache:0
 description: L1 cache
 physical id: 5
 slot: L1-Cache
 size: 384KiB
 capacity: 384KiB
 clock: 1GHz (1.0ns)
 capabilities: pipeline-burst internal write-back unified
 configuration: level=1
*-cache:1
 description: L2 cache
 physical id: 6
 slot: L2-Cache
 size: 8MiB
 capacity: 8MiB
 clock: 1GHz (1.0ns)