Re: /etc/mtab and "df -h" problem

[Grzegorz Powiedziuk]

>
> Redhat support found a reason - a bug it glibc which was fixed just a
> couple of weeks ago
>
>
It has been fixed in glibc-2.12-1.209.el6_9.1
Thank you for your suggestions
regards
Gregory

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: /etc/mtab and "df -h" problem

[Grzegorz Powiedziuk]

Redhat support found a reason - a bug it glibc which was fixed just a
couple of weeks ago

2017-04-19 21:15 GMT-04:00 Grzegorz Powiedziuk :

> Reboot didn't resolve the issue. I will have to open a ticket with redhat.
> hopefully they can figure this out
> Gregory
>
> 2017-04-18 23:31 GMT-04:00 Grzegorz Powiedziuk :
>
>> 2017-04-18 16:34 GMT-04:00 Alan Altmark :
>>
>>> On Monday, 04/17/2017 at 03:14 GMT, Grzegorz Powiedziuk
>>>  wrote:
>>> > I am having a really weird issue on one of the z redhat systems.
>>> Probably
>>> > it doesn't have anything to do with Z but we have some great minds here
>>> so
>>> > perhaps someone will help me
>>> >
>>> > the "df -h" or "df -P"  command doesn't work when run as a normal user.
>>> > "df" on it's own works fine.
>>> >
>>> > Error:
>>> > df: cannot read table of mounted file systems: Permission denied
>>> >
>>> > Now, here is when it gets really strange. Above suggests wrong
>>> permissions
>>> > on /etc/mtab which are fine:
>>> > [root@it069qz5ora lib64]# ls -la /etc/mtab
>>> > -rw-r--r-- 1 root root 2024 Apr 17 10:44 /etc/mtab
>>> >
>>> > But the "df -h" is trying to open that file with WRITE access mode !!!
>>> > strace df -h 2>&1 |grep open  | grep mtab
>>> > ...
>>> > open("/etc/mtab", O*_RDWR*|O_CLOEXEC) = -1 EACCES (Permission
>>> denied)
>>> >
>>> > On all other systems"df -h" opens that mtab file in O_RDONLY. Why this
>>> one
>>> > is different?
>>> >
>>> > And as I said, regular "df" works fine and it usies O_RDONLY flag.
>>> >
>>> > So the problem happens only with "df -h" or "df -P" . That doesn't make
>>> any
>>> > sense to me.
>>> >
>>> > Has anyone seen anything like that?
>>>
>>> Weird.  The read_file_system_list() routine in mountlist.c (called by
>>> df.c) code shows a hard-coded "r" option in the fopen(), and I don't see
>>> df calling some other "mount list" function because of -h or -P.
>>>
>>> Alan Altmark
>>>
>>
>>
>> I know .. doesn't make sense. Thanks for looking into source code. I
>> wonder if something has change in the code in the
>> newer versions of this package. Tomorrow in the evening I will reboot
>> this guy. Hopefully this will fix it.
>>
>> Aha, forgot to say. I've changed permissions of mtab file to 666 for a
>> moment just to make sure that this is the problem and of course it worked
>> this time.
>>
>> Gregory
>>
>>
>

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: /etc/mtab and "df -h" problem

[Grzegorz Powiedziuk]

Reboot didn't resolve the issue. I will have to open a ticket with redhat.
hopefully they can figure this out
Gregory

2017-04-18 23:31 GMT-04:00 Grzegorz Powiedziuk :

> 2017-04-18 16:34 GMT-04:00 Alan Altmark :
>
>> On Monday, 04/17/2017 at 03:14 GMT, Grzegorz Powiedziuk
>>  wrote:
>> > I am having a really weird issue on one of the z redhat systems.
>> Probably
>> > it doesn't have anything to do with Z but we have some great minds here
>> so
>> > perhaps someone will help me
>> >
>> > the "df -h" or "df -P"  command doesn't work when run as a normal user.
>> > "df" on it's own works fine.
>> >
>> > Error:
>> > df: cannot read table of mounted file systems: Permission denied
>> >
>> > Now, here is when it gets really strange. Above suggests wrong
>> permissions
>> > on /etc/mtab which are fine:
>> > [root@it069qz5ora lib64]# ls -la /etc/mtab
>> > -rw-r--r-- 1 root root 2024 Apr 17 10:44 /etc/mtab
>> >
>> > But the "df -h" is trying to open that file with WRITE access mode !!!
>> > strace df -h 2>&1 |grep open  | grep mtab
>> > ...
>> > open("/etc/mtab", O*_RDWR*|O_CLOEXEC) = -1 EACCES (Permission
>> denied)
>> >
>> > On all other systems"df -h" opens that mtab file in O_RDONLY. Why this
>> one
>> > is different?
>> >
>> > And as I said, regular "df" works fine and it usies O_RDONLY flag.
>> >
>> > So the problem happens only with "df -h" or "df -P" . That doesn't make
>> any
>> > sense to me.
>> >
>> > Has anyone seen anything like that?
>>
>> Weird.  The read_file_system_list() routine in mountlist.c (called by
>> df.c) code shows a hard-coded "r" option in the fopen(), and I don't see
>> df calling some other "mount list" function because of -h or -P.
>>
>> Alan Altmark
>>
>
>
> I know .. doesn't make sense. Thanks for looking into source code. I
> wonder if something has change in the code in the
> newer versions of this package. Tomorrow in the evening I will reboot this
> guy. Hopefully this will fix it.
>
> Aha, forgot to say. I've changed permissions of mtab file to 666 for a
> moment just to make sure that this is the problem and of course it worked
> this time.
>
> Gregory
>
>

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: /etc/mtab and "df -h" problem

[Marcy Cortes]

I bet the reboot fixes it. I saw some weirdness with disks on a sles 12 server 
after kernel and more patching than magically resolved after reboot.

But you should dump it prior to reboot and send the dump off to your vendor.



Marcy

-Original Message-
From: Grzegorz Powiedziuk [gpowiedz...@gmail.com]
Sent: Tuesday, April 18, 2017 10:32 PM Central Standard Time
To: LINUX-390@VM.MARIST.EDU
Subject: Re: [LINUX-390] /etc/mtab and "df -h" problem


2017-04-18 16:34 GMT-04:00 Alan Altmark :

> On Monday, 04/17/2017 at 03:14 GMT, Grzegorz Powiedziuk
>  wrote:
> > I am having a really weird issue on one of the z redhat systems.
> Probably
> > it doesn't have anything to do with Z but we have some great minds here
> so
> > perhaps someone will help me
> >
> > the "df -h" or "df -P"  command doesn't work when run as a normal user.
> > "df" on it's own works fine.
> >
> > Error:
> > df: cannot read table of mounted file systems: Permission denied
> >
> > Now, here is when it gets really strange. Above suggests wrong
> permissions
> > on /etc/mtab which are fine:
> > [root@it069qz5ora lib64]# ls -la /etc/mtab
> > -rw-r--r-- 1 root root 2024 Apr 17 10:44 /etc/mtab
> >
> > But the "df -h" is trying to open that file with WRITE access mode !!!
> > strace df -h 2>&1 |grep open  | grep mtab
> > ...
> > open("/etc/mtab", O*_RDWR*|O_CLOEXEC) = -1 EACCES (Permission
> denied)
> >
> > On all other systems"df -h" opens that mtab file in O_RDONLY. Why this
> one
> > is different?
> >
> > And as I said, regular "df" works fine and it usies O_RDONLY flag.
> >
> > So the problem happens only with "df -h" or "df -P" . That doesn't make
> any
> > sense to me.
> >
> > Has anyone seen anything like that?
>
> Weird.  The read_file_system_list() routine in mountlist.c (called by
> df.c) code shows a hard-coded "r" option in the fopen(), and I don't see
> df calling some other "mount list" function because of -h or -P.
>
> Alan Altmark
>


I know .. doesn't make sense. Thanks for looking into source code. I wonder
if something has change in the code in the
newer versions of this package. Tomorrow in the evening I will reboot this
guy. Hopefully this will fix it.

Aha, forgot to say. I've changed permissions of mtab file to 666 for a
moment just to make sure that this is the problem and of course it worked
this time.

Gregory

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: /etc/mtab and "df -h" problem

[Grzegorz Powiedziuk]

2017-04-18 16:34 GMT-04:00 Alan Altmark :

> On Monday, 04/17/2017 at 03:14 GMT, Grzegorz Powiedziuk
>  wrote:
> > I am having a really weird issue on one of the z redhat systems.
> Probably
> > it doesn't have anything to do with Z but we have some great minds here
> so
> > perhaps someone will help me
> >
> > the "df -h" or "df -P"  command doesn't work when run as a normal user.
> > "df" on it's own works fine.
> >
> > Error:
> > df: cannot read table of mounted file systems: Permission denied
> >
> > Now, here is when it gets really strange. Above suggests wrong
> permissions
> > on /etc/mtab which are fine:
> > [root@it069qz5ora lib64]# ls -la /etc/mtab
> > -rw-r--r-- 1 root root 2024 Apr 17 10:44 /etc/mtab
> >
> > But the "df -h" is trying to open that file with WRITE access mode !!!
> > strace df -h 2>&1 |grep open  | grep mtab
> > ...
> > open("/etc/mtab", O*_RDWR*|O_CLOEXEC) = -1 EACCES (Permission
> denied)
> >
> > On all other systems"df -h" opens that mtab file in O_RDONLY. Why this
> one
> > is different?
> >
> > And as I said, regular "df" works fine and it usies O_RDONLY flag.
> >
> > So the problem happens only with "df -h" or "df -P" . That doesn't make
> any
> > sense to me.
> >
> > Has anyone seen anything like that?
>
> Weird.  The read_file_system_list() routine in mountlist.c (called by
> df.c) code shows a hard-coded "r" option in the fopen(), and I don't see
> df calling some other "mount list" function because of -h or -P.
>
> Alan Altmark
>


I know .. doesn't make sense. Thanks for looking into source code. I wonder
if something has change in the code in the
newer versions of this package. Tomorrow in the evening I will reboot this
guy. Hopefully this will fix it.

Aha, forgot to say. I've changed permissions of mtab file to 666 for a
moment just to make sure that this is the problem and of course it worked
this time.

Gregory

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: /etc/mtab and "df -h" problem

[Grzegorz Powiedziuk]

2017-04-18 17:13 GMT-04:00 Scott Rohling :

> You mention nfs - so -   does 'df -hl'  also present the error? The 'l'
> limits it to local filesystems so seems an easy check to see if it's
> related to nfs..
>
> Scott Rohling
>
>
Good idea but unfortunately same thing happens. "df" works but "df -hl"
complains about permissions. Strace shows that it wants to open mtab in
O_RDWR
I have scheduled reboot of this server for tomorrow. I will let you know if
that fixes this problem ...

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: /etc/mtab and "df -h" problem

[Scott Rohling]

You mention nfs - so -   does 'df -hl'  also present the error? The 'l'
limits it to local filesystems so seems an easy check to see if it's
related to nfs..

Scott Rohling

On Mon, Apr 17, 2017 at 9:16 AM, Grzegorz Powiedziuk 
wrote:

> Thanks Scott
> Nothing unusual ext4  + nfs
> But, I've just noticed that system was updated from redhat satelite server
> few days ago. And it definitely requires a reboot (satellite claims so)
> Nonetheless other updated systems (there were a bunch of those), does not
> have this issue  .
>
> Gregory
>
> 2017-04-17 12:04 GMT-04:00 Scott Rohling :
>
> > A quick google shows similar reported (on ubuntu but just grasping here)
> >  -- had to do with fuse mounted filesystems and gvfs (and perhaps
> lightdm)
> > ...
> >
> > What kind of filesystems do you have available ​and could one of them be
> > the reason things behave differently.. ?   Is there  a filesystem here
> that
> > isn't present on your other servers..  ?
> >
> > Why the -h option would matter I'm really not sure - to me that's just an
> > adjustment of how sizes are shown.. so you'd think you'd get the error
> > regardless..   Like I said - grasping..
> >
> > Scott Rohling
> >
> > On Mon, Apr 17, 2017 at 8:12 AM, Grzegorz Powiedziuk <
> > gpowiedz...@gmail.com>
> > wrote:
> >
> > > I am having a really weird issue on one of the z redhat systems.
> Probably
> > > it doesn't have anything to do with Z but we have some great minds here
> > so
> > > perhaps someone will help me
> > >
> > > the "df -h" or "df -P"  command doesn't work when run as a normal user.
> > > "df" on it's own works fine.
> > >
> > > Error:
> > > df: cannot read table of mounted file systems: Permission denied
> > >
> > > Now, here is when it gets really strange. Above suggests wrong
> > permissions
> > > on /etc/mtab which are fine:
> > > [root@it069qz5ora lib64]# ls -la /etc/mtab
> > > -rw-r--r-- 1 root root 2024 Apr 17 10:44 /etc/mtab
> > >
> > > But the "df -h" is trying to open that file with WRITE access mode !!!
> > > strace df -h 2>&1 |grep open  | grep mtab
> > > ...
> > > open("/etc/mtab", O*_RDWR*|O_CLOEXEC) = -1 EACCES (Permission
> denied)
> > >
> > > On all other systems"df -h" opens that mtab file in O_RDONLY. Why this
> > one
> > > is different?
> > >
> > > And as I said, regular "df" works fine and it usies O_RDONLY flag.
> > >
> > > So the problem happens only with "df -h" or "df -P" . That doesn't make
> > any
> > > sense to me.
> > >
> > > Has anyone seen anything like that?
> > >
> > > thanks!
> > > Gregory
> > >
> > > --
> > > For LINUX-390 subscribe / signoff / archive access instructions,
> > > send email to lists...@vm.marist.edu with the message: INFO LINUX-390
> or
> > > visit
> > > http://www.marist.edu/htbin/wlvindex?LINUX-390
> > > --
> > > For more information on Linux on System z, visit
> > > http://wiki.linuxvm.org/
> > >
> >
> > --
> > For LINUX-390 subscribe / signoff / archive access instructions,
> > send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
> > visit
> > http://www.marist.edu/htbin/wlvindex?LINUX-390
> > --
> > For more information on Linux on System z, visit
> > http://wiki.linuxvm.org/
> >
>
> --
> For LINUX-390 subscribe / signoff / archive access instructions,
> send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
> visit
> http://www.marist.edu/htbin/wlvindex?LINUX-390
> --
> For more information on Linux on System z, visit
> http://wiki.linuxvm.org/
>

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: /etc/mtab and "df -h" problem

[Alan Altmark]

On Monday, 04/17/2017 at 03:14 GMT, Grzegorz Powiedziuk 
 wrote:
> I am having a really weird issue on one of the z redhat systems. 
Probably
> it doesn't have anything to do with Z but we have some great minds here 
so
> perhaps someone will help me
>
> the "df -h" or "df -P"  command doesn't work when run as a normal user.
> "df" on it's own works fine.
>
> Error:
> df: cannot read table of mounted file systems: Permission denied
>
> Now, here is when it gets really strange. Above suggests wrong 
permissions
> on /etc/mtab which are fine:
> [root@it069qz5ora lib64]# ls -la /etc/mtab
> -rw-r--r-- 1 root root 2024 Apr 17 10:44 /etc/mtab
>
> But the "df -h" is trying to open that file with WRITE access mode !!!
> strace df -h 2>&1 |grep open  | grep mtab
> ...
> open("/etc/mtab", O*_RDWR*|O_CLOEXEC) = -1 EACCES (Permission 
denied)
>
> On all other systems"df -h" opens that mtab file in O_RDONLY. Why this 
one
> is different?
>
> And as I said, regular "df" works fine and it usies O_RDONLY flag.
>
> So the problem happens only with "df -h" or "df -P" . That doesn't make 
any
> sense to me.
>
> Has anyone seen anything like that?

Weird.  The read_file_system_list() routine in mountlist.c (called by 
df.c) code shows a hard-coded "r" option in the fopen(), and I don't see 
df calling some other "mount list" function because of -h or -P.

Alan Altmark

Senior Managing z/VM and Linux Consultant
Lab Services System z Delivery Practice
IBM Systems & Technology Group
ibm.com/systems/services/labservices
office: 607.429.3323
mobile; 607.321.7556
alan_altm...@us.ibm.com
IBM Endicott

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: /etc/mtab and "df -h" problem

[Grzegorz Powiedziuk]

Thanks Scott
Nothing unusual ext4  + nfs
But, I've just noticed that system was updated from redhat satelite server
few days ago. And it definitely requires a reboot (satellite claims so)
Nonetheless other updated systems (there were a bunch of those), does not
have this issue  .

Gregory

2017-04-17 12:04 GMT-04:00 Scott Rohling :

> A quick google shows similar reported (on ubuntu but just grasping here)
>  -- had to do with fuse mounted filesystems and gvfs (and perhaps lightdm)
> ...
>
> What kind of filesystems do you have available ​and could one of them be
> the reason things behave differently.. ?   Is there  a filesystem here that
> isn't present on your other servers..  ?
>
> Why the -h option would matter I'm really not sure - to me that's just an
> adjustment of how sizes are shown.. so you'd think you'd get the error
> regardless..   Like I said - grasping..
>
> Scott Rohling
>
> On Mon, Apr 17, 2017 at 8:12 AM, Grzegorz Powiedziuk <
> gpowiedz...@gmail.com>
> wrote:
>
> > I am having a really weird issue on one of the z redhat systems. Probably
> > it doesn't have anything to do with Z but we have some great minds here
> so
> > perhaps someone will help me
> >
> > the "df -h" or "df -P"  command doesn't work when run as a normal user.
> > "df" on it's own works fine.
> >
> > Error:
> > df: cannot read table of mounted file systems: Permission denied
> >
> > Now, here is when it gets really strange. Above suggests wrong
> permissions
> > on /etc/mtab which are fine:
> > [root@it069qz5ora lib64]# ls -la /etc/mtab
> > -rw-r--r-- 1 root root 2024 Apr 17 10:44 /etc/mtab
> >
> > But the "df -h" is trying to open that file with WRITE access mode !!!
> > strace df -h 2>&1 |grep open  | grep mtab
> > ...
> > open("/etc/mtab", O*_RDWR*|O_CLOEXEC) = -1 EACCES (Permission denied)
> >
> > On all other systems"df -h" opens that mtab file in O_RDONLY. Why this
> one
> > is different?
> >
> > And as I said, regular "df" works fine and it usies O_RDONLY flag.
> >
> > So the problem happens only with "df -h" or "df -P" . That doesn't make
> any
> > sense to me.
> >
> > Has anyone seen anything like that?
> >
> > thanks!
> > Gregory
> >
> > --
> > For LINUX-390 subscribe / signoff / archive access instructions,
> > send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
> > visit
> > http://www.marist.edu/htbin/wlvindex?LINUX-390
> > --
> > For more information on Linux on System z, visit
> > http://wiki.linuxvm.org/
> >
>
> --
> For LINUX-390 subscribe / signoff / archive access instructions,
> send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
> visit
> http://www.marist.edu/htbin/wlvindex?LINUX-390
> --
> For more information on Linux on System z, visit
> http://wiki.linuxvm.org/
>

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: /etc/mtab and "df -h" problem

[Scott Rohling]

A quick google shows similar reported (on ubuntu but just grasping here)
 -- had to do with fuse mounted filesystems and gvfs (and perhaps lightdm)
...

What kind of filesystems do you have available ​and could one of them be
the reason things behave differently.. ?   Is there  a filesystem here that
isn't present on your other servers..  ?

Why the -h option would matter I'm really not sure - to me that's just an
adjustment of how sizes are shown.. so you'd think you'd get the error
regardless..   Like I said - grasping..

Scott Rohling

On Mon, Apr 17, 2017 at 8:12 AM, Grzegorz Powiedziuk 
wrote:

> I am having a really weird issue on one of the z redhat systems. Probably
> it doesn't have anything to do with Z but we have some great minds here so
> perhaps someone will help me
>
> the "df -h" or "df -P"  command doesn't work when run as a normal user.
> "df" on it's own works fine.
>
> Error:
> df: cannot read table of mounted file systems: Permission denied
>
> Now, here is when it gets really strange. Above suggests wrong permissions
> on /etc/mtab which are fine:
> [root@it069qz5ora lib64]# ls -la /etc/mtab
> -rw-r--r-- 1 root root 2024 Apr 17 10:44 /etc/mtab
>
> But the "df -h" is trying to open that file with WRITE access mode !!!
> strace df -h 2>&1 |grep open  | grep mtab
> ...
> open("/etc/mtab", O*_RDWR*|O_CLOEXEC) = -1 EACCES (Permission denied)
>
> On all other systems"df -h" opens that mtab file in O_RDONLY. Why this one
> is different?
>
> And as I said, regular "df" works fine and it usies O_RDONLY flag.
>
> So the problem happens only with "df -h" or "df -P" . That doesn't make any
> sense to me.
>
> Has anyone seen anything like that?
>
> thanks!
> Gregory
>
> --
> For LINUX-390 subscribe / signoff / archive access instructions,
> send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
> visit
> http://www.marist.edu/htbin/wlvindex?LINUX-390
> --
> For more information on Linux on System z, visit
> http://wiki.linuxvm.org/
>

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/