Re: "No space left on device" error, but df shows plenty of space

[Eike Lantzsch]

On Thursday, 2 February 2017 10:05:58 PYST Andy Smith wrote:
> Hi Kynn,
> 
> On Wed, Feb 01, 2017 at 10:43:37AM -0500, Kynn Jones wrote:
> > On Wed, Feb 1, 2017 at 7:24 AM, Andy Smith  wrote:
> > > On Tue, Jan 31, 2017 at 06:42:39PM -0500, Kynn Jones wrote:
> > > > Unfortunately, I'll never know what the problem was.
> > > 
> > > Do you use btrfs?
> > 
> > Not that I'm aware of.  (FWIW, if I run `mount | grep -i btrfs` (as root),
> > I get no output.)
> 
> Okay, so not a btrfs issue.
> 
> > > What does "df -i" report now, after your reboot when things are
> > > working?
> > > 
> > # df -i
> > Filesystem   Inodes   IUsedIFree IUse% Mounted on
> > /dev/sda5  24264704 1464023 228006817% /
> 
> …and that doesn't show as being anywhere near full, so it most
> likely wasn't that either.
> 
> Well I suppose it could have been processes holding open deleted
> files, though it would have to have been some really big files in
> that case, as your filesystem didn't show as being anywhere near
> full.
> 
> Cheers,
> Andy

let me guess:

facts:
/var is on the same partition as /
there is plenty of space and there are free i-nodes but anyway you got error 
apt-get resulted in "no space left on device"
you rebooted and the error disappeared

=>

/ was mounted as r/o for some reason (unclean filesystem e.g.)

Re: "No space left on device" error, but df shows plenty of space

[Andy Smith]

Hi Kynn,

On Wed, Feb 01, 2017 at 10:43:37AM -0500, Kynn Jones wrote:
> On Wed, Feb 1, 2017 at 7:24 AM, Andy Smith  wrote:
> > On Tue, Jan 31, 2017 at 06:42:39PM -0500, Kynn Jones wrote:
> > > Unfortunately, I'll never know what the problem was.
> >
> > Do you use btrfs?
> 
> Not that I'm aware of.  (FWIW, if I run `mount | grep -i btrfs` (as root),
> I get no output.)

Okay, so not a btrfs issue.

> > What does "df -i" report now, after your reboot when things are
> > working?
> 
> # df -i
> Filesystem   Inodes   IUsedIFree IUse% Mounted on
> /dev/sda5  24264704 1464023 228006817% /

…and that doesn't show as being anywhere near full, so it most
likely wasn't that either.

Well I suppose it could have been processes holding open deleted
files, though it would have to have been some really big files in
that case, as your filesystem didn't show as being anywhere near
full.

Cheers,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: "No space left on device" error, but df shows plenty of space

[Kynn Jones]

On Wed, Feb 1, 2017 at 7:24 AM, Andy Smith  wrote:

> Hi Kynn,
>
> On Tue, Jan 31, 2017 at 06:42:39PM -0500, Kynn Jones wrote:
> > After the machine rebooted, I was able to run `dpkg-reconfigure ntp`
> > without error.
> >
> > Unfortunately, I'll never know what the problem was.
>
> Do you use btrfs?
>

Not that I'm aware of.  (FWIW, if I run `mount | grep -i btrfs` (as root),
I get no output.)

What does "df -i" report now, after your reboot when things are
> working?
>

# df -i
Filesystem   Inodes   IUsedIFree IUse% Mounted on
/dev/sda5  24264704 1464023 228006817% /
udev4115384 445  41149391% /dev
tmpfs   4117499 735  41167641% /run
tmpfs   4117499  14  41174851% /dev/shm
tmpfs   4117499   5  41174941% /run/lock
tmpfs   4117499  13  41174861% /sys/fs/cgroup
tmpfs   4117499  10  41174891% /run/user/1000

Unfortunately, I don't have the output of `df -i` before I rebooted.

Best,

kj

Re: "No space left on device" error, but df shows plenty of space

[Andy Smith]

Hi Kynn,

On Tue, Jan 31, 2017 at 06:42:39PM -0500, Kynn Jones wrote:
> After the machine rebooted, I was able to run `dpkg-reconfigure ntp`
> without error.
> 
> Unfortunately, I'll never know what the problem was.

Do you use btrfs?

What does "df -i" report now, after your reboot when things are
working?

Cheers,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: "No space left on device" error, but df shows plenty of space

[Kynn Jones]

Hi everyone!  Thank you for your suggestions.  They taught some new tricks.

As it happened, I had to reboot the machine for an unrelated (?) reason
(monitor would not wake from sleep).

After the machine rebooted, I was able to run `dpkg-reconfigure ntp`
without error.

Unfortunately, I'll never know what the problem was.

Thank you all for your help.

kj


On Tue, Jan 31, 2017 at 8:28 AM, Brad Rogers  wrote:

> On Tue, 31 Jan 2017 08:01:00 -0500
> Kynn Jones  wrote:
>
> Hello Kynn,
>
> >When I tried to install `ntp`, there was a "No space left on device" at
> >the end of `apt-get`'s output, but `df` shows no shortage of space:
>
> I've no idea whether this is relevant to your case, but it's possible to
> be unable to use disk space if there are insufficient inodes left to
> store the relevant directory/file info.  AIUI, a "No space..." error is
> the result.
>
> --
>  Regards  _
>  / )   "The blindingly obvious is
> / _)radnever immediately apparent"
> Just stop and take a second
> U & Ur Hand - P!nk
>

Re: "No space left on device" error, but df shows plenty of space

[Brad Rogers]

On Tue, 31 Jan 2017 08:01:00 -0500
Kynn Jones  wrote:

Hello Kynn,

>When I tried to install `ntp`, there was a "No space left on device" at
>the end of `apt-get`'s output, but `df` shows no shortage of space:

I've no idea whether this is relevant to your case, but it's possible to
be unable to use disk space if there are insufficient inodes left to
store the relevant directory/file info.  AIUI, a "No space..." error is
the result.

-- 
 Regards  _
 / )   "The blindingly obvious is
/ _)radnever immediately apparent"
Just stop and take a second
U & Ur Hand - P!nk


pgpsfbxcxn5Rs.pgp
Description: OpenPGP digital signature

Re: "No space left on device" error, but df shows plenty of space

[Andy Smith]

Hi Kynn,

On Tue, Jan 31, 2017 at 08:01:00AM -0500, Kynn Jones wrote:
> Filesystem 1K-blocks  Used Available Use% Mounted on
> /dev/sda5  381993164 206410036 156155956  57% /

[…]

> # dpkg-reconfigure ntp
> Error: No space left on device
> 
> How can I troubleshoot this problem further?

Does /dev/sda5 have a btrfs filesystem on it? Type "mount" and look
for the "type" part. If so, these problems are common with btrfs;
see btrfs wiki and/or linux-btrfs mailing list:


https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_Btrfs_claims_I.27m_out_of_space.2C_but_it_looks_like_I_should_have_lots_left.21

If not, could be things like deleted files that are still open (see
"lsof" output, look for "(deleted)". Or maybe ran out of inodes. See
"df -i" out to check that.

Cheers,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: "No space left on device" error, but df shows plenty of space

[Tony Baldwin]

On 01/31/2017 08:03 AM, Eduardo M KALINOWSKI wrote:

On Ter, 31 Jan 2017, Kynn Jones wrote:

Hi everyone!

When I tried to install `ntp`, there was a "No space left on device"
at the
end of `apt-get`'s output, but `df` shows no shortage of space:


Try 'df -i', you're probably out of inodes.


I guess I should have read your reply before sending one suggesting the 
same precise thing.


Sorry,
Tony

--
http://tonybaldwin.me
all tony, all the time

Re: "No space left on device" error, but df shows plenty of space

[Tony Baldwin]

On 01/31/2017 08:01 AM, Kynn Jones wrote:

Hi everyone!

When I tried to install `ntp`, there was a "No space left on device" at
the end of `apt-get`'s output, but `df` shows no shortage of space:

# df
Filesystem 1K-blocks  Used Available Use% Mounted on
/dev/sda5  381993164 206410036 156155956  57% /
udev   10240 0 10240   0% /dev
tmpfs6588000  9676   6578324   1% /run
tmpfs   16469996   108  16469888   1% /dev/shm
tmpfs   5120 4  5116   1% /run/lock
tmpfs   16469996 0  16469996   0% /sys/fs/cgroup
tmpfs3294000 8   3293992   1% /run/user/1000

According to `df`, the maximum percent usage is 57%.

So I tried `dpgk-reconfigure ntp` (as root), and got the same error:

# dpkg-reconfigure ntp
Error: No space left on device
update-rc.d: warning: start and stop actions are no longer
supported; falling back to defaults
Error: No space left on device

How can I troubleshoot this problem further?


try doing
$ df -i
This shows inodes.
I once had a problem where I could no longer write to a disk, because 
all of the inodes were occupied due to not properly rotating some 
log(s). Let us know if that reveals anything seemingly significant.


Tony

--
http://tonybaldwin.me
all tony, all the time

Re: "No space left on device" error, but df shows plenty of space

[Eduardo M KALINOWSKI]

On Ter, 31 Jan 2017, Kynn Jones wrote:

Hi everyone!

When I tried to install `ntp`, there was a "No space left on device" at the
end of `apt-get`'s output, but `df` shows no shortage of space:


Try 'df -i', you're probably out of inodes.
--
Eduardo M KALINOWSKI
edua...@kalinowski.com.br

"No space left on device" error, but df shows plenty of space

[Kynn Jones]

Hi everyone!

When I tried to install `ntp`, there was a "No space left on device" at the
end of `apt-get`'s output, but `df` shows no shortage of space:

# df
Filesystem 1K-blocks  Used Available Use% Mounted on
/dev/sda5  381993164 206410036 156155956  57% /
udev   10240 0 10240   0% /dev
tmpfs6588000  9676   6578324   1% /run
tmpfs   16469996   108  16469888   1% /dev/shm
tmpfs   5120 4  5116   1% /run/lock
tmpfs   16469996 0  16469996   0% /sys/fs/cgroup
tmpfs3294000 8   3293992   1% /run/user/1000

According to `df`, the maximum percent usage is 57%.

So I tried `dpgk-reconfigure ntp` (as root), and got the same error:

# dpkg-reconfigure ntp
Error: No space left on device
update-rc.d: warning: start and stop actions are no longer supported;
falling back to defaults
Error: No space left on device

How can I troubleshoot this problem further?

# uname -ar
Linux luna 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3
(2016-01-17) x86_64 GNU/Linux

Thanks in advance!

kj