Re: OpenBSD support for xattr on file systems other than UFS ?

[J Doe]

On 2023-05-15 18:55, J Doe wrote:


On 2023-05-12, J Doe  wrote:

Hello,

I was configuring Samba on my OpenBSD 7.2 server and wanted to support
iOS/iPad OS and macOS clients.

The documentation for Samba states that the following vfs options are
required to support these clients:

  /etc/samba/smb.conf
  . . .
  vfs = catia fruit streams_xattr

... however, my server is using UFS2 (the default), which I am aware
does not support extended attributes.


OpenBSD doesn't support xattr at all.


Would it be possible to get around that by mounting an ext2 disk image
file on OpenBSD via: vnconfig and: mount and pointing: smb.conf to it ?


No.

You may be able to do something with
https://wiki.samba.org/index.php/Using_the_xattr_tdb_VFS_Module


Hi Stuart,

Thanks for your reply.  Ah, interesting!  I had not spotted that vfs 
module on the Samba site.


It appears to warn against using it in production, but I will be 
providing service to 2-3 clients at the most, so I will give it a try 
and report back on anything interesting I ran into.


Hi Stuart and list,

Just thought I'd post an update about this.  With the following: vfs stanza:

/etc/samba/smb.conf
[global]
vfs objects = catia fruit streams_xattr xattr_tdb

xattr_tdb:file = /var/samba/macosxattr.tdb

... and then the suggested: fruit config statements on the Samba 
wiki[1], I am having a better result.


While the Samba wiki does warn about using: xattr_tdb in production, I 
am servicing between 3-5 clients (iOS and macOS), not all of which are 
at the same time and it appears to be good.


Thanks,

-J

[1] 
https://wiki.samba.org/index.php/Configure_Samba_to_Work_Better_with_Mac_OS_X

Re: OpenBSD support for xattr on file systems other than UFS ?

[J Doe]

On 2023-05-12 03:22, Stuart Henderson wrote:


On 2023-05-12, J Doe  wrote:

Hello,

I was configuring Samba on my OpenBSD 7.2 server and wanted to support
iOS/iPad OS and macOS clients.

The documentation for Samba states that the following vfs options are
required to support these clients:

  /etc/samba/smb.conf
  . . .
  vfs = catia fruit streams_xattr

... however, my server is using UFS2 (the default), which I am aware
does not support extended attributes.


OpenBSD doesn't support xattr at all.


Would it be possible to get around that by mounting an ext2 disk image
file on OpenBSD via: vnconfig and: mount and pointing: smb.conf to it ?


No.

You may be able to do something with
https://wiki.samba.org/index.php/Using_the_xattr_tdb_VFS_Module


Hi Stuart,

Thanks for your reply.  Ah, interesting!  I had not spotted that vfs 
module on the Samba site.


It appears to warn against using it in production, but I will be 
providing service to 2-3 clients at the most, so I will give it a try 
and report back on anything interesting I ran into.


- J

Re: OpenBSD support for xattr on file systems other than UFS ?

[J Doe]

On 2023-05-15 17:16, Lyndon Nerenberg (VE7TFX/VE6BBM) wrote:

Marcus MERIGHI writes:


 vfs = catia fruit streams_xattr


I run a Samba server that does not have these options set - but
successfully serves iOS/macOS clients.


You need those extra attributes if you want to use your Samba
share for TimeMachine backups.

--lyndon


Hi Lyndon,

Thank you for your reply.  I figured as much for TimeMachine, but I seem 
to be running into issues saving from basic apps (ex: TextEdit), and 
dragging and dropping files to a Samba share without the: fruit vfs module.


This particular installation I won't be using TimeMachine.

- J

Re: OpenBSD support for xattr on file systems other than UFS ?

[J Doe]

On 2023-05-12 03:24, Marcus MERIGHI wrote:


Hello,

gene...@nativemethods.com (J Doe), 2023.05.12 (Fri) 04:47 (CEST):

I was configuring Samba on my OpenBSD 7.2 server and wanted to support
iOS/iPad OS and macOS clients.

The documentation for Samba states that the following vfs options are
required to support these clients:

 /etc/samba/smb.conf
 . . .
 vfs = catia fruit streams_xattr


I run a Samba server that does not have these options set - but
successfully serves iOS/macOS clients.

Apart from that, smb.conf(5) does not have the parameter "vfs", only
"vfs object"/"vfs objects" (which are aliases).

Marcus


Hi Marcus,

Yes, you are correct.  That line should have been:

/etc/samba/smb.conf
[global]
vfs objects = catia fruit streams_xattr
. . .

In regards to running Samba without the: fruit vfs module for Apple 
clients, what version of macOS/iOS/iPad OS are your clients running ?  I 
have found that running a current version of macOS (Ventura 13.3.1 (a)) 
and using the TextEdit app to save to the Samba share does *not* work in 
an install *without* fruit (I tested a Samba config with: fruit on a 
Ubuntu VM for comparison)


- J

Re: OpenBSD support for xattr on file systems other than UFS ?

[Lyndon Nerenberg (VE7TFX/VE6BBM)]

Marcus MERIGHI writes:

> > vfs = catia fruit streams_xattr
>
> I run a Samba server that does not have these options set - but
> successfully serves iOS/macOS clients.

You need those extra attributes if you want to use your Samba
share for TimeMachine backups.

--lyndon

Re: OpenBSD support for xattr on file systems other than UFS ?

[Marcus MERIGHI]

Hello, 

gene...@nativemethods.com (J Doe), 2023.05.12 (Fri) 04:47 (CEST):
> I was configuring Samba on my OpenBSD 7.2 server and wanted to support
> iOS/iPad OS and macOS clients.
> 
> The documentation for Samba states that the following vfs options are
> required to support these clients:
> 
> /etc/samba/smb.conf
> . . .
> vfs = catia fruit streams_xattr

I run a Samba server that does not have these options set - but
successfully serves iOS/macOS clients.

Apart from that, smb.conf(5) does not have the parameter "vfs", only
"vfs object"/"vfs objects" (which are aliases).

Marcus

Re: OpenBSD support for xattr on file systems other than UFS ?

[Stuart Henderson]

On 2023-05-12, J Doe  wrote:
> Hello,
>
> I was configuring Samba on my OpenBSD 7.2 server and wanted to support 
> iOS/iPad OS and macOS clients.
>
> The documentation for Samba states that the following vfs options are 
> required to support these clients:
>
>  /etc/samba/smb.conf
>  . . .
>  vfs = catia fruit streams_xattr
>
> ... however, my server is using UFS2 (the default), which I am aware 
> does not support extended attributes.

OpenBSD doesn't support xattr at all.

> Would it be possible to get around that by mounting an ext2 disk image 
> file on OpenBSD via: vnconfig and: mount and pointing: smb.conf to it ?

No.

You may be able to do something with
https://wiki.samba.org/index.php/Using_the_xattr_tdb_VFS_Module

OpenBSD support for xattr on file systems other than UFS ?

[J Doe]

Hello,

I was configuring Samba on my OpenBSD 7.2 server and wanted to support 
iOS/iPad OS and macOS clients.


The documentation for Samba states that the following vfs options are 
required to support these clients:


/etc/samba/smb.conf
. . .
vfs = catia fruit streams_xattr

... however, my server is using UFS2 (the default), which I am aware 
does not support extended attributes.


Would it be possible to get around that by mounting an ext2 disk image 
file on OpenBSD via: vnconfig and: mount and pointing: smb.conf to it ?


Thanks,

-J

Re: OpenBSD File systems , on Flash / SSD CPE (in sites with uncontrolled power (CPE customer sites)

[dirk coetzee]

We had a similar challenge. My advice would be to leave all the filesystems as 
is - and add the 'sync' flag. 

It slows down disk access - but makes the filesystem resilient to power 
outages. See sync option - https://man.openbsd.org/mount


   On Tuesday, 29 November 2022 at 09:03:28 am AWST, Yoshihiro Kawamata 
 wrote:  
 
 Hi,

In the case of CPE, I do not think that programs will be added or
removed after the start of operation, so I think that the following
partition can be mounted as read-only to reduce the impact of power
failure.

/bin
/sbin
/usr
/usr/local (with wxallowed)

For the rest, you can just mount them with mfs in conjunction with
persistent storage.

Yoshihiro Kawamata
https://fuguita.org/

  

Re: OpenBSD File systems , on Flash / SSD CPE (in sites with uncontrolled power (CPE customer sites)

[Yoshihiro Kawamata]

Hi,

In the case of CPE, I do not think that programs will be added or
removed after the start of operation, so I think that the following
partition can be mounted as read-only to reduce the impact of power
failure.

/bin
/sbin
/usr
/usr/local (with wxallowed)

For the rest, you can just mount them with mfs in conjunction with
persistent storage.

Yoshihiro Kawamata
https://fuguita.org/

Re: OpenBSD File systems , on Flash / SSD CPE (in sites with uncontrolled power (CPE customer sites)

[Brian Conway]

On Mon, Nov 28, 2022, at 4:06 PM, Tom Smyth wrote:
> /dev/sd0a / ffs rw,softdep,noatime 1 1
> /dev/sd0d /usr/local ffs rw,wxallowed,nodev,softdep,noatime 1 1

softdep is a useful option for metadata-heavy workloads, but it is not a 
magical go-fast flag. While it's possible that characterization is true for 
your /usr/local mount, I would guess it is probably not for /.

Brian

Re: OpenBSD File systems , on Flash / SSD CPE (in sites with uncontrolled power (CPE customer sites)

[Tom Smyth]

sorry  there was an omission in my /etc/fstab
i had left out the softdep,noatime flags on the filessytems that were
funning off the disk using FFS
Thanks
#begin corrected /etc/fstab##
/dev/sd0a / ffs rw,softdep,noatime 1 1
/dev/sd0d /usr/local ffs rw,wxallowed,nodev,softdep,noatime 1 1
swap /tmp mfs rw,nosuid,noexec,nodev,-s=256000,-P=/persist-fs/tmp 0 0
swap /var mfs rw,nosuid,noexec,nodev,-s=512000,-P=/persist-fs/var 0 0
swap /dev mfs rw,nosuid,noexec,-P=/persist-fs/dev,-i=2048,-s=102400 0 0

##end-corrected /etc/fstab##


On Mon, 28 Nov 2022 at 21:46, Tom Smyth 
wrote:

> Hello, Folks,
>
> Im reviewing our filesystem setup for OpenBSD CPEs that we deploy in the
> field
>
> in order to minimise the impact of Power Outages / Customer interference
> on the boxes,
> we install a 4G root partition /
> and a 2GB /usr/local (to allow the wxallowed flag for the filesystem)
>
> we use mfs for /tmp and /var  so that there the probability that there is
> a filessytem write to the SSD is reduced (so that power failures dont cause
> file system corruption)
>
> we use the following  fstabl
>
> #begin /etc/fstab/###
> /dev/sd0a / ffs rw 1 1
> /dev/sd0d /usr/local ffs rw,wxallowed,nodev 1 1
> swap /tmp mfs rw,nosuid,noexec,nodev,-s=256000,-P=/persist-fs/tmp 0 0
> swap /var mfs rw,nosuid,noexec,nodev,-s=512000,-P=/persist-fs/var 0 0
> swap /dev mfs rw,nosuid,noexec,-P=/persist-fs/dev,-i=2048,-s=102400 0 0
> #end  /etc/fstab/###
>
> and the persist-fs folders are created by installing OpenBSD, installing
> packages and running
> the following commands to copy /var /tmp and /dev to a persistent location
> on /
> ###setup commands #
> mkdir -p /persist-fs/dev
> mkdir -p /persist-fs/tmp
> mkdir -p /persist-fs/var
> cp -Rp /var/* /persist-fs/var
> cp -Rp /tmp/* /persist-fs/tmp
> cp -p /dev/MAKEDEV /persist-fs/dev/
> cd /persist-fs/dev/
> /persist-fs/dev/MAKEDEV all
>
> any feedback welcome, are there other folders that could be heavily
> written to ?
> is there shortcommings  I have ommited swap (because of flash and ssd wear
> concerns)
> I hope this helps...
> Tom Smyth
>
>
> --
> Kindest regards,
> Tom Smyth.
>


-- 
Kindest regards,
Tom Smyth.

OpenBSD File systems , on Flash / SSD CPE (in sites with uncontrolled power (CPE customer sites)

[Tom Smyth]

Hello, Folks,

Im reviewing our filesystem setup for OpenBSD CPEs that we deploy in the
field

in order to minimise the impact of Power Outages / Customer interference on
the boxes,
we install a 4G root partition /
and a 2GB /usr/local (to allow the wxallowed flag for the filesystem)

we use mfs for /tmp and /var  so that there the probability that there is a
filessytem write to the SSD is reduced (so that power failures dont cause
file system corruption)

we use the following  fstabl

#begin /etc/fstab/###
/dev/sd0a / ffs rw 1 1
/dev/sd0d /usr/local ffs rw,wxallowed,nodev 1 1
swap /tmp mfs rw,nosuid,noexec,nodev,-s=256000,-P=/persist-fs/tmp 0 0
swap /var mfs rw,nosuid,noexec,nodev,-s=512000,-P=/persist-fs/var 0 0
swap /dev mfs rw,nosuid,noexec,-P=/persist-fs/dev,-i=2048,-s=102400 0 0
#end  /etc/fstab/###

and the persist-fs folders are created by installing OpenBSD, installing
packages and running
the following commands to copy /var /tmp and /dev to a persistent location
on /
###setup commands #
mkdir -p /persist-fs/dev
mkdir -p /persist-fs/tmp
mkdir -p /persist-fs/var
cp -Rp /var/* /persist-fs/var
cp -Rp /tmp/* /persist-fs/tmp
cp -p /dev/MAKEDEV /persist-fs/dev/
cd /persist-fs/dev/
/persist-fs/dev/MAKEDEV all

any feedback welcome, are there other folders that could be heavily written
to ?
is there shortcommings  I have ommited swap (because of flash and ssd wear
concerns)
I hope this helps...
Tom Smyth


-- 
Kindest regards,
Tom Smyth.

Re: Awaiting a diff [was: Re: File systems...] Probably not gonna happen anyway

[Stuart Longland]

On 13/2/20 5:17 am, jeanfrancois wrote:
> Good evening,
> 
> Very good videos are available from one of the developer of EXT2/3/4
> recommended to see.
> 
> https://www.youtube.com/watch?v=2mYDFr5T4tY
> 
> OpenBSD's FFS code looks awesome.

It's mature, and not worth chucking out anytime soon as it'll be much
more stable than any effort to port ${FANCYFS} will be.

About the only big complaint I've heard about it is that there's no
journaling which slows down boot times after an unclean shut-down
(particularly for larger volumes).  This does concern me, but not
greatly at this point.

It's on my rather large back-log to look at, some time in the future
unless someone beats me to it.  (Contrary to others' research, pet
Python projects is not my sole software development experience.)

As it happens there's two ways I can scratch my itch (management of
OpenBSD disk partitions):

1. get OpenBSD to run on a FS that the tools I have¹ understand
   (side-benefit: OpenBSD gains support for a journalled FS)
2. get the tools I have to understand OpenBSD disklabels + ffs
   (side-benefit: people would be able to re-arrange² partitions)

As this thread already struck a few raw nerves last time, I would
suggest if there's any interest, we can collectively discuss it off-list.
-- 
Stuart Longland (aka Redhatter, VK4MSL)

I haven't lost my mind...
  ...it's backed up on a tape somewhere.

¹. Mainly what I miss is a tool for re-arranging partitions.  gparted
has served me well for this purpose.
². Primarily the goal here being that a user can "move" partitions
around to re-organise free space.  Right now one can "grow" a partition,
but shuffling the partitions around is not easily possible without
daring unsupported and dangerous acts using `dd`, `disklabel` and `growfs`.

Re: Awaiting a diff [was: Re: File systems...] Probably not gonna happen anyway

[jeanfrancois]

Good evening,

Very good videos are available from one of the developer of EXT2/3/4
recommended to see.

https://www.youtube.com/watch?v=2mYDFr5T4tY

OpenBSD's FFS code looks awesome.


Jean-François


Le 09/01/2020 à 03:25, Theo de Raadt a écrit :

Xiyue Deng  wrote:


It would be better to point out where to start, what
hard problems to solve, what work has been done in this area that people
can continue to work on.

Looking at that list, noone here owes you any of those.

Do your own homework.

Re-reading the thread is remarkable.  It's a bunch of people who
won't do the work telling us that we need to tell them what work
to do.  A bunch of garbage is coming out of your mouths.

Re: Awaiting a diff [was: Re: File systems...]

[Marc Espie]

On Fri, Jan 10, 2020 at 11:28:07AM +0100, Stefan Sperling wrote:
> On Fri, Jan 10, 2020 at 12:52:44PM +0300, Consus wrote:
> > On 20:06 Thu 09 Jan, Marc Espie wrote:
> > > It's been that way for ages. But no-one volunteered
> > > to work on this.
> > 
> > Anyone even knows about this? Aside from OpenBSD developers (who have
> > their plates full already) how an average person can find out that there
> > is rusty piece of code that should be taken care of?
> 
> Don't start looking at other people's problems before you can help yourself.
> Rewriting automounter for Theo as a first contribution is not likely to work.
> Find something small that annoys *you* and get involved by fixing that first.
> Learn how to approach and debug issues that affect you directly.


Ditto.

There are lots and lots of pieces in OpenBSD that could use some "obvious"
improvements, and not enough developers to do it all these days.

Ask anyone with an OpenBSD account how he got it, you'll get the same
story: use the OS, get annoyed by a small thing, start sending patches.

Very soon, you get to be responsible for some stuff.

No magic, just use the OS and scratch the itches.

Re: Awaiting a diff [was: Re: File systems...]

[Consus]

On 11:28 Fri 10 Jan, Stefan Sperling wrote:
> On Fri, Jan 10, 2020 at 12:52:44PM +0300, Consus wrote:
> > On 20:06 Thu 09 Jan, Marc Espie wrote:
> > > It's been that way for ages. But no-one volunteered
> > > to work on this.
> > 
> > Anyone even knows about this? Aside from OpenBSD developers (who have
> > their plates full already) how an average person can find out that there
> > is rusty piece of code that should be taken care of?
> 
> Don't start looking at other people's problems before you can help yourself.
> Rewriting automounter for Theo as a first contribution is not likely to work.
> Find something small that annoys *you* and get involved by fixing that first.
> Learn how to approach and debug issues that affect you directly.
> 
> How did I get into OpenBSD wifi?
> Because my OpenBSD access point stopped accepting new associations after
> one about week of usage. It turned out this was one of those problems
> which had been known for years and all this time people had been switching
> to non-OpenBSD APs to work around it. I had lived with the problem for
> about a year or two, resetting the wifi interface on the AP whenever it
> happened.
> 
> Then I noticed that 'ifconfig ath0 scan' showed a very long list of known
> MAC addresses whenever the problem occurred. So I looked for a way trigger
> the problem faster than in one week and succeeded by running this loop
> on the client which made the problem appear after a few minutes:
> 
>   ifconfig iwn0 up
>   while sleep 5; do ifconfig iwn0 lladdr random; done
> 
> Looking at the code I found that known MACs never expired! Once the AP had
> reached its limit of learned MACs it accepted no new associations because
> no room was made for new MAC addresses. Fix was a 3-line diff, which I
> could verify with my above test.

Neat. Though I'm using OpenBSD only for a router and since issues
preventing me from using it on a desktop (or NAS) are pretty huge to be
my "good first issues" that's not an option, sadly. So only donations /
occasional bug reports from me :D

Re: Awaiting a diff [was: Re: File systems...]

[Stefan Sperling]

On Fri, Jan 10, 2020 at 12:52:44PM +0300, Consus wrote:
> On 20:06 Thu 09 Jan, Marc Espie wrote:
> > It's been that way for ages. But no-one volunteered
> > to work on this.
> 
> Anyone even knows about this? Aside from OpenBSD developers (who have
> their plates full already) how an average person can find out that there
> is rusty piece of code that should be taken care of?

Don't start looking at other people's problems before you can help yourself.
Rewriting automounter for Theo as a first contribution is not likely to work.
Find something small that annoys *you* and get involved by fixing that first.
Learn how to approach and debug issues that affect you directly.

How did I get into OpenBSD wifi?
Because my OpenBSD access point stopped accepting new associations after
one about week of usage. It turned out this was one of those problems
which had been known for years and all this time people had been switching
to non-OpenBSD APs to work around it. I had lived with the problem for
about a year or two, resetting the wifi interface on the AP whenever it
happened.

Then I noticed that 'ifconfig ath0 scan' showed a very long list of known
MAC addresses whenever the problem occurred. So I looked for a way trigger
the problem faster than in one week and succeeded by running this loop
on the client which made the problem appear after a few minutes:

  ifconfig iwn0 up
  while sleep 5; do ifconfig iwn0 lladdr random; done

Looking at the code I found that known MACs never expired! Once the AP had
reached its limit of learned MACs it accepted no new associations because
no room was made for new MAC addresses. Fix was a 3-line diff, which I
could verify with my above test.

---
commit 6bbde753957f0b27c56cfdd15c9af836579d120b
from: stsp 
date: Wed Jan 18 14:35:34 2012 UTC
 
 Make it possible to free cached nodes which never associated (e.g. nodes
 only scanning for networks). These were never put into COLLECT state and
 were thus never evicted from the node cache in hostap mode.
 ok jsg@
 
diff ae44467745d21c295e8ffe38340d662269578502 
d62cb122bc6abf011c13400ea5c3f90c56292854
blob - 893de460bfd2a1509205c4e5d837ba6f8d5c2636
blob + 9435a287862a29a9dc79ea09ce8328b8e054c819
--- sys/net80211/ieee80211_node.c
+++ sys/net80211/ieee80211_node.c
@@ -1507,8 +1507,10 @@ ieee80211_node_leave(struct ieee80211com *ic, struct i
 * If node wasn't previously associated all we need to do is
 * reclaim the reference.
 */
-   if (ni->ni_associd == 0)
+   if (ni->ni_associd == 0) {
+   ieee80211_node_newstate(ni, IEEE80211_STA_COLLECT);
return;
+   }
 
if (ni->ni_pwrsave == IEEE80211_PS_DOZE)
ic->ic_pssta--;

Re: Awaiting a diff [was: Re: File systems...]

[Consus]

On 11:08 Fri 10 Jan, Janne Johansson wrote:
> By using the parts that OpenBSD is made up of, and not automatically moving
> to other OSes as soon as you leave the comfort zone.

I'm not sure, but it seems like from a user perspective there is nothing
wrong with amd(8). Only that it keeps using legacy code.

Re: Awaiting a diff [was: Re: File systems...]

[Janne Johansson]

Den fre 10 jan. 2020 kl 10:55 skrev Consus :

> On 20:06 Thu 09 Jan, Marc Espie wrote:
> > It's been that way for ages. But no-one volunteered
> > to work on this.
>
> Anyone even knows about this? Aside from OpenBSD developers (who have
> their plates full already) how an average person can find out that there
> is rusty piece of code that should be taken care of?
>

By using the parts that OpenBSD is made up of, and not automatically moving
to other OSes as soon as you leave the comfort zone.
Guess that is how many ports gets added. $prg exist for $other_os but not
OpenBSD, someone does the work to make it run on OpenBSD and there you go.

-- 
May the most significant bit of your life be positive.

Re: Awaiting a diff [was: Re: File systems...]

[Consus]

On 20:06 Thu 09 Jan, Marc Espie wrote:
> It's been that way for ages. But no-one volunteered
> to work on this.

Anyone even knows about this? Aside from OpenBSD developers (who have
their plates full already) how an average person can find out that there
is rusty piece of code that should be taken care of?

Re: Awaiting a diff [was: Re: File systems...]

[Marc Espie]

If you want a useful project related to filesystems,
try the automounter.

Yes, that ancient code.

Look very closely. It has tendrils in NFSv2.

And some people, most prominently Theo, use amd(8).

Write an automounter that does not depend on NFSv2,
and then, most probably we can kill NFSv2.

Small steps...

It's been that way for ages. But no-one volunteered
to work on this.

Re: Awaiting a diff [was: Re: File systems...]

[Özgür Kazancci]

On 09/01/2020 05:15, Xiyue Deng wrote:


Some guy asks whether there's any plan to improve file system
performance, the answer given is the code is right there if you want to
contribute.  Then some other guy offers a proposal to start working on
it, and the answer now becomes you are hardly qualified for such kind 
of

work.

Sorry but such kinds of answers are not helpful, and gives the
(potentially wrong) impression that OpenBSD doesn't welcome
contributions.  It would be better to point out where to start, what
hard problems to solve, what work has been done in this area that 
people

can continue to work on.


^^Totally agree on each and every word here. (And yes, noone cares)
*Unsighs

Re: Awaiting a diff [was: Re: File systems...]

[Marc Espie]

On Thu, Jan 09, 2020 at 09:07:38AM +1000, Stuart Longland wrote:
> On 9/1/20 12:56 am, Ian Darwin wrote:
> >> - If we could clean-room implement a BSD-licensed
> >> EXT3/EXT4/BTRFS/XFS/JFS/whatever, following style(8), would there be
> >> interest in supporting that in OpenBSD?
> > 
> > And which "we" are you referring to here? Did you mean yourself,
> > or are you hoping that "somebody" will do it?
> 
> I'm hoping it will be more than one person assisting in this, and yes, I
> include myself in that group.

One useful thing you could do is review the diffs that are routinely
sent to various OpenBSD mailing-lists.

For instance, I have sent a few make-related diffs recently, on
which I'm still awaiting reviews.

How is this related to filesystem work, you may ask ?

Well, it's very related, though indirectly.

See, there aren't that many people actually doing the work
in OpenBSD.  A lot of it is routine work, but vital for
the project.

Queue Ingo and mandoc.
Queue me and make... and various other things.

and lots of other developers at time. Better lld support.
continuing the work started on ctf, etc, etc.


If there were more people actually *helping* instead of talking shit,
that stuff would go forward, and maybe, just maybe, us and other
developers *could* allocate time to do other stuff.

That might (or might not) include filesystem work.

Having the whole infrastructure working better would definitely help.

But no, it's way more fun to just sit there and say "hey, I only want
to do the sexy stuff. Please pre-chew it for me to baby-vomit stage so
I can be smug about it".

Re: Awaiting a diff [was: Re: File systems...]

[Philipp Buehler]

Am 09.01.2020 16:10 schrieb Ingo Schwarze:

https://www.youtube.com/watch?v=HTD9Gow1wTU


And Bob gave a talk about VFS hacking the very same
event. Might be an eye-opener of those "proposing to help".
https://www.youtube.com/watch?v=rVb8jdlP4gE
(somehow the slides didn't made it to /papers/?)



Cool, i wasn't even aware of thib@'s talk back then.  That was the
very first year i ever took part in a hackathon, and it wasn't that


And I wasn't aware the 3-line-diff joke is at least that old.. hmm :)

Anyway.. at c2k3 (or was it 2004?) I was looking
into porting linux ciss(4) driver to OpenBSD naively.
As a more or less young gun back then: "all driver code is there, just
some... interfaces and be done!". Nope.

Well, you cannot hack storage/disk drivers without some VFS
knowledge.. mickey@ (bless him!!), art@ and niklas@ walked me
through someof that but hell.. what did I knew.
Long story short: even with help and lotsa beer I ended up with
empty hands. (eventually mickey did the "port" 1-2y later (by manpage 
3.8)


So for the "aah, cant be that hard" crowd: it IS a bloody messy
place (even IF you rip NFS out of it). I'd say hacking in this
area (arena..) requires years of experience to produce something
that can go into the tree.

In other news.. ah no, cut it.

HTH,
--
pb

Re: Awaiting a diff [was: Re: File systems...]

[Ingo Schwarze]

Hi Janne,

Janne Johansson wrote on Thu, Jan 09, 2020 at 09:49:43AM +0100:
> Den tors 9 jan. 2020 kl 02:11 skrev Ingo Schwarze :

>> Are you aware that even Bob Beck@ is seriously scared of some
>> parts of our file system code, and of touching some parts of it?
>> Yes, this Bob Beck, who isn't really all that easily scared:
>>
>>   https://www.youtube.com/watch?v=GnBbhXBDmwU
>>
>> One of our most senior developers, regularly and continuously
>> contributing since 1997, and among those who understand our
>> file system code best.

> And here I thought you would post thib@s talk literally named
> "Things that makes Bob scream" from the f2k9/Slackathon conf:
> 
> https://www.youtube.com/watch?v=HTD9Gow1wTU

Cool, i wasn't even aware of thib@'s talk back then.  That was the
very first year i ever took part in a hackathon, and it wasn't that
one.  But yeah, that talk certainly relates to the point i was
trying to make.  Not my area of work really, but as far as i
understand, some things mentioned in the talk have changed a lot,
while others didn't at all, and Bob certainly still has plenty of
opportunity for screaming.  Loudly.  Even though that talk was
certainly hilarious, file systems still aren't a laughing matter.

Thanks for the link,
  Ingo

Re: Awaiting a diff [was: Re: File systems...]

[Stefan Sperling]

On Thu, Jan 09, 2020 at 12:47:31PM +0300, Consus wrote:
> Relax, it was a joke.

Whatever, what I wrote wasn't just directed at you.

misc@ sucks a lot lately.

Re: Awaiting a diff [was: Re: File systems...]

[zap]

On 01/08/2020 09:25 PM, Theo de Raadt wrote:
> Xiyue Deng  wrote:
>
>> It would be better to point out where to start, what
>> hard problems to solve, what work has been done in this area that people
>> can continue to work on.
> Looking at that list, noone here owes you any of those.
>
> Do your own homework.
>
> Re-reading the thread is remarkable.  It's a bunch of people who
> won't do the work telling us that we need to tell them what work
> to do.  A bunch of garbage is coming out of your mouths.
>
The average user probably knows nothing about the hard work it takes to
make an OS.  Don't get me wrong, I disagree with BSD on some stances,
but I am not the one who has to do the hard work.  So there's that...

Re: Awaiting a diff [was: Re: File systems...]

[Consus]

On 10:45 Thu 09 Jan, Stefan Sperling wrote:
> On Thu, Jan 09, 2020 at 11:02:17AM +0300, Consus wrote:
> > On 18:15 Wed 08 Jan, Xiyue Deng wrote:
> > > It would be better to point out where to start, what hard problems to
> > > solve, what work has been done in this area that people can continue
> > > to work on.
> > 
> > They don't remember as there is no bugtracker.
> 
> When people report actual bugs, they get fixed so fast that tracking
> them over days or weeks would be pointless.
> 
> """
> We thank Theo de Raadt and the OpenBSD developers for their incredibly
> quick response: they published patches for these vulnerabilities less
> than 40 hours after our initial contact. 
> """ -- Qualys
> 
> Any non-critical stuff which gets reported ("my printer/wifi/usb doesn't 
> work",
> "kernel does not boot on my new laptop", "my system works but is slow") just
> lands in an endless pile of problems that someone might eventually decide to
> work on if they run into it again. Such issues happen over and over again,
> all the time. Any developer picking them up ASAP over and over would burn out,
> just like they do in companies with issue-tracker driven workflows. That's why
> people demanding loudly to get such issues fixed are told to shut up.
> 
> And keep in mind that at any given moment there are only about 50 to 100
> people doing actual work here. The rest of the world is busy elsewhere or
> slacking.

Relax, it was a joke.

Re: Awaiting a diff [was: Re: File systems...]

[Stefan Sperling]

On Thu, Jan 09, 2020 at 11:02:17AM +0300, Consus wrote:
> On 18:15 Wed 08 Jan, Xiyue Deng wrote:
> > It would be better to point out where to start, what hard problems to
> > solve, what work has been done in this area that people can continue
> > to work on.
> 
> They don't remember as there is no bugtracker.

When people report actual bugs, they get fixed so fast that tracking
them over days or weeks would be pointless.

"""
We thank Theo de Raadt and the OpenBSD developers for their incredibly
quick response: they published patches for these vulnerabilities less
than 40 hours after our initial contact. 
""" -- Qualys

Any non-critical stuff which gets reported ("my printer/wifi/usb doesn't work",
"kernel does not boot on my new laptop", "my system works but is slow") just
lands in an endless pile of problems that someone might eventually decide to
work on if they run into it again. Such issues happen over and over again,
all the time. Any developer picking them up ASAP over and over would burn out,
just like they do in companies with issue-tracker driven workflows. That's why
people demanding loudly to get such issues fixed are told to shut up.

And keep in mind that at any given moment there are only about 50 to 100
people doing actual work here. The rest of the world is busy elsewhere or
slacking.

Re: Awaiting a diff [was: Re: File systems...]

[Janne Johansson]

Den tors 9 jan. 2020 kl 02:11 skrev Ingo Schwarze :

>
> Are you aware that even Bob Beck@ is seriously scared of some
> parts of our file system code, and of touching some parts of it?
> Yes, this Bob Beck, who isn't really all that easily scared:
>
>   https://www.youtube.com/watch?v=GnBbhXBDmwU
>
> One of our most senior developers, regularly and continuously
> contributing since 1997, and among those who understand our
> file system code best.
>

And here I thought you would post thib@s talk literally named
"Things that makes Bob scream" from the f2k9/Slackathon conf:

https://www.youtube.com/watch?v=HTD9Gow1wTU

-- 
May the most significant bit of your life be positive.

Re: Awaiting a diff [was: Re: File systems...]

[Consus]

On 18:15 Wed 08 Jan, Xiyue Deng wrote:
> It would be better to point out where to start, what hard problems to
> solve, what work has been done in this area that people can continue
> to work on.

They don't remember as there is no bugtracker.

Re: Awaiting a diff [was: Re: File systems...]

[Xiyue Deng]

gwes  writes:

> Suggestion: to improve file system performance,
> first document the bad behavior in detail.
>
> Begin with examples of traces/logs of disk accesses associated
> with file system operations.
>
> Include scenarios (one hopes reproducible ones) to provoke
> bad behavior.
>
> Are reads worse than writes? Sequential vs. random?
> Interleaved r/w on one file? On more than one file simultaneously?
>
> Examples from other O/S which are better or worse?
>
> Without this very detailed data it's all noise.
>
> Being able to get good traces & correlate them with OS activity
> shows at least some competence dealing with OS internals.
>
> geoff steckel

Thanks Geoff!  I was looking for this kind of guide so that I can do
something not noisy.


signature.asc
Description: PGP signature

Re: Awaiting a diff [was: Re: File systems...]

[gwes]

Suggestion: to improve file system performance,
first document the bad behavior in detail.

Begin with examples of traces/logs of disk accesses associated
with file system operations.

Include scenarios (one hopes reproducible ones) to provoke
bad behavior.

Are reads worse than writes? Sequential vs. random?
Interleaved r/w on one file? On more than one file simultaneously?

Examples from other O/S which are better or worse?

Without this very detailed data it's all noise.

Being able to get good traces & correlate them with OS activity
shows at least some competence dealing with OS internals.

geoff steckel

Re: Awaiting a diff [was: Re: File systems...]

[Xiyue Deng]

"Theo de Raadt"  writes:

> Xiyue Deng  wrote:
>
>> Ingo Schwarze  writes:
>> 
>> > Hi Stuart,
>> >
>> > Stuart Longland wrote on Thu, Jan 09, 2020 at 09:07:38AM +1000:
>> >> Somebody wrote:
>> >
>> >>> - If we could clean-room implement a BSD-licensed
>> >>> EXT3/EXT4/BTRFS/XFS/JFS/whatever, following style(8), would there be
>> >>> interest in supporting that in OpenBSD?
>> >
>> >> I'm hoping it will be more than one person assisting in this,
>> >> and yes, I include myself in that group.
>> >
>> > schwarze@cvs $ grep -Fi longland /cvs/CVSROOT/ChangeLog*   
>> > 
>> > schwarze@cvs $
>> >
>> > And https://stuartl.longlandclan.id.au/ lists a single free software
>> > project, about 190 commits of Python code, with one single contributor.
>> >
>> >
>> > I'm sorry that i have to use somewhat strong wording here, i'm
>> > generally trying to help making our lists as friendly as possible,
>> > but in this case, a clear answer is really required.
>> >
>> > There is few code that is as difficult as a file system.
>> > There is few code that is as closely entangled with the hardest
>> > parts of there kernel like file system code.
>> > There is few code where touching it is as dangerous as touching
>> > file system code.
>> > There are few areas of the system where people get as upset
>> > when you break it as with file systems.  You literally make people
>> > lose their personal data, and when they realize something went wrong,
>> > it's usually too late, the data is usually already gone for good.
>> >
>> > You are certainly welcome to contribute if you want to: start with
>> > sending samll bugfix patches.  Progress to small feature additions
>> > or small cleanup patches in areas that are not too dangerous.
>> > Then grow.  Anything beyond that is impossible to predict.
>> >
>> > For a newbie, there is really no point in dreaming about
>> > implementing or changing file systems.
>> >
>> > You need to learn what you are capable of and then convince others
>> > of your abilities *by getting good patches committed*.  Idle talk
>> > announcing bizarre dreams doesn't really help anyone.
>> >
>> > Are you aware that even Bob Beck@ is seriously scared of some
>> > parts of our file system code, and of touching some parts of it?
>> > Yes, this Bob Beck, who isn't really all that easily scared:
>> >
>> >   https://www.youtube.com/watch?v=GnBbhXBDmwU
>> >
>> > One of our most senior developers, regularly and continuously
>> > contributing since 1997, and among those who understand our
>> > file system code best.  Most recently, he was among the main
>> > driving forces behind unveil(2).
>> >
>> >
>> > Becoming able to approximately judge the difficulty and size of
>> > tasks relative to your own abilities is extremely important when
>> > you want to contribute to free software.
>> >
>> > Even if you had, let's say, a whole year to spend full-time, you
>> > would not really be making any sense right now.  So, could we drop
>> > this thread, please?
>> >
>> > Yours,
>> >   Ingo
>> 
>> Some guy asks whether there's any plan to improve file system
>> performance, the answer given is the code is right there if you want to
>> contribute.
>
> We (the project developers) did not provide that answer.  One of you
> typical "posers" suggested it.
>
>> Then some other guy offers a proposal to start working on
>> it,
>
> Wow.  He did not offer to start anything like that.  Maybe he'll create
> a wiki, or write some words on a mailing list?
>
>> and the answer now becomes you are hardly qualified for such kind of
>> work.
>
> I suspect you are also unqualified.

I am, which is why I didn't propose anything yet.

>
>> Sorry but such kinds of answers are not helpful, and gives the
>> (potentially wrong) impression that OpenBSD doesn't welcome
>> contributions.  It would be better to point out where to start, what
>> hard problems to solve, what work has been done in this area that people
>> can continue to work on.
>
> Cut the crap.  Those of you posting in this thread are only capable of
> writing words of text to a mailing list.

I was hoping that Stuart's email was the a new start aside from that
troll thread.  But well.


signature.asc
Description: PGP signature

Re: Awaiting a diff [was: Re: File systems...]

[Stuart Longland]

On 9/1/20 12:20 pm, Theo de Raadt wrote:
>> and the answer now becomes you are hardly qualified for such kind of
>> work.
> I suspect you are also unqualified.
> 

You don't become qualified by writing words on a mailing list… and while
I acknowledge a lack of experience in the area, I do understand the
risks involved and I am willing to give it a try.  ffs is not going
anywhere any time soon.

Contrary to recently-expressed opinion, I have done kernel-level and
bare-metal coding before.  OpenBSD isn't the only OS kernel in existence.

Those interested in helping out: contact me off list, there is little to
be gained by discussing it here.
-- 
Stuart Longland (aka Redhatter, VK4MSL)

I haven't lost my mind...
  ...it's backed up on a tape somewhere.

Re: Awaiting a diff [was: Re: File systems...]

[Theo de Raadt]

Xiyue Deng  wrote:

> It would be better to point out where to start, what
> hard problems to solve, what work has been done in this area that people
> can continue to work on.

Looking at that list, noone here owes you any of those.

Do your own homework.

Re-reading the thread is remarkable.  It's a bunch of people who
won't do the work telling us that we need to tell them what work
to do.  A bunch of garbage is coming out of your mouths.

Re: Awaiting a diff [was: Re: File systems...]

[Theo de Raadt]

Xiyue Deng  wrote:

> Ingo Schwarze  writes:
> 
> > Hi Stuart,
> >
> > Stuart Longland wrote on Thu, Jan 09, 2020 at 09:07:38AM +1000:
> >> Somebody wrote:
> >
> >>> - If we could clean-room implement a BSD-licensed
> >>> EXT3/EXT4/BTRFS/XFS/JFS/whatever, following style(8), would there be
> >>> interest in supporting that in OpenBSD?
> >
> >> I'm hoping it will be more than one person assisting in this,
> >> and yes, I include myself in that group.
> >
> > schwarze@cvs $ grep -Fi longland /cvs/CVSROOT/ChangeLog*
> >
> > schwarze@cvs $
> >
> > And https://stuartl.longlandclan.id.au/ lists a single free software
> > project, about 190 commits of Python code, with one single contributor.
> >
> >
> > I'm sorry that i have to use somewhat strong wording here, i'm
> > generally trying to help making our lists as friendly as possible,
> > but in this case, a clear answer is really required.
> >
> > There is few code that is as difficult as a file system.
> > There is few code that is as closely entangled with the hardest
> > parts of there kernel like file system code.
> > There is few code where touching it is as dangerous as touching
> > file system code.
> > There are few areas of the system where people get as upset
> > when you break it as with file systems.  You literally make people
> > lose their personal data, and when they realize something went wrong,
> > it's usually too late, the data is usually already gone for good.
> >
> > You are certainly welcome to contribute if you want to: start with
> > sending samll bugfix patches.  Progress to small feature additions
> > or small cleanup patches in areas that are not too dangerous.
> > Then grow.  Anything beyond that is impossible to predict.
> >
> > For a newbie, there is really no point in dreaming about
> > implementing or changing file systems.
> >
> > You need to learn what you are capable of and then convince others
> > of your abilities *by getting good patches committed*.  Idle talk
> > announcing bizarre dreams doesn't really help anyone.
> >
> > Are you aware that even Bob Beck@ is seriously scared of some
> > parts of our file system code, and of touching some parts of it?
> > Yes, this Bob Beck, who isn't really all that easily scared:
> >
> >   https://www.youtube.com/watch?v=GnBbhXBDmwU
> >
> > One of our most senior developers, regularly and continuously
> > contributing since 1997, and among those who understand our
> > file system code best.  Most recently, he was among the main
> > driving forces behind unveil(2).
> >
> >
> > Becoming able to approximately judge the difficulty and size of
> > tasks relative to your own abilities is extremely important when
> > you want to contribute to free software.
> >
> > Even if you had, let's say, a whole year to spend full-time, you
> > would not really be making any sense right now.  So, could we drop
> > this thread, please?
> >
> > Yours,
> >   Ingo
> 
> Some guy asks whether there's any plan to improve file system
> performance, the answer given is the code is right there if you want to
> contribute.

We (the project developers) did not provide that answer.  One of you
typical "posers" suggested it.

> Then some other guy offers a proposal to start working on
> it,

Wow.  He did not offer to start anything like that.  Maybe he'll create
a wiki, or write some words on a mailing list?

> and the answer now becomes you are hardly qualified for such kind of
> work.

I suspect you are also unqualified.

> Sorry but such kinds of answers are not helpful, and gives the
> (potentially wrong) impression that OpenBSD doesn't welcome
> contributions.  It would be better to point out where to start, what
> hard problems to solve, what work has been done in this area that people
> can continue to work on.

Cut the crap.  Those of you posting in this thread are only capable of
writing words of text to a mailing list.

Re: Awaiting a diff [was: Re: File systems...]

[Xiyue Deng]

Ingo Schwarze  writes:

> Hi Stuart,
>
> Stuart Longland wrote on Thu, Jan 09, 2020 at 09:07:38AM +1000:
>> Somebody wrote:
>
>>> - If we could clean-room implement a BSD-licensed
>>> EXT3/EXT4/BTRFS/XFS/JFS/whatever, following style(8), would there be
>>> interest in supporting that in OpenBSD?
>
>> I'm hoping it will be more than one person assisting in this,
>> and yes, I include myself in that group.
>
> schwarze@cvs $ grep -Fi longland /cvs/CVSROOT/ChangeLog*  
>  
> schwarze@cvs $
>
> And https://stuartl.longlandclan.id.au/ lists a single free software
> project, about 190 commits of Python code, with one single contributor.
>
>
> I'm sorry that i have to use somewhat strong wording here, i'm
> generally trying to help making our lists as friendly as possible,
> but in this case, a clear answer is really required.
>
> There is few code that is as difficult as a file system.
> There is few code that is as closely entangled with the hardest
> parts of there kernel like file system code.
> There is few code where touching it is as dangerous as touching
> file system code.
> There are few areas of the system where people get as upset
> when you break it as with file systems.  You literally make people
> lose their personal data, and when they realize something went wrong,
> it's usually too late, the data is usually already gone for good.
>
> You are certainly welcome to contribute if you want to: start with
> sending samll bugfix patches.  Progress to small feature additions
> or small cleanup patches in areas that are not too dangerous.
> Then grow.  Anything beyond that is impossible to predict.
>
> For a newbie, there is really no point in dreaming about
> implementing or changing file systems.
>
> You need to learn what you are capable of and then convince others
> of your abilities *by getting good patches committed*.  Idle talk
> announcing bizarre dreams doesn't really help anyone.
>
> Are you aware that even Bob Beck@ is seriously scared of some
> parts of our file system code, and of touching some parts of it?
> Yes, this Bob Beck, who isn't really all that easily scared:
>
>   https://www.youtube.com/watch?v=GnBbhXBDmwU
>
> One of our most senior developers, regularly and continuously
> contributing since 1997, and among those who understand our
> file system code best.  Most recently, he was among the main
> driving forces behind unveil(2).
>
>
> Becoming able to approximately judge the difficulty and size of
> tasks relative to your own abilities is extremely important when
> you want to contribute to free software.
>
> Even if you had, let's say, a whole year to spend full-time, you
> would not really be making any sense right now.  So, could we drop
> this thread, please?
>
> Yours,
>   Ingo

Some guy asks whether there's any plan to improve file system
performance, the answer given is the code is right there if you want to
contribute.  Then some other guy offers a proposal to start working on
it, and the answer now becomes you are hardly qualified for such kind of
work.

Sorry but such kinds of answers are not helpful, and gives the
(potentially wrong) impression that OpenBSD doesn't welcome
contributions.  It would be better to point out where to start, what
hard problems to solve, what work has been done in this area that people
can continue to work on.


signature.asc
Description: PGP signature

Re: Awaiting a diff [was: Re: File systems...]

[Theo de Raadt]

Ingo Schwarze  wrote:

> Even if you had, let's say, a whole year to spend full-time, you
> would not really be making any sense right now.  So, could we drop
> this thread, please?

Ingo, you know that's impossible.

These are people on misc, their self-importance and optimism knows
no bounds.

Re: Awaiting a diff [was: Re: File systems...]

[Ingo Schwarze]

Hi Stuart,

Stuart Longland wrote on Thu, Jan 09, 2020 at 09:07:38AM +1000:
> Somebody wrote:

>> - If we could clean-room implement a BSD-licensed
>> EXT3/EXT4/BTRFS/XFS/JFS/whatever, following style(8), would there be
>> interest in supporting that in OpenBSD?

> I'm hoping it will be more than one person assisting in this,
> and yes, I include myself in that group.

schwarze@cvs $ grep -Fi longland /cvs/CVSROOT/ChangeLog*   
schwarze@cvs $

And https://stuartl.longlandclan.id.au/ lists a single free software
project, about 190 commits of Python code, with one single contributor.


I'm sorry that i have to use somewhat strong wording here, i'm
generally trying to help making our lists as friendly as possible,
but in this case, a clear answer is really required.

There is few code that is as difficult as a file system.
There is few code that is as closely entangled with the hardest
parts of there kernel like file system code.
There is few code where touching it is as dangerous as touching
file system code.
There are few areas of the system where people get as upset
when you break it as with file systems.  You literally make people
lose their personal data, and when they realize something went wrong,
it's usually too late, the data is usually already gone for good.

You are certainly welcome to contribute if you want to: start with
sending samll bugfix patches.  Progress to small feature additions
or small cleanup patches in areas that are not too dangerous.
Then grow.  Anything beyond that is impossible to predict.

For a newbie, there is really no point in dreaming about
implementing or changing file systems.

You need to learn what you are capable of and then convince others
of your abilities *by getting good patches committed*.  Idle talk
announcing bizarre dreams doesn't really help anyone.

Are you aware that even Bob Beck@ is seriously scared of some
parts of our file system code, and of touching some parts of it?
Yes, this Bob Beck, who isn't really all that easily scared:

  https://www.youtube.com/watch?v=GnBbhXBDmwU

One of our most senior developers, regularly and continuously
contributing since 1997, and among those who understand our
file system code best.  Most recently, he was among the main
driving forces behind unveil(2).


Becoming able to approximately judge the difficulty and size of
tasks relative to your own abilities is extremely important when
you want to contribute to free software.

Even if you had, let's say, a whole year to spend full-time, you
would not really be making any sense right now.  So, could we drop
this thread, please?

Yours,
  Ingo

Re: Awaiting a diff [was: Re: File systems...]

[Stuart Longland]

On 9/1/20 12:56 am, Ian Darwin wrote:
>> - If we could clean-room implement a BSD-licensed
>> EXT3/EXT4/BTRFS/XFS/JFS/whatever, following style(8), would there be
>> interest in supporting that in OpenBSD?
> 
> And which "we" are you referring to here? Did you mean yourself,
> or are you hoping that "somebody" will do it?

I'm hoping it will be more than one person assisting in this, and yes, I
include myself in that group.

Can't commit to doing anything right away, but it'll be slotted
somewhere in the back-log.

>…

>> ZFS and BTRFS are much newer, and more complicated with software RAID
>> functionality built in.  I think these would be harder to implement from
>> scratch.
> 
> Persuade the owners to release under an ISC license. Then send a diff.
> 

Yeah, I think there's been discussions about changing the license (to
GPL for Linux kernel use) and those came to a dead end.  I don't see the
copyright holders being receptive to ISC either.

-- 
Stuart Longland (aka Redhatter, VK4MSL)

I haven't lost my mind...
  ...it's backed up on a tape somewhere.

Re: File systems [was Re: OpenBSD's extremely poor network/disk performance?]

[Tom Smyth]

Hi Karel,

Thanks, for the correction...

I thought zfs was bigger than that ;)
Thanks


On Wednesday, 8 January 2020, Karel Gardas  wrote:

>
>
> On 1/8/20 12:44 PM, Tom Smyth wrote:
>
>> As far as im aware there are 2 concerns about ZFS,
>> 1) its license  is not BSD /ISC  you can use it and make money and not be
>> sued,
>> but it is more restrictive than BSD / ISC
>>
>
> Yes, CDDL seems to be a no go based on past CDDL discussion which is
> available for example in Star & OpenBSD thread on @tech:
>
> https://marc.info/?l=openbsd-tech=110806948606417=2
>
>> 2) then there is the Number of Lines of code, which I believe is far
>> longer than
>> the OpenBSD code base,  who and what team would manage the
>> introduction of that code
>> and the risks that come with that  large a code base.
>>
>
> Need to correct you a bit:
>
> ZFS: ~110k lines
> XFS: ~95k lines
> Ext4: ~38k lines
>
> while OpenBSD src/sys alone:
> ~3.7mil lines where majority is in dev. But if I subtract drm code which
> is probably the biggest contribution in dev (~1.7 mil lines), then I still
> get roughly 2 mil lines of code in sys -- which is just part of base.
>
> LInes counted by sloccount.
>
>

-- 
Kindest regards,
Tom Smyth.

Re: File systems [was Re: OpenBSD's extremely poor network/disk performance?]

[Karel Gardas]

On 1/8/20 12:44 PM, Tom Smyth wrote:

As far as im aware there are 2 concerns about ZFS,
1) its license  is not BSD /ISC  you can use it and make money and not be sued,
but it is more restrictive than BSD / ISC


Yes, CDDL seems to be a no go based on past CDDL discussion which is 
available for example in Star & OpenBSD thread on @tech:


https://marc.info/?l=openbsd-tech=110806948606417=2

2) then there is the Number of Lines of code, which I believe is far longer than
the OpenBSD code base,  who and what team would manage the
introduction of that code
and the risks that come with that  large a code base.


Need to correct you a bit:

ZFS: ~110k lines
XFS: ~95k lines
Ext4: ~38k lines

while OpenBSD src/sys alone:
~3.7mil lines where majority is in dev. But if I subtract drm code which 
is probably the biggest contribution in dev (~1.7 mil lines), then I 
still get roughly 2 mil lines of code in sys -- which is just part of base.


LInes counted by sloccount.

Awaiting a diff [was: Re: File systems...]

[Ian Darwin]

> - If we could clean-room implement a BSD-licensed
> EXT3/EXT4/BTRFS/XFS/JFS/whatever, following style(8), would there be
> interest in supporting that in OpenBSD?

And which "we" are you referring to here? Did you mean yourself,
or are you hoping that "somebody" will do it?

> There's merit in the third option, OpenBSD already supports EXT2 (which
> is also 90's vintage like ffs) as there are some platforms (e.g.
> loongson) that require it.
>...
> EXT4 is also very widespread and stable, and seems to offer decent
> performance.

So send a diff that upgrades the code to ext3 and 4.

> ZFS and BTRFS are much newer, and more complicated with software RAID
> functionality built in.  I think these would be harder to implement from
> scratch.

Persuade the owners to release under an ISC license. Then send a diff.

Re: File systems [was Re: OpenBSD's extremely poor network/disk performance?]

[Tom Smyth]

Howdy Stuart,

On Wed, 8 Jan 2020 at 11:17, Stuart Longland  wrote:
>
> On 8/1/20 1:25 am, Karel Gardas wrote:
> > And yes, ffs performance sucks, but nor me nor you provide any diff to
> > change that so we can just shut up and use what's available.
>
> Okay, question is if not ffs, then what?
>
> - Other BSDs have ZFS… is it viable to port that to OpenBSD?  (Maybe
> it's been done before?  I didn't check.)

As far as im aware there are 2 concerns about ZFS,
1) its license  is not BSD /ISC  you can use it and make money and not be sued,
but it is more restrictive than BSD / ISC
2) then there is the Number of Lines of code, which I believe is far longer than
the OpenBSD code base,  who and what team would manage the
introduction of that code
and the risks that come with that  large a code base.

> - FreeBSD has UFS2, DragonFlyBSD has HAMMER…  Could we borrow their code?
> - If we could clean-room implement a BSD-licensed

as a user I would say sweet...  but someone more knowledgable /
involved in the project would
need to see a diff before a determination can be made.


> EXT3/EXT4/BTRFS/XFS/JFS/whatever, following style(8), would there be
> interest in supporting that in OpenBSD?
what is the story with the license? if the license is not ISC / BSD I
dont think it would be in base..

as a user I would say more filessytems sweet...  but someone more
knowledgeable / involved in the project would
need to see a diff before a determination can be made.

> - Or do we implement yet another file system?  (Seems like too much work
> for not much gain IMO.)

as an OpenBSD user I would say that the performance of Network
is dependent on your hardware. / the specific hardware Driver
compatibility /capability in OpenBSD,
I have had a different performance experience depending on the
hardware I was using, and the
maturity of the Driver support for that hardware.

I have found the em(4) supported nics are pretty good ix(4) has solid
performance
vmx(4) have been good but it is dependent on the Vmware version you are using ,
and then others like vio(4) interfaces I have not had as good a
performance. but that
is more due to the age of the drivers and their capability vs what
newer virtio drivers can do.

But as a number of members of the OpenBSD Project have said to me
Diffs are welcome ...   Good Diffs will be considered
just bear in mind the the License Requirements and Coding Style  KNF
when submitting a diff and do it off current...

>
> Regards,
> --
> Stuart Longland (aka Redhatter, VK4MSL)
>
> I haven't lost my mind...
>   ...it's backed up on a tape somewhere.
>


-- 
Kindest regards,
Tom Smyth.

File systems [was Re: OpenBSD's extremely poor network/disk performance?]

[Stuart Longland]

On 8/1/20 1:25 am, Karel Gardas wrote:
> And yes, ffs performance sucks, but nor me nor you provide any diff to
> change that so we can just shut up and use what's available.

Okay, question is if not ffs, then what?

- Other BSDs have ZFS… is it viable to port that to OpenBSD?  (Maybe
it's been done before?  I didn't check.)
- FreeBSD has UFS2, DragonFlyBSD has HAMMER…  Could we borrow their code?
- If we could clean-room implement a BSD-licensed
EXT3/EXT4/BTRFS/XFS/JFS/whatever, following style(8), would there be
interest in supporting that in OpenBSD?
- Or do we implement yet another file system?  (Seems like too much work
for not much gain IMO.)

There's merit in the third option, OpenBSD already supports EXT2 (which
is also 90's vintage like ffs) as there are some platforms (e.g.
loongson) that require it.  I run BTRFS on a lot of my Linux machines,
and aside from some features that are still experimental (quotas being
one such issue), it seems to do the job.  I've also been a big XFS user
in the past.

Performance seems good and XFS in particular has seen widespread
production use, particularly in high-performance computing arenas.  (SGI
didn't exactly do things small!)

EXT4 is also very widespread and stable, and seems to offer decent
performance.

ZFS and BTRFS are much newer, and more complicated with software RAID
functionality built in.  I think these would be harder to implement from
scratch.

DIY file systems doesn't seem like a good plan for success… it'll be a
lot of work, won't be compatible with anything else, and could be as bad
if not worse than what we have now, whilst also being untested.  ffs is
at least mature and stable!

Are any of the "modern" file systems (from a design perspective,
licensing is a different matter) suitable for use as OpenBSD's root fs?
 What would be needed?

Regards,
-- 
Stuart Longland (aka Redhatter, VK4MSL)

I haven't lost my mind...
  ...it's backed up on a tape somewhere.

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

[Steve Williams]

On 11/12/2017 12:27 PM, Philip Guenther wrote:
On Mon, Dec 11, 2017 at 9:16 AM, Otto Moerbeek > wrote:


On Mon, Dec 11, 2017 at 08:30:54AM -0700, Steve Williams wrote:
> cpio has always been my "go to" for file system duplication
because it will
> re-create device nodes.

Both pax and tar do that as well.

Come on, you still remember using tar back in the 90's when it didn't 
support devices, paths were 100 bytes _total_, and they didn't include 
user/group names (only UID/GID), right?  Good times!


Philip

Yes, my habits were born of SCO Xenix, IBM's AIX for the RT PC, etc.   
Old habits die hard!!!  lol


Cheers,
Steve W.

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

[Philip Guenther]

On Mon, Dec 11, 2017 at 9:16 AM, Otto Moerbeek  wrote:

> On Mon, Dec 11, 2017 at 08:30:54AM -0700, Steve Williams wrote:
> > cpio has always been my "go to" for file system duplication because it
> will
> > re-create device nodes.
>
> Both pax and tar do that as well.
>

Come on, you still remember using tar back in the 90's when it didn't
support devices, paths were 100 bytes _total_, and they didn't include
user/group names (only UID/GID), right?  Good times!

Philip

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

[Otto Moerbeek]

On Mon, Dec 11, 2017 at 08:30:54AM -0700, Steve Williams wrote:

> Hi,
> 
> cpio has always been my "go to" for file system duplication because it will
> re-create device nodes.

Both pax and tar do that as well.

-Otto

> 
> Cheers,
> Steve Williams
> 
> 
> On 10/12/2017 11:03 AM, webmas...@bennettconstruction.us wrote:
> > Forgive problems with this email.
> > I saw how my emails showed up on marc.info
> > Scary. This is just temporary.
> > 
> > OK. I've tried to use both methods and just don't
> > get true duplication.
> > 
> > tar
> > It can't work with file and directory names
> > that are OK in filesystem, but too long for itself.
> > Quite a while back I lost a lot of unimportant files
> > and directories that had absolute paths too long.
> > Why is this happening with tar? Can this be fixed?
> > If not, I'd like to add a note about that to the FAQ.
> > 
> > dump
> > I had to move /usr/local to a bigger partition. growfs,
> > etc. I kept the /usr/local untouched and then dumped it
> > to the new partition, expecting a true duplication.
> > Nope.
> > It changed all of the program symlinks permissions.
> > Why is dump doing this? Can this be fixed?
> > Otherwise, a note about this should be added to the FAQ
> > also.
> > 
> > Question:
> > Can dd be used to do what I did with dump or tar?
> > Smaller partition copied to a bigger partition.
> > 
> > I'm willing to try and help out, but I'm going through
> > both laptop and server hell at the moment.
> > 
> > Thanks,
> > Chris Bennett

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

[Steve Williams]

Hi,

cpio has always been my "go to" for file system duplication because it 
will re-create device nodes.


Cheers,
Steve Williams


On 10/12/2017 11:03 AM, webmas...@bennettconstruction.us wrote:

Forgive problems with this email.
I saw how my emails showed up on marc.info
Scary. This is just temporary.

OK. I've tried to use both methods and just don't
get true duplication.

tar
It can't work with file and directory names
that are OK in filesystem, but too long for itself.
Quite a while back I lost a lot of unimportant files
and directories that had absolute paths too long.
Why is this happening with tar? Can this be fixed?
If not, I'd like to add a note about that to the FAQ.

dump
I had to move /usr/local to a bigger partition. growfs,
etc. I kept the /usr/local untouched and then dumped it
to the new partition, expecting a true duplication.
Nope.
It changed all of the program symlinks permissions.
Why is dump doing this? Can this be fixed?
Otherwise, a note about this should be added to the FAQ
also.

Question:
Can dd be used to do what I did with dump or tar?
Smaller partition copied to a bigger partition.

I'm willing to try and help out, but I'm going through
both laptop and server hell at the moment.

Thanks,
Chris Bennett

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

[x9p]

On Mon, December 11, 2017 4:28 am, Robert Paschedag wrote:
>>
>
> Is "rsync" not an option?
>

+1 for rsync. never had a problem.

cheers.

--

x9p | PGP : 0x03B50AF5EA4C8D80 / 5135 92C1 AD36 5293 2BDF  DDCC 0DFA 74AE
1524 E7EE

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

[vincent delft]

On Sun, Dec 10, 2017 at 10:39 PM, Philip Guenther 
wrote:


> 'pax' and 'tar' are actually the same binary so they have the same
> limitation from the file formats that are supported, as well as any purely
> internal limitations.  "pax -rw" actually has file format limitations by
> design, so it doesn't automagically free you from those limitations.
>
>
By Looking a tar man page, I do not find comments regarding "length"
constraints :(.
On pax, it stated that with the default archive format "ustar", filenames
must be 100 char max and path names must be 256 char max.

But It's not clear which archive format the command Tar is using. Is it
using the "old tar" format or the new "ustar" format as described in the
pax man pages ?


Amazing for me to see that both are the same binary

t420:~$ ls -ali /bin/tar
32778 -r-xr-xr-x  3 root  bin  433488 Oct  4 05:13 /bin/tar
t420:~$ ls -ali /bin/pax
32778 -r-xr-xr-x  3 root  bin  433488 Oct  4 05:13 /bin/pax

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

[Robert Paschedag]

Am 10. Dezember 2017 22:17:11 MEZ schrieb vincent delft 
:
>Hello,
>
>Did you tried pax ?
>some thing like: pax -rw -pe  
>
>I don't know if this the best tool, but I'm using it to duplicate a 1TB
>drive (having lot of hard links) onto an other one.
>I've done it couple of time, and I've do not see issues.
>
>rgds
>

Is "rsync" not an option?

>
>
>
>
>
>
>
>
>
>On Sun, Dec 10, 2017 at 7:03 PM, 
>wrote:
>
>> Forgive problems with this email.
>> I saw how my emails showed up on marc.info
>> Scary. This is just temporary.
>>
>> OK. I've tried to use both methods and just don't
>> get true duplication.
>>
>> tar
>> It can't work with file and directory names
>> that are OK in filesystem, but too long for itself.
>> Quite a while back I lost a lot of unimportant files
>> and directories that had absolute paths too long.
>> Why is this happening with tar? Can this be fixed?
>> If not, I'd like to add a note about that to the FAQ.
>>
>> dump
>> I had to move /usr/local to a bigger partition. growfs,
>> etc. I kept the /usr/local untouched and then dumped it
>> to the new partition, expecting a true duplication.
>> Nope.
>> It changed all of the program symlinks permissions.
>> Why is dump doing this? Can this be fixed?
>> Otherwise, a note about this should be added to the FAQ
>> also.
>>
>> Question:
>> Can dd be used to do what I did with dump or tar?
>> Smaller partition copied to a bigger partition.
>>
>> I'm willing to try and help out, but I'm going through
>> both laptop and server hell at the moment.
>>
>> Thanks,
>> Chris Bennett
>>
>>

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

[Philip Guenther]

On Sun, Dec 10, 2017 at 3:46 PM,  wrote:
>
> > Wait, you previously said your problem was with symlinks *permissions*
> but
> > now you're saying *ownership*!  I can confirm that restore(8) didn't
> > preserve the permissions (thus the patch I sent), but as long as you ran
> it
> > with sufficient privilege it should have always restored symlink
> > *ownership*.  Was that a slip of the tongue/fingers?
>
> Sorry, I was just blathering about a different unrelated problem I had with
> website symlinks. My bad.
>

Heh, no problem!

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

[webmaster]

> 
> Wait, you previously said your problem was with symlinks *permissions* but
> now you're saying *ownership*!  I can confirm that restore(8) didn't
> preserve the permissions (thus the patch I sent), but as long as you ran it
> with sufficient privilege it should have always restored symlink
> *ownership*.  Was that a slip of the tongue/fingers?
> 

Sorry, I was just blathering about a different unrelated problem I had
with
website symlinks. My bad.

Chris Bennett

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

[Philip Guenther]

On Sun, Dec 10, 2017 at 3:24 PM,  wrote:
...
>
> > > > dump
> > > > I had to move /usr/local to a bigger partition. growfs,
> > > > etc. I kept the /usr/local untouched and then dumped it
> > > > to the new partition, expecting a true duplication.
> > > > Nope.
> > > > It changed all of the program symlinks permissions.
> >
> > You do know that the mode of a symlink has *no* effect on how the kernel
> > processes it, don't you?  As far as the kernel is concerned, you can do
> > the exact same operations on a mode 0 symlink as on a mode 777 symlink.
>
> No, I didn't know. I have had lots of problems when ownership changes
> with the symlinks, so I wrote I program to delete and restore them with the
> proper owners. Thanks for letting me know. I can delete the files I had
> left on the old
> partition.
>

Wait, you previously said your problem was with symlinks *permissions* but
now you're saying *ownership*!  I can confirm that restore(8) didn't
preserve the permissions (thus the patch I sent), but as long as you ran it
with sufficient privilege it should have always restored symlink
*ownership*.  Was that a slip of the tongue/fingers?

(Symlink ownership doesn't affect the ability to follow the symlink; it
only matters for directory-entry/inode level operations, such as deciding
who can remove a symlink from a mode 1777 directory like /tmp and who can
change the group or perms on the symlink.  AFAIK, the _group_ of a symlink
is never checked.)


Philip Guenther

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

[webmaster]

> 
> 'pax' and 'tar' are actually the same binary so they have the same 
> limitation from the file formats that are supported, as well as any purely 
> internal limitations.  "pax -rw" actually has file format limitations by 
> design, so it doesn't automagically free you from those limitations.
> 
> 
> > On Sun, Dec 10, 2017 at 7:03 PM,  wrote:
> ...
> > > OK. I've tried to use both methods and just don't
> > > get true duplication.
> > >
> > > tar
> > > It can't work with file and directory names
> > > that are OK in filesystem, but too long for itself.
> > > Quite a while back I lost a lot of unimportant files
> > > and directories that had absolute paths too long.
> > > Why is this happening with tar? Can this be fixed?
> > > If not, I'd like to add a note about that to the FAQ.
> 
> tar/pax should have emitted warnings about such files when generating the 
> archive; if that didn't happen it's a bug and we should fix it.  
> Depending on the exact failure you hit there may be ways to fix what you 
> hit.

Yes, I got warnings, I was pulling all of the files off of five failing
hard drives. Luckily, the files were just some pr0n videos, but it could
have been really bad if the hard drive was on it's very last run.

> 
> > > dump
> > > I had to move /usr/local to a bigger partition. growfs,
> > > etc. I kept the /usr/local untouched and then dumped it
> > > to the new partition, expecting a true duplication.
> > > Nope.
> > > It changed all of the program symlinks permissions.
> 
> You do know that the mode of a symlink has *no* effect on how the kernel 
> processes it, don't you?  As far as the kernel is concerned, you can do 
> the exact same operations on a mode 0 symlink as on a mode 777 symlink.
> 

No, I didn't know. I have had lots of problems when ownership changes
with
the symlinks, so I wrote I program to delete and restore them with the
proper owners.
Thanks for letting me know. I can delete the files I had left on the old
partition.

> 
> > > Why is dump doing this? Can this be fixed?
> 
> restore did that because (a) it didn't matter, and (b) there was no API to 
> modify the mode of a symlink (because it didn't matter).
> 
> An API that can chmod a symlink _was_ eventually added: fchmodat(2).  The 
> diff below makes restore preserve symlink mode.
> 

Thanks,
Chris Bennett

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

[Philip Guenther]

On Sun, 10 Dec 2017, vincent delft wrote:
> Did you tried pax ?
> some thing like: pax -rw -pe  
>
> I don't know if this the best tool, but I'm using it to duplicate a 1TB
> drive (having lot of hard links) onto an other one.
> I've done it couple of time, and I've do not see issues.

'pax' and 'tar' are actually the same binary so they have the same 
limitation from the file formats that are supported, as well as any purely 
internal limitations.  "pax -rw" actually has file format limitations by 
design, so it doesn't automagically free you from those limitations.


> On Sun, Dec 10, 2017 at 7:03 PM,  wrote:
...
> > OK. I've tried to use both methods and just don't
> > get true duplication.
> >
> > tar
> > It can't work with file and directory names
> > that are OK in filesystem, but too long for itself.
> > Quite a while back I lost a lot of unimportant files
> > and directories that had absolute paths too long.
> > Why is this happening with tar? Can this be fixed?
> > If not, I'd like to add a note about that to the FAQ.

tar/pax should have emitted warnings about such files when generating the 
archive; if that didn't happen it's a bug and we should fix it.  
Depending on the exact failure you hit there may be ways to fix what you 
hit.


> > dump
> > I had to move /usr/local to a bigger partition. growfs,
> > etc. I kept the /usr/local untouched and then dumped it
> > to the new partition, expecting a true duplication.
> > Nope.
> > It changed all of the program symlinks permissions.

You do know that the mode of a symlink has *no* effect on how the kernel 
processes it, don't you?  As far as the kernel is concerned, you can do 
the exact same operations on a mode 0 symlink as on a mode 777 symlink.


> > Why is dump doing this? Can this be fixed?

restore did that because (a) it didn't matter, and (b) there was no API to 
modify the mode of a symlink (because it didn't matter).

An API that can chmod a symlink _was_ eventually added: fchmodat(2).  The 
diff below makes restore preserve symlink mode.


Philip Guenther


Index: tape.c
===
RCS file: /data/src/openbsd/src/sbin/restore/tape.c,v
retrieving revision 1.49
diff -u -p -r1.49 tape.c
--- tape.c  21 Jan 2017 08:31:44 -  1.49
+++ tape.c  10 Dec 2017 21:33:10 -
@@ -564,6 +564,8 @@ extractfile(char *name)
if (linkit(lnkbuf, name, SYMLINK) == FAIL)
return (FAIL);
(void)lchown(name, uid, gid);
+   (void)fchmodat(AT_FDCWD, name, mode,
+   AT_SYMLINK_NOFOLLOW);
return (GOOD);
}
 

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

[vincent delft]

Hello,

Did you tried pax ?
some thing like: pax -rw -pe  

I don't know if this the best tool, but I'm using it to duplicate a 1TB
drive (having lot of hard links) onto an other one.
I've done it couple of time, and I've do not see issues.

rgds










On Sun, Dec 10, 2017 at 7:03 PM,  wrote:

> Forgive problems with this email.
> I saw how my emails showed up on marc.info
> Scary. This is just temporary.
>
> OK. I've tried to use both methods and just don't
> get true duplication.
>
> tar
> It can't work with file and directory names
> that are OK in filesystem, but too long for itself.
> Quite a while back I lost a lot of unimportant files
> and directories that had absolute paths too long.
> Why is this happening with tar? Can this be fixed?
> If not, I'd like to add a note about that to the FAQ.
>
> dump
> I had to move /usr/local to a bigger partition. growfs,
> etc. I kept the /usr/local untouched and then dumped it
> to the new partition, expecting a true duplication.
> Nope.
> It changed all of the program symlinks permissions.
> Why is dump doing this? Can this be fixed?
> Otherwise, a note about this should be added to the FAQ
> also.
>
> Question:
> Can dd be used to do what I did with dump or tar?
> Smaller partition copied to a bigger partition.
>
> I'm willing to try and help out, but I'm going through
> both laptop and server hell at the moment.
>
> Thanks,
> Chris Bennett
>
>

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

[webmaster]

I'm not able to try it right now, but would gtar
accomplish what that our tar doesn't for this?
As in maybe pull something out of it into our tar?

Chris Bennett

FAQ's duplicating file systems, both methods fail to reproduce correctly

[webmaster]

Forgive problems with this email.
I saw how my emails showed up on marc.info
Scary. This is just temporary.

OK. I've tried to use both methods and just don't
get true duplication.

tar
It can't work with file and directory names
that are OK in filesystem, but too long for itself.
Quite a while back I lost a lot of unimportant files
and directories that had absolute paths too long.
Why is this happening with tar? Can this be fixed?
If not, I'd like to add a note about that to the FAQ.

dump
I had to move /usr/local to a bigger partition. growfs,
etc. I kept the /usr/local untouched and then dumped it
to the new partition, expecting a true duplication.
Nope.
It changed all of the program symlinks permissions.
Why is dump doing this? Can this be fixed?
Otherwise, a note about this should be added to the FAQ
also.

Question:
Can dd be used to do what I did with dump or tar?
Smaller partition copied to a bigger partition.

I'm willing to try and help out, but I'm going through
both laptop and server hell at the moment.

Thanks,
Chris Bennett

Re: file systems

[Nick Holland]

On 05/26/17 10:35, Friedrich Locke wrote:
> Hi folks,
> 
> does anybody here run OBSD with a file system bigger than 10TB ?
> How much time boot takes to bring the system up (i mean fsck) ?
> Are you using ffs2 ? With softdep ?
> 
> Thanks.

Error #1: assuming someone else's experience will have anything at all
to do with yours.  When doing things like this, you need to just test it
yourself with your data and see what happens.  And as I always preach,
"testing" doesn't stop at "I built it and it works!"  Experiment with
failure modes, look at how you move data to a new machine when you need
to, etc.  Your performance will depend greatly on your hw and your data.

Error #2: When you create a single file system that big, you are usually
Doing it Wrong (or at least, sub-optimally).  You really should be
"chunking" your data into more manageable chunks if at all possible.

Some of the benefits to "chunking" your data:
* If you can RO "full" file systems, your fsck time drops to zero for those.
* You buy storage when you need it instead of in advance.  Since storage
almost always gets bigger and cheaper, delaying storage purchase makes a
lot of sense.
* Even with modern hw, a gigabyte of data takes a while to move.  A
terabyte takes a long time to move.
* Someday, you will need to upgrade/replace the hardware.  Chunking your
data gives lots of options here, which might be the difference in doing
a hw replacement/upgrade in a routine hw maintenance window vs. spending
a very long weekend down.
* When something bad happens, you may spend a lot less time recreating
your data.

Nick.

Re: file systems

[Christer Solskogen]

One 27T drive?!


On May 26, 2017 17:27, "Scott Bonds"  wrote:

> I've got a 27T drive, single partition, about half full. Combination of
> big files and lots of small ones. 32G of ECC RAM. Hardware RAID5 ATM though
> I've used software RAID5 on the same array and that was good too.  I keep
> offline backups of everything. I think it takes around an hour to fsck, but
> I haven't timed it. Not using softdep. Not RO. Not sure what the file
> system is, whatever newfs chose...disklabel says fstype is 4.2BSD.
>
> On 05/26, Peter Hessler wrote:
>
>> On 2017 May 26 (Fri) at 11:35:49 -0300 (-0300), Friedrich Locke wrote:
>> :Hi folks,
>> :
>> :does anybody here run OBSD with a file system bigger than 10TB ?
>> :How much time boot takes to bring the system up (i mean fsck) ?
>> :Are you using ffs2 ? With softdep ?
>> :
>> :Thanks.
>>
>> I created a 24T disk with ff2.  I populated 2Tb of it while in async
>> mode, then pulled the power.  fsck took only 5 minutes.
>>
>> Later, I repartitioned the machine to the sizes we actually want
>> (several 5T partitions) and it is running as ftp.hostserver.de.  We
>> aren't using softdep, and generally run RO on many of the partitions.
>>
>> Please give it a try on your own hardware, paritition sizes, and
>> collection of files.
>>
>> --
>> Don't say "yes" until I finish talking.
>> -- Darryl F. Zanuck
>>
>>
>

Re: file systems

[Steven McDonald]

On Fri, 26 May 2017 11:35:49 -0300
Friedrich Locke  wrote:

> Hi folks,
> 
> does anybody here run OBSD with a file system bigger than 10TB ?
> How much time boot takes to bring the system up (i mean fsck) ?
> Are you using ffs2 ? With softdep ?
> 
> Thanks.

This depends heavily on how you plan to use the filesystem. I have an 8
TB filesystem (FFS2 with softdep) that takes several hours to fsck,
mostly in phase 2 ("Check Pathnames") because of a very large number of
hard links.

Re: file systems

[Scott Bonds]

I've got a 27T drive, single partition, about half full. Combination of 
big files and lots of small ones. 32G of ECC RAM. Hardware RAID5 ATM 
though I've used software RAID5 on the same array and that was good too.  
I keep offline backups of everything. I think it takes around an hour to 
fsck, but I haven't timed it. Not using softdep. Not RO. Not sure what 
the file system is, whatever newfs chose...disklabel says fstype is 
4.2BSD.


On 05/26, Peter Hessler wrote:

On 2017 May 26 (Fri) at 11:35:49 -0300 (-0300), Friedrich Locke wrote:
:Hi folks,
:
:does anybody here run OBSD with a file system bigger than 10TB ?
:How much time boot takes to bring the system up (i mean fsck) ?
:Are you using ffs2 ? With softdep ?
:
:Thanks.

I created a 24T disk with ff2.  I populated 2Tb of it while in async
mode, then pulled the power.  fsck took only 5 minutes.

Later, I repartitioned the machine to the sizes we actually want
(several 5T partitions) and it is running as ftp.hostserver.de.  We
aren't using softdep, and generally run RO on many of the partitions.

Please give it a try on your own hardware, paritition sizes, and
collection of files.

--
Don't say "yes" until I finish talking.
-- Darryl F. Zanuck

Re: file systems

[Peter Hessler]

On 2017 May 26 (Fri) at 11:35:49 -0300 (-0300), Friedrich Locke wrote:
:Hi folks,
:
:does anybody here run OBSD with a file system bigger than 10TB ?
:How much time boot takes to bring the system up (i mean fsck) ?
:Are you using ffs2 ? With softdep ?
:
:Thanks.

I created a 24T disk with ff2.  I populated 2Tb of it while in async
mode, then pulled the power.  fsck took only 5 minutes.

Later, I repartitioned the machine to the sizes we actually want
(several 5T partitions) and it is running as ftp.hostserver.de.  We
aren't using softdep, and generally run RO on many of the partitions.

Please give it a try on your own hardware, paritition sizes, and
collection of files.

-- 
Don't say "yes" until I finish talking.
-- Darryl F. Zanuck

file systems

[Friedrich Locke]

Hi folks,

does anybody here run OBSD with a file system bigger than 10TB ?
How much time boot takes to bring the system up (i mean fsck) ?
Are you using ffs2 ? With softdep ?

Thanks.

Books on File Systems and File System Programming

[Siju George]

Hi,

Could Some one recommend good books on  File Systems and File System
Programming Please?

Thanks

Siju

Re: Books on File Systems and File System Programming

[Luis Useche]

I can recommend some papers and books I know for theory as well as
programming.

Papers (you can find them on the internet):
A fast file system for UNIX by McKusick, a clasic one.
The design and implementation of the Log-structured File System by
Rosenblum
An Implementation of a Log- Structured File System for UNIX by Seltzer

Books (you have to buy them):
The Design and Implementation of the 4.4 BSD Operating System: A classic
one.
UNIX Filesystems: Evolution, Design, and Implementation: I have not
read this one much but it seems relevant to what you are looking for.

In case you are interested in books for other operating systems, Linux
has a couple as well:

Understanding the Linux Kernel
Linux kernel development by Love

Hope this helps.

Luis Useche
use...@gmail.com

On Thu, Aug 13, 2009 at 9:11 AM, Siju Georgesgeorge...@gmail.com wrote:
 Hi,

 Could Some one recommend good books on  File Systems and File System
 Programming Please?

 Thanks

 Siju

Re: Books on File Systems and File System Programming

[Jan Stary]

 On Thu, Aug 13, 2009 at 9:11 AM, Siju Georgesgeorge...@gmail.com wrote:
  Could Some one recommend good books on  File Systems and File System
  Programming Please?

On Aug 13 09:45:17, Luis Useche wrote:
 A fast file system for UNIX by McKusick, a clasic one.
 The design and implementation of the Log-structured File System by
 Rosenblum
 An Implementation of a Log- Structured File System for UNIX by Seltzer

Also,

Kleiman, Vnodes: an architecure for multiple file system types
McKusick, Toward a compatible filesystem interface
Rosenthal, Evolving the vnode interface
(and, of course, man vnode)

 Books (you have to buy them):
 The Design and Implementation of the 4.4 BSD Operating System: A classic
 one.
 UNIX Filesystems: Evolution, Design, and Implementation: I have not
 read this one much but it seems relevant to what you are looking for.

Re: Books on File Systems and File System Programming

[Otto Moerbeek]

On Thu, Aug 13, 2009 at 09:45:17AM -0400, Luis Useche wrote:

 I can recommend some papers and books I know for theory as well as
 programming.
 
 Papers (you can find them on the internet):
 A fast file system for UNIX by McKusick, a clasic one.

Or in /usr/share/doc/smm/05.fastfs; make ps and you have it nicely
formatted in postscript. 

-Otto

Re: Books on File Systems and File System Programming

[Siju George]

On Thu, Aug 13, 2009 at 8:04 PM, Otto Moerbeeko...@drijf.net wrote:

 Or in /usr/share/doc/smm/05.fastfs; make ps and you have it nicely
 formatted in postscript.


Thanks a million Luis, Jan and Otto.

--Siju

fsck_ffs for large file systems

[Geoff Steckel]

This is a patch to 4.3 release fsck_ffs to reduce the block map
memory usage in almost all cases. It uses a sparse representation
where regions of all zeros or all ones require no memory.
In the worst case (every region contains both ones and zeros)
it increases memory usage by less than 2%. In the best case
it reduces memory usage for the block map by 98% or so.

CPU usage is increased slightly. Since fsck is very disk-bound
I believe this will not be a problem.

This is a VERY preliminary version. It has been tested on the
few large filesystems (  30G) I have.  I do not assert that
it is acceptable for production in either format or content.
It contains debug code, #if 0, #if 1, and other constructs
which would not be present in a production version.

I would like people to try it and see how badly it fails.
Please send me any failure information and I will attempt to fix the problem.

Thanks very much.
   geoff steckel

diff -Pupr /deep/4.3/src/sbin/fsck_ffs/Makefile fsck_ffs/Makefile
--- /deep/4.3/src/sbin/fsck_ffs/MakefileSun Sep 21 07:36:37 1997
+++ fsck_ffs/Makefile   Mon May 19 15:08:41 2008
@@ -3,7 +3,7 @@
 PROG=  fsck_ffs
 MAN=   fsck_ffs.8
 SRCS=  dir.c inode.c main.c pass1.c pass1b.c pass2.c pass3.c pass4.c \
-   pass5.c fsutil.c setup.c utilities.c ffs_subr.c ffs_tables.c
+   pass5.c fsutil.c setup.c utilities.c ffs_subr.c ffs_tables.c blockmap.c
 .PATH: ${.CURDIR}/../../sys/ufs/ffs ${.CURDIR}/../fsck
 CFLAGS+= -I${.CURDIR}/../fsck
 
diff -Pupr /deep/4.3/src/sbin/fsck_ffs/blockmap.c fsck_ffs/blockmap.c
--- /deep/4.3/src/sbin/fsck_ffs/blockmap.c  Wed Dec 31 19:00:00 1969
+++ fsck_ffs/blockmap.c Mon May 19 17:45:51 2008
@@ -0,0 +1,133 @@
+/*
+ * Copyright (c) 2008 Geoff Steckel. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *notice, this list of conditions and the following disclaimer in the
+ *documentation and/or other materials provided with the distribution.
+ * 3. The names of the contributors
+ *may be used to endorse or promote products derived from this software
+ *without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL THE CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#define DKTYPENAMES
+#include sys/param.h
+#include sys/time.h
+#include ufs/ufs/dinode.h
+#include ufs/ffs/fs.h
+#include sys/stat.h
+#include sys/ioctl.h
+#include sys/disklabel.h
+
+#include errno.h
+#include fcntl.h
+#include stdio.h
+#include stdlib.h
+#include string.h
+#include ctype.h
+
+#include fsck.h
+#include extern.h
+#include fsutil.h
+
+#define ERR1   tsetbit couldn't enter a new item in blockmap\n
+#define ERR2   tclrbit couldn't find an item in blockmap\n
+
+void
+blkmapset(struct blockmapchunk **blockmap, daddr64_t blkno)
+{
+   struct blockmapchunk *thischunk;
+   daddr64_t chunkno;
+   unsigned int ofsinchunk;
+
+   if (blkno = maxfsblock + 7) {
+   printf(blkmapset called with block %lld out of range\n, 
blkno);
+   return;
+   }
+   chunkno = blkno / BLKMAPCHUNK;
+   ofsinchunk = blkno % BLKMAPCHUNK;
+   thischunk = blockmap[chunkno];
+   if ( ! thischunk) {
+   thischunk = calloc(1, sizeof *thischunk);
+   if ( ! thischunk) {
+   printf(blkmapset can't alloc for block %lld\n, blkno);
+   return;
+   }
+   blockmap[chunkno] = thischunk;
+   }
+   setbit(thischunk-bmc_bits, ofsinchunk);
+   thischunk-bmc_count++;
+   if (thischunk-bmc_count == BLKMAPCHUNK) {
+   free(thischunk);
+   blockmap[chunkno] = alloneschunk;
+   }
+}
+
+int
+blkmaptest(struct blockmapchunk **blockmap, daddr64_t blkno)
+{
+   struct blockmapchunk *thischunk;
+   daddr64_t chunkno;
+   unsigned int ofsinchunk;
+
+   if (blkno = maxfsblock + 7) {
+   printf(blkmaptest called with block %lld out of range\n, 
blkno);
+   return 0;

Re: df -h stats for same file systems display different result son AMD64 then on i386 (Source solved)

[Daniel Ouellet]

OK,

Here is the source of the problem. The cache file generated by 
webazolver is the source of the problem. Based on the information of the 
software webalizer, as this:


Cached DNS addresses have a TTL (time to live) of 3 days.  This may be
changed at compile time by editing the dns_resolv.h header file and
changing the value for DNS_CACHE_TTL.

The cache file is process each night, and the records older then 3 days 
are remove, but somehow that file become a sparse file in the process 
and when copy else where show it's real size. In my case that file was 
using a bit over 4 millions blocks more then it should have and give me 
the 4GB+ difference in mirroring the content.


So, as far as I can see it, this process of expiring the records from 
the cache file that is always reuse doesn't shrink the file really, but 
somehow just mark the records inside the file as bad, or something like 
that.


So, nothing to do with OpenBSD at all but I would think there is a bug 
in the portion of webalizer however base on what I see from it's usage.


Now the source of the problem was found and many thanks to all that 
stick with me along the way.


Always feel good to know in the end!

Thanks to Otto, Ted and Tom.

Daniel

Re: df -h stats for same file systems display different result son AMD64 then on i386 (Source solved)

[Otto Moerbeek]

On Tue, 17 Jan 2006, Daniel Ouellet wrote:

 OK,
 
 Here is the source of the problem. The cache file generated by webazolver is
 the source of the problem. Based on the information of the software webalizer,
 as this:
 
 Cached DNS addresses have a TTL (time to live) of 3 days.  This may be
 changed at compile time by editing the dns_resolv.h header file and
 changing the value for DNS_CACHE_TTL.
 
 The cache file is process each night, and the records older then 3 days are
 remove, but somehow that file become a sparse file in the process and when
 copy else where show it's real size. In my case that file was using a bit over
 4 millions blocks more then it should have and give me the 4GB+ difference in
 mirroring the content.
 
 So, as far as I can see it, this process of expiring the records from the
 cache file that is always reuse doesn't shrink the file really, but somehow
 just mark the records inside the file as bad, or something like that.
 
 So, nothing to do with OpenBSD at all but I would think there is a bug in the
 portion of webalizer however base on what I see from it's usage.
 
 Now the source of the problem was found and many thanks to all that stick with
 me along the way.

You are wrong in thinking sparse files are a problem. Having sparse
files quite a nifty feature, I would say. 


-Otto

Re: df -h stats for same file systems display different result son AMD64 then on i386 (Source solved)

[Joachim Schipper]

On Tue, Jan 17, 2006 at 02:15:57PM +0100, Otto Moerbeek wrote:
 On Tue, 17 Jan 2006, Daniel Ouellet wrote:
 
  OK,
  
  Here is the source of the problem. The cache file generated by
  webazolver is the source of the problem. Based on the information of
  the software webalizer, as this:
  
  Cached DNS addresses have a TTL (time to live) of 3 days.  This may
  be changed at compile time by editing the dns_resolv.h header file
  and changing the value for DNS_CACHE_TTL.
  
  The cache file is process each night, and the records older then 3
  days are remove, but somehow that file become a sparse file in the
  process and when copy else where show it's real size. In my case
  that file was using a bit over 4 millions blocks more then it should
  have and give me the 4GB+ difference in mirroring the content.
  
  So, as far as I can see it, this process of expiring the records
  from the cache file that is always reuse doesn't shrink the file
  really, but somehow just mark the records inside the file as bad, or
  something like that.
  
  So, nothing to do with OpenBSD at all but I would think there is a
  bug in the portion of webalizer however base on what I see from it's
  usage.
  
  Now the source of the problem was found and many thanks to all that
  stick with me along the way.
 
 You are wrong in thinking sparse files are a problem. Having sparse
 files quite a nifty feature, I would say. 

Are we talking about webazolver or OpenBSD?

I'd argue that relying on the OS handling sparse files this way instead
of handling your own log data in an efficient way *is* a problem, as
evidenced by Daniels post. After all, it's reasonable to copy data to,
say, a different drive and expect it to take about as much space as the
original.

On the other hand, I agree with you that handling sparse files
efficiently is rather neat in an OS.

Joachim

Re: df -h stats for same file systems display different result son AMD64 then on i386 (Source solved)

[Otto Moerbeek]

On Tue, 17 Jan 2006, Joachim Schipper wrote:

 On Tue, Jan 17, 2006 at 02:15:57PM +0100, Otto Moerbeek wrote:

  You are wrong in thinking sparse files are a problem. Having sparse
  files quite a nifty feature, I would say. 
 
 Are we talking about webazolver or OpenBSD?
 
 I'd argue that relying on the OS handling sparse files this way instead
 of handling your own log data in an efficient way *is* a problem, as
 evidenced by Daniels post. After all, it's reasonable to copy data to,
 say, a different drive and expect it to take about as much space as the
 original.

Now that's a wrong assumption. A file is a row of bytes. The only
thing I can assume is that if I write a byte at a certain position, I
will get the same byte back when reading the file. Furthermoe, the
file size (not the disk space used!) is the largest position written.
If I assume anything more, I'm assuming too much.

For an application, having sparse files is completely transparant. The
application doesn't even know the difference. How the OS stores the
file is up to the OS.

Again, assuming a copy of a file takes up as much space as the
original is wrong. 

 On the other hand, I agree with you that handling sparse files
 efficiently is rather neat in an OS.

-Otto

Re: df -h stats for same file systems display different result son AMD64 then on i386 (Source solved)

[Daniel Ouellet]

You are wrong in thinking sparse files are a problem. Having sparse
files quite a nifty feature, I would say. 



Are we talking about webazolver or OpenBSD?

I'd argue that relying on the OS handling sparse files this way instead
of handling your own log data in an efficient way *is* a problem, as
evidenced by Daniels post. After all, it's reasonable to copy data to,
say, a different drive and expect it to take about as much space as the
original.


Just as feedback the size showed something like 150MB or so as the 
original file on OpenBSD. Using RSYNC to copy it over makes it almost 
5GB in size, well I wouldn't call that good. But again, before I say no 
 definitely, there is always something that I may not understands, so I 
am welling to leave some space for that here. But not much! (:



On the other hand, I agree with you that handling sparse files
efficiently is rather neat in an OS.


I am not sure that the OS handle it well or not. Again, no punch 
intended, but if it was, why copy no data then? Obviously something I 
don't understand for sure.


However, here is something I didn't include in my previous email with 
all the stats and may be very interesting to know. I didn't think it was 
so important at the time, but if you talk about handling it properly, 
may be it might be relevant.


The test were done with three servers. The file showing ~150MB in size 
was on www1. Then copying it to www2 with the -S switch in rsync 
regardless got it to ~5GB. Then copying the same file from www2 to www3 
using the same rsync -S setup go that file back to the size it was on 
www1. So, why not in the www2 in that case. So, it the the OS, or is 
that the rsync. Was it handle properly or wasn't it? I am not sure. If 
it was, then the www2 file should not have been ~5GB should it?


So the picture was

www1-www2-www3

www1 cache DB show 150MB

rsync -e ssh -aSuqz --delete /var/www/sites/ [EMAIL PROTECTED]:/var/www/sites

www2 cache DB show ~5GB

rsync -e ssh -aSuqz --delete /var/www/sites/ [EMAIL PROTECTED]:/var/www/sites

www3 cache DB show ~150MB

Why not 150Mb on www2???

One think that I haven't tried and regret not have done that not to know 
is just copying that file on www1 to a different name and then copying 
it again to it's original name and check the size at the and and the 
transfer of that file as well I without the -S switch to see if the OS 
did copy the empty data or not.


I guess the question would be, should it, or shouldn't it do it?

My own opinion right now is the file should show the size it really is. 
So, if it is 5GB and only 100MB is good on it, shouldn't it show it to 
be 5GB? I don't know, better mind then me sure have the answer to this 
one, right now, I do not for sure.

Re: df -h stats for same file systems display different result son AMD64 then on i386 (Source solved)

[Joachim Schipper]

On Tue, Jan 17, 2006 at 05:49:24PM +0100, Otto Moerbeek wrote:
 On Tue, 17 Jan 2006, Joachim Schipper wrote:
 
  On Tue, Jan 17, 2006 at 02:15:57PM +0100, Otto Moerbeek wrote:
 
   You are wrong in thinking sparse files are a problem. Having sparse
   files quite a nifty feature, I would say. 
  
  Are we talking about webazolver or OpenBSD?
  
  I'd argue that relying on the OS handling sparse files this way instead
  of handling your own log data in an efficient way *is* a problem, as
  evidenced by Daniels post. After all, it's reasonable to copy data to,
  say, a different drive and expect it to take about as much space as the
  original.
 
 Now that's a wrong assumption. A file is a row of bytes. The only
 thing I can assume is that if I write a byte at a certain position, I
 will get the same byte back when reading the file. Furthermoe, the
 file size (not the disk space used!) is the largest position written.
 If I assume anything more, I'm assuming too much.
 
 For an application, having sparse files is completely transparant. The
 application doesn't even know the difference. How the OS stores the
 file is up to the OS.
 
 Again, assuming a copy of a file takes up as much space as the
 original is wrong. 
 
  On the other hand, I agree with you that handling sparse files
  efficiently is rather neat in an OS.

Okay - I understand your logic, and yes, I do know about sparse files
and how they are typically handled. And yes, you are right that
there are very good reasons for handling sparse files this way.

And yes, application are right to make use of this feature where
applicable.

However, in this case, it's a simple log file, and what the application
did, while very much technically correct, clearly violated the principle
of least astonishment, for no real reason I can see. Sure, trying to
make efficient use of every single byte may not be very efficient - but
just zeroing out the first five GB of the file is more than a little
hackish, and not really necessary.

Joachim

Re: df -h stats for same file systems display different result son AMD64 then on i386 (Source solved)

[Matthias Kilian]

On Tue, Jan 17, 2006 at 02:36:44PM -0500, Daniel Ouellet wrote:
 [...] But having a 
 file that is let say 1MB of valid data that grow very quickly to 4 and 
 6GB quickly and takes time to rsync between servers were in one instance 
 fill the fill system and create other problem. (: I wouldn't call that 
 a feature.

As Otto noted, you've distinguish between file size (that's what
stat(2) and friends report, and at the same time it's the number
of bytes you can read sequentially from the file), and a file's
disk usage.

For more explanations, see the RATIONALE section at

http://www.opengroup.org/onlinepubs/009695399/utilities/du.html

(You may have to register, but it doesn't hurt)

See also the reference to lseek(2) mentioned there.


 But at the same time, I wasn't using the -S switch in rsync, 
 so my own stupidity there. However, why spend lots of time processing 
 empty files I still don't understand that however.

Please note that -S in rsync does not *guarantee* that source and
destination files are *identical* in terms of holes or disk usage.

For example:

$ dd if=/dev/zero of=foo bs=1m count=42
$ rsync -S foo host:
$ du foo
$ ssh host du foo

Got it? The local foo is *not* sparse (no holes), but the remote
one has been optimized by rsync's -S switch.

We recently had a very controverse (and flaming) discussion at our
local UG on such optimizations (or heuristics, as in GNU cp).
IMO, if they have to be explicitely enabled (like `-S' for rsync),
that's o.k. The other direction (copy is *not* sparse by default)
is exactly what I would expect.

Telling wether a sequence of zeroes is a hole or just a (real) block
of zeroes isn't possible in userland -- it's a filesystem implementation
detail.

To copy the *exact* contents of an existing filesystem including
all holes to another disk (or system), you *have* to use
filesystem-specific tools, such as dump(8) and restore(8). Period.


 I did research on google for sparse files and try to get more 
 informations about it. In some cases I would assume like if you do round 
 database type of stuff where you have a fix file that you write in at 
 various place or something, would be good and useful, but a sparse file 
 that keep growing over time uncontrol, I may be wrong, but I don't call 
 that useful feature.

Sparse files for databases on heavy load (many insertions and
updates) ar the death of performance -- you'll get files with blocks
spreaded all over your filesystem.

OTH, *spare* databases such as quotas files (potentially large, but
growing very slowly) are good candidates for sparse files.

Ciao,
Kili

Re: df -h stats for same file systems display different result son AMD64 then on i386 (Source solved)

[Daniel Ouellet]

Hi all,

First let me start with my apology to some of you for having waisted 
your time!


As much as this was/is interesting and puzzling to me and that I am 
trying obviously to get my hands around this issue and usage of sparse 
files, the big picture of it, is obviously something missing in my 
understanding at this time.


I am doing more research on my own, so lets kill this tread and sorry to 
have waisted any of your time with my lack of understanding of this aspect!


I am not trying to be a fucking idiot on the list, but it's obvious 
that I don't understand this at this time.


So, lets drop it and I will continue my homework!

Big thanks to all that try to help me as well!

Daniel

Re: df -h stats for same file systems display different result son AMD64 then on i386

[Otto Moerbeek]

On Sun, 15 Jan 2006, Daniel Ouellet wrote:

 Otto Moerbeek wrote:
  On Sun, 15 Jan 2006, Daniel Ouellet wrote:

  Since the bsize and fsize differ, it is expected that the used kbytes of the
  file systems differ. Also, the inode table size will not be the same.
 
 Not sure that I would agree fully with that, but I differ to your judgment.
 Yes there will and should be difference in usage as if you have a lots of
 small files, you are waisting more space if you fsize are bigger, unless I
 don't understand that part. Would it mean that the df -h would take the number
 of inode in use * the fsize to display the results for human then?

I do not understand what you mean. Of course df does not do such
calculation, because it does not mean anything. Inodes allocated does
have little to do with total space in use. fsize is fragment size,
which is something different than file size.

  You're comparing apples and oranges.
 
 I don't disagree to some extend as you know better, but I still try to
 understand it however. Shouldn't the df -h display the same results however to
 human? I am not arguing, but rather try to understand it. If it is design to
 be human converted, why a human would need to know or consider the file size
 in use then to compare the results?

The human thing is only doing the conversion to megabytes and such. It
does not compensate for space wasted due to blocks not being fully used
and such.

Now I agree that the difference you are seeing is larger than I would
expect. I would run a ls -laR or du -k on the filesystems and diff the
results to see if the contents are realy the same. My bet is that
you'll discover some files that are not on the system with a smaller
usage.  It is also perfectly possible that files having holes (also
called spare files) play a role: they take less space than their
length, but depending on how you copy them, the copy does take the
full amount of blocks. 

  BTW, you don't say which version(s) you are running. That's bad. since
  some bugs were fixed in the -h display. Run df without -h to see the
  real numbers.
 
 All run 3.8. Sorry about that.
 
 the 4.6GB have 4870062 * 1024 = 4,986,943,488
 www1# df
 Filesystem  1K-blocks  Used Avail Capacity  Mounted on
 /dev/wd0a  256814 4146420251017%/
 /dev/wd0h 104815854995698 0%/home
 /dev/wd0d 1030550 2979022 0%/tmp
 /dev/wd0g 5159638310910   4590748 6%/usr
 /dev/wd0e25799860   4870062  1963980620%/var
 /dev/wd0f 1030550  1546977478 0%/var/qmail

The above display used df -k, while the one below does not. Probably
you've set some alias for df or so, or you are using the BLOCKSIZE env
var. Why are you making things more difficult than needed for us (and
yourself?).

 
 the 8.1GB have 15967148 * 512 = 8,175,179,776
 # df
 Filesystem  512-blocks  Used Avail Capacity  Mounted on
 /dev/wd0a   513628 6558842236013%/
 /dev/wd0h  186162852   1768496 0%/home
 /dev/wd0d  2061100 4   1958044 0%/tmp
 /dev/wd0g  9904156424544   8984408 5%/usr
 /dev/wd0e 33022236   1537612  29833516 5%/var
 /dev/wd1b 16412252   1937920  1365372012%/var/mysql
 /dev/wd0f  2061100 4   1958044 0%/var/qmail
 /dev/wd1a 41280348  15967148  2324918441%/var/www/sites
 
 The funny part is that the first above /var include more files then the
 /var/www/sites below and still display less space in use.
 
  To check if the inode/block/fragment free numbers add up, you could
  use dumpfs, but that is a hell of a lot of work. 
  -Otto
  
 
 It's not a huge deal and the systems works well, I am just puzzle by the
 results and want to understand it, that's all.

Re: df -h stats for same file systems display different result son AMD64 then on i386

[Ted Unangst]

run du on both filesystems and compare the results.

Re: df -h stats for same file systems display different result son AMD64 then on i386

[Daniel Ouellet]

Ted Unangst wrote:

run du on both filesystems and compare the results.



OK, just because I am curious more then think there is a problem, and 
because I am still puzzle from what Otto and Ted said, here is what I 
did and the answer to question from Otto as well.


- Both system run 3.8. (www1 was running 3.6 and updated to 3.7, then 
3.8 all good following strict step from Nick in the FAQ). www2 was a 
full clean install from scratch, full wipe out, not an upgrade.


- There isn't any hard or soft link in that section.

- On the blocksize Otto said/asked if I play with it. No, never did, 
always did fresh install by default until this time when I try the 
upgrade from Nick as this www1 was a lots of work to do fresh, but I may 
just redo it to see.


Then as there is a lots of files and comparing them manually is really a 
lots of work, I use rsync 2.6.6 to mirror them:


www1# pkg_info | grep rsync
rsync-2.6.6 mirroring/synchronization over low bandwidth links

www2# pkg_info | grep rsync
rsync-2.6.6 mirroring/synchronization over low bandwidth links

Then, I use rsync and change the setup to allow to log in and run it as 
root so that no restrictions would be there in anyway as below to be 
sure I have a full identical copy of all the files just like this:


rsync -e ssh -auqz --delete /var/www/sites/ [EMAIL PROTECTED]:/var/www/sites

Then I did:
du -h /var/www/sites on www1 and get 3.9G/var/www/sites
du -h /var/www/sites on www2 and get 7.7G/var/www/sites

Also remember that www1 was the one upgraded, and www2 fresh new install.

Now I continue to look but I am not sure what else I can do to be 100% 
sure that all the files are identical before comparing them.


I am still comparing the results from du, but that's huge!

So, may be this test is not a valid one, but then why not?

It is interesting however to say the less.

Now, to push the issue even more, I did this with a third server from 
the www1 again:


rsync on it
www3# pkg_info | grep rsync
rsync-2.6.6 mirroring/synchronization over low bandwidth links

Then mirror it:
rsync -e ssh -auqz --delete /var/www/sites/ [EMAIL PROTECTED]:/var/www/sites

Then same, I did:
du -h /var/www/sites on www1 and get 3.9G/var/www/sites
du -h /var/www/sites on www3 and get 7.7G/var/www/sites

The only difference in hardware setup is that both www2 and www3 have 
their own drive for that mount point where www1 does not and run i386 
oppose to AMD64.


www1# df | grep /var/www/sites
www1# df | grep /var
/dev/wd0e25799860   4905500  1960436820%/var
/dev/wd0f 1030550  1552977472 0%/var/qmail

www2# df | grep /var/www/sites
/dev/wd1a20640174   8495528  263843%/var/www/sites

www3# df | grep /var/www/sites
/dev/wd1a20640174   8024648  1158351841%/var/www/sites

So, the www1 have to have even more stuff on the drive compare to www2 
and www3, but show much less.


So copying from www1 to www2 and to www3 give the same results and www2 
and www3 match very well, but www1 still show much less for sure.


I am very puzzle at best.

Re: df -h stats for same file systems display different result son AMD64 then on i386

[Daniel Ouellet]

Otto Moerbeek wrote:

Now I agree that the difference you are seeing is larger than I would
expect. I would run a ls -laR or du -k on the filesystems and diff the
results to see if the contents are realy the same. My bet is that
you'll discover some files that are not on the system with a smaller
usage.  It is also perfectly possible that files having holes (also
called spare files) play a role: they take less space than their
length, but depending on how you copy them, the copy does take the
full amount of blocks. 


So, I now did the ls -laR on each system:

ls -laR /var/www/sites  /tmp/wwwx where x was the server number

Then I compare the results with diff www1 www2 as well as diff www1 and 
www3:


diff www1 www3 /tmp/check

Then look at the check file, nothing that can explain the difference 
there. The only difference in file size there are the log files as they 
change live obviously, but the rest is the same.


I still can't explain it even using the ls -LaR options and looking at 
the diff between them.


In any case definitely nothing in the order of ~ 4GB.

I guess next test would be to wipe out and reinstall fresh to be sure.

PS; I did reboot the servers in case there was any open files or 
anything like that, still same results.


Now the only thing in what Otto said that make me thing is this 
statement but depending on how you copy them.


I use rsync all the time to do this!

Like: rsync -e ssh -auqz --delete 

Would this explain it? I can't imagine it would, but I don't know 
everything for sure.

Re: df -h stats for same file systems display different result son AMD64 then on i386

[Daniel Ouellet]

Just a bit more information on this.

As I couldn't understand if that was an AMD64 issue as illogical as that 
might be, I decided to put that to the test. So, I pull out an other 
AMD64 server and it's running 3.8, same fsize and bsize, one drive, etc.


Use rsync to mirror the content and the results are consistent with the 
i386. So, that prove it's not that at a minimum.


So, the source is 4.4GB and expand to 7.7GB.

I have no logical explications what so ever.

So, I will stop here and just drop it as I have nothing else logical I 
can think of to look at why that might be.


I will just have to put it in the unknown pile and leave it alone.

One thing for sure, I will be wiping that box out to be sure next.

Unless someone have an idea, again, not a problem, but something very 
weird I was trying to understand and find some logical explication too.


I will just have to take it as such and leave it alone.

So, case close as I have no more idea or clue.

This was done on a brand new server and using rsync to copy identical 
files from the source to the destination.


==
Source:
www1# disklabel wd0
# Inside MBR partition 3: type A6 start 63 size 78156162
# /dev/rwd0c:
type: ESDI
disk: ESDI/IDE disk
label: Maxtor 6E040L0
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 16
sectors/cylinder: 1008
cylinders: 16383
total sectors: 78165360
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0   # microseconds
track-to-track seek: 0  # microseconds
drivedata: 0

16 partitions:
# sizeoffset  fstype [fsize bsize  cpg]
  a:52409763  4.2BSD   2048 16384  328 # Cyl 
0*-   519
  b:   8388576524160swap   # Cyl   520 
-  8841
  c:  78165360 0  unused  0 0  # Cyl 0 
- 77544
  d:   2097648   8912736  4.2BSD   2048 16384  328 # Cyl  8842 
- 10922
  e:  52429104  11010384  4.2BSD   2048 16384  328 # Cyl 10923 
- 62935
  f:   2097648  63439488  4.2BSD   2048 16384  328 # Cyl 62936 
- 65016
  g:  10486224  65537136  4.2BSD   2048 16384  328 # Cyl 65017 
- 75419
  h:   2132865  76023360  4.2BSD   2048 16384  328 # Cyl 75420 
- 77535*

www1# df -h
Filesystem SizeUsed   Avail Capacity  Mounted on
/dev/wd0a  251M   40.5M198M17%/
/dev/wd0h 1024M   54.0K972M 0%/home
/dev/wd0d 1006M8.5M948M 1%/tmp
/dev/wd0g  4.9G304M4.4G 6%/usr
/dev/wd0e 24.6G4.7G   18.7G20%/var
/dev/wd0f 1006M1.5M955M 0%/var/qmail

=

Destination.
# disklabel wd0
# Inside MBR partition 3: type A6 start 63 size 156296322
# /dev/rwd0c:
type: ESDI
disk: ESDI/IDE disk
label: Maxtor 6L080M0
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 16
sectors/cylinder: 1008
cylinders: 16383
total sectors: 156301488
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0   # microseconds
track-to-track seek: 0  # microseconds
drivedata: 0

16 partitions:
# sizeoffset  fstype [fsize bsize  cpg]
  a:   104825763  4.2BSD   2048 16384  328 # Cyl 
0*-  1039
  b:  16777152   1048320swap   # Cyl  1040 
- 17683
  c: 156301488 0  unused  0 0  # Cyl 0 
-155060
  d:  10486224  17825472  4.2BSD   2048 16384  328 # Cyl 17684 
- 28086
  e:  83885760  28311696  4.2BSD   2048 16384  328 # Cyl 28087 
-111306
  f:   4194288 112197456  4.2BSD   2048 16384  328 # Cyl 111307 
-115467
  g:   2097648 116391744  4.2BSD   2048 16384  328 # Cyl 115468 
-117548
  h:  20971440 118489392  4.2BSD   2048 16384  328 # Cyl 117549 
-138353
  i:  16835553 139460832  4.2BSD   2048 16384  328 # Cyl 138354 
-155055*

# df -h
Filesystem SizeUsed   Avail Capacity  Mounted on
/dev/wd0a  502M   49.3M427M10%/
/dev/wd0i  7.9G2.0K7.5G 0%/home
/dev/wd0d  4.9G2.0K4.7G 0%/tmp
/dev/wd0h  9.8G958M8.4G10%/usr
/dev/wd0e 39.4G7.7G   29.7G21%/var
/dev/wd0f  2.0G252K1.9G 0%/var/log
/dev/wd0g 1006M2.0K956M 0%/var/qmail

Re: df -h stats for same file systems display different result son AMD64 then on i386

[Otto Moerbeek]

On Mon, 16 Jan 2006, Daniel Ouellet wrote:

 Just a bit more information on this.
 
 As I couldn't understand if that was an AMD64 issue as illogical as that might
 be, I decided to put that to the test. So, I pull out an other AMD64 server
 and it's running 3.8, same fsize and bsize, one drive, etc.
 
 Use rsync to mirror the content and the results are consistent with the i386.
 So, that prove it's not that at a minimum.
 
 So, the source is 4.4GB and expand to 7.7GB.
 
 I have no logical explications what so ever.

You have been told an explanation (sparse files).

-Otto

 
 So, I will stop here and just drop it as I have nothing else logical I can
 think of to look at why that might be.
 
 I will just have to put it in the unknown pile and leave it alone.
 
 One thing for sure, I will be wiping that box out to be sure next.
 
 Unless someone have an idea, again, not a problem, but something very weird I
 was trying to understand and find some logical explication too.
 
 I will just have to take it as such and leave it alone.
 
 So, case close as I have no more idea or clue.
 
 This was done on a brand new server and using rsync to copy identical files
 from the source to the destination.
 
 ==
 Source:
 www1# disklabel wd0
 # Inside MBR partition 3: type A6 start 63 size 78156162
 # /dev/rwd0c:
 type: ESDI
 disk: ESDI/IDE disk
 label: Maxtor 6E040L0
 flags:
 bytes/sector: 512
 sectors/track: 63
 tracks/cylinder: 16
 sectors/cylinder: 1008
 cylinders: 16383
 total sectors: 78165360
 rpm: 3600
 interleave: 1
 trackskew: 0
 cylinderskew: 0
 headswitch: 0   # microseconds
 track-to-track seek: 0  # microseconds
 drivedata: 0
 
 16 partitions:
 # sizeoffset  fstype [fsize bsize  cpg]
   a:52409763  4.2BSD   2048 16384  328 # Cyl 0*-   519
   b:   8388576524160swap   # Cyl   520 -  8841
   c:  78165360 0  unused  0 0  # Cyl 0 - 77544
   d:   2097648   8912736  4.2BSD   2048 16384  328 # Cyl  8842 - 10922
   e:  52429104  11010384  4.2BSD   2048 16384  328 # Cyl 10923 - 62935
   f:   2097648  63439488  4.2BSD   2048 16384  328 # Cyl 62936 - 65016
   g:  10486224  65537136  4.2BSD   2048 16384  328 # Cyl 65017 - 75419
   h:   2132865  76023360  4.2BSD   2048 16384  328 # Cyl 75420 -
 77535*
 www1# df -h
 Filesystem SizeUsed   Avail Capacity  Mounted on
 /dev/wd0a  251M   40.5M198M17%/
 /dev/wd0h 1024M   54.0K972M 0%/home
 /dev/wd0d 1006M8.5M948M 1%/tmp
 /dev/wd0g  4.9G304M4.4G 6%/usr
 /dev/wd0e 24.6G4.7G   18.7G20%/var
 /dev/wd0f 1006M1.5M955M 0%/var/qmail
 
 =
 
 Destination.
 # disklabel wd0
 # Inside MBR partition 3: type A6 start 63 size 156296322
 # /dev/rwd0c:
 type: ESDI
 disk: ESDI/IDE disk
 label: Maxtor 6L080M0
 flags:
 bytes/sector: 512
 sectors/track: 63
 tracks/cylinder: 16
 sectors/cylinder: 1008
 cylinders: 16383
 total sectors: 156301488
 rpm: 3600
 interleave: 1
 trackskew: 0
 cylinderskew: 0
 headswitch: 0   # microseconds
 track-to-track seek: 0  # microseconds
 drivedata: 0
 
 16 partitions:
 # sizeoffset  fstype [fsize bsize  cpg]
   a:   104825763  4.2BSD   2048 16384  328 # Cyl 0*-  1039
   b:  16777152   1048320swap   # Cyl  1040 - 17683
   c: 156301488 0  unused  0 0  # Cyl 0 -155060
   d:  10486224  17825472  4.2BSD   2048 16384  328 # Cyl 17684 - 28086
   e:  83885760  28311696  4.2BSD   2048 16384  328 # Cyl 28087 -111306
   f:   4194288 112197456  4.2BSD   2048 16384  328 # Cyl 111307
 -115467
   g:   2097648 116391744  4.2BSD   2048 16384  328 # Cyl 115468
 -117548
   h:  20971440 118489392  4.2BSD   2048 16384  328 # Cyl 117549
 -138353
   i:  16835553 139460832  4.2BSD   2048 16384  328 # Cyl 138354
 -155055*
 # df -h
 Filesystem SizeUsed   Avail Capacity  Mounted on
 /dev/wd0a  502M   49.3M427M10%/
 /dev/wd0i  7.9G2.0K7.5G 0%/home
 /dev/wd0d  4.9G2.0K4.7G 0%/tmp
 /dev/wd0h  9.8G958M8.4G10%/usr
 /dev/wd0e 39.4G7.7G   29.7G21%/var
 /dev/wd0f  2.0G252K1.9G 0%/var/log
 /dev/wd0g 1006M2.0K956M 0%/var/qmail

Re: df -h stats for same file systems display different result son AMD64 then on i386

[Daniel Ouellet]

Otto Moerbeek wrote:

On Mon, 16 Jan 2006, Daniel Ouellet wrote:


Just a bit more information on this.

As I couldn't understand if that was an AMD64 issue as illogical as that might
be, I decided to put that to the test. So, I pull out an other AMD64 server
and it's running 3.8, same fsize and bsize, one drive, etc.

Use rsync to mirror the content and the results are consistent with the i386.
So, that prove it's not that at a minimum.

So, the source is 4.4GB and expand to 7.7GB.

I have no logical explications what so ever.


You have been told an explanation (sparse files).

-Otto



Yes you are right and thanks to Tom Cosgrove that reminded me of that 
over look part in your previous answer, I did this again with:


rsync -e ssh -aSuqz --delete /var/www/sites/ [EMAIL PROTECTED]:/var/www/sites

instead of:

rsync -e ssh -auqz --delete /var/www/sites/ [EMAIL PROTECTED]:/var/www/sites

and it shrink to:

/dev/wd1a 19.7G3.9G   14.8G21%/var/www/sites

from:

/dev/wd1a 19.7G8.0G   10.7G43%/var/www/sites

Thanks for your patience with me!

That's a lots of waisted space for sure.

I am learning... sometime slow, but I am.

My next step is now trying to see where that's from.

Thanks

Daniel

df -h stats for same file systems display different result son AMD64 then on i386

[Daniel Ouellet]

Here is something I can't put my hands around to well and I don't really 
understand why that is, other then may be the fize of each mount point 
not process properly on AMD64, but that's just an idea. See lower below 
for why I think it might be the case. In any case, I would welcome a 
logical explication why that might be however.


I mirror a mount point for three servers, one AMD64 and two i386. Then I 
do df -h for each one, but I get way different results when I do it on 
AMD64, or when I do it on i386, but I can't understand why.


When I do the df -i however, I do get the same amount of inode, so there 
is the same amount of files. I even use rsync to make a perfect mirror 
of them and still I get way different results.


AMD64 give me 4.6GB as the i386 gives me 8.1GB. The funny part is that 
the AMD64 should give me more as the file system include a bit more stuff


AMD64 mount point file system is for /var/www as the mirror one is for 
/var/www/sites and the amd does include all of sites files.


However is I log in with WinSCP and do the calculate stuff on both 
server to the location


/var/www/sites, I do get the same results.

dev.
52584 files, 2799 folders
location /var/www
7,685 MB (8,059,054,473)

www2
52584 files, 2799 folders
location /var/www
7,683 MB (8,056,394,923)

The difference in size is the logs files that are process not in sync of 
each others, but locally on each one.


I can't explain this one.

This is really weird.

I thought to delete the file system and recreate it with the additional 
mount to to see, but the results should be good as it is now as the 
/var/www/sites is inside the /var/www one on the AMD64.


i386 display:
# df -h
Filesystem SizeUsed   Avail Capacity  Mounted on
/dev/wd0a  247M   27.8M206M12%/
/dev/wd0h  4.6G3.2M4.3G 0%/home
/dev/wd0d  495M1.0K470M 0%/tmp
/dev/wd0g  4.4G206M3.9G 5%/usr
/dev/wd0e 12.6G745M   11.2G 6%/var
/dev/wd1b  7.8G2.0K7.4G 0%/var/mysql
/dev/wd0f  991M1.1M940M 0%/var/qmail
/dev/wd1a 19.7G8.1G   10.6G43%/var/www/sites


AMD64 display:
www1# df -h
Filesystem SizeUsed   Avail Capacity  Mounted on
/dev/wd0a  251M   40.5M198M17%/
/dev/wd0h 1024M   54.0K972M 0%/home
/dev/wd0d 1006M2.0K956M 0%/tmp
/dev/wd0g  4.9G304M4.4G 6%/usr
/dev/wd0e 24.6G4.6G   18.7G20%/var
/dev/wd0f 1006M1.5M955M 0%/var/qmail


I also thought about files still open, but I rebooted the system to be 
safe and still the same results.


May be the disklabel is not seen right, or calculate right on AMD64. I 
am not sure I understand this right, but if the file system use fsize of 
 2048 on AMD64 and display almost 1/2 the size of the i386 that use 
fsize of 1024, may be that's just the part of the fsize that is missing 
in the calculation.


So, far I couldn't come up with a different explication.

www1# disklabel wd0
# Inside MBR partition 3: type A6 start 63 size 78156162
# /dev/rwd0c:
type: ESDI
disk: ESDI/IDE disk
label: Maxtor 6E040L0
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 16
sectors/cylinder: 1008
cylinders: 16383
total sectors: 78165360
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0   # microseconds
track-to-track seek: 0  # microseconds
drivedata: 0

16 partitions:
# sizeoffset  fstype [fsize bsize  cpg]
  a:52409763  4.2BSD   2048 16384  328 # Cyl 
0*-   519
  b:   8388576524160swap   # Cyl   520 
-  8841
  c:  78165360 0  unused  0 0  # Cyl 0 
- 77544
  d:   2097648   8912736  4.2BSD   2048 16384  328 # Cyl  8842 
- 10922
  e:  52429104  11010384  4.2BSD   2048 16384  328 # Cyl 10923 
- 62935
  f:   2097648  63439488  4.2BSD   2048 16384  328 # Cyl 62936 
- 65016
  g:  10486224  65537136  4.2BSD   2048 16384  328 # Cyl 65017 
- 75419
  h:   2132865  76023360  4.2BSD   2048 16384  328 # Cyl 75420 
- 77535*



oppose to i386:
# disklabel wd0
# Inside MBR partition 3: type A6 start 63 size 58621122
# /dev/rwd0c:
type: ESDI
disk: ESDI/IDE disk
label: QUANTUM FIREBALL
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 16
sectors/cylinder: 1008
cylinders: 16383
total sectors: 58633344
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0   # microseconds
track-to-track seek: 0  # microseconds
drivedata: 0

16 partitions:
# sizeoffset  fstype [fsize bsize  cpg]
  a:52409763  4.2BSD   1024  8192   86 # Cyl 
0*-   519
  b:   8388576524160swap   # Cyl   520 
-  8841
  c:  58633344 0  unused  0 0  # Cyl 0 
- 58167
  d:   1048320   8912736  4.2BSD   1024  8192   86 # Cyl 

Re: df -h stats for same file systems display different result son AMD64 then on i386

[Otto Moerbeek]

On Sun, 15 Jan 2006, Daniel Ouellet wrote:

[snip lots of talk by a confused person]

 16 partitions:
 # sizeoffset  fstype [fsize bsize  cpg]
   a:52409763  4.2BSD   2048 16384  328 # Cyl 0*-   519
   b:   8388576524160swap   # Cyl   520 -  8841
   c:  78165360 0  unused  0 0  # Cyl 0 - 77544
   d:   2097648   8912736  4.2BSD   2048 16384  328 # Cyl  8842 - 10922
   e:  52429104  11010384  4.2BSD   2048 16384  328 # Cyl 10923 - 62935
   f:   2097648  63439488  4.2BSD   2048 16384  328 # Cyl 62936 - 65016
   g:  10486224  65537136  4.2BSD   2048 16384  328 # Cyl 65017 - 75419
   h:   2132865  76023360  4.2BSD   2048 16384  328 # Cyl 75420 -
 77535*

 16 partitions:
 # sizeoffset  fstype [fsize bsize  cpg]
   a:52409763  4.2BSD   1024  8192   86 # Cyl 0*-   519
   b:   8388576524160swap   # Cyl   520 -  8841
   c:  58633344 0  unused  0 0  # Cyl 0 - 58167
   d:   1048320   8912736  4.2BSD   1024  8192   86 # Cyl  8842 -  9881
   e:  27263376   9961056  4.2BSD   1024  8192   86 # Cyl  9882 - 36928
   f:   2097648  37224432  4.2BSD   1024  8192   86 # Cyl 36929 - 39009
   g:   9436896  39322080  4.2BSD   1024  8192   86 # Cyl 39010 - 48371
   h:   9874368  48758976  4.2BSD   1024  8192   86 # Cyl 48372 - 58167

Since the bsize and fsize differ, it is expected that the used kbytes of the
file systems differ. Also, the inode table size will not be the same.

You're comparing apples and oranges.

BTW, you don't say which version(s) you are running. That's bad. since
some bugs were fixed in the -h display. Run df without -h to see the
real numbers.

To check if the inode/block/fragment free numbers add up, you could
use dumpfs, but that is a hell of a lot of work. 

-Otto

Re: df -h stats for same file systems display different result son AMD64 then on i386

[Daniel Ouellet]

Otto Moerbeek wrote:

On Sun, 15 Jan 2006, Daniel Ouellet wrote:

[snip lots of talk by a confused person]


16 partitions:
# sizeoffset  fstype [fsize bsize  cpg]
  a:52409763  4.2BSD   2048 16384  328 # Cyl 0*-   519
  b:   8388576524160swap   # Cyl   520 -  8841
  c:  78165360 0  unused  0 0  # Cyl 0 - 77544
  d:   2097648   8912736  4.2BSD   2048 16384  328 # Cyl  8842 - 10922
  e:  52429104  11010384  4.2BSD   2048 16384  328 # Cyl 10923 - 62935
  f:   2097648  63439488  4.2BSD   2048 16384  328 # Cyl 62936 - 65016
  g:  10486224  65537136  4.2BSD   2048 16384  328 # Cyl 65017 - 75419
  h:   2132865  76023360  4.2BSD   2048 16384  328 # Cyl 75420 -
77535*



16 partitions:
# sizeoffset  fstype [fsize bsize  cpg]
  a:52409763  4.2BSD   1024  8192   86 # Cyl 0*-   519
  b:   8388576524160swap   # Cyl   520 -  8841
  c:  58633344 0  unused  0 0  # Cyl 0 - 58167
  d:   1048320   8912736  4.2BSD   1024  8192   86 # Cyl  8842 -  9881
  e:  27263376   9961056  4.2BSD   1024  8192   86 # Cyl  9882 - 36928
  f:   2097648  37224432  4.2BSD   1024  8192   86 # Cyl 36929 - 39009
  g:   9436896  39322080  4.2BSD   1024  8192   86 # Cyl 39010 - 48371
  h:   9874368  48758976  4.2BSD   1024  8192   86 # Cyl 48372 - 58167


Since the bsize and fsize differ, it is expected that the used kbytes of the
file systems differ. Also, the inode table size will not be the same.


Not sure that I would agree fully with that, but I differ to your 
judgment. Yes there will and should be difference in usage as if you 
have a lots of small files, you are waisting more space if you fsize are 
bigger, unless I don't understand that part. Would it mean that the df 
-h would take the number of inode in use * the fsize to display the 
results for human then?



You're comparing apples and oranges.


I don't disagree to some extend as you know better, but I still try to 
understand it however. Shouldn't the df -h display the same results 
however to human? I am not arguing, but rather try to understand it. If 
it is design to be human converted, why a human would need to know or 
consider the file size in use then to compare the results?



BTW, you don't say which version(s) you are running. That's bad. since
some bugs were fixed in the -h display. Run df without -h to see the
real numbers.


All run 3.8. Sorry about that.

the 4.6GB have 4870062 * 1024 = 4,986,943,488
www1# df
Filesystem  1K-blocks  Used Avail Capacity  Mounted on
/dev/wd0a  256814 4146420251017%/
/dev/wd0h 104815854995698 0%/home
/dev/wd0d 1030550 2979022 0%/tmp
/dev/wd0g 5159638310910   4590748 6%/usr
/dev/wd0e25799860   4870062  1963980620%/var
/dev/wd0f 1030550  1546977478 0%/var/qmail


the 8.1GB have 15967148 * 512 = 8,175,179,776
# df
Filesystem  512-blocks  Used Avail Capacity  Mounted on
/dev/wd0a   513628 6558842236013%/
/dev/wd0h  186162852   1768496 0%/home
/dev/wd0d  2061100 4   1958044 0%/tmp
/dev/wd0g  9904156424544   8984408 5%/usr
/dev/wd0e 33022236   1537612  29833516 5%/var
/dev/wd1b 16412252   1937920  1365372012%/var/mysql
/dev/wd0f  2061100 4   1958044 0%/var/qmail
/dev/wd1a 41280348  15967148  2324918441%/var/www/sites

The funny part is that the first above /var include more files then the 
/var/www/sites below and still display less space in use.



To check if the inode/block/fragment free numbers add up, you could
use dumpfs, but that is a hell of a lot of work. 


-Otto



It's not a huge deal and the systems works well, I am just puzzle by the 
results and want to understand it, that's all.