Re: Any way to "passive commit" to RAM without syncing ?

[Hal Murray]

swiftgri...@gmail.com said:
> Let's say I've got a machine with a boatload of RAM and I have an
> application which wants extremely fast writes, but doesn't really care  that
> much about data integrity, and this case, nor do I. Having the system  come
> back in the same state after rebooting is "nice to have" in this  thought
> experiment. Soo, here is the question:
...
> In other words, operate out of a RAM disk until my human operator gives  the
> go-ahead to flush out the writes to disk or the RAM disk gets full. I  think
> Sprite did something like this back in the Jurassic Period. I also  remember
> CacheFS for Solaris doing something similar for NFS.

I had a setup for a while to keep working log files in RAM.  I have forgotten 
the details.

The general idea was that the working log files were in RAM, old ones were on 
disk.  A script would rsync things from RAM to disk and delete stuff more than 
a few days old.  It was called by cron, the script that stopped the server, or 
by hand.  At boot time, another script would preload the RAM from disk so 
appending kept going at the right place..  If the system crashed, you lost the 
stuff in RAM that hadn't been backed up yet.

Overall, it was simple enough so that I (far from a wizard) could set it up 
without a lot of effort.

You might be able to do something similar for any specific application.  Please 
let us know if you come up with a good solution.


-- 
These are my opinions.  I hate spam.

Re: Any way to "passive commit" to RAM without syncing ?

[Robert Elz]

Sounds like you want a tmpfs, possibly union mounted over the
"stable" filesystem, with "rsync" being your :sync" command.

kre

Re: Any way to "passive commit" to RAM without syncing ?

[Malcolm Herbert]

Sounds like you're looking for a combo of overlay mounts on top of tmpfs
- the flush to disk you'd have to manage manually but I've done this
with some success in the past and I do seem to recall it being discussed
to provide pseudo read-write over a read-only medium such as a CDROM

http://netbsd.gw.com/cgi-bin/man-cgi?mount_overlay+8+NetBSD-current

the only caution is that I found it tricky to grok the sense of which
filesystem gets overlaid on what and where writes end up, but some
workshopping should show that up pretty quickly

Regards,
Malcolm

On Wed, Mar 9, 2016, at 10:49, Swift Griggs wrote:
> 
> I know NetBSD supports RAM disks. I also know it uses them for 
> installation, but I'm not sure if they are overlays on top of the static 
> disk images or if they get loaded/populated with the image after the 
> system boots. The effect is the same, either way. You end up with a 
> writable file system in RAM which is predicated on a disk image. I was 
> thinking about this in the context of another problem.
> 
> Let's say I've got a machine with a boatload of RAM and I have an 
> application which wants extremely fast writes, but doesn't really care 
> that much about data integrity, and this case, nor do I. Having the
> system 
> come back in the same state after rebooting is "nice to have" in this 
> thought experiment. Soo, here is the question:
> 
> Since a RAM disk is probably the fastest option, is there a way to 
> basically leave uncommitted data until I run "sync" or something calls 
> fsync() ? Even better, is there a way to fake a sync until I run
> something 
> else ?
> 
> In other words, operate out of a RAM disk until my human operator gives 
> the go-ahead to flush out the writes to disk or the RAM disk gets full. I 
> think Sprite did something like this back in the Jurassic Period. I also 
> remember CacheFS for Solaris doing something similar for NFS.
> 
> I can think of a few applications:
> 
> - Internet cafe style browsing where the machines are using older 
> hardware. They might benefit from using a small RAM disk overlay rather 
> than having to use something like a PATA disks for it's cache.
> 
> - Editing high bandwidth data like video or audio. Most of the time my 
> editing sessions don't last more than an hour and once I get a final cut
> I 
> just save the project to a server and don't care much about the recording 
> or editing workstation.
> 
> - Any kind of temporary storage that needs RAM disk speeds, but still 
> needs an option to sync/flush the file system back to long term storage.
> 
> I'm guessing this would be a little tricky with something like UFS since
> a 
> process could alter/write to a file and it'd need to reflect that when 
> another process wanted to do the same, fseek(), or dozens of other 
> operations. However, I just wonder if there is already a way to do this 
> and I'm just not aware of it.
> 
> I wonder if the LiveCD's for Linux that let you flush you changes to USB 
> do something similar. I doubt it. I'm guessing they just load a RAM disk 
> image and you operate on that until you reboot and it's lost.
> 
> -Swift


-- 
Malcolm Herbert
m...@mjch.net

Any way to "passive commit" to RAM without syncing ?

[Swift Griggs]

I know NetBSD supports RAM disks. I also know it uses them for 
installation, but I'm not sure if they are overlays on top of the static 
disk images or if they get loaded/populated with the image after the 
system boots. The effect is the same, either way. You end up with a 
writable file system in RAM which is predicated on a disk image. I was 
thinking about this in the context of another problem.


Let's say I've got a machine with a boatload of RAM and I have an 
application which wants extremely fast writes, but doesn't really care 
that much about data integrity, and this case, nor do I. Having the system 
come back in the same state after rebooting is "nice to have" in this 
thought experiment. Soo, here is the question:


Since a RAM disk is probably the fastest option, is there a way to 
basically leave uncommitted data until I run "sync" or something calls 
fsync() ? Even better, is there a way to fake a sync until I run something 
else ?


In other words, operate out of a RAM disk until my human operator gives 
the go-ahead to flush out the writes to disk or the RAM disk gets full. I 
think Sprite did something like this back in the Jurassic Period. I also 
remember CacheFS for Solaris doing something similar for NFS.


I can think of a few applications:

- Internet cafe style browsing where the machines are using older 
hardware. They might benefit from using a small RAM disk overlay rather 
than having to use something like a PATA disks for it's cache.


- Editing high bandwidth data like video or audio. Most of the time my 
editing sessions don't last more than an hour and once I get a final cut I 
just save the project to a server and don't care much about the recording 
or editing workstation.


- Any kind of temporary storage that needs RAM disk speeds, but still 
needs an option to sync/flush the file system back to long term storage.


I'm guessing this would be a little tricky with something like UFS since a 
process could alter/write to a file and it'd need to reflect that when 
another process wanted to do the same, fseek(), or dozens of other 
operations. However, I just wonder if there is already a way to do this 
and I'm just not aware of it.


I wonder if the LiveCD's for Linux that let you flush you changes to USB 
do something similar. I doubt it. I'm guessing they just load a RAM disk 
image and you operate on that until you reboot and it's lost.


-Swift

Re: SSD TRIM / "discard" works after a remount with mount -a?

[Benny Siegert]

> I have a scenario where I have several NetBSD systems which were just 
> upgraded to SSDs. I forgot to turn on discard (TRIM) support on the drives 
> when I first put them in. 

I don’t think that’s a big deal. For the record, I have run my SSD without TRIM 
on NetBSD for a long while now. In fact, your email taught me about the 
„discard“ option :¬


> 1. Edit /etc/fstab, and add the "discard" option to all FFS file systems

OK.

> 2. Run "mount -a" which should re-mount the affected file systems  with
>   the newly added option. I'm assuming this work for the trim/discard
>   opton.

Reboot?

Otherwise, to enable TRIM without rebooting, try:

mount -uo discard /

> 3. dd if=/dev/zero bs=4096k of=/my/affected/file/system/DELETEME.000
>   I'm assuming short blocks get written as partials.

I would skip this step and the following ones. They likely do more harm than 
good.

—Benny.

Re: GPT, wedges and RAID-1 on new server with NetBSD 7

[Greg Troxel]

Gerard Lally  writes:

> If I want separate / swap and /home, I presume I should delete raid1a
> and create raid1a, raid1b and raid1e instead (by clicking on its
> parent raid1 and selecting "Edit BSD partitions")?

Separately from how to use sysinst for this (which I've never done), I
think it is sensible to have partitions within a raid.   I typically
have wd0a/wd1a as type raid, being mostly the whole disk, and then
within raid0 have a/b/e/f.

> I create these and proceed with installation as normal, selecting
> raid1 as the available disk on which I want to install NetBSD. But
> each time I do this I get the dreaded error "FATAL: No bootable medium
> found! System halted."
>
> Where am I going wrong? One thing I note is that I am not asked at any
> point to install bootcode to the disks as I would be with non-RAID
> setups.

Probably you can boot to utility and run installboot manually on wd0a/wd1a.

> If it's not possible to do this with sysinst is it at least possible
> to do it by dropping to a shell?

When I want to set up a new raid system, I tend to get a bootable disk
with a minimal system and boot that and do the whole disk setup
including bootblocks by hand.  But I suspect you are just missing
bootblocks.

> Ideally I would like to use GPT with the RAID-1 setup as well, since I
> will be on 2 x 2TB disks and I anticipate this getting bigger, not
> smaller in years to come. I have successfully set up NetBSD with GPT
> by dropping to a shell but I don't know where to add RAID into that
> mix.

As others commented it seems disklabel-in-raid-in-gpt works.  So that
leads to having two raid sets.  One is small enoguh to fit in 2T, and
would have root, swap, /var, /usr sorts of things.  The other would be
just bare raid in gpt, and have a filesystem in raid0d.  or maybe gpt
inside raid.  The point is that the >2T raid doesn't have a disklabel
(because it's too big) and doesn't have root (because the bootblocks
can't yet find it).


signature.asc
Description: PGP signature

Re: Reformatting little USB-harddisks

[Swift Griggs]

On Tue, 8 Mar 2016, Andrew Cagney wrote:
One got ya!  They can be slow to come online and may not be ready if 
you're trying to use or mount it during boot.


This is especially slow for some reason on WD drives with SES. I have a 
2TB WD USB3 drive for backups that has this issue. It's fine otherwise. 
NetBSD used to be a bit flaky with the SES operations (a long time ago) 
but it's great these days. My only wish was that it ran at USB3 speeds, 
but most USB3 drives can't exceed the USB2 spec (or not except for 
spikes). So, it's not a big deal to me.


-Swift

Re: SSD TRIM / "discard" works after a remount with mount -a?

[Swift Griggs]

On Tue, 8 Mar 2016, Benny Siegert wrote:
I don?t think that?s a big deal. For the record, I have run my SSD 
without TRIM on NetBSD for a long while now. In fact, your email taught 
me about the ?discard? option :?


Well, I have also on other systems. I've also read that some SSDs don't 
actually need TRIM at all. I use Samsung 850 Pros.



Otherwise, to enable TRIM without rebooting, try:
mount -uo discard /


I think that the mount -a does the same thing. What I did to test was to 
try it on one system and after editing the /etc/fstab and running mount -a 
I see that "discard" is listed in the "mount" output. However, I'm a 
little suspicous that it's not actually on. Paranoia.



3. dd if=/dev/zero bs=4096k of=/my/affected/file/system/DELETEME.000
  I'm assuming short blocks get written as partials.
I would skip this step and the following ones. They likely do more harm 
than good.


I'd like to know where you are coming from on that. I get that SSDs have 
to be wear leveled, but I'm wondering what's going to happen the next time 
that I write to a block that was deleted but not properly trim'd. Wouldn't 
that cause a pause / blocking while the TRIM operation had to complete? 
Maybe I have a jacked up understanding of the dynamics, but hey, that's 
why I asked :-)


Thanks for the feedback.

-Swift

Re: Reformatting little USB-harddisks

[Andrew Cagney]

On 8 March 2016 at 11:13, herbert langhans  wrote:
> Hi List,
> you sure know these little external USB-harddisks, often used for laptops
> or basic backups. Like WD-Passport and Seagate Expansion and whatever
> they name them.
>
> They come FAT formatted, right? Has anyone of you tried to
> reformat them for NetBSD? I guess its not a big thing, but I wonder if I
> can buy any brand (need it for portable server backup and need GID/UID) or
> there are some models what make trouble with formatting, partitioning or
> mounting.

One got ya!  They can be slow to come online and may not be ready if
you're trying to use or mount it during boot.

> I had some trouble with USB-Sticks. Thats another story, but close
> enough the be alert just buying any plug n play solution what may not
> work later.
>
> Any experience?
> Thanks!
> herb langhans
>

Re: Reformatting little USB-harddisks

[herbert langhans]

Most of these that I've used work just fine. You end up mounting them
as /dev/sdXe and use mount_msdos. /dev/sdXe is just a made up
disklabel that the machine makes to temporarily deal with it.

Some come formatted with ExFAT now which is not compatible with
NetBSD, and might never be as far as I know.

You can wipe them and use fdisk and disklabel to turn them into "real"
NetBSD disks without any problem. There might be a few that cause
issues but most work in my experience.
Andy

Ok, thats fairly easy - fdisk and disklabel. Will be a 1T harddisk. Just
for keeping backups in a separate location.

Not the fastest solution I guess. Rsync will take care of it - and I still 
have the slower USB2 on my server...

Cheers
herb langhans

Re: Reformatting little USB-harddisks

[herbert langhans]

From: Travis Paul  [160308 18:36]

> They come FAT formatted, right? Has anyone of you tried to
> reformat them for NetBSD? I guess its not a big thing, but I wonder if I
> can buy any brand (need it for portable server backup and need GID/UID) 
or 
> there are some models what make trouble with formatting, partitioning or 
> mounting.
> 


I have used several brands, WD, Seagate, Kingston and they have all worked 
with NetBSD.
I've even used a Kingston SATA to USB converter with an SanDisk SSD and had 
no issues.

Best of luck,
Travis

Thats great, thanks for the hint, Travis. I will fetch a WD or Seagate.
Will miss my tape streamer though, it has style...
   
Cheers
herb langhans
 

Re: Reformatting little USB-harddisks

[Greg Troxel]

Andy Ruhl  writes:

> On Tue, Mar 8, 2016 at 9:13 AM, herbert langhans  wrote:
>> Hi List,
>> you sure know these little external USB-harddisks, often used for laptops
>> or basic backups. Like WD-Passport and Seagate Expansion and whatever
>> they name them.
>>
>> They come FAT formatted, right? Has anyone of you tried to
>> reformat them for NetBSD? I guess its not a big thing, but I wonder if I
>> can buy any brand (need it for portable server backup and need GID/UID) or
>> there are some models what make trouble with formatting, partitioning or
>> mounting.
>>
>> I had some trouble with USB-Sticks. Thats another story, but close
>> enough the be alert just buying any plug n play solution what may not
>> work later.
>>
>
> Most of these that I've used work just fine. You end up mounting them
> as /dev/sdXe and use mount_msdos. /dev/sdXe is just a made up
> disklabel that the machine makes to temporarily deal with it.
>
> Some come formatted with ExFAT now which is not compatible with
> NetBSD, and might never be as far as I know.
>
> You can wipe them and use fdisk and disklabel to turn them into "real"
> NetBSD disks without any problem. There might be a few that cause
> issues but most work in my experience.

Agreed.

The only issue is disks with 4K sectors may need newer NetBSD (7) to
work right.   But 512-byte sectors should have no issues.

I have seen a 3T disk (WD elements) with 4K sectors that I can't get
NetBSD 7 to recognize.  But I think that's an odd disk, not a general
issue.  And it's not about the filesystem; it's about the disk itself.


signature.asc
Description: PGP signature

Re: Reformatting little USB-harddisks

[Andy Ruhl]

On Tue, Mar 8, 2016 at 9:13 AM, herbert langhans  wrote:
> Hi List,
> you sure know these little external USB-harddisks, often used for laptops
> or basic backups. Like WD-Passport and Seagate Expansion and whatever
> they name them.
>
> They come FAT formatted, right? Has anyone of you tried to
> reformat them for NetBSD? I guess its not a big thing, but I wonder if I
> can buy any brand (need it for portable server backup and need GID/UID) or
> there are some models what make trouble with formatting, partitioning or
> mounting.
>
> I had some trouble with USB-Sticks. Thats another story, but close
> enough the be alert just buying any plug n play solution what may not
> work later.
>

Most of these that I've used work just fine. You end up mounting them
as /dev/sdXe and use mount_msdos. /dev/sdXe is just a made up
disklabel that the machine makes to temporarily deal with it.

Some come formatted with ExFAT now which is not compatible with
NetBSD, and might never be as far as I know.

You can wipe them and use fdisk and disklabel to turn them into "real"
NetBSD disks without any problem. There might be a few that cause
issues but most work in my experience.

Andy

Reformatting little USB-harddisks

[herbert langhans]

Hi List,
you sure know these little external USB-harddisks, often used for laptops
or basic backups. Like WD-Passport and Seagate Expansion and whatever
they name them.

They come FAT formatted, right? Has anyone of you tried to
reformat them for NetBSD? I guess its not a big thing, but I wonder if I
can buy any brand (need it for portable server backup and need GID/UID) or 
there are some models what make trouble with formatting, partitioning or 
mounting.

I had some trouble with USB-Sticks. Thats another story, but close
enough the be alert just buying any plug n play solution what may not
work later.

Any experience?
Thanks!
herb langhans

Re: Simple IPSEC client with certificate - phase 1 time out

[Christos Zoulas]

On Mar 8,  2:26pm, fr...@phoenix.owl.de (Frank Wille) wrote:
-- Subject: Re: Simple IPSEC client with certificate - phase 1 time out

| http://www.netbsd.org/docs/network/ipsec/
| In section "Configuring IPsec kernel".
| 
| It also mentions IPSEC_ESP, which was removed together with IPSEC_NAT_T
| after netbsd-6.

Fixed.

christos

Re: Simple IPSEC client with certificate - phase 1 time out

[Frank Wille]

Greg Troxel wrote:

> It seems to me that if a kernel with "option IPSEC" and w/o swcrypto
> doesn't work, then perhaps it should fail to config, or log an error at
> runtime.  (Perhaps swcrypto isn't required, and it's just that there
> must be some crypto provider.)

As far as I understand swcrypto registers software encryption algorithms in
the kernel for 3des, aes, blowfish, etc.. They are not explicitely required
as you may also use hardware for that.

-- 
Frank Wille

Re: Simple IPSEC client with certificate - phase 1 time out

[Greg Troxel]

Frank Wille  writes:

> BTW, my problem with setkey on macppc was caused by the missing swcrypto
> pseudo device in the kernel.
>
> Our IPsec FAQ should mention that you need that, besides "option IPSEC". I
> know that amd64, i386 and sparc64 have these enabled by default now, but no
> other port has.

It seems to me that if a kernel with "option IPSEC" and w/o swcrypto
doesn't work, then perhaps it should fail to config, or log an error at
runtime.  (Perhaps swcrypto isn't required, and it's just that there
must be some crypto provider.)


signature.asc
Description: PGP signature

Re: Simple IPSEC client with certificate - phase 1 time out

[Christos Zoulas]

On Mar 8, 12:14pm, fr...@phoenix.owl.de (Frank Wille) wrote:
-- Subject: Re: Simple IPSEC client with certificate - phase 1 time out

| Christos Zoulas wrote:
| 
| >>| > If your server is behind NAT, I think that got broken at some point.
| >>| 
| >>| Oh no! :(
| >>
| >>Yes, it is almost working... The tunnel is up, and 3 out of 4 SAD's are
| >>present; the 4th one comes up as larval and then times out... 
| >
| >And it is now fixed and tested on little endian. I have done no testing
| >on big endian. I guess I could boot my sparc64 box and see if the extended
| >rest made the hardware more reliable :-)
| 
| Indeed. It is! Many thanks for your great work! Much appreciated. :)

Great!

| IPsec with Racoon behind NAT is confirmed to work now. Tested on macppc, so
| there is no endian problem.
| 
| Do we get a pullup for netbsd-7, and maybe netbsd-6?

I asked for them just now.

| BTW, my problem with setkey on macppc was caused by the missing swcrypto
| pseudo device in the kernel.
| 
| Our IPsec FAQ should mention that you need that, besides "option IPSEC". I
| know that amd64, i386 and sparc64 have these enabled by default now, but no
| other port has.

URL?

christos