ZFS ARC and mmap/page cache coherency question

[Paul Koch]

We are trying to understand a performance issue when syncing large mmap'ed
files on ZFS.

Example test box setup:
 FreeBSD 10.3-p5
 Intel i7-5820K 3.30GHz with 64G RAM
 6 * 2 Tbyte Seagate ST2000DM001-1ER164 in a ZFS stripe

Read performance of a sequentially written large file on the pool is
typically around 950Mbytes/sec using dd.

Our software mmap's some large database files using MAP_NOSYNC, and we call
fsync() every 10 minutes when we know the file system is mostly idle.  In
our test setup, the database files are 1.1G, 2G, 1.4G, 12G, 4.7G and ~20
small files (under 10M). Most of the memory pages in the mmap'ed files are
updated every minute with massive amounts of collected ping and snmp mib
values.  

When the 10 minute fsync() occurs, gstat typically shows very little disk
reads and very high write speeds, which is what we expect.  But, every 80
minutes we process the data in the large mmap'ed files and store it in highly
compressed blocks of a ~300G file using pread/pwrite (i.e. not mmap'ed).
After that, the performance of the next fsync() of the mmap'ed files falls
off a cliff.  We are assuming it is because the ARC has thrown away the
cached data of the mmap'ed files.  gstat shows lots of read/write contention
and lots of things tend to stall waiting for disk.

Is this just a lack of ZFS ARC and page cache coherency ??

Is there a way to prime the ARC with the mmap'ed files again before we call
fsync() ?

We've tried cat and read() on the mmap'ed files but doesn't seem to touch the
disk at all and the fsync() performance is still poor, so it looks like the
ARC is not being filled.  msync() doesn't seem to be much different.
mincore() stats show the mmap'ed data is entirely incore and referenced.

Paul.
-- 
Paul Koch | Founder | CEO
AKIPS Network Monitor | akips.com
Brisbane, Australia
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: unbound and ntp issuse

[David Wolfskill]

On Tue, Jun 14, 2016 at 07:55:34AM -0700, Chris H wrote:
> I'm playing catchup on my INBOX, so apologies in advance, if this has
> already been satisfactorily answered...
> On Mon, 6 Jun 2016 16:50:18 +0300 Slawa Olhovchenkov  wrote
> ...
> > What I am missing?
> > Need to fix unbound setup scripts? bsdinstall scripts?
> > As I see unbound setup scripts detects 127.0.0.1 in resolv.conf and
> > configured unbound as fully recursive DNS server.
> May I suggest ntpdate(8)?
> Find a reliable time server in your region, and once found add it
> *early* in your rc.conf(5). Well, ahead of your unbound stanza. ie;
> 
> hostname="..."
> ifconfig_re0="inet ... netmask ..."
> defaultrouter="..."
> 

Errr...  The sequence of assignments in /etc/rc.conf* has no bearing on
the sequence in which /etc/rc starts things.

Please refer to rcorder(8) for further information on this.

Peace,
david
-- 
David H. Wolfskill  da...@catwhisker.org
Those who would murder in the name of God or prophet are blasphemous cowards.

See http://www.catwhisker.org/~david/publickey.gpg for my public key.


signature.asc
Description: PGP signature

Re: unbound and ntp issuse

[Chris H]

I'm playing catchup on my INBOX, so apologies in advance, if this has
already been satisfactorily answered...
On Mon, 6 Jun 2016 16:50:18 +0300 Slawa Olhovchenkov  wrote

> On Mon, Jun 06, 2016 at 09:33:02AM -0400, Lowell Gilbert wrote:
> 
> > Slawa Olhovchenkov  writes:
> > 
> > > On Fri, Jun 03, 2016 at 02:34:18PM -0400, Lowell Gilbert wrote:
> > >
> > >> Slawa Olhovchenkov  writes:
> > >> 
> > >> > Default install with local_unbound and ntpd can't be functional with
> > >> > incorrect date/time in BIOS:
> > >> >
> > >> > Unbound requred correct time for DNSSEC check and refuseing queries
> > >> > ("Jul 1 20:17:29 yellowrat unbound: [3444:0] info: failed to prime
> > >> > trust anchor -- DNSKEY rrset is not secure . DNSKEY IN")
> > >> >
> > >> > ntpd don't have any numeric IP of ntp servers in ntp.conf -- only
> > >> > symbolic names like 0.freebsd.pool.ntp.org, as result -- can't
> > >> > resolve (see above, about DNSKEY).
> > >> 
> > >> I can't see how this would happen. DNSSEC doesn't seem to be required in
> > >> a regular install as far as I can see. Certainly I don't have any
> > >
> > > I don't know reasson for enforcing DNSSEC in regular install.
> > > I am just select 'local_unbound' at setup time and enter '127.0.0.1' as
> > > nameserver address.
> > 
> > That's not enough to configure unbound as a fully recursive DNS
> > server.
> 
> What I am missing?
> Need to fix unbound setup scripts? bsdinstall scripts?
> As I see unbound setup scripts detects 127.0.0.1 in resolv.conf and
> configured unbound as fully recursive DNS server.
May I suggest ntpdate(8)?
Find a reliable time server in your region, and once found add it
*early* in your rc.conf(5). Well, ahead of your unbound stanza. ie;

hostname="..."
ifconfig_re0="inet ... netmask ..."
defaultrouter="..."
ntpdate_enable="YES"
ntpdate_hosts="a reliable regional time server"

..

unbound_enable="YES"
..

ALSO. Since you're upstream will, in all likelihood have informed
you of a preferred set of 2 name servers. Place one of them in your
hosts(5) file. This will help ensure that ntpdate(8) can reliably
discover your regional time server.

That should get you where you want to go. :-)

--Chris
> 
> > If your system gets its address through DHCP, it is probably
> > getting DNS server addresses as well, and would work fine *without* your
> > configuring any of the DNS state.
> 
> I am have static address and don't getting DNS server address.
> 
> > >> problem on any of my systems, and I've never configured an anchor on the
> > >> internal systems.
> > >> 
> > >> > IMHO, ntp.conf need to include some numeric IP of public ntp servers.
> > >> 
> > >> Ouch; that's a terrible idea, for several different reasons.
> > >
> > > What else?
> > 
> > All the normal reasons that hard-coding IP addresses is a bad idea; they
> > can change, you're encouraging a lot of people to use the same ones, etc.
> 
> And how to resolve this issuse:
> 
> - default install with unbound as recursive DNS server (by default
>   enforcing DNSSEC)
> - ntp time synchronisation
> - stale CMOS time (2008 year)


___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: unbound and ntp issuse

[Slawa Olhovchenkov]

On Tue, Jun 14, 2016 at 07:55:34AM -0700, Chris H wrote:

> I'm playing catchup on my INBOX, so apologies in advance, if this has
> already been satisfactorily answered...

Main question not about how I am can resolve my current issuse.
Main question about deadloop after setup.

> On Mon, 6 Jun 2016 16:50:18 +0300 Slawa Olhovchenkov  wrote
> 
> > On Mon, Jun 06, 2016 at 09:33:02AM -0400, Lowell Gilbert wrote:
> > 
> > > Slawa Olhovchenkov  writes:
> > > 
> > > > On Fri, Jun 03, 2016 at 02:34:18PM -0400, Lowell Gilbert wrote:
> > > >
> > > >> Slawa Olhovchenkov  writes:
> > > >> 
> > > >> > Default install with local_unbound and ntpd can't be functional with
> > > >> > incorrect date/time in BIOS:
> > > >> >
> > > >> > Unbound requred correct time for DNSSEC check and refuseing queries
> > > >> > ("Jul 1 20:17:29 yellowrat unbound: [3444:0] info: failed to prime
> > > >> > trust anchor -- DNSKEY rrset is not secure . DNSKEY IN")
> > > >> >
> > > >> > ntpd don't have any numeric IP of ntp servers in ntp.conf -- only
> > > >> > symbolic names like 0.freebsd.pool.ntp.org, as result -- can't
> > > >> > resolve (see above, about DNSKEY).
> > > >> 
> > > >> I can't see how this would happen. DNSSEC doesn't seem to be required 
> > > >> in
> > > >> a regular install as far as I can see. Certainly I don't have any
> > > >
> > > > I don't know reasson for enforcing DNSSEC in regular install.
> > > > I am just select 'local_unbound' at setup time and enter '127.0.0.1' as
> > > > nameserver address.
> > > 
> > > That's not enough to configure unbound as a fully recursive DNS
> > > server.
> > 
> > What I am missing?
> > Need to fix unbound setup scripts? bsdinstall scripts?
> > As I see unbound setup scripts detects 127.0.0.1 in resolv.conf and
> > configured unbound as fully recursive DNS server.
> May I suggest ntpdate(8)?
> Find a reliable time server in your region, and once found add it
> *early* in your rc.conf(5). Well, ahead of your unbound stanza. ie;



> hostname="..."
> ifconfig_re0="inet ... netmask ..."
> defaultrouter="..."
> ntpdate_enable="YES"
> ntpdate_hosts="a reliable regional time server"

Already pointed about draw back using IP address of NTP servers.

> 
> unbound_enable="YES"
> ..
> 
> ALSO. Since you're upstream will, in all likelihood have informed
> you of a preferred set of 2 name servers. Place one of them in your
> hosts(5) file. This will help ensure that ntpdate(8) can reliably

ok. i.e. cut-off unbound from FreeBSD tree. We don't need unbound and
will always use name servers from upstream, yes?

> discover your regional time server.
> 
> That should get you where you want to go. :-)

I am want working setup after FreeBSD installer.

I think best solution is disable enforciment in case of STA_UNSYNC.

% ntptime
ntp_gettime() returns code 0 (OK)
  time db0a9e2b.4bd3a1d4  Tue, Jun 14 2016 18:15:55.296, (.296198421),
  maximum error 569983 us, estimated error 2912 us, TAI offset 0
ntp_adjtime() returns code 0 (OK)
  modes 0x0 (),
  offset 3993.151 us, frequency 0.240 ppm, interval 1 s,
  maximum error 569983 us, estimated error 2912 us,
  status 0x2001 (PLL,NANO),
^^  -- OK, may be enforciment.
  time constant 10, precision 0.001 us, tolerance 496 ppm,
  

Not only for unbound, for SSL too. And may be in the other places.

> --Chris
> > 
> > > If your system gets its address through DHCP, it is probably
> > > getting DNS server addresses as well, and would work fine *without* your
> > > configuring any of the DNS state.
> > 
> > I am have static address and don't getting DNS server address.
> > 
> > > >> problem on any of my systems, and I've never configured an anchor on 
> > > >> the
> > > >> internal systems.
> > > >> 
> > > >> > IMHO, ntp.conf need to include some numeric IP of public ntp servers.
> > > >> 
> > > >> Ouch; that's a terrible idea, for several different reasons.
> > > >
> > > > What else?
> > > 
> > > All the normal reasons that hard-coding IP addresses is a bad idea; they
> > > can change, you're encouraging a lot of people to use the same ones, etc.
> > 
> > And how to resolve this issuse:
> > 
> > - default install with unbound as recursive DNS server (by default
> >   enforcing DNSSEC)
> > - ntp time synchronisation
> > - stale CMOS time (2008 year)
> 
> 
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"