Re: RHEL7 issues maybe???

[Frank Swasey]

Mark,

Today at 10:01am, Mark Cairney wrote:


Hi Frank,

...


My present /etc/systemd/journald.conf contains the following- these 
are the defaults from our config management system and I haven't found 
a need to adjust them yet (although they do look a bit looser than the 
RH defaults):



...


For day-to-day operations I'm not noticing any limiting kicking in.
During operations that generate a lot of logs e.g. re-syncing a replica,
SLAMD benchmarking logs do indeed get lost but this is preferable to the
alternative where the logger becomes the bottleneck and slapd is
essentially sitting there waiting for logs to be written before continuing.


Thanks!  That's similar to one of the configs I had tried (though I did 
not make the journald log persistent).  I'll try again.




RE: bringing up a replica I'd avoid using syncrepl from scratch- I find
building the slave from a dump of the master is far quicker and more
reliable. I tend to do (as a brief/trivial example):


...

Importing this with slapadd takes me about 5 mins for a 1.5G ldif file
using mdb- it took about 15-20 mins when I was using bdb. With any luck
the replica will then pick up any changes that have been made to the
master since the dump when you start slapd.
The usual disclaimer that this is just what's working for me and not a
definitive "this is how it should be done"


Yeah, I do the same in production.  In my test environment, firing up an 
empty replica is a good way to generate a lot of traffic that needs to 
be logged.


--
Frank Swasey| http://www.uvm.edu/~fcs
Sr Systems Administrator| Always remember: You are UNIQUE,
University of Vermont   |just like everyone else.
  "I am not young enough to know everything." - Oscar Wilde (1854-1900)

Re: RHEL7 issues maybe???

[Mark Cairney]

Hi Frank,


On 01/07/16 14:15, Frank Swasey wrote:
> Today at 7:15am, Mark Cairney wrote:
> 
>> Another gotcha is w.r.t. rate limiting as both journald and rsyslog
>> implement this independently of one another. Disabling this completely
>> can make the bottleneck mentioned above even more apparent! Setting your
>> OpenLDAP logging level appropriately can mitigate this (I log at Stats+
>> Sync)
> 
> I am also running with
> 
> loglevel stats sync
> 
> and my servers are busy enough that journald misses a lot of the
> messages even with ryslogd's rate limiting turned off.  Since Red Hat's
> advice is to *not* remove the rate limiting in journald, I've not found
> a sweet spot yet where the sync logging does not get limited when the
> empty consumer is drinking from the fire hose.  ... any experience you
> can help me with would be greatly appreciated.
> 

My present /etc/systemd/journald.conf contains the following- these are
the defaults from our config management system and I haven't found a
need to adjust them yet (although they do look a bit looser than the RH
defaults):

[Journal]
Storage=persistent
SplitMode=login
RateLimitInterval=10s
RateLimitBurst=3000

My corresponding rsyslog settings are:
# Standard preamble

$ModLoad imuxsock
$ModLoad imjournal
$ModLoad immark

# Tuning
$OmitLocalLogging on
$IMJournalStateFile imjournal.state

$SystemLogRateLimitInterval 0

# Discard audit log entries
user.info ~

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Configure imjournal rate limiting
$imjournalRatelimitInterval 60
$imjournalRatelimitBurst 10

# CAuth logs
local4.* -/usr/local/authz/var/log/openldap.log



For day-to-day operations I'm not noticing any limiting kicking in.
During operations that generate a lot of logs e.g. re-syncing a replica,
SLAMD benchmarking logs do indeed get lost but this is preferable to the
alternative where the logger becomes the bottleneck and slapd is
essentially sitting there waiting for logs to be written before continuing.

RE: bringing up a replica I'd avoid using syncrepl from scratch- I find
building the slave from a dump of the master is far quicker and more
reliable. I tend to do (as a brief/trivial example):
(master)
slapcat -n 1 -o ldif-wrap=no  -l /tmp/master

scp it across to the slave
(slave)
rm -f /var/openldap-data/database/*.mdb
rm -f /var/openldap-data/accesslog/*.mdb
slapadd -q -w -n1 -l /tmp/master.ldif

Importing this with slapadd takes me about 5 mins for a 1.5G ldif file
using mdb- it took about 15-20 mins when I was using bdb. With any luck
the replica will then pick up any changes that have been made to the
master since the dump when you start slapd.
The usual disclaimer that this is just what's working for me and not a
definitive "this is how it should be done"


-- 
/

Mark Cairney
ITI Enterprise Services
Information Services
University of Edinburgh

Tel: 0131 650 6565
Email: mark.cair...@ed.ac.uk
PGP: 0x435A9621

***/

The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.



signature.asc
Description: OpenPGP digital signature

Re: RHEL7 issues maybe???

[Frank Swasey]

Today at 7:15am, Mark Cairney wrote:


Another gotcha is w.r.t. rate limiting as both journald and rsyslog
implement this independently of one another. Disabling this completely
can make the bottleneck mentioned above even more apparent! Setting your
OpenLDAP logging level appropriately can mitigate this (I log at Stats+
Sync)


I am also running with

loglevel stats sync

and my servers are busy enough that journald misses a lot of the 
messages even with ryslogd's rate limiting turned off.  Since Red Hat's 
advice is to *not* remove the rate limiting in journald, I've not found 
a sweet spot yet where the sync logging does not get limited when the 
empty consumer is drinking from the fire hose.  ... any experience you 
can help me with would be greatly appreciated.


--
Frank Swasey| http://www.uvm.edu/~fcs
Sr Systems Administrator| Always remember: You are UNIQUE,
University of Vermont   |just like everyone else.
  "I am not young enough to know everything." - Oscar Wilde (1854-1900)

Re: RHEL7 issues maybe???

[Mark Cairney]

Hi Toby/Quanah,

I'm currently using a RHEL7 derivative (SL7) on our production OpenLDAP
servers i.e. Central Authorisation Service.

We use rsyslog partly out of familiarity but also because it allows
sending logs to a central rsyslog server.

The way this setup appears to work with a system using systemd + rsyslog
is that all the things are logged via systemd's journalctl.

The downside of this approach is that you are effectively running 2
logging daemons in parallel. As logging seems to be a potential
bottleneck for OpenLDAP anyway it potentially exacerbates this even further.

Another gotcha is w.r.t. rate limiting as both journald and rsyslog
implement this independently of one another. Disabling this completely
can make the bottleneck mentioned above even more apparent! Setting your
OpenLDAP logging level appropriately can mitigate this (I log at Stats+
Sync)

From my experience I'd say RHEL7 is a stable system to run OpenLDAP on.
If you have a heavily-loaded system or don't need centralised logging
though I'd try and get away with journalctl on it's own and only
introduce rsyslog logging if you need it.

Kind regards,

Mark



On 01/07/16 10:21, Toby Blake wrote:
> On Thu, 30 Jun 2016, Quanah Gibson-Mount wrote:
> 
>> On a side note, we've been moving customers off of RHEL7 back to
>> RHEL6, as we've simply found it too unstable for production use.
> 
> Hi Quanah,
> 
> This is concerning - can you provide a little more detail on this?
> 
> Cheers
> Toby Blake
> School of Informatics
> University of Edinburgh
> 

-- 
/

Mark Cairney
ITI Enterprise Services
Information Services
University of Edinburgh

Tel: 0131 650 6565
Email: mark.cair...@ed.ac.uk
PGP: 0x435A9621

***/

The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.



signature.asc
Description: OpenPGP digital signature

Re: RHEL7 issues maybe???

[Toby Blake]

On Thu, 30 Jun 2016, Quanah Gibson-Mount wrote:


On a side note, we've been moving customers off of RHEL7 back to
RHEL6, as we've simply found it too unstable for production use.


Hi Quanah,

This is concerning - can you provide a little more detail on this?

Cheers
Toby Blake
School of Informatics
University of Edinburgh

--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

Re: RHEL7 issues maybe???

[Frank Swasey]

Today at 12:28pm, Quanah Gibson-Mount wrote:

--On Thursday, June 30, 2016 10:13 AM -0700 Quanah Gibson-Mount 
 wrote:



Again, I have no clue why RHEL7 does this. Literally I can install a
series of VMs on RHEL7. Some correctly log out to rsyslog, some don't,
without tweaking them.  I.e., some default to only logging to journald
and have to get reset to use rsyslog.


Here we go: 

The first comment details what was done to fix the problem.

On a side note, we've been moving customers off of RHEL7 back to RHEL6, as 
we've simply found it too unstable for production use.


Quanah,

  As always - thanks for helping me see the light!

--
Frank Swasey| http://www.uvm.edu/~fcs
Sr Systems Administrator| Always remember: You are UNIQUE,
University of Vermont   |just like everyone else.
  "I am not young enough to know everything." - Oscar Wilde (1854-1900)

Re: RHEL7 issues maybe???

[Quanah Gibson-Mount]

--On Thursday, June 30, 2016 10:13 AM -0700 Quanah Gibson-Mount 
 wrote:



Again, I have no clue why RHEL7 does this. Literally I can install a
series of VMs on RHEL7. Some correctly log out to rsyslog, some don't,
without tweaking them.  I.e., some default to only logging to journald
and have to get reset to use rsyslog.


Here we go: 

The first comment details what was done to fix the problem.

On a side note, we've been moving customers off of RHEL7 back to RHEL6, as 
we've simply found it too unstable for production use.


--Quanah

--

Quanah Gibson-Mount
Platform Architect
Manager, Systems Team
Zimbra, Inc.

Zimbra ::  the leader in open source messaging and collaboration
A division of Synacor, Inc

Re: RHEL7 issues maybe???

[Quanah Gibson-Mount]

--On Thursday, June 30, 2016 1:10 PM -0400 Frank Swasey 
 wrote:



Today at 11:35am, Quanah Gibson-Mount wrote:


--On Thursday, June 30, 2016 12:27 PM -0400 Frank Swasey
 wrote:


Folks,

   I am working on moving my OpenLDAP 2.4.44 environment from RHEL6 to
RHEL7.  I want to double check an assumption I have made before I open a
report with Red Hat about the quality of their product.

   I am assuming that with "loglevel sync" and an empty consumer, that
there should be one sincrepl_entry line that has "inserted UUID" in the
syslog output for every entry created.  Is that a valid assumption?


You're assuming RHEL7 logs to syslog by default.  My experience so far
is  that it may or may not.  I'm not sure why sometimes it does and
sometimes it  doesn't, and instead logs to journald.


I'm not assuming that completely.  I have configured rsyslogd to send
LOCAL6 to /var/log/slapd.log and am passing "-l LOCAL6" to slapd in the
startup.  I am assuming the system is actually going to do what I told it
to (and that I've told it correctly).  I'm asking the question to see if
my assumption is faulty or not.


Right, we always configure things to use syslog.  I'm simply telling you, 
in my experience, regardless of whether or not we had things configured to 
use syslog, RHEL7 *may or may not* honor that.  There's some article I 
found before that details how to fix it so rsyslog works again under RHEL7 
when you hit this case, that I'm trying to track down and send to you.


Again, I have no clue why RHEL7 does this. Literally I can install a series 
of VMs on RHEL7. Some correctly log out to rsyslog, some don't, without 
tweaking them.  I.e., some default to only logging to journald and have to 
get reset to use rsyslog.


--Quanah

--

Quanah Gibson-Mount
Platform Architect
Manager, Systems Team
Zimbra, Inc.

Zimbra ::  the leader in open source messaging and collaboration
A division of Synacor, Inc

Re: RHEL7 issues maybe???

[Frank Swasey]

Today at 11:35am, Quanah Gibson-Mount wrote:

--On Thursday, June 30, 2016 12:27 PM -0400 Frank Swasey 
 wrote:



Folks,

   I am working on moving my OpenLDAP 2.4.44 environment from RHEL6 to
RHEL7.  I want to double check an assumption I have made before I open a
report with Red Hat about the quality of their product.

   I am assuming that with "loglevel sync" and an empty consumer, that
there should be one sincrepl_entry line that has "inserted UUID" in the
syslog output for every entry created.  Is that a valid assumption?


You're assuming RHEL7 logs to syslog by default.  My experience so far is 
that it may or may not.  I'm not sure why sometimes it does and sometimes it 
doesn't, and instead logs to journald.


I'm not assuming that completely.  I have configured rsyslogd to send 
LOCAL6 to /var/log/slapd.log and am passing "-l LOCAL6" to slapd in the 
startup.  I am assuming the system is actually going to do what I told 
it to (and that I've told it correctly).  I'm asking the question to see 
if my assumption is faulty or not.


Thanks,

--
Frank Swasey| http://www.uvm.edu/~fcs
Sr Systems Administrator| Always remember: You are UNIQUE,
University of Vermont   |just like everyone else.
  "I am not young enough to know everything." - Oscar Wilde (1854-1900)

Re: RHEL7 issues maybe???

[Quanah Gibson-Mount]

--On Thursday, June 30, 2016 12:27 PM -0400 Frank Swasey 
 wrote:



Folks,

   I am working on moving my OpenLDAP 2.4.44 environment from RHEL6 to
RHEL7.  I want to double check an assumption I have made before I open a
report with Red Hat about the quality of their product.

   I am assuming that with "loglevel sync" and an empty consumer, that
there should be one sincrepl_entry line that has "inserted UUID" in the
syslog output for every entry created.  Is that a valid assumption?


You're assuming RHEL7 logs to syslog by default.  My experience so far is 
that it may or may not.  I'm not sure why sometimes it does and sometimes 
it doesn't, and instead logs to journald.


--Quanah



--

Quanah Gibson-Mount
Platform Architect
Manager, Systems Team
Zimbra, Inc.

Zimbra ::  the leader in open source messaging and collaboration
A division of Synacor, Inc

RHEL7 issues maybe???

[Frank Swasey]

Folks,

  I am working on moving my OpenLDAP 2.4.44 environment from RHEL6 to 
RHEL7.  I want to double check an assumption I have made before I open a 
report with Red Hat about the quality of their product.


  I am assuming that with "loglevel sync" and an empty consumer, that 
there should be one sincrepl_entry line that has "inserted UUID" in the 
syslog output for every entry created.  Is that a valid assumption?


 Thanks,
--
Frank Swasey| http://www.uvm.edu/~fcs
Sr Systems Administrator| Always remember: You are UNIQUE,
University of Vermont   |just like everyone else.
  "I am not young enough to know everything." - Oscar Wilde (1854-1900)