Re: LibreNMS 500 Internal Server Error

[Stuart Henderson]

On 2016/12/14 14:27, Predrag Punosevac wrote:
> Stuart Henderson wrote:
> 
> Hi Stuard,
> 
> Thank you so very much for your thoughtful insight. I have few follow up
> questions.
> 
> > On 2016-12-14, Predrag Punosevac  wrote:
> > > I followed pkg-readmes and mysql, php, and nginx are correctly
> > > configured and working in the INSECURE mode(no chroot). Please no
> > > remarks if you are not familiar with LibreNMS and the fact that it has
> > > to be run in the insecure mode.
> > 
> > The pkg-readme only says to run php outside the chroot, nginx is OK as
> > normal.
> > 
> 
> Fair enough. I understand that part. Could you than please clarify to me
> if I have to make changes to MariaDB's /etc/my.cnf file since Nginx will
> have to access it from the chroot. I obviously know how to set up
> MariaDB, PHP, and Nginx in chroot but LibreNMS is complicating things.

My standard config for servers running mariadb + web is to create
/var/www/var/run/mysql, place this in my.cnf:

[mysqld]
socket  = /var/www/var/run/mysql/mysql.sock

and "ln -s /var/www/var/run/mysql /var/run/" in rc.local (because /var/run
is cleared at boot).

> Also another thing about MariaDB is this paragraph
> 
> # Uncomment the following if you are using InnoDB tables
> innodb_data_home_dir = /var/mysql
> innodb_file_per_table=1
> sql-mode=""
> #innodb_data_file_path = ibdata1:10M:autoextend
> #innodb_log_group_home_dir = /var/mysql
> # You can set .._buffer_pool_size up to 50 - 80 %
> # of RAM but beware of setting memory usage too high
> #innodb_buffer_pool_size = 16M
> #innodb_additional_mem_pool_size = 2M
> # Set .._log_file_size to 25 % of buffer pool size
> #innodb_log_file_size = 5M
> #innodb_log_buffer_size = 8M
> #innodb_flush_log_at_trx_commit = 1
> #innodb_lock_wait_timeout = 50
> 
> Should I increase buffer pool for performance or any other parameters?
> Any sugestions for MariaDB tuning?

I don't know much about mariadb tuning, I used my-medium.cnf,
uncommented those lines and changed the socket path, that's about
it I think.

> At this point I do have a working LibreNMS server collecting info from
> about 40 devices (mostly servers) but I think I need some serious
> performance tuning. First off my CPU load is 100% on 4 cores (I have a
> dedicated small Atom server). What do you see on your servers? I am
> afraid something is miss-configured and it is just eating CPU time.
> 
> Regarding Collectd (since I am displaying Collectd RRD on this LibreNMS
> server). Do you recommend that I turn on rrdcached? Should I enable
> RRDCacheD plugin on the LibreNMS server?
> 
> https://collectd.org/wiki/index.php/Plugin:RRDCacheD 

I haven't used collectd so can't comment on that.

I find most non-trivial use of rrdtool fairly heavy on the machine
without rrdcached. (Only tried it on OpenBSD so I don't know if this
is the same everywhere). On my librenms boxes I have rrdcached and still
found it pretty heavy, currently running it with delayed writes (-w 1800
-z 1800) with the files on SSD and it's not so bad, but I still wouldn't
really call it lightweight.

Permissions/ownership were a bit awkward for rrdcached, I currently
have rrd/* owned by _librenms:_rrdcached and _librenms is in groups
_librenms and _rrdcached (istr having some problems with new files
getting created with wrong ownership at some point though it seems
ok at the moment with current rrdtool+librenms versions).

> > Try normal chrooted nginx (remove the -u flag) and undo your changes to
> > the
> > "location" block, I have it working on multiple installations like that.
> > 
> > For un-chrooted nginx you will have to make some other changes, though I
> > don't
> > see any reason to do that.
> > 
> 
> Agreed! Could you just clarify if I need to use handle @librenms or
> @lnms because my server name is lnms.int.autonlab.org. I will admit I
> have to reread nginx rewrite module documentation Mihai Popescu
> mentioned in the previous e-mail. Also in the term of PHP performances.
> I have increased the number of children from 5 to 25 in
> /etc/php-fpm.conf but more or less everything else is default. Any
> suggestions. I feel based upon my previous experience on running TurnKey
> Observium that I should be able to get better performance on this
> hardware for 40 devices. 

It's just creating a location context to use in try_files, the name
doesn't matter - you can call it @rhubarb if you like :-) 

> > >  The one other thing I did different was my
> > > /var/www/librenms is owned by _librenms:_librenms.
> > 
> > Only the logs/ and rrd/ directories should be writable by the user
> > running
> > the php code.
> 
> Thanks for clarifying this!

One of the biggest problems with webapp security in general is to
have too many things writable by the uid running the webapp. All too
often people are told to chown things so they're writable by that
user as a quick fix, but it's often opening a door to remote code
execution (especially in 

Re: LibreNMS 500 Internal Server Error

[Predrag Punosevac]

Stuart Henderson wrote:

Hi Stuard,

Thank you so very much for your thoughtful insight. I have few follow up
questions.

> On 2016-12-14, Predrag Punosevac  wrote:
> > I followed pkg-readmes and mysql, php, and nginx are correctly
> > configured and working in the INSECURE mode(no chroot). Please no
> > remarks if you are not familiar with LibreNMS and the fact that it has
> > to be run in the insecure mode.
> 
> The pkg-readme only says to run php outside the chroot, nginx is OK as
> normal.
> 

Fair enough. I understand that part. Could you than please clarify to me
if I have to make changes to MariaDB's /etc/my.cnf file since Nginx will
have to access it from the chroot. I obviously know how to set up
MariaDB, PHP, and Nginx in chroot but LibreNMS is complicating things.

Also another thing about MariaDB is this paragraph

# Uncomment the following if you are using InnoDB tables
innodb_data_home_dir = /var/mysql
innodb_file_per_table=1
sql-mode=""
#innodb_data_file_path = ibdata1:10M:autoextend
#innodb_log_group_home_dir = /var/mysql
# You can set .._buffer_pool_size up to 50 - 80 %
# of RAM but beware of setting memory usage too high
#innodb_buffer_pool_size = 16M
#innodb_additional_mem_pool_size = 2M
# Set .._log_file_size to 25 % of buffer pool size
#innodb_log_file_size = 5M
#innodb_log_buffer_size = 8M
#innodb_flush_log_at_trx_commit = 1
#innodb_lock_wait_timeout = 50

Should I increase buffer pool for performance or any other parameters?
Any sugestions for MariaDB tuning?

At this point I do have a working LibreNMS server collecting info from
about 40 devices (mostly servers) but I think I need some serious
performance tuning. First off my CPU load is 100% on 4 cores (I have a
dedicated small Atom server). What do you see on your servers? I am
afraid something is miss-configured and it is just eating CPU time.

Regarding Collectd (since I am displaying Collectd RRD on this LibreNMS
server). Do you recommend that I turn on rrdcached? Should I enable
RRDCacheD plugin on the LibreNMS server?

https://collectd.org/wiki/index.php/Plugin:RRDCacheD 





> Try normal chrooted nginx (remove the -u flag) and undo your changes to
> the
> "location" block, I have it working on multiple installations like that.
> 
> For un-chrooted nginx you will have to make some other changes, though I
> don't
> see any reason to do that.
> 

Agreed! Could you just clarify if I need to use handle @librenms or
@lnms because my server name is lnms.int.autonlab.org. I will admit I
have to reread nginx rewrite module documentation Mihai Popescu
mentioned in the previous e-mail. Also in the term of PHP performances.
I have increased the number of children from 5 to 25 in
/etc/php-fpm.conf but more or less everything else is default. Any
suggestions. I feel based upon my previous experience on running TurnKey
Observium that I should be able to get better performance on this
hardware for 40 devices. 





> >  The one other thing I did different was my
> > /var/www/librenms is owned by _librenms:_librenms.
> 
> Only the logs/ and rrd/ directories should be writable by the user
> running
> the php code.


Thanks for clarifying this!

Best,
Predrag

Re: LibreNMS 500 Internal Server Error

[Stuart Henderson]

On 2016-12-14, Predrag Punosevac  wrote:
> I followed pkg-readmes and mysql, php, and nginx are correctly
> configured and working in the INSECURE mode(no chroot). Please no
> remarks if you are not familiar with LibreNMS and the fact that it has
> to be run in the insecure mode.

The pkg-readme only says to run php outside the chroot, nginx is OK as normal.

Try normal chrooted nginx (remove the -u flag) and undo your changes to the
"location" block, I have it working on multiple installations like that.

For un-chrooted nginx you will have to make some other changes, though I don't
see any reason to do that.

>  The one other thing I did different was my
> /var/www/librenms is owned by _librenms:_librenms.

Only the logs/ and rrd/ directories should be writable by the user running
the php code.

Re: LibreNMS 500 Internal Server Error

[Mihai Popescu]

Maybe a shot in the dark:

"But if these directives are put inside the “ ... ” location, the last
flag should be replaced by break, or otherwise nginx will make 10
cycles and return the 500 error: "

>From this: [1] [2].
[1] http://nginx.org/en/docs/http/ngx_http_rewrite_module.html
[2] https://www.nginx.com/blog/creating-nginx-rewrite-rules/

Re: LibreNMS 500 Internal Server Error

[Predrag Punosevac]

I think I made a progress. Namely I stumbled upon this post 

https://github.com/librenms/librenms/issues/3525

Replacing 

 location / {
   try_files $uri $uri/ @librenms;
}

with 

 location / {
   try_files $uri $uri/ /index.php;
}

Did the trick for me. I would still appreciate if people who are running
LibreNMS in production can explain me what am I doing wrong.

Predrag