Send netdisco-users mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.sourceforge.net/lists/listinfo/netdisco-users
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of netdisco-users digest..."
Today's Topics:
1. Re: ND2 Worker configuration and Server Performance problems
(Oliver Gorwits)
2. Re: ND2 Worker configuration and Server Performance problems
(Oliver Gorwits)
3. Re: ND2 Worker configuration and Server Performance problems
(Joseph Bernard)
4. Re: ND2 Worker configuration and Server Performance problems
(Oliver Gorwits)
5. Re: ND2 Worker configuration and Server Performance problems
(Tobias Gerlach)
6. Re: ND2 Worker configuration and Server Performance problems
(Tobias Gerlach)
7. Re: ND2 Worker configuration and Server Performance problems
(Charles Goldsmith)
8. Re: ND2 Worker configuration and Server Performance problems
(Ben Nelson)
--- Begin Message ---
Hi Tobias,
On 2015-04-01 11:29, Tobias Gerlach wrote:
workers:
tasks: 'AUTO * 2'
sleep_time: 1
queue: PostgreSQL
...
Based on 8 installed CPUs I expect 16 netdisco workers, but
independent if I increase or decrease the worker count,
all the time I see 8 workers in the process table
Just on this point, it seems the behaviour of one of the modules we use
inside Netdisco has changed, which might be causing this.
Please can you set "tasks: 16" and see if that helps?
regards,
oliver.
--- End Message ---
--- Begin Message ---
Hi Tobias,
Have you looked at the "Poller Performance" and "Slowest Devices"
reports in the Admin menu? Sometimes slow devices can tie up pollers and
affect the overall runtime.
regards,
oliver.
On 2015-04-01 11:29, Tobias Gerlach wrote:
The problem is that after a few hours the RAM begins to swap
as a result the server uptime is increasing and the jobs didn't
finish
in proper time
--- End Message ---
--- Begin Message ---
Maybe the AUTO value is broken. I took out “AUTO * “ and just put in the value
I wanted, and it worked.
Thanks,
Joseph B.
On Apr 1, 2015, at 10:08 AM, Charles Goldsmith <[email protected]> wrote:
> Tobias, that's a pretty big network. I have not worked with ND2 on
> such a size, but I have with ND1, and we had to do quite a bit of
> tuning on pgsql to optimize for ram.
>
> Have you done any tuning yet?
>
>> From your PS, looks like the cpu's are quite busy, what process(es)
> are using the most ram and causing the swap?
>
> On Wed, Apr 1, 2015 at 4:29 AM, Tobias Gerlach <[email protected]> wrote:
>> Hello All,
>>
>> I'm using
>>
>> App::Netdisco 2.032001
>> DB Schema v40
>> Dancer 1.3132
>> Bootstrap 2.3.1
>> PostgreSQL PostgreSQL 9.1.12 on x86_64-suse-linux-gnu, compiled by gcc
>> (SUSE Linux) 4.3.2
>> [gcc-4_3-branch revision 141291], 64-bit.
>> DBI 1.632, DBD::Pg 2.19.3
>> SNMP::Info 3.26
>> Perl 5.010000
>>
>> in our network environment
>>
>> Statistics for this installation
>> 5,524 devices with 236,925 interfaces using 19,316 IPs 19,915 layer 2
>> links between devices
>> 96,146 nodes in 161,029 entries 159,622 IPs in 528,795 entries
>>
>> on a SLES 11.3 Server
>>
>> 8x Intel(R) Xeon(R) CPU E7- 4820 @ 2.00GHz
>> 16 GB RAM
>>
>> The problem is that after a few hours the RAM begins to swap
>>
>> netdisco@server:~> free
>> total used free shared buffers cached
>> Mem: 16336160 1331484 15004676 34996 5732 367940
>> -/+ buffers/cache: 957812 15378348
>> Swap: 16777212 334176 16443036
>>
>> as a result the server uptime is increasing and the jobs didn't finish
>> in proper time
>>
>> netdisco@server:~> uptime
>> 11:59am up 1 day 21:47, 1 user, load average: 1.67, 2.19, 1.68
>>
>> I try to play around with the worker configuration, but it looks like
>> that the settings are not working
>>
>> ~environments/deployment.yml
>> ...
>> workers:
>> tasks: 'AUTO * 2'
>> sleep_time: 1
>> queue: PostgreSQL
>> ...
>>
>> Based on 8 installed CPUs I expect 16 netdisco workers, but
>> independent if I increase or decrease the worker count,
>> all the time I see 8 workers in the process table
>>
>> netdisco@server:~> ps aux | grep "netdisco-daemon: worker"
>> netdisco 3058 0.0 0.2 173052 43276 ? S 11:53 0:00
>> netdisco-daemon: worker #1 scheduler: idle
>> netdisco 3059 25.8 0.4 208940 65420 ? S 11:53 0:00
>> netdisco-daemon: worker #2 manager: idle
>> netdisco 3060 46.7 0.4 210788 68416 ? S 11:53 0:00
>> netdisco-daemon: worker #3 poller: working on #16169554 ...
>> netdisco 3061 57.8 0.4 215476 72960 ? S 11:53 0:01
>> netdisco-daemon: worker #4 poller: working on #16168032 ...
>> netdisco 3062 57.3 0.4 215464 72944 ? S 11:53 0:01
>> netdisco-daemon: worker #5 poller: working on #16167271 ...
>> netdisco 3063 54.3 0.4 210628 68200 ? S 11:53 0:01
>> netdisco-daemon: worker #6 poller: working on #16168522 ...
>> netdisco 3064 62.9 0.4 215536 72972 ? S 11:53 0:01
>> netdisco-daemon: worker #7 poller: working on #16170079 ...
>> netdisco 3065 60.9 0.4 215224 72800 ? S 11:53 0:01
>> netdisco-daemon: worker #8 poller: working on #16170677 ...
>> netdisco 3066 48.4 0.4 210692 68208 ? S 11:53 0:00
>> netdisco-daemon: worker #9 poller: working on #16170967 ...
>> netdisco 3067 67.3 0.4 216832 74552 ? R 11:53 0:01
>> netdisco-daemon: worker #10 poller: working on #16167410 ...
>>
>>
>> How much RAM / CPU is recommended for such network environment (see
>> above) and what could be the reason that the worker configurations is
>> not working?
>> Thanks in advance for any tip!
>>
>> ------------------------------------------------------------------------------
>> Dive into the World of Parallel Programming The Go Parallel Website,
>> sponsored
>> by Intel and developed in partnership with Slashdot Media, is your hub for
>> all
>> things parallel software development, from weekly thought leadership blogs to
>> news, videos, case studies, tutorials and more. Take a look and join the
>> conversation now. http://goparallel.sourceforge.net/
>> _______________________________________________
>> Netdisco mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/netdisco-users
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website, sponsored
> by Intel and developed in partnership with Slashdot Media, is your hub for all
> things parallel software development, from weekly thought leadership blogs to
> news, videos, case studies, tutorials and more. Take a look and join the
> conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> Netdisco mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/netdisco-users
--- End Message ---
--- Begin Message ---
On 2015-04-01 16:08, Joseph Bernard wrote:
Maybe the AUTO value is broken. I took out “AUTO * “ and just put in
the value I wanted, and it worked.
Thanks for the confirmation Joseph, I've created a ticket for the bug
report:
https://sourceforge.net/p/netdisco/netdisco2/209/
regards,
oliver.
--- End Message ---
--- Begin Message ---
Hi Oliver,
when I configure "tasks: 16" instead of "tasks: 'AUTO * 2'" then it's
working and 16 tasks are used.
I also checked the "Poller Performance" and "Slowest Devices". There
are some slow devices but
not too much that they could affect the overall runtime so
dramatically, I would say.
Hi Charles,
I haven't tuned the PostgreSQL DB yet, but I found the article in the
Netdisco documentation about the tool pgtune. I'll work on that and
see if it helps.
Regarding the swapping processes, these are the Top 20 at that moment:
starman 43332 kB
starman 35844 kB
starman 32492 kB
starman 32488 kB
starman 30660 kB
perl 10952 kB
httpd2-prefork 10108 kB
httpd2-prefork 7940 kB
httpd2-prefork 7936 kB
starman 7760 kB
httpd2-prefork 6020 kB
httpd2-prefork 5280 kB
httpd2-prefork 5280 kB
httpd2-prefork 5248 kB
httpd2-prefork 5204 kB
httpd2-prefork 5152 kB
httpd2-prefork 5140 kB
httpd2-prefork 5124 kB
haveged 4100 kB
postgres 2992 kB
--- End Message ---
--- Begin Message ---
Hi Oliver,
when I configure "tasks: 16" instead of "tasks: 'AUTO * 2'" then it's
working and 16 tasks are used.
I also checked the "Poller Performance" and "Slowest Devices". There
are some slow devices but
not too much that they could affect the overall runtime so
dramatically, I would say.
Hi Charles,
I haven't tuned the PostgreSQL DB yet, but I found the article in the
Netdisco documentation about the tool pgtune. I'll work on that and
see if it helps.
Regarding the swapping processes, these are the Top 20 at that moment:
starman 43332 kB
starman 35844 kB
starman 32492 kB
starman 32488 kB
starman 30660 kB
perl 10952 kB
httpd2-prefork 10108 kB
httpd2-prefork 7940 kB
httpd2-prefork 7936 kB
starman 7760 kB
httpd2-prefork 6020 kB
httpd2-prefork 5280 kB
httpd2-prefork 5280 kB
httpd2-prefork 5248 kB
httpd2-prefork 5204 kB
httpd2-prefork 5152 kB
httpd2-prefork 5140 kB
httpd2-prefork 5124 kB
haveged 4100 kB
postgres 2992 kB
--- End Message ---
--- Begin Message ---
Tobias, those are just the processes using swap, they may not be what
is causing the swap. My guess is that it's pgsql causing the swap,
but that's a guess.
On Wed, Apr 1, 2015 at 9:43 AM, Tobias Gerlach <[email protected]> wrote:
> Hi Oliver,
> when I configure "tasks: 16" instead of "tasks: 'AUTO * 2'" then it's
> working and 16 tasks are used.
> I also checked the "Poller Performance" and "Slowest Devices". There
> are some slow devices but
> not too much that they could affect the overall runtime so
> dramatically, I would say.
>
> Hi Charles,
> I haven't tuned the PostgreSQL DB yet, but I found the article in the
> Netdisco documentation about the tool pgtune. I'll work on that and
> see if it helps.
> Regarding the swapping processes, these are the Top 20 at that moment:
>
> starman 43332 kB
> starman 35844 kB
> starman 32492 kB
> starman 32488 kB
> starman 30660 kB
> perl 10952 kB
> httpd2-prefork 10108 kB
> httpd2-prefork 7940 kB
> httpd2-prefork 7936 kB
> starman 7760 kB
> httpd2-prefork 6020 kB
> httpd2-prefork 5280 kB
> httpd2-prefork 5280 kB
> httpd2-prefork 5248 kB
> httpd2-prefork 5204 kB
> httpd2-prefork 5152 kB
> httpd2-prefork 5140 kB
> httpd2-prefork 5124 kB
> haveged 4100 kB
> postgres 2992 kB
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website, sponsored
> by Intel and developed in partnership with Slashdot Media, is your hub for all
> things parallel software development, from weekly thought leadership blogs to
> news, videos, case studies, tutorials and more. Take a look and join the
> conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> Netdisco mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/netdisco-users
--- End Message ---
--- Begin Message ---
Hey Tobias,
We have a fairly large network and I had to do a good deal of tuning of
nd2 worker threads and postgres memory settings due to swapping. One of
the latest releases really helped by restarting worker threads daily
(thanks Oliver!).
pgtune is a good place to start, but I would suggest tuning down
effective_cache_size, work_mem, and shared_buffers in the postgres
config likely further.
With 22 worker threads, the total memory that nd2 was using on our
machine reached about 5G, so you might look at tuning down the worker
thread count as well.
If you have the option of getting more memory that would of course make
things better too.
Cheers,
Ben
On 4/1/15 12:27 PM, Charles Goldsmith wrote:
> Tobias, those are just the processes using swap, they may not be what
> is causing the swap. My guess is that it's pgsql causing the swap,
> but that's a guess.
>
> On Wed, Apr 1, 2015 at 9:43 AM, Tobias Gerlach <[email protected]> wrote:
>> Hi Oliver,
>> when I configure "tasks: 16" instead of "tasks: 'AUTO * 2'" then it's
>> working and 16 tasks are used.
>> I also checked the "Poller Performance" and "Slowest Devices". There
>> are some slow devices but
>> not too much that they could affect the overall runtime so
>> dramatically, I would say.
>>
>> Hi Charles,
>> I haven't tuned the PostgreSQL DB yet, but I found the article in the
>> Netdisco documentation about the tool pgtune. I'll work on that and
>> see if it helps.
>> Regarding the swapping processes, these are the Top 20 at that moment:
>>
>> starman 43332 kB
>> starman 35844 kB
>> starman 32492 kB
>> starman 32488 kB
>> starman 30660 kB
>> perl 10952 kB
>> httpd2-prefork 10108 kB
>> httpd2-prefork 7940 kB
>> httpd2-prefork 7936 kB
>> starman 7760 kB
>> httpd2-prefork 6020 kB
>> httpd2-prefork 5280 kB
>> httpd2-prefork 5280 kB
>> httpd2-prefork 5248 kB
>> httpd2-prefork 5204 kB
>> httpd2-prefork 5152 kB
>> httpd2-prefork 5140 kB
>> httpd2-prefork 5124 kB
>> haveged 4100 kB
>> postgres 2992 kB
>>
>> ------------------------------------------------------------------------------
>> Dive into the World of Parallel Programming The Go Parallel Website,
>> sponsored
>> by Intel and developed in partnership with Slashdot Media, is your hub for
>> all
>> things parallel software development, from weekly thought leadership blogs to
>> news, videos, case studies, tutorials and more. Take a look and join the
>> conversation now. http://goparallel.sourceforge.net/
>> _______________________________________________
>> Netdisco mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/netdisco-users
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website, sponsored
> by Intel and developed in partnership with Slashdot Media, is your hub for all
> things parallel software development, from weekly thought leadership blogs to
> news, videos, case studies, tutorials and more. Take a look and join the
> conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> Netdisco mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/netdisco-users
--
Ben Nelson, Systems Engineer
Indiana University GlobalNOC
<[email protected]>
--- End Message ---
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Netdisco mailing list - Digest Mode
[email protected]
https://lists.sourceforge.net/lists/listinfo/netdisco-users
