netdisco-users Digest, Vol 132, Issue 3

Thu, 01 Jun 2017 21:02:37 -0700 

Send netdisco-users mailing list submissions to
        [email protected]

To subscribe or unsubscribe via the World Wide Web, visit
        https://lists.sourceforge.net/lists/listinfo/netdisco-users
or, via email, send a message with subject or body 'help' to
        [email protected]

You can reach the person managing the list at
        [email protected]

When replying, please edit your Subject line so it is more specific
than "Re: Contents of netdisco-users digest..."

Today's Topics:

   1. Re: Random speedup tricks (Brian Marshall)
   2. Failed to upgrade DB schema 2.034001 (Nicol?s Matsunaga)
   3. Re: Failed to upgrade DB schema 2.034001 (Andy Ruhl)
--- Begin Message --- 
Hello Joseph,

Thanks for the kind words
I'm currently running a distributed system in a dozen branch offices. Each one is a Debian 8 VM with 1 vcpu and 4gb RAM. The VM hosts are a combination of Dell and Nutanix nodes (E5-2650/2665/2690) with a VMWare hypervisor. The DB is on each VM alongside netdisco. 

Minor tweaks on postgres.conf based on pgtune:
   shared_buffers = 128MB
   maintenance_work_mem = 60MB

   checkpoint_completion_target = 0.9

   effective_cache_size = 704MB

   work_mem = 128MB

   checkpoint_segments = 32

   shared_buffers = 240MB
They are also running NFSight for sflow/geoip data on each switch, guacamole for html5 ssh switch access, rancid for configs and smokeping/mrtg for port/WAN stats, so they are under load for other reasons as well. (as an aside I have written integration plugins and scripts for use with these tools so that the netdisco discovery process autoconfigures everything without user intervention)
They also force ssl and use kerberos/LDAP against our corporate AD for sso auth, which I'm sure adds some additional load and latency. 

None of the sites are particularly large:


53 devices with 1,963 interfaces using 154 IPs

180 layer 2 links between devices

1,376 nodes in 5,242 entries

932 IPs in 1,890 entries

Statistics took 0 seconds to process



6 devices with 342 interfaces using 16 IPs

26 layer 2 links between devices

321 nodes in 416 entries

169 IPs in 633 entries

Statistics took 0 seconds to process.



36 devices with 842 interfaces using 75 IPs

128 layer 2 links between devices

979 nodes in 1,831 entries

588 IPs in 1,862 entries

Statistics took 0 seconds to process.



67 devices with 1,809 interfaces using 152 IPs

200 layer 2 links between devices

1,287 nodes in 2,126 entries

1,078 IPs in 4,007 entries

Statistics took 0 seconds to process.



22 devices with 822 interfaces using 78 IPs

92 layer 2 links between devices

942 nodes in 2,134 entries

531 IPs in 943 entries

Statistics took 0 seconds to process.



44 devices with 1,294 interfaces using 109 IPs

152 layer 2 links between devices

916 nodes in 1,567 entries

510 IPs in 1,557 entries

Statistics took 1 seconds to process.



19 devices with 1,419 interfaces using 55 IPs

136 layer 2 links between devices

793 nodes in 1,772 entries

531 IPs in 1,311 entries

Statistics took 0 seconds to process.



30 devices with 668 interfaces using 101 IPs

61 layer 2 links between devices

284 nodes in 366 entries

779 IPs in 2,191 entries

Statistics took 0 seconds to process.



27 devices with 835 interfaces using 77 IPs

102 layer 2 links between devices

512 nodes in 872 entries

327 IPs in 858 entries

Statistics took 0 seconds to process.



21 devices with 853 interfaces using 54 IPs

85 layer 2 links between devices

543 nodes in 743 entries

340 IPs in 917 entries

Statistics took 0 seconds to process.



10 devices with 497 interfaces using 34 IPs

56 layer 2 links between devices

593 nodes in 932 entries

402 IPs in 1,296 entries

Statistics took 0 seconds to process.
I think one of the reasons swappiness works to speed up my install so much is that the entire DB can fit in memory (Even my largest PG DB is under 200MB). Setting swappiness down to 1 ensures that it won't start swapping what it considers "hotter" data into memory and swapping DB pages out.
It would be very interesting to see what kind of an impact these changes could have on an installation of your size! 

   -Brian Marshall

On 01/06/17 01:29 PM, Joseph Bernard wrote:

What kind of setup are you running?  We have ours running as a VM with 8 vCPUs 
(Intel Xeon CPU E5-2680 v3 @ 2.50GHz) and 16GB RAM.

Statistics for this installation:
2,443 devices with 132,490 interfaces using 7,813 IPs
13,407 layer 2 links between devices
44,310 nodes in 58,295 entries
5,077,480 IPs in 17,408,486 entries
Statistics took 16 seconds to process.


Thanks,
Joseph B.


On 6/1/17, 3:36 PM, "Brian Marshall" <[email protected]> wrote:

    Hello List,

    I thought I'd share a few speedup tricks that I'd used on my Netdisco
    server that have made life much more pleasant.

    1. Making static assets go thru regular apache and not starman
    2. Making internet-hosted static assets local to the server
    3. Prewarm the postgres caches every 5 minutes (pg_prewarm function)
    4. Turning the linux VM swappiness down to 1

    I have submitted patches for the first 3, and the fourth is easy to do
    and made a HUGE difference. The server would be a crapshoot after not
    having been used for a few hours, sometimes taking 10 seconds plus to
    respond. With swappiness turned down to 1 it is almost instant every time.
    Hope this makes some people's lives easier!

        -Brian Marshall

    
------------------------------------------------------------------------------
    Check out the vibrant tech community on one of the world's most
    engaging tech sites, Slashdot.org! http://sdm.link/slashdot
    _______________________________________________
    Netdisco mailing list
    [email protected]
    https://lists.sourceforge.net/lists/listinfo/netdisco-users


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Netdisco mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/netdisco-users

--- End Message ---
--- Begin Message --- 
Hi,

I have no knowledge of PostreSQL.
Sometime ago I updated netdisco from version 2.033002 to  2.034001
When I did the upgrade I executed "~/bin/netdisco-deploy"
Now I tried it to give netdisco another chance (because it is not working fine) and I want to have the correct DB version and I am not able to update the schema doing "~netdisco/bin/netdisco-db-deploy " 

How can I fix this?

Regards,
Nicolás


[netdisco@server ~]$ ~netdisco/bin/netdisco-db-deploy
DBIx::Class::Schema::Versioned::_on_connect(): Versions out of sync. This is 41, your database contains version 40, please call upgrade on your Schema. at /home/netdisco/perl5/lib/perl5/DBICx/Sugar.pm line 85 DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (40) is lower than the schema version (41). Attempting upgrade. 


[netdisco@server ~]$ ~netdisco/bin/netdisco-db-deploy --redeploy-all
DBIx::Class::Schema::Versioned::_on_connect(): Versions out of sync. This is 41, your database contains version 40, please call upgrade on your Schema. at /home/netdisco/perl5/lib/perl5/DBICx/Sugar.pm line 85 NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "dbix_class_schema_versions_pkey" for table "dbix_class_schema_versions" DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (2) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (3) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (4) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (5) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (6) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (7) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (8) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (9) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (10) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (11) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (12) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (13) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (14) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (15) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (16) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (17) is lower than the schema version (41). Attempting upgrade. NOTICE: ALTER TABLE / ADD PRIMARY KEY will create implicit index "admin_pkey" for table "admin" DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (18) is lower than the schema version (41). Attempting upgrade. NOTICE: ALTER TABLE / ADD PRIMARY KEY will create implicit index "node_pkey" for table "node" DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (19) is lower than the schema version (41). Attempting upgrade. NOTICE: ALTER TABLE / ADD PRIMARY KEY will create implicit index "node_wireless_pkey" for table "node_wireless" DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (20) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (21) is lower than the schema version (41). Attempting upgrade. NOTICE: ALTER TABLE / ADD UNIQUE will create implicit index "topology_dev1_port1" for table "topology" DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (22) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (23) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (24) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (25) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (26) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (27) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (28) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (29) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (30) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (31) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (32) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (33) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (34) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (35) is lower than the schema version (41). Attempting upgrade. NOTICE: ALTER TABLE / ADD PRIMARY KEY will create implicit index "device_port_vlan_pkey" for table "device_port_vlan" DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (36) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (37) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (38) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (39) is lower than the schema version (41). Attempting upgrade. DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (40) is lower than the schema version (41). Attempting upgrade.

--- End Message ---
--- Begin Message --- 
Use netdisco-deploy, not netdisco-db-deploy.

Andy

On Jun 1, 2017 8:41 PM, "Nicolás Matsunaga" <[email protected]> wrote:

Hi,

I have no knowledge of PostreSQL.
Sometime ago I updated netdisco from version 2.033002 to  2.034001
When I did the upgrade I executed "~/bin/netdisco-deploy"
Now I tried it to give netdisco another chance (because it is not working
fine) and I want to have the correct DB version and I am not able to update
the schema doing "~netdisco/bin/netdisco-db-deploy "

How can I fix this?

Regards,
Nicolás


[netdisco@server ~]$ ~netdisco/bin/netdisco-db-deploy
DBIx::Class::Schema::Versioned::_on_connect(): Versions out of sync. This
is 41, your database contains version 40, please call upgrade on your
Schema. at /home/netdisco/perl5/lib/perl5/DBICx/Sugar.pm line 85
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (40) is
lower than the schema version (41). Attempting upgrade.


[netdisco@server ~]$ ~netdisco/bin/netdisco-db-deploy --redeploy-all
DBIx::Class::Schema::Versioned::_on_connect(): Versions out of sync. This
is 41, your database contains version 40, please call upgrade on your
Schema. at /home/netdisco/perl5/lib/perl5/DBICx/Sugar.pm line 85
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index
"dbix_class_schema_versions_pkey" for table "dbix_class_schema_versions"
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (2) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (3) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (4) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (5) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (6) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (7) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (8) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (9) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (10) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (11) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (12) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (13) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (14) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (15) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (16) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (17) is
lower than the schema version (41). Attempting upgrade.
NOTICE:  ALTER TABLE / ADD PRIMARY KEY will create implicit index
"admin_pkey" for table "admin"
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (18) is
lower than the schema version (41). Attempting upgrade.
NOTICE:  ALTER TABLE / ADD PRIMARY KEY will create implicit index
"node_pkey" for table "node"
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (19) is
lower than the schema version (41). Attempting upgrade.
NOTICE:  ALTER TABLE / ADD PRIMARY KEY will create implicit index
"node_wireless_pkey" for table "node_wireless"
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (20) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (21) is
lower than the schema version (41). Attempting upgrade.
NOTICE:  ALTER TABLE / ADD UNIQUE will create implicit index
"topology_dev1_port1" for table "topology"
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (22) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (23) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (24) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (25) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (26) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (27) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (28) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (29) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (30) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (31) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (32) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (33) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (34) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (35) is
lower than the schema version (41). Attempting upgrade.
NOTICE:  ALTER TABLE / ADD PRIMARY KEY will create implicit index
"device_port_vlan_pkey" for table "device_port_vlan"
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (36) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (37) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (38) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (39) is
lower than the schema version (41). Attempting upgrade.
DBIx::Class::Schema::Versioned::upgrade_single_step(): DB version (40) is
lower than the schema version (41). Attempting upgrade.


------------------------------------------------------------
------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Netdisco mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/netdisco-users

--- End Message ---
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Netdisco mailing list - Digest Mode
[email protected]
https://lists.sourceforge.net/lists/listinfo/netdisco-users

Reply via email to