Send netdisco-users mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.sourceforge.net/lists/listinfo/netdisco-users
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of netdisco-users digest..."
Today's Topics:
1. Re: Listing ports with connected nodes never finishes (Andy Ruhl)
2. Re: Listing ports with connected nodes never finishes
(Oliver Gorwits)
3. Re: Listing ports with connected nodes never finishes
(Oliver Gorwits)
--- Begin Message ---
Just as another data point:
I renamed the current database, loaded the users and user_log into a
new one called "netdisco" (after doing netdisco-deploy of course) and
rediscovered all devices.
Our largest switch - a 4 chassis Juniper EX8216 virtual chassis -
takes a while to return a result for ports with connected nodes and
devices, but it eventually does.
I'm not 100% sure if this is a "size of result" problem or a "bad data" problem.
I counted rows in the device_port and device_port_vlan tables between
old (broken) and new, and it's not fantastically different. There are
a few less switches in the new database due to cleanup of auto
discovered stuff we don't need.
Andy
On Sat, May 13, 2017 at 6:17 AM, Christian Ramseyer <[email protected]> wrote:
> On 06/05/17 18:00, Oliver Gorwits wrote:
>
>> We could do with someone with good SQL chops taking a look at that -
>> maybe some correlated subqueries or common table expressions need
>> adding. Also we don't do paging in the UI so it gets all results and
>> sends them to the client in one go.
>>
>
> Hi Oliver & Andy
>
> I think this is the same issue we already talked about a while ago here:
> https://sourceforge.net/p/netdisco/mailman/message/34714207/
>
> Main problem seemed to not be the SQL, but that the whole result is
> fetched from the database and this creates an astronomical amount of
> DBIx::Class objects for big switches.
>
> Steven Xu mentioned a private patch (presumably adding pagination) that
> he "just needed to clean up a bit", but unfortunately that probably
> didn't happen. Maybe we could get him to just release a diff.
>
> Christian
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Netdisco mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/netdisco-users
--- End Message ---
--- Begin Message ---
Hi Andy,
On 2017-05-13 15:08, Andy Ruhl wrote:
I renamed the current database, loaded the users and user_log into a
new one called "netdisco" (after doing netdisco-deploy of course) and
rediscovered all devices.
Many thanks for taking the time to run this test, we really appreciate
it.
To narrow down on whether it's bad data, we could trace the web engine
(log:debug and DBIC_TRACE=1) and see where it pauses. If this is in the
SQL then it's likely to be volume of data that's the issue. If it pauses
in the web engine it could be anything, but there are known hotspots in
the code, before page render, for sure. My gut feeling is still that
it's the volume rather than any issue with a bad field.
One problem I have as a dev is that where I work now doesn't have large
numbers of nodes so I don't stand a chance of reproducing this issue. If
anyone is willing and able to give me a copy of their database, it would
help (even better where it exhibits this behaviour!). I realise the
security concerns in this, though. Anyone reading - me an email off-list
to discuss.
regards,
oliver.
Our largest switch - a 4 chassis Juniper EX8216 virtual chassis -
takes a while to return a result for ports with connected nodes and
devices, but it eventually does.
I'm not 100% sure if this is a "size of result" problem or a "bad data"
problem.
I counted rows in the device_port and device_port_vlan tables between
old (broken) and new, and it's not fantastically different. There are
a few less switches in the new database due to cleanup of auto
discovered stuff we don't need.
Andy
On Sat, May 13, 2017 at 6:17 AM, Christian Ramseyer <[email protected]>
wrote:
On 06/05/17 18:00, Oliver Gorwits wrote:
We could do with someone with good SQL chops taking a look at that -
maybe some correlated subqueries or common table expressions need
adding. Also we don't do paging in the UI so it gets all results and
sends them to the client in one go.
Hi Oliver & Andy
I think this is the same issue we already talked about a while ago
here:
https://sourceforge.net/p/netdisco/mailman/message/34714207/
Main problem seemed to not be the SQL, but that the whole result is
fetched from the database and this creates an astronomical amount of
DBIx::Class objects for big switches.
Steven Xu mentioned a private patch (presumably adding pagination)
that
he "just needed to clean up a bit", but unfortunately that probably
didn't happen. Maybe we could get him to just release a diff.
Christian
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Netdisco mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/netdisco-users
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Netdisco mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/netdisco-users
--- End Message ---
--- Begin Message ---
Oh, I forgot to mention... your fresh install will be missing archived
data, which may or may not affect the DB search and resultset size.
regards,
Oliver.
On 2017-05-13 15:08, Andy Ruhl wrote:
Just as another data point:
I renamed the current database, loaded the users and user_log into a
new one called "netdisco" (after doing netdisco-deploy of course) and
rediscovered all devices.
Our largest switch - a 4 chassis Juniper EX8216 virtual chassis -
takes a while to return a result for ports with connected nodes and
devices, but it eventually does.
I'm not 100% sure if this is a "size of result" problem or a "bad data"
problem.
I counted rows in the device_port and device_port_vlan tables between
old (broken) and new, and it's not fantastically different. There are
a few less switches in the new database due to cleanup of auto
discovered stuff we don't need.
Andy
On Sat, May 13, 2017 at 6:17 AM, Christian Ramseyer <[email protected]>
wrote:
On 06/05/17 18:00, Oliver Gorwits wrote:
We could do with someone with good SQL chops taking a look at that -
maybe some correlated subqueries or common table expressions need
adding. Also we don't do paging in the UI so it gets all results and
sends them to the client in one go.
Hi Oliver & Andy
I think this is the same issue we already talked about a while ago
here:
https://sourceforge.net/p/netdisco/mailman/message/34714207/
Main problem seemed to not be the SQL, but that the whole result is
fetched from the database and this creates an astronomical amount of
DBIx::Class objects for big switches.
Steven Xu mentioned a private patch (presumably adding pagination)
that
he "just needed to clean up a bit", but unfortunately that probably
didn't happen. Maybe we could get him to just release a diff.
Christian
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Netdisco mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/netdisco-users
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Netdisco mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/netdisco-users
--- End Message ---
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Netdisco mailing list - Digest Mode
[email protected]
https://lists.sourceforge.net/lists/listinfo/netdisco-users
