Re: Test Lab Best Practices

[Aaron1]

I love the built-in Wireshark capability in EVE-NG.  BTW, EVE-NG Community is 
free.  You just have to get images for anything you want to emulate.  Virtual 
images for various vendor products are sometimes freely available, with trail 
licenses.  For instance Junipers vMX was freely available for a while with a 60 
day license.  …also vSRX, vQFX, and the new vJunos-switch (I think vEX). 

Aaron

> On Sep 28, 2023, at 3:16 PM, Mark Prosser  wrote:
> 
> ++ all that was said thus far. Physical equipment with console access is the 
> best way to test software/firmware issues. As for virtualization, it's great 
> for expanding your topology quickly.
> 
> Use a virtual bridge in GNS3 or EVE-NG and you can make your smaller 
> footprint physical lab into a larger topology with ease -- especially around 
> cabling. It also allows you to do packet generation & link simulation (packet 
> loss, jitter) much easier. You can even couple it with T-Rex.
> 
> - Mark
> 

Re: Test Lab Best Practices

[Brad Dreisbach]

On Thu, Sep 28, 2023 at 10:14:24AM -0400, Kenneth Vedder wrote:

Hello NANOG,

We have been struggling with firmware bugs from a specific router vendor. I
am looking to set up a test lab of our core network and a few remote site
routers.  Protocols would include SR-MPLS, ISIS, EVPN MPLS and L3VPN with a
little OSPF sprinkled in. I'd be grateful for any tips or resources anyone
has that might cover testing strategies and/or best practices.


if you are just testing control plane features the virtualized stuff works
well. i personally use vrnetlab.

if you are trying to test hw features and use actual hardware:
remote controlled power
optical cross connect boxes(calient/telescent)
if you are testing optical stuff with distance limitation something like a
timbercon spool
ixia/spirent

-b

Re: Test Lab Best Practices

[Mark Prosser]

++ all that was said thus far. Physical equipment with console access is 
the best way to test software/firmware issues. As for virtualization, 
it's great for expanding your topology quickly.


Use a virtual bridge in GNS3 or EVE-NG and you can make your smaller 
footprint physical lab into a larger topology with ease -- especially 
around cabling. It also allows you to do packet generation & link 
simulation (packet loss, jitter) much easier. You can even couple it 
with T-Rex.


- Mark

Re: Test Lab Best Practices

[Aaron Gould]

I agree with others here...

Physical lab - gotta have console server for the most control - perle 
console server is good, and also good ole fashion cisco terminal server 
(2509/2511 or 2600 with asynch module)


Virtual labs are great for testing features and functionality

- Juniper vLabs

- Cisco DevNet sandbox

- Cisco CML (i think fka VIRL)

- EVE-NG

- GNS3

I use these virtual environments a lot and do videos about them on my 
youtube channel, where I try to cover some SP-related topics.  Hope it helps


https://jlabs.juniper.net/vlabs/

https://developer.cisco.com/site/sandbox/

https://www.youtube.com/@aarontechtalk

https://www.youtube.com/playlist?list=PL2ZMKm7ZEEWI8YyRWm9fnYNtRaV-fi-7x

https://www.youtube.com/playlist?list=PL2ZMKm7ZEEWLMVxuZqeXzciRu59C02NAc


-Aaron


On 9/28/2023 9:14 AM, Kenneth Vedder wrote:

Hello NANOG,

We have been struggling with firmware bugs from a specific router 
vendor. I am looking to set up a test lab of our core network and a 
few remote site routers.  Protocols would include SR-MPLS, ISIS, EVPN 
MPLS and L3VPN with a little OSPF sprinkled in. I'd be grateful for 
any tips or resources anyone has that might cover testing strategies 
and/or best practices.


Thanks,
Ken


--
-Aaron

Re: Test Lab Best Practices

[Tom Beecher]

Appliance virtualization is perfectly acceptable for a lot of things. But
there are large sets of problems that you will never catch that way.

To the OP :

With respect to 'strategies' :

1. Test something to make sure it works.
2. Then test it to see where and how it breaks.

Lots of people do #1. Not enough do #2. (Looking at you, equipment vendors.
)



On Thu, Sep 28, 2023 at 12:04 PM Nickolas Stevermer via NANOG <
nanog@nanog.org> wrote:

> If you're able to virtualize some of these appliances, almost any
> virtualization platform can be setup to connect nodes together. However I
> really enjoy using Eve-ng . It's a network
> virtualization web app that can be deployed on bare metal or in a VM. It
> lets you diagram, connect and configure setups and do testing with
> different network nodes. What is pretty nice with this is you can then just
> click on any of the connected links and wireshark pops up and you're
> viewing packet captures instantly. It kind of reminds me of Cisco's VIRL,
> but with maybe less overhead?  It has enabled me to run and connect several
> vendor appliances and build up configuration prior to acquiring the
> physical hardware for deployment.
>
> - Nick Stevermer
>
>
> On Thu, Sep 28, 2023 at 10:46 AM Mel Beckman  wrote:
>
>> In any lab,I find concurrent access to serial ports is still an essential
>> diagnostic tool. In a pinch you can get a used Cisco 2811 for $100, but
>> there are multiport devices from lots of vendors. These let you SSH into
>> the server and then connect to any serial port, giving you separate serial
>> port windows all on the same screen. I’ve become fond of the WiFi-capable
>> multiport modules from get-console.com. The ability to record logs from
>> these serial ports in real-time helps a lot for documenting regression
>> tests.
>>
>>  -mel beckman
>>
>> > On Sep 28, 2023, at 7:25 AM, Kenneth Vedder  wrote:
>> >
>> > 
>> > Hello NANOG,
>> >
>> > We have been struggling with firmware bugs from a specific router
>> vendor. I am looking to set up a test lab of our core network and a few
>> remote site routers.  Protocols would include SR-MPLS, ISIS, EVPN MPLS and
>> L3VPN with a little OSPF sprinkled in. I'd be grateful for any tips or
>> resources anyone has that might cover testing strategies and/or best
>> practices.
>> >
>> > Thanks,
>> > Ken
>>
>
>
> --
> *Nick Stevermer*
> Network Engineer
> Technology Department
> Duluth Public Schools
> 713 Portia Johnson Dr.
> Duluth, MN 55811
> Phone: 218-336-8754 ext 1227
>
>
>
> Confidentiality Notice: This E-mail message, including any attachments, is
> for the sole use of the intended recipient(s) and may contain confidential
> and privileged information. Any unauthorized review, use, disclosure or
> distribution is prohibited. If you are not the intended recipient, please
> contact the sender by reply E-mail and destroy all copies of the original
> message.

Re: Test Lab Best Practices

[Nickolas Stevermer via NANOG]

If you're able to virtualize some of these appliances, almost any
virtualization platform can be setup to connect nodes together. However I
really enjoy using Eve-ng . It's a network
virtualization web app that can be deployed on bare metal or in a VM. It
lets you diagram, connect and configure setups and do testing with
different network nodes. What is pretty nice with this is you can then just
click on any of the connected links and wireshark pops up and you're
viewing packet captures instantly. It kind of reminds me of Cisco's VIRL,
but with maybe less overhead?  It has enabled me to run and connect several
vendor appliances and build up configuration prior to acquiring the
physical hardware for deployment.

- Nick Stevermer


On Thu, Sep 28, 2023 at 10:46 AM Mel Beckman  wrote:

> In any lab,I find concurrent access to serial ports is still an essential
> diagnostic tool. In a pinch you can get a used Cisco 2811 for $100, but
> there are multiport devices from lots of vendors. These let you SSH into
> the server and then connect to any serial port, giving you separate serial
> port windows all on the same screen. I’ve become fond of the WiFi-capable
> multiport modules from get-console.com. The ability to record logs from
> these serial ports in real-time helps a lot for documenting regression
> tests.
>
>  -mel beckman
>
> > On Sep 28, 2023, at 7:25 AM, Kenneth Vedder  wrote:
> >
> > 
> > Hello NANOG,
> >
> > We have been struggling with firmware bugs from a specific router
> vendor. I am looking to set up a test lab of our core network and a few
> remote site routers.  Protocols would include SR-MPLS, ISIS, EVPN MPLS and
> L3VPN with a little OSPF sprinkled in. I'd be grateful for any tips or
> resources anyone has that might cover testing strategies and/or best
> practices.
> >
> > Thanks,
> > Ken
>


-- 
*Nick Stevermer*
Network Engineer
Technology Department
Duluth Public Schools
713 Portia Johnson Dr.
Duluth, MN 55811
Phone: 218-336-8754 ext 1227

-- 
Confidentiality Notice: This E-mail message, including any attachments, is 
for the sole use of the intended recipient(s) and may contain confidential 
and privileged information. Any unauthorized review, use, disclosure or 
distribution is prohibited. If you are not the intended recipient, please 
contact the sender by reply E-mail and destroy all copies of the original 
message.

Re: Test Lab Best Practices

[Mel Beckman]

In any lab,I find concurrent access to serial ports is still an essential 
diagnostic tool. In a pinch you can get a used Cisco 2811 for $100, but there 
are multiport devices from lots of vendors. These let you SSH into the server 
and then connect to any serial port, giving you separate serial port windows 
all on the same screen. I’ve become fond of the WiFi-capable multiport modules 
from get-console.com. The ability to record logs from these serial ports in 
real-time helps a lot for documenting regression tests. 

 -mel beckman

> On Sep 28, 2023, at 7:25 AM, Kenneth Vedder  wrote:
> 
> 
> Hello NANOG,
> 
> We have been struggling with firmware bugs from a specific router vendor. I 
> am looking to set up a test lab of our core network and a few remote site 
> routers.  Protocols would include SR-MPLS, ISIS, EVPN MPLS and L3VPN with a 
> little OSPF sprinkled in. I'd be grateful for any tips or resources anyone 
> has that might cover testing strategies and/or best practices. 
> 
> Thanks,
> Ken

Test Lab Best Practices

[Kenneth Vedder]

Hello NANOG,

We have been struggling with firmware bugs from a specific router vendor. I
am looking to set up a test lab of our core network and a few remote site
routers.  Protocols would include SR-MPLS, ISIS, EVPN MPLS and L3VPN with a
little OSPF sprinkled in. I'd be grateful for any tips or resources anyone
has that might cover testing strategies and/or best practices.

Thanks,
Ken