Re: Some doubts on large scale BGP/AS design and black hole routing risk

[Bill Woodcock]

Are respondents to suppose that the customer base and address space are evenly 
divided between the two cities, and that the ISP is too clueless to originate 
each /23 from the city that uses it, in iBGP?


-Bill


> On Apr 3, 2016, at 15:04, "magicb...@hotmail.com"  
> wrote:
> 
> Hi everybody!
> 
> 
> as part of laboratory work at the university,  I'm working on a BGP design 
> study, and I would like to post some questions regarding IP address space 
> allocation and its impact on BGP which are breaking my mind :)
> 
> Let's suppose we have an ISP/AS with two POPs: PARIS and LONDON. These two 
> POPs are connected with redundant leased lines. Each POP has a BGP router 
> speaking eBGP to different ISP providers/upstreams and also, each POP run its 
> own OSPF area/ISIS area. Something like this:
> 
> 
>  ---eBGP---===redundant leased lines (ospf 
> area0)===---eBGP---
> 
> Now, this AS/ISP gets one /22 prefix from it RIR (RIPE in this case), and 
> starts to announce it to its upstreams in PARIS and LONDON at the same time.
> 
> 
> My questions are:
> 
> 1. What could happen in the case of total failure in the redundant leased 
> lines? Black hole routing between POPs?
> 
> 2. What are the best design methods to avoid this scenario?
> 
>   2.1: adding a third POP creating a triangle? What if a POP looses 
> connection with the other two POPs at the same time? Another black hole?
> 
>   2.2: requesting another prefix and allocating 1:1 prefix:POP, so in the 
> scenario each POP only would announce its prefix to the upstreams?
> 
>   2.3: other?
> 
> 
> 
> Thanks in advance!
> J.
> 

Re: Someone Please Help Me Understand

[Faisal Imtiaz]

Hi Eric,

With this type of connectivity you have to pay attention to Traffic 
Engineering...

And when I say, traffic engineering, I mean both ways.. how you are sending 
traffic to them
along with how they are sending traffic to you... (sometimes a bit more 
challenging to do).

I will give you two specific example, just to illustrate the point...

We are located in the east coast, we have ip transit to Cogent network, via one 
intermediary ASN.
We also have IP Transit with GTT and Hibernia networks.
We also have direct peering on multiple Peering Fabrics.

1st cases...
We have our outbound traffic engineered to prefer direct routes.. e.g. when 
sending traffic to Cogent, we send
it out via the intermediary ASN to Cogent.
However when traffic is coming back from Cogent they see our prefixes via 
intermediary ASN as well as Hibernia Networks,
since Hibernia networks is a lower ASN, they prefer that route 
So, one can say, no big deal, except, Hibernia Networks connects to Cogent on 
the West Coast !... so our return traffic is going
from the east coast to west coast and them back to east coast 
So one can easily say... Houston we have a problem !...

2nd Case..
We are peered with some networks at Telx TIE, via one of our (intermediary) 
ASN...So while we can send traffic over to that network via our ASN, however 
that networks sees our prefixes via our (intermediary) ASN as Hibernia as 
well Hibernia being a lower ASN, they send traffic back to us via them...

In both cases we use communities to take corrective action

Moral of the story is. just because you have multiple peers, and peer with 
folks on the Peering Fabric, the default configuration of BGP will not 
AUTOMAGICALY  optimize the paths in your favor 

And thus the condition you describe will be the result...

Faisal Imtiaz
Snappy Internet & Telecom
7266 SW 48 Street
Miami, FL 33155
Tel: 305 663 5518 x 232

Help-desk: (305)663-5518 Option 2 or Email: supp...@snappytelecom.net

- Original Message -
> From: "Eric Rogers" 
> To: "nanog list" 
> Sent: Saturday, April 2, 2016 1:54:40 PM
> Subject: Someone Please Help Me Understand

> Ok, I'm trying to learn, so bear with me.
> 
> 
> 
> We are an ISP in Indianapolis that has full routes from 3 different
> providers HE.Net in Columbus OH being one.  We also are peered with 2
> peering exchanges, including EquinixIX in Chicago.  The problem is
> Instagram and Facebook (same company, I know) for our customers seems
> very slow.
> 
> 
> 
> This is where I need a way to troubleshoot/understand more.  I did a
> traceroute to the IP that is serving the pictures, and it resolves to
> the FBCDN servers in Dallas, and is showing packet loss and pings once
> it hits Dallas, and are in the 1xxs of ms.
> 
> 
> 
> Tracing route to instagram-p3-shv-01-dfw1.fbcdn.net [31.13.66.52]
> 
> over a maximum of 30 hops:
> 
> 
> 
>  1 4 ms 3 ms 4 ms  10.7.0.1
> 
>  220 ms43 ms42 ms  inmtvlobs-rtr-01.dynamic.pdsconnect.me
> [192.69.57.1]
> 
>  325 ms47 ms29 ms
> inmtvlmwt-rtr-01.infrastructure.pdsconnect.me [192.69.48.162]
> 
>  446 ms32 ms58 ms
> inindyhen-core1.infrastructure.pdsconnect.me [192.69.48.193]
> 
>  536 ms53 ms51 ms  ge2-4.core1.cmh1.he.net [184.105.32.1]
> 
>  647 ms41 ms75 ms  10ge1-2.core1.chi1.he.net
> [184.105.222.165]
> 
>  757 ms57 ms53 ms  100ge14-1.core2.chi1.he.net
> [184.105.81.97]
> 
>  857 ms73 ms84 ms  100ge12-1.core1.mci3.he.net
> [184.105.81.209]
> 
>  975 ms73 ms   102 ms  10ge15-6.core1.dal1.he.net
> [184.105.222.10]
> 
> 1093 ms   103 ms92 ms  eqix-da1.facebook.com [206.223.118.176]
> 
> 11   102 ms   101 ms * psw01c.dfw1.tfbnw.net [173.252.65.196]
> 
> 1292 ms97 ms   105 ms  msw1aq.01.dfw1.tfbnw.net [204.15.21.89]
> 
> 13   110 ms *   98 ms  instagram-p3-shv-01-dfw1.fbcdn.net
> [31.13.66.52]
> 
> 
> 
> Since I am peered with the route servers in EquinixIX Chicago, shouldn't
> the data be coming from there, or at least hit their routers?  In my
> trace, it shows HE to Chicago, then to Dallas.  How does FB decide what
> IP the content gets displayed from, and is there anything I can do as a
> provider?  If it is DNS, I can obviously clear the cache to see if it
> gets new IPs.  If I'm not getting FB peering IPs in Chicago, do I need
> to peer directly?  Should I get FaceBook involved?
> 
> 
> 
> Eric Rogers
> 
> PDS Connect
> 
> (317) 831-3000 x200

Re: What services does Microsoft AS8075 provide when peering at IXPs?

[Eric A Louie via NANOG]

My direct peering is within my control, the traffic has to follow basically the 
same path internally to Internet and IXP.  Internet path is definitely 
variable.  IXP path is definitely fixed.  It's already there, just needed to 
know what Microsoft provided (if anyone knew).  Actually, a MS peering engineer 
answered the question for me privately.
My marketing people aren't terribly concerned with security, that wasn't the 
issue.  The issue was shorter path/better performance.  I'm still not clear if 
it's public or private Azure that the customer is trying to access.
 

On Sunday, April 3, 2016 4:04 PM, "valdis.kletni...@vt.edu" 
 wrote:
 
 

 On Fri, 01 Apr 2016 18:02:56 -, Eric A Louie via NANOG said:
> I suppose we have a customer who is an Azure customer that wants to know if 
> their Azure traffic will stay in our network or still go through the Internet.

As a practical matter, if they're using the answer for a security baseline,
they're doing it wrong - they should be planning that based on the assumption
that their traffic *will* ride the rails of the commodity Internet (due to
outages or whatever).

Similarly, if they're looking at it for performance/latency, they need to
fix their assumptions - your direct peering can be slow and congested, while
there's actually a longer but faster path through someplace else


 

Re: What services does Microsoft AS8075 provide when peering at IXPs?

[Mike Hammett]

"your direct peering can be slow and congested, while 
there's actually a longer but faster path through someplace else" 

Possible, but unlikely for most networks. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


- Original Message -

From: "Valdis Kletnieks"  
To: "Eric A Louie"  
Cc: "NANOG List"  
Sent: Sunday, April 3, 2016 6:04:14 PM 
Subject: Re: What services does Microsoft AS8075 provide when peering at IXPs? 

On Fri, 01 Apr 2016 18:02:56 -, Eric A Louie via NANOG said: 
> I suppose we have a customer who is an Azure customer that wants to know if 
> their Azure traffic will stay in our network or still go through the 
> Internet. 

As a practical matter, if they're using the answer for a security baseline, 
they're doing it wrong - they should be planning that based on the assumption 
that their traffic *will* ride the rails of the commodity Internet (due to 
outages or whatever). 

Similarly, if they're looking at it for performance/latency, they need to 
fix their assumptions - your direct peering can be slow and congested, while 
there's actually a longer but faster path through someplace else 

Re: What services does Microsoft AS8075 provide when peering at IXPs?

[Valdis . Kletnieks]

On Fri, 01 Apr 2016 18:02:56 -, Eric A Louie via NANOG said:
> I suppose we have a customer who is an Azure customer that wants to know if 
> their Azure traffic will stay in our network or still go through the Internet.

As a practical matter, if they're using the answer for a security baseline,
they're doing it wrong - they should be planning that based on the assumption
that their traffic *will* ride the rails of the commodity Internet (due to
outages or whatever).

Similarly, if they're looking at it for performance/latency, they need to
fix their assumptions - your direct peering can be slow and congested, while
there's actually a longer but faster path through someplace else



pgp0aAYcBI4i5.pgp
Description: PGP signature

Re: Some doubts on large scale BGP/AS design and black hole routing risk

[Mark Tinka]

On 31/Mar/16 10:12, magicb...@hotmail.com wrote:
 
>
>
> My questions are:
>
> 1. What could happen in the case of total failure in the redundant
> leased lines? Black hole routing between POPs?

If you have redundant backhaul that completely fails, you've got real
problems.

However, if that does happen, any traffic coming into each individual
PoP destined for users in the other PoP will fail. Only traffic
terminating for customers at that PoP will succeed.


>
> 2. What are the best design methods to avoid this scenario?

Work on your backhaul.

Originate specific routes that cover customers present in each PoP, with
the aggregate as a backup route.

You can run a tunnel across the Internet to simulate a backbone between
both PoP's, using your side of your upstream's IP addresses as the
tunnel end-point. Not elegant, but keeps you up.

>
>2.1: adding a third POP creating a triangle? What if a POP looses
> connection with the other two POPs at the same time? Another black hole?

Your fixation on a complete backhaul outage is interesting.

Purchase backhaul from different service providers to increase your
chances of uptime.


>
>2.2: requesting another prefix and allocating 1:1 prefix:POP, so in
> the scenario each POP only would announce its prefix to the upstreams?

See above re: originating more specific routes based on the customers
you have at each PoP.


>
>2.3: other?

Work harder on your backhaul.

Yes, bad things can happen, and they do happen. But more than likely, if
a 3-PoP network loses all connectivity from each other, I think routing
will be a much smaller problem to solve in the grand scheme of things.

Mark.

Re: Colocation Server Lifts

[Sam Oduor]

Yes, I would expect a lift at a colo but in terms of regulations (safety) I
do not think it is a mandatory requirement for most colo's

Allowing customers to use them can be a yes or no ; it requires some basic
operational skills to operate - you just cant trust a client visiting to
use it unless some vetting is in place for this.

It should be free and a datacentre operator needs to assist or at-least
supervise in usage.

The weight depends on the model of the lift .

Some helpful resources:-

https://www.youtube.com/watch?v=5uLWVMOfY0U

http://www.serverlift.com/





On Tue, Mar 29, 2016 at 3:23 PM, Jason Lee  wrote:

> Hi NANOG community,
>
> A few questions I have for the community regarding server lifts at colo
> facilities.
>
> 1. Is a server lift something you would typically expect a colo facility to
> provide?
>
> if yes,
>
> 2. Do colo facilities typically allow customers to just use them or provide
> an operator?
> 3. Is it a free offering or something they rent out?
> 4. What would be the typical device weight you would lift?
> 5. What would be the max device weight you would lift?
>
> Thanks,
>
> Jason
>



-- 
Samson Oduor

Re: Capacity planning , transit vs last mile

[takashi tome]

Like. Good question.
There are a lot of papers on traffic model, but it is still an open issue...

takashi.tome


2016-03-31 3:51 GMT-03:00 Jean-Francois Mezei :

>
> Canada is to hold a 3 week long hearing on discussing whether the
> internet is important and whether the farcical 5/1 speed promoted by the
> government is adequate.
>
> In this day and age, it would be easy to just set FTTP as target
> technology and be done with it, but too many want to have a policy that
> is technologically neutral.
>
> To this end, I will not only be proposing that subsidized deployments
> not only meet advertised service speed standards, but also a capacity
> per end user metric for the last mile technology  as well as for the
> backhaul/transit.
>
> (One of the often subsidized companies deploys fixed wireless which
> delivers the advertised speed for the first week, but routinely gets
> oversubscribed after a while and customers feel like on dialup.)
>
>
> I know that for sufficiently large ISPs, they currently provision just
> over 1mbps of transit capacity per end user (so 800-1000 customers per
> 1gbps of transit). The number rises by over 30% a year as usage grows.
> (The CRTC can get exact figure from telecom operators and generate
> aggregate industry-wide growth in traffic to do yearly standard
> adjustment).
>
> QUESTION:
>
> Say the policy is 1mbps per customer if 1000 customer or more.  Is there
> some formula (approx or precise) to calculate how that 1mbps changes for
> smaller samples ? (like 500 customers, 200 ? )
>
>
>
> And on the last mile portion where one has typically few users on each
> shared capacity segment (fixed wireless, FTTP, cable), are there fairly
> standard oversubscription ratios based on average service speed that is
> sold in that neighbourhood ? (for instance if I have 100 customers with
> average subscibed speed of 15mbps, how much capacity should the antenna
> serving those customers have ?
>
>
> I realise that each ISP guards its oversubscription ratios as very
> proprietary, but aren't there generic industry-wide recommendations ? My
> goal is to have some basic standards that prevent gross over
> subscription that result in unusable service.
>
> As well, I want that a company pitching a broadband deployment be able
> to demonstrate that the technology being deployed will last X years
> because it has sufficient capacity to handle the number  of customers as
> well as the predicted growth in usage each year.
>
>
> Any help ? comments on whether this is crazy ? sanity check ?
>
>
>
>
>
>

Re: PlayStation Network blocking an IP

[J]

I've been seeing abuse reports from Sony lately, indicating IPs are blacklisted 
on the Playstation Network, mainly for attempted account compromise.

Indicating to check for traffic towards some endpoints, resolve the complaint 
else blacklist again, etc, etc.  The address they direct inquiries to is 
snei-noc-ab...@am.sony.com - perhaps this is a reason and vector for your issue 
as well?

 On Fri, 01 Apr 2016 17:31:16 -0500 Velocity Lists 
voli...@staff.velocityonline.net wrote  

Can someone form Sony's Playstation network give me call or contact me 
offlist. 
 
One of our apartment complexes has been reporting errors of PS4s not 
working for a few days then they start working again. 
 
PSN Support is telling the users to call us. 
We have diagnosed it and PSN is blocking the IP of the complex and it has 
nothing to do with us. 
 
 
Velocity Online 
Rodger Lewis rcle...@velocityonline.net 
850-205-4638 x201 

RE: Fri AM AT outage

[Lowe,Chris]

When I talked to Frontier about our Office in Tampa, I was told there was a 
fiber cut that was effecting SoCal and Florida.  Not sure how the 2 were 
related but that is what I was told.  Our Tampa office has been down as well 
since 7:00am local time.

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Mel Beckman
Sent: Friday, April 01, 2016 2:25 PM
To: James Laszko
Cc: North American Network Operators' Group
Subject: Re: Fri AM AT outage

Was this in SoCal? I had 40% packet loss Friday midnight to 11am on my FIOS 
sites. 

 -mel beckman

> On Apr 1, 2016, at 2:18 PM, James Laszko  wrote:
> 
> I just saw that some of our AT tickets were updated as "Frontier is 
> experiencing an outage in Oxnard, CA"
> 
> 
> James
> 
> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Jay R. Ashworth
> Sent: Friday, April 01, 2016 13:43
> To: North American Network Operators' Group
> Subject: Fri AM AT outage
> 
> I heard speculation from many quarters that this may have been related to the 
> Verizon splashcut to Frontier -- which, regardless of what Frontier was 
> telling us, I was pretty sure would be more than just varying which light 
> switch for a building sign was the one turned on.
> 
> Has anyone heard anything they're allowed to repeat, yet, which confirms or 
> denies?
> 
> On a related[1] topic: I see on Craigs that Frontier is hiring through a sub 
> for transport staff for their new NOC, which I think is going to be located 
> in SPBGFLXA89H; that building has about 3 completely empty floors, and has 
> for over 20 years.  (OK, it did when I toured it in 91; dunno what's there 
> now.  :-)
> 
> Cheers,
> -- jra
> [1]maybe
> 
> -- 
> Jay R. Ashworth  Baylink   
> j...@baylink.com
> Designer The Things I Think   RFC 2100
> Ashworth & Associates   http://www.bcp38.info  2000 Land Rover DII
> St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647 1274

What services does Microsoft AS8075 provide when peering at IXPs?

[Eric A Louie via NANOG]

I had this question posed by a marketing type in my office.  Does anyone know 
the answer?
Is it microsoft.com, msdn, outllook365, msn.com, outlook.com, azure, windows 
updates, xbox?  what else is possibly covered or omitted in their peering 
service?  I suppose we have a customer who is an Azure customer that wants to 
know if their Azure traffic will stay in our network or still go through the 
Internet.
I've tried asking peer...@microsoft.com twice, but it looks like they have 
their ignore filter up.

Thanks, -e-

Re: how to deal with port scan and brute force attack from AS 8075 ?

[cyrus ramirez via NANOG]

You could use Shields Up to view your vulnerabilities... obvious ones, and 
remedy... Cyrus Ramirez

 

On Thursday, March 31, 2016 10:21 AM, "valdis.kletni...@vt.edu" 
 wrote:
 

 On Thu, 31 Mar 2016 10:02:05 +0200, "marcel.duregards--- via NANOG" said:

> We consider port scan and brute force on ssh port as an attack, and even

So explain to me why you don't have ACLs that silently drop inbound SYN
packets on port 22 from outside your allocated address space?  (And if
you can't do it at your border because you sub-allocate address space
to customers, figure out how to use iptables or similar to block it on
the target hosts, or only apply the ACL for your own subnets).

If you have a *legitimate* business case for needing to SSH in from outside,
there are fine products such as OpenVPN (and not-so-fine like the one we
have in production - although it's mostly usable too, and achieves the goal
of presenting you as being inside our corporate address space)

Also, move your SSH service to some port other than 22, and consider
putting 'Password Authentication no/PubKeyAuthentication yes' in your
sshd_config.

I admit never understanding why people run their systems in a low-hanging
fruit configuration, and then are surprised that miscreants go looking for
low hanging fruit.

(For the record, our border routers drop inbound SYN on port 22 on *both*
ipv4 and ipv6 address spaces.  It's amazing how few brute force
attempts we see on our servers... :)

Re: how to deal with port scan and brute force attack from AS 8075 ?

[Davide Davini]

On 31/03/2016 10:02, marcel.duregards--- via NANOG wrote:
> We are facing a lot of port scan and brute force attack on port 22 (but
> not limited to)

Maybe not super useful in your case but talking about SSH the sysadmin
solution would be to disable password login and use just keys.

Also, as someone else said, fail2ban... because it's a lot of fun. :)

Ciao,
Davide

Re: how to deal with port scan and brute force attack from AS 8075 ?

[DV]

I have noticed this and especially the strange format of the packets with a
SYN/ECE/CWR flag combination: http://pastebin.com/jFCDAmdr

This may be $whoever trying to establish network performance/congestion via
ECN or it could be something else like a fast scan technique or OS
fingerprinting


On Thu, Mar 31, 2016 at 5:50 AM, marcel.duregards--- via NANOG <
nanog@nanog.org> wrote:

> I can not blame them to not answer to all of the thousands emails
> destined to their abuse mailbox. And the goal of my email was not to
> call them on public forum, but rather to know how others ops deal with
> it, and also if MS (and competitors) have automatic detection of such
> 'illegal' traffic, and if not why ?
>
>
>
>
>
> On 31.03.2016 10:18, Todd Crane wrote:
> > Oh and,
> >
> > I’m assuming you contacted Microsoft’s abuse? If not, it’s not cool, not
> to mention unprofessional, to publicly call them out on such a public forum
> without giving them an opportunity to correct it first.
> >
> >> On Mar 31, 2016, at 1:15 AM, Todd Crane  wrote:
> >>
> >> Marcel
> >>
> >> Depending on what is on those machines, I would just recommend using
> fail2ban. The default is that if an ip address fails ssh auth 3 times in 5
> minutes, their ip gets blocked via iptables for 5 minutes. This is enough
> to thwart most scripted attacks, especially those from a certain government
> in Asia. This is configurable to various applications, timing schemes, and
> blocking/jailing mechanisms.
> >>
> >> -Todd
> >>> On Mar 31, 2016, at 1:02 AM, marcel.duregards--- via NANOG <
> nanog@nanog.org> wrote:
> >>>
> >>> Dear Nanog'er,
> >>>
> >>> We are facing a lot of port scan and brute force attack on port 22 (but
> >>> not limited to) from Microsoft AS 8075 range toward our own infra, or
> >>> toward our customers.
> >>> We have sent email to ab...@microsoft.com, but no answer.
> >>>
> >>> source ip are:
> >>> NetRange:   40.74.0.0 - 40.125.127.255
> >>> CIDR:   40.74.0.0/15, 40.112.0.0/13, 40.124.0.0/16,
> >>> 40.76.0.0/14, 40.80.0.0/12, 40.125.0.0/17, 40.96.0.0/12, 40.120.0.0/14
> >>> NetName:MSFT
> >>>
> >>>
> >>>
> >>> We consider port scan and brute force on ssh port as an attack, and
> even
> >>> as a pre-DDOS phase (could be use to install botnet, detect unpatched
> >>> host, and so one).
> >>>
> >>> It's one thing to propose services and make money over an infra, it's
> an
> >>> other thing to take care that you clients do not use this infra to make
> >>> illegal stuffs.
> >>>
> >>>
> >>> How do you deal with such massive amount of 'illegal' traffic ?
> >>>
> >>> Thank,
> >>> Best Regards
> >>> Marcel
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> He are some examples (we have more than 3000 such packets per day just
> >>> from them, probably Azure), and source ip is always differents of
> course:
> >>>
> >>>
> >>> Flow Filtering Expression
> >>> src AS 8075 and dst port 22 and packets=1
> >>> Limit Flows
> >>> 4
> >>> Sorting
> >>> By Date
> >>>
>
> >>
> >
>

Some doubts on large scale BGP/AS design and black hole routing risk

[magicb...@hotmail.com]

Hi everybody!


as part of laboratory work at the university,  I'm working on a BGP 
design study, and I would like to post some questions regarding IP 
address space allocation and its impact on BGP which are breaking my mind :)


Let's suppose we have an ISP/AS with two POPs: PARIS and LONDON. These 
two POPs are connected with redundant leased lines. Each POP has a BGP 
router speaking eBGP to different ISP providers/upstreams and also, each 
POP run its own OSPF area/ISIS area. Something like this:



  ---eBGP---===redundant leased lines 
(ospf area0)===---eBGP---


Now, this AS/ISP gets one /22 prefix from it RIR (RIPE in this case), 
and starts to announce it to its upstreams in PARIS and LONDON at the 
same time.



My questions are:

1. What could happen in the case of total failure in the redundant 
leased lines? Black hole routing between POPs?


2. What are the best design methods to avoid this scenario?

   2.1: adding a third POP creating a triangle? What if a POP looses 
connection with the other two POPs at the same time? Another black hole?


   2.2: requesting another prefix and allocating 1:1 prefix:POP, so in 
the scenario each POP only would announce its prefix to the upstreams?


   2.3: other?



Thanks in advance!
J.

Colocation Server Lifts

[Jason Lee]

Hi NANOG community,

A few questions I have for the community regarding server lifts at colo
facilities.

1. Is a server lift something you would typically expect a colo facility to
provide?

if yes,

2. Do colo facilities typically allow customers to just use them or provide
an operator?
3. Is it a free offering or something they rent out?
4. What would be the typical device weight you would lift?
5. What would be the max device weight you would lift?

Thanks,

Jason