Re: Heptio Contour

2018-01-24 Thread Clayton Coleman 
On Jan 24, 2018, at 2:05 PM, Srinivas Naga Kotaru (skotaru) <
skot...@cisco.com> wrote:

Clayton



Good analyses. That is exactly am looking for. Thanks for great info.



Also happy that, you already did prototype and compared with current OCP
routing solution.



Also can u share your thoughts on how ambassador fit this eco system? my
research shows, ambassador would be a good fit for north/south ingress
controller where as Istio would be great fit for east/west service traffic.
Both use Envoy internally.


I do expect all of these proxies to become more interesting over time, and
I agree east/west and north/south are obvious places for differentiation.
We might have multiple interlocking proxies with different focuses.



Then ambassador would be another competitor to contour?



I knew lot of moving parts on routes, ingress and services side but none is
prime time ready for high scale workloads.



-- 

*Srinivas Kotaru*

*From: *Clayton Coleman 
*Date: *Wednesday, January 24, 2018 at 10:32 AM
*To: *Srinivas Naga Kotaru 
*Cc: *users 
*Subject: *Re: Heptio Contour



At this point in time, contour is still pretty new, so expect some rough
edges.  I did a prototype of routes with envoy (similar to contour, but
preserving the router features) a few months back, and identified a set of
challenges which made it not a great fit as a replacement for the OOTB
openshift router



In general, when comparing to haproxy and the general state, here's the
list



PROs (envoy):



* supports http2 natively

* deeper insight into traffic passing through



CONs (envoy):



* scale is not great right now - even using dynamic programming i couldn't
get much above 100 backends before hitting the wall (wouldn't scale to very
large, dense clusters)

* memory use is much higher than haproxy - so another density challenge (I
was using 30GB at 1k backends, vs 5GB for 15k backends that we see with
haproxy)

* web sockets can't be transparent - so you have to run another port for
them instead of sharing the HTTP port

* SNI passthrough not ready, maybe in 6mo

* reencrypt was really hacky, I couldn't get it to work right now (again,
6mo should be fixed)

* general fragility - was easy to break when programming config



CONs (contour, vs openshift router):



* None of the security isolation stuff (preventing one tenant from using
someone else's hostname)

* None of the protection against bad certs (preventing someone from
breaking the router by using someone else's tenant)

* No status reporting back



I think the biggest long term challenge with envoy will be pure scale -
HAProxy is at least two orders of magnitude more efficient right now, and I
think it will be a while before envoy even gets close.  So if you have 10k+
frontends, haproxy is the only game in town.  On ingress vs routes, routes
are still more polished, so it's really just a "do you want the features
routes have that ingress doesn't.



On the other downsides to envoy, I expect to see progress over the next
year or two to fixing it.  I had originally done the prototype expecting
that maybe we would use envoy as the "out of the box" plugin to the router
(continuing to support routes and ingress and all the features, but with
envoy underneath), but the biggest challenge is that envoy isn't really
better than haproxy for the feature set that ingress and routes expose.
Where envoy really shines is in something like istio, where you have a
richer model for the ingress / service definition that can use the extra
bells and whistles.  Ultimately ingress + annotations and routes are both
targeted at "simple, high scale" use of web frontends.  I would expect a
lot of people to have their apps "grow up" and use node ports or istio
ingress as the apps get bigger and more important.  I don't see them as
directly competing.







On Fri, Jan 19, 2018 at 1:36 PM, Srinivas Naga Kotaru (skotaru) <
skot...@cisco.com> wrote:

How it is different than Openshift router and what extra benefits it
brings? Anyone educate me to understand differences or possible use cases
where it fit into eco system? Or replacing ingress controller or will it
solve ingress controller 244 address limitations?



https://blog.heptio.com/announcing-contour-0-3-37f4aa7bc6f7







-- 

*Srinivas Kotaru*


___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Heptio Contour

2018-01-24 Thread Clayton Coleman 
3.9 will have haproxy 1.8, but the current level of http2 doesn’t
really help applications that aren’t willing to do passthrough.  That
said, passthrough http2 should work.

> On Jan 24, 2018, at 2:54 PM, Tobias Brunner  wrote:
>
>> On 24.01.2018 19:31, Clayton Coleman wrote:
>> In general, when comparing to haproxy and the general state, here's the list
>>
>> PROs (envoy):
>>
>> * supports http2 natively
>
> I would say this is a non-issue:
> https://www.mail-archive.com/haproxy@formilux.org/msg28004.html
>
> ---
> HTTP/2 (Willy Tarreau) : HTTP/2 is automatically detected and processed
> in HTTP frontends negociating the "h2" protocol name based on the ALPN
> or NPN TLS extensions. At the moment the HTTP/2 frames are converted to
> HTTP/1.1 requests before processing, so they will always appear as 1.1
> in the logs (and in server logs). No HTTP/2 is supported for now on the
> backend, though this is scheduled for the next steps. HTTP/2 support is
> still considered EXPERIMENTAL, so just like for multi-threading, in case
> of problem you may end up having to disable it for the time it takes to
> solve the issue.
> ---
>
> Now we just need to wait until HAProxy >= 1.8.0 is included in
> OpenShift. Is there any rough ETA for this upgrade available?
>
> Cheers,
> Tobias
>
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users

___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Heptio Contour

2018-01-24 Thread Tobias Brunner 
On 24.01.2018 19:31, Clayton Coleman wrote:
> In general, when comparing to haproxy and the general state, here's the list
> 
> PROs (envoy):
> 
> * supports http2 natively

I would say this is a non-issue:
https://www.mail-archive.com/haproxy@formilux.org/msg28004.html

---
HTTP/2 (Willy Tarreau) : HTTP/2 is automatically detected and processed
in HTTP frontends negociating the "h2" protocol name based on the ALPN
or NPN TLS extensions. At the moment the HTTP/2 frames are converted to
HTTP/1.1 requests before processing, so they will always appear as 1.1
in the logs (and in server logs). No HTTP/2 is supported for now on the
backend, though this is scheduled for the next steps. HTTP/2 support is
still considered EXPERIMENTAL, so just like for multi-threading, in case
of problem you may end up having to disable it for the time it takes to
solve the issue.
---

Now we just need to wait until HAProxy >= 1.8.0 is included in
OpenShift. Is there any rough ETA for this upgrade available?

Cheers,
Tobias

___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Heptio Contour

2018-01-24 Thread Srinivas Naga Kotaru (skotaru) 
Clayton

Good analyses. That is exactly am looking for. Thanks for great info.

Also happy that, you already did prototype and compared with current OCP 
routing solution.

Also can u share your thoughts on how ambassador fit this eco system? my 
research shows, ambassador would be a good fit for north/south ingress 
controller where as Istio would be great fit for east/west service traffic. 
Both use Envoy internally.

Then ambassador would be another competitor to contour?

I knew lot of moving parts on routes, ingress and services side but none is 
prime time ready for high scale workloads.

--
Srinivas Kotaru
From: Clayton Coleman 
Date: Wednesday, January 24, 2018 at 10:32 AM
To: Srinivas Naga Kotaru 
Cc: users 
Subject: Re: Heptio Contour

At this point in time, contour is still pretty new, so expect some rough edges. 
 I did a prototype of routes with envoy (similar to contour, but preserving the 
router features) a few months back, and identified a set of challenges which 
made it not a great fit as a replacement for the OOTB openshift router

In general, when comparing to haproxy and the general state, here's the list

PROs (envoy):

* supports http2 natively
* deeper insight into traffic passing through

CONs (envoy):

* scale is not great right now - even using dynamic programming i couldn't get 
much above 100 backends before hitting the wall (wouldn't scale to very large, 
dense clusters)
* memory use is much higher than haproxy - so another density challenge (I was 
using 30GB at 1k backends, vs 5GB for 15k backends that we see with haproxy)
* web sockets can't be transparent - so you have to run another port for them 
instead of sharing the HTTP port
* SNI passthrough not ready, maybe in 6mo
* reencrypt was really hacky, I couldn't get it to work right now (again, 6mo 
should be fixed)
* general fragility - was easy to break when programming config

CONs (contour, vs openshift router):

* None of the security isolation stuff (preventing one tenant from using 
someone else's hostname)
* None of the protection against bad certs (preventing someone from breaking 
the router by using someone else's tenant)
* No status reporting back

I think the biggest long term challenge with envoy will be pure scale - HAProxy 
is at least two orders of magnitude more efficient right now, and I think it 
will be a while before envoy even gets close.  So if you have 10k+ frontends, 
haproxy is the only game in town.  On ingress vs routes, routes are still more 
polished, so it's really just a "do you want the features routes have that 
ingress doesn't.

On the other downsides to envoy, I expect to see progress over the next year or 
two to fixing it.  I had originally done the prototype expecting that maybe we 
would use envoy as the "out of the box" plugin to the router (continuing to 
support routes and ingress and all the features, but with envoy underneath), 
but the biggest challenge is that envoy isn't really better than haproxy for 
the feature set that ingress and routes expose.  Where envoy really shines is 
in something like istio, where you have a richer model for the ingress / 
service definition that can use the extra bells and whistles.  Ultimately 
ingress + annotations and routes are both targeted at "simple, high scale" use 
of web frontends.  I would expect a lot of people to have their apps "grow up" 
and use node ports or istio ingress as the apps get bigger and more important.  
I don't see them as directly competing.



On Fri, Jan 19, 2018 at 1:36 PM, Srinivas Naga Kotaru (skotaru) 
> wrote:
How it is different than Openshift router and what extra benefits it brings? 
Anyone educate me to understand differences or possible use cases where it fit 
into eco system? Or replacing ingress controller or will it solve ingress 
controller 244 address limitations?

https://blog.heptio.com/announcing-contour-0-3-37f4aa7bc6f7



--
Srinivas Kotaru

___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Heptio Contour

2018-01-24 Thread Clayton Coleman 
At this point in time, contour is still pretty new, so expect some rough
edges.  I did a prototype of routes with envoy (similar to contour, but
preserving the router features) a few months back, and identified a set of
challenges which made it not a great fit as a replacement for the OOTB
openshift router

In general, when comparing to haproxy and the general state, here's the list

PROs (envoy):

* supports http2 natively
* deeper insight into traffic passing through

CONs (envoy):

* scale is not great right now - even using dynamic programming i couldn't
get much above 100 backends before hitting the wall (wouldn't scale to very
large, dense clusters)
* memory use is much higher than haproxy - so another density challenge (I
was using 30GB at 1k backends, vs 5GB for 15k backends that we see with
haproxy)
* web sockets can't be transparent - so you have to run another port for
them instead of sharing the HTTP port
* SNI passthrough not ready, maybe in 6mo
* reencrypt was really hacky, I couldn't get it to work right now (again,
6mo should be fixed)
* general fragility - was easy to break when programming config

CONs (contour, vs openshift router):

* None of the security isolation stuff (preventing one tenant from using
someone else's hostname)
* None of the protection against bad certs (preventing someone from
breaking the router by using someone else's tenant)
* No status reporting back

I think the biggest long term challenge with envoy will be pure scale -
HAProxy is at least two orders of magnitude more efficient right now, and I
think it will be a while before envoy even gets close.  So if you have 10k+
frontends, haproxy is the only game in town.  On ingress vs routes, routes
are still more polished, so it's really just a "do you want the features
routes have that ingress doesn't.

On the other downsides to envoy, I expect to see progress over the next
year or two to fixing it.  I had originally done the prototype expecting
that maybe we would use envoy as the "out of the box" plugin to the router
(continuing to support routes and ingress and all the features, but with
envoy underneath), but the biggest challenge is that envoy isn't really
better than haproxy for the feature set that ingress and routes expose.
Where envoy really shines is in something like istio, where you have a
richer model for the ingress / service definition that can use the extra
bells and whistles.  Ultimately ingress + annotations and routes are both
targeted at "simple, high scale" use of web frontends.  I would expect a
lot of people to have their apps "grow up" and use node ports or istio
ingress as the apps get bigger and more important.  I don't see them as
directly competing.



On Fri, Jan 19, 2018 at 1:36 PM, Srinivas Naga Kotaru (skotaru) <
skot...@cisco.com> wrote:

> How it is different than Openshift router and what extra benefits it
> brings? Anyone educate me to understand differences or possible use cases
> where it fit into eco system? Or replacing ingress controller or will it
> solve ingress controller 244 address limitations?
>
>
>
> https://blog.heptio.com/announcing-contour-0-3-37f4aa7bc6f7
>
>
>
>
>
>
>
> --
>
> *Srinivas Kotaru*
>
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
>
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users