The DNS, point the DNS record for crsplabweb2.example.com to the public IP of haproxy.
On Mon, Oct 29, 2018 at 2:36 PM Imam Toufique <[email protected]> wrote: > " Well you need to point crsplabweb2.example.com to the haproxy IP that's > the whole point of it running behind a proxy. Or am I missing something? " > > Well, I am not sure what you meant by that comment above. > > On Sun, Oct 28, 2018 at 8:07 PM Igor Cicimov < > [email protected]> wrote: > >> Well you need to point crsplabweb2.example.com to the haproxy IP that's >> the whole point of it running behind a proxy. Or am I missing something? >> >> On Mon, Oct 29, 2018 at 1:28 PM Imam Toufique <[email protected]> >> wrote: >> >>> Hi Igor, >>> >>> Thank you so much, I will definitely try your suggestions, but I am not >>> sure how it will help my situation. shibboleth SP looks for, let's >>> suppose, https://crsplabweb2.example.com/Shibboleth.sso - for it it's >>> single sign-on. for apache or nginx to talk to the SP, SP needs to run in >>> the same node ( as far as I know ). So, I am not sure how shibboleth will >>> be able to communicate with the HAP for its SSO calls. >>> >>> --imam >>> >>> >>> >>> On Sun, Oct 28, 2018 at 5:21 PM Igor Cicimov < >>> [email protected]> wrote: >>> >>>> Hi Imam, >>>> >>>> On Sat, Oct 27, 2018 at 4:42 PM Imam Toufique <[email protected]> >>>> wrote: >>>> >>>>> Hi Igor, >>>>> >>>>> Thanks very much for offering to help! I will do this in sections, >>>>> hopefully, I can keep this from being too cluttered. >>>>> >>>>> haproxy.cfg: >>>>> >>>>> -------------------------------------------------------------------------------------- >>>>> global >>>>> #log /dev/log local0 debug >>>>> #log /dev/log local1 debug >>>>> log 127.0.0.1 local2 >>>>> chroot /var/lib/haproxy >>>>> stats timeout 30s >>>>> user haproxy >>>>> group haproxy >>>>> tune.ssl.default-dh-param 2048 >>>>> daemon >>>>> >>>>> defaults >>>>> log global >>>>> mode http >>>>> option tcplog >>>>> option dontlognull >>>>> timeout connect 5000 >>>>> timeout client 50000 >>>>> timeout server 50000 >>>>> timeout tunnel 9h >>>>> option tcp-check >>>>> >>>>> frontend http_front >>>>> bind :80 >>>>> bind 0.0.0.0:443 ssl crt /etc/haproxy/crsplab2_1.pem >>>>> stats uri /haproxy?stats >>>>> default_backend web1_cluster >>>>> option httplog >>>>> log global >>>>> #option dontlognull >>>>> log /dev/log local0 debug >>>>> mode http >>>>> option forwardfor # forward IP >>>>> http-request set-header X-Forwarded-Port %[dst_port] >>>>> http-request add-header X-Forwarded-Proto https if { ssl_fc } >>>>> redirect scheme https if !{ ssl_fc } >>>>> >>>>> acl host_web2 hdr(host) -i crsplab2.oit.uci.edu/webdav >>>>> use_backend webdav_cluster if host_web2 >>>>> >>>>> acl host_web3 path_beg /jhub >>>>> use_backend web3_cluster if host_web3 >>>>> >>>>> >>>>> backend webdav_cluster >>>>> balance roundrobin >>>>> server web1 10.1.100.156:8080 check inter 2000 cookie w1 >>>>> server web2 10.1.100.160:8080 check inter 2000 cookie w2 >>>>> >>>>> backend web3_cluster >>>>> server publicIP:443 check ssl verify none inter 2000 cookie w1 >>>>> >>>>> ----------------------------------------------------------------------------------------------------- >>>>> Note: I have a single backend node, as it was easy to test with just >>>>> one node, instead of making changes to 2 nodes at a time. >>>>> >>>>> Here is my apache config: >>>>> >>>>> in httpd.conf, only change I have made is ( the rest is a stock centos >>>>> 7.5 httpd.conf ): >>>>> ------------------------------------- >>>>> ServerName 10.1.100.160:80 ( Internal IP of the backend node) >>>>> Redirect permanent /jhub https://crsplabweb1.domain.com/jhub >>>>> ------------------------------------- >>>>> >>>>> in my ssl.conf, where I access the jupyterhub instance running in >>>>> 127.0.0.1:8000 . Also, note that the backend is running shibboleth >>>>> SP. One of the issues I encountered is, If I did not have SSL , i was >>>>> getting a browser warning for not having SSL. >>>>> >>>>> Here is my ssl.conf: >>>>> >>>>> >>>>> -------------------------------------------------------------------------- >>>>> Listen 443 https >>>>> SSLPassPhraseDialog exec:/usr/libexec/httpd-ssl-pass-dialog >>>>> SSLSessionCache shmcb:/run/httpd/sslcache(512000) >>>>> SSLSessionCacheTimeout 300 >>>>> SSLRandomSeed startup file:/dev/urandom 256 >>>>> SSLRandomSeed connect builtin >>>>> SSLCryptoDevice builtin >>>>> >>>>> <VirtualHost _default_:443> >>>>> >>>>> UseCanonicalName on >>>>> ServerName crsplabweb1.domain.com:443 >>>>> >>>>> ErrorLog logs/ssl_error_log >>>>> TransferLog logs/ssl_access_log >>>>> LogLevel warn >>>>> >>>>> SSLEngine on >>>>> >>>>> SSLProtocol all -SSLv2 -SSLv3 >>>>> SSLCipherSuite HIGH:3DES:!aNULL:!MD5:!SEED:!IDEA >>>>> SSLCertificateFile /etc/pki/tls/certs/crsplabweb1.domain.com_cert.cer >>>>> SSLCertificateKeyFile /etc/pki/tls/certs/crsplabweb2.key >>>>> SSLCertificateChainFile >>>>> /etc/pki/tls/certs/crsplabweb1.domain.com_interm_reverse.c >>>>> >>>>> <Files ~ "\.(cgi|shtml|phtml|php3?)$"> >>>>> SSLOptions +StdEnvVars >>>>> </Files> >>>>> <Directory "/var/www/cgi-bin"> >>>>> SSLOptions +StdEnvVars >>>>> </Directory> >>>>> >>>>> <Location /jhub> >>>>> ProxyPass http://127.0.0.1:8000/jhub >>>>> ProxyPassReverse http://127.0.0.1:8000/jhub >>>>> RequestHeader unset Accept-Encoding >>>>> ProxyPreserveHost on >>>>> AuthType shibboleth >>>>> ShibRequestSetting requireSession 1 >>>>> Require shibboleth >>>>> ShibUseHeaders On >>>>> ShibBasicHijack On >>>>> RewriteEngine On >>>>> RequestHeader set X-Remote-User %{REMOTE_USER}s >>>>> </Location> >>>>> >>>>> <LocationMatch >>>>> "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)"> >>>>> ProxyPassMatch ws://127.0.0.1:8000/jhub/$1/$2$3 >>>>> ProxyPassReverse ws://127.0.0.1:8000/jhub/$1/$2$3 >>>>> </LocationMatch> >>>>> >>>>> BrowserMatch "MSIE [2-5]" \ >>>>> nokeepalive ssl-unclean-shutdown \ >>>>> downgrade-1.0 force-response-1.0 >>>>> >>>>> CustomLog logs/ssl_request_log \ >>>>> "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b" >>>>> </VirtualHost> >>>>> >>>>> ---------------------------------------------------------------------------------- >>>>> >>>>> Thanks >>>>> >>>> >>>> Your problem is that you are not using the Forwarded headers set by HAP >>>> in Apache thus you get http response instead ssl. >>>> >>>> First for haproxy create a directory where you will keep all your SSL >>>> certs, lets say /etc/haproxy/ssl.d/, and put the crsplab2.oit.uci.edu >>>> and crsplabweb1.domain.com certificates inside. More details on >>>> setting SSL certificates in Haproxy can be found here: >>>> https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#5.1-crt >>>> >>>> The config will then look something like this: >>>> >>>> frontend http_front >>>> bind *:80 >>>> bind *:443 ssl crt /etc/haproxy/ssl.d/ no-sslv3 no-tls-tickets ... >>>> >>>> backend web3_cluster >>>> server shibboleth1 10.1.100.160:80 check inter 2000 >>>> >>>> On the apache side remove the ssl settings (since now HAP will be >>>> terminating SSL) and set a SSL redirect, something like this: >>>> >>>> <VirtualHost *:80> >>>> ServerName crsplabweb1.domain.com >>>> ServerAlias www.crsplabweb1.domain.com >>>> >>>> SetEnvIfNoCase X-Forwarded-Proto https HTTPS=on >>>> # Insure the pages requested over ssl are always over ssl >>>> RewriteEngine On >>>> RewriteCond %{HTTP_X_Forwarded_Proto} ^https$ >>>> RewriteRule ^(.*)$ https://%{HTTP_HOST}$1 [R,L] >>>> ... >>>> </VirtualHost> >>>> Let me know if any further questions. >>>> >>>> >>>>> On Fri, Oct 26, 2018 at 8:34 PM Igor Cicimov < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Imam, >>>>>> >>>>>> On Sat, Oct 27, 2018 at 9:37 AM Imam Toufique <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I came up with the following config, things seem to be working now, >>>>>>> for the most part. >>>>>>> >>>>>>> frontend http_front >>>>>>> bind :80 >>>>>>> bind 0.0.0.0:443 ssl crt /etc/haproxy/crsplab2_1.pem >>>>>>> stats uri /haproxy?stats >>>>>>> default_backend web1_cluster >>>>>>> option httplog >>>>>>> log global >>>>>>> #option dontlognull >>>>>>> log /dev/log local0 debug >>>>>>> mode http >>>>>>> option forwardfor # forward IP >>>>>>> http-request set-header X-Forwarded-Port %[dst_port] >>>>>>> http-request add-header X-Forwarded-Proto https if { ssl_fc } >>>>>>> redirect scheme https if !{ ssl_fc } >>>>>>> acl host_web3 path_beg /jhub >>>>>>> use_backend web3_cluster if host_web3 >>>>>>> >>>>>>> web3_cluster >>>>>>> >>>>>>> backend web3_cluster >>>>>>> mode http >>>>>>> balance source >>>>>>> server crsplabweb1.domain.com publicIP:443 check ssl verify none >>>>>>> inter 2000 cookie w1 >>>>>>> >>>>>>> The above config gets me to the backend node -- where I have a >>>>>>> jupyterhub instance running + . Shibboleth SP running for >>>>>>> authentication. >>>>>>> As I could not get shibboleth SP to work by staying in my private >>>>>>> network, >>>>>>> I had to set up a public IP for the backend node, get SSL certs - so >>>>>>> shibboleth authentication could be done. I am sure there is a better >>>>>>> approach to this, but I don't know what it is. I will be trying out >>>>>>> SNAT >>>>>>> to see if that will allow me to keep using my private IP for the backend >>>>>>> nodes. If any of you know how to do SNAT, please chime in, it would be >>>>>>> worth the time/effort to try it out. >>>>>>> >>>>>>> Now, the interesting thing I have noticed with the above setup -- >>>>>>> when I connect to HAProxy, let's say with https://proxy.domain.com >>>>>>> , I authenticate with shibboleth, and then the URL in the browser >>>>>>> points to >>>>>>> the backend node. >>>>>>> >>>>>>> For example: >>>>>>> >>>>>>> my proxy address: https://proxy.domain.com/jhub >>>>>>> >>>>>>> after I connect to the backend, the URL turns into - >>>>>>> https://crsplabweb1.domain.com/jhub/tree? >>>>>>> >>>>>>> ...and everything works thereafter. >>>>>>> >>>>>>> I tried the rewrite method that Igor has suggested before, that did >>>>>>> not make any difference. But what I noticed is, after I connect, no >>>>>>> traffic go through the proxy anymore, my client ( i.e. laptop) connects >>>>>>> directly to the backend server. Not sure if this good or bad though (?) >>>>>>> , >>>>>>> but, I am not sure how to configure this so that I will go through a >>>>>>> proxy but still be connected in the backend via a private IP and I can ( >>>>>>> still ) authenticate via shibboleth. >>>>>>> >>>>>>> So, when I change the 'web3_cluster' backend to : >>>>>>> >>>>>>> server crsplabweb1 privateIP:80 inter 2000 cookie w1 >>>>>>> >>>>>>> and, I set backend apache to accept connection on port 80, then I >>>>>>> break shibboleth authentication. >>>>>>> >>>>>>> Any inputs here? >>>>>>> >>>>>>> thanks, guys! >>>>>>> >>>>>>> >>>>>> I think it is time for you to provide the full HAP and Apache configs >>>>>> so we can see what is going on (please obfuscate any sensitive data). >>>>>> Also >>>>>> the use of the "cookie w1" is not clear since you are not setting it >>>>>> in HAP and is kinda redundant for single backend setup. >>>>>> >>>>>> >>>>>>> >>>>>>> On Thu, Oct 25, 2018 at 1:21 AM Igor Cicimov < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Oct 25, 2018 at 6:31 PM Igor Cicimov < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, 25 Oct 2018 6:13 pm Imam Toufique <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> so I almost got this to work, based on the situation I am in. To >>>>>>>>>> elaborate just a bit, my setup involves a shibboleth SP that I need >>>>>>>>>> to >>>>>>>>>> authenticate my application. Since I can't set up the HA proxy node >>>>>>>>>> with >>>>>>>>>> shibboleth SP - I had to wrap my application in the backend with >>>>>>>>>> apache so >>>>>>>>>> I can pass REMOTE_USER to the application. the application I have >>>>>>>>>> is - >>>>>>>>>> jupyterhub and it start with its own proxy. Long story short, here >>>>>>>>>> is my >>>>>>>>>> current setup: >>>>>>>>>> >>>>>>>>>> frontend >>>>>>>>>> bind :80 >>>>>>>>>> bind :443 ssl crt /etc/haproxy/crsplab2_1.pem >>>>>>>>>> stats uri /haproxy?stats >>>>>>>>>> default_backend web1_cluster >>>>>>>>>> option httplog >>>>>>>>>> log global >>>>>>>>>> #option dontlognull >>>>>>>>>> log /dev/log local0 debug >>>>>>>>>> mode http >>>>>>>>>> option forwardfor # forward IP >>>>>>>>>> http-request set-header X-Forwarded-Port %[dst_port] >>>>>>>>>> http-request add-header X-Forwarded-Proto https if { ssl_fc } >>>>>>>>>> redirect scheme https if !{ ssl_fc } >>>>>>>>>> >>>>>>>>>> acl host_web3 path_beg /jhub >>>>>>>>>> use_backend web3_cluster if host_web3 >>>>>>>>>> >>>>>>>>>> backend >>>>>>>>>> server web1.oit.uci.edu 128.110.80.5:80 check >>>>>>>>>> >>>>>>>>>> this works for the most part. But I am confused with a problem. >>>>>>>>>> when I get to my application, my backend IP address shows up in the >>>>>>>>>> browser >>>>>>>>>> URL. >>>>>>>>>> >>>>>>>>>> for example, I see this in my browser: >>>>>>>>>> >>>>>>>>>> http://128.110.80.5/jhub/user/itoufiqu/tree? >>>>>>>>>> >>>>>>>>>> whereas, I was expecting that it would show the original URL, >>>>>>>>>> such as: >>>>>>>>>> >>>>>>>>>> http://crsplab2.domain.com/jhub/user/itoufiqu/tree? ( where >>>>>>>>>> crsplab2.domain.com is the URL to get HAproxy ) >>>>>>>>>> >>>>>>>>> >>>>>>>>> You need to tell your backend app that it runs behind reverse >>>>>>>>> proxy with ssl termination and that it's domain/url is >>>>>>>>> https://crsplab2.domain.com >>>>>>>>> <http://crsplab2.domain.com/jhub/user/itoufiqu/tree>. How you do >>>>>>>>> that depends on the backend app you are using but most of them like >>>>>>>>> apache2, tomcat etc. have specific configs that you can find in their >>>>>>>>> documentation. For example if your backend is apache2 I bet you don't >>>>>>>>> have >>>>>>>>> the DomainName set in the config in which case it defaults to the >>>>>>>>> host ip >>>>>>>>> address. >>>>>>>>> >>>>>>>> >>>>>>>> You can also try: >>>>>>>> >>>>>>>> rspirep ^Location:\ http://(.*):80(.*) Location:\ https:// >>>>>>>> crsplab2.domain.com >>>>>>>> <http://crsplab2.domain.com/jhub/user/itoufiqu/tree>:443\2 if { >>>>>>>> ssl_fc } >>>>>>>> >>>>>>>> to fix the URL but note that this will not save you from hard coded >>>>>>>> url's in the returned html pages the way apache does. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>>> While I am no expert in HA proxy world, I think this might due to >>>>>>>>>> the fact that my backend does not have SSL and HAproxy frontend does >>>>>>>>>> have >>>>>>>>>> SSL. At this point, I would avoid that IP address showing up in the >>>>>>>>>> browser. what is the best way to accomplish this? >>>>>>>>>> >>>>>>>>>> thanks for your continues help! >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Oct 23, 2018 at 8:35 AM Aleksandar Lazic < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi. >>>>>>>>>>> >>>>>>>>>>> Am 23.10.2018 um 09:04 schrieb Imam Toufique: >>>>>>>>>>> > I am looking for some help on how to write the following >>>>>>>>>>> apache proxypass rules >>>>>>>>>>> > in HAproxy. Not to mention I am at a bit of loss with my >>>>>>>>>>> first try :-) . Here >>>>>>>>>>> > are my current proxypass rules: >>>>>>>>>>> > >>>>>>>>>>> > ProxyPass http://10.1.100.156:8000/jhub >>>>>>>>>>> > ProxyPassReverse http://10.1.100.156:8000/jhub >>>>>>>>>>> >>>>>>>>>>> Well ProxyPass and ProxyPassReverse do a lot of thinks not just >>>>>>>>>>> rewrites, as >>>>>>>>>>> mentioned in the doc >>>>>>>>>>> >>>>>>>>>>> https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass >>>>>>>>>>> >>>>>>>>>>> https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypassreverse >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> > <LocationMatch >>>>>>>>>>> "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)"> >>>>>>>>>>> > ProxyPassMatch ws://10.1.100.156:8000/jhub/$1/$2$3 >>>>>>>>>>> > ProxyPassReverse ws://10.1.100.156:8000/jhub/$1/$2$3 >>>>>>>>>>> > </LocationMatch> >>>>>>>>>>> > >>>>>>>>>>> > As I am not well versed in the massive HAproxy configuration >>>>>>>>>>> guide, if any of >>>>>>>>>>> > you can give me a hand with this, I would very much appreciate >>>>>>>>>>> it. >>>>>>>>>>> >>>>>>>>>>> I'm also not "that" expert but I would try the following, >>>>>>>>>>> untested. >>>>>>>>>>> >>>>>>>>>>> ### >>>>>>>>>>> defaults >>>>>>>>>>> mode http >>>>>>>>>>> log global >>>>>>>>>>> >>>>>>>>>>> #... maybe some other settings >>>>>>>>>>> timeout tunnel 10h >>>>>>>>>>> >>>>>>>>>>> frontend https_001 >>>>>>>>>>> >>>>>>>>>>> #... maybe some other settings >>>>>>>>>>> >>>>>>>>>>> acl websocket path_beg /jhub >>>>>>>>>>> >>>>>>>>>>> #... maybe some other acls >>>>>>>>>>> >>>>>>>>>>> use_backend websocket_001 if websocket >>>>>>>>>>> >>>>>>>>>>> backend websocket_001 >>>>>>>>>>> >>>>>>>>>>> reqrep "^([^\ :]*) >>>>>>>>>>> /jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)" >>>>>>>>>>> "/jhub/\1/\2\3" >>>>>>>>>>> >>>>>>>>>>> # You will need to replace the first column with the response >>>>>>>>>>> from the >>>>>>>>>>> # backend response >>>>>>>>>>> # rspirep "^Location: >>>>>>>>>>> /jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)" >>>>>>>>>>> "Location: >>>>>>>>>>> /jhub/\1/\2\3" >>>>>>>>>>> # OR >>>>>>>>>>> # http-response replace-header Location >>>>>>>>>>> "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)" >>>>>>>>>>> "/jhub/\1/\2\3" >>>>>>>>>>> >>>>>>>>>>> # add some checks >>>>>>>>>>> >>>>>>>>>>> server ws_01 10.1.100.156:8000 check >>>>>>>>>>> ### >>>>>>>>>>> >>>>>>>>>>> Here are some links which may help you also. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/ >>>>>>>>>>> >>>>>>>>>>> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-reqirep >>>>>>>>>>> >>>>>>>>>>> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-rspirep >>>>>>>>>>> >>>>>>>>>>> I would run haproxy in Debug mode and see how the request pass >>>>>>>>>>> haproxy and adopt >>>>>>>>>>> the config. >>>>>>>>>>> >>>>>>>>>>> It would be nice when you show us the working conf ;-) >>>>>>>>>>> >>>>>>>>>>> It would be nice to have a >>>>>>>>>>> >>>>>>>>>>> http-request replace-uri <match-regex> <replace-fmt> >>>>>>>>>>> >>>>>>>>>>> to replace the reqrep. >>>>>>>>>>> >>>>>>>>>>> > thanks >>>>>>>>>>> >>>>>>>>>>> Hth >>>>>>>>>>> Aleks >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Regards, >>>>>>>>>> *Imam Toufique* >>>>>>>>>> *213-700-5485* >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Igor Cicimov | DevOps >>>>>>>> >>>>>>>> >>>>>>>> p. +61 (0) 433 078 728 >>>>>>>> e. [email protected] >>>>>>>> <http://encompasscorporation.com/> >>>>>>>> w*.* www.encompasscorporation.com >>>>>>>> a. Level 4, 65 York Street, Sydney 2000 >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Regards, >>>>>>> *Imam Toufique* >>>>>>> *213-700-5485* >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Igor Cicimov | DevOps >>>>>> >>>>>> >>>>>> p. +61 (0) 433 078 728 >>>>>> e. [email protected] <http://encompasscorporation.com/> >>>>>> w*.* www.encompasscorporation.com >>>>>> a. Level 4, 65 York Street, Sydney 2000 >>>>>> >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> *Imam Toufique* >>>>> *213-700-5485* >>>>> >>>> >>>> >>>> -- >>>> Igor Cicimov | DevOps >>>> >>>> >>>> p. +61 (0) 433 078 728 >>>> e. [email protected] <http://encompasscorporation.com/> >>>> w*.* www.encompasscorporation.com >>>> a. Level 4, 65 York Street, Sydney 2000 >>>> >>> >>> >>> -- >>> Regards, >>> *Imam Toufique* >>> *213-700-5485* >>> >> >> >> -- >> Igor Cicimov | DevOps >> >> >> p. +61 (0) 433 078 728 >> e. [email protected] <http://encompasscorporation.com/> >> w*.* www.encompasscorporation.com >> a. Level 4, 65 York Street, Sydney 2000 >> > > > -- > Regards, > *Imam Toufique* > *213-700-5485* > -- Igor Cicimov | DevOps p. +61 (0) 433 078 728 e. [email protected] <http://encompasscorporation.com/> w*.* www.encompasscorporation.com a. Level 4, 65 York Street, Sydney 2000
