i found something here, might be of good interest, this might work. It does have some similarities with what you have mentioned . I will try this out.
https://docs.shib.ncsu.edu/docs/advanced/spscale.html On Sun, Oct 28, 2018 at 9:21 PM Igor Cicimov <[email protected]> wrote: > The DNS, point the DNS record for crsplabweb2.example.com to the public > IP of haproxy. > > On Mon, Oct 29, 2018 at 2:36 PM Imam Toufique <[email protected]> wrote: > >> " Well you need to point crsplabweb2.example.com to the haproxy IP >> that's the whole point of it running behind a proxy. Or am I missing >> something? " >> >> Well, I am not sure what you meant by that comment above. >> >> On Sun, Oct 28, 2018 at 8:07 PM Igor Cicimov < >> [email protected]> wrote: >> >>> Well you need to point crsplabweb2.example.com to the haproxy IP that's >>> the whole point of it running behind a proxy. Or am I missing something? >>> >>> On Mon, Oct 29, 2018 at 1:28 PM Imam Toufique <[email protected]> >>> wrote: >>> >>>> Hi Igor, >>>> >>>> Thank you so much, I will definitely try your suggestions, but I am not >>>> sure how it will help my situation. shibboleth SP looks for, let's >>>> suppose, https://crsplabweb2.example.com/Shibboleth.sso - for it it's >>>> single sign-on. for apache or nginx to talk to the SP, SP needs to run in >>>> the same node ( as far as I know ). So, I am not sure how shibboleth will >>>> be able to communicate with the HAP for its SSO calls. >>>> >>>> --imam >>>> >>>> >>>> >>>> On Sun, Oct 28, 2018 at 5:21 PM Igor Cicimov < >>>> [email protected]> wrote: >>>> >>>>> Hi Imam, >>>>> >>>>> On Sat, Oct 27, 2018 at 4:42 PM Imam Toufique <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Igor, >>>>>> >>>>>> Thanks very much for offering to help! I will do this in sections, >>>>>> hopefully, I can keep this from being too cluttered. >>>>>> >>>>>> haproxy.cfg: >>>>>> >>>>>> -------------------------------------------------------------------------------------- >>>>>> global >>>>>> #log /dev/log local0 debug >>>>>> #log /dev/log local1 debug >>>>>> log 127.0.0.1 local2 >>>>>> chroot /var/lib/haproxy >>>>>> stats timeout 30s >>>>>> user haproxy >>>>>> group haproxy >>>>>> tune.ssl.default-dh-param 2048 >>>>>> daemon >>>>>> >>>>>> defaults >>>>>> log global >>>>>> mode http >>>>>> option tcplog >>>>>> option dontlognull >>>>>> timeout connect 5000 >>>>>> timeout client 50000 >>>>>> timeout server 50000 >>>>>> timeout tunnel 9h >>>>>> option tcp-check >>>>>> >>>>>> frontend http_front >>>>>> bind :80 >>>>>> bind 0.0.0.0:443 ssl crt /etc/haproxy/crsplab2_1.pem >>>>>> stats uri /haproxy?stats >>>>>> default_backend web1_cluster >>>>>> option httplog >>>>>> log global >>>>>> #option dontlognull >>>>>> log /dev/log local0 debug >>>>>> mode http >>>>>> option forwardfor # forward IP >>>>>> http-request set-header X-Forwarded-Port %[dst_port] >>>>>> http-request add-header X-Forwarded-Proto https if { ssl_fc } >>>>>> redirect scheme https if !{ ssl_fc } >>>>>> >>>>>> acl host_web2 hdr(host) -i crsplab2.oit.uci.edu/webdav >>>>>> use_backend webdav_cluster if host_web2 >>>>>> >>>>>> acl host_web3 path_beg /jhub >>>>>> use_backend web3_cluster if host_web3 >>>>>> >>>>>> >>>>>> backend webdav_cluster >>>>>> balance roundrobin >>>>>> server web1 10.1.100.156:8080 check inter 2000 cookie w1 >>>>>> server web2 10.1.100.160:8080 check inter 2000 cookie w2 >>>>>> >>>>>> backend web3_cluster >>>>>> server publicIP:443 check ssl verify none inter 2000 cookie w1 >>>>>> >>>>>> ----------------------------------------------------------------------------------------------------- >>>>>> Note: I have a single backend node, as it was easy to test with just >>>>>> one node, instead of making changes to 2 nodes at a time. >>>>>> >>>>>> Here is my apache config: >>>>>> >>>>>> in httpd.conf, only change I have made is ( the rest is a stock >>>>>> centos 7.5 httpd.conf ): >>>>>> ------------------------------------- >>>>>> ServerName 10.1.100.160:80 ( Internal IP of the backend node) >>>>>> Redirect permanent /jhub https://crsplabweb1.domain.com/jhub >>>>>> ------------------------------------- >>>>>> >>>>>> in my ssl.conf, where I access the jupyterhub instance running in >>>>>> 127.0.0.1:8000 . Also, note that the backend is running shibboleth >>>>>> SP. One of the issues I encountered is, If I did not have SSL , i was >>>>>> getting a browser warning for not having SSL. >>>>>> >>>>>> Here is my ssl.conf: >>>>>> >>>>>> >>>>>> -------------------------------------------------------------------------- >>>>>> Listen 443 https >>>>>> SSLPassPhraseDialog exec:/usr/libexec/httpd-ssl-pass-dialog >>>>>> SSLSessionCache shmcb:/run/httpd/sslcache(512000) >>>>>> SSLSessionCacheTimeout 300 >>>>>> SSLRandomSeed startup file:/dev/urandom 256 >>>>>> SSLRandomSeed connect builtin >>>>>> SSLCryptoDevice builtin >>>>>> >>>>>> <VirtualHost _default_:443> >>>>>> >>>>>> UseCanonicalName on >>>>>> ServerName crsplabweb1.domain.com:443 >>>>>> >>>>>> ErrorLog logs/ssl_error_log >>>>>> TransferLog logs/ssl_access_log >>>>>> LogLevel warn >>>>>> >>>>>> SSLEngine on >>>>>> >>>>>> SSLProtocol all -SSLv2 -SSLv3 >>>>>> SSLCipherSuite HIGH:3DES:!aNULL:!MD5:!SEED:!IDEA >>>>>> SSLCertificateFile /etc/pki/tls/certs/crsplabweb1.domain.com_cert.cer >>>>>> SSLCertificateKeyFile /etc/pki/tls/certs/crsplabweb2.key >>>>>> SSLCertificateChainFile >>>>>> /etc/pki/tls/certs/crsplabweb1.domain.com_interm_reverse.c >>>>>> >>>>>> <Files ~ "\.(cgi|shtml|phtml|php3?)$"> >>>>>> SSLOptions +StdEnvVars >>>>>> </Files> >>>>>> <Directory "/var/www/cgi-bin"> >>>>>> SSLOptions +StdEnvVars >>>>>> </Directory> >>>>>> >>>>>> <Location /jhub> >>>>>> ProxyPass http://127.0.0.1:8000/jhub >>>>>> ProxyPassReverse http://127.0.0.1:8000/jhub >>>>>> RequestHeader unset Accept-Encoding >>>>>> ProxyPreserveHost on >>>>>> AuthType shibboleth >>>>>> ShibRequestSetting requireSession 1 >>>>>> Require shibboleth >>>>>> ShibUseHeaders On >>>>>> ShibBasicHijack On >>>>>> RewriteEngine On >>>>>> RequestHeader set X-Remote-User %{REMOTE_USER}s >>>>>> </Location> >>>>>> >>>>>> <LocationMatch >>>>>> "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)"> >>>>>> ProxyPassMatch ws://127.0.0.1:8000/jhub/$1/$2$3 >>>>>> ProxyPassReverse ws://127.0.0.1:8000/jhub/$1/$2$3 >>>>>> </LocationMatch> >>>>>> >>>>>> BrowserMatch "MSIE [2-5]" \ >>>>>> nokeepalive ssl-unclean-shutdown \ >>>>>> downgrade-1.0 force-response-1.0 >>>>>> >>>>>> CustomLog logs/ssl_request_log \ >>>>>> "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b" >>>>>> </VirtualHost> >>>>>> >>>>>> ---------------------------------------------------------------------------------- >>>>>> >>>>>> Thanks >>>>>> >>>>> >>>>> Your problem is that you are not using the Forwarded headers set by >>>>> HAP in Apache thus you get http response instead ssl. >>>>> >>>>> First for haproxy create a directory where you will keep all your SSL >>>>> certs, lets say /etc/haproxy/ssl.d/, and put the crsplab2.oit.uci.edu >>>>> and crsplabweb1.domain.com certificates inside. More details on >>>>> setting SSL certificates in Haproxy can be found here: >>>>> https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#5.1-crt >>>>> >>>>> The config will then look something like this: >>>>> >>>>> frontend http_front >>>>> bind *:80 >>>>> bind *:443 ssl crt /etc/haproxy/ssl.d/ no-sslv3 no-tls-tickets ... >>>>> >>>>> backend web3_cluster >>>>> server shibboleth1 10.1.100.160:80 check inter 2000 >>>>> >>>>> On the apache side remove the ssl settings (since now HAP will be >>>>> terminating SSL) and set a SSL redirect, something like this: >>>>> >>>>> <VirtualHost *:80> >>>>> ServerName crsplabweb1.domain.com >>>>> ServerAlias www.crsplabweb1.domain.com >>>>> >>>>> SetEnvIfNoCase X-Forwarded-Proto https HTTPS=on >>>>> # Insure the pages requested over ssl are always over ssl >>>>> RewriteEngine On >>>>> RewriteCond %{HTTP_X_Forwarded_Proto} ^https$ >>>>> RewriteRule ^(.*)$ https://%{HTTP_HOST}$1 [R,L] >>>>> ... >>>>> </VirtualHost> >>>>> Let me know if any further questions. >>>>> >>>>> >>>>>> On Fri, Oct 26, 2018 at 8:34 PM Igor Cicimov < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Imam, >>>>>>> >>>>>>> On Sat, Oct 27, 2018 at 9:37 AM Imam Toufique <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I came up with the following config, things seem to be working now, >>>>>>>> for the most part. >>>>>>>> >>>>>>>> frontend http_front >>>>>>>> bind :80 >>>>>>>> bind 0.0.0.0:443 ssl crt /etc/haproxy/crsplab2_1.pem >>>>>>>> stats uri /haproxy?stats >>>>>>>> default_backend web1_cluster >>>>>>>> option httplog >>>>>>>> log global >>>>>>>> #option dontlognull >>>>>>>> log /dev/log local0 debug >>>>>>>> mode http >>>>>>>> option forwardfor # forward IP >>>>>>>> http-request set-header X-Forwarded-Port %[dst_port] >>>>>>>> http-request add-header X-Forwarded-Proto https if { ssl_fc } >>>>>>>> redirect scheme https if !{ ssl_fc } >>>>>>>> acl host_web3 path_beg /jhub >>>>>>>> use_backend web3_cluster if host_web3 >>>>>>>> >>>>>>>> web3_cluster >>>>>>>> >>>>>>>> backend web3_cluster >>>>>>>> mode http >>>>>>>> balance source >>>>>>>> server crsplabweb1.domain.com publicIP:443 check ssl verify >>>>>>>> none inter 2000 cookie w1 >>>>>>>> >>>>>>>> The above config gets me to the backend node -- where I have a >>>>>>>> jupyterhub instance running + . Shibboleth SP running for >>>>>>>> authentication. >>>>>>>> As I could not get shibboleth SP to work by staying in my private >>>>>>>> network, >>>>>>>> I had to set up a public IP for the backend node, get SSL certs - so >>>>>>>> shibboleth authentication could be done. I am sure there is a better >>>>>>>> approach to this, but I don't know what it is. I will be trying out >>>>>>>> SNAT >>>>>>>> to see if that will allow me to keep using my private IP for the >>>>>>>> backend >>>>>>>> nodes. If any of you know how to do SNAT, please chime in, it would be >>>>>>>> worth the time/effort to try it out. >>>>>>>> >>>>>>>> Now, the interesting thing I have noticed with the above setup -- >>>>>>>> when I connect to HAProxy, let's say with https://proxy.domain.com >>>>>>>> , I authenticate with shibboleth, and then the URL in the browser >>>>>>>> points to >>>>>>>> the backend node. >>>>>>>> >>>>>>>> For example: >>>>>>>> >>>>>>>> my proxy address: https://proxy.domain.com/jhub >>>>>>>> >>>>>>>> after I connect to the backend, the URL turns into - >>>>>>>> https://crsplabweb1.domain.com/jhub/tree? >>>>>>>> >>>>>>>> ...and everything works thereafter. >>>>>>>> >>>>>>>> I tried the rewrite method that Igor has suggested before, that did >>>>>>>> not make any difference. But what I noticed is, after I connect, no >>>>>>>> traffic go through the proxy anymore, my client ( i.e. laptop) connects >>>>>>>> directly to the backend server. Not sure if this good or bad though >>>>>>>> (?) , >>>>>>>> but, I am not sure how to configure this so that I will go through a >>>>>>>> proxy but still be connected in the backend via a private IP and I can >>>>>>>> ( >>>>>>>> still ) authenticate via shibboleth. >>>>>>>> >>>>>>>> So, when I change the 'web3_cluster' backend to : >>>>>>>> >>>>>>>> server crsplabweb1 privateIP:80 inter 2000 cookie w1 >>>>>>>> >>>>>>>> and, I set backend apache to accept connection on port 80, then I >>>>>>>> break shibboleth authentication. >>>>>>>> >>>>>>>> Any inputs here? >>>>>>>> >>>>>>>> thanks, guys! >>>>>>>> >>>>>>>> >>>>>>> I think it is time for you to provide the full HAP and Apache >>>>>>> configs so we can see what is going on (please obfuscate any sensitive >>>>>>> data). Also the use of the "cookie w1" is not clear since you are >>>>>>> not setting it in HAP and is kinda redundant for single backend setup. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> On Thu, Oct 25, 2018 at 1:21 AM Igor Cicimov < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Oct 25, 2018 at 6:31 PM Igor Cicimov < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, 25 Oct 2018 6:13 pm Imam Toufique <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> so I almost got this to work, based on the situation I am in. >>>>>>>>>>> To elaborate just a bit, my setup involves a shibboleth SP that I >>>>>>>>>>> need to >>>>>>>>>>> authenticate my application. Since I can't set up the HA proxy >>>>>>>>>>> node with >>>>>>>>>>> shibboleth SP - I had to wrap my application in the backend with >>>>>>>>>>> apache so >>>>>>>>>>> I can pass REMOTE_USER to the application. the application I have >>>>>>>>>>> is - >>>>>>>>>>> jupyterhub and it start with its own proxy. Long story short, here >>>>>>>>>>> is my >>>>>>>>>>> current setup: >>>>>>>>>>> >>>>>>>>>>> frontend >>>>>>>>>>> bind :80 >>>>>>>>>>> bind :443 ssl crt /etc/haproxy/crsplab2_1.pem >>>>>>>>>>> stats uri /haproxy?stats >>>>>>>>>>> default_backend web1_cluster >>>>>>>>>>> option httplog >>>>>>>>>>> log global >>>>>>>>>>> #option dontlognull >>>>>>>>>>> log /dev/log local0 debug >>>>>>>>>>> mode http >>>>>>>>>>> option forwardfor # forward IP >>>>>>>>>>> http-request set-header X-Forwarded-Port %[dst_port] >>>>>>>>>>> http-request add-header X-Forwarded-Proto https if { ssl_fc } >>>>>>>>>>> redirect scheme https if !{ ssl_fc } >>>>>>>>>>> >>>>>>>>>>> acl host_web3 path_beg /jhub >>>>>>>>>>> use_backend web3_cluster if host_web3 >>>>>>>>>>> >>>>>>>>>>> backend >>>>>>>>>>> server web1.oit.uci.edu 128.110.80.5:80 check >>>>>>>>>>> >>>>>>>>>>> this works for the most part. But I am confused with a problem. >>>>>>>>>>> when I get to my application, my backend IP address shows up in the >>>>>>>>>>> browser >>>>>>>>>>> URL. >>>>>>>>>>> >>>>>>>>>>> for example, I see this in my browser: >>>>>>>>>>> >>>>>>>>>>> http://128.110.80.5/jhub/user/itoufiqu/tree? >>>>>>>>>>> >>>>>>>>>>> whereas, I was expecting that it would show the original URL, >>>>>>>>>>> such as: >>>>>>>>>>> >>>>>>>>>>> http://crsplab2.domain.com/jhub/user/itoufiqu/tree? ( where >>>>>>>>>>> crsplab2.domain.com is the URL to get HAproxy ) >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> You need to tell your backend app that it runs behind reverse >>>>>>>>>> proxy with ssl termination and that it's domain/url is >>>>>>>>>> https://crsplab2.domain.com >>>>>>>>>> <http://crsplab2.domain.com/jhub/user/itoufiqu/tree>. How you do >>>>>>>>>> that depends on the backend app you are using but most of them like >>>>>>>>>> apache2, tomcat etc. have specific configs that you can find in their >>>>>>>>>> documentation. For example if your backend is apache2 I bet you >>>>>>>>>> don't have >>>>>>>>>> the DomainName set in the config in which case it defaults to the >>>>>>>>>> host ip >>>>>>>>>> address. >>>>>>>>>> >>>>>>>>> >>>>>>>>> You can also try: >>>>>>>>> >>>>>>>>> rspirep ^Location:\ http://(.*):80(.*) Location:\ https:// >>>>>>>>> crsplab2.domain.com >>>>>>>>> <http://crsplab2.domain.com/jhub/user/itoufiqu/tree>:443\2 if { >>>>>>>>> ssl_fc } >>>>>>>>> >>>>>>>>> to fix the URL but note that this will not save you from hard >>>>>>>>> coded url's in the returned html pages the way apache does. >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>>> While I am no expert in HA proxy world, I think this might due >>>>>>>>>>> to the fact that my backend does not have SSL and HAproxy frontend >>>>>>>>>>> does >>>>>>>>>>> have SSL. At this point, I would avoid that IP address showing up >>>>>>>>>>> in the >>>>>>>>>>> browser. what is the best way to accomplish this? >>>>>>>>>>> >>>>>>>>>>> thanks for your continues help! >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Oct 23, 2018 at 8:35 AM Aleksandar Lazic < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi. >>>>>>>>>>>> >>>>>>>>>>>> Am 23.10.2018 um 09:04 schrieb Imam Toufique: >>>>>>>>>>>> > I am looking for some help on how to write the following >>>>>>>>>>>> apache proxypass rules >>>>>>>>>>>> > in HAproxy. Not to mention I am at a bit of loss with my >>>>>>>>>>>> first try :-) . Here >>>>>>>>>>>> > are my current proxypass rules: >>>>>>>>>>>> > >>>>>>>>>>>> > ProxyPass http://10.1.100.156:8000/jhub >>>>>>>>>>>> > ProxyPassReverse http://10.1.100.156:8000/jhub >>>>>>>>>>>> >>>>>>>>>>>> Well ProxyPass and ProxyPassReverse do a lot of thinks not just >>>>>>>>>>>> rewrites, as >>>>>>>>>>>> mentioned in the doc >>>>>>>>>>>> >>>>>>>>>>>> https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass >>>>>>>>>>>> >>>>>>>>>>>> https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypassreverse >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > <LocationMatch >>>>>>>>>>>> "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)"> >>>>>>>>>>>> > ProxyPassMatch ws://10.1.100.156:8000/jhub/$1/$2$3 >>>>>>>>>>>> > ProxyPassReverse ws://10.1.100.156:8000/jhub/$1/$2$3 >>>>>>>>>>>> > </LocationMatch> >>>>>>>>>>>> > >>>>>>>>>>>> > As I am not well versed in the massive HAproxy configuration >>>>>>>>>>>> guide, if any of >>>>>>>>>>>> > you can give me a hand with this, I would very much >>>>>>>>>>>> appreciate it. >>>>>>>>>>>> >>>>>>>>>>>> I'm also not "that" expert but I would try the following, >>>>>>>>>>>> untested. >>>>>>>>>>>> >>>>>>>>>>>> ### >>>>>>>>>>>> defaults >>>>>>>>>>>> mode http >>>>>>>>>>>> log global >>>>>>>>>>>> >>>>>>>>>>>> #... maybe some other settings >>>>>>>>>>>> timeout tunnel 10h >>>>>>>>>>>> >>>>>>>>>>>> frontend https_001 >>>>>>>>>>>> >>>>>>>>>>>> #... maybe some other settings >>>>>>>>>>>> >>>>>>>>>>>> acl websocket path_beg /jhub >>>>>>>>>>>> >>>>>>>>>>>> #... maybe some other acls >>>>>>>>>>>> >>>>>>>>>>>> use_backend websocket_001 if websocket >>>>>>>>>>>> >>>>>>>>>>>> backend websocket_001 >>>>>>>>>>>> >>>>>>>>>>>> reqrep "^([^\ :]*) >>>>>>>>>>>> /jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)" >>>>>>>>>>>> "/jhub/\1/\2\3" >>>>>>>>>>>> >>>>>>>>>>>> # You will need to replace the first column with the response >>>>>>>>>>>> from the >>>>>>>>>>>> # backend response >>>>>>>>>>>> # rspirep "^Location: >>>>>>>>>>>> /jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)" >>>>>>>>>>>> "Location: >>>>>>>>>>>> /jhub/\1/\2\3" >>>>>>>>>>>> # OR >>>>>>>>>>>> # http-response replace-header Location >>>>>>>>>>>> "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)" >>>>>>>>>>>> "/jhub/\1/\2\3" >>>>>>>>>>>> >>>>>>>>>>>> # add some checks >>>>>>>>>>>> >>>>>>>>>>>> server ws_01 10.1.100.156:8000 check >>>>>>>>>>>> ### >>>>>>>>>>>> >>>>>>>>>>>> Here are some links which may help you also. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/ >>>>>>>>>>>> >>>>>>>>>>>> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-reqirep >>>>>>>>>>>> >>>>>>>>>>>> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-rspirep >>>>>>>>>>>> >>>>>>>>>>>> I would run haproxy in Debug mode and see how the request pass >>>>>>>>>>>> haproxy and adopt >>>>>>>>>>>> the config. >>>>>>>>>>>> >>>>>>>>>>>> It would be nice when you show us the working conf ;-) >>>>>>>>>>>> >>>>>>>>>>>> It would be nice to have a >>>>>>>>>>>> >>>>>>>>>>>> http-request replace-uri <match-regex> <replace-fmt> >>>>>>>>>>>> >>>>>>>>>>>> to replace the reqrep. >>>>>>>>>>>> >>>>>>>>>>>> > thanks >>>>>>>>>>>> >>>>>>>>>>>> Hth >>>>>>>>>>>> Aleks >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Regards, >>>>>>>>>>> *Imam Toufique* >>>>>>>>>>> *213-700-5485* >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Igor Cicimov | DevOps >>>>>>>>> >>>>>>>>> >>>>>>>>> p. +61 (0) 433 078 728 >>>>>>>>> e. [email protected] >>>>>>>>> <http://encompasscorporation.com/> >>>>>>>>> w*.* www.encompasscorporation.com >>>>>>>>> a. Level 4, 65 York Street, Sydney 2000 >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Regards, >>>>>>>> *Imam Toufique* >>>>>>>> *213-700-5485* >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Igor Cicimov | DevOps >>>>>>> >>>>>>> >>>>>>> p. +61 (0) 433 078 728 >>>>>>> e. [email protected] <http://encompasscorporation.com/> >>>>>>> w*.* www.encompasscorporation.com >>>>>>> a. Level 4, 65 York Street, Sydney 2000 >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Regards, >>>>>> *Imam Toufique* >>>>>> *213-700-5485* >>>>>> >>>>> >>>>> >>>>> -- >>>>> Igor Cicimov | DevOps >>>>> >>>>> >>>>> p. +61 (0) 433 078 728 >>>>> e. [email protected] <http://encompasscorporation.com/> >>>>> w*.* www.encompasscorporation.com >>>>> a. Level 4, 65 York Street, Sydney 2000 >>>>> >>>> >>>> >>>> -- >>>> Regards, >>>> *Imam Toufique* >>>> *213-700-5485* >>>> >>> >>> >>> -- >>> Igor Cicimov | DevOps >>> >>> >>> p. +61 (0) 433 078 728 >>> e. [email protected] <http://encompasscorporation.com/> >>> w*.* www.encompasscorporation.com >>> a. Level 4, 65 York Street, Sydney 2000 >>> >> >> >> -- >> Regards, >> *Imam Toufique* >> *213-700-5485* >> > > > -- > Igor Cicimov | DevOps > > > p. +61 (0) 433 078 728 > e. [email protected] <http://encompasscorporation.com/> > w*.* www.encompasscorporation.com > a. Level 4, 65 York Street, Sydney 2000 > -- Regards, *Imam Toufique* *213-700-5485*
