Re: Caching servers in Local ISPs !!
ok, but i have no idea why ISP is asking for BGP and matter of fact is,
i'll have to make BGP work somehow, so local caching server will fetch the
new subnets from ISP router automatically (and i don't know how).
Btw, our local ISP provided us with some testing ip prefixes to check nginx
based caching. i.e
geo {
default 0;
10.0.0.0/8
39.23.2.0/24 1;
112.50.192.0/18 1;
}
Now whenever we add the prefix 112.50.192.0/18 in geo {} , all the requests
coming from the 39.23.2.0/24 and 10.0.0.0/8 returns 504 gateway error and
videos failed to stream. To resolve this issue, we have to remove
112.50.192.0/18 1; from geo block.
On Wed, Jun 18, 2014 at 12:50 AM, itpp2012 nginx-fo...@nginx.us wrote:
You don't need to do anything with a dns that is only local to the clients
served by the ISP.
Suppose I am in Africa;
Question to my ISP: I'd like to go to new-york
ISP: new-york is located in south-Africa
Suppose I am in the US;
Question to my ISP: I'd like to go to new-york
ISP: new-york is located in the US
The DNS is just a pointer, where ever you have an edge server make the dns
name point to it, when not point the dns to origin.
Every ISP client gets the DNS servers from their ISP, its really simple.
Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,249997,250957#msg-250957
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
shahzaib1232 Wrote:
---
Btw, our local ISP provided us with some testing ip prefixes to check
nginx
based caching. i.e
geo {
default 0;
10.0.0.0/8
39.23.2.0/24 1;
112.50.192.0/18 1;
}
Typo??
geo {
default 0;
10.0.0.0/8 1;
39.23.2.0/24 1;
112.50.192.0/18 1;
}
Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,249997,251019#msg-251019
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
geo {
default 0;
10.0.0.0/8 1;
39.23.2.0/24 1;
112.50.192.0/18 1;
}
Sorry i didn't write accurately here but it is 10.0.0.0/8 1; in nginx
config, so the problem is not the wrong syntax for geo {}.
On Wed, Jun 18, 2014 at 11:24 PM, itpp2012 nginx-fo...@nginx.us wrote:
shahzaib1232 Wrote:
---
Btw, our local ISP provided us with some testing ip prefixes to check
nginx
based caching. i.e
geo {
default 0;
10.0.0.0/8
39.23.2.0/24 1;
112.50.192.0/18 1;
}
Typo??
geo {
default 0;
10.0.0.0/8 1;
39.23.2.0/24 1;
112.50.192.0/18 1;
}
Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,249997,251019#msg-251019
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
RE: Caching servers in Local ISPs !!
Hi, ok, but i have no idea why ISP is asking for BGP and matter of fact is, i'll have to make BGP work somehow, so local caching server will fetch the new subnets from ISP router automatically (and i don't know how). I strongly suggest you hire some consultant who can help you setting all those things up, because this is clearly a task too complex for a single mailing list thread and some nginx configurations. Also, why not host those file on a professional CDN instead of in-house? https://www.google.com/search?q=mp4+streaming+cdn Lukas ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
why not host those file on a professional CDN instead of in-house? Because 80% of the traffic is from our country and 50% of that traffic is from the ISP we're talking to and this is the reason we deployed the caching box on this ISP edge. On Thu, Jun 19, 2014 at 12:35 AM, Lukas Tribus luky...@hotmail.com wrote: Hi, ok, but i have no idea why ISP is asking for BGP and matter of fact is, i'll have to make BGP work somehow, so local caching server will fetch the new subnets from ISP router automatically (and i don't know how). I strongly suggest you hire some consultant who can help you setting all those things up, because this is clearly a task too complex for a single mailing list thread and some nginx configurations. Also, why not host those file on a professional CDN instead of in-house? https://www.google.com/search?q=mp4+streaming+cdn Lukas ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
On 18 Jun 2014 20:45, shahzaib shahzaib shahzaib...@gmail.com wrote: why not host those file on a professional CDN instead of in-house? Because 80% of the traffic is from our country and 50% of that traffic is from the ISP we're talking to and this is the reason we deployed the caching box on this ISP edge. But, as this now pretty off-topic thread is repeatedly demonstrating, you haven't deployed diddly squat. You've just chucked a server in a rack and are having to rely on unpaid, debugging-by-email advice from an pseudonymous mailing list to get it even near functional. Let alone properly defined and understood. If your *business* needs to do this, pay a professional person or organisation to help you like others have suggested. The alternative, which you appear to be ending up with, is a black box of hacks known only to yourself and potentially understood by no-one, which will SPoF on you, personally, until you leave that organisation. You don't want that. Trust me. /sysadmin Just my 2 cents, Jonathan ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
@Jonathon, yes you're right i should not post off-topic here, offcourse i
thought as nginx has tremendous amount of capabilities and there might be
alternative possibility of BGP too but i was wrong. I would be thankful if
you help me on ngx-http_geo_module as it is related to nginx and help me
with the following problem :-
-
Our local ISP provided us with some testing ip prefixes to check nginx
based caching. i.e
geo {
default 0;
10.0.0.0/8 1;
39.23.2.0/24 1;
112.50.192.0/18 1;
}
Now whenever we add the prefix 112.50.192.0/18 in geo {} , all the requests
coming from the 39.23.2.0/24 and 10.0.0.0/8 returns nginx 504 gateway error
and videos failed to stream. To resolve this issue, we have to remove
112.50.192.0/18 1; from geo block.
On Thu, Jun 19, 2014 at 1:05 AM, Jonathan Matthews cont...@jpluscplusm.com
wrote:
On 18 Jun 2014 20:45, shahzaib shahzaib shahzaib...@gmail.com wrote:
why not host those file on a professional CDN instead of in-house?
Because 80% of the traffic is from our country and 50% of that traffic
is from the ISP we're talking to and this is the reason we deployed the
caching box on this ISP edge.
But, as this now pretty off-topic thread is repeatedly demonstrating, you
haven't deployed diddly squat. You've just chucked a server in a rack and
are having to rely on unpaid, debugging-by-email advice from an
pseudonymous mailing list to get it even near functional. Let alone
properly defined and understood.
If your *business* needs to do this, pay a professional person or
organisation to help you like others have suggested. The alternative, which
you appear to be ending up with, is a black box of hacks known only to
yourself and potentially understood by no-one, which will SPoF on you,
personally, until you leave that organisation. You don't want that. Trust
me.
/sysadmin
Just my 2 cents,
Jonathan
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
shahzaib1232 Wrote: --- Thanks itpp but the issue is still same and still the ip is from the main server in inspect element as well as in local-caching nginx access logs, i am getting the client ip as main-server's ip instead of original client ip It could be the case the traffic you are getting is from the ISP proxy which could mean that any traffic is from that ISP only, which makes it easier to determine which to serve from local. Ask the ISP from where the traffic is coming from, if it is a proxy then proxy=local. So, the specific subnet coming from our ISP to the main server will be routed to the local caching server and our ISP will have to tell us each time to add specific ip prefix in the nginx config to route them towards their caching server. So the problem is, whenever few hundreds ip prefixes are added to their network, they'll have to provide us those prefixes in order to enable caching for newly added ips. See above, if this is not the case look into https://github.com/flant/nginx-http-rdns if a client has something like 'p1234.adsl-pool2-auckland.au' you can redirect based on a part of the client dns name, your ISP can tell you which DHCP named pools there are. If you can't get the client IP of hostname you gonna need to do some wiresharking to see where the info is, if it is anywhere. If the ISP is using a proxy to pass clients to your server ask them to add a header with the client ip/hostname. Posted at Nginx Forum: http://forum.nginx.org/read.php?2,249997,250950#msg-250950 ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
shahzaib1232 Wrote: --- i don't think the solution rdns will be suitable for us. I have checked the zebra software to make linux a BGP router http://www.techrepublic.com/article/use-zebra-to-set-up-a-linux-bgp-os pf-router/ Could you tell me if BGP is capable of doing what we want? Because our local ISP supports this method and i have no idea how to implement it. Functionality we need, is to auto detect the new ip prefixes from local ISP so they'll not have to provide us thousands of prefixes on daily basis. Why not use a DNS for the clients? your making things too complicated. Client-1-request@ISP-1 - edge1.streaming.au ISP-1-DNS - 12.34.56.78 (which is your edge box) Client-1-request@ISP-2 - edge1.streaming.au ISP-2-DNS - 99.88.77.66 (which is your box in the US) Anyone from ISP-1 will always be directed to the edge systems, anyone else to where-ever you point the dns. ISP's also use regional DNS servers which allows you more edge systems closer to the users. Anyway, BGP see http://bird.network.cz/ (netflix solution) Posted at Nginx Forum: http://forum.nginx.org/read.php?2,249997,250953#msg-250953 ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
Why not use a DNS for the clients? How i would be sure that request coming from the ISP-1 on the DNS server and then point it to the local caching server? I mean i can use View directive of BIND to route specific ips (local ISP clients) to the local caching server and what if tomorrow the ISP has added more clients to their network ? I'll also have those new ip prefixes to DNS server. Please correct me if i am wrong. On Tue, Jun 17, 2014 at 11:32 PM, itpp2012 nginx-fo...@nginx.us wrote: shahzaib1232 Wrote: --- i don't think the solution rdns will be suitable for us. I have checked the zebra software to make linux a BGP router http://www.techrepublic.com/article/use-zebra-to-set-up-a-linux-bgp-os pf-router/ Could you tell me if BGP is capable of doing what we want? Because our local ISP supports this method and i have no idea how to implement it. Functionality we need, is to auto detect the new ip prefixes from local ISP so they'll not have to provide us thousands of prefixes on daily basis. Why not use a DNS for the clients? your making things too complicated. Client-1-request@ISP-1 - edge1.streaming.au ISP-1-DNS - 12.34.56.78 (which is your edge box) Client-1-request@ISP-2 - edge1.streaming.au ISP-2-DNS - 99.88.77.66 (which is your box in the US) Anyone from ISP-1 will always be directed to the edge systems, anyone else to where-ever you point the dns. ISP's also use regional DNS servers which allows you more edge systems closer to the users. Anyway, BGP see http://bird.network.cz/ (netflix solution) Posted at Nginx Forum: http://forum.nginx.org/read.php?2,249997,250953#msg-250953 ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
Why not use a DNS for the clients? How i would be sure that request coming from the ISP-1 on the DNS server ? and then point it to the local caching server? I mean i can use View directive of BIND to route specific ips (local ISP clients) to the local caching server and what if tomorrow the ISP has added more clients to their network ? I'll also have to add those new ip prefixes to DNS server. On Tue, Jun 17, 2014 at 11:45 PM, shahzaib shahzaib shahzaib...@gmail.com wrote: Why not use a DNS for the clients? How i would be sure that request coming from the ISP-1 on the DNS server and then point it to the local caching server? I mean i can use View directive of BIND to route specific ips (local ISP clients) to the local caching server and what if tomorrow the ISP has added more clients to their network ? I'll also have those new ip prefixes to DNS server. Please correct me if i am wrong. On Tue, Jun 17, 2014 at 11:32 PM, itpp2012 nginx-fo...@nginx.us wrote: shahzaib1232 Wrote: --- i don't think the solution rdns will be suitable for us. I have checked the zebra software to make linux a BGP router http://www.techrepublic.com/article/use-zebra-to-set-up-a-linux-bgp-os pf-router/ Could you tell me if BGP is capable of doing what we want? Because our local ISP supports this method and i have no idea how to implement it. Functionality we need, is to auto detect the new ip prefixes from local ISP so they'll not have to provide us thousands of prefixes on daily basis. Why not use a DNS for the clients? your making things too complicated. Client-1-request@ISP-1 - edge1.streaming.au ISP-1-DNS - 12.34.56.78 (which is your edge box) Client-1-request@ISP-2 - edge1.streaming.au ISP-2-DNS - 99.88.77.66 (which is your box in the US) Anyone from ISP-1 will always be directed to the edge systems, anyone else to where-ever you point the dns. ISP's also use regional DNS servers which allows you more edge systems closer to the users. Anyway, BGP see http://bird.network.cz/ (netflix solution) Posted at Nginx Forum: http://forum.nginx.org/read.php?2,249997,250953#msg-250953 ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
You don't need to do anything with a dns that is only local to the clients served by the ISP. Suppose I am in Africa; Question to my ISP: I'd like to go to new-york ISP: new-york is located in south-Africa Suppose I am in the US; Question to my ISP: I'd like to go to new-york ISP: new-york is located in the US The DNS is just a pointer, where ever you have an edge server make the dns name point to it, when not point the dns to origin. Every ISP client gets the DNS servers from their ISP, its really simple. Posted at Nginx Forum: http://forum.nginx.org/read.php?2,249997,250957#msg-250957 ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
Hello itpp,i have been abled to use ngx_http_geo_module. Now the request
coming from local ISP will first go to the main server (US) and then main
server will check if the ip is 1.2.3.4 so it'll direct the request to the
local caching server and than caching server will check if the file is
cached or it should again get the file from main server and cache it
locally.
When i tested it locally, it worked fine but the file URL in firebug is
coming from MAIN server when it should have come from the local caching
server. I can also see the caching directory size increases when the
matching client via geo module is directed to the local caching server but
the URL remains the same in firebug.
US config :-
geo $TW {
default 0;
192.168.1.0/24 1;
}
server {
listen 80;
server_name 002.files.com;
# limit_rate 600k;
location / {
root /var/www/html/files;
index index.html index.htm index.php;
# autoindex on;
}
location ~ \.(mp4|jpeg|jpg)$ {
mp4;
root /var/www/html/files;
if ($TW) {
proxy_pass http://192.168.22.32:80;
}
expires 7d;
valid_referers none blocked domain.com *.domain.com blog.domain.com
*.facebook.com *.twitter.com *.files.com *.pump.net domain.tv *.domain.tv
domainmedia.tv www.domainmedia.tv embed.domainmedia.tv;
if ($invalid_referer) {
return 403;
}
}
}
Edge config :-
proxy_ignore_headers Set-Cookie;
proxy_cache_path /data/nginx/cache levels=1:2 keys_zone=static:100m
loader_threshold=300 loader_files=10 inactive=1d
max_size=30m;
proxy_temp_path /data/nginx/tmp 1 2;
add_header X-Cache-Status $upstream_cache_status;
add_header Accept-Ranges bytes;
max_ranges 512;
server {
listen 80;
server_name 192.168.22.32;
root /var/www/html/files;
location ~ \.(mp4|jpeg|jpg)$ {
root /var/www/html/files;
mp4;
try_files $uri @getfrom_origin;
}
location @getfrom_origin {
proxy_pass http://002.files.com:80;
proxy_cache_valid 200 302 60m;
proxy_cache_valid any 1m;
proxy_cache static;
proxy_cache_min_uses 1;
}
Maybe i need to add some variable to get original server ip ?
http://nginx.org/en/docs/http/ngx_http_geo_module.html
http://nginx.org/en/docs/http/ngx_http_geo_module.html
On Fri, Jun 6, 2014 at 8:56 PM, shahzaib shahzaib shahzaib...@gmail.com
wrote:
Thanks a lot itpp. :) I'll look into it and get back to you.
Thanks again for quick solution :)
On Fri, Jun 6, 2014 at 8:26 PM, itpp2012 nginx-fo...@nginx.us wrote:
shahzaib1232 Wrote:
---
@itpp I am currenlty proceeding with proxy_cache method just because i
had
to done this in emergency mode due to boss pressure :-|. I have a
quick
question, can i make nginx to cache files for specific clients ?
Like, if our caching servers are deployed by only single ISP named
ptcl.
So if ip from ptcl client is browsing video, only his requested file
should
be cached not for any other client, does nginx support that ??
You could do this based on some IP ranges or via
https://github.com/flant/nginx-http-rdns
See
http://serverfault.com/questions/380642/nginx-how-to-redirect-users-with-certain-ip-to-special-page
and
http://www.cyberciti.biz/faq/nginx-redirect-backend-traffic-based-upon-client-ip-address/
Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,249997,250707#msg-250707
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
shahzaib1232 Wrote: --- Maybe i need to add some variable to get original server ip ? https://www.google.nl/#q=nginx+geo+remote+ip+address http://www.cyberciti.biz/faq/nginx-redirect-backend-traffic-based-upon-client-ip-address/ Posted at Nginx Forum: http://forum.nginx.org/read.php?2,249997,250895#msg-250895 ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
@itpp I am currenlty proceeding with proxy_cache method just because i had
to done this in emergency mode due to boss pressure :-|. I have a quick
question, can i make nginx to cache files for specific clients ?
Like, if our caching servers are deployed by only single ISP named ptcl.
So if ip from ptcl client is browsing video, only his requested file should
be cached not for any other client, does nginx support that ??
I know its kind of funny, but i've to complete this task :(
On Thu, Jun 5, 2014 at 12:23 AM, shahzaib shahzaib shahzaib...@gmail.com
wrote:
Also sync to a temp folder and move after completion or nginx will
attempt
to send partial files.
Oh right. Thanks for quick help and suggestion :). I'll look into wanproxy
now.
On Thu, Jun 5, 2014 at 12:19 AM, itpp2012 nginx-fo...@nginx.us wrote:
shahzaib1232 Wrote:
---
@itpp, i just used your method try_files and it worked flawlessly :).
Following is the testing config :
server {
listen 80;
server_name domain.com;
root /var/www/html/files;
location / {
location ~* (\.mp3|\.avi|\.mp4)$ {
Should i use rsync or lsync for mirroring the files between Origin and
caching server ?
Whatever works for you, I'd prefer rsync since that's easier to schedule
for
off-peek hours.
Also sync to a temp folder and move after completion or nginx will attempt
to send partial files.
see also http://wanproxy.org/
Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,249997,250645#msg-250645
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
shahzaib1232 Wrote: --- @itpp I am currenlty proceeding with proxy_cache method just because i had to done this in emergency mode due to boss pressure :-|. I have a quick question, can i make nginx to cache files for specific clients ? Like, if our caching servers are deployed by only single ISP named ptcl. So if ip from ptcl client is browsing video, only his requested file should be cached not for any other client, does nginx support that ?? You could do this based on some IP ranges or via https://github.com/flant/nginx-http-rdns See http://serverfault.com/questions/380642/nginx-how-to-redirect-users-with-certain-ip-to-special-page and http://www.cyberciti.biz/faq/nginx-redirect-backend-traffic-based-upon-client-ip-address/ Posted at Nginx Forum: http://forum.nginx.org/read.php?2,249997,250707#msg-250707 ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
Thanks a lot itpp. :) I'll look into it and get back to you. Thanks again for quick solution :) On Fri, Jun 6, 2014 at 8:26 PM, itpp2012 nginx-fo...@nginx.us wrote: shahzaib1232 Wrote: --- @itpp I am currenlty proceeding with proxy_cache method just because i had to done this in emergency mode due to boss pressure :-|. I have a quick question, can i make nginx to cache files for specific clients ? Like, if our caching servers are deployed by only single ISP named ptcl. So if ip from ptcl client is browsing video, only his requested file should be cached not for any other client, does nginx support that ?? You could do this based on some IP ranges or via https://github.com/flant/nginx-http-rdns See http://serverfault.com/questions/380642/nginx-how-to-redirect-users-with-certain-ip-to-special-page and http://www.cyberciti.biz/faq/nginx-redirect-backend-traffic-based-upon-client-ip-address/ Posted at Nginx Forum: http://forum.nginx.org/read.php?2,249997,250707#msg-250707 ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
@itpp, i just used your method try_files and it worked flawlessly :).
Following is the testing config :
server {
listen 80;
server_name domain.com;
root /var/www/html/files;
location / {
root /var/www/html/files;
try_files $uri @getfrom_origin;
}
location @getfrom_origin {
proxy_pass http://127.0.0.1:8080;
}
}
if proxy_pass worked for localhost, i hope it will also work for remote
host to forward request if the file doesn't exist on local caching server.
:-)
Would you suggest me to add some more configs for tweaking on nginx ?
Btw, proxy_pass should only be for mp4 and jpeg, cause the caching is only
for video files.
Should i use rsync or lsync for mirroring the files between Origin and
caching server ?
Suggestion will be highly appreciated.
Regards.
Shahzaib
On Wed, May 28, 2014 at 4:19 PM, shahzaib shahzaib shahzaib...@gmail.com
wrote:
Right. I'll proceed with my research and get back to you with better
approach . :)
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
Also sync to a temp folder and move after completion or nginx will attempt
to send partial files.
Oh right. Thanks for quick help and suggestion :). I'll look into wanproxy
now.
On Thu, Jun 5, 2014 at 12:19 AM, itpp2012 nginx-fo...@nginx.us wrote:
shahzaib1232 Wrote:
---
@itpp, i just used your method try_files and it worked flawlessly :).
Following is the testing config :
server {
listen 80;
server_name domain.com;
root /var/www/html/files;
location / {
location ~* (\.mp3|\.avi|\.mp4)$ {
Should i use rsync or lsync for mirroring the files between Origin and
caching server ?
Whatever works for you, I'd prefer rsync since that's easier to schedule
for
off-peek hours.
Also sync to a temp folder and move after completion or nginx will attempt
to send partial files.
see also http://wanproxy.org/
Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,249997,250645#msg-250645
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
shahzaib1232 Wrote:
---
Thanks itpp. So according to you the more preferable way is
non-caching
replication of main content servers to local ISP servers and than run
rsync
periodically for replication. So if file not found in local ISP nginx,
it
should forward request to main content server using try_files method ?
@itpp22, proxy_cache is eating too much memory and cpu on local testing.
1
nginx process was using 1G of Ram and load-avg reached to 22, eventually
local server got killed.
Local server specs are : 8cores and 4G ram.
Looks like proxy_cache will not be suitable method for 3000+ concerrent
connections on local ISP caching server.
A HLD should have pointed that out :)
You should change your thinking what caching actually is, just because some
functionality is called 'caching' it doesn't mean anything else is not.
If exist do {
present local via local interfaces
} else {
present remote via remote interfaces
}
Another (none-nginx) process (provisioning) controls which remote components
flows to local (or is removed) via provisioning interfaces.
Look at something simple like a filesystem with replication, in it's basics
it's not a cache but more a glorified 2way rsync solution. However global
files are always local to be determined to which resource you are connecting
to, yet the filesystem doesn't need to cache much no matter how many files
it serves, which doesn't defer much from how nxing would work.
Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,249997,250423#msg-250423
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
Right. I'll proceed with my research and get back to you with better approach . :) ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
Hello , Came after a long time to this thread. I have finished the caching method. I've used bind-geoip to resolve all requests from asia against the local ISP caching server ips. Now, here's the most critical part. Suppose a request came from asia for the video file which is not yet downloaded on local caching server, how will the local nginx will deal with it ? I want it to forward that request to the main Server and keep forwarding request until the next rsync runs and new files propagate to the local caching server. Please point me to some nginx guide if this feature works well with nginx. Regards. On Sat, May 10, 2014 at 3:39 PM, itpp2012 nginx-fo...@nginx.us wrote: See http://en.wikipedia.org/wiki/Content_delivery_network and http://en.wikipedia.org/wiki/File:Akamaiprocess.png Make yourself a HLD (high level design) before getting to technology. Posted at Nginx Forum: http://forum.nginx.org/read.php?2,249997,250007#msg-250007 ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
I'd call that provisioning, a simple try_files directive can deal with that while the provisioning service syncs. http://nginx.org/en/docs/http/ngx_http_core_module.html#try_files Then jump left for cached contents or jump right for non-cached contents. Posted at Nginx Forum: http://forum.nginx.org/read.php?2,249997,250385#msg-250385 ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
@itpp, thanks for reply. I am working on proxy_cache . My all webservers in
local ISP will be proxy servers and all requests regarding flv,mp4,jpeg
from asia will be route through the proxy servers and data will be cache in
proxy servers. I think, this will help me with caching. I am testing it on
local and its working as it should. Following is the edge server config :
proxy_cache_path /data/nginx/cache levels=1:2 keys_zone=static:1000m;
proxy_temp_path /tmp;
server {
listen 80;
server_name edge.com;
# proxy_cache one;
root /var/www/html/test;
access_log /var/log/nginx/access.log ;
error_log /var/log/nginx/error.log;
location / {
proxy_pass http://origin.com:80;
proxy_cache static;
proxy_cache_min_uses 1;
proxy_cache_valid 200 301 302 120m;
#proxy_cache_valid 404 1m;
}
}
Please let me know if the method is correct.
On Tue, May 27, 2014 at 3:55 PM, itpp2012 nginx-fo...@nginx.us wrote:
I'd call that provisioning, a simple try_files directive can deal with that
while the provisioning service syncs.
http://nginx.org/en/docs/http/ngx_http_core_module.html#try_files
Then jump left for cached contents or jump right for non-cached contents.
Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,249997,250385#msg-250385
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
I'm not seeing where you determine if a file is present on an edge system,
the try_files directive allows you to redirect or not (locally present or
not). I don't really see a point for using (proxy)cache, the file is present
or it is not.
location / {
try_files $uri @getfrom_origin;
}
location @getfrom_origin {
proxy_pass ..;
}
Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,249997,250393#msg-250393
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
Hello itpp,
the Local caching server is nothing but a proxy server which
will route all traffic from asia to origin server and will cache the
requested file in /data/nginx/cache folder. So when the request made again,
it'll be served from cache.
I'm not seeing where you determine if a file is present on an edge system,
Edge system will route request for new file to origin server and will cache
the resulted file to edge server,so when the new request for same file
will arrive, nginx edge server will serve it from /data/nginx/cache folder.
I did experiment on locally and by checking the logs, i think its working .
I got hint from following link :
http://nginx.com/resources/admin-guide/caching/
On Tue, May 27, 2014 at 6:00 PM, itpp2012 nginx-fo...@nginx.us wrote:
I'm not seeing where you determine if a file is present on an edge system,
the try_files directive allows you to redirect or not (locally present or
not). I don't really see a point for using (proxy)cache, the file is
present
or it is not.
location / {
try_files $uri @getfrom_origin;
}
location @getfrom_origin {
proxy_pass ..;
}
Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,249997,250393#msg-250393
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
Of course this will work but when nginx restarts or the cache gets damaged/corrupted the whole shebang starts again And you can't control the cached files versus try_files and a normal filesystem. Posted at Nginx Forum: http://forum.nginx.org/read.php?2,249997,250395#msg-250395 ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
Thanks for replying guyz. Can i use nginx (origin and edge) ? As the question in following link. http://stackoverflow.com/questions/10024981/distributed-cached-mp4-pseudostreaming-seeking-with-nginx If i use the origin and edge method, i think i'll change my application codes to redirect local country traffic to edge webservers (ISP caching server for video files) and that edge server will check if the requested file is not in cache and it'll fetch the requested video file from origin web-server located in U.S and cache it to local. For this procedure, I'll have to configure DNS A entries against local ISP caching servers and put those DNS to my application code to stream videos from those LOCAL CACHING SERVERS for specific country. Please correct me if i am wrong. On Sat, May 10, 2014 at 9:24 AM, Steve Holdoway st...@greengecko.co.nzwrote: You might want to look at lsyncd - a GZSOC project - to ease the synchronisation. I have had good results with it. Steve On Sat, 2014-05-10 at 00:22 +0500, shahzaib shahzaib wrote: @itpp thanks for replying. So on easy note, i would have to assign those machines the preferred dns and use rsync on regular basis in order to make identical data between local caching machines and main front end content servers ? What if a client request a video which is not in local caching server ? Does nginx has the configuration for it to check the files locally and then forward the request to main content servers if requested file is not cached locally ? I need a bit of guidance in order to configure nginx this way. Shahzaib On Fri, May 9, 2014 at 11:49 PM, itpp2012 nginx-fo...@nginx.us wrote: Its quite simple, think of it this way, a DNS entry does not have to point to the same IP everywhere. Place your cache machines at a ISP, have them assign its IP to your preferred dns name, thats about it. The rest like distribution works like a reverse riverbed with a master mirror, rsync or the likes. And of course this can all be done with nginx at all locations. Posted at Nginx Forum: http://forum.nginx.org/read.php?2,249997,25#msg-25 ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx -- Steve Holdoway BSc(Hons) MIITP http://www.greengecko.co.nz Linkedin: http://www.linkedin.com/in/steveholdoway Skype: sholdowa ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
See http://en.wikipedia.org/wiki/Content_delivery_network and http://en.wikipedia.org/wiki/File:Akamaiprocess.png Make yourself a HLD (high level design) before getting to technology. Posted at Nginx Forum: http://forum.nginx.org/read.php?2,249997,250007#msg-250007 ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
@Rainer, we're already in contact with one of our Country's ISP(80% of the country users are using that ISP services) .So, they can do much better work than the DNS sites you provided because we only required caching for our country. On Fri, May 9, 2014 at 10:18 PM, Rainer Duffner rai...@ultra-secure.dewrote: Am 09.05.2014 um 16:58 schrieb shahzaib shahzaib shahzaib...@gmail.com: Hello, We're running a high traffic website similar to youtube.com. Due to high bandwidth utilization over the network, we're in contact with the local ISP in order to put caching server to reduce bandwidth utilization for file streaming. Our main front end content servers (nginx) are located in U.S and we want to put caching servers in ASIA as most of the traffic is originating from asia. We've no idea how this caching would work. Would the caching servers will be configured and deployed by Local ISP ? or we're required to do some work with our application coding ? I know it is bit off topic on nginx forum. But this is the only forum that is intensively active and helpful. Please guide me a bit, i am new to this caching environment. I think you would need to move your DNS to somebody like easydns.com or dyn.com (just of the top of my head, there are probably lots more) and use their geo-location feature. That way, somebody with an IP from Asia will receive the IP of your Asian server when it asks DNS about the IP of your domain-name. Maybe somebody who has done this before can comment - I’ve looked into it, but never seriously. ;-) ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
So on easy note, i would have to assign those machines the preferred dns and use rsync on regular basis in order to make identical data between local caching machines and main front end content servers ? Yep. What if a client request a video which is not in local caching server ? You need to maintain a cache index on each cache machine in order to determine what is available to the users, for most content you need to do this anyway since not all content can legally be everywhere and you also might want to customize what you present for each region. Does nginx has the configuration for it to check the files locally and then forward the request to main content servers if requested file is not cached locally ? There are many ways to do this with nginx and Lua but a independent cache index would be much better, with it you can do much more like redirect a content source from elsewhere depending on load and demand. You simply feed nginx the cache index. A very simplistic cache index system is abusing a local (local to nginx) dns server, assign local IP's to resources and change them according to load and demand. Again for a local DNS you can assign whatever you want to a dns name, with a local ttl of 15 seconds and nginx loadbalancing between 4 regional resources it will be peanuts to change the load based on demand (provided you have monitoring in place which can act on such data). Basically a DIY BGP :) Posted at Nginx Forum: http://forum.nginx.org/read.php?2,249997,250002#msg-250002 ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
Re: Caching servers in Local ISPs !!
You might want to look at lsyncd - a GZSOC project - to ease the synchronisation. I have had good results with it. Steve On Sat, 2014-05-10 at 00:22 +0500, shahzaib shahzaib wrote: @itpp thanks for replying. So on easy note, i would have to assign those machines the preferred dns and use rsync on regular basis in order to make identical data between local caching machines and main front end content servers ? What if a client request a video which is not in local caching server ? Does nginx has the configuration for it to check the files locally and then forward the request to main content servers if requested file is not cached locally ? I need a bit of guidance in order to configure nginx this way. Shahzaib On Fri, May 9, 2014 at 11:49 PM, itpp2012 nginx-fo...@nginx.us wrote: Its quite simple, think of it this way, a DNS entry does not have to point to the same IP everywhere. Place your cache machines at a ISP, have them assign its IP to your preferred dns name, thats about it. The rest like distribution works like a reverse riverbed with a master mirror, rsync or the likes. And of course this can all be done with nginx at all locations. Posted at Nginx Forum: http://forum.nginx.org/read.php?2,249997,25#msg-25 ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx -- Steve Holdoway BSc(Hons) MIITP http://www.greengecko.co.nz Linkedin: http://www.linkedin.com/in/steveholdoway Skype: sholdowa ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
