Inspired by unveiltech.com I have tried to write a StoreID http
interface and to try to help cache\de-duplicate couple websites.
The helpers I have written can be found at:
http://www1.ngtech.co.il/squid/storeid/ng-storeid.pl
http://www1.ngtech.co.il/squid/storeid/ng-storeid.py
http://www1.ngtech.co.il/squid/storeid/ng-storeid.rb
The interface supports keep-alive http connections if you want to use it
in your helper.
The helpers use concurrency and only with concurrency.
Example settings for storeid in squid:
#START
store_id_program /opt/storeid/ng-storeid.rb
store_id_children 25 startup=10 idle=5 concurrency=50
acl storeiddoms dstdomain .youtube.com .googlevideo.com .vimeo.com
.media-imdb.com
store_id_access allow storeiddoms !CONNECT
store_id_access deny all
refresh_pattern -i ^http:\/\/[a-zA-Z0-9\-\_\.]+\.squid\.internal\/.*
3600 95% 86400 reload-into-ims ignore-reload ignore-no-store ignore-private
#END
About the service:
- The service do not guaranteed to provide what you want or need.
- The service is in testing\debugging state for now and until future notice.
- The service is gathering information for analysis and statistics.
- The service is open for anyone to use with some certain basic
constrains of the hardware and spare bandwidth of the service provider.
- Keep in mind that for now there is DOS monitoring on the service and
you can get white-listed via email request.
- The service is not 100% bullet proof and will might get down for
maintenance at the first stages of operation.(updated helper with
failure detection and crash prevention will be provided later).
- The service do not and will not ever help to de-duplicate\cache or
analyze in-appropriate content in any form or format.
In the case you want me to analyze a specific website for StoreID
compatibility feel free to contact me via my email and I will provide
you with the changes needed for the purpose of analyzing it(not
promising it is possible for any website).
Any Comments are more then just welcomed,
Eliezer Croitoru
