Re: Fwd: [squid-users] store-id.pl doesnt cache youtube
On Tue, Jul 9, 2013 at 1:32 PM, Amos Jeffries squ...@treenet.co.nz wrote: On 8/07/2013 6:34 p.m., Alan wrote: On Mon, Jul 8, 2013 at 6:25 AM, Eliezer Croitoru elie...@ngtech.co.il wrote: try this if you want to try something new. https://github.com/elico/squid-helpers/blob/master/squid_helpers/store-id.pl Eliezer Hi Eliezer, I read your script, and I have a suggestion. How about you store the matching urls in a separate file or database? That way the script would remain the same even if some website changes their url scheme. When the squid admin wants to update the file/db he can just issue squid -k reconfigure and the script would reload the file/db. I just came up with this simple script, based on yours. I haven't tested it though, since 3.4 head segfaults for me (btw, which revision are you using?). The invocation for my script should be, for example: store_id_program /etc/squid/storeid.pl /etc/squid/storeid.txt By the way, thanks for your contributions in this mailing list, they are very helpful. Best regards, Alan PS: Had to resend to the mailing list because it doesn't allow attachments. Here are the attachments: storeid.pl script: http://pastebin.ca/2420563 storeid.txt file: http://pastebin.ca/2420565 Nice. If you would like to nominate a GPLv2 compatible license to this .pl script I would be happy adding it to the Squid package. I have only been rejecting the store-URL scripts earlier only because they hard-coded the patterns. Amos Hi Amos, Feel free to include it in Squid using the GPLv2 license, that would be an honor. But I just noticed it is missing a $|=1 to prevent Perl's output buffering. And in order to make the DB less Perl-centric, I wrote an alternative version that you can see here: http://pastebin.ca/2422099 The other version had Perl code in the DB, which can be quite powerful, but less language independent. Which one is better? I guess it depends more or less on what kind of things we will see in the DB. This new version uses a DB that looks like this: http://pastebin.ca/2422105 Best regards, Alan
Re: Fwd: [squid-users] store-id.pl doesnt cache youtube
FYI: Alan has provided a slightly more polished helper and I have accepted it into the main Squid sources. You will find it being built by the next 3.HEAD snapshots under the name storeid_file_rewrite. Please send further additions or alterations to the helper itself through the regular code auditing process: mail to squid-dev list a unified diff patch against the 3.HEAD code with [PATCH] in the email subject and a descriptive commit message in the email text. Cheers Amos
Re: Fwd: [squid-users] store-id.pl doesnt cache youtube
Thanks Amos, I am very happy there is a helper which Is more plural then my ruby helper. I do understand the differences between ruby and other languages. I think that compared to all the helpers that already existed the one that I wrote gives a great example of how we can describe things in a way someone else can understand. Now That we have a simple general usage helper we can just decrypt the mysteries of the web. Just reminding that in the old days SSL-BUMP was illegal and now a days the only place you won't see it done is on enterprises with squid since they don't want the users to know that they are bumping the traffic and they still know it.. Eliezer On 07/09/2013 02:20 PM, Amos Jeffries wrote: FYI: Alan has provided a slightly more polished helper and I have accepted it into the main Squid sources. You will find it being built by the next 3.HEAD snapshots under the name storeid_file_rewrite. Please send further additions or alterations to the helper itself through the regular code auditing process: mail to squid-dev list a unified diff patch against the 3.HEAD code with [PATCH] in the email subject and a descriptive commit message in the email text. Cheers Amos
Fwd: [squid-users] store-id.pl doesnt cache youtube
On Mon, Jul 8, 2013 at 6:25 AM, Eliezer Croitoru elie...@ngtech.co.il wrote: try this if you want to try something new. https://github.com/elico/squid-helpers/blob/master/squid_helpers/store-id.pl Eliezer Hi Eliezer, I read your script, and I have a suggestion. How about you store the matching urls in a separate file or database? That way the script would remain the same even if some website changes their url scheme. When the squid admin wants to update the file/db he can just issue squid -k reconfigure and the script would reload the file/db. I just came up with this simple script, based on yours. I haven't tested it though, since 3.4 head segfaults for me (btw, which revision are you using?). The invocation for my script should be, for example: store_id_program /etc/squid/storeid.pl /etc/squid/storeid.txt By the way, thanks for your contributions in this mailing list, they are very helpful. Best regards, Alan PS: Had to resend to the mailing list because it doesn't allow attachments. Here are the attachments: storeid.pl script: http://pastebin.ca/2420563 storeid.txt file: http://pastebin.ca/2420565
Re: Fwd: [squid-users] store-id.pl doesnt cache youtube
Happy for the responses. On 07/08/2013 09:34 AM, Alan wrote: On Mon, Jul 8, 2013 at 6:25 AM, Eliezer Croitoru elie...@ngtech.co.il wrote: try this if you want to try something new. https://github.com/elico/squid-helpers/blob/master/squid_helpers/store-id.pl Eliezer Hi Eliezer, I read your script, and I have a suggestion. How about you store the matching urls in a separate file or database? Which I was thinking about long ago.. That way the script would remain the same even if some website changes their url scheme. When the squid admin wants to update the file/db he can just issue squid -k reconfigure and the script would reload the file/db. This is what happens today actually when you reload and use a new script. I just came up with this simple script, based on yours. I haven't tested it though, since 3.4 head segfaults for me (btw, which revision are you using?). # squid -v Squid Cache: Version 3.HEAD-20130520-r12839 configure options: '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--exec_prefix=/usr' '--libexecdir=/usr/lib64/squid' '--localstatedir=/var' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--with-logdir=$(localstatedir)/log/squid' '--with-pidfile=$(localstatedir)/run/squid.pid' '--disable-dependency-tracking' '--enable-eui' '--enable-follow-x-forwarded-for' '--enable-auth' '--enable-auth-basic=DB,LDAP,MSNT,MSNT-multi-domain,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB,getpwnam' '--enable-auth-ntlm=smb_lm,fake' '--enable-auth-digest=file,LDAP,eDirectory' '--enable-auth-negotiate=kerberos,wrapper' '--enable-external-acl-helpers=wbinfo_group,kerberos_ldap_group,AD_group' '--enable-cache-digests' '--enable-cachemgr-hostname=localhost' '--enable-delay-pools' '--enable-epoll' '--enable-icap-client' '--enable-ident-lookups' '--enable-linux-netfilter' '--enable-removal-policies=heap,lru' '--enable-snmp' '--enable-ssl' '--enable-storeio=aufs,diskd,ufs' '--enable-wccpv2' '--enable-esi' '--with-aio' '--with-default-user=squid' '--with-filedescriptors=16384' '--with-dl' '--with-pthreads' '--enable-ecap' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'target_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fpie' 'LDFLAGS=-pie' 'CXXFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fpie' 'PKG_CONFIG_PATH=/usr/lib64/pkgconfig:/usr/share/pkgconfig' The invocation for my script should be, for example: store_id_program /etc/squid/storeid.pl /etc/squid/storeid.txt By the way, thanks for your contributions in this mailing list, they are very helpful. I am working on a new document which will describe squid internals that have been used in order to allow StoreID. Since I am good at patterns your examples are helpful but I am working on other things I wouldn't be able to design the DB in a sec. Eliezer Best regards, Alan PS: Had to resend to the mailing list because it doesn't allow attachments. Here are the attachments: storeid.pl script: http://pastebin.ca/2420563 storeid.txt file: http://pastebin.ca/2420565
Re: Fwd: [squid-users] store-id.pl doesnt cache youtube
On 8/07/2013 6:34 p.m., Alan wrote: I just came up with this simple script, based on yours. I haven't tested it though, since 3.4 head segfaults for me (btw, which revision are you using?). backtrcace from that crash to bugzilla please. Amos
Re: Fwd: [squid-users] store-id.pl doesnt cache youtube
On 07/08/2013 09:34 AM, Alan wrote: On Mon, Jul 8, 2013 at 6:25 AM, Eliezer Croitoru elie...@ngtech.co.il wrote: try this if you want to try something new. https://github.com/elico/squid-helpers/blob/master/squid_helpers/store-id.pl Eliezer SNIP Hey Alan, I have just updated the StoreID wiki and my helper at the wiki to help others understand the idea of the DB and since it should work I do not see any problem with using it, just test it first a bit. The next updates are at: http://wiki.squid-cache.org/Features/StoreID my ruby helper: http://wiki.squid-cache.org/Features/StoreID/Helper And since your DB design is pretty simple It can be used even if just in order to post new patterns at: http://wiki.squid-cache.org/Features/StoreID/DB I will try my best at using this DB to post new patterns. Notice that in this DB you can use Many To One which means that couple sites\urls\patterns can refer to the same file which I have seen in the past. Couple very big sites refer to the same objects which has even a hash in the url and for example: http://www1.example.com/some_garbage_collection/MD5_HASH_OF_THE_FILE/some_more_garbage.hch?even_more_stuff http://www.subdomain.super.server.example_test.net/MD5_HASH_OF_THE_FILE/some_more_garbage.exe?even_more_stuff That you can refer to as only the http://hash_db.squid.internal/md5_hash; Best Regards, Eliezer Alan PS: Had to resend to the mailing list because it doesn't allow attachments. Here are the attachments: storeid.pl script: http://pastebin.ca/2420563 storeid.txt file: http://pastebin.ca/2420565
Re: Fwd: [squid-users] store-id.pl doesnt cache youtube
On 8/07/2013 6:34 p.m., Alan wrote: On Mon, Jul 8, 2013 at 6:25 AM, Eliezer Croitoru elie...@ngtech.co.il wrote: try this if you want to try something new. https://github.com/elico/squid-helpers/blob/master/squid_helpers/store-id.pl Eliezer Hi Eliezer, I read your script, and I have a suggestion. How about you store the matching urls in a separate file or database? That way the script would remain the same even if some website changes their url scheme. When the squid admin wants to update the file/db he can just issue squid -k reconfigure and the script would reload the file/db. I just came up with this simple script, based on yours. I haven't tested it though, since 3.4 head segfaults for me (btw, which revision are you using?). The invocation for my script should be, for example: store_id_program /etc/squid/storeid.pl /etc/squid/storeid.txt By the way, thanks for your contributions in this mailing list, they are very helpful. Best regards, Alan PS: Had to resend to the mailing list because it doesn't allow attachments. Here are the attachments: storeid.pl script: http://pastebin.ca/2420563 storeid.txt file: http://pastebin.ca/2420565 Nice. If you would like to nominate a GPLv2 compatible license to this .pl script I would be happy adding it to the Squid package. I have only been rejecting the store-URL scripts earlier only because they hard-coded the patterns. Amos
Re: Fwd: [squid-users] store-id.pl doesnt cache youtube
On 07/09/2013 07:32 AM, Amos Jeffries wrote: On 8/07/2013 6:34 p.m., Alan wrote: On Mon, Jul 8, 2013 at 6:25 AM, Eliezer Croitoru elie...@ngtech.co.il wrote: try this if you want to try something new. https://github.com/elico/squid-helpers/blob/master/squid_helpers/store-id.pl Eliezer Hi Eliezer, I read your script, and I have a suggestion. How about you store the matching urls in a separate file or database? That way the script would remain the same even if some website changes their url scheme. When the squid admin wants to update the file/db he can just issue squid -k reconfigure and the script would reload the file/db. I just came up with this simple script, based on yours. I haven't tested it though, since 3.4 head segfaults for me (btw, which revision are you using?). The invocation for my script should be, for example: store_id_program /etc/squid/storeid.pl /etc/squid/storeid.txt By the way, thanks for your contributions in this mailing list, they are very helpful. Best regards, Alan PS: Had to resend to the mailing list because it doesn't allow attachments. Here are the attachments: storeid.pl script: http://pastebin.ca/2420563 storeid.txt file: http://pastebin.ca/2420565 Nice. If you would like to nominate a GPLv2 compatible license to this .pl script I would be happy adding it to the Squid package. I have only been rejecting the store-URL scripts earlier only because they hard-coded the patterns. Amos Well Then we need to make sure the algorithm of the patterns is a bit more efficient. Then we can have the DB in wiki?? or sharing the patters between users can be done only on pastebin and mailing lists? Eliezer
Re: Fwd: [squid-users] store-id.pl doesnt cache youtube
On 9/07/2013 4:51 p.m., Eliezer Croitoru wrote: On 07/09/2013 07:32 AM, Amos Jeffries wrote: On 8/07/2013 6:34 p.m., Alan wrote: On Mon, Jul 8, 2013 at 6:25 AM, Eliezer Croitoru elie...@ngtech.co.il wrote: try this if you want to try something new. https://github.com/elico/squid-helpers/blob/master/squid_helpers/store-id.pl Eliezer Hi Eliezer, I read your script, and I have a suggestion. How about you store the matching urls in a separate file or database? That way the script would remain the same even if some website changes their url scheme. When the squid admin wants to update the file/db he can just issue squid -k reconfigure and the script would reload the file/db. I just came up with this simple script, based on yours. I haven't tested it though, since 3.4 head segfaults for me (btw, which revision are you using?). The invocation for my script should be, for example: store_id_program /etc/squid/storeid.pl /etc/squid/storeid.txt By the way, thanks for your contributions in this mailing list, they are very helpful. Best regards, Alan PS: Had to resend to the mailing list because it doesn't allow attachments. Here are the attachments: storeid.pl script: http://pastebin.ca/2420563 storeid.txt file: http://pastebin.ca/2420565 Nice. If you would like to nominate a GPLv2 compatible license to this .pl script I would be happy adding it to the Squid package. I have only been rejecting the store-URL scripts earlier only because they hard-coded the patterns. Amos Well Then we need to make sure the algorithm of the patterns is a bit more efficient. Then we can have the DB in wiki?? or sharing the patters between users can be done only on pastebin and mailing lists? As long as it is legal I don't have any opinion on how you decide to distribute the DB. It is just that the source code is semi-permanent once it commits and the patterns far too volatile for that type of publishing. Amos
Re: Fwd: [squid-users] store-id.pl doesnt cache youtube
On 9/07/2013 1:55 a.m., Amos Jeffries wrote: On 8/07/2013 6:34 p.m., Alan wrote: I just came up with this simple script, based on yours. I haven't tested it though, since 3.4 head segfaults for me (btw, which revision are you using?). Now fixed. Thank you. Amos
