Jenkins build is back to normal : website-builds #949
See http://build.squid-cache.org/job/website-builds/949/
Re: [RFC] or ACLs
On 09/25/2012 09:02 PM, Amos Jeffries wrote: So, if we change the name to any/one-of/first-of/etc or use the is/= syntax above, will you be OK with adding OR ACLs? Does 'is' mean OR or AND or IF or equals ? is means what it means in English: equality or definition. Does '=' means OR or AND or assignment ? = means what it means in programming: equality or assignment. The expression on the right hand side determines what is being assigned. Since neither of us liked or acl1 acl2 style, I proposed is acl1 or acl2 style because it is natural and will allow us to support more complex expressions later. I now understand that you do not like that direction, so I will use one-of you suggested unless others help form a different consensus. Please consider names that provide you with easily distinguishable set of types that still match the underlying semantics. one-of/all-of at least hint at the OR/AND set semantics. I will use your one-of/all-of names. To summarise: Yes I'm okay with adding OR type. Provided the larger picture is considered when adding them. You may as well add the AND type as well, since they only differ in match() strategy. Then you have grounds for adding a Conditional.h/cc to src/acl which defines these and any future boolean node types. I am glad AND/OR ACLs will be allowed. It is unfortunate that our views on what the ideal Squid configuration language should provide (and how to get to that ideal) differ so much. I focus on maximizing flexibility and expressiveness of the language while you focus on minimizing misuse and abuse. I cannot think of any real-world example where humanity succeeded optimizing in _both_ directions. While both expressiveness and safety are good principles and usually co-exist, one principle has to dominate for the design to flourish. Now that the solution to an immediate practical problem has been agreed on, I do not think it makes sense to continue this discussion between the two of us. Hopefully, others will chime in and help form a consensus, but I am not optimistic. Thank you, Alex.
Re: [RFC] or ACLs
On 9/26/2012 5:07 PM, Alex Rousskov wrote: I am glad AND/OR ACLs will be allowed. It is unfortunate that our views on what the ideal Squid configuration language should provide (and how to get to that ideal) differ so much. I focus on maximizing flexibility and expressiveness of the language while you focus on minimizing misuse and abuse. I cannot think of any real-world example where humanity succeeded optimizing in_both_ directions. While both expressiveness and safety are good principles and usually co-exist, one principle has to dominate for the design to flourish. Now that the solution to an immediate practical problem has been agreed on, I do not think it makes sense to continue this discussion between the two of us. Hopefully, others will chime in and help form a consensus, but I am not optimistic. Thank you, Alex. I was thinking and what AND/OR different from the ACLs as it now? today it's a strict AND for the whole line so in a sense of calculator it's: http_access allow AND acl1 acl2 acl4 (+1,2,3,4) which is the better choice if you do ask me for acl validation instead of using: http_access allow AND acl1 AND acl2 AND acl4 (1+2+3+4) (I dont remember the literal way of the different ways to this calculation thing) If you do ask me one of the best things I do like about squid configuration is not having a programming language like structure. Ff you will take for example varnish configuration. you will need to learn how it all fits together to even write the basic configurations while with squid you can simply write: one acl for src allow one cache_peer directive + allow\deny acl one http_acces \ https_access and refresh_pattern if needed at all So I think that adding the feature OR like this: http_access allow acl1 acl2 acl4 OR acl5 acl6 http_access allow OR acl7 acl8 when the OR is only on acls after the OR will be accounted as OR can be understandable and will not effect any current configurations. Flexibility is a good thing but IN steps. What I mean is that the current configuration syntax is based on static statements which can be understood in seconds for almost anyone who just read it. The way system administrators looks at the product can be seen from couple angles like: while they need the product. while they need the product to do something specific. while they have a product the was used before. and maybe some other points of view of course. While they prefer the software to have all they need to do a specific task they would admit that in other points encountering the product they prefer simplicity. I would ask just to now if i'm wrong or not: all this big list of http_access deny w1 r2 http_access deny w2 r2 http_access deny w3 r2 http_access deny w4 r2 http_access deny w5 r2 can be replaced with one external_acl helper? (since I do not know what w1 and r1 acls are). Eliezer -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations eliezer at ngtech.co.il
Re: [RFC] or ACLs
On 27.09.2012 03:18, Kinkie wrote:
Here's my cue :-)
I tend to favor expressiveness, and I'd love to see the access rules
evolve
to a tree-like structure, with sub expressions and explicit Boolean
operators.
But I also think that the one-of / all-of proposal is clear and is
more
expressive than what we have now, so I support it.
On Wednesday, September 26, 2012, Alex Rousskov wrote:
On 09/25/2012 09:02 PM, Amos Jeffries wrote:
So, if we change the name to any/one-of/first-of/etc or use the
is/=
syntax above, will you be OK with adding OR ACLs?
Does 'is' mean OR or AND or IF or equals ?
is means what it means in English: equality or definition.
Does '=' means OR or AND or assignment ?
= means what it means in programming: equality or assignment.
The expression on the right hand side determines what is being
assigned.
Since neither of us liked or acl1 acl2 style, I proposed is acl1
or
acl2 style because it is natural and will allow us to support more
complex expressions later. I now understand that you do not like
that
direction, so I will use one-of you suggested unless others help
form
a different consensus.
Please consider names that provide you with easily distinguishable
set
of types that still match the underlying semantics.
one-of/all-of at
least hint at the OR/AND set semantics.
I will use your one-of/all-of names.
To summarise: Yes I'm okay with adding OR type. Provided the
larger
picture is considered when adding them.
You may as well add the AND type as well, since they only differ
in
match() strategy. Then you have grounds for adding a
Conditional.h/cc to
src/acl which defines these and any future boolean node types.
I am glad AND/OR ACLs will be allowed.
It is unfortunate that our views on what the ideal Squid
configuration
language should provide (and how to get to that ideal) differ so
much. I
focus on maximizing flexibility and expressiveness of the language
while
you focus on minimizing misuse and abuse. I cannot think of any
real-world example where humanity succeeded optimizing in _both_
directions. While both expressiveness and safety are good principles
and
usually co-exist, one principle has to dominate for the design to
flourish.
On the contrary. The safe route I would I would very much like to see
is one day to have the very flexible and expressive syntax:
acl label = (condition)
Where condition contains at least 'or', 'and', '(', ')', '!' operators
to construct a true boolean tree structure for the ACL test. That syntax
has much wider understanding than our existing definition structure and
will cause far less confusion overall.
If you want this project to jump straight to that for 3.4 I have no
problem with the naming. It is only for this half-stage where its almost
there but missing vital bracket/scoping operators that I am concerned
about understanding and migration problems.
IMHO its not that much work to add a Conditional data type which hold
ACL node pointers instead of data values to test against. With a
strategy for each operator type. The parser would need to be
semi-recursive like any boolean parser - but that is not a big problem.
HTH
Amos
some notes and help to think is needed + Test results of testing basics in store_url_rewrite.
Well I was reading here and there the store code and other things then started testing some theories about how store_url can be and should be implemented. If you do remember or not I refactored the source in couple places to use originalUrl and original_url to check all the places where original url is used and then decide where the store_url should be placed if needed. (memobject-original_url and has memobject-store_url) the main place of all I have seen that should be used with the store_url is the http://bazaar.launchpad.net/~squid/squid/3-trunk/view/head:/src/store_key_md5.cc#L144 which gets requests from couple places and mainly the setPublic for store entry (Does bazaar has search option in it ?) so I changed the code to use the store_url in a case it exists in the memobject instead of the urlCanonical(request). The result is good since it makes the predicted md5 hash for the url based on the store_url and not the original_url. At this point I started looking up for more places that original_url might should be replaced since maybe there is a place I missed like when storing or fetching the file for any reason there is that I cant see and think of. I placed a test file on my server called v.flv which is a static text file. I used a store_url helper that will use the arguments id itag and range (something I know and have script for) to rewrite the url. from 'http://www1.ngtech.co.il/v.flv?id=1000itag=34range=0-100icu=0tag1' into 'http://ngtech.squid.internal/id=1000itag=34range=0-100' which I later will change the id itag and range on the request and check what happens in my extended logs. so as expected: 2012/09/27 04:34:17 kid1| A candidate recived from Store URL rewriter accepted: http://ngtech.squid.internal/id=1000itag=34range=0-100 2012/09/27 04:34:17.157 kid1| storeKeypublicreuest could be with URL and method: GET http://ngtech.squid.internal/id=1000itag=34range=0-100 2012/09/27 04:34:17.157 kid1| storeKeypublicreuest with URL and method: GET http://ngtech.squid.internal/id=1000itag=34range=0-100 2012/09/27 04:34:17.157 kid1| storeKeypublicreuest was created using method and url from request 2012/09/27 04:34:17.157 kid1| storeGet: looking up 4F2F79AE5171037CBE02EA95ECD5B86C the ^^^ hexdigest is what expected to be looked for. instead of looking up for: 7F77FC394B370386D2E6AD9F8B748846 its looking for: 4F2F79AE5171037CBE02EA95ECD5B86C which is a very good progress and the file was cached based on the store_url hash. I started seeing somethings with large files so I reduced the file size to 702 bytes and then I will be able to look over things without too much stuff running because of the size. a new session new file and other url which result a good 304 validation and the web server logs shows 200 for the first and 304 for the second. ##access.log 1348715219.641205 127.0.0.1 TCP_MISS/200 1088 GET http://www1.ngtech.co.il/v.flv?id=100itag=34ii=1 - HIER_DIRECT/79.181.248.214 video/x-flv 1348715278.783 11 127.0.0.1 TCP_REFRESH_UNMODIFIED/200 1096 GET http://www1.ngtech.co.il/v.flv?id=100itag=34ii=1 - HIER_DIRECT/79.181.248.214 video/x-flv (you can skip this log or jump over what you know) ##cache.log A candidate recived from Store URL rewriter accepted: http://ngtech.squid.internal/id=100itag=34 storeKeypublicreuest could be with URL and method: GET http://ngtech.squid.internal/id=100itag=34 storeKeypublicreuest with URL and method: GET http://ngtech.squid.internal/id=100itag=34 storeKeypublicreuest was created using method and url from request storeGet: looking up A5E995AF63D94AC8E7B40ACA255948F1 src/store_dir.cc(748) get: got in-transit entry: -1@-1=1/2/0/0 StoreEntry::lock: key 'A5E995AF63D94AC8E7B40ACA255948F1' count=1 storeCreateEntry: 'http://www1.ngtech.co.il/v.flv?id=100itag=34ii=1' src/store.cc(395) StoreEntry: new StoreEntry 0x247c080 src/MemObject.cc(109) MemObject: new MemObject 0x2788050 storeKeyPrivate: GET http://www1.ngtech.co.il/v.flv?id=100itag=34ii=1 storeKeyPrivate was created using method and url is: StoreEntry::hashInsert: Inserting Entry 0x247c080 key '56E7D8D797D8F666B16138EF1686E5B7' StoreEntry::lock: key '56E7D8D797D8F666B16138EF1686E5B7' count=2 StoreEntry::lock: key '56E7D8D797D8F666B16138EF1686E5B7' count=3 src/store.cc(1677) originalUrl: A storeEntry::origianlUrl being used here using URL:http://www1.ngtech.co.il/v.flv?id=100itag=34ii=1 src/store.cc(1678) originalUrl: A storeEntry::origianlUrl Could be used with URL:http://ngtech.squid.internal/id=100itag=34 StoreEntry::lock: key '56E7D8D797D8F666B16138EF1686E5B7' count=4 StoreEntry::unlock: key '56E7D8D797D8F666B16138EF1686E5B7' count=3 src/store.cc(1677) originalUrl: A storeEntry::origianlUrl being used here using URL:http://www1.ngtech.co.il/v.flv?id=100itag=34ii=1 src/store.cc(1678) originalUrl: A storeEntry::origianlUrl Could be used with URL:http://ngtech.squid.internal/id=100itag=34 src/store.cc(1677) originalUrl: A storeEntry::origianlUrl
Re: some notes and help to think is needed + Test results of testing basics in store_url_rewrite.
On Thu, Sep 27, 2012 at 4:09 PM, Eliezer Croitoru elie...@ngtech.co.il wrote: Well I was reading here and there the store code and other things then started testing some theories about how store_url can be and should be implemented. If you do remember or not I refactored the source in couple places to use originalUrl and original_url to check all the places where original url is used and then decide where the store_url should be placed if needed. (memobject-original_url and has memobject-store_url) the main place of all I have seen that should be used with the store_url is the http://bazaar.launchpad.net/~squid/squid/3-trunk/view/head:/src/store_key_md5.cc#L144 which gets requests from couple places and mainly the setPublic for store entry (Does bazaar has search option in it ?) bzr grep (if you install the bzr-grep plugin) will search all your source code. bzr search (if you install the bzr-search plugin) finds references anywhere in the project history. -Rob
Re: some notes and help to think is needed + Test results of testing basics in store_url_rewrite.
On 27/09/2012 4:09 p.m., Eliezer Croitoru wrote: Well I was reading here and there the store code and other things then started testing some theories about how store_url can be and should be implemented. If you do remember or not I refactored the source in couple places to use originalUrl and original_url to check all the places where original url is used and then decide where the store_url should be placed if needed. (memobject-original_url and has memobject-store_url) the main place of all I have seen that should be used with the store_url is the http://bazaar.launchpad.net/~squid/squid/3-trunk/view/head:/src/store_key_md5.cc#L144 which gets requests from couple places and mainly the setPublic for store entry (Does bazaar has search option in it ?) Doxygen provides all the references/referenced-by information. http://www.squid-cache.org/Doc/code/store__key__md5_8cc.html#a5461ad8f7a969c8bec3008a70a74c42a so I changed the code to use the store_url in a case it exists in the memobject instead of the urlCanonical(request). The result is good since it makes the predicted md5 hash for the url based on the store_url and not the original_url. At this point I started looking up for more places that original_url might should be replaced since maybe there is a place I missed like when storing or fetching the file for any reason there is that I cant see and think of. I placed a test file on my server called v.flv which is a static text file. I used a store_url helper that will use the arguments id itag and range (something I know and have script for) to rewrite the url. from 'http://www1.ngtech.co.il/v.flv?id=1000itag=34range=0-100icu=0tag1' into 'http://ngtech.squid.internal/id=1000itag=34range=0-100' which I later will change the id itag and range on the request and check what happens in my extended logs. so as expected: 2012/09/27 04:34:17 kid1| A candidate recived from Store URL rewriter accepted: http://ngtech.squid.internal/id=1000itag=34range=0-100 2012/09/27 04:34:17.157 kid1| storeKeypublicreuest could be with URL and method: GET http://ngtech.squid.internal/id=1000itag=34range=0-100 2012/09/27 04:34:17.157 kid1| storeKeypublicreuest with URL and method: GET http://ngtech.squid.internal/id=1000itag=34range=0-100 2012/09/27 04:34:17.157 kid1| storeKeypublicreuest was created using method and url from request 2012/09/27 04:34:17.157 kid1| storeGet: looking up 4F2F79AE5171037CBE02EA95ECD5B86C the ^^^ hexdigest is what expected to be looked for. instead of looking up for: 7F77FC394B370386D2E6AD9F8B748846 its looking for: 4F2F79AE5171037CBE02EA95ECD5B86C which is a very good progress and the file was cached based on the store_url hash. I started seeing somethings with large files so I reduced the file size to 702 bytes and then I will be able to look over things without too much stuff running because of the size. a new session new file and other url which result a good 304 validation and the web server logs shows 200 for the first and 304 for the second. ##access.log 1348715219.641205 127.0.0.1 TCP_MISS/200 1088 GET http://www1.ngtech.co.il/v.flv?id=100itag=34ii=1 - HIER_DIRECT/79.181.248.214 video/x-flv 1348715278.783 11 127.0.0.1 TCP_REFRESH_UNMODIFIED/200 1096 GET http://www1.ngtech.co.il/v.flv?id=100itag=34ii=1 - HIER_DIRECT/79.181.248.214 video/x-flv (you can skip this log or jump over what you know) ##cache.log A candidate recived from Store URL rewriter accepted: http://ngtech.squid.internal/id=100itag=34 storeKeypublicreuest could be with URL and method: GET http://ngtech.squid.internal/id=100itag=34 storeKeypublicreuest with URL and method: GET http://ngtech.squid.internal/id=100itag=34 storeKeypublicreuest was created using method and url from request storeGet: looking up A5E995AF63D94AC8E7B40ACA255948F1 src/store_dir.cc(748) get: got in-transit entry: -1@-1=1/2/0/0 StoreEntry::lock: key 'A5E995AF63D94AC8E7B40ACA255948F1' count=1 storeCreateEntry: 'http://www1.ngtech.co.il/v.flv?id=100itag=34ii=1' src/store.cc(395) StoreEntry: new StoreEntry 0x247c080 src/MemObject.cc(109) MemObject: new MemObject 0x2788050 storeKeyPrivate: GET http://www1.ngtech.co.il/v.flv?id=100itag=34ii=1 storeKeyPrivate was created using method and url is: StoreEntry::hashInsert: Inserting Entry 0x247c080 key '56E7D8D797D8F666B16138EF1686E5B7' StoreEntry::lock: key '56E7D8D797D8F666B16138EF1686E5B7' count=2 StoreEntry::lock: key '56E7D8D797D8F666B16138EF1686E5B7' count=3 src/store.cc(1677) originalUrl: A storeEntry::origianlUrl being used here using URL:http://www1.ngtech.co.il/v.flv?id=100itag=34ii=1 src/store.cc(1678) originalUrl: A storeEntry::origianlUrl Could be used with URL:http://ngtech.squid.internal/id=100itag=34 StoreEntry::lock: key '56E7D8D797D8F666B16138EF1686E5B7' count=4 StoreEntry::unlock: key '56E7D8D797D8F666B16138EF1686E5B7' count=3 src/store.cc(1677) originalUrl: A storeEntry::origianlUrl being used here using
