Re: [squid-dev] Drop cache_object protocol support

[Eduard Bagdasaryan]

On 25.01.2023 16:58, Eduard Bagdasaryan wrote:

On 25.01.2023 15:29, Amos Jeffries wrote:

On 25/01/2023 5:34 pm, Alex Rousskov wrote:

On 1/24/23 20:57, Amos Jeffries wrote:

Blocker #1: The cachemgr_passwd directly still needs to be cleanly 
removed, eg replaced by a manager_access ACL based mechanism.


I do not see a relationship: I have not tested it, but the existing 
CacheManager::ParseHeaders() code already extracts authentication 
information from cache manager requests that use "http" scheme 
AFAICT. Can you detail why the cachemgr_passwd directive/code cannot 
continue to work essentially as it works today after cache_object 
scheme support is removed from Squid?


We should check that then. It may not be as impactful as I am recalling. 



My test showed that CacheManager::ParseHeaders() extracts password 
from the Authorization header incorrectly: params.password gets an 
extra '\n' symbol and CacheManager::CheckPassword() fails. This, 
however, seems easy to fix.




My previous test was incorrect - please ignore this comment. I retested 
it with a properly encoded base64 password now.



Eduard.
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] Drop cache_object protocol support

[Eduard Bagdasaryan]

On 25.01.2023 15:29, Amos Jeffries wrote:

On 25/01/2023 5:34 pm, Alex Rousskov wrote:

On 1/24/23 20:57, Amos Jeffries wrote:

Blocker #1:  The cachemgr_passwd directly still needs to be cleanly 
removed, eg replaced by a manager_access ACL based mechanism.


I do not see a relationship: I have not tested it, but the existing 
CacheManager::ParseHeaders() code already extracts authentication 
information from cache manager requests that use "http" scheme 
AFAICT. Can you detail why the cachemgr_passwd directive/code cannot 
continue to work essentially as it works today after cache_object 
scheme support is removed from Squid?


We should check that then. It may not be as impactful as I am recalling. 



My test showed that CacheManager::ParseHeaders() extracts password from 
the Authorization header incorrectly: params.password gets an extra '\n' 
symbol and CacheManager::CheckPassword() fails. This, however, seems 
easy to fix.



Eduard.
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] Drop cache_object protocol support

[Eduard Bagdasaryan]

Hello,

Today we can query cache manager in two ways:

1. with cache_object:// URL scheme
2. with an HTTP request having the 'squid-internal-mgr' path prefix.

I guess that when (2) was initially added at e37bd29, its implementation 
was somewhat incomplete compared to the old cache_object scheme (e.g., 
it lacked authentication) and both methods existed. Since then, however, 
(2) has been improved and it should be equivalent to (1) by now.  If so, 
can we completely remove the non-standard cache_object scheme support 
from Squid? This would simplify request forwarding logic, including code 
paths where the existing code complexity may result in vulnerability issues.



Thanks,

Eduard.

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] RFC: Adding a new line to a regex

[Eduard Bagdasaryan]

I would concur with Alex that (4) is preferable: It does not break old 
configurations, re-uses existing mechanisms and allows to apply it only 
when/where required. I have one more option for your consideration: 
escaping with a backtick (e.g., `n) instead of a backslash. This 
approach is used, e.g., in PowerShell.


5a. Recognize just `n escape sequence in squid.conf regexes.

5b. Recognize all '`'-based escape sequences in squid.conf regexes.

Pros:  Easier upgrade: backtick is rare in regular expressions (compared 
to '%' or '/'), probably there is no need to convert old regexes at all.

Pros:  Simplicity: no double-escaping is required (as in (1b)).
Cons: Though it should be straightforward to specify common escape 
sequences, such as `n, `r or `t, we still need to devise a way of 
providing arbitrary character (i.e., its code) in this way.



HTH,

Eduard.


On 20.01.2022 00:32, Alex Rousskov wrote:

Here is a fairly representative sample:

1a. Recognize just \n escape sequence in squid.conf regexes
Pros: Simple.
Cons: Converting old regexes[1] requires careful checking[2].
Cons: Cannot detect typos in escape sequences. \r is accepted.
Cons: Cannot address other, similar use cases (e.g., ASCII CR).

1b. Recognize all C escape sequences in squid.conf regexes
Pros: Can detect typos -- unsupported escape sequences.
Cons: Poor readability: Double-escaping of all for-regex backslashes!
Cons: Converting old regexes requires non-trivial automation.


2a. Recognize %byte{n} logformat-like sequence in squid.conf regexes
Pros: Simple.
Cons: Converting old regexes[1] requires careful checking[3].
Cons: Cannot detect typos in logformat-like sequences.
Cons: Does not support other advanced use cases (e.g., %tr).

2b. Recognize %byte{n} and logformat sequences in squid.conf regexes
Pros: Can detect typos -- unsupported logformat sequences.
Cons: The need to escape % in regexes will surprise admins.
Cons: Converting old regexes requires (simple) automation.


3. Use composition to combine regexes and some special strings:
regex1 + "\n" + regex2
or
regex1 + %byte{10} + regex2
Pros: Old regexes can be safely used without any conversions.
Cons: Requires new, complex composition expressions/syntax.
Cons: A bit difficult to read.
Cons: Requires a lot of development.


4. Use 2b but only when regex is given to a special function:
substitute_logformat_codes(regex)
Pros: Old regexes can be safely used without any conversions.
Pros: New regexes do not need to escape % (by default).
Pros: Extendable to old regex configuration contexts.
Pros: Extendable to non-regex configuration contexts.
Pros: Reusing the existing parameters(...)-like call syntax.
Cons: A bit more difficult to read than 1a or 2a.
Cons: Duplicates "quoted string" approach in some directives[4].
Cons: Requires arguing about the new function name:-).


Given all the pros and cons, I think we should use option 4 above.

Do you see any better options?

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] Incremental parsing of chunked quoted extensions

[Eduard Bagdasaryan]

Since there have not been any objections so far, I am going
to start implementing the incremental parsing approach,
outlined here.


Eduard.

On 05.10.2018 18:34, Alex Rousskov wrote:

I doubt that writing code to explicitly buffer the whole line before
parsing extensions is necessary. It is definitely not desirable because
it slows things down in_common_  cases. I would adjust the code to
follow this sketch instead:

 while (!atEof()) {
 if (skipLineTerminator())
 return true; // reached the end of extensions (if any)
 if (parseChunkExtension())
 continue; // got one extension; there may be more
 if (tok.skipAll(NotLF) && skip(LF))
 throw "cannot parse chunk extension"; //  CR*LF
 return false; // need more data to finish parsing extension
 }
 return false; // need data to start parsing extension or CRLF

where skipLineTerminator() is, essentially,

 return tok.skip(CRLF)) || (relaxed_parser && tok.skipOne(LF));


As you can see, the above sketch is optimized for the common cases and
blindly seeks to the end of the line only in rare cases of partially
received or malformed extensions.

Please note that whether the above sketch is used incrementally is an
_orthogonal_  question:

* In incremental parsing, a successful parseChunkExtension() would
update object state to remember the parsed extension and advance the
parsing position to never come back to the same spot. The loop caller
would advance the parsing position/state if (and only if) the loop
returns true.

* In non-incremental parsing, a successful parseChunkExtension() would
update its parameter (not shown) to remember the parsed extension. The
loop caller would update object state ("committing" that not shown
parameter info) and advance the parsing position if (and only if) the
loop returns true. [ Alternatively, a successful parseChunkExtension()
would update object state directly, sometimes overwriting the previous
(identical) value many times. The current useOriginBody code uses that
simpler scheme. However, even with this simpler alternative, the
advancement is controlled by the loop caller. ]

Should we fix and continue to use incremental parsing here or abandon
it? The answer probably depends on overall code simplicity. Since we
have to use incremental parsing for parsing chunked encoding as a whole
(as discussed in the beginning of this email), we may continue to use it
for parsing extensions if (and only if) doing so does not complicate things.


___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] Incremental parsing of chunked quoted extensions

[Eduard Bagdasaryan]

Hello all,

There is a bug in Squid during incremental parsing of quoted chunked
extensions, resulting in unexpected throwing in
One::Parser::skipLineTerminator(). The underlying problem comes from
the fact that Http::One::Tokenizer::qdText() cannot parse
incrementally because it cannot distinguish "an invalid input" from
"need more data" case.

I see two approaches for fixing this problem:

* Low-level.
  Supply Parser::Tokenizer and its children with incremental parsing
  functionality. It looks like this would require much non-trivial work.

* High-level.
  Avoid incremental parsing of chunked extensions and parse them all
  at once. In other words, we need to adjust the Tokenizer caller
  to buffer the whole chunk extension line before passing it to the
  Tokenizer in order to avoid "need more data" case, causing that bug.

Is the incremental parsing of chunked extensions all-important here?
If no, can I proceed with (2) approach fixing this bug?


Thanks,
Eduard.
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] Use MAX_URL for first line limitation

[Eduard Bagdasaryan]

Yes, %>ru is consistent for small and large URLs.

FYI: this problem existed in the past for %>ru, but was fixed at master
r11274. I have a feeling that the way how setLogUri() fixes malformed
requests was not the best choice (in fact, there is a code duplication
between setLogUri() and urlParse(), and, moreover, I suspect that
encoding methods like rfc1738_escape_unescaped() may be applied
several times with destructive consequences).

However, probably in this small step we could leave those problems
aside and just unify %>ru and %ru 'cleanup' procedure, as suggested.
I see a simple fix: in One::Server::buildHttpRequest() initialize
our new ALE::virginUrlForMissingRequest with http->log_uri just
after setLogUri() (i.e., when the URL was clean-upped already).


Eduard.


On 08.06.2018 11:46, Amos Jeffries wrote:

On 08/06/18 11:18, Alex Rousskov wrote:

On 06/07/2018 04:13 PM, Eduard Bagdasaryan wrote:


in %>ru Squid logs large and small URLs differently.  For example,
Squid strips whitespaces from small URLs, while keeping them for
large ones.

Is %ru logging consistent with regard to small and large URLs?

* If it is, should we use the same approach to make %>ru consistent?


For a quick fix yes.

Longer term; all Squid code using char* buffers and MAX_URL for URL
storage is obsolete and needs fixing to use class URL (or an SBuf) instead.

Amos
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev


___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Purge cache entries in SMP-aware caches

[Eduard Bagdasaryan]

On 23.07.2017 02:04, Alex Rousskov wrote:

+if (!flags.cachable)
+EBIT_SET(e->flags, RELEASE_REQUEST);

This release request feels out of place and direct flags setting goes
around the existing releaseRequest() API. Please check all callers --
perhaps we do not need the above because all callers already do an
equivalent action (e.g., makePrivate()) for "uncachable" requests?


I don't think this lines are 'out of place': storeCreatePureEntry() just
initializes the just created StoreEntry fields (including
StoreEntry::flags) with correct values.  If we definitely know a
this moment that 'flags' should have RELEASE_REQUEST set, why do we need
to postpone this to many callers, hoping that all of them will do that
work correctly?  There are lots of storeCreateEntry() calls and it is
hardly possible to track that all of them end up with
'releaseRequest()', when flags.cachable is false.  BTW, at the time of
StoreEntry initialization we do not need to do most of the work
releaseRequest() does. E.g., there are no connected storages to
disconnect from, no public keys to make them private, etc. The only
thing to do is RELEASE_REQUEST flag setting.


+SWAPOUT_NONE, ///< the store entry has not been stored on a disk

This definition seems to contradict the "else" usage in
Rock::SwapDir::disconnect(). What does SWAPOUT_NONE state correspond to
in that "else" clause? A readable disk entry? It may be tempting to use
SWAPOUT_DONE in that "else" clause inside disconnect() but I suspect
that another worker may still be writing that disk entry (i.e., there is
a SWAPOUT_WITING in another worker). We may need to either

* broaden SWAPOUT_NONE definition to cover that "else" clause or
* call anchor.complete() and use SWAPOUT_DONE/SWAPOUT_WITING in that
"else" clause, similar to what Rock::SwapDir::anchorEntry() does.

Please investigate and suggest any necessary changes.


This patch introduces an invariant StoreEntry::checkDisk() for disk
fields. According to this rule, a disconnected/unlinked entry
(swap_dirn < 0) must have SWAPOUT_NONE.  So I assume we should correct
SWAPOUT_NONE definition, e.g.,:

/// a store entry which is either unlinked the disk Store or
/// has not been stored on a disk.


Thanks,

Eduard.

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] [PATCH] Retries of failed re-forwardable transactions

[Eduard Bagdasaryan]

Hello,


This is a bug 4223 fix.

This change fixes Store to delay writing (and, therefore, sharing) of
re-triable errors (e.g., 504 Gateway Timeout responses) to clients:
once we start sharing the response with a client, we cannot re-try the
transaction. Since ENTRY_FWD_HDR_WAIT flag purpose is to delay response
sharing with Store clients, this patch fixes and clarifies its usage:

1. Removes unconditional clearing from startWriting().
2. Adds a conditional clearing to StoreEntry::write().
3. Sets it only for may-be-rejected responses.

(2) adds ENTRY_FWD_HDR_WAIT clearing to detect responses that filled the
entire read buffer and must be shared now with the clients because
they can no longer be retried with the current re-forwarding mechanisms
(which rely on completing the bad response transaction first) and will
get stuck. Such large re-triable error responses (>32KB with default
read_ahead_gap) should be uncommon. They were getting stuck prior to
trunk r12501.1.48. That revision started clearing ENTRY_FWD_HDR_WAIT in
StoreEntry startWriting() unconditionally, allowing all errors to be
sent to Store clients without a delay, and effectively disabling
retries.

Mgr::Forwarder was always setting ENTRY_FWD_HDR_WAIT, probably mimicking
similarly aggressive FwdState behavior that we are now removing. Since
the forwarder never rewrites the entry content, it should not need to
set that flag. The forwarder and associated Server classes must not
communicate with the mgr client during the client-Squid connection
descriptor handoff to Coordinator, but ENTRY_FWD_HDR_WAIT is not the
right mechanism to block such Squid-client communication. The flag does
not work well for this purpose because those Server classes may (and
do?) substitute the "blocked" StoreEntry with another one to
write an error message to the client.

Also moved ENTRY_FWD_HDR_WAIT clearance from many StoreEntry::complete()
callers to that method itself. StoreEntry::complete() is meant to be the
last call when forming the entry. Any post-complete entry modifications
such as retries are prohibited.


Regards,

Eduard.

Bug 4223: Fixed retries of failed re-forwardable transactions.

This change fixes Store to delay writing (and, therefore, sharing) of
re-triable errors (e.g., 504 Gateway Timeout responses) to clients:
once we start sharing the response with a client, we cannot re-try the
transaction. Since ENTRY_FWD_HDR_WAIT flag purpose is to delay response
sharing with Store clients, this patch fixes and clarifies its usage:

1. Removes unconditional clearing from startWriting().
2. Adds a conditional clearing to StoreEntry::write().
3. Sets it only for may-be-rejected responses.

(2) adds ENTRY_FWD_HDR_WAIT clearing to detect responses that filled the
entire read buffer and must be shared now with the clients because
they can no longer be retried with the current re-forwarding mechanisms
(which rely on completing the bad response transaction first) and will
get stuck. Such large re-triable error responses (>32KB with default
read_ahead_gap) should be uncommon. They were getting stuck prior to
trunk r12501.1.48. That revision started clearing ENTRY_FWD_HDR_WAIT in
StoreEntry startWriting() unconditionally, allowing all errors to be
sent to Store clients without a delay, and effectively disabling
retries.

Mgr::Forwarder was always setting ENTRY_FWD_HDR_WAIT, probably mimicking
similarly aggressive FwdState behavior that we are now removing. Since
the forwarder never rewrites the entry content, it should not need to
set that flag. The forwarder and associated Server classes must not
communicate with the mgr client during the client-Squid connection
descriptor handoff to Coordinator, but ENTRY_FWD_HDR_WAIT is not the
right mechanism to block such Squid-client communication. The flag does
not work well for this purpose because those Server classes may (and
do?) substitute the "blocked" StoreEntry with another one to
write an error message to the client.

Also moved ENTRY_FWD_HDR_WAIT clearance from many StoreEntry::complete()
callers to that method itself. StoreEntry::complete() is meant to be the
last call when forming the entry. Any post-complete entry modifications
such as retries are prohibited.

=== modified file 'src/FwdState.cc'
--- src/FwdState.cc	2017-06-26 14:34:57 +
+++ src/FwdState.cc	2017-06-28 09:11:30 +
@@ -114,61 +114,60 @@ FwdState::abort(void* d)
 fwd->serverDestinations.clear();
 fwd->stopAndDestroy("store entry aborted");
 }
 
 void
 FwdState::closeServerConnection(const char *reason)
 {
 debugs(17, 3, "because " << reason << "; " << serverConn);
 comm_remove_close_handler(serverConn->fd, closeHandler);
 closeHandler = NULL;
 fwdPconnPool->noteUses(fd_table[serverConn->fd].pconn.uses);
 serverConn->close();
 }
 
 / PUBLIC INTERFACE /
 
 FwdState::FwdState(const Comm::ConnectionPointer , StoreEntry * e, HttpRequest * r, const 

Re: [squid-dev] [PATCH] Fix 'miss_access' and 'cache' checks when no ACL rules matched

[Eduard Bagdasaryan]

Any remarks about latest patch before applying?


Eduard.

On 12.06.2017 14:14, Eduard Bagdasaryan wrote:


On 01.06.2017 17:37, Amos Jeffries wrote:
The admin intention here is almost invariably to prevent certain 
users getting cached data. Preventing cache being used simply for 
lack of credentials is not a good occurance. 


Ok, I am leaving 'cache' and 'miss_access' as is. So the re-attached
patch has no functionality changes, just allowed()/denied() API
improvements.


Eduard.



___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Fix 'miss_access' and 'cache' checks when no ACL rules matched

[Eduard Bagdasaryan]

On 01.06.2017 17:37, Amos Jeffries wrote:
The admin intention here is almost invariably to prevent certain users 
getting cached data. Preventing cache being used simply for lack of 
credentials is not a good occurance. 


Ok, I am leaving 'cache' and 'miss_access' as is. So the re-attached
patch has no functionality changes, just allowed()/denied() API
improvements.


Eduard.
Do not use direct comparisons with ACCESS_DENIED and ACCESS_ALLOWED.

This patch introduces new allow_t API, substituting direct checks
against ACCESS_ALLOWED and ACCESS_DENIED and helping to better
distinguish between them. No functionality changes expected.

Since most of existing ACL directives check against ACCESS_ALLOWED, the
initial intention was to unify these checks, removing checking against
ACCESS_DENIED for "miss_access" and "cache" directives. However, this
idea was rejected, because different Squid installations could suddenly
stop caching or start receiving 403s. For details see squid-dev
discussion:

http://lists.squid-cache.org/pipermail/squid-dev/2017-May/008576.html

=== modified file 'src/DelayId.cc'
--- src/DelayId.cc	2017-01-01 00:12:22 +
+++ src/DelayId.cc	2017-06-11 09:33:23 +
@@ -74,61 +74,61 @@ DelayId::DelayClient(ClientHttpRequest *
 if (r->client_addr.isNoAddr()) {
 debugs(77, 2, "delayClient: WARNING: Called with 'NO_ADDR' address, ignoring");
 return DelayId();
 }
 
 for (pool = 0; pool < DelayPools::pools(); ++pool) {
 
 /* pools require explicit 'allow' to assign a client into them */
 if (!DelayPools::delay_data[pool].access) {
 debugs(77, DBG_IMPORTANT, "delay_pool " << pool <<
" has no delay_access configured. This means that no clients will ever use it.");
 continue;
 }
 
 ACLFilledChecklist ch(DelayPools::delay_data[pool].access, r, NULL);
 if (reply) {
 ch.reply = reply;
 HTTPMSGLOCK(reply);
 }
 #if FOLLOW_X_FORWARDED_FOR
 if (Config.onoff.delay_pool_uses_indirect_client)
 ch.src_addr = r->indirect_client_addr;
 else
 #endif /* FOLLOW_X_FORWARDED_FOR */
 ch.src_addr = r->client_addr;
 ch.my_addr = r->my_addr;
 
 if (http->getConn() != NULL)
 ch.conn(http->getConn());
 
-if (DelayPools::delay_data[pool].theComposite().getRaw() && ch.fastCheck() == ACCESS_ALLOWED) {
+if (DelayPools::delay_data[pool].theComposite().getRaw() && ch.fastCheck().allowed()) {
 
 DelayId result (pool + 1);
 CompositePoolNode::CompositeSelectionDetails details;
 details.src_addr = ch.src_addr;
 #if USE_AUTH
 details.user = r->auth_user_request;
 #endif
 details.tag = r->tag;
 result.compositePosition(DelayPools::delay_data[pool].theComposite()->id(details));
 return result;
 }
 }
 
 return DelayId();
 }
 
 void
 DelayId::setNoDelay(bool const newValue)
 {
 markedAsNoDelay = newValue;
 }
 
 /*
  * this returns the number of bytes the client is permitted. it does not take
  * into account bytes already buffered - that is up to the caller.
  */
 int
 DelayId::bytesWanted(int minimum, int maximum) const
 {
 /* unlimited */

=== modified file 'src/FwdState.cc'
--- src/FwdState.cc	2017-06-09 04:38:40 +
+++ src/FwdState.cc	2017-06-12 10:53:49 +
@@ -305,61 +305,61 @@ FwdState::~FwdState()
 closeServerConnection("~FwdState");
 
 serverDestinations.clear();
 
 debugs(17, 3, "FwdState destructed, this=" << this);
 }
 
 /**
  * This is the entry point for client-side to start forwarding
  * a transaction.  It is a static method that may or may not
  * allocate a FwdState.
  */
 void
 FwdState::Start(const Comm::ConnectionPointer , StoreEntry *entry, HttpRequest *request, const AccessLogEntryPointer )
 {
 /** \note
  * client_addr == no_addr indicates this is an "internal" request
  * from peer_digest.c, asn.c, netdb.c, etc and should always
  * be allowed.  yuck, I know.
  */
 
 if ( Config.accessList.miss && !request->client_addr.isNoAddr() &&
 !request->flags.internal && request->url.getScheme() != AnyP::PROTO_CACHE_OBJECT) {
 /**
  * Check if this host is allowed to fetch MISSES from us (miss_access).
  * Intentionally replace the src_addr automatically selected by the checklist code
  * we do NOT want the indirect client address to be tested here.
  */
 ACLFilledChecklist ch(Config.accessList.miss, request, NULL);
 ch.src_addr = request->client_addr;
-if (ch.fastCheck() == ACCESS_DENIED) {
+if (ch.fastCheck().denied()) {
 err_type page_id;
 page_id = aclGetDenyInfoPage(, AclMatchedName, 1);
 
 if (page_id == ERR_NONE)
 page_id = ERR_FORWARDING_DENIED;
 
 ErrorState *anErr = new ErrorState(page_id, 

Re: [squid-dev] [PATCH] Fix 'miss_access' and 'cache' checks when no ACL rules matched

[Eduard Bagdasaryan]

Since there is no clear consensus on this issue, I assume
we should leave 'miss_access' as is. I am still speculating
what to do with 'cache', because there are unanswered
questions in my previous post. Amos, what do you
think about them?

Eduard.


On 12.05.2017 20:24, Eduard Bagdasaryan wrote:


On 12.05.2017 07:54, Amos Jeffries wrote:
 Also going through the process to deprecate cache directive formally 
might be worth beginning - most old uses should be a straight rename 
to store_miss, but some will be send_hit or a mix of the two. Just 
adding a debugs statement to that effect on startup, reconfigure, and 
-k parse should do for a while.



If there are plans about deprecating 'cache' directive, good, but it is a
possibly a future work which is unrelated here.  BTW, I think we can't 
simply
'rename' it to 'store_miss' or a mix of the two directives because 
'cache'

directive supports both fast and slow checks, but both 'store_miss' and
'send_hit' only fast ones.  Probably I miss or do_not_understand 
something

important.


On 12.05.2017 17:32, Amos Jeffries wrote:
That r14984 was itself carefully designed to _revert_ unintentional 
side effects hostVerify had on cache directive behaviour. Your patch 
is reverting those DUNNO occurances back to the code which had many, 
many complaints.


I looked through bug 3940 discussion but have not found any connection 
between

ACCESS_DUNNO/ACCESS_AUTH_REQUIRED/checkNoCacheDone() and
hostHeaderVerify()/hostHeaderVerifyFailed() methods. Yes, all these 
methods

may change RequestFlags flags, and that what
fix of r14984 is about, but how, roughly, hostHeaderVerifyFailed() may 
cause
ACCESS_DUNNO? In other words, I have not found any hints about your 
primary

change ACCESS_ALLOWED into ACCESS_DENIED: no in the patch preamble, no
in the bug discussion. I assume there are another threads/bugs where 
this change
probably was discussed. If so, could you please post such references 
here?



Eduard.
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev


___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] Empty directories during bzr-to-git conversion

[Eduard Bagdasaryan]

Hello,

Working on upcoming bzr-to-git migration we noticed empty directories
presence in bzr history. Therefore, since git does not care about
directories (and creates them only for existing files), this conversion
does not exactly preserve bzr history and may theoretically cause
problems (e.g., build problems) for the affected revisions.
Fortunately, there were probably few such cases in the whole history
and all of them are old. Our analysis for Squid trunk showed the last
such revision is 9440 dated 2009-01-18, containing empty
'errors/Japanese' folder. Here is the generated list for trunk:

errors/Japanese, r9401-9440
errors/Czech, r9436-9438
lib/cppunit-1.10.0/contrib, r6693-8121
lib/cppunit-1.10.0/include/msvc6, r6693-8121
lib/cppunit-1.10.0/src/msvc6, r6693-8121
helpers/external_acl, r5495-5516
helpers, r5482-5494
helpers/external_acl, r5482-5494
helpers, r4765-5481

There are 2 possible ways to address this problem:

1. Leave as is.
2. During conversion, add dummy files to the history when the empty
   directory is created and remove those dummy files when the empty
   directory is either removed or becomes non-empty. This requires extra
   development cycles.

Note that both the missing (empty) directories and extra dummy files may
break builds for the related revisions.

If nobody is against, to minimize development overhead,
we are going to follow (1) in the above plan.

Thanks,

Eduard.

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Make PID file check/creation atomic

[Eduard Bagdasaryan]

Note that all PID-related methods are invoked before mainInitialize():
creating PID file ASAP was our goal here. Also, PID file operation
is a separate activity which has nothing common with DiskIO. In other
words, since DiskIO code knows nothing about PID file, it can't do any
harm with it.
I think your suggestion may be valuable but only after somebody
starts using File class in a threaded environment.


Eduard.

On 16.05.2017 15:21, Amos Jeffries wrote:

On 16/05/17 01:42, Eduard Bagdasaryan wrote:
Could you clarify, what are these processes that 'operate as 
threads'? AFAIK,
Squid spawns its children by fork() and execlp(), and each new 
process is a
common process with its own address space. As for atomicity, the OS 
itself

guarantees that only one process can have an exclusive file lock at a
given time. Where do you suggest to use std::atomic then?



The main Squid binary is threaded via the DiskIO implementations (at 
the least). These threads may be operational at any time after the 
first StoreInit() call, and unfortunately do reach other parts of the 
startup logic.



An static std::atomic lock/flag around the File::File() try{...} 
internals to make it threadsafe should resolve any potential issues 
there, the fnctl locks should be able to cope with anything once the 
lock() method returns.



Amos

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev


___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] A new 'has' ACL

[Eduard Bagdasaryan]

I did this change a bit differently, making these static SBufs
ACLHasComponentData class const members so that I could
reuse them in two methods, instead of duplicating C strings.
Reattaching the patch.

Eduard.

On 15.05.2017 14:38, Amos Jeffries wrote:

On 15/05/17 20:47, Eduard Bagdasaryan wrote:


Just a reminder: any remarks before this feature can be applied?




The SBuf in dump() should be static locals for performance. Otherwise +1. 


A new 'has' ACL was implemented.

This ACL detects presence of request, response or ALE transaction
components. Since many ACLs require some of these components, lack of
them in a transaction may spoil the check and confuse admin with
warnings like "... ACL is used in context without an HTTP request".
Using 'has' ACL should help dealing with these problems caused by
component-less transactions.

Also: addressed TODO in item #3 of trunk revision 14752.

=== modified file 'src/AclRegs.cc'
--- src/AclRegs.cc	2017-01-30 12:46:15 +
+++ src/AclRegs.cc	2017-05-15 19:04:56 +
@@ -17,60 +17,62 @@
 #include "acl/AdaptationService.h"
 #include "acl/AdaptationServiceData.h"
 #endif
 #include "acl/AllOf.h"
 #include "acl/AnnotateClient.h"
 #include "acl/AnnotateTransaction.h"
 #include "acl/AnnotationData.h"
 #include "acl/AnyOf.h"
 #if USE_SQUID_EUI
 #include "acl/Arp.h"
 #include "acl/Eui64.h"
 #endif
 #if USE_OPENSSL
 #include "acl/AtStep.h"
 #include "acl/AtStepData.h"
 #endif
 #include "acl/Asn.h"
 #include "acl/Browser.h"
 #include "acl/Checklist.h"
 #include "acl/ConnectionsEncrypted.h"
 #include "acl/Data.h"
 #include "acl/DestinationAsn.h"
 #include "acl/DestinationDomain.h"
 #include "acl/DestinationIp.h"
 #include "acl/DomainData.h"
 #if USE_AUTH
 #include "acl/ExtUser.h"
 #endif
 #include "acl/FilledChecklist.h"
 #include "acl/Gadgets.h"
+#include "acl/HasComponent.h"
+#include "acl/HasComponentData.h"
 #include "acl/HierCode.h"
 #include "acl/HierCodeData.h"
 #include "acl/HttpHeaderData.h"
 #include "acl/HttpRepHeader.h"
 #include "acl/HttpReqHeader.h"
 #include "acl/HttpStatus.h"
 #include "acl/IntRange.h"
 #include "acl/Ip.h"
 #include "acl/LocalIp.h"
 #include "acl/LocalPort.h"
 #include "acl/MaxConnection.h"
 #include "acl/Method.h"
 #include "acl/MethodData.h"
 #include "acl/MyPortName.h"
 #include "acl/Note.h"
 #include "acl/NoteData.h"
 #include "acl/PeerName.h"
 #include "acl/Protocol.h"
 #include "acl/ProtocolData.h"
 #include "acl/Random.h"
 #include "acl/Referer.h"
 #include "acl/RegexData.h"
 #include "acl/ReplyHeaderStrategy.h"
 #include "acl/ReplyMimeType.h"
 #include "acl/RequestHeaderStrategy.h"
 #include "acl/RequestMimeType.h"
 #include "acl/SourceAsn.h"
 #include "acl/SourceDomain.h"
 #include "acl/SourceIp.h"
 #include "acl/SquidError.h"
@@ -217,30 +219,33 @@ ACLMaxUserIP ACLMaxUserIP::RegistryEntry
 
 ACL::Prototype ACLTag::RegistryProtoype(::RegistryEntry_, "tag");
 ACLStrategised ACLTag::RegistryEntry_(new ACLStringData, ACLTagStrategy::Instance(), "tag");
 
 ACL::Prototype Acl::AnyOf::RegistryProtoype(::AnyOf::RegistryEntry_, "any-of");
 Acl::AnyOf Acl::AnyOf::RegistryEntry_;
 
 ACL::Prototype Acl::AllOf::RegistryProtoype(::AllOf::RegistryEntry_, "all-of");
 Acl::AllOf Acl::AllOf::RegistryEntry_;
 
 ACL::Prototype ACLNote::RegistryProtoype(::RegistryEntry_, "note");
 ACLStrategised ACLNote::RegistryEntry_(new ACLNoteData, ACLNoteStrategy::Instance(), "note");
 
 ACL::Prototype ACLAnnotateClient::RegistryProtoype(::RegistryEntry_, "annotate_client");
 ACLStrategised ACLAnnotateClient::RegistryEntry_(new ACLAnnotationData, ACLAnnotateClientStrategy::Instance(), "annotate_client");
 
 ACL::Prototype ACLAnnotateTransaction::RegistryProtoype(::RegistryEntry_, "annotate_transaction");
 ACLStrategised ACLAnnotateTransaction::RegistryEntry_(new ACLAnnotationData, ACLAnnotateTransactionStrategy::Instance(), "annotate_transaction");
 
 #if USE_ADAPTATION
 ACL::Prototype ACLAdaptationService::RegistryProtoype(::RegistryEntry_, "adaptation_service");
 ACLStrategised ACLAdaptationService::RegistryEntry_(new ACLAdaptationServiceData, ACLAdaptationServiceStrategy::Instance(), "adaptation_service");
 #endif
 
 ACL::Prototype ACLSquidError::RegistryProtoype(::RegistryEntry_, "squid_error");
 ACLStrategised ACLSquidError::RegistryEntry_(new ACLSquidErrorData, ACLSquidEr

Re: [squid-dev] [PATCH] Make PID file check/creation atomic

[Eduard Bagdasaryan]

Ssl::Lock class resides in 
src/security/cert_generators/file/certificate_db.h.



Eduard.

On 15.05.2017 15:04, Amos Jeffries wrote:


A new File class guarantees atomic PID file operations using locks. We
tried to generalize/reuse Ssl::Lock from the certificate generation


What Ssl::Lock class?


___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Do not revive unconditionally dead peers after DNS refresh

[Eduard Bagdasaryan]

I followed your plan with few adjustments and reattached the patch.
Since the new CachePeer::reprobe is a kind of 'helper' flag with
default value, I think no other initialization/reporting/cloning steps
needed (as for similar CachePeer::tcp_up).

Eduard.

On 06.05.2017 00:33, Alex Rousskov wrote:

On 04/27/2017 02:39 PM, Eduard Bagdasaryan wrote:

+// always start probing in order to effectively detect
+// dead or revived peers
+(void)peerProbeConnect(p);

I think we can simplify that comment while making it more precise:

 peerProbeConnect(p); // detect any died or revived peers ASAP

The (void) cast was correct, but is no longer needed after v5 r15133.

The above are minor polishing touches that could be done during commit,
but I am writing this message because I think we may be able to cover
one more corner case without significant changes...

Imagine a situation where the peers are already being probed during
peerDNSConfigure(). Patches Squid will do nothing nothing in this case.
However, the current probes may be using old/wrong IP addresses (or
other peer configuration parameters) and, hence, may produce wrong
result. This lack of a fresh probe may delay the discovery of a died or
revived peer and might even cause an HTTP transaction failure (in the
"the peer has recently died" case).

On the other hand, we still do not want to create too many concurrent or
too frequent probes so we want to continue to honor the current
peerProbeConnect() concurrency and rate limits.

Please see if the following small change allows Squid to handle the
above case better:

1. Add a boolean CachePeer::reprobe field, defaulted to false. It can be
described as "whether to do another TCP probe after the current TCP probes".

2. peerProbeConnect() should clear the reprobe field after passing the
two if-statement guards and before entering the loop (even if there are
no IPs to probe). The right timing here would eliminate any implicit
reprobing loops while providing the desired functionality.

3. After closing the connection, peerProbeConnectDone() should call
peerProbeConnect() if reprobe is true.

4. Add a boolean reprobeIfBusy parameter to peerProbeConnect() with
false as the default value. peerDNSConfigure() will call
peerProbeConnect() with a true parameter value. peerProbeConnect() will
start by setting reprobe to (reprobe || reprobeIfBusy).

There may be more initialization/reporting/cloning steps needed, just
like for any other CachePeer data member, but the above reflects the
core logic. Adjust as needed, of course.


Thank you,

Alex.


Do not revive unconditionally dead peers after DNS refresh.

Every hour, peerRefreshDNS() performs a DNS lookup of all cache_peer
addresses. Before this patch, even if DNS lookup results did not change,
the associated peerDNSConfigure() code silently cleared dead peer
marking, if any (CachePeer::tcp_up counter).  Forcefully reviving dead
peers every hour can lead to transaction delays (and delays may lead to
failures) due to connection timeouts when using a still dead peer.

This patch starts standard TCP probing instead of pointless dead peer
reviving, correctly refreshing peer state.  The primary goal is to cover
situation when DNS refresh changes peer address list. However TCP
probing may be useful for other situations either, without much overhead
(that is why it starts unconditionally).  For example, we need it when
DNS refresh returns the same addresses list but in different order. Also
it should help with dead idle peers detection.

Also: restart peer probing if peerDNSConfigure() is invoked when peers
are already being probed. This change should avoid situation when
current probes may produce wrong results due to using old/wrong IP
addresses.

=== modified file 'src/CachePeer.h'
--- src/CachePeer.h	2017-04-14 14:35:11 +
+++ src/CachePeer.h	2017-05-15 09:55:25 +
@@ -118,60 +118,62 @@ public:
 bool originserver = false;
 bool no_tproxy = false;
 #if PEER_MULTICAST_SIBLINGS
 bool mcast_siblings = false;
 #endif
 bool auth_no_keytab = false;
 } options;
 
 int weight = 1;
 int basetime = 0;
 
 struct {
 double avg_n_members = 0.0;
 int n_times_counted = 0;
 int n_replies_expected = 0;
 int ttl = 0;
 int id = 0;
 
 struct {
 bool count_event_pending = false;
 bool counting = false;
 } flags;
 } mcast;
 
 #if USE_CACHE_DIGESTS
 PeerDigest *digest = nullptr;
 char *digest_url = nullptr;
 #endif
 
 int tcp_up = 0; /* 0 if a connect() fails */
+/// whether to do another TCP probe after current TCP probes
+bool reprobe = false;
 
 Ip::Address addresses[10];
 int n_addresses = 0;
 int rr_count = 0;
 CachePeer *next = nullptr;
 int testing_now = 0;
 
 struct {
 unsigned int hash = 0;
 double load_multiplier = 0.0;
 doub

Re: [squid-dev] [PATCH] Fix 'miss_access' and 'cache' checks when no ACL rules matched

[Eduard Bagdasaryan]

On 12.05.2017 07:54, Amos Jeffries wrote:
The other access lists which obviously treat non-allowed as denied are 
very recent additions. So using them as a template to re-write 
existing and widely used directives behaviour is not great. 


Frankly speaking, the "cache" directive behavior changed rather
recently (r14984), as I noted above.  Can we say that it became 'widely' 
used

since then? On the contrary, I suspect that this change broke (or eventually
will break) some existing installations.
I so, the only "miss_access" directive check change may
break some installations.  Should we make an exception for this single
directive or formalize the rules, making them identical for all directives?
Probably the latter would be better for long term.  We can postpone this 
change

of course, adding a warning message for admin, that
'dunno' or 'auth required' outcomes will be denied in future releases.


Eduard.
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] [PATCH] Fix reopened bug 2833

[Eduard Bagdasaryan]

Hello,

This patch fixes [reopened] bug 2833.

A security fix made in r14979 had a negative effect on collapsed
forwarding. All "private" entries were considered automatically
non-shareable among collapsed clients. However this is not true: there
are many situations when collapsed forwarding should work despite of
"private" entry status: 304/5xx responses are good examples of that.
This patch fixes that by means of a new StoreEntry::shareableWhenPrivate
flag.

The suggested fix is not complete: to cover all possible situations we
need to decide whether StoreEntry::shareableWhenPrivate is true or not
for all contexts where StoreEntry::setPrivateKey() is used. This patch
fixes only few important cases inside http.cc, making CF (as well
collapsed revalidation) work for some [non-cacheable] response status
codes, including 3xx, 5xx and some others.

Also: avoid sending 304 responses for non-conditional requests.
Before this change, the original 'non-conditional' HttpRequest was still
marked (and processed) as 'conditional' after revalidation completion.
That happened because 'Last-Modified' and 'ETag' values were not
saved/restored while performing internal revalidation request.


Regards,

Eduard.

Fix [reopened] bug 2833.

A security fix made in r14979 had a negative effect on collapsed
forwarding. All "private" entries were considered automatically
non-shareable among collapsed clients. However this is not true: there
are many situations when collapsed forwarding should work despite of
"private" entry status: 304/5xx responses are good examples of that.
This patch fixes that by means of a new StoreEntry::shareableWhenPrivate
flag.

The suggested fix is not complete: to cover all possible situations we
need to decide whether StoreEntry::shareableWhenPrivate is true or not
for all contexts where StoreEntry::setPrivateKey() is used. This patch
fixes only few important cases inside http.cc, making CF (as well
collapsed revalidation) work for some [non-cacheable] response status
codes, including 3xx, 5xx and some others.

Also: avoid sending 304 responses for non-conditional requests.
Before this change, the original 'non-conditional' HttpRequest was still
marked (and processed) as 'conditional' after revalidation completion.
That happened because 'Last-Modified' and 'ETag' values were not
saved/restored while performing internal revalidation request.

=== modified file 'src/MemStore.cc'
--- src/MemStore.cc	2017-02-20 04:56:00 +
+++ src/MemStore.cc	2017-05-07 22:08:14 +
@@ -447,61 +447,61 @@ MemStore::updateCollapsedWith(StoreEntry
 }
 
 /// anchors StoreEntry to an already locked map entry
 void
 MemStore::anchorEntry(StoreEntry , const sfileno index, const Ipc::StoreMapAnchor )
 {
 const Ipc::StoreMapAnchor::Basics  = anchor.basics;
 
 e.swap_file_sz = basics.swap_file_sz;
 e.lastref = basics.lastref;
 e.timestamp = basics.timestamp;
 e.expires = basics.expires;
 e.lastModified(basics.lastmod);
 e.refcount = basics.refcount;
 e.flags = basics.flags;
 
 assert(e.mem_obj);
 if (anchor.complete()) {
 e.store_status = STORE_OK;
 e.mem_obj->object_sz = e.swap_file_sz;
 e.setMemStatus(IN_MEMORY);
 } else {
 e.store_status = STORE_PENDING;
 assert(e.mem_obj->object_sz < 0);
 e.setMemStatus(NOT_IN_MEMORY);
 }
 assert(e.swap_status == SWAPOUT_NONE); // set in StoreEntry constructor
 e.ping_status = PING_NONE;
 
 EBIT_CLR(e.flags, RELEASE_REQUEST);
-EBIT_CLR(e.flags, KEY_PRIVATE);
+e.clearPrivate();
 EBIT_SET(e.flags, ENTRY_VALIDATED);
 
 MemObject::MemCache  = e.mem_obj->memCache;
 mc.index = index;
 mc.io = MemObject::ioReading;
 }
 
 /// copies the entire entry from shared to local memory
 bool
 MemStore::copyFromShm(StoreEntry , const sfileno index, const Ipc::StoreMapAnchor )
 {
 debugs(20, 7, "mem-loading entry " << index << " from " << anchor.start);
 assert(e.mem_obj);
 
 // emulate the usual Store code but w/o inapplicable checks and callbacks:
 
 Ipc::StoreMapSliceId sid = anchor.start; // optimize: remember the last sid
 bool wasEof = anchor.complete() && sid < 0;
 int64_t sliceOffset = 0;
 while (sid >= 0) {
 const Ipc::StoreMapSlice  = map->readableSlice(index, sid);
 // slice state may change during copying; take snapshots now
 wasEof = anchor.complete() && slice.next < 0;
 const Ipc::StoreMapSlice::Size wasSize = slice.size;
 
 debugs(20, 9, "entry " << index << " slice " << sid << " eof " <<
wasEof << " wasSize " << wasSize << " <= " <<
anchor.basics.swap_file_sz << " sliceOffset " << sliceOffset <<
" mem.endOffset " << e.mem_obj->endOffset());
 

=== modified file 'src/Store.h'
--- src/Store.h	2017-04-12 00:00:22 +
+++ src/Store.h	2017-05-07 22:08:14 +
@@ -56,69 +56,73 @@ public:
  * \retval true   Store contains 0 bytes of data.
  * 

[squid-dev] [PATCH] Make PID file check/creation atomic

[Eduard Bagdasaryan]

This patch makes PID file check/creation atomic to avoid associated
race conditions.

Authors: Alex Rousskov, Eduard Bagdasaryan

After this change, if N Squid instances are concurrently started shortly
after time TS, then exactly one Squid instance (X) will run (and have
the corresponding PID file). If another Squid instance has already been
running (with the corresponding PID file) at TS, then X will be that
"old" Squid instance. If no Squid instances were running at TS, then X
will be one of those new N Squids started after TS.

Lack of atomic PID file operations caused unexpected Squid behavior:
* Mismatch between started Squid instance and stored PID file.
* Unexpected crashes due to failed allocation of shared resources,
  such as listening TCP ports or shared memory segments.

A new File class guarantees atomic PID file operations using locks. We
tried to generalize/reuse Ssl::Lock from the certificate generation
helper, but that was a bad idea: Helpers cannot use a lot of Squid code
(e.g., debugs(), TextException, SBuf, and enter_suid()), and the old
Ssl::Lock class cannot support shared locking without a major rewrite.

File locks on Solaris cannot work well (see bug #4212 comment #14), but
those problems do not affect PID file management code. Solaris- and
Windows-specific File code has not been tested and may not build.

Failure to write a PID file is now fatal. It used to be fatal only when
Squid was started with the -C command line option. In the increasingly
SMP world, running without a PID file leads to difficult-to-triage
errors. An admin who does not care about PID files should disable them.

Squid now exits with a non-zero error code if another Squid is running.


Also removed PID file rewriting during reconfiguration in non-daemon
mode. Squid daemons do not support PID file reconfiguration since trunk
r13867, but that revision (accidentally?) left behind half-broken
reconfiguration code for non-daemon mode. Fixing that code is difficult,
and supporting PID reconfigure in non-daemons is probably unnecessary.

Also fixed "is Squid running?" check when kill(0) does not have
permissions to signal the other instance. This does happen when Squid is
started (e.g., on the command line) by a different user than the user
Squid normally runs as or, perhaps, when the other Squid instance enters
a privileged section at the time of the check (untested). The bug could
result in undelivered signals or multiple running Squid instances.

These changes do not alter partially broken enter/leave_suid() behavior
of main.cc. That old code will need to be fixed separately!

PID file-related cache.log messages have changed slightly to improve
consistency with other DBG_IMPORTANT messages and to simplify code.
Squid no longer lies about creating a non-configured PID file.


* Terminal errors should throw instead of calling exit()

Squid used to call exit() in many PID-related error cases. Using exit()
as an error handling mechanism creates several problems:

1. exit() does not unwind the stack, possibly executing atexit()
   handlers in the wrong (e.g., privileged) context, possibly leaving
   some RAII-controller resources in bad state, and complicating triage;
2. Using exit() complicates code by adding a yet another error handling
   mechanism to the (appropriate) exceptions and assertions.
3. Spreading exit() calls around the code obscures unreachable code
   areas, complicates unifying exit codes, and confuses code checkers.

Long-term, it is best to use exceptions for nearly all error handling.
Reaching that goal will take time, but we can and should move in that
direction: The adjusted SquidMainSafe() treats exceptions as fatal
errors, without dumping core or assuming that no exception can reach
SquidMainSafe() on purpose. This trivial-looking change significantly
simplified (and otherwise improved) PID-file handling code!


Thanks,
Eduard.
Make PID file check/creation atomic to avoid associated race conditions.

After this change, if N Squid instances are concurrently started shortly
after time TS, then exactly one Squid instance (X) will run (and have
the corresponding PID file). If another Squid instance has already been
running (with the corresponding PID file) at TS, then X will be that
"old" Squid instance. If no Squid instances were running at TS, then X
will be one of those new N Squids started after TS.

Lack of atomic PID file operations caused unexpected Squid behavior:
* Mismatch between started Squid instance and stored PID file.
* Unexpected crashes due to failed allocation of shared resources,
  such as listening TCP ports or shared memory segments.

A new File class guarantees atomic PID file operations using locks. We
tried to generalize/reuse Ssl::Lock from the certificate generation
helper, but that was a bad idea: Helpers cannot use a lot of Squid code
(e.g., debugs(), TextException, SBuf, and enter_suid()), and the old
Ssl::Lock class cannot support 

[squid-dev] [PATCH] Fix 'miss_access' and 'cache' checks when no ACL rules matched

[Eduard Bagdasaryan]

Hello,

This patch fixes "miss_access" and "cache" checks when no ACL rules
matched.

The miss_access code allowed transactions to reach origin server when no
"allow" rules matched (and no "deny" rule matched either). This could
happen when a miss_access directive ACL could not make a "match" or
"mismatch" decision (e.g., the admin used a slow ACL or an ACL that
required authentication when the user was not authenticated).

Similarly, the "cache" directive code allowed requests to be served from
the cache (and responses to be stored in the cache) when no "allow" (and
"deny") rules matched. This behavior (established in v5 r14984) does not
contradict "Requests denied by this directive will not be..."
documentation, though.

I believe that both "miss_access" and "cache" directives should behave
like all other directives in similar contexts, i.e., "deny" in such
indeterminate cases because:

* It avoids problems with configurations like:

   # broken if badGuys fails to match or mismatch (allows bad guys)
   acl_driven_option allow !badGuys

  That is what trunk r12176 has partially fixed.

* It makes ACL checking rules consistent with each other, especially if
  there is no good reason for making exception for these two cases.

To help avoid similar problems in the future, and to improve code
consistency, I added an API that eliminates the need for direct
comparison with ACCESS_ALLOWED and ACCESS_DENIED.

Thanks,

Eduard.

Fixed "miss_access" and "cache" checks when no ACL rules matched.

The miss_access code allowed transactions to reach origin server when no
"allow" rules matched (and no "deny" rule matched either). This could
happen when a miss_access directive ACL could not make a "match" or
"mismatch" decision (e.g., the admin used a slow ACL or an ACL that
required authentication when the user was not authenticated).

Similarly, the "cache" directive code allowed requests to be served from
the cache (and responses to be stored in the cache) when no "allow" (and
"deny") rules matched. This behavior (established in v5 r14984) does not
contradict "Requests denied by this directive will not be..."
documentation, though.

I believe that both "miss_access" and "cache" directives should behave
similarly as all other directives, i.e., "deny" in such indeterminate
cases because: 

* It avoids problems with configurations like:

   # broken if badGuys fails to match or mismatch (allows bad guys)
   acl_driven_option allow !badGuys

  That is what trunk r12176 has partially fixed.

* It makes ACL checking rules consistent with each other, especially if
  there is no good reason for making exception for these two cases.

To help avoid similar problems in the future, and to improve code
consistency, I added an API that eliminates the need for direct
comparison with ACCESS_ALLOWED and ACCESS_DENIED.

=== modified file 'src/DelayId.cc'
--- src/DelayId.cc	2017-01-01 00:12:22 +
+++ src/DelayId.cc	2017-05-04 17:14:41 +
@@ -74,61 +74,61 @@ DelayId::DelayClient(ClientHttpRequest *
 if (r->client_addr.isNoAddr()) {
 debugs(77, 2, "delayClient: WARNING: Called with 'NO_ADDR' address, ignoring");
 return DelayId();
 }
 
 for (pool = 0; pool < DelayPools::pools(); ++pool) {
 
 /* pools require explicit 'allow' to assign a client into them */
 if (!DelayPools::delay_data[pool].access) {
 debugs(77, DBG_IMPORTANT, "delay_pool " << pool <<
" has no delay_access configured. This means that no clients will ever use it.");
 continue;
 }
 
 ACLFilledChecklist ch(DelayPools::delay_data[pool].access, r, NULL);
 if (reply) {
 ch.reply = reply;
 HTTPMSGLOCK(reply);
 }
 #if FOLLOW_X_FORWARDED_FOR
 if (Config.onoff.delay_pool_uses_indirect_client)
 ch.src_addr = r->indirect_client_addr;
 else
 #endif /* FOLLOW_X_FORWARDED_FOR */
 ch.src_addr = r->client_addr;
 ch.my_addr = r->my_addr;
 
 if (http->getConn() != NULL)
 ch.conn(http->getConn());
 
-if (DelayPools::delay_data[pool].theComposite().getRaw() && ch.fastCheck() == ACCESS_ALLOWED) {
+if (DelayPools::delay_data[pool].theComposite().getRaw() && ch.fastCheck().allowed()) {
 
 DelayId result (pool + 1);
 CompositePoolNode::CompositeSelectionDetails details;
 details.src_addr = ch.src_addr;
 #if USE_AUTH
 details.user = r->auth_user_request;
 #endif
 details.tag = r->tag;
 result.compositePosition(DelayPools::delay_data[pool].theComposite()->id(details));
 return result;
 }
 }
 
 return DelayId();
 }
 
 void
 DelayId::setNoDelay(bool const newValue)
 {
 markedAsNoDelay = newValue;
 }
 
 /*
  * this returns the number of bytes the client is permitted. it does not take
  * into account bytes already buffered - that is up to the caller.
  */
 int
 

[squid-dev] [PATCH] A new 'has' ACL

[Eduard Bagdasaryan]

Hello,

This patch implements a new 'has' ACL.

This ACL detects presence of request, response or ALE transaction
components.

The feature specification and related discussion can be found at:

http://lists.squid-cache.org/pipermail/squid-dev/2017-May/008559.html


Regards,

Eduard.

A new 'has' ACL was implemented.

This ACL detects presence of request, response or ALE transaction
components. Since many ACLs require some of these components, lack of
them in a transaction may spoil the check and confuse admin with
warnings like "... ACL is used in context without an HTTP request".
Using 'has' ACL should help dealing with these problems caused by
component-less transactions.

Also: addressed TODO in item #3 of trunk revision 14752.

=== modified file 'src/AclRegs.cc'
--- src/AclRegs.cc	2017-01-30 12:46:15 +
+++ src/AclRegs.cc	2017-04-30 11:11:06 +
@@ -17,60 +17,62 @@
 #include "acl/AdaptationService.h"
 #include "acl/AdaptationServiceData.h"
 #endif
 #include "acl/AllOf.h"
 #include "acl/AnnotateClient.h"
 #include "acl/AnnotateTransaction.h"
 #include "acl/AnnotationData.h"
 #include "acl/AnyOf.h"
 #if USE_SQUID_EUI
 #include "acl/Arp.h"
 #include "acl/Eui64.h"
 #endif
 #if USE_OPENSSL
 #include "acl/AtStep.h"
 #include "acl/AtStepData.h"
 #endif
 #include "acl/Asn.h"
 #include "acl/Browser.h"
 #include "acl/Checklist.h"
 #include "acl/ConnectionsEncrypted.h"
 #include "acl/Data.h"
 #include "acl/DestinationAsn.h"
 #include "acl/DestinationDomain.h"
 #include "acl/DestinationIp.h"
 #include "acl/DomainData.h"
 #if USE_AUTH
 #include "acl/ExtUser.h"
 #endif
 #include "acl/FilledChecklist.h"
 #include "acl/Gadgets.h"
+#include "acl/HasComponent.h"
+#include "acl/HasComponentData.h"
 #include "acl/HierCode.h"
 #include "acl/HierCodeData.h"
 #include "acl/HttpHeaderData.h"
 #include "acl/HttpRepHeader.h"
 #include "acl/HttpReqHeader.h"
 #include "acl/HttpStatus.h"
 #include "acl/IntRange.h"
 #include "acl/Ip.h"
 #include "acl/LocalIp.h"
 #include "acl/LocalPort.h"
 #include "acl/MaxConnection.h"
 #include "acl/Method.h"
 #include "acl/MethodData.h"
 #include "acl/MyPortName.h"
 #include "acl/Note.h"
 #include "acl/NoteData.h"
 #include "acl/PeerName.h"
 #include "acl/Protocol.h"
 #include "acl/ProtocolData.h"
 #include "acl/Random.h"
 #include "acl/Referer.h"
 #include "acl/RegexData.h"
 #include "acl/ReplyHeaderStrategy.h"
 #include "acl/ReplyMimeType.h"
 #include "acl/RequestHeaderStrategy.h"
 #include "acl/RequestMimeType.h"
 #include "acl/SourceAsn.h"
 #include "acl/SourceDomain.h"
 #include "acl/SourceIp.h"
 #include "acl/SquidError.h"
@@ -217,30 +219,33 @@ ACLMaxUserIP ACLMaxUserIP::RegistryEntry
 
 ACL::Prototype ACLTag::RegistryProtoype(::RegistryEntry_, "tag");
 ACLStrategised ACLTag::RegistryEntry_(new ACLStringData, ACLTagStrategy::Instance(), "tag");
 
 ACL::Prototype Acl::AnyOf::RegistryProtoype(::AnyOf::RegistryEntry_, "any-of");
 Acl::AnyOf Acl::AnyOf::RegistryEntry_;
 
 ACL::Prototype Acl::AllOf::RegistryProtoype(::AllOf::RegistryEntry_, "all-of");
 Acl::AllOf Acl::AllOf::RegistryEntry_;
 
 ACL::Prototype ACLNote::RegistryProtoype(::RegistryEntry_, "note");
 ACLStrategised ACLNote::RegistryEntry_(new ACLNoteData, ACLNoteStrategy::Instance(), "note");
 
 ACL::Prototype ACLAnnotateClient::RegistryProtoype(::RegistryEntry_, "annotate_client");
 ACLStrategised ACLAnnotateClient::RegistryEntry_(new ACLAnnotationData, ACLAnnotateClientStrategy::Instance(), "annotate_client");
 
 ACL::Prototype ACLAnnotateTransaction::RegistryProtoype(::RegistryEntry_, "annotate_transaction");
 ACLStrategised ACLAnnotateTransaction::RegistryEntry_(new ACLAnnotationData, ACLAnnotateTransactionStrategy::Instance(), "annotate_transaction");
 
 #if USE_ADAPTATION
 ACL::Prototype ACLAdaptationService::RegistryProtoype(::RegistryEntry_, "adaptation_service");
 ACLStrategised ACLAdaptationService::RegistryEntry_(new ACLAdaptationServiceData, ACLAdaptationServiceStrategy::Instance(), "adaptation_service");
 #endif
 
 ACL::Prototype ACLSquidError::RegistryProtoype(::RegistryEntry_, "squid_error");
 ACLStrategised ACLSquidError::RegistryEntry_(new ACLSquidErrorData, ACLSquidErrorStrategy::Instance(), "squid_error");
 
 ACL::Prototype Acl::ConnectionsEncrypted::RegistryProtoype(::ConnectionsEncrypted::RegistryEntry_, "connections_encrypted");
 Acl::ConnectionsEncrypted Acl::ConnectionsEncrypted::RegistryEntry_("connections_encrypted");
 
+ACL::Prototype ACLHasComponent::RegistryProtoype(::RegistryEntry_, "has");
+ACLStrategised ACLHasComponent::RegistryEntry_(new ACLHasComponentData, ACLHasComponentStrategy::Instance(), "has");
+

=== added file 'src/acl/HasComponent.cc'
--- src/acl/HasComponent.cc	1970-01-01 00:00:00 +
+++ src/acl/HasComponent.cc	2017-05-04 14:28:12 +
@@ -0,0 +1,28 @@
+/*
+ * Copyright (C) 1996-2017 The Squid Software Foundation and contributors
+ *
+ * Squid software is distributed under GPLv2+ license and includes
+ * contributions from numerous individuals and organizations.
+ * Please see 

Re: [squid-dev] A new 'has' ACL

[Eduard Bagdasaryan]

On 02.05.2017 23:53, Alex Rousskov wrote:

AFAICT, r14752 deals with transactions originated by client requests and
broken client connections. Even if all those transactions are fully
covered, there are other transactions (e.g., ESI and Downloader
requests) that may or may not be covered (and seem to be completely
unrelated to r14752 changes!). Is there some kind of common interface
that essentially requires all transactions to have ALE now?

I have not found any ALE manipulations inside ESI and Downloader
code, so that is probably unrelated.
ClientHttpRequest is the object used in transactions, (including
Downloader and ESI code). This object owns its ALE, which is
created in the constructor. I have not found 'common interface'
you are talking about, but if all Squid transactions require
ClientHttpRequest object (which is probably true) then they surely
will have the corresponding ALE.

Normally it is stored in
ClientHttpRequest::al.  However, I see that many ACLFilledChecklist
objects does not initialize their ACLFilledChecklist::al (assigning from
the transaction ALE). That works for now in most cases, probably
because there is the only ACLExternal which needs it and overrides
requiresAle() to return true. On the other hand, the lack of proper
ACLFilledChecklist::al initialization makes impossible correct using
ALE component in our 'has' ACL.

Thank you for researching this. Does the following summary accurately
reflect your findings as far as the current code is concerned?

"The current ACL-driven code may feed an ACLFilledChecklist object
without ALE to an (ACLExternal) ACL that requires ALE."


Yes.


Evidently, if we fix this (e.g., with mandatory initializing
ACLFilledChecklist::al in its constructor), we would not need to
check ALE component either.

The question is whether we can "fix this" (i.e., guarantee correct ALE
presence when ALE is needed) _quickly_. I doubt that all
ACLFilledChecklist objects can easily reach the right ALE from their
constructing context. I am worried that fixing this will turn into a
serious, large project. Are you sure it will not?

I agree that it is possible large project, because I am not
sure that every ACL checking context have its ALE near at hand.

If you are not sure, then I suggest keeping "has ALE" support.


... and have our new 'has' ACL broken from the very beginning too.
As I tried to explain earlier, how can we rely on
ACLFilledChecklist::hasAle(), if the ALE itself have not been provided to
ACLFilledChecklist before?


  We can
easily make "has ALE" a no-op when/if the code is fixed to always have
access to ALE when ALE is needed.

I see no reason to resist supporting "has ALE" if Squid support for ALE
is currently broken and cannot be fixed easily/quickly. If you see flaws
in this logic, please point them out. I may be missing something.



Eduard.
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] A new 'has' ACL

[Eduard Bagdasaryan]

As I noted before, after r14752 each transaction(even a failed one,
see logAcceptError()) has its ALE created. Normally it is stored in
ClientHttpRequest::al.  However, I see that many ACLFilledChecklist
objects does not initialize their ACLFilledChecklist::al (assigning from
the transaction ALE). That works for now in most cases, probably
because there is the only ACLExternal which needs it and overrides
requiresAle() to return true. On the other hand, the lack of proper
ACLFilledChecklist::al initialization makes impossible correct using
ALE component in our 'has' ACL.

Evidently, if we fix this (e.g., with mandatory initializing
ACLFilledChecklist::al in its constructor), we would not need to
check ALE component either.


Eduard.

On 01.05.2017 19:11, Alex Rousskov wrote:

On 04/30/2017 10:03 PM, Amos Jeffries wrote:


Is there an explicit need you have found for ALE to be on the
component list? Since ALE is currently standing in as a "master
transaction" object for most of the Squid code. It needs to be either
created or provided/fetched from elsewhere whenever it is used.
Having an ACL that bypasses that would defeat bug-finding of places
where it is broken.

I would like to polish that question and correct its justification
logic: We can abuse admins as bug-finding guinea pigs only if we give
them tools to turn those bugs off. The "has" ACL is such a tool. Thus,
we need a "has" ACL component for any object that any ACL or ACL-driven
code uses (or should be using) conditionally. Hence, the question is:

Does Squid ACL or ACL-driven code correctly use (or should be using) ALE
conditionally?

If the answer is "yes", then ALE should be added to "has". If the answer
is "maybe", then ALE may be added to "has".

Please note that it is not necessary to have a specific crash or warning
bug report for the answer to become "yes" or "maybe", justifying ALE
support in the "has" ACL. Squid code analysis is sufficient.

I cannot answer that question without doing more research, but I am sure
Eduard can find the right answer.


Thank you,

Alex.

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev


___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Do not forward HTTP requests to dead idle peers

[Eduard Bagdasaryan]

There are still no remarks about this patch.
Is there something to be fixed before it can go in?


Eduard.

On 18.04.2017 13:40, Eduard Bagdasaryan wrote:

Hello,

This patch removes "use dead idle peer" heuristic since
nobody has spoken in defense of this feature on Squid mailing lists:
http://lists.squid-cache.org/pipermail/squid-users/2017-March/014785.html
http://lists.squid-cache.org/pipermail/squid-dev/2017-March/008308.html

Note that the removed functionality was not used to detect revived
peers. All peer revival mechanisms (such as TCP probes) remain intact.


Regards,

Eduard.



___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev


___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] A new 'has' ACL

[Eduard Bagdasaryan]

Hello,


I am working on a new 'has' ACL:

  acl aclname has 

where "component" is one of the following three tokens: request, 
response, or ALE.

For example:

  acl hasRequest has request

Multiple components on one ACL line are not supported because they
would have to be ORed and ORing components would probably go against
admin expectaions in most cases:

  acl hasWhatMyLoggingDaemonNeeds has request response # XXX: Likely to 
be wrong!


Multiple same-name ACL lines would still be ORed to support arguably
rare use cases where ORing is appropriate:

  # OK, this strange logging daemon needs request or response,
  # but can work without either a request or a response:
  acl hasWhatMyLoggingDaemonNeeds has request
  acl hasWhatMyLoggingDaemonNeeds has response

This new ACL addresses a TODO in item #3 of trunk revision 14752,
adding that 'missing' configuration option.

For example, the following configuration:

  acl logMe note important_transaction
  access_log ... logformat=detailed logMe

gives such 'noise' warning messages:

  2017/01/06 12:54:46 kid2| WARNING: logMe ACL is used in context 
without an HTTP request. Assuming mismatch.
  2017/01/06 12:54:46 kid1| WARNING: logMe ACL is used in context 
without an HTTP request. Assuming mismatch.


Once the "has" ACL is available, the admin can do either:

  acl logMe note important_transaction
  access_log ... logformat=detailed hasRequest logMe

  or even

  acl logMe note important_transaction
  access_log ... logformat=detailed hasRequest logMe
  access_log ... logformat=brief !hasRequest logMe

 If you think these specs miss something important, let's discuss.


 Thanks,
 Eduard.

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Honor peer timeouts when forwarding CONNECTs

[Eduard Bagdasaryan]

Adjusted the patch accordingly.


Eduard.


On 22.03.2017 15:58, Amos Jeffries wrote:

On 17/03/2017 2:11 a.m., Eduard Bagdasaryan wrote:

On 16.03.2017 10:15, Amos Jeffries wrote:


in src/neighbours.cc:

* peerConnectTimeout() should be a member of the CachePeer class yes?

The initial patch version did as you suggest, but then we decided
to use separate method instead, to avoid 'heavy' dependency on
SquidConfig.h (which periodically causes linkage problems, IIRC).


Hmm. Okay for now, but please add a TODO or XXX about that to the function.


in src/tunnel.cc:

* "start" is an action name and we use it (almost?) exclusively for Job
initiation. By comparison "started" means/implies more clearly a state
or time point. The Tunnel member stores a time point.
  - IMO both are bad, but "started" is better than "start". Better still
would be a name that actually describes *what* has been start(ed).

I do not mind renaming it, e.g., into 'creationTime'.


Sounds good to me.


in src/PeerPoolMgr.cc:

* in PeerPoolMgr::handleOpenedConnection() I see a line doing max(1,
(peerTimeout - timeUsed)); just like the code in FwdState.cc
FwdState::connectDone().
  BUT, this PoolMgr is (wrongly) using int instead of time_t. Perhapse
that should be changed and something de-duplicate that max(1, x) usage?

I am not against changing int to time_t there. As for 'max', how about
creating inside neighbors.cc a helper method:
time_t PositiveTimeout(const time_t timeout) { return max(1, timeout); }


Sounds good. I did a few tests to see if we could also avoid the
static_cast, but it seems not in any simple way. So dont forget that in
the new function.



in src/comm/Connection.cc:

* I dont see why EnoughTimeToReForward() should be in the Comm::
namespace. It is about messageing layer timing. So is more appropriate
to be in FwdState:: or maybe SquidTime.h/time.cc.
  - if you pick time.cc that would also mean the fairly generic time_t
static functions it calls would move to time.cc where they seem more
appropriate.

Again, I am not against moving this method and ForwardTimeout() into
FwdState.cc (keeping related '*Forward' methods there makes some sense).


Okay, I can live with that.

Amos

Count failures and use peer-specific connect timeouts when tunneling.

Fixed two bugs with tunneling CONNECT requests (or equivalent traffic)
through a cache_peer:

1. Not detecting dead cache_peers due to missing code to count peer
   connect failures. SSL-level failures were detected (for "tls"
   cache_peers) but TCP/IP connect(2) failures were not (for all peers).

2. Origin server connect_timeout used instead of peer_connect_timeout or
   a peer-specific connect-timeout=N (where configured).

The regular forwarding code path does not have the above bugs. This
change reduces code duplication across the two code paths (that
duplication probably caused these bugs in the first place), but a lot
more work is needed in that direction.

The 5-second forwarding timeout hack has been in Squid since
forward_timeout inception (r6733). It is not without problems (now
marked with an XXX), but I left it as is to avoid opening another
Pandora box. The hack now applies to the tunneling code path as well.

=== modified file 'src/CachePeer.h'
--- src/CachePeer.h	2017-03-18 09:41:16 +
+++ src/CachePeer.h	2017-03-24 19:01:53 +
@@ -144,55 +144,55 @@ public:
 char *digest_url = nullptr;
 #endif
 
 int tcp_up = 0; /* 0 if a connect() fails */
 
 Ip::Address addresses[10];
 int n_addresses = 0;
 int rr_count = 0;
 CachePeer *next = nullptr;
 int testing_now = 0;
 
 struct {
 unsigned int hash = 0;
 double load_multiplier = 0.0;
 double load_factor = 0.0; ///< normalized weight value
 } carp;
 #if USE_AUTH
 struct {
 unsigned int hash = 0;
 double load_multiplier = 0.0;
 double load_factor = 0.0; ///< normalized weight value
 } userhash;
 #endif
 struct {
 unsigned int hash = 0;
 double load_multiplier = 0.0;
 double load_factor = 0.0; ///< normalized weight value
 } sourcehash;
 
 char *login = nullptr;/* Proxy authorization */
-time_t connect_timeout = 0;
+time_t connect_timeout_raw; ///< connect_timeout; use peerConnectTimeout() instead!
 int connect_fail_limit = 0;
 int max_conn = 0;
 
 /// optional "cache_peer standby=limit" feature
 struct {
 PconnPool *pool = nullptr;///< idle connection pool for this peer
 CbcPointer mgr;  ///< pool manager
 int limit = 0;///< the limit itself
 bool waitingForClose = false; ///< a conn must close before we open a standby conn
 } standby;
 
 char *domain = nullptr; ///< Forced domain
 
 /// security settings for peer connection
 Security::PeerOptions secure;
 Securit

Re: [squid-dev] [PATCH] Honor peer timeouts when forwarding CONNECTs

[Eduard Bagdasaryan]

On 16.03.2017 10:15, Amos Jeffries wrote:

>in src/neighbours.cc:
>
> * peerConnectTimeout() should be a member of the CachePeer class yes?

The initial patch version did as you suggest, but then we decided
to use separate method instead, to avoid 'heavy' dependency on
SquidConfig.h (which periodically causes linkage problems, IIRC).


> in src/tunnel.cc:
>
> * "start" is an action name and we use it (almost?) exclusively for Job
> initiation. By comparison "started" means/implies more clearly a state
> or time point. The Tunnel member stores a time point.
>  - IMO both are bad, but "started" is better than "start". Better still
> would be a name that actually describes *what* has been start(ed).

I do not mind renaming it, e.g., into 'creationTime'.


> in src/PeerPoolMgr.cc:
>
> * in PeerPoolMgr::handleOpenedConnection() I see a line doing max(1,
> (peerTimeout - timeUsed)); just like the code in FwdState.cc
> FwdState::connectDone().
>  BUT, this PoolMgr is (wrongly) using int instead of time_t. Perhapse
> that should be changed and something de-duplicate that max(1, x) usage?

I am not against changing int to time_t there. As for 'max', how about
creating inside neighbors.cc a helper method:
time_t PositiveTimeout(const time_t timeout) { return max(1, timeout); }


> in src/comm/Connection.cc:
>
> * I dont see why EnoughTimeToReForward() should be in the Comm::
> namespace. It is about messageing layer timing. So is more appropriate
> to be in FwdState:: or maybe SquidTime.h/time.cc.
>  - if you pick time.cc that would also mean the fairly generic time_t
> static functions it calls would move to time.cc where they seem more
> appropriate.

Again, I am not against moving this method and ForwardTimeout() into
FwdState.cc (keeping related '*Forward' methods there makes some sense).



Eduard.

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] VIA creation code duplication

[Eduard Bagdasaryan]

I would avoid using magic numbers like 64*1024.
If you are sure that this check is much better/informative,
consider making String::SizeMax_ public and use it instead.


Eduard.

On 16.03.2017 14:15, Amos Jeffries wrote:

On 16/03/2017 11:03 p.m., Eduard Bagdasaryan wrote:

If throwing when String exceeds its 64K limit is
acceptable then why not just use my previous t2 patch?
HttpHeader::addVia() calls strListAdd() similarly as the old
did: it throws if str->canGrowBy(itemSize) returns false.


I was hoping not to need a Must() at all. We still have the option of
passing buf instead of strVia to putStr().


Also, compare the information content of the exception messages:

Exception: str->canGrowBy(2)

Exception: str->canGrowBy(itemSize)

Exception: strVia.length() > 64*1024


If one was presented with only one of them. How would one go about
replicating the traffic to debug and fix it?

Only the latter offers the chance of a few minutes turnaround on a fix
if the Must() is acually hit in production traffic.

(I kind of hate those low-level operations with Must() and assert() like
the former - far too little context to be useful debugging with such big
impact when they are triggered.)

Amos

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev


___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] VIA creation code duplication

[Eduard Bagdasaryan]

If throwing when String exceeds its 64K limit is
acceptable then why not just use my previous t2 patch?
HttpHeader::addVia() calls strListAdd() similarly as the old
did: it throws if str->canGrowBy(itemSize) returns false.


Eduard.

On 16.03.2017 10:38, Amos Jeffries wrote:

On 14/03/2017 5:24 a.m., Alex Rousskov wrote:

On 03/13/2017 08:25 AM, Eduard Bagdasaryan wrote:

On 14.02.2017 04:22, Amos Jeffries wrote:

The problem is with proxy where the admin has configured large headers
to be allowed, and receives a Via just under the 6KB liit. Our append
pushing it over by even one byte would assert.

Yes, except s/6/64/ and no special configuration is probably needed:
reply_header_max_size is 64KB by default (and so is String::SizeMax_).



I am attaching a patch using SBuf instead of String for Via
accumulating, as you previously suggested. I am not
sure this is a much better solution since we have to do an
extra copy into SBuf.

Will this new patch assert in String::setBuffer(?) when the final Via
value exceeds String::SizeMax_ and putStr() is called with a 64KB+1 or
longer value string?

Yes I think it still misses the size check there.

Any objections to applying this with this added:

   // XXX: putStr() still has String 64KB limits
   Must(strVia.length() < 64*1024);

Amos

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev


___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] VIA creation code duplication

[Eduard Bagdasaryan]

On 14.02.2017 04:22, Amos Jeffries wrote:

The problem is with proxy where the admin has configured large headers
to be allowed, and receives a Via just under the 6KB liit. Our append
pushing it over by even one byte would assert.


I am attaching a patch using SBuf instead of String for Via
accumulating, as you previously suggested. I am not
sure this is a much better solution since we have to do an
extra copy into SBuf.

> The older bbuf code
> cropping at 1KB was nasty but would not crash Squid.

I don't see the code preventing Squid from String overflow
there. The bbuf you are talking about was appended to via string
by strListAdd() without any checks...


Thanks,
Eduard.

Fixed appending Http::HdrType::VIA code duplication.

=== modified file 'src/HttpHeader.cc'
--- src/HttpHeader.cc	2017-02-15 18:54:02 +
+++ src/HttpHeader.cc	2017-03-13 13:42:36 +
@@ -1,57 +1,58 @@
 /*
  * Copyright (C) 1996-2017 The Squid Software Foundation and contributors
  *
  * Squid software is distributed under GPLv2+ license and includes
  * contributions from numerous individuals and organizations.
  * Please see the COPYING and CONTRIBUTORS files for details.
  */
 
 /* DEBUG: section 55HTTP Header */
 
 #include "squid.h"
 #include "base/EnumIterator.h"
 #include "base64.h"
 #include "globals.h"
 #include "http/ContentLengthInterpreter.h"
 #include "HttpHdrCc.h"
 #include "HttpHdrContRange.h"
 #include "HttpHdrScTarget.h" // also includes HttpHdrSc.h
 #include "HttpHeader.h"
 #include "HttpHeaderFieldInfo.h"
 #include "HttpHeaderStat.h"
 #include "HttpHeaderTools.h"
 #include "MemBuf.h"
 #include "mgr/Registration.h"
 #include "mime_header.h"
 #include "profiler/Profiler.h"
 #include "rfc1123.h"
+#include "sbuf/StringConvert.h"
 #include "SquidConfig.h"
 #include "StatHist.h"
 #include "Store.h"
 #include "StrList.h"
 #include "TimeOrTag.h"
 #include "util.h"
 
 #include 
 
 /* XXX: the whole set of API managing the entries vector should be rethought
  *  after the parse4r-ng effort is complete.
  */
 
 /*
  * On naming conventions:
  *
  * HTTP/1.1 defines message-header as
  *
  * message-header = field-name ":" [ field-value ] CRLF
  * field-name = token
  * field-value= *( field-content | LWS )
  *
  * HTTP/1.1 does not give a name name a group of all message-headers in a message.
  * Squid 1.1 seems to refer to that group _plus_ start-line as "headers".
  *
  * HttpHeader is an object that represents all message-headers in a message.
  * HttpHeader does not manage start-line.
  *
  * HttpHeader is implemented as a collection of header "entries".
  * An entry is a (field_id, field_name, field_value) triplet.
@@ -979,60 +980,84 @@ HttpHeader::getListMember(Http::HdrType
 const char *item;
 int ilen;
 int mlen = strlen(member);
 
 assert(any_registered_header(id));
 
 header = getStrOrList(id);
 String result;
 
 while (strListGetItem(, separator, , , )) {
 if (strncmp(item, member, mlen) == 0 && item[mlen] == '=') {
 result.append(item + mlen + 1, ilen - mlen - 1);
 break;
 }
 }
 
 header.clean();
 return result;
 }
 
 /* test if a field is present */
 int
 HttpHeader::has(Http::HdrType id) const
 {
 assert(any_registered_header(id));
 debugs(55, 9, this << " lookup for " << id);
 return CBIT_TEST(mask, id);
 }
 
 void
+HttpHeader::addVia(const AnyP::ProtocolVersion , const HttpHeader *from)
+{
+// TODO: do not add Via header for messages where Squid itself
+// generated the message (i.e., Downloader or ESI) there should be no Via header added at all.
+
+if (Config.onoff.via) {
+SBuf buf;
+// RFC 7230 section 5.7.1.: protocol-name is omitted when
+// the received protocol is HTTP.
+if (ver.protocol > AnyP::PROTO_NONE && ver.protocol < AnyP::PROTO_UNKNOWN &&
+ver.protocol != AnyP::PROTO_HTTP && ver.protocol != AnyP::PROTO_HTTPS)
+buf.appendf("%s/", AnyP::ProtocolType_str[ver.protocol]);
+buf.appendf("%d.%d %s", ver.major, ver.minor, ThisCache);
+const HttpHeader *hdr = from ? from : this;
+SBuf strVia = StringToSBuf(hdr->getList(Http::HdrType::VIA));
+if (!strVia.isEmpty())
+strVia.append(", ");
+strVia.append(buf);
+delById(Http::HdrType::VIA);
+putStr(Http::HdrType::VIA, strVia.c_str());
+}
+}
+
+void
 HttpHeader::putInt(Http::HdrType id, int number)
 {
 assert(any_registered_header(id));
 assert(Http::HeaderLookupTable.lookup(id).type == Http::HdrFieldType::ftInt);  /* must be of an appropriate type */
 assert(number >= 0);
 addEntry(new HttpHeaderEntry(id, NULL, xitoa(number)));
 }
 
 void
 HttpHeader::putInt64(Http::HdrType id, int64_t number)
 {
 assert(any_registered_header(id));
 assert(Http::HeaderLookupTable.lookup(id).type == Http::HdrFieldType::ftInt64);/* must be of an appropriate type */
 assert(number >= 0);
 

[squid-dev] [PATCH] Honor peer timeouts when forwarding CONNECTs

[Eduard Bagdasaryan]

Hello,

This patch fixes two bugs with tunneling CONNECT requests (or equivalent
traffic) through a cache_peer:

1. Not detecting dead cache_peers due to missing code to count peer
   connect failures. SSL-level failures were detected (for "tls"
   cache_peers) but TCP/IP connect(2) failures were not (for all peers).

2. Origin server connect_timeout used instead of peer_connect_timeout or
   a peer-specific connect-timeout=N (where configured).

The regular forwarding code path does not have the above bugs. This
change reduces code duplication across the two code paths (that
duplication probably caused these bugs in the first place), but a lot
more work is needed in that direction.

This patch applies to v5 r15094. I assume v4 should be fixed as well:
the same patch applies to v4 r14999.


Thanks,
Eduard.
Count failures and use peer-specific connect timeouts when tunneling.

Fixed two bugs with tunneling CONNECT requests (or equivalent traffic)
through a cache_peer:

1. Not detecting dead cache_peers due to missing code to count peer
   connect failures. SSL-level failures were detected (for "tls"
   cache_peers) but TCP/IP connect(2) failures were not (for all peers).

2. Origin server connect_timeout used instead of peer_connect_timeout or
   a peer-specific connect-timeout=N (where configured).

The regular forwarding code path does not have the above bugs. This
change reduces code duplication across the two code paths (that
duplication probably caused these bugs in the first place), but a lot
more work is needed in that direction.

The 5-second forwarding timeout hack has been in Squid since
forward_timeout inception (r6733). It is not without problems (now
marked with an XXX), but I left it as is to avoid opening another
Pandora box. The hack now applies to the tunneling code path as well.

=== modified file 'src/CachePeer.cc'
--- src/CachePeer.cc	2017-01-01 00:12:22 +
+++ src/CachePeer.cc	2017-03-12 14:31:13 +
@@ -9,61 +9,61 @@
 #include "squid.h"
 #include "acl/Gadgets.h"
 #include "CachePeer.h"
 #include "defines.h"
 #include "NeighborTypeDomainList.h"
 #include "pconn.h"
 #include "PeerPoolMgr.h"
 
 CBDATA_CLASS_INIT(CachePeer);
 
 CachePeer::CachePeer() :
 index(0),
 name(NULL),
 host(NULL),
 type(PEER_NONE),
 http_port(CACHE_HTTP_PORT),
 typelist(NULL),
 access(NULL),
 weight(1),
 basetime(0),
 #if USE_CACHE_DIGESTS
 digest(NULL),
 digest_url(NULL),
 #endif
 tcp_up(0),
 n_addresses(0),
 rr_count(0),
 next(NULL),
 testing_now(false),
 login(NULL),
-connect_timeout(0),
+connect_timeout_raw(0),
 connect_fail_limit(0),
 max_conn(0),
 domain(NULL),
 front_end_https(0),
 connection_auth(2 /* auto */)
 {
 memset(, 0, sizeof(stats));
 stats.logged_state = PEER_ALIVE;
 
 memset(, 0, sizeof(icp));
 icp.port = CACHE_ICP_PORT;
 icp.version = ICP_VERSION_CURRENT;
 
 #if USE_HTCP
 memset(, 0, sizeof(htcp));
 #endif
 memset(, 0, sizeof(options));
 memset(, 0, sizeof(mcast));
 memset(, 0, sizeof(carp));
 #if USE_AUTH
 memset(, 0, sizeof(userhash));
 #endif
 memset(, 0, sizeof(sourcehash));
 
 standby.pool = NULL;
 standby.limit = 0;
 standby.waitingForClose = false;
 }
 
 CachePeer::~CachePeer()

=== modified file 'src/CachePeer.h'
--- src/CachePeer.h	2017-01-01 00:12:22 +
+++ src/CachePeer.h	2017-03-12 14:31:13 +
@@ -143,52 +143,52 @@ public:
 char *digest_url;
 #endif
 
 int tcp_up; /* 0 if a connect() fails */
 
 Ip::Address addresses[10];
 int n_addresses;
 int rr_count;
 CachePeer *next;
 int testing_now;
 
 struct {
 unsigned int hash;
 double load_multiplier;
 double load_factor; /* normalized weight value */
 } carp;
 #if USE_AUTH
 struct {
 unsigned int hash;
 double load_multiplier;
 double load_factor; /* normalized weight value */
 } userhash;
 #endif
 struct {
 unsigned int hash;
 double load_multiplier;
 double load_factor; /* normalized weight value */
 } sourcehash;
 
 char *login;/* Proxy authorization */
-time_t connect_timeout;
+time_t connect_timeout_raw; ///< connect_timeout; use peerConnectTimeout() instead!
 int connect_fail_limit;
 int max_conn;
 struct {
 PconnPool *pool; ///< idle connection pool for this peer
 CbcPointer mgr; ///< pool manager
 int limit; ///< the limit itself
 bool waitingForClose; ///< a conn must close before we open a standby conn
 } standby; ///< optional "cache_peer standby=limit" feature
 char *domain;   /* Forced domain */
 
 /// security settings for peer connection
 Security::PeerOptions secure;
 Security::ContextPointer sslContext;
 Security::SessionStatePointer sslSession;
 
 int front_end_https;
 int connection_auth;
 };
 
 #endif /* SQUID_CACHEPEER_H_ */
 

[squid-dev] [PATCH] Fix broken build for ufsdump

[Eduard Bagdasaryan]

Hello,

ufsdump build is broken now and seems that it became broken quite a long
time ago (though I have not tested when exactly), probably because is
not built by default. This patch fixes this, however I am not sure that
does it in a best possible way. For example, someone may argue that
inlining storeKeyText() is wrong. I am posting it in hope that somebody
else will polish and apply it.

BTW, running ufsdump produces an error:

FATAL: StoreMeta.cc required for use of Factory

Looks like it requires StoreMeta.cc instead of stub_StoreMeta.cc,
currently used.

Thanks,

Eduard.
Fix broken build for ufsdump utility.

=== modified file 'src/Makefile.am'
--- src/Makefile.am	2017-02-19 17:13:27 +
+++ src/Makefile.am	2017-03-05 18:49:54 +
@@ -600,87 +600,93 @@
 
 ## What requires what..
 ## many things want ACLChecklist.cc
 ## ACLChecklist.cc wants AuthUserRequest.cc
 ## ACLChecklist.cc wants AuthScheme.cc
 ## ACLChecklist.cc wants ACLProxyAuth.cc directly
 ## ACLProxyAuth.cc wants ACLUserData
 ## ACLProxyAuth.cc wants ACLRegexData
 ## cache_cf.cc wants $(AUTH_LIBS)
 ## cache_cf.cc wants store/libstore.la
 ## cache_cf.cc wants AnyP::PortCfg
 ## client_side wants client_db
 ## client_db wants SNMP_SOURCE
 ## snmp_core wants ACLStringData
 ## tools.cc wants ip/libip.la
 ## client_side.cc wants ip/libip.la
 ## libbase.la wants cbdata.*
 ## libbase.la wants MemBuf.*
 ufsdump_SOURCES = \
 	ClientInfo.h \
 	cbdata.h \
 	cbdata.cc \
 	debug.cc \
 	int.h \
 	int.cc \
 	mem/forward.h \
 	MemBuf.cc \
 	MemBuf.h \
 	Parsing.h \
 	store_key_md5.h \
-	store_key_md5.cc \
 	tests/stub_StoreMeta.cc \
 	StoreMetaUnpacker.cc \
+	StatHist.cc \
+	StatHist.h \
 	String.cc \
 	SquidNew.cc \
 	tests/stub_time.cc \
 	ufsdump.cc \
 	dlink.h \
 	dlink.cc \
 	helper/ChildConfig.h \
 	tests/stub_HelperChildConfig.cc \
 	RemovalPolicy.cc \
 	$(WIN32_SOURCE) \
 	fd.h \
 	tests/stub_fd.cc
 ufsdump_LDADD = \
 	ident/libident.la \
 	acl/libacls.la \
 	eui/libeui.la \
 	acl/libstate.la \
 	acl/libapi.la \
 	base/libbase.la \
 	libsquid.la \
 	ip/libip.la \
 	fs/libfs.la \
 	ipc/libipc.la \
 	mgr/libmgr.la \
+	sbuf/libsbuf.la \
+	mem/libmem.la \
+	SquidConfig.o \
+	$(top_builddir)/lib/libmiscutil.la \
+	$(top_builddir)/lib/libmiscencoding.la \
 	$(XTRA_OBJS) \
 	$(REPL_OBJS) \
 	$(NETTLELIB) \
 	$(CRYPTLIB) \
 	$(REGEXLIB) \
 	$(SSLLIB) \
 	$(COMPAT_LIB) \
 	$(EPOLL_LIBS) \
 	$(MINGW_LIBS) \
 	$(XTRA_LIBS)
 ufsdump_DEPENDENCIES = \
 	ident/libident.la \
 	acl/libacls.la \
 	eui/libeui.la \
 	acl/libstate.la \
 	acl/libapi.la \
 	base/libbase.la \
 	libsquid.la \
 	ip/libip.la \
 	fs/libfs.la \
 	ipc/libipc.la \
 	mgr/libmgr.la \
 	DiskIO/libdiskio.la \
 	$(REPL_OBJS)
 
 nodist_ufsdump_SOURCES = \
 	globals.cc
 
 sysconf_DATA = \
 	squid.conf.default \

=== modified file 'src/store_key_md5.cc'
--- src/store_key_md5.cc	2017-01-01 00:12:22 +
+++ src/store_key_md5.cc	2017-03-05 17:27:12 +
@@ -1,63 +1,47 @@
 /*
  * Copyright (C) 1996-2017 The Squid Software Foundation and contributors
  *
  * Squid software is distributed under GPLv2+ license and includes
  * contributions from numerous individuals and organizations.
  * Please see the COPYING and CONTRIBUTORS files for details.
  */
 
 /* DEBUG: section 20Storage Manager MD5 Cache Keys */
 
 #include "squid.h"
 #include "HttpRequest.h"
-#include "md5.h"
 #include "store_key_md5.h"
 #include "URL.h"
 
 static cache_key null_key[SQUID_MD5_DIGEST_LENGTH];
 
-const char *
-storeKeyText(const cache_key *key)
-{
-if (!key)
-return "[null_store_key]";
-
-static char buf[SQUID_MD5_DIGEST_LENGTH * 2+1];
-int i;
-
-for (i = 0; i < SQUID_MD5_DIGEST_LENGTH; ++i)
-snprintf([i*2],sizeof(buf) - i*2, "%02X", *(key + i));
-
-return buf;
-}
-
 const cache_key *
 storeKeyScan(const char *buf)
 {
 static unsigned char digest[SQUID_MD5_DIGEST_LENGTH];
 int i;
 int j = 0;
 char t[3];
 
 for (i = 0; i < SQUID_MD5_DIGEST_LENGTH; ++i) {
 t[0] = *(buf + (j++));
 t[1] = *(buf + (j++));
 t[2] = '\0';
 *(digest + i) = (unsigned char) strtol(t, NULL, 16);
 }
 
 return digest;
 }
 
 int
 storeKeyHashCmp(const void *a, const void *b)
 {
 const unsigned char *A = (const unsigned char *)a;
 const unsigned char *B = (const unsigned char *)b;
 int i;
 
 for (i = 0; i < SQUID_MD5_DIGEST_LENGTH; ++i) {
 if (A[i] < B[i])
 return -1;
 
 if (A[i] > B[i])

=== modified file 'src/store_key_md5.h'
--- src/store_key_md5.h	2017-02-26 10:24:15 +
+++ src/store_key_md5.h	2017-03-05 17:28:34 +
@@ -1,40 +1,56 @@
 /*
  * Copyright (C) 1996-2017 The Squid Software Foundation and contributors
  *
  * Squid software is distributed under GPLv2+ license and includes
  * contributions from numerous individuals and organizations.
  * Please see the COPYING and CONTRIBUTORS files for details.
  */
 
 /* DEBUG: section 20Storage Manager MD5 Cache Keys */
 
 #ifndef SQUID_STORE_KEY_MD5_H_
 

[squid-dev] [PATCH] Detail swapfile header inconsistencies

[Eduard Bagdasaryan]

Hello,

This patch improves Squid to better distinguish error cases when loading
cache entry metadata is failed. Knowing the exact failure reason may
help triage and guide development.  Refactoring also reduced code
duplication and  fixed a null pointer dereference inside ufsdump.cc (but
ufsdump does not even build right now for reasons unrelated to
these changes).

This patch applies to both v4 and v5.


Regards,

Eduard.

Detail swapfile header inconsistencies.

Squid may fail to load cache entry metadata for several very different
reasons, including the following two relatively common ones:

* A cache_dir entry corruption.
* Huge cache_dir entry metadata that does not fit into the I/O buffer
  used for loading entry metadata.

Knowing the exact failure reason may help triage and guide development.
We refactored existing checks to distinguish various error cases,
including the two above. Refactoring also reduced code duplication.

These improvements also uncovered and fixed a null pointer dereference
inside ufsdump.cc (but ufsdump does not even build right now for reasons
unrelated to these changes).

=== modified file 'src/StoreMetaUnpacker.cc'
--- src/StoreMetaUnpacker.cc	2017-02-10 23:37:05 +
+++ src/StoreMetaUnpacker.cc	2017-03-03 21:31:09 +
@@ -1,75 +1,72 @@
 /*
  * Copyright (C) 1996-2017 The Squid Software Foundation and contributors
  *
  * Squid software is distributed under GPLv2+ license and includes
  * contributions from numerous individuals and organizations.
  * Please see the COPYING and CONTRIBUTORS files for details.
  */
 
 /* DEBUG: section 20Storage Manager Swapfile Unpacker */
 
 #include "squid.h"
+#include "base/TextException.h"
 #include "Debug.h"
 #include "defines.h"
 #include "StoreMeta.h"
 #include "StoreMetaUnpacker.h"
 
 int const StoreMetaUnpacker::MinimumBufferLength = sizeof(char) + sizeof(int);
 
 /// useful for meta stored in pre-initialized (with zeros) db files
 bool
 StoreMetaUnpacker::isBufferZero()
 {
 // We could memcmp the entire buffer, but it is probably safe enough
 // to test a few bytes because if we do not detect a corrupted entry
 // it is not a big deal. Empty entries are not isBufferSane anyway.
 const int depth = 10;
 if (buflen < depth)
 return false; // cannot be sure enough
 
 for (int i = 0; i < depth; ++i) {
 if (buf[i])
 return false;
 }
 return true;
 }
 
-bool
-StoreMetaUnpacker::isBufferSane()
+void
+StoreMetaUnpacker::checkBuffer()
 {
-if (buf[0] != (char) STORE_META_OK)
-return false;
-
+assert(buf); // paranoid; already checked in the constructor
+if (buf[0] != static_cast(STORE_META_OK))
+throw TexcHere("store entry metadata is corrupted");
 /*
  * sanity check on 'buflen' value.  It should be at least big
  * enough to hold one type and one length.
  */
 getBufferLength();
-
 if (*hdr_len < MinimumBufferLength)
-return false;
-
+throw TexcHere("store entry metadata is too small");
 if (*hdr_len > buflen)
-return false;
-
-return true;
+throw TexcHere("store entry metadata is too big");
 }
 
 void
 StoreMetaUnpacker::getBufferLength()
 {
 memcpy(hdr_len, [1], sizeof(int));
 }
 
 StoreMetaUnpacker::StoreMetaUnpacker(char const *aBuffer, ssize_t aLen, int *anInt) :
 buf(aBuffer),
 buflen(aLen),
 hdr_len(anInt),
 position(1 + sizeof(int)),
 type('\0'),
 length(0),
 tail(NULL)
 {
 assert(aBuffer != NULL);
 }
 
@@ -105,35 +102,38 @@ StoreMetaUnpacker::doOneEntry()
 tail = StoreMeta::Add (tail, newNode);
 
 position += length;
 
 return true;
 }
 
 bool
 StoreMetaUnpacker::moreToProcess() const
 {
 return *hdr_len - position - MinimumBufferLength >= 0;
 }
 
 StoreMeta *
 StoreMetaUnpacker::createStoreMeta ()
 {
 tlv *TLV = NULL;
 tail = 
 assert(hdr_len != NULL);
 
-if (!isBufferSane())
-return NULL;
+checkBuffer();
 
 getBufferLength();
 
 assert (position == 1 + sizeof(int));
 
 while (moreToProcess()) {
 if (!doOneEntry())
 break;
 }
 
+if (!TLV)
+   throw TexcHere("store entry metadata is empty");
+
+assert(TLV);
 return TLV;
 }
 

=== modified file 'src/StoreMetaUnpacker.h'
--- src/StoreMetaUnpacker.h	2017-02-10 23:37:05 +
+++ src/StoreMetaUnpacker.h	2017-03-03 21:31:09 +
@@ -1,42 +1,43 @@
 /*
  * Copyright (C) 1996-2017 The Squid Software Foundation and contributors
  *
  * Squid software is distributed under GPLv2+ license and includes
  * contributions from numerous individuals and organizations.
  * Please see the COPYING and CONTRIBUTORS files for details.
  */
 
 #ifndef SQUID_TYPELENGTHVALUEUNPACKER_H
 #define SQUID_TYPELENGTHVALUEUNPACKER_H
 
 class StoreMeta;
 class StoreEntry;
 
 class StoreMetaUnpacker
 {
 
 public:
 StoreMetaUnpacker (const char *buf, ssize_t bufferLength, int *hdrlen);
 StoreMeta *createStoreMeta();

Re: [squid-dev] [PATCH] Case-insensitive URI schemes

[Eduard Bagdasaryan]

Any more suggestions/remarks here before this patch
can be applied?


Thanks,
Eduard.


On 07.02.2017 18:10, Eduard Bagdasaryan wrote:


Checked that it is ok to move AnyP::UriScheme::Init() as
you suggested. Re-attached the patch (v5 r15037).


Eduard.

On 02.02.2017 22:12, Alex Rousskov wrote:

> We should avoid this code duplication [...]
> However, please check whether we can move the
> call up, to place it above storeFsInit().
> Both of the above changes can be done during commit.



___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev


___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Compilation error after r15057

[Eduard Bagdasaryan]

There are also 'make check' problems reported by clang,
fix attached.


Eduard.


On 21.02.2017 01:24, Alex Rousskov wrote:

On 02/20/2017 02:37 PM, Eduard Bagdasaryan wrote:

That applied fix missed one case, attaching patch for it.

Committed to v5 (r15062).

Alex.


On 20.02.2017 21:06, Alex Rousskov wrote:

Attaching compilation fix for r15057.

Committed to v5 (r15061).

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev


Fix 'make check' to work with clang after r15057.

=== modified file 'src/tests/stub_DelayId.cc'
--- src/tests/stub_DelayId.cc	2017-01-01 00:12:22 +
+++ src/tests/stub_DelayId.cc	2017-02-21 09:16:49 +
@@ -1,25 +1,30 @@
 /*
  * Copyright (C) 1996-2017 The Squid Software Foundation and contributors
  *
  * Squid software is distributed under GPLv2+ license and includes
  * contributions from numerous individuals and organizations.
  * Please see the COPYING and CONTRIBUTORS files for details.
  */
 
 /* DEBUG: section 20Storage Manager */
 
 #include "squid.h"
 
 #if USE_DELAY_POOLS
+#include "BandwidthBucket.h"
 #include "DelayId.h"
 
 #define STUB_API "stub_DelayId.cc"
 #include "tests/STUB.h"
 
 DelayId::DelayId(): pool_(0), compositeId(NULL), markedAsNoDelay(false) {}
 DelayId::~DelayId() {}
 
 void DelayId::delayRead(DeferredRead const&) STUB_NOP
+void BandwidthBucket::refillBucket() STUB
+bool BandwidthBucket::applyQuota(int &, Comm::IoCallback *) STUB_RETVAL(false)
+BandwidthBucket *BandwidthBucket::SelectBucket(fde *) STUB_RETVAL(nullptr)
+void BandwidthBucket::reduceBucket(const int len) STUB
 
 #endif /* USE_DELAY_POOLS */
 

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Compilation error after r15057

[Eduard Bagdasaryan]

Hello,

That applied fix missed one case, attaching patch for it.

Eduard.


On 20.02.2017 21:06, Alex Rousskov wrote:

Attaching compilation fix for r15057.

Committed to v5 (r15061).




Compilation error fix after r15057.

=== modified file 'src/client_db.cc'
--- src/client_db.cc	2017-02-19 17:13:27 +
+++ src/client_db.cc	2017-02-20 21:24:10 +
@@ -413,61 +413,61 @@
 
 debugs(49, 2, "clientdbGC: Removed " << cleanup_removed << " entries");
 }
 }
 
 static void
 clientdbStartGC(void)
 {
 max_clients = statCounter.client_http.clients;
 cleanup_running = 1;
 cleanup_removed = 0;
 clientdbGC(NULL);
 }
 
 #if SQUID_SNMP
 
 Ip::Address *
 client_entry(Ip::Address *current)
 {
 char key[MAX_IPSTRLEN];
 hash_first(client_table);
 
 if (current) {
 current->toStr(key,MAX_IPSTRLEN);
 while (hash_link *hash = hash_next(client_table)) {
 if (!strcmp(key, hashKeyStr(hash)))
 break;
 }
 }
 
-ClientInfo *c = reinterpret_cast(hash_next(client_table));
+ClientInfo *c = static_cast(hash_next(client_table));
 
 hash_last(client_table);
 
 return c ? >addr : nullptr;
 }
 
 variable_list *
 snmp_meshCtblFn(variable_list * Var, snint * ErrP)
 {
 char key[MAX_IPSTRLEN];
 ClientInfo *c = NULL;
 Ip::Address keyIp;
 
 *ErrP = SNMP_ERR_NOERROR;
 MemBuf tmp;
 debugs(49, 6, HERE << "Current : length=" << Var->name_length << ": " << snmpDebugOid(Var->name, Var->name_length, tmp));
 if (Var->name_length == 16) {
 oid2addr(&(Var->name[12]), keyIp, 4);
 } else if (Var->name_length == 28) {
 oid2addr(&(Var->name[12]), keyIp, 16);
 } else {
 *ErrP = SNMP_ERR_NOSUCHNAME;
 return NULL;
 }
 
 keyIp.toStr(key, sizeof(key));
 debugs(49, 5, HERE << "[" << key << "] requested!");
 c = (ClientInfo *) hash_lookup(client_table, key);
 
 if (c == NULL) {

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] [PATCH] Compilation error after r15057

[Eduard Bagdasaryan]

Hello,

Attaching compilation fix for r15057.


Regards,

Eduard.
Compilation errors fix after r15057.

=== modified file 'src/MessageBucket.h'
--- src/MessageBucket.h	2017-02-19 17:13:27 +
+++ src/MessageBucket.h	2017-02-20 07:20:53 +
@@ -2,40 +2,40 @@
  * Copyright (C) 1996-2017 The Squid Software Foundation and contributors
  *
  * Squid software is distributed under GPLv2+ license and includes
  * contributions from numerous individuals and organizations.
  * Please see the COPYING and CONTRIBUTORS files for details.
  */
 
 #ifndef MESSAGEBUCKET_H
 #define MESSAGEBUCKET_H
 
 #if USE_DELAY_POOLS
 
 #include "BandwidthBucket.h"
 #include "base/RefCount.h"
 #include "comm/forward.h"
 #include "MessageDelayPools.h"
 
 /// Limits Squid-to-client bandwidth for each matching response
 class MessageBucket : public RefCountable, public BandwidthBucket
 {
 MEMPROXY_CLASS(MessageBucket);
 
 public:
 typedef RefCount Pointer;
 
 MessageBucket(const int speed, const int initialLevelPercent, const double sizeLimit, MessageDelayPool::Pointer pool);
 
 /* BandwidthBucket API */
 virtual int quota() override;
 virtual void scheduleWrite(Comm::IoCallback *state) override;
-virtual void reduceBucket(int len);
+virtual void reduceBucket(int len) override;
 
 private:
 MessageDelayPool::Pointer theAggregate;
 };
 
 #endif /* USE_DELAY_POOLS */
 
 #endif
 

=== modified file 'src/client_db.cc'
--- src/client_db.cc	2017-02-19 17:13:27 +
+++ src/client_db.cc	2017-02-20 07:26:00 +
@@ -249,61 +249,61 @@
 
 debugs(1, DBG_CRITICAL, "WARNING: Probable misconfigured neighbor at " << key);
 
 debugs(1, DBG_CRITICAL, "WARNING: " << ND << " of the last " << NR <<
" ICP replies are DENIED");
 
 debugs(1, DBG_CRITICAL, "WARNING: No replies will be sent for the next " <<
CUTOFF_SECONDS << " seconds");
 
 c->cutoff.time = squid_curtime;
 
 c->cutoff.n_req = c->Icp.n_requests;
 
 c->cutoff.n_denied = c->Icp.result_hist[LOG_UDP_DENIED];
 
 return 1;
 }
 
 void
 clientdbDump(StoreEntry * sentry)
 {
 const char *name;
 int icp_total = 0;
 int icp_hits = 0;
 int http_total = 0;
 int http_hits = 0;
 storeAppendPrintf(sentry, "Cache Clients:\n");
 hash_first(client_table);
 
 while (hash_link *hash = hash_next(client_table)) {
-const ClientInfo *c = reinterpret_cast(hash);
+const ClientInfo *c = static_cast(hash);
 storeAppendPrintf(sentry, "Address: %s\n", hashKeyStr(hash));
 if ( (name = fqdncache_gethostbyaddr(c->addr, 0)) ) {
 storeAppendPrintf(sentry, "Name:%s\n", name);
 }
 storeAppendPrintf(sentry, "Currently established connections: %d\n",
   c->n_established);
 storeAppendPrintf(sentry, "ICP  Requests %d\n",
   c->Icp.n_requests);
 
 for (LogTags_ot l = LOG_TAG_NONE; l < LOG_TYPE_MAX; ++l) {
 if (c->Icp.result_hist[l] == 0)
 continue;
 
 icp_total += c->Icp.result_hist[l];
 
 if (LOG_UDP_HIT == l)
 icp_hits += c->Icp.result_hist[l];
 
 storeAppendPrintf(sentry, "%-20.20s %7d %3d%%\n", LogTags(l).c_str(), c->Icp.result_hist[l], Math::intPercent(c->Icp.result_hist[l], c->Icp.n_requests));
 }
 
 storeAppendPrintf(sentry, "HTTP Requests %d\n", c->Http.n_requests);
 
 for (LogTags_ot l = LOG_TAG_NONE; l < LOG_TYPE_MAX; ++l) {
 if (c->Http.result_hist[l] == 0)
 continue;
 
 http_total += c->Http.result_hist[l];
 
 if (LogTags(l).isTcpHit())

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] [PATCH] Annotation value parsing fix

[Eduard Bagdasaryan]

Hello,

This patch fixes two bugs, detected by Coverity:

* Do not leak Note::Value::Value::valueFormat.
* Throw if annotation value parsing failures.


Regards,
Eduard.
Fixed bugs introduced by r15024.

* Do not leak Note::Value::Value::valueFormat.
* Throw if annotation value parsing failures.

Detected by Coverity Scan:
 * CID 1399758:  Error handling issues (CHECKED_RETURN)
 * CID 1399759:  Resource leaks (CTOR_DTOR_LEAK)

=== modified file 'src/Notes.cc'
--- src/Notes.cc	2017-01-31 00:12:18 +
+++ src/Notes.cc	2017-02-13 22:00:50 +
@@ -1,68 +1,74 @@
 /*
  * Copyright (C) 1996-2017 The Squid Software Foundation and contributors
  *
  * Squid software is distributed under GPLv2+ license and includes
  * contributions from numerous individuals and organizations.
  * Please see the COPYING and CONTRIBUTORS files for details.
  */
 
 #include "squid.h"
 #include "AccessLogEntry.h"
 #include "acl/FilledChecklist.h"
 #include "acl/Gadgets.h"
 #include "client_side.h"
 #include "ConfigParser.h"
 #include "globals.h"
 #include "http/Stream.h"
 #include "HttpReply.h"
 #include "HttpRequest.h"
 #include "parser/Tokenizer.h"
 #include "sbuf/StringConvert.h"
 #include "SquidConfig.h"
 #include "Store.h"
 #include "StrList.h"
 
 #include 
 #include 
 
 Note::Value::~Value()
 {
 aclDestroyAclList();
+delete valueFormat;
 }
 
 Note::Value::Value(const char *aVal, const bool quoted, const char *descr, const Method m)
 : aclList(nullptr), valueFormat(nullptr), theValue(aVal), theMethod(m)
 {
 if (quoted) {
 valueFormat = new Format::Format(descr ? descr : "Notes");
-valueFormat->parse(theValue.c_str());
+if (!valueFormat->parse(theValue.c_str())) {
+delete valueFormat;
+SBuf exceptionMsg;
+exceptionMsg.Printf("failed to parse annotation value %s", theValue.c_str());
+throw TexcHere(exceptionMsg.c_str());
+}
 }
 }
 
 const SBuf &
 Note::Value::format(const AccessLogEntryPointer )
 {
 if (al && valueFormat) {
 static MemBuf mb;
 mb.reset();
 valueFormat->assemble(mb, al, 0);
 theFormattedValue.assign(mb.content());
 return theFormattedValue;
 }
 return theValue;
 }
 
 Note::Value::Pointer
 Note::addValue(const char *value, const bool quoted, const char *descr, const Value::Method m)
 {
 values.push_back(new Value(value, quoted, descr, m));
 return values.back();
 }
 
 bool
 Note::match(HttpRequest *request, HttpReply *reply, const AccessLogEntry::Pointer , SBuf )
 {
 ACLFilledChecklist ch(nullptr, request, nullptr);
 ch.reply = reply;
 if (reply)
 HTTPMSGLOCK(ch.reply);

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] VIA creation code duplication

[Eduard Bagdasaryan]

I see that String::append asserts when String is unable to "grow":
String has hardcoded ~64Kb limit for that. It is hardly possible since
most of web servers have header length limit less than this value.
Theoretically a buggy upstream server could generate such huge Via.
However any other header may assert as well, since HttpHeaderEntry
stores its value in String. Why do you think we should care only about
Via header overflow?


Eduard.


On 13.02.2017 00:56, Amos Jeffries wrote:

Thank you. But if we are going to keep the copy+append behaviour we have
to consider the assertion in String::append which goes off if the client
has supplied a large header already. We need to either assemble in the
SBuf, or to check the strVia length before appending the new parts.

Amos

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev


___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] VIA creation code duplication

[Eduard Bagdasaryan]

On 09.02.2017 20:19, Amos Jeffries wrote:

> Since Via is a list header we should be able to just append a new Via
> header to the header list with putStr. No need to use getList, String,
> delById to inject on the end of existing Via string.

Doing so will change the Via generation way currently used. Instead of
generating a single Via header with comma-separated value list Squid
would produce several Via header fields. Though this is equivalent from
the RFC point of view, I would prefer to leave these things "as is" for now.

> Please also take the opportunity to fix a long standing protocol
> violation bug in Via header produced by Squid:
> The version part of the header is only supposed to elide the protocol
> label if the that label would be "HTTP/" (ie. for messages received via
> HTTP and HTTPS).
>
> eg. for ICY replies:
>   Via: ICY/1.1 squid
>
> eg. for FTP gatewayed replies (or relayed requests):
>   Via: FTP/2 squid

Done.

> After those calls that require String are gone please assemble the value
> in an SBuf instead of static char* buffer.

Done.

> Also, for messages where Squid itself generated the message (ie
> Downloader or ESI) there should be no Via header added at all.

Marked as TODO.


Eduard.

Fixed appending Http::HdrType::VIA code duplication.

=== modified file 'src/HttpHeader.cc'
--- src/HttpHeader.cc	2017-01-25 22:29:03 +
+++ src/HttpHeader.cc	2017-02-11 19:30:36 +
@@ -976,60 +976,83 @@
 const char *item;
 int ilen;
 int mlen = strlen(member);
 
 assert(any_registered_header(id));
 
 header = getStrOrList(id);
 String result;
 
 while (strListGetItem(, separator, , , )) {
 if (strncmp(item, member, mlen) == 0 && item[mlen] == '=') {
 result.append(item + mlen + 1, ilen - mlen - 1);
 break;
 }
 }
 
 header.clean();
 return result;
 }
 
 /* test if a field is present */
 int
 HttpHeader::has(Http::HdrType id) const
 {
 assert(any_registered_header(id));
 debugs(55, 9, this << " lookup for " << id);
 return CBIT_TEST(mask, id);
 }
 
 void
+HttpHeader::addVia(const AnyP::ProtocolVersion , const HttpHeader *from)
+{
+// TODO: do not add Via header for messages where Squid itself
+// generated the message (i.e., Downloader or ESI) there should be no Via header added at all.
+
+if (Config.onoff.via) {
+SBuf buf;
+// RFC 7230 section 5.7.1.: protocol-name is omitted when
+// the received protocol is HTTP.
+if (ver.protocol > AnyP::PROTO_NONE && ver.protocol < AnyP::PROTO_UNKNOWN &&
+ver.protocol != AnyP::PROTO_HTTP && ver.protocol != AnyP::PROTO_HTTPS)
+buf.appendf("%s/", AnyP::ProtocolType_str[ver.protocol]);
+buf.appendf("%d.%d %s", ver.major, ver.minor, ThisCache);
+String strVia;
+const HttpHeader *hdr = from ? from : this;
+hdr->getList(Http::HdrType::VIA, );
+strListAdd(, buf.c_str(), ',');
+delById(Http::HdrType::VIA);
+putStr(Http::HdrType::VIA, strVia.termedBuf());
+}
+}
+
+void
 HttpHeader::putInt(Http::HdrType id, int number)
 {
 assert(any_registered_header(id));
 assert(Http::HeaderLookupTable.lookup(id).type == Http::HdrFieldType::ftInt);  /* must be of an appropriate type */
 assert(number >= 0);
 addEntry(new HttpHeaderEntry(id, NULL, xitoa(number)));
 }
 
 void
 HttpHeader::putInt64(Http::HdrType id, int64_t number)
 {
 assert(any_registered_header(id));
 assert(Http::HeaderLookupTable.lookup(id).type == Http::HdrFieldType::ftInt64);/* must be of an appropriate type */
 assert(number >= 0);
 addEntry(new HttpHeaderEntry(id, NULL, xint64toa(number)));
 }
 
 void
 HttpHeader::putTime(Http::HdrType id, time_t htime)
 {
 assert(any_registered_header(id));
 assert(Http::HeaderLookupTable.lookup(id).type == Http::HdrFieldType::ftDate_1123);/* must be of an appropriate type */
 assert(htime >= 0);
 addEntry(new HttpHeaderEntry(id, NULL, mkrfc1123(htime)));
 }
 
 void
 HttpHeader::putStr(Http::HdrType id, const char *str)
 {
 assert(any_registered_header(id));

=== modified file 'src/HttpHeader.h'
--- src/HttpHeader.h	2017-01-01 00:12:22 +
+++ src/HttpHeader.h	2017-02-11 16:53:40 +
@@ -1,41 +1,42 @@
 /*
  * Copyright (C) 1996-2017 The Squid Software Foundation and contributors
  *
  * Squid software is distributed under GPLv2+ license and includes
  * contributions from numerous individuals and organizations.
  * Please see the COPYING and CONTRIBUTORS files for details.
  */
 
 #ifndef SQUID_HTTPHEADER_H
 #define SQUID_HTTPHEADER_H
 
+#include "anyp/ProtocolVersion.h"
 #include "base/LookupTable.h"
 #include "http/RegisteredHeaders.h"
 /* because we pass a spec by value */
 #include "HttpHeaderMask.h"
 #include "mem/forward.h"
 #include "sbuf/forward.h"
 #include "SquidString.h"
 
 #include 
 
 /* class forward declarations */
 class HttpHdrCc;
 class HttpHdrContRange;
 class 

Re: [squid-dev] [PATCH] Case-insensitive URI schemes

[Eduard Bagdasaryan]

Checked that it is ok to move AnyP::UriScheme::Init() as
you suggested. Re-attached the patch (v5 r15037).


Eduard.

On 02.02.2017 22:12, Alex Rousskov wrote:

> We should avoid this code duplication [...]
> However, please check whether we can move the
> call up, to place it above storeFsInit().
> Both of the above changes can be done during commit.

Fix URI scheme case-sensitivity treatment broken since r14802.

A parsed value for the AnyP::UriScheme image constructor parameter was
stored without toLower() canonicalization for known protocols (e.g.,
after successful "HTTP://EXAMPLE.COM/" parsing in urlParseFinish).
Without that canonicalization step, Squid violated various HTTP caching
rules related to URI comparison (and served fewer hits) when dealing
with absolute URLs containing non-lowercase HTTP scheme.

According to my limited tests, URL-based ACLs are not affected by this
bug, but I have not investigated how URL-based ACL code differs from
caching code when it comes to stored URL access and whether some ACLs
are actually affected in some environments.

=== modified file 'src/anyp/UriScheme.cc'
--- src/anyp/UriScheme.cc	2017-01-01 00:12:22 +
+++ src/anyp/UriScheme.cc	2017-02-07 14:57:49 +
@@ -1,61 +1,78 @@
 /*
  * Copyright (C) 1996-2017 The Squid Software Foundation and contributors
  *
  * Squid software is distributed under GPLv2+ license and includes
  * contributions from numerous individuals and organizations.
  * Please see the COPYING and CONTRIBUTORS files for details.
  */
 
 /* DEBUG: section 23URL Scheme parsing */
 
 #include "squid.h"
 #include "anyp/UriScheme.h"
 
+AnyP::UriScheme::LowercaseSchemeNames AnyP::UriScheme::LowercaseSchemeNames_;
+
 AnyP::UriScheme::UriScheme(AnyP::ProtocolType const aScheme, const char *img) :
 theScheme_(aScheme)
 {
-if (img)
-// image could be provided explicitly (case-sensitive)
-image_ = img;
-
-else if (theScheme_ == AnyP::PROTO_UNKNOWN)
-// image could be actually unknown and not provided
-image_ = "(unknown)";
-
-else if (theScheme_ > AnyP::PROTO_NONE && theScheme_ < AnyP::PROTO_MAX) {
-// image could be implied by a registered transfer protocol
-// which use upper-case labels, so down-case for scheme image
-image_ = AnyP::ProtocolType_str[theScheme_];
-image_.toLower();
+// RFC 3986 section 3.1: schemes are case-insensitive.
+
+// To improve diagnostic, remember exactly how an unsupported scheme looks like.
+// XXX: Object users may rely on toLower() canonicalization that we refuse to provide.
+if (img && theScheme_ == AnyP::PROTO_UNKNOWN)
+ image_ = img;
+
+// XXX: A broken caller supplies an image of an absent scheme?
+// XXX: We assume that the caller is using a lower-case image.
+else if (img && theScheme_ == AnyP::PROTO_NONE)
+ image_ = img;
+
+else if (theScheme_ > AnyP::PROTO_NONE && theScheme_ < AnyP::PROTO_MAX)
+image_ = LowercaseSchemeNames_.at(theScheme_);
+// else, the image remains empty (e.g., "://example.com/")
+// hopefully, theScheme_ is PROTO_NONE here
+}
+
+void
+AnyP::UriScheme::Init()
+{
+if (LowercaseSchemeNames_.empty()) {
+LowercaseSchemeNames_.reserve(sizeof(SBuf) * AnyP::PROTO_MAX);
+// TODO: use base/EnumIterator.h if possible
+for (int i = AnyP::PROTO_NONE; i < AnyP::PROTO_MAX; ++i) {
+SBuf image(ProtocolType_str[i]);
+image.toLower();
+LowercaseSchemeNames_.emplace_back(image);
+}
 }
-// else, image is an empty string ("://example.com/")
 }
 
 unsigned short
 AnyP::UriScheme::defaultPort() const
 {
 switch (theScheme_) {
 
 case AnyP::PROTO_HTTP:
 return 80;
 
 case AnyP::PROTO_HTTPS:
 return 443;
 
 case AnyP::PROTO_FTP:
 return 21;
 
 case AnyP::PROTO_COAP:
 case AnyP::PROTO_COAPS:
 // coaps:// default is TBA as of draft-ietf-core-coap-08.
 // Assuming IANA policy of allocating same port for base and TLS protocol versions will occur.
 return 5683;
 
 case AnyP::PROTO_GOPHER:
 return 70;
 
 case AnyP::PROTO_WAIS:
 return 210;
 
 case AnyP::PROTO_CACHE_OBJECT:
 return CACHE_HTTP_PORT;

=== modified file 'src/anyp/UriScheme.h'
--- src/anyp/UriScheme.h	2017-01-01 00:12:22 +
+++ src/anyp/UriScheme.h	2017-02-07 14:59:52 +
@@ -1,68 +1,78 @@
 /*
  * Copyright (C) 1996-2017 The Squid Software Foundation and contributors
  *
  * Squid software is distributed under GPLv2+ license and includes
  * contributions from numerous individuals and organizations.
  * Please see the COPYING and CONTRIBUTORS files for details.
  */
 
 #ifndef SQUID_ANYP_URISCHEME_H
 #define SQUID_ANYP_URISCHEME_H
 
 #include "anyp/ProtocolType.h"
 #include "sbuf/SBuf.h"
 
 #include 
 
 namespace AnyP
 {
 
 /** This class represents a URI Scheme such as http:// https://, wais://, urn: etc.
  * It does not 

[squid-dev] [PATCH] VIA creation code duplication

[Eduard Bagdasaryan]

Hello,

This patch fixes VIA appending code duplication, moving common
code into a separate method.


Regards,
Eduard.
Fixed appending Http::HdrType::VIA code duplication.

=== modified file 'src/HttpHeader.cc'
--- src/HttpHeader.cc	2017-01-25 22:29:03 +
+++ src/HttpHeader.cc	2017-02-02 14:47:12 +
@@ -976,60 +976,78 @@
 const char *item;
 int ilen;
 int mlen = strlen(member);
 
 assert(any_registered_header(id));
 
 header = getStrOrList(id);
 String result;
 
 while (strListGetItem(, separator, , , )) {
 if (strncmp(item, member, mlen) == 0 && item[mlen] == '=') {
 result.append(item + mlen + 1, ilen - mlen - 1);
 break;
 }
 }
 
 header.clean();
 return result;
 }
 
 /* test if a field is present */
 int
 HttpHeader::has(Http::HdrType id) const
 {
 assert(any_registered_header(id));
 debugs(55, 9, this << " lookup for " << id);
 return CBIT_TEST(mask, id);
 }
 
 void
+HttpHeader::addVia(const AnyP::ProtocolVersion , const HttpHeader *from)
+{
+if (Config.onoff.via) {
+LOCAL_ARRAY(char, bbuf, MAX_URL + 32);
+String strVia;
+const HttpHeader *hdr = from ? from : this;
+hdr->getList(Http::HdrType::VIA, );
+snprintf(bbuf, MAX_URL + 32, "%d.%d %s",
+ver.major,
+ver.minor,
+ThisCache);
+strListAdd(, bbuf, ',');
+delById(Http::HdrType::VIA);
+putStr(Http::HdrType::VIA, strVia.termedBuf());
+}
+}
+
+void
 HttpHeader::putInt(Http::HdrType id, int number)
 {
 assert(any_registered_header(id));
 assert(Http::HeaderLookupTable.lookup(id).type == Http::HdrFieldType::ftInt);  /* must be of an appropriate type */
 assert(number >= 0);
 addEntry(new HttpHeaderEntry(id, NULL, xitoa(number)));
 }
 
 void
 HttpHeader::putInt64(Http::HdrType id, int64_t number)
 {
 assert(any_registered_header(id));
 assert(Http::HeaderLookupTable.lookup(id).type == Http::HdrFieldType::ftInt64);/* must be of an appropriate type */
 assert(number >= 0);
 addEntry(new HttpHeaderEntry(id, NULL, xint64toa(number)));
 }
 
 void
 HttpHeader::putTime(Http::HdrType id, time_t htime)
 {
 assert(any_registered_header(id));
 assert(Http::HeaderLookupTable.lookup(id).type == Http::HdrFieldType::ftDate_1123);/* must be of an appropriate type */
 assert(htime >= 0);
 addEntry(new HttpHeaderEntry(id, NULL, mkrfc1123(htime)));
 }
 
 void
 HttpHeader::putStr(Http::HdrType id, const char *str)
 {
 assert(any_registered_header(id));

=== modified file 'src/HttpHeader.h'
--- src/HttpHeader.h	2017-01-01 00:12:22 +
+++ src/HttpHeader.h	2017-02-02 14:27:48 +
@@ -1,41 +1,42 @@
 /*
  * Copyright (C) 1996-2017 The Squid Software Foundation and contributors
  *
  * Squid software is distributed under GPLv2+ license and includes
  * contributions from numerous individuals and organizations.
  * Please see the COPYING and CONTRIBUTORS files for details.
  */
 
 #ifndef SQUID_HTTPHEADER_H
 #define SQUID_HTTPHEADER_H
 
+#include "anyp/ProtocolVersion.h"
 #include "base/LookupTable.h"
 #include "http/RegisteredHeaders.h"
 /* because we pass a spec by value */
 #include "HttpHeaderMask.h"
 #include "mem/forward.h"
 #include "sbuf/forward.h"
 #include "SquidString.h"
 
 #include 
 
 /* class forward declarations */
 class HttpHdrCc;
 class HttpHdrContRange;
 class HttpHdrRange;
 class HttpHdrSc;
 class Packable;
 
 /** Possible owners of http header */
 typedef enum {
 hoNone =0,
 #if USE_HTCP
 hoHtcpReply,
 #endif
 hoRequest,
 hoReply,
 #if USE_OPENSSL
 hoErrorDetail,
 #endif
 hoEnd
 } http_hdr_owner_type;
@@ -86,60 +87,63 @@
 int parse(const char *header_start, size_t len);
 /// Parses headers stored in a buffer.
 /// \returns 1 and sets hdr_sz on success
 /// \returns 0 when needs more data
 /// \returns -1 on error
 int parse(const char *buf, size_t buf_len, bool atEnd, size_t _sz);
 void packInto(Packable * p, bool mask_sensitive_info=false) const;
 HttpHeaderEntry *getEntry(HttpHeaderPos * pos) const;
 HttpHeaderEntry *findEntry(Http::HdrType id) const;
 int delByName(const char *name);
 int delById(Http::HdrType id);
 void delAt(HttpHeaderPos pos, int _deleted);
 void refreshMask();
 void addEntry(HttpHeaderEntry * e);
 void insertEntry(HttpHeaderEntry * e);
 String getList(Http::HdrType id) const;
 bool getList(Http::HdrType id, String *s) const;
 bool conflictingContentLength() const { return conflictingContentLength_; }
 String getStrOrList(Http::HdrType id) const;
 String getByName(const SBuf ) const;
 String getByName(const char *name) const;
 String getById(Http::HdrType id) const;
 /// sets value and returns true iff a [possibly empty] field identified by id is there
 bool getByIdIfPresent(Http::HdrType id, String ) const;
 /// sets value and returns true 

Re: [squid-dev] [PATCH] Case-insensitive URI schemes

[Eduard Bagdasaryan]

I applied your polishing suggestions to my latest patch
re-attached it.


Eduard.


On 02.02.2017 11:33, Amos Jeffries wrote:

On 1/02/2017 2:18 a.m., Eduard Bagdasaryan wrote:

Optimized with static array as you suggested and
re-attached the patch.


Thank you, looks like this one removes most of the performance regression.

Just some polishing for src/anyp/UriScheme.cc ;


in AnyP::UriScheme::UriScheme

* this TODO seems reasonable and simple enough, please do it:

  +// the caller knows nothing about the scheme
+// TODO: Avoid special casing by storing this string in the
generated ProtocolType_str?
  else if (theScheme_ == AnyP::PROTO_UNKNOWN)
-// image could be actually unknown and not provided
  image_ = "(unknown)";


in AnyP::UriScheme::LowercaseScheme:

* please use the Squid coding style of parameters being on a line before
the function name.

* please do use emplace_back instead of push_back. Simple as it is the
SBuf is not a pointer.

* please add a TODO note about making the ProtocolType enum use
base/EnumIterator.h instead of an int for-loop.


+1. I don't think this needs another review, just the polish.

Amos

Fix URI scheme case-sensitivity treatment broken since r14802.

A parsed value for the AnyP::UriScheme image constructor parameter was
stored without toLower() canonicalization for known protocols (e.g.,
after successful "HTTP://EXAMPLE.COM/" parsing in urlParseFinish).
Without that canonicalization step, Squid violated various HTTP caching
rules related to URI comparison (and served fewer hits) when dealing
with absolute URLs containing non-lowercase HTTP scheme.

According to my limited tests, URL-based ACLs are not affected by this
bug, but I have not investigated how URL-based ACL code differs from
caching code when it comes to stored URL access and whether some ACLs
are actually affected in some environments.

=== modified file 'src/anyp/UriScheme.cc'
--- src/anyp/UriScheme.cc	2017-01-01 00:12:22 +
+++ src/anyp/UriScheme.cc	2017-02-02 10:37:44 +
@@ -1,61 +1,78 @@
 /*
  * Copyright (C) 1996-2017 The Squid Software Foundation and contributors
  *
  * Squid software is distributed under GPLv2+ license and includes
  * contributions from numerous individuals and organizations.
  * Please see the COPYING and CONTRIBUTORS files for details.
  */
 
 /* DEBUG: section 23URL Scheme parsing */
 
 #include "squid.h"
 #include "anyp/UriScheme.h"
 
+std::vector AnyP::UriScheme::LowercaseSchemeNames;
+
 AnyP::UriScheme::UriScheme(AnyP::ProtocolType const aScheme, const char *img) :
 theScheme_(aScheme)
 {
-if (img)
-// image could be provided explicitly (case-sensitive)
-image_ = img;
-
-else if (theScheme_ == AnyP::PROTO_UNKNOWN)
-// image could be actually unknown and not provided
-image_ = "(unknown)";
-
-else if (theScheme_ > AnyP::PROTO_NONE && theScheme_ < AnyP::PROTO_MAX) {
-// image could be implied by a registered transfer protocol
-// which use upper-case labels, so down-case for scheme image
-image_ = AnyP::ProtocolType_str[theScheme_];
-image_.toLower();
+// RFC 3986 section 3.1: schemes are case-insensitive.
+
+// To improve diagnostic, remember exactly how an unsupported scheme looks like.
+// XXX: Object users may rely on toLower() canonicalization that we refuse to provide.
+if (img && theScheme_ == AnyP::PROTO_UNKNOWN)
+ image_ = img;
+
+// XXX: A broken caller supplies an image of an absent scheme?
+// XXX: We assume that the caller is using a lower-case image.
+else if (img && theScheme_ == AnyP::PROTO_NONE)
+ image_ = img;
+
+else if (theScheme_ > AnyP::PROTO_NONE && theScheme_ < AnyP::PROTO_MAX)
+image_ = LowercaseSchemeNames.at(theScheme_);
+// else, the image remains empty (e.g., "://example.com/")
+// hopefully, theScheme_ is PROTO_NONE here
+}
+
+void
+AnyP::UriScheme::Init()
+{
+if (LowercaseSchemeNames.empty()) {
+LowercaseSchemeNames.reserve(sizeof(SBuf) * AnyP::PROTO_MAX);
+// TODO: use base/EnumIterator.h if possible
+for (int i = AnyP::PROTO_NONE; i < AnyP::PROTO_MAX; ++i) {
+SBuf image(ProtocolType_str[i]);
+image.toLower();
+LowercaseSchemeNames.emplace_back(image);
+}
 }
-// else, image is an empty string ("://example.com/")
 }
 
 unsigned short
 AnyP::UriScheme::defaultPort() const
 {
 switch (theScheme_) {
 
 case AnyP::PROTO_HTTP:
 return 80;
 
 case AnyP::PROTO_HTTPS:
 return 443;
 
 case AnyP::PROTO_FTP:
 return 21;
 
 case AnyP::PROTO_COAP:
 case AnyP::PROTO_COAPS:
 // coaps:// default is TBA as of draft-ietf-core-coap-08.
 // Assuming IANA policy of allocating same port for base and TLS protocol versio

Re: [squid-dev] [PATCH] Case-insensitive URI schemes

[Eduard Bagdasaryan]

This is a bit improved version of previous patch:
fill static schemes array at configuration phase.


Eduard.


On 31.01.2017 16:18, Eduard Bagdasaryan wrote:

Optimized with static array as you suggested and
re-attached the patch.


Eduard.


On 30.01.2017 19:24, Alex Rousskov wrote:

On 01/29/2017 07:10 AM, Amos Jeffries wrote:


I'm thinking the quick-and-dirty way is to just lowercase the 'proto'
variable in url.cc urlParse() function. Doing that in the for-loop 
where

it is copied from 'src' would be easiest.
  - it breaks the case preservation on unknown schemes a litte bit. But
since they are supposed to be insensitive anyway the harm is minimal.


The UriScheme constructor is broken. No quick-and-dirty fix in some
constructor caller will address that problem. Eduard's patch fixes that
problem the right way. Unfortunately, the fixed constructor is
expensive. Fortunately, it is easy to optimize it, addressing another
old problem (that the patch has documented but did not address). With
that optimization, there will be no motivation for quick-and-dirty
workarounds.

Eduard, please create and use a UriScheme::SchemeImages or similar
static array with lowercase versions of ProtocolType_str.


Thank you,

Alex.




Fix URI scheme case-sensitivity treatment broken since r14802.

A parsed value for the AnyP::UriScheme image constructor parameter was
stored without toLower() canonicalization for known protocols (e.g.,
after successful "HTTP://EXAMPLE.COM/" parsing in urlParseFinish).
Without that canonicalization step, Squid violated various HTTP caching
rules related to URI comparison (and served fewer hits) when dealing
with absolute URLs containing non-lowercase HTTP scheme.

According to my limited tests, URL-based ACLs are not affected by this
bug, but I have not investigated how URL-based ACL code differs from
caching code when it comes to stored URL access and whether some ACLs
are actually affected in some environments.

=== modified file 'src/anyp/UriScheme.cc'
--- src/anyp/UriScheme.cc	2017-01-01 00:12:22 +
+++ src/anyp/UriScheme.cc	2017-02-01 14:26:36 +
@@ -1,61 +1,81 @@
 /*
  * Copyright (C) 1996-2017 The Squid Software Foundation and contributors
  *
  * Squid software is distributed under GPLv2+ license and includes
  * contributions from numerous individuals and organizations.
  * Please see the COPYING and CONTRIBUTORS files for details.
  */
 
 /* DEBUG: section 23URL Scheme parsing */
 
 #include "squid.h"
 #include "anyp/UriScheme.h"
 
+std::vector AnyP::UriScheme::LowercaseSchemeNames;
+
 AnyP::UriScheme::UriScheme(AnyP::ProtocolType const aScheme, const char *img) :
 theScheme_(aScheme)
 {
-if (img)
-// image could be provided explicitly (case-sensitive)
-image_ = img;
-
+// RFC 3986 section 3.1: schemes are case-insensitive.
+
+// To improve diagnostic, remember exactly how an unsupported scheme looks like.
+// XXX: Object users may rely on toLower() canonicalization that we refuse to provide.
+if (img && theScheme_ == AnyP::PROTO_UNKNOWN)
+ image_ = img;
+
+// XXX: A broken caller supplies an image of an absent scheme?
+// XXX: We assume that the caller is using a lower-case image.
+else if (img && theScheme_ == AnyP::PROTO_NONE)
+ image_ = img;
+
+// the caller knows nothing about the scheme
+// TODO: Avoid special casing by storing this string in the generated ProtocolType_str?
 else if (theScheme_ == AnyP::PROTO_UNKNOWN)
-// image could be actually unknown and not provided
 image_ = "(unknown)";
 
-else if (theScheme_ > AnyP::PROTO_NONE && theScheme_ < AnyP::PROTO_MAX) {
-// image could be implied by a registered transfer protocol
-// which use upper-case labels, so down-case for scheme image
-image_ = AnyP::ProtocolType_str[theScheme_];
-image_.toLower();
+else if (theScheme_ > AnyP::PROTO_NONE && theScheme_ < AnyP::PROTO_MAX)
+image_ = LowercaseSchemeNames.at(theScheme_);
+// else, the image remains empty (e.g., "://example.com/")
+// hopefully, theScheme_ is PROTO_NONE here
+}
+
+void
+AnyP::UriScheme::Init()
+{
+if (LowercaseSchemeNames.empty()) {
+for (int i = AnyP::PROTO_NONE; i < AnyP::PROTO_MAX; ++i) {
+SBuf image(ProtocolType_str[i]);
+image.toLower();
+LowercaseSchemeNames.push_back(image);
+}
 }
-// else, image is an empty string ("://example.com/")
 }
 
 unsigned short
 AnyP::UriScheme::defaultPort() const
 {
 switch (theScheme_) {
 
 case AnyP::PROTO_HTTP:
 return 80;
 
 case AnyP::PROTO_HTTPS:
 return 443;
 
 case AnyP::PROTO_FTP:
 return 21;
 
 case AnyP::PROTO_COAP:
 case AnyP::PROTO_COAPS:
 // coaps:// default is TBA as of draft-ietf-core-coap-08.
 // Assuming IANA poli

Re: [squid-dev] [PATCH] Case-insensitive URI schemes

[Eduard Bagdasaryan]

Optimized with static array as you suggested and
re-attached the patch.


Eduard.


On 30.01.2017 19:24, Alex Rousskov wrote:

On 01/29/2017 07:10 AM, Amos Jeffries wrote:


I'm thinking the quick-and-dirty way is to just lowercase the 'proto'
variable in url.cc urlParse() function. Doing that in the for-loop where
it is copied from 'src' would be easiest.
  - it breaks the case preservation on unknown schemes a litte bit. But
since they are supposed to be insensitive anyway the harm is minimal.


The UriScheme constructor is broken. No quick-and-dirty fix in some
constructor caller will address that problem. Eduard's patch fixes that
problem the right way. Unfortunately, the fixed constructor is
expensive. Fortunately, it is easy to optimize it, addressing another
old problem (that the patch has documented but did not address). With
that optimization, there will be no motivation for quick-and-dirty
workarounds.

Eduard, please create and use a UriScheme::SchemeImages or similar
static array with lowercase versions of ProtocolType_str.


Thank you,

Alex.


Fix URI scheme case-sensitivity treatment broken since r14802.

A parsed value for the AnyP::UriScheme image constructor parameter was
stored without toLower() canonicalization for known protocols (e.g.,
after successful "HTTP://EXAMPLE.COM/" parsing in urlParseFinish).
Without that canonicalization step, Squid violated various HTTP caching
rules related to URI comparison (and served fewer hits) when dealing
with absolute URLs containing non-lowercase HTTP scheme.

According to my limited tests, URL-based ACLs are not affected by this
bug, but I have not investigated how URL-based ACL code differs from
caching code when it comes to stored URL access and whether some ACLs
are actually affected in some environments.

=== modified file 'src/anyp/UriScheme.cc'
--- src/anyp/UriScheme.cc	2017-01-01 00:12:22 +
+++ src/anyp/UriScheme.cc	2017-01-31 12:49:52 +
@@ -1,61 +1,81 @@
 /*
  * Copyright (C) 1996-2017 The Squid Software Foundation and contributors
  *
  * Squid software is distributed under GPLv2+ license and includes
  * contributions from numerous individuals and organizations.
  * Please see the COPYING and CONTRIBUTORS files for details.
  */
 
 /* DEBUG: section 23URL Scheme parsing */
 
 #include "squid.h"
 #include "anyp/UriScheme.h"
 
+std::vector AnyP::UriScheme::LowercaseSchemeNames;
+
 AnyP::UriScheme::UriScheme(AnyP::ProtocolType const aScheme, const char *img) :
 theScheme_(aScheme)
 {
-if (img)
-// image could be provided explicitly (case-sensitive)
-image_ = img;
-
+// RFC 3986 section 3.1: schemes are case-insensitive.
+
+// To improve diagnostic, remember exactly how an unsupported scheme looks like.
+// XXX: Object users may rely on toLower() canonicalization that we refuse to provide.
+if (img && theScheme_ == AnyP::PROTO_UNKNOWN)
+ image_ = img;
+
+// XXX: A broken caller supplies an image of an absent scheme?
+// XXX: We assume that the caller is using a lower-case image.
+else if (img && theScheme_ == AnyP::PROTO_NONE)
+ image_ = img;
+
+// the caller knows nothing about the scheme
+// TODO: Avoid special casing by storing this string in the generated ProtocolType_str?
 else if (theScheme_ == AnyP::PROTO_UNKNOWN)
-// image could be actually unknown and not provided
 image_ = "(unknown)";
 
-else if (theScheme_ > AnyP::PROTO_NONE && theScheme_ < AnyP::PROTO_MAX) {
-// image could be implied by a registered transfer protocol
-// which use upper-case labels, so down-case for scheme image
-image_ = AnyP::ProtocolType_str[theScheme_];
-image_.toLower();
+else if (theScheme_ > AnyP::PROTO_NONE && theScheme_ < AnyP::PROTO_MAX)
+image_ = LowercaseScheme(theScheme_);
+// else, the image remains empty (e.g., "://example.com/")
+// hopefully, theScheme_ is PROTO_NONE here
+}
+
+const SBuf ::UriScheme::LowercaseScheme(const AnyP::ProtocolType protoType)
+{
+if (LowercaseSchemeNames.empty()) {
+for (int i = AnyP::PROTO_NONE; i < AnyP::PROTO_MAX; ++i) {
+SBuf image(ProtocolType_str[i]);
+image.toLower();
+LowercaseSchemeNames.push_back(image);
+}
 }
-// else, image is an empty string ("://example.com/")
+return LowercaseSchemeNames.at(protoType);
 }
 
 unsigned short
 AnyP::UriScheme::defaultPort() const
 {
 switch (theScheme_) {
 
 case AnyP::PROTO_HTTP:
 return 80;
 
 case AnyP::PROTO_HTTPS:
 return 443;
 
 case AnyP::PROTO_FTP:
 return 21;
 
 case AnyP::PROTO_COAP:
 case AnyP::PROTO_COAPS:
 // coaps:// default is TBA as of draft-ietf-core-coap-08.
 // Assuming IANA policy of allocating same port for base and TLS protocol versions will occur.
 return 5683;
 
 case AnyP::PROTO_GOPHER:
 return 70;
 
 case AnyP::PROTO_WAIS:
 

Re: [squid-dev] [PATCH] Response delay pools

[Eduard Bagdasaryan]

On 22.01.2017 22:52, Amos Jeffries wrote:

> as I understood it the existing delay pools design is that multiple
> pools can apply to traffic. In which case on each write() attempt the
> bucket with smallest available amount determins the write size and all
> buckets have the actually consumed amount removed.

I assume that you are mixing up "pools" and "buckets". A pool may have
several "cascading" buckets. The existing server pools have several
buckets for classes > 1.  We can configure several server delay pools,
but only one first 'matched' pool is assigned to the transaction. Then
the "smallest" bucket of the matched pool determines the final speed
limiting (as you correctly noted).  Following this logic, old "client"
and new "response" pools should be independent (as if they would
represent different client pool classes), i.e., finally we
should select only one "first matched" client-side delay pool. Within
this selected pool, we select bucket with the smallest level (individual
or aggregate) similarly as we do for server pools of classes 2
or 3). However we do not do this selection if old client pool matched,
because it has the only one bucket.

> ClientDelayConfig::parsePoolCount() please use emplace_back instead of
> push_back.

AFAIK, emplace_back() would have a benefit over push_back if there was
an object itself (thus avoiding copying the object). In this case a
pointer is inserted so the benefit is negligible. I would prefer using
old push_back() here.


> is there a specific reason this has to be a list of raw-pointers
> instead of a list of objects like it was?

This change was a consequence of another fix: ClientDelayPool::access member
was not destroyed on shutdown causing "indirectly lost" Valgrind error 
messages.
ClientDelayPool class does not follow copy semantics(and probably should 
not)
due to acl_access pointer, requiring by stl vector. So I had to use 
pointers instead.

However my latest checks with test-builds.sh showed another linker problems,
because SquidConfig got unnecessary dependency on ClientDelayConfig
destructor. Finally I had to re-work ClientDelayConfig module,
introducing new ClientDelayPools class (a storage for ClientDelayPool
objects) and making ClientDelayPool ref-countable.

>  which brings up a design contradiction: "why are there required
> 'options'?
> so the answer is that they should not be required, the value -1 in
> pools just means "no limit" or "infinity".
> - the for-loop checking for missing options should be removed and use
> of the "none" magic word as value for these options should be supported.

I agree that we should not require using all response_delay_pool
options.  For example, it would be useful to allow only "individual" or
"aggregate" options, e.g.:

#no aggregate limiting
response_delay_pool slowPool1  individual-restore=5 \
  individual-maximum=10

#no individual limiting, split aggregate bandwidth among clients
response_delay_pool slowPool2  aggregate-restore=10  \
  aggregate-maximum=20

No need for "none" world: just remove unneeded options (or specify -1
value). Using approach the existing server delay pools
adhere to ("none" for speed/maximum pair), we should not use "restore"
option without "maximum" for both "aggregate" and "individual" cases.
BTW, it is possible to specify "no limitation" at all for a pool
(i.e., no individual and aggregate limitation). The existing server
pools work similarly, allowing to specify "none" for all buckets
(without any warning message).

> The limitation configured with initial_fill_level= was previously
> configured through directive client_delay_initial_bucket_level.

"client_delay_initial_bucket_level" relates to only old client delay
pools.  I don't think we should reuse it in the new response pools: one
may need to configure both pools with different initial level parameter.


Thanks for the detailed review. I tried to address all other remarks,
renamed parameters according to the suggested terminology,
merged with latest v5 r15027 and re-attached the patch.


Eduard.

Added response delay pools feature for Squid-to-client speed limiting.

The feature restricts Squid-to-client bandwidth only.  It applies to
both cache hits and misses.

  * Rationale *

  This may be useful for specific response(s) bandwidth limiting.
  There are situations when doing this is hardly possible
  (or impossible) by means of netfilter/iptables operating with
  TCP/IP packets and IP addresses information for filtering. In other
  words, sometimes it is problematic to 'extract' a single response from
  TCP/IP data flow at system level. For example, a single Squid-to-client
  TCP connection can transmit multiple responses (persistent connections,
  pipelining or HTTP/2 connection multiplexing) or be encrypted
  (HTTPS proxy mode).

  * Description *

  When Squid starts delivering the final HTTP response to a client,
  Squid checks 

[squid-dev] [PATCH] Response delay pools

[Eduard Bagdasaryan]

Hello,

This patch introduces a new "response_delay_pools" feature. I have
posted it(with detailed description) recently to the thread for
preliminary review with "preview" flag.  This patch conforms to latest
v5 (r15011) and also has some problems fixed:

* MessageBucket::theAggregate pointer became invalid during
  reconfiguration.

* Response delay pools got stuck after reconfiguration, consuming
  no more data.

* A performance improvement: MessageDelayPools::Update() was called
  every second(via eventAdd) consuming CPU cycles even when no
  connections were served.


Regards,
Eduard.

Added response delay pools feature for Squid-to-client speed limiting.

The feature restricts Squid-to-client bandwidth only.  It applies to
both cache hits and misses.

When Squid starts delivering the final HTTP response to a client, Squid
checks response_delay_pool_access rules (supporting fast ACLs only), in
the order they were declared. The first rule with a matching ACL wins.
If (and only if) an "allow" rule won, Squid assigns the response to the
corresponding named delay pool.

If a response is assigned to a delay pool, the response becomes subject
to the configured bucket and aggregate bandwidth limits of that pool,
similar to the current "class 2" server-side delay pools, but with a
brand new, dedicated "individual" filled bucket assigned to the
matched response.

The new feature serves the same purpose as the existing client-side
pools: both features limit Squid-to-client bandwidth. Their common
interface was placed into a new base BandwidthBucket class.  The
difference is that client-side pools do not aggregate clients and always
use one bucket per client IP. It is possible that a response becomes a
subject of both these pools. In such situations only matched response
delay pool will be used for Squid-to-client speed limiting.

The accurate SMP support (with the aggregate bucket shared among
workers) is outside this patch scope. In SMP configurations,
Squid should automatically divide the aggregate_speed_limit and
max_aggregate_size values among the configured number of Squid
workers. 

=== added file 'src/BandwidthBucket.cc'
--- src/BandwidthBucket.cc	1970-01-01 00:00:00 +
+++ src/BandwidthBucket.cc	2016-10-26 14:09:11 +
@@ -0,0 +1,111 @@
+/*
+ * Copyright (C) 1996-2016 The Squid Software Foundation and contributors
+ *
+ * Squid software is distributed under GPLv2+ license and includes
+ * contributions from numerous individuals and organizations.
+ * Please see the COPYING and CONTRIBUTORS files for details.
+ */
+
+#include "squid.h"
+
+#if USE_DELAY_POOLS
+
+#include "BandwidthBucket.h"
+#include "ClientInfo.h"
+#include "comm/Connection.h"
+#include "Debug.h"
+#include "fde.h"
+
+extern double current_dtime;
+
+BandwidthBucket::BandwidthBucket(const int aWriteSpeedLimit, const double anInitialBurst,
+ const double aHighWatermark) :
+bucketSize(anInitialBurst),
+selectWaiting(false),
+writeSpeedLimit(aWriteSpeedLimit),
+bucketSizeLimit(aHighWatermark)
+{
+getCurrentTime();
+/* put current time to have something sensible here */
+prevTime = current_dtime;
+}
+
+void
+BandwidthBucket::refillBucket()
+{
+// all these times are in seconds, with double precision
+const double currTime = current_dtime;
+const double timePassed = currTime - prevTime;
+
+// Calculate allowance for the time passed. Use double to avoid
+// accumulating rounding errors for small intervals. For example, always
+// adding 1 byte instead of 1.4 results in 29% bandwidth allocation error.
+const double gain = timePassed * writeSpeedLimit;
+
+// XXX: Decide whether to add 'hash' field like ClientInfo::hash
+//  debugs(77,5, HERE << currTime << " clt" << (const char*)hash.key << ": " <<
+// bucketSize << " + (" << timePassed << " * " << writeSpeedLimit <<
+// " = " << gain << ')');
+
+// to further combat error accumulation during micro updates,
+// quit before updating time if we cannot add at least one byte
+if (gain < 1.0)
+return;
+
+prevTime = currTime;
+
+// for "first" connections, drain initial fat before refilling but keep
+// updating prevTime to avoid bursts after the fat is gone
+if (bucketSize > bucketSizeLimit) {
+debugs(77,4, HERE << "not refilling while draining initial fat");
+return;
+}
+
+bucketSize += gain;
+
+// obey quota limits
+if (bucketSize > bucketSizeLimit)
+bucketSize = bucketSizeLimit;
+}
+
+bool
+BandwidthBucket::applyQuota(int , Comm::IoCallback *state)
+{
+const int q = quota();
+if (!q)
+return false;
+const int nleft_corrected = min(nleft, q);
+if (nleft != nleft_corrected) {
+debugs(77, 5, state->conn << " writes only " <<
+   nleft_corrected << " out of " << nleft);
+nleft = nleft_corrected;
+}
+return true;
+}
+
+void

Re: [squid-dev] [PATCH] Memory overlap Valgrind-reported errors

[Eduard Bagdasaryan]

Thank you.
Also I believe the same patch could be applied to v4 and v3.5
(with a little adjustment). Are you planning to do this?


Eduard.

On 15.01.2017 20:37, Amos Jeffries wrote:

On 16/01/2017 5:15 a.m., Eduard Bagdasaryan wrote:

Fixed "Source and destination overlap in memcpy" Valgrind errors.

Before this patch, source and destination arguments in
log_quoted_string() could point to the same static memory area, causing
multiple Valgrind-reported errors. Fixed by creating another buffer for
storing the quoted-processed output string. The buffer size should be
enough to hold all processed escape sequences.


Ouch. Thanks for finding this.

+1 and Applied to v5 as rev.15010

Amos

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev


___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] [PATCH] Memory overlap Valgrind-reported errors

[Eduard Bagdasaryan]

Fixed "Source and destination overlap in memcpy" Valgrind errors.

Before this patch, source and destination arguments in
log_quoted_string() could point to the same static memory area, causing
multiple Valgrind-reported errors. Fixed by creating another buffer for
storing the quoted-processed output string. The buffer size should be
enough to hold all processed escape sequences.


Regards,
Eduard.

Fixed "Source and destination overlap in memcpy" Valgrind errors.

Before this patch, source and destination arguments in
log_quoted_string() could point to the same static memory area, causing
multiple Valgrind-reported errors. Fixed by creating another buffer to
store quoted-processed output string.

=== modified file 'src/format/Format.cc'
--- src/format/Format.cc	2017-01-01 00:12:22 +
+++ src/format/Format.cc	2017-01-15 15:24:28 +
@@ -1440,76 +1440,81 @@
 
 case LFT_EXT_ACL_DATA:
 if (!al->lastAclData.isEmpty())
 out = al->lastAclData.c_str();
 break;
 }
 
 if (dooff) {
 snprintf(tmp, sizeof(tmp), "%0*" PRId64, fmt->zero && fmt->widthMin >= 0 ? fmt->widthMin : 0, outoff);
 out = tmp;
 
 } else if (doint) {
 snprintf(tmp, sizeof(tmp), "%0*ld", fmt->zero && fmt->widthMin >= 0 ? fmt->widthMin : 0, outint);
 out = tmp;
 } else if (doMsec) {
 if (fmt->widthMax < 0) {
 snprintf(tmp, sizeof(tmp), "%0*ld", fmt->widthMin , tvToMsec(outtv));
 } else {
 int precision = fmt->widthMax;
 snprintf(tmp, sizeof(tmp), "%0*" PRId64 ".%0*" PRId64 "", fmt->zero && (fmt->widthMin - precision - 1 >= 0) ? fmt->widthMin - precision - 1 : 0, static_cast(outtv.tv_sec * 1000 + outtv.tv_usec / 1000), precision, static_cast((outtv.tv_usec % 1000 )* (1000 / fmt->divisor)));
 }
 out = tmp;
 } else if (doSec) {
 int precision = fmt->widthMax >=0 ? fmt->widthMax :3;
 snprintf(tmp, sizeof(tmp), "%0*" PRId64 ".%0*d", fmt->zero && (fmt->widthMin - precision - 1 >= 0) ? fmt->widthMin - precision - 1 : 0, static_cast(outtv.tv_sec), precision, (int)(outtv.tv_usec / fmt->divisor));
 out = tmp;
 }
 
 if (out && *out) {
 if (quote || fmt->quote != LOG_QUOTE_NONE) {
+// Do not write to the tmp buffer because it may contain the to-be-quoted value.
+static char quotedOut[2 * sizeof(tmp)];
+static_assert(sizeof(quotedOut) > 0, "quotedOut has zero length");
+quotedOut[0] = '\0';
+
 char *newout = NULL;
 int newfree = 0;
 
 switch (fmt->quote) {
 
 case LOG_QUOTE_NONE:
 newout = rfc1738_escape_unescaped(out);
 break;
 
 case LOG_QUOTE_QUOTES: {
 size_t out_len = static_cast(strlen(out)) * 2 + 1;
 if (out_len >= sizeof(tmp)) {
 newout = (char *)xmalloc(out_len);
 newfree = 1;
 } else
-newout = tmp;
+newout = quotedOut;
 log_quoted_string(out, newout);
 }
 break;
 
 case LOG_QUOTE_MIMEBLOB:
 newout = QuoteMimeBlob(out);
 newfree = 1;
 break;
 
 case LOG_QUOTE_URL:
 newout = rfc1738_escape(out);
 break;
 
 case LOG_QUOTE_SHELL: {
 MemBuf mbq;
 mbq.init();
 strwordquote(, out);
 newout = mbq.content();
 mbq.stolen = 1;
 newfree = 1;
 }
 break;
 
 case LOG_QUOTE_RAW:
 break;
 }
 
 if (newout) {
 if (dofree)
 safe_free(out);

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PREVIEW] Response delay pools

[Eduard Bagdasaryan]

This new feature is useful for specific response(s) bandwidth limiting.
AFAIK, there are situations when doing this is hardly possible
(or impossible) by means of netfilter/iptables operating with
TCP/IP packets and IP addresses information for filtering. In other
words, sometimes it is problematic to 'extract' a single response from
TCP/IP data flow at system level. For example, a single Squid-to-client
TCP connection can transmit multiple responses (persistent connections,
pipelining or HTTP/2 connection multiplexing) or be encrypted
(HTTPS proxy mode).


HTH,
Eduard.


On 09.01.2017 09:33, Amos Jeffries wrote:

On 24/12/2016 9:26 p.m., Eduard Bagdasaryan wrote:

Hello,

This patch introduces a new "response_delay_pools" feature.
The feature restricts Squid-to-client bandwidth and applies to both
cache hits and misses.

When Squid starts delivering the final HTTP response to a client, Squid
checks response_delay_pool_access rules (supporting fast ACLs only), in
the order they were declared. The first rule with a matching ACL wins.
If (and only if) an "allow" rule won, Squid assigns the response to the
corresponding named delay pool.

If a response is assigned to a delay pool, the response becomes subject
to the configured bucket and aggregate bandwidth limits of that pool,
similar to the current "class 2" server-side delay pools, but with a
brand new, dedicated "individual" filled bucket assigned to the
matched response.

The new feature serves the same purpose as the existing client-side
pools: both features limit Squid-to-client bandwidth. Their common
interface was placed into a new base BandwidthBucket class.  The
difference is that client-side pools do not aggregate clients and always
use one bucket per client IP. It is possible that a response becomes a
subject of both these pools. In such situations only matched response
delay pool will be used for Squid-to-client speed limiting.

The accurate SMP support (with the aggregate bucket shared among
workers) is outside this patch scope. In SMP configurations,
Squid should automatically divide the aggregate_speed_limit and
max_aggregate_size values among the configured number of Squid
workers.

I have not synced with last v5 yet: this patch applies to v4 r14793.
To speed up the process and initiate the discussion I am posting with
"preview" flag and meanwhile start merging with the current v5 version.


Thanks. Sorry I didn't get a chance to look at it yet. That should now
happen in the next few days.

Can you please add to the above an explanation of why this is being
added instead of improving ways to send TOS/NFMARK to feed the system
QoS bandwidth controls. We really should not be re-inventing QoS in
Squid (for about the third time AFAICT).

Amos
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev


___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Case-insensitive URI schemes

[Eduard Bagdasaryan]

On 06.01.2017 15:27, Amos Jeffries wrote:

As a result, the code responsible for lower-case

transformation was not executed.


That is intentional behaviour for several reasons;

1) it improves transparency and reduces risks from proxy 
fingerprinting by systems probing the URI scheme handling by the 
transport agents (ie, fingerprinting Squid).


2) unknown URI schemes are not necessarily handled properly as 
case-insensitive by the experimental agents sending and receiving the 
messages.


also, (and more importantly);


The patch does not change this, i.e., "unknown" images are still stored 
without

down-casing.



3) the transport protocol label and URI scheme label are still 
conflated. The scheme down-casing procedure is _only_ applicable when 
translating from ProtocolType_str labels (upper case) to scheme label 
(lower case).


To avoid misunderstanding I pay your attention that the unpatched Squid 
did not
down-case at all (i.e. for known ProtocolType_str schemes too). In other 
words, when
receiving HTTP://example.com "HTTP" was not down-cased. Just this 
violates HTTP
caching rules: two different cache entries were created for 
HTTP://example.com

and http://example.com requests.




4) storing the down-cased string for registered protocols of each URI 
avoids many explicit down-casing operations on use/display of the URI 
scheme. Note that is specific to the known protocols.


 - There are many more points of code displaying the scheme than 
setting it. So this is a significant performance gain despite the 
overhead of allocating and own-casing a new SBuf per UriScheme object 
your patch notes with an XXX.


I am not against allocating and storing down-cased SBuf "image_" (for 
performance sake).
The related XXX is about  allocating SBuf which we probably can avoid in 
future optimization.
For example, we could do this by converting ProtocolType_str to a const 
array of SBufs, thus

avoiding image_ member allocation when dealing with known protocols.





There are a couple of XXX and one of them (about "broken caller" for
PROTO_NONE) needs clarification. This caller uses PROTO_NONE
(i.e., absent scheme) together with "http" scheme image inside
clientReplyContext::createStoreEntry() which looks inconsistent and
probably is a bug. Am I missing anything there?


Thats an odd case. I'm not exactly sure what we should be doing there. 
Thus the unusual parameter combo. It was done that way to minimize 
breakage with old code - so the request can be detected as missing 
scheme (avoid confusing with a real HTTP URL) if anything tries to 
compare it, but still generates a reasonably correct URL if its dumped 
to store files.




Since you agree that it is an "odd" case (and we are not going to fix it 
right now) I would leave

the "broken caller" XXX.

Please note that the major problem here is probably the caching problem 
as I noted above.
I have not fully investigated whether it has security aspects, i.e., 
affects some URL based ACLs

while comparing with stored URLs.


Eduard.

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] [PATCH] Case-insensitive URI schemes

[Eduard Bagdasaryan]

Hello,

This patch fixes URI schemes to be case-insensitive (r14802 regression).

The bug was introduced by r14802 (better support of unknown
URL schemes). AnyP::UriScheme constructor had a problem: the
explicitly-specified img parameter (always provided by URL::setScheme())
was used for all schemes, not only for unknown ones (as it obviously
should be). As a result, the code responsible for lower-case
transformation was not executed.

There are a couple of XXX and one of them (about "broken caller" for
PROTO_NONE) needs clarification. This caller uses PROTO_NONE
(i.e., absent scheme) together with "http" scheme image inside
clientReplyContext::createStoreEntry() which looks inconsistent and
probably is a bug. Am I missing anything there?


Thanks,
Eduard.

Fix URI schemes to be case-insensitive (caused by r14802).

The patch fixes a bug introduced by r14802 (better support of unknown
URL schemes). AnyP::UriScheme constructor had a problem: the
explicitly-specified img parameter (always provided by URL::setScheme())
was used for all schemes, not only for unknown ones (as it obviously
should be). As a result, the code responsible for lower-case
transformation was not executed.

=== modified file 'src/anyp/UriScheme.cc'
--- src/anyp/UriScheme.cc	2016-08-17 00:38:25 +
+++ src/anyp/UriScheme.cc	2017-01-05 14:57:10 +
@@ -1,61 +1,71 @@
 /*
  * Copyright (C) 1996-2016 The Squid Software Foundation and contributors
  *
  * Squid software is distributed under GPLv2+ license and includes
  * contributions from numerous individuals and organizations.
  * Please see the COPYING and CONTRIBUTORS files for details.
  */
 
 /* DEBUG: section 23URL Scheme parsing */
 
 #include "squid.h"
 #include "anyp/UriScheme.h"
 
 AnyP::UriScheme::UriScheme(AnyP::ProtocolType const aScheme, const char *img) :
 theScheme_(aScheme)
 {
-if (img)
-// image could be provided explicitly (case-sensitive)
-image_ = img;
-
+// RFC 3986 section 3.1: schemes are case-insensitive.
+
+// To improve diagnostic, remember exactly how an unsupported scheme looks like.
+// XXX: Object users may rely on toLower() canonicalization that we refuse to provide.
+if (img && theScheme_ == AnyP::PROTO_UNKNOWN)
+ image_ = img;
+
+// XXX: A broken caller supplies an image of an absent scheme?
+// XXX: We assume that the caller is using a lower-case image.
+else if (img && theScheme_ == AnyP::PROTO_NONE)
+ image_ = img;
+
+// the caller knows nothing about the scheme
+// TODO: Avoid special casing by storing this string in the generated ProtocolType_str?
 else if (theScheme_ == AnyP::PROTO_UNKNOWN)
-// image could be actually unknown and not provided
-image_ = "(unknown)";
+ image_ = "(unknown)";
 
+// use the standard image (and ignore supplied one, if any) for known transfer protocols
 else if (theScheme_ > AnyP::PROTO_NONE && theScheme_ < AnyP::PROTO_MAX) {
-// image could be implied by a registered transfer protocol
-// which use upper-case labels, so down-case for scheme image
+// XXX: do not allocate (and lower-case) a new SBuf every time we need a well-known image
 image_ = AnyP::ProtocolType_str[theScheme_];
 image_.toLower();
 }
-// else, image is an empty string ("://example.com/")
+// else, the image remains empty (e.g., "://example.com/")
+// hopefully, theScheme_ is PROTO_NONE here
 }
 
 unsigned short
 AnyP::UriScheme::defaultPort() const
 {
 switch (theScheme_) {
 
 case AnyP::PROTO_HTTP:
 return 80;
 
 case AnyP::PROTO_HTTPS:
 return 443;
 
 case AnyP::PROTO_FTP:
 return 21;
 
 case AnyP::PROTO_COAP:
 case AnyP::PROTO_COAPS:
 // coaps:// default is TBA as of draft-ietf-core-coap-08.
 // Assuming IANA policy of allocating same port for base and TLS protocol versions will occur.
 return 5683;
 
 case AnyP::PROTO_GOPHER:
 return 70;
 
 case AnyP::PROTO_WAIS:
 return 210;
 
 case AnyP::PROTO_CACHE_OBJECT:
 return CACHE_HTTP_PORT;

=== modified file 'src/anyp/UriScheme.h'
--- src/anyp/UriScheme.h	2016-08-17 00:38:25 +
+++ src/anyp/UriScheme.h	2017-01-03 18:04:03 +
@@ -1,56 +1,57 @@
 /*
  * Copyright (C) 1996-2016 The Squid Software Foundation and contributors
  *
  * Squid software is distributed under GPLv2+ license and includes
  * contributions from numerous individuals and organizations.
  * Please see the COPYING and CONTRIBUTORS files for details.
  */
 
 #ifndef SQUID_ANYP_URISCHEME_H
 #define SQUID_ANYP_URISCHEME_H
 
 #include "anyp/ProtocolType.h"
 #include "sbuf/SBuf.h"
 
 #include 
 
 namespace AnyP
 {
 
 /** This class represents a URI Scheme such as http:// https://, wais://, urn: etc.
  * It does not represent the PROTOCOL that such schemes refer to.
  */
 class UriScheme
 {
 public:
 UriScheme() : theScheme_(AnyP::PROTO_NONE) {}
+/// 

[squid-dev] [PREVIEW] Response delay pools

[Eduard Bagdasaryan]

Hello,

This patch introduces a new "response_delay_pools" feature.
The feature restricts Squid-to-client bandwidth and applies to both
cache hits and misses.

When Squid starts delivering the final HTTP response to a client, Squid
checks response_delay_pool_access rules (supporting fast ACLs only), in
the order they were declared. The first rule with a matching ACL wins.
If (and only if) an "allow" rule won, Squid assigns the response to the
corresponding named delay pool.

If a response is assigned to a delay pool, the response becomes subject
to the configured bucket and aggregate bandwidth limits of that pool,
similar to the current "class 2" server-side delay pools, but with a
brand new, dedicated "individual" filled bucket assigned to the
matched response.

The new feature serves the same purpose as the existing client-side
pools: both features limit Squid-to-client bandwidth. Their common
interface was placed into a new base BandwidthBucket class.  The
difference is that client-side pools do not aggregate clients and always
use one bucket per client IP. It is possible that a response becomes a
subject of both these pools. In such situations only matched response
delay pool will be used for Squid-to-client speed limiting.

The accurate SMP support (with the aggregate bucket shared among
workers) is outside this patch scope. In SMP configurations,
Squid should automatically divide the aggregate_speed_limit and
max_aggregate_size values among the configured number of Squid
workers.

I have not synced with last v5 yet: this patch applies to v4 r14793.
To speed up the process and initiate the discussion I am posting with
"preview" flag and meanwhile start merging with the current v5 version.


Thank you,
Eduard.

Added response delay pools feature for Squid-to-client speed limiting.

The feature restricts Squid-to-client bandwidth only.  It applies to
both cache hits and misses.

When Squid starts delivering the final HTTP response to a client, Squid
checks response_delay_pool_access rules (supporting fast ACLs only), in
the order they were declared. The first rule with a matching ACL wins.
If (and only if) an "allow" rule won, Squid assigns the response to the
corresponding named delay pool.

If a response is assigned to a delay pool, the response becomes subject
to the configured bucket and aggregate bandwidth limits of that pool,
similar to the current "class 2" server-side delay pools, but with a
brand new, dedicated "individual" filled bucket assigned to the
matched response.

The new feature serves the same purpose as the existing client-side
pools: both features limit Squid-to-client bandwidth. Their common
interface was placed into a new base BandwidthBucket class.  The
difference is that client-side pools do not aggregate clients and always
use one bucket per client IP. It is possible that a response becomes a
subject of both these pools. In such situations only matched response
delay pool will be used for Squid-to-client speed limiting.

The accurate SMP support (with the aggregate bucket shared among
workers) is outside this patch scope. In SMP configurations,
Squid should automatically divide the aggregate_speed_limit and
max_aggregate_size values among the configured number of Squid
workers. 

=== added file 'src/BandwidthBucket.cc'
--- src/BandwidthBucket.cc	1970-01-01 00:00:00 +
+++ src/BandwidthBucket.cc	2016-12-23 18:13:43 +
@@ -0,0 +1,111 @@
+/*
+ * Copyright (C) 1996-2016 The Squid Software Foundation and contributors
+ *
+ * Squid software is distributed under GPLv2+ license and includes
+ * contributions from numerous individuals and organizations.
+ * Please see the COPYING and CONTRIBUTORS files for details.
+ */
+
+#include "squid.h"
+
+#if USE_DELAY_POOLS
+
+#include "BandwidthBucket.h"
+#include "ClientInfo.h"
+#include "comm/Connection.h"
+#include "Debug.h"
+#include "fde.h"
+
+extern double current_dtime;
+
+BandwidthBucket::BandwidthBucket(const int aWriteSpeedLimit, const double anInitialBurst,
+ const double aHighWatermark) :
+bucketSize(anInitialBurst),
+selectWaiting(false),
+writeSpeedLimit(aWriteSpeedLimit),
+bucketSizeLimit(aHighWatermark)
+{
+getCurrentTime();
+/* put current time to have something sensible here */
+prevTime = current_dtime;
+}
+
+void
+BandwidthBucket::refillBucket()
+{
+// all these times are in seconds, with double precision
+const double currTime = current_dtime;
+const double timePassed = currTime - prevTime;
+
+// Calculate allowance for the time passed. Use double to avoid
+// accumulating rounding errors for small intervals. For example, always
+// adding 1 byte instead of 1.4 results in 29% bandwidth allocation error.
+const double gain = timePassed * writeSpeedLimit;
+
+// XXX: Decide whether to add 'hash' field like ClientInfo::hash
+//  debugs(77,5, HERE << currTime << " clt" << (const char*)hash.key << ": " <<
+// 

Re: [squid-dev] [PATCH] Compilation fix for v5 r14973

[Eduard Bagdasaryan]

Attached a patch witch removes SquidConfig  dependency
on vector and uses vector *.
instead. This fixes pinger linking error.


Eduard.

On 10.12.2016 23:55, Amos Jeffries wrote:

On 11/12/2016 6:12 a.m., Christos Tsantilas wrote:

I applied the patch, however still exist problem. The icmp pinger does
not build correctly.
We should add libsbuf library to pinger libraries, but still there are
references to HistStat.cc file (maybe add HistStat stub files for pinger?).

pinger does not use the Auth:: things, so it really should not pull them
+ dependencies in.

The correct fix I think is to refactor the Auth::Config so that the
various global auth* directives can all be stored there. I'm working on
that right now.

Amos

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev


A linker fix for building pinger, caused by r14973.

Jenkins revealed an linker error when building pinger (--enable-icmp
option is on).  Before this patch, SquidConfig had a dependency
on vector which required additional libraries at linker
stage, e.g., libsbuf.  The suggested solution is to get rid of such
dependency using vector * instead.

=== modified file 'src/SquidConfig.h'
--- src/SquidConfig.h	2016-12-10 04:48:25 +
+++ src/SquidConfig.h	2016-12-11 08:27:58 +
@@ -518,54 +518,54 @@
 #endif
 } ssl_client;
 
 char *accept_filter;
 int umask;
 int max_filedescriptors;
 int workers;
 CpuAffinityMap *cpuAffinityMap;
 
 #if USE_LOADABLE_MODULES
 wordlist *loadable_module_names;
 #endif
 
 int client_ip_max_connections;
 
 char *redirector_extras;
 
 struct UrlHelperTimeout {
 int action;
 char *response;
 } onUrlRewriteTimeout;
 
 char *storeId_extras;
 
 struct {
 int v4_first;   ///< Place IPv4 first in the order of DNS results.
 ssize_t packet_max; ///< maximum size EDNS advertised for DNS replies.
 } dns;
 
 #if USE_AUTH
-Auth::SchemesConfigs authSchemesConfigs;
+Auth::SchemesConfigs *authSchemesConfigs;
 #endif
 };
 
 extern SquidConfig Config;
 
 class SquidConfig2
 {
 public:
 void clear() {
 *this = SquidConfig2();
 }
 
 struct {
 int enable_purge = 0;
 } onoff;
 uid_t effectiveUserID = 0;
 gid_t effectiveGroupID = 0;
 };
 
 extern SquidConfig2 Config2;
 
 #endif /* SQUID_SQUIDCONFIG_H_ */
 

=== modified file 'src/auth/UserRequest.cc'
--- src/auth/UserRequest.cc	2016-12-10 04:48:25 +
+++ src/auth/UserRequest.cc	2016-12-11 08:20:38 +
@@ -446,61 +446,61 @@
 if (*aUR == NULL)
 *aUR = t;
 
 if (request->auth_user_request == NULL && t->lastReply == AUTH_AUTHENTICATED) {
 request->auth_user_request = t;
 }
 return t->lastReply;
 }
 
 // ok, call the actual authenticator routine.
 AuthAclState result = authenticate(aUR, headertype, request, conn, src_addr, al);
 
 // auth process may have changed the UserRequest we are dealing with
 t = authTryGetUser(*aUR, conn, request);
 
 if (t != NULL && result != AUTH_ACL_CANNOT_AUTHENTICATE && result != AUTH_ACL_HELPER)
 t->lastReply = result;
 
 return result;
 }
 
 static Auth::ConfigVector &
 schemesConfig(HttpRequest *request, HttpReply *rep)
 {
 if (::Config.accessList.authSchemes) {
 ACLFilledChecklist ch(NULL, request, NULL);
 ch.reply = rep;
 HTTPMSGLOCK(ch.reply);
 const allow_t answer = ch.fastCheck(::Config.accessList.authSchemes);
 if (answer == ACCESS_ALLOWED)
-return ::Config.authSchemesConfigs.at(answer.kind).authConfigs;
+return ::Config.authSchemesConfigs->at(answer.kind).authConfigs;
 }
 return Auth::TheConfig;
 }
 
 void
 Auth::UserRequest::addReplyAuthHeader(HttpReply * rep, Auth::UserRequest::Pointer auth_user_request, HttpRequest * request, int accelerated, int internal)
 /* send the auth types we are configured to support (and have compiled in!) */
 {
 Http::HdrType type;
 
 switch (rep->sline.status()) {
 
 case Http::scProxyAuthenticationRequired:
 /* Proxy authorisation needed */
 type = Http::HdrType::PROXY_AUTHENTICATE;
 break;
 
 case Http::scUnauthorized:
 /* WWW Authorisation needed */
 type = Http::HdrType::WWW_AUTHENTICATE;
 break;
 
 default:
 /* Keep GCC happy */
 /* some other HTTP status */
 type = Http::HdrType::BAD_HDR;
 break;
 }
 
 debugs(29, 9, "headertype:" << type << " authuser:" << auth_user_request);

=== modified file 'src/cache_cf.cc'
--- src/cache_cf.cc	2016-12-10 16:27:09 +
+++ src/cache_cf.cc	2016-12-11 09:20:44 +
@@ -919,65 +919,67 @@
(uint32_t)Config.maxRequestBufferSize, (uint32_t)Config.maxRequestHeaderSize);
 }
 
 /*
  * Disable client side request pipelining if 

[squid-dev] [PATCH] Compilation fix for v5 r14973

[Eduard Bagdasaryan]

Hello,

Attached a typo fix for r14973 which caused Jenkins errors.

Regards,
Eduard.
A typo fix for r14973 revealed by Jenkins.

Should use 'ConfigParser::LastTokenWasQuoted()' instead of
'ConfigParser::LastTokenWasQuoted'.

=== modified file 'src/cache_cf.cc'
--- src/cache_cf.cc	2016-12-10 06:12:06 +
+++ src/cache_cf.cc	2016-12-10 11:53:07 +
@@ -1817,61 +1817,61 @@
 }
 
 static void
 free_authparam(Auth::ConfigVector * cfg)
 {
 /* Wipe the Auth globals and Detach/Destruct component config + state. */
 cfg->clear();
 
 /* on reconfigure initialize new auth schemes for the new config. */
 if (reconfiguring) {
 Auth::Init();
 }
 }
 
 static void
 dump_authparam(StoreEntry * entry, const char *name, Auth::ConfigVector cfg)
 {
 for (Auth::ConfigVector::iterator  i = cfg.begin(); i != cfg.end(); ++i)
 (*i)->dump(entry, name, (*i));
 }
 
 static void
 parse_AuthSchemes(acl_access **authSchemes)
 {
 const char *tok = ConfigParser::NextQuotedToken();
 if (!tok) {
 debugs(29, DBG_CRITICAL, "FATAL: auth_schemes missing the parameter");
 self_destruct();
 return;
 }
-Config.authSchemesConfigs.push_back(Auth::SchemesConfig(tok, ConfigParser::LastTokenWasQuoted));
+Config.authSchemesConfigs.push_back(Auth::SchemesConfig(tok, ConfigParser::LastTokenWasQuoted()));
 const allow_t action = allow_t(ACCESS_ALLOWED, Config.authSchemesConfigs.size() - 1);
 ParseAclWithAction(authSchemes, action, "auth_schemes");
 }
 
 static void
 free_AuthSchemes(acl_access **authSchemes)
 {
 Config.authSchemesConfigs.clear();
 free_acl_access(authSchemes);
 }
 
 static void
 dump_AuthSchemes(StoreEntry *entry, const char *name, acl_access *authSchemes)
 {
 if (authSchemes)
 dump_SBufList(entry, authSchemes->treeDump(name, [](const allow_t ) {
 return Config.authSchemesConfigs.at(action.kind).rawSchemes;
 }));
 }
 
 #endif /* USE_AUTH */
 
 static void
 ParseAclWithAction(acl_access **access, const allow_t , const char *desc, ACL *acl)
 {
 assert(access);
 SBuf name;
 if (!*access) {
 *access = new Acl::Tree;
 name.Printf("(%s rules)", desc);

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] auth_schemes directive

[Eduard Bagdasaryan]

Attached two patches for v5 after splitting. Please apply
SQUID-242-refactor-custom-acl-actions-cfg-t1.patch first.


Regards,
Eduard


On 02.12.2016 19:41, Alex Rousskov wrote:

Directive naming aside, since what you have proposed is already
implemented, we will proceed with splitting the patch into two parts, as
you have requested earlier. This will give you and others more time to
suggest a better name than auth_schemes.



Added auth_schemes to control schemes presence and order in 401s/407s.

The new squid.conf directive may be used to customize authentication
schemes presence and order in Squid's HTTP 401 (Unauthorized) and 407
(Proxy Authentication Required) responses. The defaults remain the same.

=== modified file 'src/SquidConfig.h'
--- src/SquidConfig.h	2016-11-07 05:24:26 +
+++ src/SquidConfig.h	2016-12-05 14:40:03 +
@@ -1,42 +1,45 @@
 /*
  * Copyright (C) 1996-2016 The Squid Software Foundation and contributors
  *
  * Squid software is distributed under GPLv2+ license and includes
  * contributions from numerous individuals and organizations.
  * Please see the COPYING and CONTRIBUTORS files for details.
  */
 
 #ifndef SQUID_SQUIDCONFIG_H_
 #define SQUID_SQUIDCONFIG_H_
 
 #include "acl/forward.h"
+#if USE_AUTH
+#include "auth/SchemesConfig.h"
+#endif
 #include "base/RefCount.h"
 #include "base/YesNoNone.h"
 #include "ClientDelayConfig.h"
 #include "DelayConfig.h"
 #include "helper/ChildConfig.h"
 #include "HttpHeaderTools.h"
 #include "ip/Address.h"
 #include "Notes.h"
 #include "security/forward.h"
 #include "SquidTime.h"
 #if USE_OPENSSL
 #include "ssl/support.h"
 #endif
 #include "store/forward.h"
 
 #if USE_OPENSSL
 class sslproxy_cert_sign;
 class sslproxy_cert_adapt;
 #endif
 
 namespace Mgr
 {
 class ActionPasswordList;
 } // namespace Mgr
 class CachePeer;
 class CustomLog;
 class CpuAffinityMap;
 class external_acl;
 class HeaderManglers;
 class RefreshPattern;
@@ -370,60 +373,63 @@
 #endif
 acl_access *redirector;
 acl_access *store_id;
 acl_access *reply;
 Acl::Address *outgoing_address;
 #if USE_HTCP
 
 acl_access *htcp;
 acl_access *htcp_clr;
 #endif
 
 #if USE_OPENSSL
 acl_access *ssl_bump;
 #endif
 #if FOLLOW_X_FORWARDED_FOR
 acl_access *followXFF;
 #endif /* FOLLOW_X_FORWARDED_FOR */
 
 /// acceptible PROXY protocol clients
 acl_access *proxyProtocol;
 
 /// spoof_client_ip squid.conf acl.
 /// nil unless configured
 acl_access* spoof_client_ip;
 acl_access *on_unsupported_protocol;
 
 acl_access *ftp_epsv;
 
 acl_access *forceRequestBodyContinuation;
 acl_access *serverPconnForNonretriable;
+#if USE_AUTH
+acl_access *authSchemes;
+#endif
 } accessList;
 AclDenyInfoList *denyInfoList;
 
 struct {
 size_t list_width;
 int list_wrap;
 char *anon_user;
 int passive;
 int epsv_all;
 int epsv;
 int eprt;
 int sanitycheck;
 int telnet;
 } Ftp;
 RefreshPattern *Refresh;
 
 Store::DiskConfig cacheSwap;
 
 struct {
 char *directory;
 int use_short_names;
 } icons;
 char *errorDirectory;
 #if USE_ERR_LOCALES
 char *errorDefaultLanguage;
 int errorLogMissingLanguages;
 #endif
 char *errorStylesheet;
 
 struct {
@@ -511,52 +517,55 @@
 sslproxy_cert_adapt *cert_adapt;
 #endif
 } ssl_client;
 
 char *accept_filter;
 int umask;
 int max_filedescriptors;
 int workers;
 CpuAffinityMap *cpuAffinityMap;
 
 #if USE_LOADABLE_MODULES
 wordlist *loadable_module_names;
 #endif
 
 int client_ip_max_connections;
 
 char *redirector_extras;
 
 struct UrlHelperTimeout {
 int action;
 char *response;
 } onUrlRewriteTimeout;
 
 char *storeId_extras;
 
 struct {
 int v4_first;   ///< Place IPv4 first in the order of DNS results.
 ssize_t packet_max; ///< maximum size EDNS advertised for DNS replies.
 } dns;
 
+#if USE_AUTH
+Auth::SchemesConfigs authSchemesConfigs;
+#endif
 };
 
 extern SquidConfig Config;
 
 class SquidConfig2
 {
 public:
 void clear() {
 *this = SquidConfig2();
 }
 
 struct {
 int enable_purge = 0;
 } onoff;
 uid_t effectiveUserID = 0;
 gid_t effectiveGroupID = 0;
 };
 
 extern SquidConfig2 Config2;
 
 #endif /* SQUID_SQUIDCONFIG_H_ */
 

=== modified file 'src/auth/Config.cc'
--- src/auth/Config.cc	2016-01-01 00:12:18 +
+++ src/auth/Config.cc	2016-12-05 14:40:03 +
@@ -38,60 +38,69 @@
 Auth::Config *config = Find(proxy_auth);
 
 if (config == NULL || !config->active()) {
 debugs(29, (shutting_down?3:DBG_IMPORTANT), (shutting_down?"":"WARNING: ") <<
"Unsupported or unconfigured/inactive proxy-auth scheme, '" << proxy_auth << "'");
 return NULL;
 }
 static MemBuf rmb;
 rmb.reset();
 if 

Re: [squid-dev] [PATCH] auth_schemes directive

[Eduard Bagdasaryan]

 2016-11-19 12:15 GMT+03:00 Amos Jeffries :
>  To add auth scheme access controls in a way that does not actively
> prevent (1) from being implemented later IMO we need to add access
> controls as a member of the Config object, *not* as a global access list.
>
> The config would look like:
>  auth_param foo access allow blah blah
>  auth_param foo access deny bar
>
>  auth_param foo2 access deny blah
>  auth_param foo2 access allow all

If I understood you correctly, we have foo and foo2 auth schemes, and do
not want foo to be used with "bar", foo2 with "blah" and want both
schemes for others. I don't see problem here because with new
auth_schemes we similarly get:

 auth_param foo ...
 auth_param foo2 ...

 auth_schemes foo blah
 auth_schemes foo2 bar


Eduard.

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] [PATCH] auth_schemes directive

[Eduard Bagdasaryan]

Hello,

This patch introduces a new 'auth_schemes' squid.conf directive.

This directive may be used to customize authentication
schemes presence and order in Squid's HTTP 401 (Unauthorized) and 407
(Proxy Authentication Required) responses. The defaults remain the same.

Also refactored configuration code for options with custom
ACL-controlled actions:

* Reduced parsing code duplication by adding ParseAclWithAction().

* Used functors for generic action-to-string conversions. It is possible
  now to perform such conversions providing lambda expressions.

* Used vectors of strings instead of C-style arrays for storing
  conversion tables and check against bounds with vector::at().


Regards,
Eduard.

Added auth_schemes to control schemes presence and order in 401s/407s.

The new squid.conf directive may be used to customize authentication
schemes presence and order in Squid's HTTP 401 (Unauthorized) and 407
(Proxy Authentication Required) responses. The defaults remain the same.

Also refactored configuration code for options with custom
ACL-controlled actions:

* Reduced parsing code duplication by adding ParseAclWithAction().

* Used functors for generic action-to-string conversions. It is possible
  now to perform such conversions providing lambda expressions.

* Used vectors of strings instead of C-style arrays for storing
  conversion tables and check against bounds with vector::at().

=== modified file 'src/SquidConfig.h'
--- src/SquidConfig.h	2016-11-07 05:24:26 +
+++ src/SquidConfig.h	2016-11-18 15:03:18 +
@@ -1,42 +1,45 @@
 /*
  * Copyright (C) 1996-2016 The Squid Software Foundation and contributors
  *
  * Squid software is distributed under GPLv2+ license and includes
  * contributions from numerous individuals and organizations.
  * Please see the COPYING and CONTRIBUTORS files for details.
  */
 
 #ifndef SQUID_SQUIDCONFIG_H_
 #define SQUID_SQUIDCONFIG_H_
 
 #include "acl/forward.h"
+#if USE_AUTH
+#include "auth/SchemesConfig.h"
+#endif
 #include "base/RefCount.h"
 #include "base/YesNoNone.h"
 #include "ClientDelayConfig.h"
 #include "DelayConfig.h"
 #include "helper/ChildConfig.h"
 #include "HttpHeaderTools.h"
 #include "ip/Address.h"
 #include "Notes.h"
 #include "security/forward.h"
 #include "SquidTime.h"
 #if USE_OPENSSL
 #include "ssl/support.h"
 #endif
 #include "store/forward.h"
 
 #if USE_OPENSSL
 class sslproxy_cert_sign;
 class sslproxy_cert_adapt;
 #endif
 
 namespace Mgr
 {
 class ActionPasswordList;
 } // namespace Mgr
 class CachePeer;
 class CustomLog;
 class CpuAffinityMap;
 class external_acl;
 class HeaderManglers;
 class RefreshPattern;
@@ -370,60 +373,63 @@
 #endif
 acl_access *redirector;
 acl_access *store_id;
 acl_access *reply;
 Acl::Address *outgoing_address;
 #if USE_HTCP
 
 acl_access *htcp;
 acl_access *htcp_clr;
 #endif
 
 #if USE_OPENSSL
 acl_access *ssl_bump;
 #endif
 #if FOLLOW_X_FORWARDED_FOR
 acl_access *followXFF;
 #endif /* FOLLOW_X_FORWARDED_FOR */
 
 /// acceptible PROXY protocol clients
 acl_access *proxyProtocol;
 
 /// spoof_client_ip squid.conf acl.
 /// nil unless configured
 acl_access* spoof_client_ip;
 acl_access *on_unsupported_protocol;
 
 acl_access *ftp_epsv;
 
 acl_access *forceRequestBodyContinuation;
 acl_access *serverPconnForNonretriable;
+#if USE_AUTH
+acl_access *authSchemes;
+#endif
 } accessList;
 AclDenyInfoList *denyInfoList;
 
 struct {
 size_t list_width;
 int list_wrap;
 char *anon_user;
 int passive;
 int epsv_all;
 int epsv;
 int eprt;
 int sanitycheck;
 int telnet;
 } Ftp;
 RefreshPattern *Refresh;
 
 Store::DiskConfig cacheSwap;
 
 struct {
 char *directory;
 int use_short_names;
 } icons;
 char *errorDirectory;
 #if USE_ERR_LOCALES
 char *errorDefaultLanguage;
 int errorLogMissingLanguages;
 #endif
 char *errorStylesheet;
 
 struct {
@@ -511,52 +517,55 @@
 sslproxy_cert_adapt *cert_adapt;
 #endif
 } ssl_client;
 
 char *accept_filter;
 int umask;
 int max_filedescriptors;
 int workers;
 CpuAffinityMap *cpuAffinityMap;
 
 #if USE_LOADABLE_MODULES
 wordlist *loadable_module_names;
 #endif
 
 int client_ip_max_connections;
 
 char *redirector_extras;
 
 struct UrlHelperTimeout {
 int action;
 char *response;
 } onUrlRewriteTimeout;
 
 char *storeId_extras;
 
 struct {
 int v4_first;   ///< Place IPv4 first in the order of DNS results.
 ssize_t packet_max; ///< maximum size EDNS advertised for DNS replies.
 } dns;
 
+#if USE_AUTH
+Auth::SchemesConfigs authSchemesConfigs;
+#endif
 };
 
 extern SquidConfig Config;
 
 class SquidConfig2
 {
 public:
 void clear() {
 *this = SquidConfig2();
 }
 
 struct {
 int 

Re: [squid-dev] [PATCH] ICAP trailer support

[Eduard Bagdasaryan]

2016-11-11 8:27 GMT+03:00 Amos Jeffries :

> In Adaptation::Icap::ModXact::expectIcapTrailers() please use
> getByIdIfPresent(Http::TRAILER, ...) since Trailer is a registered
> header.

Fixed.

> In answer to "TODO: should we add a new Http::scInvalidTrailer?" only if
> the draft defines an HTTP 5xx status code called "Ivalid > Trailer".
> Since the Http::sc* are HTTP status codes.

The draft does not define HTTP 5xx status codes for invalid trailers.
Also please note that scInvalidHeader (600 code) is marked for
internal/Squid-only use. Probably we could similarly use
scInvalidTrailer with code 602 (for example).

> Only if all the callers of this function are handling the case where it
> produces non-1 and adding a terminator LF explicitly for a re-parse
> attempt.

I have only found such 're-parse' logic inside
HttpStateData::processReplyHeader(), marked with the comment. However
this code does not use the patch-affected HttpHeader::parse(): it uses
Http::One::Parser::grabMimeBlock() instead. So, nothing to fix here, IMO.


Eduard.

Initial ICAP trailer support.

ICAP trailers are currently specified by
https://datatracker.ietf.org/doc/draft-rousskov-icap-trailers/

This implementation complies with version -01 of that draft:

- Squid unconditionally announces ICAP Trailer support via the ICAP
  Allow request header field.

- Squid parses the ICAP response trailer if and only if the ICAP server
  signals its presence by sending both Trailer header and Allow/trailers
  in the ICAP response.

Squid logs and ignores all parsed ICAP header fields (for now).

Also refactored HttpHeader parsing methods in order to reuse them for
ICAP trailer parsing.

Also simplified and fixed headers isolating code while dealing with
empty (i.e. zero header fields) headers. Old httpMsgIsolateHeaders()
tried to re-implement header end detection/processing logic that is
actually covered by headersEnd(). Old httpMsgIsolateHeaders() could
return success for some garbage input (e.g., a buffer of several CRs)
even if no end of headers was found.

=== modified file 'src/HttpHeader.cc'
--- src/HttpHeader.cc	2016-10-06 21:02:32 +
+++ src/HttpHeader.cc	2016-11-08 13:37:55 +
@@ -1,54 +1,55 @@
 /*
  * Copyright (C) 1996-2016 The Squid Software Foundation and contributors
  *
  * Squid software is distributed under GPLv2+ license and includes
  * contributions from numerous individuals and organizations.
  * Please see the COPYING and CONTRIBUTORS files for details.
  */
 
 /* DEBUG: section 55HTTP Header */
 
 #include "squid.h"
 #include "base/EnumIterator.h"
 #include "base64.h"
 #include "globals.h"
 #include "http/ContentLengthInterpreter.h"
 #include "HttpHdrCc.h"
 #include "HttpHdrContRange.h"
 #include "HttpHdrScTarget.h" // also includes HttpHdrSc.h
 #include "HttpHeader.h"
 #include "HttpHeaderFieldInfo.h"
 #include "HttpHeaderStat.h"
 #include "HttpHeaderTools.h"
 #include "MemBuf.h"
 #include "mgr/Registration.h"
+#include "mime_header.h"
 #include "profiler/Profiler.h"
 #include "rfc1123.h"
 #include "SquidConfig.h"
 #include "StatHist.h"
 #include "Store.h"
 #include "StrList.h"
 #include "TimeOrTag.h"
 #include "util.h"
 
 #include 
 
 /* XXX: the whole set of API managing the entries vector should be rethought
  *  after the parse4r-ng effort is complete.
  */
 
 /*
  * On naming conventions:
  *
  * HTTP/1.1 defines message-header as
  *
  * message-header = field-name ":" [ field-value ] CRLF
  * field-name = token
  * field-value= *( field-content | LWS )
  *
  * HTTP/1.1 does not give a name name a group of all message-headers in a message.
  * Squid 1.1 seems to refer to that group _plus_ start-line as "headers".
  *
  * HttpHeader is an object that represents all message-headers in a message.
  * HttpHeader does not manage start-line.
  *
@@ -289,60 +290,105 @@
 
 const HttpHeaderEntry *e;
 HttpHeaderPos pos = HttpHeaderInitPos;
 
 while ((e = fresh->getEntry())) {
 /* deny bad guys (ok to check for Http::HdrType::OTHER) here */
 
 if (skipUpdateHeader(e->id))
 continue;
 
 if (e->id != Http::HdrType::OTHER)
 delById(e->id);
 else
 delByName(e->name.termedBuf());
 }
 
 pos = HttpHeaderInitPos;
 while ((e = fresh->getEntry())) {
 /* deny bad guys (ok to check for Http::HdrType::OTHER) here */
 
 if (skipUpdateHeader(e->id))
 continue;
 
 debugs(55, 7, "Updating header '" << Http::HeaderLookupTable.lookup(e->id).name << "' in cached entry");
 
 addEntry(e->clone());
 }
 return true;
 }
 
+bool
+HttpHeader::Isolate(const char **parse_start, size_t l, const char **blk_start, const char **blk_end)
+{
+/*
+ * parse_start points to the first line of HTTP message *headers*,
+ * not including the request or status lines
+ */
+const size_t end = headersEnd(*parse_start, l);
+
+if (end) {
+*blk_start = 

[squid-dev] [PATCH] ICAP trailer support

[Eduard Bagdasaryan]

Hello,

This patch introduces the initial ICAP trailer support. ICAP trailers are
currently specified by
https://datatracker.ietf.org/doc/draft-rousskov-icap-trailers/. For now,
Squid logs and ignores all parsed ICAP header fields.  In future we plan
to add code for using ICAP trailers for transaction annotations and
access logging.

Also refactored HttpHeader parsing methods in order to reuse them for
ICAP trailer parsing.

Also simplified and fixed headers isolating code while dealing with
empty (i.e. zero header fields) headers. Old httpMsgIsolateHeaders()
tried to re-implement header end detection/processing logic that is
actually covered by headersEnd(). Old httpMsgIsolateHeaders() could
return success for some garbage input (e.g., a buffer of several CRs)
even if no end of headers was found.

Regards,
Eduard.

Initial ICAP trailer support.

ICAP trailers are currently specified by
https://datatracker.ietf.org/doc/draft-rousskov-icap-trailers/

This implementation complies with version -01 of that draft:

- Squid unconditionally announces ICAP Trailer support via the ICAP
  Allow request header field.

- Squid parses the ICAP response trailer if and only if the ICAP server
  signals its presence by sending both Trailer header and Allow/trailers
  in the ICAP response.

Squid logs and ignores all parsed ICAP header fields (for now).

Also refactored HttpHeader parsing methods in order to reuse them for
ICAP trailer parsing.

Also simplified and fixed headers isolating code while dealing with
empty (i.e. zero header fields) headers. Old httpMsgIsolateHeaders()
tried to re-implement header end detection/processing logic that is
actually covered by headersEnd(). Old httpMsgIsolateHeaders() could
return success for some garbage input (e.g., a buffer of several CRs)
even if no end of headers was found.

=== modified file 'src/HttpHeader.cc'
--- src/HttpHeader.cc	2016-10-06 21:02:32 +
+++ src/HttpHeader.cc	2016-11-07 21:20:57 +
@@ -1,54 +1,55 @@
 /*
  * Copyright (C) 1996-2016 The Squid Software Foundation and contributors
  *
  * Squid software is distributed under GPLv2+ license and includes
  * contributions from numerous individuals and organizations.
  * Please see the COPYING and CONTRIBUTORS files for details.
  */
 
 /* DEBUG: section 55HTTP Header */
 
 #include "squid.h"
 #include "base/EnumIterator.h"
 #include "base64.h"
 #include "globals.h"
 #include "http/ContentLengthInterpreter.h"
 #include "HttpHdrCc.h"
 #include "HttpHdrContRange.h"
 #include "HttpHdrScTarget.h" // also includes HttpHdrSc.h
 #include "HttpHeader.h"
 #include "HttpHeaderFieldInfo.h"
 #include "HttpHeaderStat.h"
 #include "HttpHeaderTools.h"
 #include "MemBuf.h"
 #include "mgr/Registration.h"
+#include "mime_header.h"
 #include "profiler/Profiler.h"
 #include "rfc1123.h"
 #include "SquidConfig.h"
 #include "StatHist.h"
 #include "Store.h"
 #include "StrList.h"
 #include "TimeOrTag.h"
 #include "util.h"
 
 #include 
 
 /* XXX: the whole set of API managing the entries vector should be rethought
  *  after the parse4r-ng effort is complete.
  */
 
 /*
  * On naming conventions:
  *
  * HTTP/1.1 defines message-header as
  *
  * message-header = field-name ":" [ field-value ] CRLF
  * field-name = token
  * field-value= *( field-content | LWS )
  *
  * HTTP/1.1 does not give a name name a group of all message-headers in a message.
  * Squid 1.1 seems to refer to that group _plus_ start-line as "headers".
  *
  * HttpHeader is an object that represents all message-headers in a message.
  * HttpHeader does not manage start-line.
  *
@@ -289,60 +290,105 @@
 
 const HttpHeaderEntry *e;
 HttpHeaderPos pos = HttpHeaderInitPos;
 
 while ((e = fresh->getEntry())) {
 /* deny bad guys (ok to check for Http::HdrType::OTHER) here */
 
 if (skipUpdateHeader(e->id))
 continue;
 
 if (e->id != Http::HdrType::OTHER)
 delById(e->id);
 else
 delByName(e->name.termedBuf());
 }
 
 pos = HttpHeaderInitPos;
 while ((e = fresh->getEntry())) {
 /* deny bad guys (ok to check for Http::HdrType::OTHER) here */
 
 if (skipUpdateHeader(e->id))
 continue;
 
 debugs(55, 7, "Updating header '" << Http::HeaderLookupTable.lookup(e->id).name << "' in cached entry");
 
 addEntry(e->clone());
 }
 return true;
 }
 
+bool
+HttpHeader::Isolate(const char **parse_start, size_t l, const char **blk_start, const char **blk_end)
+{
+/*
+ * parse_start points to the first line of HTTP message *headers*,
+ * not including the request or status lines
+ */
+const size_t end = headersEnd(*parse_start, l);
+
+if (end) {
+*blk_start = *parse_start;
+*blk_end = *parse_start + end - 1;
+assert(**blk_end == '\n');
+// Point blk_end to the first character after the last header field.
+// In other words, blk_end should point to the CR?LF header 

Re: [squid-dev] [PATCH] Older response must not update

[Eduard Bagdasaryan]

2016-10-05 20:17 GMT+03:00 Amos Jeffries :

> Well, Alex and I are really disagreeeing on long-term things. As patch
> author on the spot for this that final choice is yours in regards to
> what goes in right now.
>
> If you could pick something and submit a final patch in the next few
> days I will apply it. I would really like to get this in the new
> releases this weekend (both 3.5 and 4.0).

I reverted that sub-struct to "Errors" since its flags mean a "failure"
in a way. Also attaching v3.5 port.


Eduard.
Squid should ignore a [revalidation] response with an older Date header.

Before this patch, Squid violated the RFC 7234 section 4 MUST
requirement: "When more than one suitable response is stored, a cache
MUST use the most recent response (as determined by the Date header
field)." This problem may be damaging in cache hierarchies where
parent caches may have different responses. Trusting the older response
may lead to excessive IMS requests, cache thrashing and other problems.

=== modified file 'src/HttpReply.cc'
--- src/HttpReply.cc	2016-09-23 18:14:24 +
+++ src/HttpReply.cc	2016-10-06 21:33:06 +
@@ -601,30 +601,38 @@
 
 // warning-value MUST end with quote
 if (p >= item && *p == '"') {
 const char *const warnDateEnd = p;
 --p;
 while (p >= item && *p != '"') --p; // find the next quote
 
 const char *warnDateBeg = p + 1;
 --p;
 while (p >= item && xisspace(*p)) --p; // skip whitespace
 
 if (p >= item && *p == '"' && warnDateBeg - p > 2) {
 // found warn-text
 String warnDate;
 warnDate.append(warnDateBeg, warnDateEnd - warnDateBeg);
 const time_t time = parse_rfc1123(warnDate.termedBuf());
 keep = (time > 0 && time == date); // keep valid and matching date
 }
 }
 
 if (keep) {
 if (newValue.size())
 newValue.append(", ");
 newValue.append(item, len);
 }
 }
 
 return newValue;
 }
 
+bool
+HttpReply::olderThan(const HttpReply *them) const
+{
+if (!them || !them->date || !date)
+return false;
+return date < them->date;
+}
+

=== modified file 'src/HttpReply.h'
--- src/HttpReply.h	2016-09-23 15:28:42 +
+++ src/HttpReply.h	2016-10-06 21:33:06 +
@@ -85,60 +85,64 @@
 HttpReply *make304() const;
 
 void redirect(Http::StatusCode, const char *);
 
 int64_t bodySize(const HttpRequestMethod&) const;
 
 /** Checks whether received body exceeds known maximum size.
  * Requires a prior call to calcMaxBodySize().
  */
 bool receivedBodyTooLarge(HttpRequest&, int64_t receivedBodySize);
 
 /** Checks whether expected body exceeds known maximum size.
  * Requires a prior call to calcMaxBodySize().
  */
 bool expectedBodyTooLarge(HttpRequest& request);
 
 int validatorsMatch (HttpReply const *other) const;
 
 void packHeadersInto(Packer * p) const;
 
 /** Clone this reply.
  *  Could be done as a copy-contructor but we do not want to accidently copy a HttpReply..
  */
 HttpReply *clone() const;
 
 /// Remove Warnings with warn-date different from Date value
 void removeStaleWarnings();
 
 virtual void hdrCacheInit();
 
+/// whether our Date header value is smaller than theirs
+/// \returns false if any information is missing
+bool olderThan(const HttpReply *them) const;
+
 private:
 /** initialize */
 void init();
 
 void clean();
 
 void hdrCacheClean();
 
 void packInto(Packer * p);
 
 /* ez-routines */
 /** \return construct 304 reply and pack it into a MemBuf */
 MemBuf *packed304Reply();
 
 /* header manipulation */
 time_t hdrExpirationTime();
 
 /** Calculates and stores maximum body size if needed.
  * Used by receivedBodyTooLarge() and expectedBodyTooLarge().
  */
 void calcMaxBodySize(HttpRequest& request) const;
 
 String removeStaleWarningValues(const String );
 
 mutable int64_t bodySizeMax; /**< cached result of calcMaxBodySize */
 
 protected:
 virtual void packFirstLineInto(Packer * p, bool) const { sline.packInto(p); }
 
 virtual bool parseFirstLine(const char *start, const char *end);

=== modified file 'src/LogTags.h'
--- src/LogTags.h	2016-01-01 00:14:27 +
+++ src/LogTags.h	2016-10-06 21:33:07 +
@@ -1,57 +1,58 @@
 /*
  * Copyright (C) 1996-2016 The Squid Software Foundation and contributors
  *
  * Squid software is distributed under GPLv2+ license and includes
  * contributions from numerous individuals and organizations.
  * Please see the COPYING and CONTRIBUTORS files for details.
  */
 
 #ifndef SQUID_SRC_LOGTAGS_H
 #define SQUID_SRC_LOGTAGS_H
 
 /** Squid transaction result code/tag set.
  *
  * These codes indicate how the request was received
  * and some details about its processing pathway.
  *
  * 

Re: [squid-dev] [PATCH] Handling syntactically valid requests with higher-than-supported HTTP versions

[Eduard Bagdasaryan]

> 2016-09-06 17:03 GMT+03:00 Amos Jeffries :
>
>  > > This patch fixes Squid to return 505 (Version Not Supported) error
>  > > code for HTTP/2+ versions.
>  > >
>  > > Before this change, when a syntactically valid HTTP/1 request 
indicated

>  > > HTTP major version 2, Squid mangled and forwarded the request as an
>  > > HTTP/1 message.
>  >
>  > That is not true. The trunk code returns false the same as this 
patched

>  > version. Because the version field for HTTP/2 does not start with the
>  > static magic string "HTTP/1.
>
> Disagree: the trunk Http::One::RequestParser::parseHttpVersionField()
> returns true, because it treats HTTP/2 message as HTTP/0.9
> message. You can see this in my Co-Advisor attachment in "HTTP/1.1
> device MUST NOT use 2.1 HTTP version in requests" test case.
>
>  > > Since Squid does not and cannot correctly interpret an
>  > > HTTP/1 request using HTTP/2 rules, returning an error and 
closing the

>  > > connection appears to be the only correct action possible.
>  >
>  > That is not correct either. The old logic requires method to be 
GET for

>  > full HTTP/0.9 handling. HTTP/2 pseudo-method is PRI. If anything other
>  > than PRI is seen it is not valid HTTP/2.
>  >  At most we should have logic passing HTTP/2 transparently through the
>  > proxy (configurable). Which is the relay behaviour of HTTP/0.9
>  > processing, but not its header mangling behaviour (because PRI, 
not GET

>  > method).
>
> Not sure that is what Http::One::RequestParser::parseHttpVersionField()
> should care of. In my understanding, this method should parse the input
> with HTTP/1 rules, returning false if the input does not conform to
> these rules.
>
>  > Also, HTTP/2 does not obsolete HTTP/1.x RFC's, so there is an explicit
>  > possibility that HTTP/1.2+ will exist at some point. The IETF WG
>  > expectation is/was that 1.2+ (if any) would contain compatibility for
>  > HTTP/2+ features over HTTP/1.x syntax.
>
> OK, and the patched Squid would return 505 (HTTP Version Not Supported)
> for HTTP/1.2+.(it is not yet implemented).
>
>  > Just noticed that the unit test for "GET / HTTP/2.0" request-line is
>  > also failing due to this parser change now accepting that invalid
>  > version as valid.
>
> I suspect that test should be changed to check for 505 error code instead
> of 400.
>
>  > You should use tok.skip(Http1::MagicPrefix) instead of trying to count
>  > digits in the major version.
>
> I count digits in order to differentiate syntactically valid(but
> unsupported by Squid) HTTP versions and invalid(multi-digit)
> ones, returning 505 or 400 codes respectively.


Just a reminder of this issue. Amos, is there anything to do
to move it forward? In my point of view, only the related unit test
is to be fixed, but probably I am still misunderstand something.

Thanks,
Eduard.
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] [PATCH] Null pointer dereference with Coverity

[Eduard Bagdasaryan]

Hello,

This patch addresses the following problem, disclosed by
recent Coverity scan:

> *** CID 1372977:  Null pointer dereferences  (FORWARD_NULL)
> /src/adaptation/icap/ModXact.cc: 1278 in
> Adaptation::Icap::ModXact::finalizeLogInfo()()
>  [ ... ]
>
>   CID 1372977:  Null pointer dereferences  (FORWARD_NULL)
> >>> Comparing "virgin_request_" to null implies that
> >>> "virgin_request_" might be null.
>  [ ... ]


Regards,
Eduard.
Coverity issue fix.

Adjusted Adaptation::Icap::ModXact::finalizeLogInfo(), fixing possible
"Null pointer dereference". 

=== modified file 'src/adaptation/icap/ModXact.cc'
--- src/adaptation/icap/ModXact.cc	2016-09-13 17:30:03 +
+++ src/adaptation/icap/ModXact.cc	2016-09-16 09:31:39 +
@@ -1240,69 +1240,69 @@
 Adaptation::Icap::ModXact::~ModXact()
 {
 delete bodyParser;
 }
 
 // internal cleanup
 void Adaptation::Icap::ModXact::swanSong()
 {
 debugs(93, 5, HERE << "swan sings" << status());
 
 stopWriting(false);
 stopSending(false);
 
 if (theInitiator.set()) // we have not sent the answer to the initiator
 detailError(ERR_DETAIL_ICAP_XACT_OTHER);
 
 // update adaptation history if start was called and we reserved a slot
 Adaptation::History::Pointer ah = virginRequest().adaptLogHistory();
 if (ah != NULL && adaptHistoryId >= 0)
 ah->recordXactFinish(adaptHistoryId);
 
 Adaptation::Icap::Xaction::swanSong();
 }
 
 void prepareLogWithRequestDetails(HttpRequest *, AccessLogEntry::Pointer &);
 
 void Adaptation::Icap::ModXact::finalizeLogInfo()
 {
 HttpRequest *adapted_request_ = nullptr;
 HttpReply *adapted_reply_ = nullptr;
-HttpRequest *virgin_request_ = (virgin.cause ? virgin.cause : dynamic_cast(virgin.header));
+HttpRequest *virgin_request_ = const_cast(());
 if (!(adapted_request_ = dynamic_cast(adapted.header))) {
 // if the request was not adapted, use virgin request to simplify
 // the code further below
 adapted_request_ = virgin_request_;
 adapted_reply_ = dynamic_cast(adapted.header);
 }
 
-Adaptation::Icap::History::Pointer h = (virgin_request_ ? virgin_request_->icapHistory() : NULL);
+Adaptation::Icap::History::Pointer h = virgin_request_->icapHistory();
 Must(h != NULL); // ICAPXaction::maybeLog calls only if there is a log
 al.icp.opcode = ICP_INVALID;
 al.url = h->log_uri.termedBuf();
 const Adaptation::Icap::ServiceRep   = service();
 al.icap.reqMethod = s.cfg().method;
 
 al.cache.caddr = virgin_request_->client_addr;
 
 al.request = virgin_request_;
 HTTPMSGLOCK(al.request);
 al.adapted_request = adapted_request_;
 HTTPMSGLOCK(al.adapted_request);
 
 if (adapted_reply_) {
 al.reply = adapted_reply_;
 HTTPMSGLOCK(al.reply);
 } else
 al.reply = NULL;
 
 if (h->rfc931.size())
 al.cache.rfc931 = h->rfc931.termedBuf();
 
 #if USE_OPENSSL
 if (h->ssluser.size())
 al.cache.ssluser = h->ssluser.termedBuf();
 #endif
 al.cache.code = h->logType;
 
 const HttpMsg *virgin_msg = dynamic_cast(virgin.header);
 if (!virgin_msg)

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Incorrect logging of request size

[Eduard Bagdasaryan]

2016-09-13 20:42 GMT+03:00 Alex Rousskov :
> Committed to trunk (r14838)

I am attaching v3.5 port of this r14838 and also r14839 and r14840,
containing several related fixes.


Eduard.
Fix logged request size (%http::>st) and other size-related %codes.

The %[http::]>st logformat code should log the actual number of
[dechunked] bytes received from the client. However, for requests with
Content-Length, Squid was logging the sum of the request header size and
the Content-Length header field value instead. The logged value was
wrong when the client sent fewer than Content-Length body bytes.

Also:

* Fixed %http::>h and %http::h logformat code, but since ICAP
  services deal with (and log) both HTTP requests and responses, that
  focus on the HTTP message kind actually complicates ICAP logging
  configuration and will eventually require introduction of new %http
  logformat codes that would be meaningless in access.log context.

  - In ICAP context, %http::>h should log to-be-adapted HTTP message
headers embedded in an ICAP request (HTTP request headers in REQMOD;
HTTP response headers in RESPMOD). Before this change, %http::>h
logged constant/"FYI" HTTP request headers in RESPMOD.

  - Similarly, %http::" and "<" characters
  should indicate ICAP message kind (where the being-logged HTTP message
  is embedded), not HTTP message kind, even for %http codes.

  TODO: %http::>h code logs "-" in RESPMOD because AccessLogEntry does
  not store virgin HTTP response headers.

* Correctly initialize ICAP ALE HTTP fields related to HTTP message
  sizes for icap_log, including %http::>st, %http::sh, and
  %http::>sh format codes.

* Initialize HttpMsg::hdr_sz in HTTP header parsing code. Among other
  uses, this field is needed to initialize HTTP fields inside ICAP ALE.

* Synced default icap_log format documentation with the current code.

This is a v3.5 port of trunk r14838, r14839 and r14840.

=== modified file 'src/adaptation/icap/ModXact.cc'
--- src/adaptation/icap/ModXact.cc	2016-01-01 00:14:27 +
+++ src/adaptation/icap/ModXact.cc	2016-09-15 10:48:05 +
@@ -1232,125 +1232,130 @@
 }
 
 Adaptation::Icap::ModXact::~ModXact()
 {
 delete bodyParser;
 }
 
 // internal cleanup
 void Adaptation::Icap::ModXact::swanSong()
 {
 debugs(93, 5, HERE << "swan sings" << status());
 
 stopWriting(false);
 stopSending(false);
 
 if (theInitiator.set()) // we have not sent the answer to the initiator
 detailError(ERR_DETAIL_ICAP_XACT_OTHER);
 
 // update adaptation history if start was called and we reserved a slot
 Adaptation::History::Pointer ah = virginRequest().adaptLogHistory();
 if (ah != NULL && adaptHistoryId >= 0)
 ah->recordXactFinish(adaptHistoryId);
 
 Adaptation::Icap::Xaction::swanSong();
 }
 
 void prepareLogWithRequestDetails(HttpRequest *, AccessLogEntry::Pointer &);
 
 void Adaptation::Icap::ModXact::finalizeLogInfo()
 {
-HttpRequest * request_ = NULL;
-HttpRequest * adapted_request_ = NULL;
-HttpReply * reply_ = NULL;
-request_ = (virgin.cause? virgin.cause: dynamic_cast(virgin.header));
+HttpRequest *adapted_request_ = nullptr;
+HttpReply *adapted_reply_ = nullptr;
+HttpRequest *virgin_request_ = (virgin.cause ? virgin.cause : dynamic_cast(virgin.header));
 if (!(adapted_request_ = dynamic_cast(adapted.header))) {
-adapted_request_ = request_;
-reply_ = dynamic_cast(adapted.header);
+// if the request was not adapted, use virgin request to simplify
+// the code further below
+adapted_request_ = virgin_request_;
+adapted_reply_ = dynamic_cast(adapted.header);
 }
 
-Adaptation::Icap::History::Pointer h = (request_ ? request_->icapHistory() : NULL);
+Adaptation::Icap::History::Pointer h = (virgin_request_ ? virgin_request_->icapHistory() : NULL);
 Must(h != NULL); // ICAPXaction::maybeLog calls only if there is a log
 al.icp.opcode = ICP_INVALID;
 al.url = h->log_uri.termedBuf();
 const Adaptation::Icap::ServiceRep   = service();
 al.icap.reqMethod = s.cfg().method;
 
-al.cache.caddr = request_->client_addr;
+al.cache.caddr = virgin_request_->client_addr;
 
-al.request = request_;
+al.request = virgin_request_;
 HTTPMSGLOCK(al.request);
 al.adapted_request = adapted_request_;
 HTTPMSGLOCK(al.adapted_request);
 
-if (reply_) {
-al.reply = reply_;
+if (adapted_reply_) {
+al.reply = adapted_reply_;
 HTTPMSGLOCK(al.reply);
 } else
 al.reply = NULL;
 
 if (h->rfc931.size())
 al.cache.rfc931 = h->rfc931.termedBuf();
 
 #if USE_OPENSSL
 if (h->ssluser.size())
 al.cache.ssluser = h->ssluser.termedBuf();
 #endif
 al.cache.code = h->logType;
-// XXX: should use icap-specific counters instead ?
-

Re: [squid-dev] [PATCH] Older response must not update

[Eduard Bagdasaryan]

2016-09-10 17:20 GMT+03:00 Amos Jeffries :

> Well, I'm still in slight disagreement with Alex on how to group things
> -though that is mostly because we have not discussed it properly.
>
> If a sub-struct name cannot be agreed then bool members in the LogTags
> object is better than calling a sub-struct "flags" and only putting
> flags for failure conditions into it.
>
> I leave the final choice to you Eduard, we can always change it again
> after later discussions.

If you think that "Flags" is too "neutral" for the sub-struct naming,
and it is questionable whether "ignored" flag can be numbered
as "Errors", we could name with, e.g., "Problems" or "Faults". Or
just leave "Errors", as it was before.


Eduard.
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Revalidate without Last-Modified

[Eduard Bagdasaryan]

Made a couple of fixes:

* avoid needless updates when a [revalidation] reply lacks Last-Modified

* compilation errors while running test-builds.sh

Also attached a port to v3.5.


Eduard.
Bug 4471: revalidation doesn't work when expired cached object lacks Last-Modified.

The bug was caused by the fact that Squid used only Last-Modified header
value for evaluating entry's last modification time while making an
internal revalidation request. So, without Last-Modified it was not
possible to correctly fill If-Modified-Since header value. As a result,
the revalidation request was not initiated when it should be.

To fix this problem, we should generate If-Modified-Since based on other
data, showing how "old" the cached response is, such as Date and Age
header values. Both Date and Age header values are utilized while
calculating entry's timestamp. Therefore, we should use the timestamp if
Last-Modified is unavailable.

TODO: HttpRequest::imslen support looks broken/deprecated. Using this
field inside StoreEntry::modifiedSince() leads to several useless checks
and probably may cause other [hidden] problems.

=== modified file 'src/MemStore.cc'
--- src/MemStore.cc	2016-05-01 21:37:52 +
+++ src/MemStore.cc	2016-09-08 14:11:18 +
@@ -429,61 +429,61 @@
 // already disconnected from the cache, no need to update
 if (index < 0)
 return true;
 
 if (!map)
 return false;
 
 const Ipc::StoreMapAnchor  = map->readableEntry(index);
 return updateCollapsedWith(collapsed, index, anchor);
 }
 
 /// updates collapsed entry after its anchor has been located
 bool
 MemStore::updateCollapsedWith(StoreEntry , const sfileno index, const Ipc::StoreMapAnchor )
 {
 collapsed.swap_file_sz = anchor.basics.swap_file_sz;
 const bool copied = copyFromShm(collapsed, index, anchor);
 return copied;
 }
 
 /// anchors StoreEntry to an already locked map entry
 void
 MemStore::anchorEntry(StoreEntry , const sfileno index, const Ipc::StoreMapAnchor )
 {
 const Ipc::StoreMapAnchor::Basics  = anchor.basics;
 
 e.swap_file_sz = basics.swap_file_sz;
 e.lastref = basics.lastref;
 e.timestamp = basics.timestamp;
 e.expires = basics.expires;
-e.lastmod = basics.lastmod;
+e.lastModified(basics.lastmod);
 e.refcount = basics.refcount;
 e.flags = basics.flags;
 
 assert(e.mem_obj);
 if (anchor.complete()) {
 e.store_status = STORE_OK;
 e.mem_obj->object_sz = e.swap_file_sz;
 e.setMemStatus(IN_MEMORY);
 } else {
 e.store_status = STORE_PENDING;
 assert(e.mem_obj->object_sz < 0);
 e.setMemStatus(NOT_IN_MEMORY);
 }
 assert(e.swap_status == SWAPOUT_NONE); // set in StoreEntry constructor
 e.ping_status = PING_NONE;
 
 EBIT_CLR(e.flags, RELEASE_REQUEST);
 EBIT_CLR(e.flags, KEY_PRIVATE);
 EBIT_SET(e.flags, ENTRY_VALIDATED);
 
 MemObject::MemCache  = e.mem_obj->memCache;
 mc.index = index;
 mc.io = MemObject::ioReading;
 }
 
 /// copies the entire entry from shared to local memory
 bool
 MemStore::copyFromShm(StoreEntry , const sfileno index, const Ipc::StoreMapAnchor )
 {
 debugs(20, 7, "mem-loading entry " << index << " from " << anchor.start);

=== modified file 'src/Store.h'
--- src/Store.h	2016-07-23 13:36:56 +
+++ src/Store.h	2016-09-08 22:28:50 +
@@ -104,78 +104,89 @@
 const char *url() const;
 /// Satisfies cachability requirements shared among disk and RAM caches.
 /// Encapsulates common checks of mayStartSwapOut() and memoryCachable().
 /// TODO: Rename and make private so only those two methods can call this.
 bool checkCachable();
 int checkNegativeHit() const;
 int locked() const;
 int validToSend() const;
 bool memoryCachable(); ///< checkCachable() and can be cached in memory
 
 /// if needed, initialize mem_obj member w/o URI-related information
 MemObject *makeMemObject();
 
 /// initialize mem_obj member (if needed) and supply URI-related info
 void createMemObject(const char *storeId, const char *logUri, const HttpRequestMethod );
 
 void dump(int debug_lvl) const;
 void hashDelete();
 void hashInsert(const cache_key *);
 void registerAbort(STABH * cb, void *);
 void reset();
 void setMemStatus(mem_status_t);
 bool timestampsSet();
 void unregisterAbort();
 void destroyMemObject();
 int checkTooSmall();
 
 void delayAwareRead(const Comm::ConnectionPointer , char *buf, int len, AsyncCall::Pointer callback);
 
 void setNoDelay (bool const);
-bool modifiedSince(HttpRequest * request) const;
+void lastModified(const time_t when) { lastModified_ = when; }
+/// \returns entry's 'effective' modification time
+time_t lastModified() const {
+// may still return -1 if timestamp is not set
+return lastModified_ < 0 ? timestamp : lastModified_;
+}
+/// \returns a formatted string with entry's timestamps
+const 

Re: [squid-dev] [PATCH] Handling syntactically valid requests with higher-than-supported HTTP versions

[Eduard Bagdasaryan]

2016-09-06 17:03 GMT+03:00 Amos Jeffries :

> > This patch fixes Squid to return 505 (Version Not Supported) error
> > code for HTTP/2+ versions.
> >
> > Before this change, when a syntactically valid HTTP/1 request indicated
> > HTTP major version 2, Squid mangled and forwarded the request as an
> > HTTP/1 message.
>
> That is not true. The trunk code returns false the same as this patched
> version. Because the version field for HTTP/2 does not start with the
> static magic string "HTTP/1.

Disagree: the trunk Http::One::RequestParser::parseHttpVersionField()
returns true, because it treats HTTP/2 message as HTTP/0.9
message. You can see this in my Co-Advisor attachment in "HTTP/1.1
device MUST NOT use 2.1 HTTP version in requests" test case.

> > Since Squid does not and cannot correctly interpret an
> > HTTP/1 request using HTTP/2 rules, returning an error and closing the
> > connection appears to be the only correct action possible.
>
> That is not correct either. The old logic requires method to be GET for
> full HTTP/0.9 handling. HTTP/2 pseudo-method is PRI. If anything other
> than PRI is seen it is not valid HTTP/2.
>  At most we should have logic passing HTTP/2 transparently through the
> proxy (configurable). Which is the relay behaviour of HTTP/0.9
> processing, but not its header mangling behaviour (because PRI, not GET
> method).

Not sure that is what Http::One::RequestParser::parseHttpVersionField()
should care of. In my understanding, this method should parse the input
with HTTP/1 rules, returning false if the input does not conform to
these rules.

> Also, HTTP/2 does not obsolete HTTP/1.x RFC's, so there is an explicit
> possibility that HTTP/1.2+ will exist at some point. The IETF WG
> expectation is/was that 1.2+ (if any) would contain compatibility for
> HTTP/2+ features over HTTP/1.x syntax.

OK, and the patched Squid would return 505 (HTTP Version Not Supported)
for HTTP/1.2+.(it is not yet implemented).

> Just noticed that the unit test for "GET / HTTP/2.0" request-line is
> also failing due to this parser change now accepting that invalid
> version as valid.

I suspect that test should be changed to check for 505 error code instead
of 400.

> You should use tok.skip(Http1::MagicPrefix) instead of trying to count
> digits in the major version.

I count digits in order to differentiate syntactically valid(but
unsupported by Squid) HTTP versions and invalid(multi-digit)
ones, returning 505 or 400 codes respectively.


Eduard.

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Must revalidate CC:no-cache responses

[Eduard Bagdasaryan]

2016-09-05 13:38 GMT+03:00 Amos Jeffries :
> However we still have people wanting the nasty refresh_pattern
> ignore-private option. In order to minimize the security issues that
> causes anything marked as CC:private that does get into cache needs to
> be revalidated on every use just like CC:no-cache.

Ok, thank you.

2016-09-05 13:38 GMT+03:00 Amos Jeffries :
> * ccPrivate is only cacheable in the same conditions as
> ccNoCacheNoParams so should be a ENTRY_REVALIDATE_ALWAYS as well.
>  - this also affects the debugs() in src/refresh.cc syign why it needs
> to revalidate.

Updated the patch with these remarks.

Eduard.
MUST always revalidate Cache-Control:no-cache responses.

Squid MUST NOT use a CC:no-cache response for satisfying subsequent
requests without successful validation with the origin server. Squid
violated this MUST because Squid mapped both "no-cache" and
"must-revalidate"/etc. directives to the same ENTRY_REVALIDATE flag
which was interpreted as "revalidate when the cached response becomes
stale" rather than "revalidate on every request".

This patch splits ENTRY_REVALIDATE into:
- ENTRY_REVALIDATE_ALWAYS, for entries with CC:no-cache and
- ENTRY_REVALIDATE_STALE, for entries with other related CC directives
  like CC:must-revalidate and CC:proxy-revalidate.

Reusing ENTRY_CACHABLE_RESERVED_FOR_FUTURE_USE value for the more
forgiving ENTRY_REVALIDATE_STALE (instead of the more aggressive
ENTRY_REVALIDATE_ALWAYS) fixes the bug for the already cached entries -
they will be revalidated and stored with the correct flag on the next
request.

=== modified file 'src/enums.h'
--- src/enums.h	2016-04-01 17:54:10 +
+++ src/enums.h	2016-09-02 21:17:57 +
@@ -40,65 +40,65 @@
 PING_NONE,
 PING_WAITING,
 PING_DONE
 } ping_status_t;
 
 typedef enum {
 STORE_OK,
 STORE_PENDING
 } store_status_t;
 
 typedef enum {
 SWAPOUT_NONE,
 SWAPOUT_WRITING,
 SWAPOUT_DONE
 } swap_status_t;
 
 typedef enum {
 STORE_NON_CLIENT,
 STORE_MEM_CLIENT,
 STORE_DISK_CLIENT
 } store_client_t;
 
 /*
  * These are for StoreEntry->flag, which is defined as a SHORT
  *
  * NOTE: These flags are written to swap.state, so think very carefully
  * about deleting or re-assigning!
  */
 enum {
 ENTRY_SPECIAL,
-ENTRY_REVALIDATE,
+ENTRY_REVALIDATE_ALWAYS,
 DELAY_SENDING,
 RELEASE_REQUEST,
 REFRESH_REQUEST,
-ENTRY_CACHABLE_RESERVED_FOR_FUTURE_USE,
+ENTRY_REVALIDATE_STALE,
 ENTRY_DISPATCHED,
 KEY_PRIVATE,
 ENTRY_FWD_HDR_WAIT,
 ENTRY_NEGCACHED,
 ENTRY_VALIDATED,
 ENTRY_BAD_LENGTH,
 ENTRY_ABORTED
 };
 
 /*
  * These are for client Streams. Each node in the stream can be queried for
  * its status
  */
 typedef enum {
 STREAM_NONE,/* No particular status */
 STREAM_COMPLETE,/* All data has been flushed, no more reads allowed */
 /* an unpredicted end has occured, no more
  * reads occured, but no need to tell
  * downstream that an error occured
  */
 STREAM_UNPLANNED_COMPLETE,
 /* An error has occured in this node or an above one,
  * and the node is not generating an error body / it's
  * midstream
  */
 STREAM_FAILED
 } clientStream_status_t;
 
 /* stateful helper callback response codes */
 typedef enum {

=== modified file 'src/http.cc'
--- src/http.cc	2016-08-15 11:26:37 +
+++ src/http.cc	2016-09-05 09:12:50 +
@@ -961,72 +961,74 @@
 if (Config.negativeTtl > 0)
 entry->cacheNegatively();
 else
 #endif
 entry->makePrivate();
 break;
 
 default:
 assert(0);
 break;
 }
 }
 
 if (!ignoreCacheControl) {
 if (rep->cache_control) {
 // We are required to revalidate on many conditions.
 // For security reasons we do so even if storage was caused by refresh_pattern ignore-* option
 
 // CC:must-revalidate or CC:proxy-revalidate
 const bool ccMustRevalidate = (rep->cache_control->proxyRevalidate() || rep->cache_control->mustRevalidate());
 
 // CC:no-cache (only if there are no parameters)
 const bool ccNoCacheNoParams = (rep->cache_control->hasNoCache() && rep->cache_control->noCache().size()==0);
 
 // CC:s-maxage=N
 const bool ccSMaxAge = rep->cache_control->hasSMaxAge();
 
 // CC:private (yes, these can sometimes be stored)
 const bool ccPrivate = rep->cache_control->hasPrivate();
 
-if (ccMustRevalidate || ccNoCacheNoParams || ccSMaxAge || ccPrivate)
-EBIT_SET(entry->flags, ENTRY_REVALIDATE);
+if (ccNoCacheNoParams || ccPrivate)
+EBIT_SET(entry->flags, ENTRY_REVALIDATE_ALWAYS);
+else if (ccMustRevalidate || ccSMaxAge)
+EBIT_SET(entry->flags, ENTRY_REVALIDATE_STALE);
 }
 #if 

Re: [squid-dev] [PATCH] Must revalidate CC:no-cache responses

[Eduard Bagdasaryan]

2016-09-04 18:31 GMT+03:00 Amos Jeffries :

> * ccPrivate is only cacheable in the same conditions as
> ccNoCacheNoParams so should be a ENTRY_REVALIDATE_ALWAYS as well

It is unclear what are these "same" conditions. RFC 7234 5.2.2.6:

   The "private" response directive indicates that the response message
   is intended for a single user and MUST NOT be stored by a shared
   cache.

In my understanding Squid (as a shared cache) must not store "private"
responses at all (while user agents could). Is this correct? If yes,
currently Squid violates this MUST.

On the other hand, "no-cache" without field-names does not impose
constraints on storing in the cache, but restricts the cache to always
revalidate.


Eduard.
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] [PATCH] Must revalidate CC:no-cache responses

[Eduard Bagdasaryan]

Hello,

This patch forces Squid to always revalidate Cache-Control:no-cache
responses.

Squid MUST NOT use a CC:no-cache response for satisfying subsequent
requests without successful validation with the origin server. Squid
violated this MUST because Squid mapped both "no-cache" and
"must-revalidate"/etc. directives to the same ENTRY_REVALIDATE flag
which was interpreted as "revalidate when the cached response becomes
stale" rather than "revalidate on every request".

This patch splits ENTRY_REVALIDATE into:
- ENTRY_REVALIDATE_ALWAYS, for entries with CC:no-cache and
- ENTRY_REVALIDATE_STALE, for entries with other related CC directives
  like CC:must-revalidate and CC:proxy-revalidate.

Reusing ENTRY_CACHABLE_RESERVED_FOR_FUTURE_USE value for the more
forgiving ENTRY_REVALIDATE_STALE (instead of the more aggressive
ENTRY_REVALIDATE_ALWAYS) fixes the bug for the already cached entries -
they will be revalidated and stored with the correct flag on the next
request.


Regards,
Eduard.

MUST always revalidate Cache-Control:no-cache responses.

Squid MUST NOT use a CC:no-cache response for satisfying subsequent
requests without successful validation with the origin server. Squid
violated this MUST because Squid mapped both "no-cache" and
"must-revalidate"/etc. directives to the same ENTRY_REVALIDATE flag
which was interpreted as "revalidate when the cached response becomes
stale" rather than "revalidate on every request".

This patch splits ENTRY_REVALIDATE into:
- ENTRY_REVALIDATE_ALWAYS, for entries with CC:no-cache and
- ENTRY_REVALIDATE_STALE, for entries with other related CC directives
  like CC:must-revalidate and CC:proxy-revalidate.

Reusing ENTRY_CACHABLE_RESERVED_FOR_FUTURE_USE value for the more
forgiving ENTRY_REVALIDATE_STALE (instead of the more aggressive
ENTRY_REVALIDATE_ALWAYS) fixes the bug for the already cached entries -
they will be revalidated and stored with the correct flag on the next
request.

=== modified file 'src/enums.h'
--- src/enums.h	2016-04-01 17:54:10 +
+++ src/enums.h	2016-09-02 21:17:57 +
@@ -40,65 +40,65 @@
 PING_NONE,
 PING_WAITING,
 PING_DONE
 } ping_status_t;
 
 typedef enum {
 STORE_OK,
 STORE_PENDING
 } store_status_t;
 
 typedef enum {
 SWAPOUT_NONE,
 SWAPOUT_WRITING,
 SWAPOUT_DONE
 } swap_status_t;
 
 typedef enum {
 STORE_NON_CLIENT,
 STORE_MEM_CLIENT,
 STORE_DISK_CLIENT
 } store_client_t;
 
 /*
  * These are for StoreEntry->flag, which is defined as a SHORT
  *
  * NOTE: These flags are written to swap.state, so think very carefully
  * about deleting or re-assigning!
  */
 enum {
 ENTRY_SPECIAL,
-ENTRY_REVALIDATE,
+ENTRY_REVALIDATE_ALWAYS,
 DELAY_SENDING,
 RELEASE_REQUEST,
 REFRESH_REQUEST,
-ENTRY_CACHABLE_RESERVED_FOR_FUTURE_USE,
+ENTRY_REVALIDATE_STALE,
 ENTRY_DISPATCHED,
 KEY_PRIVATE,
 ENTRY_FWD_HDR_WAIT,
 ENTRY_NEGCACHED,
 ENTRY_VALIDATED,
 ENTRY_BAD_LENGTH,
 ENTRY_ABORTED
 };
 
 /*
  * These are for client Streams. Each node in the stream can be queried for
  * its status
  */
 typedef enum {
 STREAM_NONE,/* No particular status */
 STREAM_COMPLETE,/* All data has been flushed, no more reads allowed */
 /* an unpredicted end has occured, no more
  * reads occured, but no need to tell
  * downstream that an error occured
  */
 STREAM_UNPLANNED_COMPLETE,
 /* An error has occured in this node or an above one,
  * and the node is not generating an error body / it's
  * midstream
  */
 STREAM_FAILED
 } clientStream_status_t;
 
 /* stateful helper callback response codes */
 typedef enum {

=== modified file 'src/http.cc'
--- src/http.cc	2016-08-15 11:26:37 +
+++ src/http.cc	2016-09-02 19:44:34 +
@@ -961,72 +961,74 @@
 if (Config.negativeTtl > 0)
 entry->cacheNegatively();
 else
 #endif
 entry->makePrivate();
 break;
 
 default:
 assert(0);
 break;
 }
 }
 
 if (!ignoreCacheControl) {
 if (rep->cache_control) {
 // We are required to revalidate on many conditions.
 // For security reasons we do so even if storage was caused by refresh_pattern ignore-* option
 
 // CC:must-revalidate or CC:proxy-revalidate
 const bool ccMustRevalidate = (rep->cache_control->proxyRevalidate() || rep->cache_control->mustRevalidate());
 
 // CC:no-cache (only if there are no parameters)
 const bool ccNoCacheNoParams = (rep->cache_control->hasNoCache() && rep->cache_control->noCache().size()==0);
 
 // CC:s-maxage=N
 const bool ccSMaxAge = rep->cache_control->hasSMaxAge();
 
 // CC:private (yes, these can sometimes be stored)
 const bool ccPrivate = rep->cache_control->hasPrivate();
 
-if (ccMustRevalidate || ccNoCacheNoParams || ccSMaxAge || 

[squid-dev] [PATCH] Incorrect logging of request size

[Eduard Bagdasaryan]

Hello,

This patch fixes logged request size (%http::>st) and other size-related
%codes.

The %[http::]>st logformat code should log the actual number of
[dechunked] bytes received from the client. However, for requests with
Content-Length, Squid was logging the sum of the request header size and
the Content-Length header field value instead. The logged value was
wrong when the client sent fewer than Content-Length body bytes.

Also:

* Fixed %http::>h and %http::h logformat code, but since ICAP
  services deal with (and log) both HTTP requests and responses, that
  focus on the HTTP message kind actually complicates ICAP logging
  configuration and will eventually require introduction of new %http
  logformat codes that would be meaningless in access.log context.

  - In ICAP context, %http::>h should log to-be-adapted HTTP message
headers embedded in an ICAP request (HTTP request headers in REQMOD;
HTTP response headers in RESPMOD). Before this change, %http::>h
logged constant/"FYI" HTTP request headers in RESPMOD.

  - Similarly, %http::" and "<" characters
  should indicate ICAP message kind (where the being-logged HTTP message
  is embedded), not HTTP message kind, even for %http codes.

  TODO: %http::>h code logs "-" in RESPMOD because AccessLogEntry does
  not store virgin HTTP response headers.

* Correctly initialize ICAP ALE HTTP fields related to HTTP message
  sizes for icap_log, including %http::>st, %http::sh, and
  %http::>sh format codes.

* Initialize HttpMsg::hdr_sz in HTTP header parsing code. Among other
  uses, this field is needed to initialize HTTP fields inside ICAP ALE.

* Synced default icap_log format documentation with the current code.


Regards,
Eduard.
Fix logged request size (%http::>st), other size-related %codes.

The %[http::]>st logformat code should log the actual number of
[dechunked] bytes received from the client. However, for requests with
Content-Length, Squid was logging the sum of the request header size and
the Content-Length header field value instead. The logged value was
wrong when the client sent fewer than Content-Length body bytes.

Also:

* Fixed %http::>h and %http::h logformat code, but since ICAP
  services deal with (and log) both HTTP requests and responses, that
  focus on the HTTP message kind actually complicates ICAP logging
  configuration and will eventually require introduction of new %http
  logformat codes that would be meaningless in access.log context.

  - In ICAP context, %http::>h should log to-be-adapted HTTP message
headers embedded in an ICAP request (HTTP request headers in REQMOD;
HTTP response headers in RESPMOD). Before this change, %http::>h
logged constant/"FYI" HTTP request headers in RESPMOD.

  - Similarly, %http::" and "<" characters
  should indicate ICAP message kind (where the being-logged HTTP message
  is embedded), not HTTP message kind, even for %http codes.

  TODO: %http::>h code logs "-" in RESPMOD because AccessLogEntry does
  not store virgin HTTP response headers.

* Correctly initialize ICAP ALE HTTP fields related to HTTP message
  sizes for icap_log, including %http::>st, %http::sh, and
  %http::>sh format codes.

* Initialize HttpMsg::hdr_sz in HTTP header parsing code. Among other
  uses, this field is needed to initialize HTTP fields inside ICAP ALE.

* Synced default icap_log format documentation with the current code.

=== modified file 'src/HttpMsg.cc'
--- src/HttpMsg.cc	2016-03-21 04:48:44 +
+++ src/HttpMsg.cc	2016-07-30 14:48:54 +
@@ -287,60 +287,62 @@
 
 if (!header.parse(blk_start, blk_end-blk_start)) {
 PROF_stop(HttpMsg_httpMsgParseStep);
 return httpMsgParseError();
 }
 
 hdrCacheInit();
 
 *parse_end_ptr = parse_start;
 
 hdr_sz = *parse_end_ptr - buf;
 
 ++pstate;
 }
 
 PROF_stop(HttpMsg_httpMsgParseStep);
 return 1;
 }
 
 bool
 HttpMsg::parseHeader(Http1::Parser )
 {
 // HTTP/1 message contains "zero or more header fields"
 // zero does not need parsing
 // XXX: c_str() reallocates. performance regression.
 if (hp.headerBlockSize() && !header.parse(hp.mimeHeader().c_str(), hp.headerBlockSize())) {
 pstate = psError;
 return false;
 }
 
+// XXX: we are just parsing HTTP headers, not the whole message prefix here
+hdr_sz = hp.messageHeaderSize();
 pstate = psParsed;
 hdrCacheInit();
 return true;
 }
 
 /* handy: resets and returns -1 */
 int
 HttpMsg::httpMsgParseError()
 {
 reset();
 return -1;
 }
 
 void
 HttpMsg::setContentLength(int64_t clen)
 {
 header.delById(Http::HdrType::CONTENT_LENGTH); // if any
 header.putInt64(Http::HdrType::CONTENT_LENGTH, clen);
 content_length = clen;
 }
 
 bool
 HttpMsg::persistent() const
 {
 if (http_ver > Http::ProtocolVersion(1,0)) {
 /*
  * for modern versions of HTTP: persistent unless there is
  * a 

Re: [squid-dev] [PATCH] Revalidate without Last-Modified

[Eduard Bagdasaryan]

2016-08-28 1:12 GMT+03:00 Alex Rousskov :

> I worry about the message recipient comparing received
> effective LMT with the actual (absent!) LMT of the object they have in
> the cache and then deciding that the resource has changed (and their
> cached copy is now stale) because the resource now has LMT and it did
> not have it before.
>
> Not all HTCP clients are Squids, but how does Squid code treat such an
> HTCP TST response? In other words, will Squid itself make a different
> decision if it receives an effective LMT header value instead of no LMT
> header field at all? Does your own patch affect the receiving Squid
> behavior?

It seems that Squid does not care whether HTCP response contains LMT or
not. Moreover, it looks that all TST response headers are only parsed
inside htcpHandleTstResponse() without using them further for making
any decisions (probably they just intended to detect parse errors?).
Thus I see no problems for Squid as HTCP client when receiving effective
LMT in HTCP responses.


Eduard.
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Revalidate without Last-Modified

[Eduard Bagdasaryan]

2016-08-25 18:52 GMT+03:00 Alex Rousskov :

> 3. Sending an HTCP message to another service.
>
> > -hdr.putTime(Http::HdrType::LAST_MODIFIED, e->lastmod);
> > +if (e && e->lastModified() > -1)
> > +hdr.putTime(Http::HdrType::LAST_MODIFIED, 
e->lastModified());

>
> Is this a conditional/revalidation request? If yes, then should we use
> an effective modification time instead, like we do in use case #1?

I have not found what is the "conditional/revalidation" request from 
HTCP point
of view. The code snippet is taken from htcpTstReply(), serving replies 
for TST

requests (TST - test for the presence in the cache). If the entry is cached,
this method fills HTCP reply packet with some of cached entry's headers 
(Age,
Expires, Last-Modified etc.). According to the documentation, these 
headers are
added in order to give client an ability to calculate object's freshness 
itself:
HTCP response does not provide such information explicitly. On the other 
hand, I
have not found any strict requirements of what HTTP headers HTCP reply 
should

contain.

Would the "effective" Last-Modified information help HTCP client to 
calculate
object's freshness? It looks that some more information in this case is 
better

that lack of it.


Eduard.
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Older response must not update

[Eduard Bagdasaryan]

2016-08-25 20:05 GMT+03:00 Alex Rousskov :

> The "_IGNORED" flag Amos is talking about should go into LogTags::Errors.
> I recommend renaming and re-documenting that subclass

Updated the patch with your suggestions.


Eduard.

Squid should ignore a [revalidation] response with an older Date header.

Before this patch, Squid violated the RFC 7234 section 4 MUST
requirement: "When more than one suitable response is stored, a cache
MUST use the most recent response (as determined by the Date header
field)." This problem may be damaging in cache hierarchies where
parent caches may have different responses. Trusting the older response
may lead to excessive IMS requests, cache thrashing and other problems.

=== modified file 'src/HttpReply.cc'
--- src/HttpReply.cc	2016-07-23 13:36:56 +
+++ src/HttpReply.cc	2016-08-25 14:12:11 +
@@ -600,30 +600,37 @@
 
 // warning-value MUST end with quote
 if (p >= item && *p == '"') {
 const char *const warnDateEnd = p;
 --p;
 while (p >= item && *p != '"') --p; // find the next quote
 
 const char *warnDateBeg = p + 1;
 --p;
 while (p >= item && xisspace(*p)) --p; // skip whitespace
 
 if (p >= item && *p == '"' && warnDateBeg - p > 2) {
 // found warn-text
 String warnDate;
 warnDate.append(warnDateBeg, warnDateEnd - warnDateBeg);
 const time_t time = parse_rfc1123(warnDate.termedBuf());
 keep = (time > 0 && time == date); // keep valid and matching date
 }
 }
 
 if (keep) {
 if (newValue.size())
 newValue.append(", ");
 newValue.append(item, len);
 }
 }
 
 return newValue;
 }
 
+bool
+HttpReply::olderThan(const HttpReply *them) const
+{
+if (!them || !them->date || !date)
+return false;
+return date < them->date;
+}

=== modified file 'src/HttpReply.h'
--- src/HttpReply.h	2016-07-23 13:36:56 +
+++ src/HttpReply.h	2016-08-25 14:12:11 +
@@ -85,60 +85,64 @@
 HttpReply *make304() const;
 
 void redirect(Http::StatusCode, const char *);
 
 int64_t bodySize(const HttpRequestMethod&) const;
 
 /** Checks whether received body exceeds known maximum size.
  * Requires a prior call to calcMaxBodySize().
  */
 bool receivedBodyTooLarge(HttpRequest&, int64_t receivedBodySize);
 
 /** Checks whether expected body exceeds known maximum size.
  * Requires a prior call to calcMaxBodySize().
  */
 bool expectedBodyTooLarge(HttpRequest& request);
 
 int validatorsMatch (HttpReply const *other) const;
 
 void packHeadersInto(Packable * p) const;
 
 /** Clone this reply.
  *  Could be done as a copy-contructor but we do not want to accidently copy a HttpReply..
  */
 HttpReply *clone() const;
 
 /// Remove Warnings with warn-date different from Date value
 void removeStaleWarnings();
 
 virtual void hdrCacheInit();
 
+/// whether our Date header value is smaller than theirs
+/// \returns false if any information is missing
+bool olderThan(const HttpReply *them) const;
+
 private:
 /** initialize */
 void init();
 
 void clean();
 
 void hdrCacheClean();
 
 void packInto(Packable * p) const;
 
 /* ez-routines */
 /** \return construct 304 reply and pack it into a MemBuf */
 MemBuf *packed304Reply() const;
 
 /* header manipulation */
 time_t hdrExpirationTime();
 
 /** Calculates and stores maximum body size if needed.
  * Used by receivedBodyTooLarge() and expectedBodyTooLarge().
  */
 void calcMaxBodySize(HttpRequest& request) const;
 
 String removeStaleWarningValues(const String );
 
 mutable int64_t bodySizeMax; /**< cached result of calcMaxBodySize */
 
 protected:
 virtual void packFirstLineInto(Packable * p, bool) const { sline.packInto(p); }
 
 virtual bool parseFirstLine(const char *start, const char *end);

=== modified file 'src/LogTags.cc'
--- src/LogTags.cc	2016-06-02 09:49:19 +
+++ src/LogTags.cc	2016-08-26 12:09:16 +
@@ -28,52 +28,55 @@
 "TCP_DENIED_REPLY",
 "TCP_OFFLINE_HIT",
 "TCP_REDIRECT",
 "TCP_TUNNEL",
 "UDP_HIT",
 "UDP_MISS",
 "UDP_DENIED",
 "UDP_INVALID",
 "UDP_MISS_NOFETCH",
 "ICP_QUERY",
 "TYPE_MAX"
 };
 
 /*
  * This method is documented in http://wiki.squid-cache.org/SquidFaq/SquidLogs#Squid_result_codes
  * Please keep the wiki up to date
  */
 const char *
 LogTags::c_str() const
 {
 static char buf[1024];
 *buf = 0;
 int pos = 0;
 
 // source tags
 if (oldType && oldType < LOG_TYPE_MAX)
 pos += snprintf(buf, sizeof(buf), "%s",Str_[oldType]);
 else
 pos += snprintf(buf, sizeof(buf), "NONE");
 
+if (flags.ignored)
+pos += snprintf(buf+pos,sizeof(buf)-pos, "_IGNORED");
+
 

Re: [squid-dev] [PATCH] Older response must not update

[Eduard Bagdasaryan]

2016-08-24 18:20 GMT+03:00 Amos Jeffries :

> in src/LogTags.cc:
> * instead of adding new enum entry please extend LogTags with a new bool
> flag and the c_str() to append the "IGNORED" when that flag is true.
>  - TCP_REFRESH should be set when refresh was started.
>  - whoever merges this patch will also need to update the wiki
> SquidFaq/SquidLogs page to document what "IGNORED" means.

Added a new LogTags::ignored for this. I wonder whether the
LogTags assignment operator should copy this member too.

> in src/HttpReply.cc:
> * please use Squid syntax.
>   -  return type on separate line above the method/function name.
>
> in src/HttpReply.h:
> * please use doxygen syntax "\returns" instead of "returns" in the
> comment text.
> in src/client_side_reply.cc:
> * please remove the "handleIMSReply: " bit from touched debugs messages.

2016-08-24 22:05 GMT+03:00 Alex Rousskov :

> If Eduard does that, he should remove that bit from all related debugs
> messages to keep them all consistent IMO.


Addressed these comments and updated the patch.


Eduard.
Squid should ignore a [revalidation] response with an older Date header.

Before this patch, Squid violated the RFC 7234 section 4 MUST
requirement: "When more than one suitable response is stored, a cache
MUST use the most recent response (as determined by the Date header
field)." This problem may be damaging in cache hierarchies where
parent caches may have different responses. Trusting the older response
may lead to excessive IMS requests, cache thrashing and other problems.

=== modified file 'src/HttpReply.cc'
--- src/HttpReply.cc	2016-07-23 13:36:56 +
+++ src/HttpReply.cc	2016-08-25 10:16:52 +
@@ -600,30 +600,37 @@
 
 // warning-value MUST end with quote
 if (p >= item && *p == '"') {
 const char *const warnDateEnd = p;
 --p;
 while (p >= item && *p != '"') --p; // find the next quote
 
 const char *warnDateBeg = p + 1;
 --p;
 while (p >= item && xisspace(*p)) --p; // skip whitespace
 
 if (p >= item && *p == '"' && warnDateBeg - p > 2) {
 // found warn-text
 String warnDate;
 warnDate.append(warnDateBeg, warnDateEnd - warnDateBeg);
 const time_t time = parse_rfc1123(warnDate.termedBuf());
 keep = (time > 0 && time == date); // keep valid and matching date
 }
 }
 
 if (keep) {
 if (newValue.size())
 newValue.append(", ");
 newValue.append(item, len);
 }
 }
 
 return newValue;
 }
 
+bool
+HttpReply::olderThan(const HttpReply *them) const
+{
+if (!them || !them->date || !date)
+return false;
+return date < them->date;
+}

=== modified file 'src/HttpReply.h'
--- src/HttpReply.h	2016-07-23 13:36:56 +
+++ src/HttpReply.h	2016-08-25 11:08:10 +
@@ -85,60 +85,64 @@
 HttpReply *make304() const;
 
 void redirect(Http::StatusCode, const char *);
 
 int64_t bodySize(const HttpRequestMethod&) const;
 
 /** Checks whether received body exceeds known maximum size.
  * Requires a prior call to calcMaxBodySize().
  */
 bool receivedBodyTooLarge(HttpRequest&, int64_t receivedBodySize);
 
 /** Checks whether expected body exceeds known maximum size.
  * Requires a prior call to calcMaxBodySize().
  */
 bool expectedBodyTooLarge(HttpRequest& request);
 
 int validatorsMatch (HttpReply const *other) const;
 
 void packHeadersInto(Packable * p) const;
 
 /** Clone this reply.
  *  Could be done as a copy-contructor but we do not want to accidently copy a HttpReply..
  */
 HttpReply *clone() const;
 
 /// Remove Warnings with warn-date different from Date value
 void removeStaleWarnings();
 
 virtual void hdrCacheInit();
 
+/// whether our Date header value is smaller than theirs
+/// \returns false if any information is missing
+bool olderThan(const HttpReply *them) const;
+
 private:
 /** initialize */
 void init();
 
 void clean();
 
 void hdrCacheClean();
 
 void packInto(Packable * p) const;
 
 /* ez-routines */
 /** \return construct 304 reply and pack it into a MemBuf */
 MemBuf *packed304Reply() const;
 
 /* header manipulation */
 time_t hdrExpirationTime();
 
 /** Calculates and stores maximum body size if needed.
  * Used by receivedBodyTooLarge() and expectedBodyTooLarge().
  */
 void calcMaxBodySize(HttpRequest& request) const;
 
 String removeStaleWarningValues(const String );
 
 mutable int64_t bodySizeMax; /**< cached result of calcMaxBodySize */
 
 protected:
 virtual void packFirstLineInto(Packable * p, bool) const { sline.packInto(p); }
 
 virtual bool parseFirstLine(const char *start, const char *end);

=== modified file 'src/LogTags.cc'
--- 

Re: [squid-dev] [PATCH] Revalidate without Last-Modified

[Eduard Bagdasaryan]

2016-08-21 15:58 GMT+03:00 Amos Jeffries :

> please dont move the StoreEntry 'lastmod' member variable which exists
> between the "ON-DISK STORE_META_STD TLV field" marker comments.

> A delta should be the actual difference value -N < 0 < +N.

> please take the opportunity to cleanup any touched debugs() messages.

> please add a note "XXX BUG 1890 objects without Date do not get one
> added".

Addressed these suggestions and refreshed the patch.


Eduard.
Bug 4471: revalidation doesn't work when expired cached object lacks Last-Modified.

The bug was caused by the fact that Squid used only Last-Modified header
value for evaluating entry's last modification time while making an
internal revalidation request. So, without Last-Modified it was not
possible to correctly fill If-Modified-Since header value. As a result,
the revalidation request was not initiated when it should be.

To fix this problem, we should generate If-Modified-Since based on other
data, showing how "old" the cached response is, such as Date and Age
header values. Both Date and Age header values are utilized while
calculating entry's timestamp. Therefore, we could use the timestamp if
Last-Modified is unavailable.

TODO: HttpRequest::imslen support looks broken/deprecated. Using this
field inside StoreEntry::modifiedSince() leads to several useless checks
and probably may cause other [hidden] problems.

=== modified file 'src/MemStore.cc'
--- src/MemStore.cc	2016-05-01 21:37:52 +
+++ src/MemStore.cc	2016-08-25 06:30:42 +
@@ -429,61 +429,61 @@
 // already disconnected from the cache, no need to update
 if (index < 0)
 return true;
 
 if (!map)
 return false;
 
 const Ipc::StoreMapAnchor  = map->readableEntry(index);
 return updateCollapsedWith(collapsed, index, anchor);
 }
 
 /// updates collapsed entry after its anchor has been located
 bool
 MemStore::updateCollapsedWith(StoreEntry , const sfileno index, const Ipc::StoreMapAnchor )
 {
 collapsed.swap_file_sz = anchor.basics.swap_file_sz;
 const bool copied = copyFromShm(collapsed, index, anchor);
 return copied;
 }
 
 /// anchors StoreEntry to an already locked map entry
 void
 MemStore::anchorEntry(StoreEntry , const sfileno index, const Ipc::StoreMapAnchor )
 {
 const Ipc::StoreMapAnchor::Basics  = anchor.basics;
 
 e.swap_file_sz = basics.swap_file_sz;
 e.lastref = basics.lastref;
 e.timestamp = basics.timestamp;
 e.expires = basics.expires;
-e.lastmod = basics.lastmod;
+e.lastModified(basics.lastmod);
 e.refcount = basics.refcount;
 e.flags = basics.flags;
 
 assert(e.mem_obj);
 if (anchor.complete()) {
 e.store_status = STORE_OK;
 e.mem_obj->object_sz = e.swap_file_sz;
 e.setMemStatus(IN_MEMORY);
 } else {
 e.store_status = STORE_PENDING;
 assert(e.mem_obj->object_sz < 0);
 e.setMemStatus(NOT_IN_MEMORY);
 }
 assert(e.swap_status == SWAPOUT_NONE); // set in StoreEntry constructor
 e.ping_status = PING_NONE;
 
 EBIT_CLR(e.flags, RELEASE_REQUEST);
 EBIT_CLR(e.flags, KEY_PRIVATE);
 EBIT_SET(e.flags, ENTRY_VALIDATED);
 
 MemObject::MemCache  = e.mem_obj->memCache;
 mc.index = index;
 mc.io = MemObject::ioReading;
 }
 
 /// copies the entire entry from shared to local memory
 bool
 MemStore::copyFromShm(StoreEntry , const sfileno index, const Ipc::StoreMapAnchor )
 {
 debugs(20, 7, "mem-loading entry " << index << " from " << anchor.start);

=== modified file 'src/Store.h'
--- src/Store.h	2016-07-23 13:36:56 +
+++ src/Store.h	2016-08-25 09:16:52 +
@@ -104,78 +104,87 @@
 const char *url() const;
 /// Satisfies cachability requirements shared among disk and RAM caches.
 /// Encapsulates common checks of mayStartSwapOut() and memoryCachable().
 /// TODO: Rename and make private so only those two methods can call this.
 bool checkCachable();
 int checkNegativeHit() const;
 int locked() const;
 int validToSend() const;
 bool memoryCachable(); ///< checkCachable() and can be cached in memory
 
 /// if needed, initialize mem_obj member w/o URI-related information
 MemObject *makeMemObject();
 
 /// initialize mem_obj member (if needed) and supply URI-related info
 void createMemObject(const char *storeId, const char *logUri, const HttpRequestMethod );
 
 void dump(int debug_lvl) const;
 void hashDelete();
 void hashInsert(const cache_key *);
 void registerAbort(STABH * cb, void *);
 void reset();
 void setMemStatus(mem_status_t);
 bool timestampsSet();
 void unregisterAbort();
 void destroyMemObject();
 int checkTooSmall();
 
 void delayAwareRead(const Comm::ConnectionPointer , char *buf, int len, AsyncCall::Pointer callback);
 
 void setNoDelay (bool const);
-bool modifiedSince(HttpRequest * request) const;
+time_t effectiveModificationTime() const {
+return 

Re: [squid-dev] [PATCH] Incorrect processing of long URIs

[Eduard Bagdasaryan]

2016-08-23 17:50 GMT+03:00 Alex Rousskov :

> s/request-line/request-line: URI/ for consistency and clarity sake.

> I wonder whether we should make this variable static to avoid repeated
> function calls on a performance-sensitive code path. Same for the old
> "delimiters" variable left inside parseRequestFirstLine(), of course.


2016-08-24 17:30 GMT+03:00 Amos Jeffries :

> Option 1, is to use a fixed and slightly generic where message:
> "before protocol version"


Fixed and refreshed the patch.


Eduard.

MUST respond with 414 (URI Too Long) when request target exceeds limits.

Before the fix, Squid simply closed client connection after receiving a
huge URI (or a huge request-line), violating the RFC 7230 MUST. This
happened because a high-level Must(have buffer space) check in
ConnStateData::clientParseRequests() would throw an exception. Now these
problems are detected inside the low-level RequestParser code, where we
can distinguish huge URIs from huge methods.

=== modified file 'src/http/one/RequestParser.cc'
--- src/http/one/RequestParser.cc	2016-05-20 08:28:33 +
+++ src/http/one/RequestParser.cc	2016-08-24 23:12:45 +
@@ -58,60 +58,65 @@
 while (!buf_.isEmpty() && (buf_[0] == '\n' || (buf_[0] == '\r' && buf_[1] == '\n'))) {
 buf_.consume(1);
 }
 }
 }
 
 /**
  * Attempt to parse the method field out of an HTTP message request-line.
  *
  * Governed by:
  *  RFC 1945 section 5.1
  *  RFC 7230 section 2.6, 3.1 and 3.5
  */
 bool
 Http::One::RequestParser::parseMethodField(Http1::Tokenizer )
 {
 // method field is a sequence of TCHAR.
 // Limit to 32 characters to prevent overly long sequences of non-HTTP
 // being sucked in before mismatch is detected. 32 is itself annoyingly
 // big but there are methods registered by IANA that reach 17 bytes:
 //  http://www.iana.org/assignments/http-methods
 static const size_t maxMethodLength = 32; // TODO: make this configurable?
 
 SBuf methodFound;
 if (!tok.prefix(methodFound, CharacterSet::TCHAR, maxMethodLength)) {
 debugs(33, ErrorLevel(), "invalid request-line: missing or malformed method");
 parseStatusCode = Http::scBadRequest;
 return false;
 }
 method_ = HttpRequestMethod(methodFound);
+
+static const CharacterSet  = DelimiterCharacters();
+if (!skipDelimiter(tok.skipAll(delimiters), "after method"))
+return false;
+
 return true;
 }
 
 /// the characters which truly are valid within URI
 static const CharacterSet &
 UriValidCharacters()
 {
 /* RFC 3986 section 2:
  * "
  *   A URI is composed from a limited set of characters consisting of
  *   digits, letters, and a few graphic symbols.
  * "
  */
 static const CharacterSet UriChars =
 CharacterSet("URI-Chars","") +
 // RFC 3986 section 2.2 - reserved characters
 CharacterSet("gen-delims", ":/?#[]@") +
 CharacterSet("sub-delims", "!$&'()*+,;=") +
 // RFC 3986 section 2.3 - unreserved characters
 CharacterSet::ALPHA +
 CharacterSet::DIGIT +
 CharacterSet("unreserved", "-._~") +
 // RFC 3986 section 2.1 - percent encoding "%" HEXDIG
 CharacterSet("pct-encoded", "%") +
 CharacterSet::HEXDIG;
 
 return UriChars;
 }
 
 /// characters which Squid will accept in the HTTP request-target (URI)
@@ -185,139 +190,149 @@
 
 SBuf digit;
 // Searching for Http1magic precludes detecting HTTP/2+ versions.
 // Rewrite if we ever _need_ to return 505 (Version Not Supported) errors.
 if (tok.suffix(digit, CharacterSet::DIGIT) && tok.skipSuffix(Http1magic)) {
 msgProtocol_ = Http::ProtocolVersion(1, (*digit.rawContent() - '0'));
 return true;
 }
 
 // A GET request might use HTTP/0.9 syntax
 if (method_ == Http::METHOD_GET) {
 // RFC 1945 - no HTTP version field at all
 tok = savedTok; // in case the URI ends with a digit
 // report this assumption as an error if configured to triage parsing
 debugs(33, ErrorLevel(), "assuming HTTP/0.9 request-line");
 msgProtocol_ = Http::ProtocolVersion(0,9);
 return true;
 }
 
 debugs(33, ErrorLevel(), "invalid request-line: not HTTP");
 parseStatusCode = Http::scBadRequest;
 return false;
 }
 
 /**
  * Skip characters separating request-line fields.
  * To handle bidirectional parsing, the caller does the actual skipping and
  * we just check how many character the caller has skipped.
  */
 bool
-Http::One::RequestParser::skipDelimiter(const size_t count)
+Http::One::RequestParser::skipDelimiter(const size_t count, const char *where)
 {
 if (count <= 0) {
-debugs(33, ErrorLevel(), "invalid request-line: missing delimiter");
+debugs(33, ErrorLevel(), "invalid request-line: missing delimiter " << where);
 parseStatusCode = Http::scBadRequest;
 

Re: [squid-dev] [PATCH] Incorrect processing of long URIs

[Eduard Bagdasaryan]

2016-08-23 18:01 GMT+03:00 Alex Rousskov :

> invalid request-line: missing delimiter before "HTTP/1"

In order to generate "where" with such detalization (i.e. the specific 
protocol version or method) we would need to pass skipDelimiter() the 
parsed AnyP::ProtocolVersion or HttpRequestMethod objects, which would 
require converting skipDelimiter() to a template:


   skipDelimiter(size_t, const char *where, const C 
);


Otherwise (keeping it simple without templates), the skipDelimiter() the 
debugging would be:


  invalid request-line: missing delimiter before HTTP-version

I would prefer the latter.


Eduard.
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Revalidate without Last-Modified

[Eduard Bagdasaryan]

2016-08-21 15:58 GMT+03:00 Amos Jeffries :

> To change anything between those markers we have to do a full cache
> versioning and up/down-grade compatibility dance.

Could you please clarify what versioning problems you are talking about?  It
seems that StoreEntry's STORE_META_STD fields are not used directly while
storing/restoring metainformation. I found only places where field-by-field
assignment is performed (e.g., InitStoreEntry::operator()), so making
StoreEntry::lastmod 'private' should not do any harm there.  So probably we
could mark with  "START OF ON-DISK STORE_META_STD TLV field" (or 
similar) the

StoreEntry::lastmod, which is now private.

> lastModifiedDelta() returning -1 when the actual delta is any -N value
> is wrong conceptually. A delta should be the actual difference value 
-N < 0 < +N.


Since we need and use only the positive outcome of this method inside
refreshStaleness(), probably rename it to ageWhenCached() (or similar)?
Returning '-1' would mean that it is impossible to calculate the age 
('lastmod'

is absent or greater than timestamp).


Eduard.

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Incorrect processing of long URIs

[Eduard Bagdasaryan]

2016-08-23 3:08 GMT+03:00 Alex Rousskov :

> I do not understand why you decided to use maxMethodLength in
> parseRequestFirstLine(). AFAICT, parseMethodField() already does
> everything we need: It logs an error message and sets parseStatusCode
> accordingly.

Yes, it does partly what we need but it does not parse the delimiter(
and inform about possible failure). skipDelimiter() does this with
message:

> invalid request-line: missing delimiter

which does not hint that our case is a 'bad' method and
less informative than:

> invalid request-line: method exceeds 32-byte limit

Probably this does not make much sense, so I followed your
sketch and refreshed the patch.


Eduard.
MUST respond with 414 (URI Too Long) when request target exceeds limits.

Before the fix, Squid simply closed client connection after receiving a
huge URI (or a huge request-line), violating the RFC 7230 MUST. This
happened because a high-level Must(have buffer space) check in
ConnStateData::clientParseRequests() would throw an exception. Now these
problems are detected inside the low-level RequestParser code, where we
can distinguish huge URIs from huge methods.

=== modified file 'src/http/one/RequestParser.cc'
--- src/http/one/RequestParser.cc	2016-05-20 08:28:33 +
+++ src/http/one/RequestParser.cc	2016-08-23 09:01:24 +
@@ -58,60 +58,65 @@
 while (!buf_.isEmpty() && (buf_[0] == '\n' || (buf_[0] == '\r' && buf_[1] == '\n'))) {
 buf_.consume(1);
 }
 }
 }
 
 /**
  * Attempt to parse the method field out of an HTTP message request-line.
  *
  * Governed by:
  *  RFC 1945 section 5.1
  *  RFC 7230 section 2.6, 3.1 and 3.5
  */
 bool
 Http::One::RequestParser::parseMethodField(Http1::Tokenizer )
 {
 // method field is a sequence of TCHAR.
 // Limit to 32 characters to prevent overly long sequences of non-HTTP
 // being sucked in before mismatch is detected. 32 is itself annoyingly
 // big but there are methods registered by IANA that reach 17 bytes:
 //  http://www.iana.org/assignments/http-methods
 static const size_t maxMethodLength = 32; // TODO: make this configurable?
 
 SBuf methodFound;
 if (!tok.prefix(methodFound, CharacterSet::TCHAR, maxMethodLength)) {
 debugs(33, ErrorLevel(), "invalid request-line: missing or malformed method");
 parseStatusCode = Http::scBadRequest;
 return false;
 }
 method_ = HttpRequestMethod(methodFound);
+
+const CharacterSet  = DelimiterCharacters();
+if (!skipDelimiter(tok.skipAll(delimiters)))
+return false;
+
 return true;
 }
 
 /// the characters which truly are valid within URI
 static const CharacterSet &
 UriValidCharacters()
 {
 /* RFC 3986 section 2:
  * "
  *   A URI is composed from a limited set of characters consisting of
  *   digits, letters, and a few graphic symbols.
  * "
  */
 static const CharacterSet UriChars =
 CharacterSet("URI-Chars","") +
 // RFC 3986 section 2.2 - reserved characters
 CharacterSet("gen-delims", ":/?#[]@") +
 CharacterSet("sub-delims", "!$&'()*+,;=") +
 // RFC 3986 section 2.3 - unreserved characters
 CharacterSet::ALPHA +
 CharacterSet::DIGIT +
 CharacterSet("unreserved", "-._~") +
 // RFC 3986 section 2.1 - percent encoding "%" HEXDIG
 CharacterSet("pct-encoded", "%") +
 CharacterSet::HEXDIG;
 
 return UriChars;
 }
 
 /// characters which Squid will accept in the HTTP request-target (URI)
@@ -243,73 +248,83 @@
 return false;
 }
 }
 return true;
 }
 
 /**
  * Attempt to parse the first line of a new request message.
  *
  * Governed by:
  *  RFC 1945 section 5.1
  *  RFC 7230 section 2.6, 3.1 and 3.5
  *
  * \retval -1  an error occurred. parseStatusCode indicates HTTP status result.
  * \retval  1  successful parse. member fields contain the request-line items
  * \retval  0  more data is needed to complete the parse
  */
 int
 Http::One::RequestParser::parseRequestFirstLine()
 {
 debugs(74, 5, "parsing possible request: buf.length=" << buf_.length());
 debugs(74, DBG_DATA, buf_);
 
 SBuf line;
 
 // Earlier, skipGarbageLines() took care of any leading LFs (if allowed).
 // Now, the request line has to end at the first LF.
 static const CharacterSet lineChars = CharacterSet::LF.complement("notLF");
 ::Parser::Tokenizer lineTok(buf_);
 if (!lineTok.prefix(line, lineChars) || !lineTok.skip('\n')) {
+if (buf_.length() >= Config.maxRequestHeaderSize) {
+/* who should we blame for our failure to parse this line? */
+
+Http1::Tokenizer methodTok(buf_);
+if (!parseMethodField(methodTok))
+return -1; // blame a bad method (or its delimiter)
+
+// assume it is the URI
+debugs(74, ErrorLevel(), "invalid request-line exceeds " <<
+

[squid-dev] [PATCH] Incorrect processing of long URIs

[Eduard Bagdasaryan]

Hello,

This patch makes Squid respond with 414 (URI Too Long) when request
target exceeds limits.

Before the fix, Squid simply closed client connection after receiving a
huge URI (or a huge request-line), violating the RFC 7230 MUST. This
happened because a high-level Must(have buffer space) check in
ConnStateData::clientParseRequests() would throw an exception. Now these
problems are detected inside the low-level RequestParser code, where we
can distinguish huge URIs from huge methods.

Regards,
Eduard.
MUST respond with 414 (URI Too Long) when request target exceeds limits.

Before the fix, Squid simply closed client connection after receiving a
huge URI (or a huge request-line), violating the RFC 7230 MUST. This
happened because a high-level Must(have buffer space) check in
ConnStateData::clientParseRequests() would throw an exception. Now these
problems are detected inside the low-level RequestParser code, where we
can distinguish huge URIs from huge methods.

=== modified file 'src/http/one/RequestParser.cc'
--- src/http/one/RequestParser.cc	2016-05-20 08:28:33 +
+++ src/http/one/RequestParser.cc	2016-08-22 22:14:35 +
@@ -45,68 +45,63 @@
  * Parsing state is stored between calls to avoid repeating buffer scans.
  * If garbage is found the parsing offset is incremented.
  */
 void
 Http::One::RequestParser::skipGarbageLines()
 {
 if (Config.onoff.relaxed_header_parser) {
 if (Config.onoff.relaxed_header_parser < 0 && (buf_[0] == '\r' || buf_[0] == '\n'))
 debugs(74, DBG_IMPORTANT, "WARNING: Invalid HTTP Request: " <<
"CRLF bytes received ahead of request-line. " <<
"Ignored due to relaxed_header_parser.");
 // Be tolerant of prefix empty lines
 // ie any series of either \n or \r\n with no other characters and no repeated \r
 while (!buf_.isEmpty() && (buf_[0] == '\n' || (buf_[0] == '\r' && buf_[1] == '\n'))) {
 buf_.consume(1);
 }
 }
 }
 
 /**
  * Attempt to parse the method field out of an HTTP message request-line.
  *
  * Governed by:
  *  RFC 1945 section 5.1
  *  RFC 7230 section 2.6, 3.1 and 3.5
  */
 bool
 Http::One::RequestParser::parseMethodField(Http1::Tokenizer )
 {
 // method field is a sequence of TCHAR.
-// Limit to 32 characters to prevent overly long sequences of non-HTTP
-// being sucked in before mismatch is detected. 32 is itself annoyingly
-// big but there are methods registered by IANA that reach 17 bytes:
-//  http://www.iana.org/assignments/http-methods
-static const size_t maxMethodLength = 32; // TODO: make this configurable?
 
 SBuf methodFound;
-if (!tok.prefix(methodFound, CharacterSet::TCHAR, maxMethodLength)) {
+if (!tok.prefix(methodFound, CharacterSet::TCHAR, MaxMethodLength)) {
 debugs(33, ErrorLevel(), "invalid request-line: missing or malformed method");
 parseStatusCode = Http::scBadRequest;
 return false;
 }
 method_ = HttpRequestMethod(methodFound);
 return true;
 }
 
 /// the characters which truly are valid within URI
 static const CharacterSet &
 UriValidCharacters()
 {
 /* RFC 3986 section 2:
  * "
  *   A URI is composed from a limited set of characters consisting of
  *   digits, letters, and a few graphic symbols.
  * "
  */
 static const CharacterSet UriChars =
 CharacterSet("URI-Chars","") +
 // RFC 3986 section 2.2 - reserved characters
 CharacterSet("gen-delims", ":/?#[]@") +
 CharacterSet("sub-delims", "!$&'()*+,;=") +
 // RFC 3986 section 2.3 - unreserved characters
 CharacterSet::ALPHA +
 CharacterSet::DIGIT +
 CharacterSet("unreserved", "-._~") +
 // RFC 3986 section 2.1 - percent encoding "%" HEXDIG
 CharacterSet("pct-encoded", "%") +
 CharacterSet::HEXDIG;
@@ -242,67 +237,83 @@
 parseStatusCode = Http::scBadRequest;
 return false;
 }
 }
 return true;
 }
 
 /**
  * Attempt to parse the first line of a new request message.
  *
  * Governed by:
  *  RFC 1945 section 5.1
  *  RFC 7230 section 2.6, 3.1 and 3.5
  *
  * \retval -1  an error occurred. parseStatusCode indicates HTTP status result.
  * \retval  1  successful parse. member fields contain the request-line items
  * \retval  0  more data is needed to complete the parse
  */
 int
 Http::One::RequestParser::parseRequestFirstLine()
 {
 debugs(74, 5, "parsing possible request: buf.length=" << buf_.length());
 debugs(74, DBG_DATA, buf_);
 
 SBuf line;
 
 // Earlier, skipGarbageLines() took care of any leading LFs (if allowed).
 // Now, the request line has to end at the first LF.
 static const CharacterSet lineChars = CharacterSet::LF.complement("notLF");
 ::Parser::Tokenizer lineTok(buf_);
+const CharacterSet  = DelimiterCharacters();
+
 if (!lineTok.prefix(line, lineChars) || !lineTok.skip('\n')) {
+if 

[squid-dev] [PATCH] Revalidate without Last-Modified

[Eduard Bagdasaryan]

Hello,

This patch fixes bug 4471.

The bug was caused by the fact that Squid used only Last-Modified header
value for evaluating entry's last modification time while making an
internal revalidation request. So, without Last-Modified it was not
possible to correctly fill If-Modified-Since header value. As a result,
the revalidation request was not initiated when it should be.

To fix this problem, we should generate If-Modified-Since based on other
data, showing how "old" the cached response is, such as Date and Age
header values.  Both Date and Age header values are utilized while
calculating entry's timestamp.  Therefore, we could use the timestamp if
Last-Modified is unavailable.

XXX: HttpRequest::imslen support looks broken/deprecated. Using this
field inside StoreEntry::modifiedSince() leads to several useless checks
and probably may cause other [hidden] problems.

Regards,
Eduard.
Bug 4471 fix.

The bug was caused by the fact that Squid used only Last-Modified header
value for evaluating entry's last modification time while making an
internal revalidation request. So, without Last-Modified it was not
possible to correctly fill If-Modified-Since header value. As a result,
the revalidation request was not initiated when it should be.

To fix this problem, we should generate If-Modified-Since based on other
data, showing how "old" the cached response is, such as Date and Age
header values. Both Date and Age header values are utilized while
calculating entry's timestamp. Therefore, we could use the timestamp if
Last-Modified is unavailable.

XXX: HttpRequest::imslen support looks broken/deprecated. Using this
field inside StoreEntry::modifiedSince() leads to several useless checks
and probably may cause other [hidden] problems.

=== modified file 'src/MemStore.cc'
--- src/MemStore.cc	2016-05-01 21:37:52 +
+++ src/MemStore.cc	2016-08-19 10:28:10 +
@@ -429,61 +429,61 @@
 // already disconnected from the cache, no need to update
 if (index < 0)
 return true;
 
 if (!map)
 return false;
 
 const Ipc::StoreMapAnchor  = map->readableEntry(index);
 return updateCollapsedWith(collapsed, index, anchor);
 }
 
 /// updates collapsed entry after its anchor has been located
 bool
 MemStore::updateCollapsedWith(StoreEntry , const sfileno index, const Ipc::StoreMapAnchor )
 {
 collapsed.swap_file_sz = anchor.basics.swap_file_sz;
 const bool copied = copyFromShm(collapsed, index, anchor);
 return copied;
 }
 
 /// anchors StoreEntry to an already locked map entry
 void
 MemStore::anchorEntry(StoreEntry , const sfileno index, const Ipc::StoreMapAnchor )
 {
 const Ipc::StoreMapAnchor::Basics  = anchor.basics;
 
 e.swap_file_sz = basics.swap_file_sz;
 e.lastref = basics.lastref;
 e.timestamp = basics.timestamp;
 e.expires = basics.expires;
-e.lastmod = basics.lastmod;
+e.lastModified(basics.lastmod);
 e.refcount = basics.refcount;
 e.flags = basics.flags;
 
 assert(e.mem_obj);
 if (anchor.complete()) {
 e.store_status = STORE_OK;
 e.mem_obj->object_sz = e.swap_file_sz;
 e.setMemStatus(IN_MEMORY);
 } else {
 e.store_status = STORE_PENDING;
 assert(e.mem_obj->object_sz < 0);
 e.setMemStatus(NOT_IN_MEMORY);
 }
 assert(e.swap_status == SWAPOUT_NONE); // set in StoreEntry constructor
 e.ping_status = PING_NONE;
 
 EBIT_CLR(e.flags, RELEASE_REQUEST);
 EBIT_CLR(e.flags, KEY_PRIVATE);
 EBIT_SET(e.flags, ENTRY_VALIDATED);
 
 MemObject::MemCache  = e.mem_obj->memCache;
 mc.index = index;
 mc.io = MemObject::ioReading;
 }
 
 /// copies the entire entry from shared to local memory
 bool
 MemStore::copyFromShm(StoreEntry , const sfileno index, const Ipc::StoreMapAnchor )
 {
 debugs(20, 7, "mem-loading entry " << index << " from " << anchor.start);

=== modified file 'src/Store.h'
--- src/Store.h	2016-07-23 13:36:56 +
+++ src/Store.h	2016-08-20 15:50:30 +
@@ -104,78 +104,86 @@
 const char *url() const;
 /// Satisfies cachability requirements shared among disk and RAM caches.
 /// Encapsulates common checks of mayStartSwapOut() and memoryCachable().
 /// TODO: Rename and make private so only those two methods can call this.
 bool checkCachable();
 int checkNegativeHit() const;
 int locked() const;
 int validToSend() const;
 bool memoryCachable(); ///< checkCachable() and can be cached in memory
 
 /// if needed, initialize mem_obj member w/o URI-related information
 MemObject *makeMemObject();
 
 /// initialize mem_obj member (if needed) and supply URI-related info
 void createMemObject(const char *storeId, const char *logUri, const HttpRequestMethod );
 
 void dump(int debug_lvl) const;
 void hashDelete();
 void hashInsert(const cache_key *);
 void registerAbort(STABH * cb, void *);
 void reset();
 void setMemStatus(mem_status_t);
 bool timestampsSet();
 

Re: [squid-dev] [PATCH] Make Squid death due to overloaded helpers optional

[Eduard Bagdasaryan]

2016-08-10 19:03 GMT+03:00 Alex Rousskov :

> As Amos has noted, we do need to restore the old "unknown" behavior when
> the helper is _missing_ (and not overloaded), but that is a completely
> different problem with a simple solution: SubmissionFailure() should use
> Helper::Unknown when its hlp parameter is nil and Helper::Error.

Adjusted accordingly and updated the patch.


Eduard.
Make Squid death due to overloaded helpers optional.

Added on-persistent-overload=action option to helpers. Helper overload
is defined as running with an overflowing queue. Persistent helper
overload is [still] defined as being overloaded for more than 3 minutes.

The default behavior is unchanged(*) -- Squid worker dies with a fatal
error at the attempt to submit a new request to a persistenly overloaded
helper. This default behavior can also be configured explicitly using
on-persistent-overload=die.

With on-persistent-overload=err, when dealing with a persistently
overloaded helper, Squid immediately drops the helper request and sends
an empty response to the caller which should handle that empty response
as an error. Squid informs the admin when it starts and when it stops
dropping helper requests due to persistent overload.

The code had conflicting notions of an "overloaded helper". The external
ACL helper, the URL rewriter, and the store ID code used queueFull() to
test whether the new request would overflow the queue (and, hence,
overload the helper), but queueFull() itself did not check whether the
queue was full! It checked whether the queue was already overflowing.
This confusion resulted in that code scheduling one extra helper request
before enabling bypass. The code and its documentation are now more
consistent (and better match the "overload" terminology used by the new
configuration option, which also feels better than calling the helper
"full").

(*) Resolving the above confusion resulted in minor (one request)
differences in the number of helper requests queued by Squid for
external ACL, URL rewriting, and store ID helpers, with the adjusted
behavior [better] matching the documentation.

=== modified file 'src/cf.data.pre'
--- src/cf.data.pre	2016-08-01 11:11:47 +
+++ src/cf.data.pre	2016-08-11 10:11:31 +
@@ -539,89 +539,107 @@
 		can be used. In practice, a %macro expands as a dash (-) if
 		the helper request is sent before the required macro
 		information is available to Squid.
 
 		By default, Squid uses request formats provided in
 		scheme-specific examples below (search for %credentials).
 
 		The expanded key_extras value is added to the Squid credentials
 		cache and, hence, will affect authentication. It can be used to
 		autenticate different users with identical user names (e.g.,
 		when user authentication depends on http_port).
 
 		Avoid adding frequently changing information to key_extras. For
 		example, if you add user source IP, and it changes frequently
 		in your environment, then max_user_ip ACL is going to treat
 		every user+IP combination as a unique "user", breaking the ACL
 		and wasting a lot of memory on those user records. It will also
 		force users to authenticate from scratch whenever their IP
 		changes.
 
 	"realm" string
 		Specifies the protection scope (aka realm name) which is to be
 		reported to the client for the authentication scheme. It is
 		commonly part of the text the user will see when prompted for
 		their username and password.
 
 		For Basic the default is "Squid proxy-caching web server".
 		For Digest there is no default, this parameter is mandatory.
 		For NTLM and Negotiate this parameter is ignored.
 
-	"children" numberofchildren [startup=N] [idle=N] [concurrency=N] [queue-size=N]
+	"children" numberofchildren [startup=N] [idle=N] [concurrency=N]
+		[queue-size=N] [on-persistent-overload=action]
 
 		The maximum number of authenticator processes to spawn. If
 		you start too few Squid will have to wait for them to process
 		a backlog of credential verifications, slowing it down. When
 		password verifications are done via a (slow) network you are
 		likely to need lots of authenticator processes.
 
 		The startup= and idle= options permit some skew in the exact
 		amount run. A minimum of startup=N will begin during startup
 		and reconfigure. Squid will start more in groups of up to
 		idle=N in an attempt to meet traffic needs and to keep idle=N
 		free above those traffic needs up to the maximum.
 
 		The concurrency= option sets the number of concurrent requests
 		the helper can process.  The default of 0 is used for helpers
 		who only supports one request at a time. Setting this to a
 		number greater than 0 changes the protocol used to include a
 		channel ID field first on the request/response line, allowing
 		multiple requests to be sent to the same helper in parallel
 		without waiting for the response.
 
 		Concurrency must not be set unless it's known the helper
 		supports the input format 

Re: [squid-dev] [PATCH] Make Squid death due to overloaded helpers optional

[Eduard Bagdasaryan]

2016-08-08 23:17 GMT+03:00 Amos Jeffries :

> s/temporary exceed/temporarily exceed/

Done.

> please remove the above/below wording.

Removed.

> the lines documenting 'die' and 'err' action look a bit squashed up.

Adjusted accordingly.

> looks wrong documentation for the overloaded() method

Removed that comment at all, since the method looks simple and
self-documented.

> please revert to sending helper::Unknown code

Reverted. Note that for helper::Unknown some callers print a bit
misleading error message, when requests are dropped, e.g.:
"ERROR: storeID helper returned invalid result code. Wrong helper?"

> Also, can we please add a "wait" action as well. Which means just keep
> queueing things and ignore the overload.

I don't think we need this extra option. Currently we have a hard-coded
3 min. timeout, within which requests are keep queuing ignoring
[possible] overload. Probably you need another option to configure this
timeout? That would be a separate task, IMO.



Eduard.
Make Squid death due to overloaded helpers optional.

Added on-persistent-overload=action option to helpers. Helper overload
is defined as running with an overflowing queue. Persistent helper
overload is [still] defined as being overloaded for more than 3 minutes.

The default behavior is unchanged(*) -- Squid worker dies with a fatal
error at the attempt to submit a new request to a persistenly overloaded
helper. This default behavior can also be configured explicitly using
on-persistent-overload=die.

With on-persistent-overload=err, when dealing with a persistently
overloaded helper, Squid immediately drops the helper request and sends
an empty response to the caller which should handle that empty response
as an error. Squid informs the admin when it starts and when it stops
dropping helper requests due to persistent overload.

The code had conflicting notions of an "overloaded helper". The external
ACL helper, the URL rewriter, and the store ID code used queueFull() to
test whether the new request would overflow the queue (and, hence,
overload the helper), but queueFull() itself did not check whether the
queue was full! It checked whether the queue was already overflowing.
This confusion resulted in that code scheduling one extra helper request
before enabling bypass. The code and its documentation are now more
consistent (and better match the "overload" terminology used by the new
configuration option, which also feels better than calling the helper
"full").

(*) Resolving the above confusion resulted in minor (one request)
differences in the number of helper requests queued by Squid for
external ACL, URL rewriting, and store ID helpers, with the adjusted
behavior [better] matching the documentation.

=== modified file 'src/cf.data.pre'
--- src/cf.data.pre	2016-08-01 11:11:47 +
+++ src/cf.data.pre	2016-08-09 11:01:15 +
@@ -539,89 +539,107 @@
 		can be used. In practice, a %macro expands as a dash (-) if
 		the helper request is sent before the required macro
 		information is available to Squid.
 
 		By default, Squid uses request formats provided in
 		scheme-specific examples below (search for %credentials).
 
 		The expanded key_extras value is added to the Squid credentials
 		cache and, hence, will affect authentication. It can be used to
 		autenticate different users with identical user names (e.g.,
 		when user authentication depends on http_port).
 
 		Avoid adding frequently changing information to key_extras. For
 		example, if you add user source IP, and it changes frequently
 		in your environment, then max_user_ip ACL is going to treat
 		every user+IP combination as a unique "user", breaking the ACL
 		and wasting a lot of memory on those user records. It will also
 		force users to authenticate from scratch whenever their IP
 		changes.
 
 	"realm" string
 		Specifies the protection scope (aka realm name) which is to be
 		reported to the client for the authentication scheme. It is
 		commonly part of the text the user will see when prompted for
 		their username and password.
 
 		For Basic the default is "Squid proxy-caching web server".
 		For Digest there is no default, this parameter is mandatory.
 		For NTLM and Negotiate this parameter is ignored.
 
-	"children" numberofchildren [startup=N] [idle=N] [concurrency=N] [queue-size=N]
+	"children" numberofchildren [startup=N] [idle=N] [concurrency=N]
+		[queue-size=N] [on-persistent-overload=action]
 
 		The maximum number of authenticator processes to spawn. If
 		you start too few Squid will have to wait for them to process
 		a backlog of credential verifications, slowing it down. When
 		password verifications are done via a (slow) network you are
 		likely to need lots of authenticator processes.
 
 		The startup= and idle= options permit some skew in the exact
 		amount run. A minimum of startup=N will begin during startup
 		and reconfigure. Squid will start more in groups of up to
 		idle=N in an attempt to meet 

[squid-dev] [PATCH] Make Squid death due to overloaded helpers optional

[Eduard Bagdasaryan]

This patch allows to configure Squid so that it reject overloaded helper
requests instead of [default] crashing.

Added on-persistent-overload=action option to helpers. Helper overload
is defined as running with an overflowing queue. Persistent helper
overload is [still] defined as being overloaded for more than 3 minutes.

The default behavior is unchanged(*) -- Squid worker dies with a fatal
error at the attempt to submit a new request to a persistenly overloaded
helper. This default behavior can also be configured explicitly using
on-persistent-overload=die.

With on-persistent-overload=err, when dealing with a persistently
overloaded helper, Squid immediately drops the helper request and sends
an empty response to the caller which should handle that empty response
as an error. Squid informs the admin when it starts and when it stops
dropping helper requests due to persistent overload.

The code had conflicting notions of an "overloaded helper". The external
ACL helper, the URL rewriter, and the store ID code used queueFull() to
test whether the new request would overflow the queue (and, hence,
overload the helper), but queueFull() itself did not check whether the
queue was full! It checked whether the queue was already overflowing.
This confusion resulted in that code scheduling one extra helper request
before enabling bypass. The code and its documentation are now more
consistent (and better match the "overload" terminology used by the new
configuration option, which also feels better than calling the helper
"full").

(*) Resolving the above confusion resulted in minor (one request)
differences in the number of helper requests queued by Squid for
external ACL, URL rewriting, and store ID helpers, with the adjusted
behavior [better] matching the documentation.

Regards,
Eduard.
Make Squid death due to overloaded helpers optional.

Added on-persistent-overload=action option to helpers. Helper overload
is defined as running with an overflowing queue. Persistent helper
overload is [still] defined as being overloaded for more than 3 minutes.

The default behavior is unchanged(*) -- Squid worker dies with a fatal
error at the attempt to submit a new request to a persistenly overloaded
helper. This default behavior can also be configured explicitly using
on-persistent-overload=die.

With on-persistent-overload=err, when dealing with a persistently
overloaded helper, Squid immediately drops the helper request and sends
an empty response to the caller which should handle that empty response
as an error. Squid informs the admin when it starts and when it stops
dropping helper requests due to persistent overload.

The code had conflicting notions of an "overloaded helper". The external
ACL helper, the URL rewriter, and the store ID code used queueFull() to
test whether the new request would overflow the queue (and, hence,
overload the helper), but queueFull() itself did not check whether the
queue was full! It checked whether the queue was already overflowing.
This confusion resulted in that code scheduling one extra helper request
before enabling bypass. The code and its documentation are now more
consistent (and better match the "overload" terminology used by the new
configuration option, which also feels better than calling the helper
"full").

(*) Resolving the above confusion resulted in minor (one request)
differences in the number of helper requests queued by Squid for
external ACL, URL rewriting, and store ID helpers, with the adjusted
behavior [better] matching the documentation.

=== modified file 'src/cf.data.pre'
--- src/cf.data.pre	2016-08-01 11:11:47 +
+++ src/cf.data.pre	2016-08-08 10:59:46 +
@@ -539,89 +539,103 @@
 		can be used. In practice, a %macro expands as a dash (-) if
 		the helper request is sent before the required macro
 		information is available to Squid.
 
 		By default, Squid uses request formats provided in
 		scheme-specific examples below (search for %credentials).
 
 		The expanded key_extras value is added to the Squid credentials
 		cache and, hence, will affect authentication. It can be used to
 		autenticate different users with identical user names (e.g.,
 		when user authentication depends on http_port).
 
 		Avoid adding frequently changing information to key_extras. For
 		example, if you add user source IP, and it changes frequently
 		in your environment, then max_user_ip ACL is going to treat
 		every user+IP combination as a unique "user", breaking the ACL
 		and wasting a lot of memory on those user records. It will also
 		force users to authenticate from scratch whenever their IP
 		changes.
 
 	"realm" string
 		Specifies the protection scope (aka realm name) which is to be
 		reported to the client for the authentication scheme. It is
 		commonly part of the text the user will see when prompted for
 		their username and password.
 
 		For Basic the default is "Squid proxy-caching web server".
 		For Digest there is no default, this parameter is 

Re: [squid-dev] [PATCH] Some failed transactions are not logged

[Eduard Bagdasaryan]

2016-07-20 18:23 GMT+03:00 Alex Rousskov <rouss...@measurement-factory.com>:
> On 07/20/2016 09:06 AM, Amos Jeffries wrote:
> > On 21/07/2016 2:44 a.m., Eduard Bagdasaryan wrote:
> >> Amos,
> >> just to clarify: any more touches from my side?
>
> > There are the bits where Alex and I agreed on a change. I think 
that was

> > all cosmetic documentation stuff.
>
> In summary:
>
> 1. Rename the error URL to "error:accept-client-connection".
> 2. Adjust the source code comments using the "Given the new condition, I
> would do" sketch.
> 3. Use the getter method for receivedFirstByte_.

Updated the patch with first two changes. We do not have a 'getter' for
receivedFirstByte_. It easy to add it of course, but I assume the current
receivedFirstByte() should be refactored then, adding a 'bool' parameter,
smth. like:

void
ConnStateData::receivedFirstByte(bool on)
{
if (on == receivedFirstByte_)
return;
receivedFirstByte_ = on;
if (!receivedFirstByte_)
return;
// Set timeout to Config.Timeout.request
typedef CommCbMemFunT<ConnStateData, CommTimeoutCbParams> 
TimeoutDialer;

AsyncCall::Pointer timeoutCall =  JobCallback(33, 5,
  TimeoutDialer, this, 
ConnStateData::requestTimeout);
commSetConnTimeout(clientConnection, Config.Timeout.request, 
timeoutCall);

}

Then go through the code and use this new method instead of
receivedFirstByte_ assignment. Do we need all this?

BTW, in server.cc "if" check is redundant:
  if (!receivedFirstByte_)
  receivedFirstByte();



Eduard.
Some failed transactions are not logged.

There are situations when Squid logs nothing to access.log after an
[abnormal] transaction termination. Such "stealthy" transactions may be
a security risk and an accounting problem.

ClientHttpRequest is responsible for logging most transactions but that
object is created only after the HTTP request headers are successfully
parsed. Request header parsing errors may be detected and logged
appropriately, but the job handling the incoming transaction may
terminate for reasons outside the parsing code control (e.g., a job-
killing exception thrown when there are no request headers to start
parsing yet or when the job waits for more request headers to finishing
parsing).

This change adds access logging for three cases:

1. accept(2) system call errors (before ConnStateData job is created).

2. Unexpected ConnStateData job termination, when there is no
   ClientHttpRequest to log the failure.

3. Connections which send no bytes: these connections drain Squid
   resources and, hence, should be logged.
   TODO: make this behavior configurable because some browsers are known to
   routinely create such connections(and, hence, logging them may create
   too much noise in some environments).

=== modified file 'doc/release-notes/release-4.sgml'
--- doc/release-notes/release-4.sgml	2016-07-01 11:30:00 +
+++ doc/release-notes/release-4.sgml	2016-07-20 22:20:25 +
@@ -199,60 +199,66 @@
 
 	request_start_timeout
 	New directive controlling how long Squid waits for the first request
 	   bytes to arrive after initial connection establishment by a client.
 
 	server_pconn_for_nonretriable
 	New directive to provide fine-grained control over persistent connection
 	   reuse when forwarding HTTP requests that Squid cannot retry. It is useful
 	   in environments where opening new connections is very expensive
 	   and race conditions associated with persistent connections are very rare
 	   and/or only cause minor problems.
 
 	shared_memory_locking
 	New directive to ensure shared memory is all available immediately
 	   on startup. Protects against SIGBUS errors, but delays startup.
 
 	tls_outgoing_options
 	New directive to define TLS security context options for outgoing
 	   connections. For example to HTTPS servers.
 
 	url_rewrite_timeout
 	Squid times active requests to redirector. This option sets
 	   the timeout value and the Squid reaction to a timed out
 	   request.
 
 
 
 Changes to existing tags
 
 
+	access_log
+	TCP accept(2) errors logged with URI error:accept-client-connection
+	Unused connections received in http_port or https_port
+	   or transactions terminated before reading[parsing] request headers
+	   logged with URI error:transaction-end-before-headers
+
 	acl
 	New -m flag for note ACL to match substrings.
 
 	auth_param
 	New parameter queue-size= to set the maximum number
 	   of queued requests.
 
 	cache_peer
 	New option auth-no-keytab to let GSSAPI implementation determine
 	   which Kerberos credentials to use, instead of specifying a keytab.
 	New option tls-min-version=1.N to set minimum TLS version allowed.
 	New option tls-default-ca replaces sslflags=NO_DEFAULT_CA
 	New option tls-no-npn to disable sending TLS NPN extension.
 	All ssloptions= values for SSLv2 configurati

Re: [squid-dev] [PATCH] Some failed transactions are not logged

[Eduard Bagdasaryan]

>   2016-07-20 7:36 GMT+03:00 Amos Jeffries <squ...@treenet.co.nz>:
>On 20/07/2016 5:01 a.m., Alex Rousskov wrote:
>> On 07/19/2016 08:10 AM, Amos Jeffries wrote:
>>> On 20/07/2016 1:44 a.m., Eduard Bagdasaryan wrote:
>>>> 2016-07-19 16:17 GMT+03:00 Amos Jeffries:
>>>>> Is this patch going to include the new config option to prevent 
logging

>>>>> the new things? or do it in a followup?
>>>>
>>>> For now we are not planning to add this option(that is why 
initially the

>>>> patch did not perform logging for "no-bytes" connections).
>>>> Probably there should be a (new?) ACL, controlling number of received
>>>> bytes(instead of separate option). If so, implementing this requires
>>>> solving non-trivial separate task.
>>
>>> Okay. I was just thinking an on/off directive for now. So people could
>>> restore the old behaviour, or go ahead with the new logs.
>>
>> An ACL is the right way to control what gets logged -- this is the
>> access_log configuration interface we already support. If no existing
>> ACL(s) can match the transactions we want users to be able to match,
>> then we need to add more ACL(s). I suspect those new ACL(s) would be
>> useful for more than logging.
>>
>> A new config directive is not something we can add "for now" -- it would
>> have to be maintained for a long time while the overlap between the
>> existing ACL-driven interface and the new directive will cause pains for
>> developers, documentation writers, and admins.
>>
>>
>> Which ACL(s) to add depends on the final version of the new logging
>> code, requires careful thinking, and may require non-trivial
>> development. We want to keep all of that outside this project scope.
>> Amos, if we assume that this patch does not add new ACLs or directives,
>> are you happy with what it logs now or do you want Eduard to exclude
>> something?
>>
>
>Hmm. Good point.
>
>The 'problem' with ACLs was not the lack of checks, it is that the URL
>ones we already have use exclusively the HttpRequest URL and complain
>when one is not existing. It might be easier to extend those to use the
>error:* URL, in the same way the log itself is getting it for display.
>
>>
>>> I'll give Alex another day to point out any issues, with intention to
>>> apply this in ~24hrs.
>>
>> Thank you.
>>
>>
>>> error:accept-user-connection
>>
>> vs. recently discussed
>>
>>> annotate_client or annotate_client_connection
>>
>> All of these are about the same kind of connection to a Squid
>> protocol_port. Should we use "user" or "client"? I do not think just
>> "annotate_user" works because folks will think it is about the
>> [authenticated] user [agent] as a whole.
>
>Agreed. User is a specific subset of client. That 'does it require a
>person' is the first filter I am applying whenever anyone suggests
>"user" for anything.
>
>
>> Thus, we have the following
>> naming options:
>>
>> 1) error:accept-user-connection and annotate_user_connection
>>
>> 2) error:accept-client-connection and annotate_client
>>
>> 3) error:accept-client-connection and annotate_client_connection
>>
>>
>> FYI: I found a few references to "user" and ~30 references to "client"
>> in squid.conf directives and ACLS:
>>
>> * user_max_ip, user_cert, ext_user, ftp_user, and cache_effective_user;
>>
>> * client_dst_passthru, client_delay_*, *_uses_indirect_client,
>> client_db, client_idle_pconn_timeout, client_ip_max_connections,
>> client_lifetime, client_netmask, client_persistent_connections, and
>> client_request_buffer_max_size.
>>
>> Given the above, (2) or (3) seems like a better choice. Amos, please
>> pick one or confirm that you still want (1).
>>
>
>My inbox(es) contain no mention of these annotate_client / annotate_user
>you speak of being discussed. And my memory is also drawing a bank right
>now.
>
>The latest proposed patch of this thread failed the filter mentioned
>above for me. I'm asking for "error:accept-client-connection", which
>would be (2) or (3) of your offered list.
>
>
>>
>>> +// do not log connections that closed after a transaction (it 
is normal)

>> ...
>>> +// XXX: TLS bytes are consumed without going through inBuf
>>> +// XXX: PROXY protocol bytes are consumed without going 
through inBuf

>>> +if (receivedFirstByte_ 

Re: [squid-dev] [PATCH] Collapse internal revalidation requests (SMP-unaware caches)

[Eduard Bagdasaryan]

2016-07-20 16:21 GMT+03:00 Amos Jeffries :
> * smpAware() is documented as indicating whether CF is allowed. Yet the
> the new comment inside the loop of Store::Disks::smpAware() seems to be
> saying the opposite.

As this patch says, collapsing for revalidation requests is not
supported for SMP-aware caches, i.e. if we have configured several
caches and even one of them is SMP-aware, the new feature should be
disabled.

> * collapsing requires Transients to exist yes?
>  So why disable Transients initialization when !smpAware() ?

Transients is not required for collapsing to work for:
* Non-SMP Squid.
* SMP Squid with SMP-unaware-caches(like if we had several independent
   Squid instances). In this configuration no sharing among workes
   is supposed. Since Transients is a sharing mechanism, it should be
   disabled in this case.


Eduard.
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Some failed transactions are not logged

[Eduard Bagdasaryan]

2016-07-19 16:17 GMT+03:00 Amos Jeffries :
> Is this patch going to include the new config option to prevent logging
> the new things? or do it in a followup?

For now we are not planning to add this option(that is why initially the
patch did not perform logging for "no-bytes" connections).
Probably there should be a (new?) ACL, controlling number of received
bytes(instead of separate option). If so, implementing this requires
solving non-trivial separate task.


Eduard.
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Some failed transactions are not logged

[Eduard Bagdasaryan]

Addressed discussion concerns and refreshed the patch.

2016-07-19 8:13 GMT+03:00 Amos Jeffries :
> Is ftp_port traffic another one?

IMO this is not a case because Ftp::Server (similarly to Http::One::Server)
cares about inBuf consumtion and ClientHttpRequest objects creation.


Eduard.
Some failed transactions are not logged.

There are situations when Squid logs nothing to access.log after an
[abnormal] transaction termination. Such "stealthy" transactions may be
a security risk and an accounting problem.

ClientHttpRequest is responsible for logging most transactions but that
object is created only after the HTTP request headers are successfully
parsed. Request header parsing errors may be detected and logged
appropriately, but the job handling the incoming transaction may
terminate for reasons outside the parsing code control (e.g., a job-
killing exception thrown when there are no request headers to start
parsing yet or when the job waits for more request headers to finishing
parsing).

This change adds access logging for three cases:

1. accept(2) system call errors (before ConnStateData job is created).

2. Unexpected ConnStateData job termination, when there is no
   ClientHttpRequest to log the failure.

3. Connections which send no bytes: these connections drain Squid
   resources and, hence, should be logged.
   TODO: make this behavior configurable because some browsers are known to
   routinely create such connections(and, hence, logging them by default may
   create too much noise).

=== modified file 'doc/release-notes/release-4.sgml'
--- doc/release-notes/release-4.sgml	2016-07-01 11:30:00 +
+++ doc/release-notes/release-4.sgml	2016-07-19 10:29:44 +
@@ -199,60 +199,66 @@
 
 	request_start_timeout
 	New directive controlling how long Squid waits for the first request
 	   bytes to arrive after initial connection establishment by a client.
 
 	server_pconn_for_nonretriable
 	New directive to provide fine-grained control over persistent connection
 	   reuse when forwarding HTTP requests that Squid cannot retry. It is useful
 	   in environments where opening new connections is very expensive
 	   and race conditions associated with persistent connections are very rare
 	   and/or only cause minor problems.
 
 	shared_memory_locking
 	New directive to ensure shared memory is all available immediately
 	   on startup. Protects against SIGBUS errors, but delays startup.
 
 	tls_outgoing_options
 	New directive to define TLS security context options for outgoing
 	   connections. For example to HTTPS servers.
 
 	url_rewrite_timeout
 	Squid times active requests to redirector. This option sets
 	   the timeout value and the Squid reaction to a timed out
 	   request.
 
 
 
 Changes to existing tags
 
 
+	access_log
+	TCP accept(2) errors logged with URI error:accept-user-connection
+	Unused connections received in http_port or https_port
+	   or transactions terminated before reading[parsing] request headers
+	   logged with URI error:transaction-end-before-headers
+
 	acl
 	New -m flag for note ACL to match substrings.
 
 	auth_param
 	New parameter queue-size= to set the maximum number
 	   of queued requests.
 
 	cache_peer
 	New option auth-no-keytab to let GSSAPI implementation determine
 	   which Kerberos credentials to use, instead of specifying a keytab.
 	New option tls-min-version=1.N to set minimum TLS version allowed.
 	New option tls-default-ca replaces sslflags=NO_DEFAULT_CA
 	New option tls-no-npn to disable sending TLS NPN extension.
 	All ssloptions= values for SSLv2 configuration or disabling
 	   have been removed.
 	Removed sslversion= option. Use tls-options= instead.
 	Manual squid.conf update may be required on upgrade.
 	Replaced sslcafile= with tls-cafile= which takes multiple entries.
 
 	external_acl_type
 	New parameter queue-size= to set the maximum number
 	   of queued requests.
 	Format field updated to accept any logformat %macro code.
 
 	http_port
 	New option tls-min-version=1.N to set minimum TLS version allowed.
 	New option tls-default-ca replaces sslflags=NO_DEFAULT_CA
 	New option tls-no-npn to disable sending TLS NPN extension.
 	All option= values for SSLv2 configuration or disabling
 	   have been removed.

=== modified file 'src/Pipeline.cc'
--- src/Pipeline.cc	2016-01-24 17:41:43 +
+++ src/Pipeline.cc	2016-07-19 07:51:03 +
@@ -9,55 +9,67 @@
 /*
  * DEBUG: section 33Client Request Pipeline
  */
 #include "squid.h"
 #include "anyp/PortCfg.h"
 #include "client_side.h"
 #include "Debug.h"
 #include "http/Stream.h"
 #include "Pipeline.h"
 
 void
 Pipeline::add(const Http::StreamPointer )
 {
 requests.push_back(c);
 ++nrequests;
 debugs(33, 3, "Pipeline " << (void*)this << " add request " << nrequests << ' ' << c);
 }
 
 Http::StreamPointer
 Pipeline::front() const
 {
 if (requests.empty()) {
 debugs(33, 3, "Pipeline " << (void*)this << " empty");
 return Http::StreamPointer();
   

Re: [squid-dev] [PATCH] Some failed transactions are not logged

[Eduard Bagdasaryan]

Hello Amos,

2016-07-17 12:34 GMT+03:00 Amos Jeffries :
>  can these URI be logged with url_regex ACLs on access_log lines?

No because ACLUrlStrategy::requiresRequest() returns true.

> why is pipeline.back() being checked instead of pipeline.front() ?
> [not saying its wrong yet, just asking for a good reason.]

Since the pipeline acts as a FIFO we need the last-added pipeline
element to check whether it uses connection. I.e., if we pipelined
several requests, the CONNECT request (if we got it) will be the last,
because it starts using connection.

> logAcceptError() should use a distinct "error:" URI labeling this as a
> TCP accept error, since it has nothing to do with HTTP state.
Something like "error:user-connection-establishment" or
"error:open-user-connection"?  What do you think?


Eduard.
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] [PATCH] Some failed transactions are not logged

[Eduard Bagdasaryan]

Hello,

There are situations when Squid logs nothing to access.log after an
[abnormal] transaction termination. Such "stealthy" transactions may be
a security risk and an accounting problem.

ClientHttpRequest is responsible for logging most transactions but that
object is created only after the HTTP request headers are successfully
parsed. Request header parsing errors may be detected and logged
appropriately, but the job handling the incoming transaction may
terminate for reasons outside the parsing code control (e.g., a job-
killing exception thrown when there are no request headers to start
parsing yet or when the job waits for more request headers to finishing
parsing).

This change adds access logging for two cases:

1. accept(2) system call errors (before ConnStateData job is created);

2. unexpected ConnStateData job termination, when there is no
   ClientHttpRequest to log the failure.

TODO: Squid still logs nothing when the connection closes before reading
request header data. We should probably make that behavior configurable
because such connections drain Squid resources (and, hence, should be
logged) but some browsers are known to routinely create them (and,
hence, logging them by default may create too much noise).

Regards,
Eduard.

Some failed transactions are not logged.

There are situations when Squid logs nothing to access.log after an
[abnormal] transaction termination. Such "stealthy" transactions may be
a security risk and an accounting problem.

ClientHttpRequest is responsible for logging most transactions but that
object is created only after the HTTP request headers are successfully
parsed. Request header parsing errors may be detected and logged
appropriately, but the job handling the incoming transaction may
terminate for reasons outside the parsing code control (e.g., a job-
killing exception thrown when there are no request headers to start
parsing yet or when the job waits for more request headers to finishing
parsing).

This change adds access logging for two cases:

1. accept(2) system call errors (before ConnStateData job is created);

2. unexpected ConnStateData job termination, when there is no
   ClientHttpRequest to log the failure.

TODO: Squid still logs nothing when the connection closes before reading
request header data. We should probably make that behavior configurable
because such connections drain Squid resources (and, hence, should be
logged) but some browsers are known to routinely create them (and,
hence, logging them by default may create too much noise).

=== modified file 'src/Pipeline.cc'
--- src/Pipeline.cc	2016-01-24 17:41:43 +
+++ src/Pipeline.cc	2016-07-15 09:41:40 +
@@ -9,55 +9,67 @@
 /*
  * DEBUG: section 33Client Request Pipeline
  */
 #include "squid.h"
 #include "anyp/PortCfg.h"
 #include "client_side.h"
 #include "Debug.h"
 #include "http/Stream.h"
 #include "Pipeline.h"
 
 void
 Pipeline::add(const Http::StreamPointer )
 {
 requests.push_back(c);
 ++nrequests;
 debugs(33, 3, "Pipeline " << (void*)this << " add request " << nrequests << ' ' << c);
 }
 
 Http::StreamPointer
 Pipeline::front() const
 {
 if (requests.empty()) {
 debugs(33, 3, "Pipeline " << (void*)this << " empty");
 return Http::StreamPointer();
 }
 
 debugs(33, 3, "Pipeline " << (void*)this << " front " << requests.front());
 return requests.front();
 }
 
+Http::StreamPointer
+Pipeline::back() const
+{
+if (requests.empty()) {
+debugs(33, 3, "Pipeline " << (void*)this << " empty");
+return Http::StreamPointer();
+}
+
+debugs(33, 3, "Pipeline " << (void*)this << " back " << requests.back());
+return requests.back();
+}
+
 void
 Pipeline::terminateAll(int xerrno)
 {
 while (!requests.empty()) {
 Http::StreamPointer context = requests.front();
 debugs(33, 3, "Pipeline " << (void*)this << " notify(" << xerrno << ") " << context);
 context->noteIoError(xerrno);
 context->finished();  // cleanup and self-deregister
 assert(context != requests.front());
 }
 }
 
 void
 Pipeline::popMe(const Http::StreamPointer )
 {
 if (requests.empty())
 return;
 
 debugs(33, 3, "Pipeline " << (void*)this << " drop " << requests.front());
 // in reality there may be multiple contexts doing processing in parallel.
 // XXX: pipeline still assumes HTTP/1 FIFO semantics are obeyed.
 assert(which == requests.front());
 requests.pop_front();
 }
 

=== modified file 'src/Pipeline.h'
--- src/Pipeline.h	2016-01-24 17:41:43 +
+++ src/Pipeline.h	2016-07-15 09:41:40 +
@@ -19,53 +19,56 @@
  *
  * Transactions in the queue may be fully processed, but not yet delivered,
  * or only partially processed.
  *
  * - HTTP/1 pipelined requests can be processed out of order but
  *   responses MUST be written to the client in-order.
  *   The front() context is for the response writing transaction.
  *   The back context may still be reading a 

[squid-dev] Dealing with RegisteredHeadersHash.gperf

[Eduard Bagdasaryan]

Hello,

In my current task I need to change some of header definitions, but it is
unclear how to do this correctly. My understanding is that I need to
modify http/RegisteredHeadersHash.gperf, and then run
"make gperf-files" to generate RegisteredHeadersHash.cci. Is this correct?
The http/RegisteredHeadersHash.gperf has a comment that it was 
auto-generated

from RegisteredHeadersHash.gperf. How could it be generated from itself?

Thank you,
Eduard.
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Uninitialised errors during Squid startup

[Eduard Bagdasaryan]

Hello Amos,

> How does this interact with objects that begin their life as private
> then get converted to public keys?
I think that it is a typical situation: a request that is a miss
firstly gets a private key(so that other requests to the same URL still 
got misses).
It is converted to a public key when Squid gets headers of a [cachable] 
response. As I wrote
the private key is required to be unique, but I see no correlation 
between the private and

the public keys in such scenario.

> the comment about kidID and PID in the new private key blob as being
> just an optimization I dont think is correct. Without them I think we
> could have ID collisions in collapsed_forwarding, shared cacheh_mem, or
> rock across workers. That would make them required for proper operations
> in SMP.
AFAIK, private store entries are not cachable: see KEY_PRIVATE check in
StoreEntry::checkCachable(); this check is performed before entry is written
to shared memory or disk cache. So private entries are not shared among 
workers and

a unique per-process counter should be sufficient.

> How much testing has this had?
I tested with Valgrind and ran test-builds.sh.

Eduard.
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] [PATCH] Uninitialised errors during Squid startup

[Eduard Bagdasaryan]

Hello,

This patch fixes valgrind-discovered trunk errors.

During start-up, Valgrind reported many errors with a similar message:
"Use of uninitialised value of size 8...". These errors were caused by
HttpRequestMethod& parameter during private key generation:
it was used as a raw void* memory there. HttpRequestMethod is a non-POD
type and has some "padding" bytes, which are not initialized by the 
constructor.


The patch simplifies private key generation by using a counter for this,
which is sufficient to create a unique key within a single Squid process.
Except fixing Valgrind errors, this solution saves many CPU cycles wasted on
generating MD5 hashes just to "guarantee" uniqueness within a single 
process.


Regards,
Eduard.
Fix for valgrind-discovered trunk errors.

During start-up, Valgrind reported many errors with a similar
message: "Use of uninitialised value of size 8...". These errors
were caused by HttpRequestMethod& parameter during private
key generation: it was used as a raw void* memory there. HttpRequestMethod
is a non-POD type and has some "padding" bytes, which are not initialized
by the constructor.
The patch simplifies private key generation by using a simple counter for this,
which is sufficient to create a unique key within a single Squid process.
Except fixing Valgrind errors, this solution saves many CPU cycles wasted on
generating MD5 hashes just to "guarantee" uniqueness within a single process.

=== modified file 'src/store.cc'
--- src/store.cc	2016-05-01 21:37:52 +
+++ src/store.cc	2016-06-08 10:29:45 +
@@ -572,81 +572,76 @@
 if (e == NULL && req->method == Http::METHOD_HEAD)
 /* We can generate a HEAD reply from a cached GET object */
 e = storeGetPublicByRequestMethod(req, Http::METHOD_GET);
 
 return e;
 }
 
 static int
 getKeyCounter(void)
 {
 static int key_counter = 0;
 
 if (++key_counter < 0)
 key_counter = 1;
 
 return key_counter;
 }
 
 /* RBC 20050104 AFAICT this should become simpler:
  * rather than reinserting with a special key it should be marked
  * as 'released' and then cleaned up when refcounting indicates.
  * the StoreHashIndex could well implement its 'released' in the
  * current manner.
  * Also, clean log writing should skip over ia,t
  * Otherwise, we need a 'remove from the index but not the store
  * concept'.
  */
 void
 StoreEntry::setPrivateKey()
 {
-const cache_key *newkey;
-
 if (key && EBIT_TEST(flags, KEY_PRIVATE))
 return; /* is already private */
 
 if (key) {
 setReleaseFlag(); // will markForUnlink(); all caches/workers will know
 
 // TODO: move into SwapDir::markForUnlink() already called by Root()
 if (swap_filen > -1)
 storeDirSwapLog(this, SWAP_LOG_DEL);
 
 hashDelete();
 }
 
-if (mem_obj && mem_obj->hasUris()) {
+if (mem_obj && mem_obj->hasUris())
 mem_obj->id = getKeyCounter();
-newkey = storeKeyPrivate(mem_obj->storeId(), mem_obj->method, mem_obj->id);
-} else {
-newkey = storeKeyPrivate("JUNK", Http::METHOD_NONE, getKeyCounter());
-}
+const cache_key *newkey = storeKeyPrivate();
 
 assert(hash_lookup(store_table, newkey) == NULL);
 EBIT_SET(flags, KEY_PRIVATE);
 hashInsert(newkey);
 }
 
 void
 StoreEntry::setPublicKey()
 {
 const cache_key *newkey;
 
 if (key && !EBIT_TEST(flags, KEY_PRIVATE))
 return; /* is already public */
 
 assert(mem_obj);
 
 /*
  * We can't make RELEASE_REQUEST objects public.  Depending on
  * when RELEASE_REQUEST gets set, we might not be swapping out
  * the object.  If we're not swapping out, then subsequent
  * store clients won't be able to access object data which has
  * been freed from memory.
  *
  * If RELEASE_REQUEST is set, setPublicKey() should not be called.
  */
 #if MORE_DEBUG_OUTPUT
 
 if (EBIT_TEST(flags, RELEASE_REQUEST))
 debugs(20, DBG_IMPORTANT, "assertion failed: RELEASE key " << key << ", url " << mem_obj->url);
 

=== modified file 'src/store_key_md5.cc'
--- src/store_key_md5.cc	2016-03-25 20:11:29 +
+++ src/store_key_md5.cc	2016-06-08 11:44:52 +
@@ -53,72 +53,72 @@
 storeKeyHashCmp(const void *a, const void *b)
 {
 const unsigned char *A = (const unsigned char *)a;
 const unsigned char *B = (const unsigned char *)b;
 int i;
 
 for (i = 0; i < SQUID_MD5_DIGEST_LENGTH; ++i) {
 if (A[i] < B[i])
 return -1;
 
 if (A[i] > B[i])
 return 1;
 }
 
 return 0;
 }
 
 unsigned int
 storeKeyHashHash(const void *key, unsigned int n)
 {
 /* note, n must be a power of 2! */
 const unsigned char *digest = (const unsigned char *)key;
 unsigned int i = digest[0]
  | digest[1] << 8
  | digest[2] << 16
  | digest[3] << 24;
 return (i & (--n));
 }
 
 const cache_key *

[squid-dev] [PATCH] Use TCP_REFRESH_PENDING while waiting for IMS reply

[Eduard Bagdasaryan]

Hello,

This patch marks refresh-waiting transactions with TCP_REFRESH_PENDING.

Before this change, transactions initiating a refresh were still marked
as TCP_HIT*. If such a transaction was terminated (for any reason)
before receiving an IMS reply, it was logged with that misleading tag.
Now, such transactions are logged using TCP_REFRESH_PENDING[_ABORTED].

After the refresh (successful or otherwise), the tag changes to one of
the other TCP_REFRESH_* values, as before.

Regards,
Eduard.
Mark refresh-waiting transactions with TCP_REFRESH_PENDING.

Before this change, transactions initiating a refresh were still marked
as TCP_HIT*. If such a transaction was terminated (for any reason)
before receiving an IMS reply, it was logged with that misleading tag.
Now, such transactions are logged using TCP_REFRESH_PENDING[_ABORTED].

After the refresh (successful or otherwise), the tag changes to one of
the other TCP_REFRESH_* values, as before.

=== modified file 'src/LogTags.cc'
--- src/LogTags.cc	2016-01-01 00:12:18 +
+++ src/LogTags.cc	2016-05-31 19:37:07 +
@@ -18,6 +18,7 @@
 "TCP_REFRESH_FAIL_OLD",
 "TCP_REFRESH_FAIL_ERR",
 "TCP_REFRESH_MODIFIED",
+"TCP_REFRESH_PENDING",
 "TCP_CLIENT_REFRESH_MISS",
 "TCP_IMS_HIT",
 "TCP_SWAPFAIL_MISS",

=== modified file 'src/LogTags.h'
--- src/LogTags.h	2016-01-01 00:12:18 +
+++ src/LogTags.h	2016-05-31 19:54:28 +
@@ -25,6 +25,7 @@
 LOG_TCP_REFRESH_FAIL_OLD,   // refresh from origin failed, stale reply sent
 LOG_TCP_REFRESH_FAIL_ERR,   // refresh from origin failed, error forwarded
 LOG_TCP_REFRESH_MODIFIED,   // refresh from origin replaced existing entry
+LOG_TCP_REFRESH_PENDING,// refresh from origin still pending when transaction aborted
 LOG_TCP_CLIENT_REFRESH_MISS,
 LOG_TCP_IMS_HIT,
 LOG_TCP_SWAPFAIL_MISS,

=== modified file 'src/client_side_reply.cc'
--- src/client_side_reply.cc	2016-05-10 23:06:48 +
+++ src/client_side_reply.cc	2016-05-31 19:37:07 +
@@ -278,6 +278,7 @@
 return;
 }
 
+http->logType = LOG_TCP_REFRESH_PENDING;
 http->request->flags.refresh = true;
 #if STORE_CLIENT_LIST_DEBUG
 /* Prevent a race with the store client memory free routines

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] [PATCH] bug 4485 fix

[Eduard Bagdasaryan]

Hello,

This patch fixes bug 4485 and adjusts related test cases
to fully check Parser::Tokenizer::int64() post-conditions.

Regards,
Eduard.
Fixed bug 4485:
off-by-one out-of-bounds Parser::Tokenizer::int64() read errors.
Also adjusted related test cases to fully check Parser::Tokenizer::int64()
post-conditions.

=== modified file 'src/parser/Tokenizer.cc'
--- src/parser/Tokenizer.cc	2016-01-01 00:12:18 +
+++ src/parser/Tokenizer.cc	2016-05-21 12:00:33 +
@@ -200,85 +200,86 @@
 debugs(24, 8, "no match when trying to skip " << skippable.name);
 return 0;
 }
 debugs(24, 8, "skipping in " << skippable.name << " len " << suffixLen);
 return successTrailing(suffixLen);
 }
 
 /* reworked from compat/strtoll.c */
 bool
 Parser::Tokenizer::int64(int64_t & result, int base, bool allowSign, const SBuf::size_type limit)
 {
 if (atEnd() || limit == 0)
 return false;
 
 const SBuf range(buf_.substr(0,limit));
 
 //fixme: account for buf_.size()
 bool neg = false;
 const char *s = range.rawContent();
 const char *end = range.rawContent() + range.length();
 
 if (allowSign) {
 if (*s == '-') {
 neg = true;
 ++s;
 } else if (*s == '+') {
 ++s;
 }
 if (s >= end) return false;
 }
-if (( base == 0 || base == 16) && *s == '0' && (s+1 <= end ) &&
+if (( base == 0 || base == 16) && *s == '0' && (s+1 < end ) &&
 tolower(*(s+1)) == 'x') {
 s += 2;
 base = 16;
 }
 if (base == 0) {
 if ( *s == '0') {
 base = 8;
 ++s;
 } else {
 base = 10;
 }
 }
 if (s >= end) return false;
 
 uint64_t cutoff;
 
 cutoff = neg ? -static_cast(INT64_MIN) : INT64_MAX;
 const int cutlim = cutoff % static_cast(base);
 cutoff /= static_cast(base);
 
 int any = 0, c;
 int64_t acc = 0;
-for (c = *s++; s <= end; c = *s++) {
+do {
+c = *s;
 if (xisdigit(c)) {
 c -= '0';
 } else if (xisalpha(c)) {
 c -= xisupper(c) ? 'A' - 10 : 'a' - 10;
 } else {
 break;
 }
 if (c >= base)
 break;
 if (any < 0 || static_cast(acc) > cutoff || (static_cast(acc) == cutoff && c > cutlim))
 any = -1;
 else {
 any = 1;
 acc *= base;
 acc += c;
 }
-}
+} while (++s < end);
 
 if (any == 0) // nothing was parsed
 return false;
 if (any < 0) {
 acc = neg ? INT64_MIN : INT64_MAX;
 errno = ERANGE;
 return false;
 } else if (neg)
 acc = -acc;
 
 result = acc;
-return success(s - range.rawContent() - 1);
+return success(s - range.rawContent());
 }
 

=== modified file 'src/tests/testTokenizer.cc'
--- src/tests/testTokenizer.cc	2016-01-01 00:12:18 +
+++ src/tests/testTokenizer.cc	2016-05-21 20:59:43 +
@@ -177,134 +177,142 @@
 
 // match until the end of the sample
 CPPUNIT_ASSERT(t.suffix(s, all));
 CPPUNIT_ASSERT_EQUAL(SBuf(), t.remaining());
 
 // an empty buffer does not end with a token
 s = canary;
 CPPUNIT_ASSERT(!t.suffix(s, all));
 CPPUNIT_ASSERT_EQUAL(canary, s); // no parameter changes
 
 // we cannot skip an empty suffix, even in an empty buffer
 CPPUNIT_ASSERT(!t.skipSuffix(SBuf()));
 }
 
 void
 testTokenizer::testCharacterSet()
 {
 
 }
 
 void
 testTokenizer::testTokenizerInt64()
 {
 // successful parse in base 10
 {
 int64_t rv;
 Parser::Tokenizer t(SBuf("1234"));
 const int64_t benchmark = 1234;
 CPPUNIT_ASSERT(t.int64(rv, 10));
 CPPUNIT_ASSERT_EQUAL(benchmark,rv);
+CPPUNIT_ASSERT(t.buf().isEmpty());
 }
 
 // successful parse, autodetect base
 {
 int64_t rv;
 Parser::Tokenizer t(SBuf("1234"));
 const int64_t benchmark = 1234;
 CPPUNIT_ASSERT(t.int64(rv));
 CPPUNIT_ASSERT_EQUAL(benchmark,rv);
+CPPUNIT_ASSERT(t.buf().isEmpty());
 }
 
 // successful parse, autodetect base
 {
 int64_t rv;
 Parser::Tokenizer t(SBuf("01234"));
 const int64_t benchmark = 01234;
 CPPUNIT_ASSERT(t.int64(rv));
 CPPUNIT_ASSERT_EQUAL(benchmark,rv);
+CPPUNIT_ASSERT(t.buf().isEmpty());
 }
 
 // successful parse, autodetect base
 {
 int64_t rv;
 Parser::Tokenizer t(SBuf("0x12f4"));
 const int64_t benchmark = 0x12f4;
 CPPUNIT_ASSERT(t.int64(rv));
 CPPUNIT_ASSERT_EQUAL(benchmark,rv);
+CPPUNIT_ASSERT(t.buf().isEmpty());
 }
 
 // API mismatch: don't eat leading space
 {
 int64_t rv;
 Parser::Tokenizer t(SBuf(" 1234"));
 CPPUNIT_ASSERT(!t.int64(rv));
+CPPUNIT_ASSERT_EQUAL(SBuf(" 1234"), t.buf());
 }
 
 // API mismatch: don't eat multiple leading spaces
 {
 

[squid-dev] [PATCH] Do not hide important/critical messages

[Eduard Bagdasaryan]

Hello,

This is a trunk port for Alex's v3.5 reentrant debugging fix 
.


Regards,
Eduard.
Do not allow low-level debugging to hide important/critical messages.

Removed debugs() side effects that inadvertently resulted in some
important/critical messages logged at the wrong debugging level and,
hence, becoming invisible to the admin. The removed side effects set the
"current" debugging level when a debugs() parameter called a function
that also called debugs(). The last nested debugs() called affected the
level of all debugs() still in progress!

Related changes:

* Reentrant debugging messages no longer clobber parent messages. Each
  debugging message is logged separately, in the natural order of
  debugs() calls that would have happened if debugs() were a function
  (that gets already evaluated arguments) and not a macro (that
  evaluates its arguments in the middle of the call). This order is
  "natural" because good macros work like functions from the caller
  point of view.

* Assertions hit while evaluating debugs() parameters are now logged
  instead of being lost with the being-built debugs() log line.

* 10-20% faster debugs() performance because we no longer allocate a new
  std::ostringstream buffer for the vast majority of debugs() calls.
  Only reentrant calls get a new buffer.

* Removed old_debug(), addressing an old "needs to die" to-do.

* Removed do_debug() that changed debugging level while testing whether
  debugging is needed. Use side-effect-free Debug::Enabled() instead.

Also removed the OutStream wrapper class. The wrapper was added in trunk 
revision 13767 that promised to (but did not?) MemPool the debug output
buffers. We no longer "new" the buffer stream so a custom new() method
would be unused. Besides, the r13767 explanation implied that providing
a Child::new() method would somehow overwrite Parent::allocator_type,
which did not compute for me. Finally, Squid "new"s other allocator-
enabled STL objects without overriding their new methods so either the
same problem is still there or it did not exist (or was different?).

Also removed Debug::xassert() because the debugs() assertions now work
OK without that hack.

=== modified file 'src/Debug.h'
--- src/Debug.h	2016-01-01 00:12:18 +
+++ src/Debug.h	2016-05-19 08:48:18 +
@@ -25,153 +25,164 @@
 #if PURIFY
 #define assert(EX) ((void)0)
 #elif defined(NODEBUG)
 #define assert(EX) ((void)0)
 #elif STDC_HEADERS
 #define assert(EX)  ((EX)?((void)0):xassert( # EX , __FILE__, __LINE__))
 #else
 #define assert(EX)  ((EX)?((void)0):xassert("EX", __FILE__, __LINE__))
 #endif
 
 /* context-based debugging, the actual type is subject to change */
 typedef int Ctx;
 Ctx ctx_enter(const char *descr);
 void ctx_exit(Ctx ctx);
 
 /* defined debug section limits */
 #define MAX_DEBUG_SECTIONS 100
 
 /* defined names for Debug Levels */
 #define DBG_CRITICAL0   /**< critical messages always shown when they occur */
 #define DBG_IMPORTANT   1   /**< important messages always shown when their section is being checked */
 /* levels 2-8 are still being discussed amongst the developers */
 #define DBG_DATA9   /**< output is a large data dump only necessary for advanced debugging */
 
 #define DBG_PARSE_NOTE(x) (opt_parse_cfg_only?0:(x)) /**< output is always to be displayed on '-k parse' but at level-x normally. */
 
 class Debug
 {
 
 public:
+/// meta-information for debugs() or a similar debugging call
+class Context
+{
+public:
+Context(const int aSectionLevel, const int aLevel);
+
+int level; ///< minimum debugging level required by the debugs() call
+int sectionLevel; ///< maximum debugging level allowed during the call
+
+private:
+friend class Debug;
+void rewind(const int aSection, const int aLevel);
+void formatStream();
+Context *upper; ///< previous or parent record in nested debugging calls
+std::ostringstream buf; ///< debugs() output sink
+};
+
+/// whether debugging the given section and the given level produces output
+static bool Enabled(const int section, const int level)
+{
+return level <= Debug::Levels[section];
+}
+
 static char *debugOptions;
 static char *cache_log;
 static int rotateNumber;
 static int Levels[MAX_DEBUG_SECTIONS];
-static int level; ///< minimum debugging level required by debugs() call
-static int sectionLevel; ///< maximum debugging level allowed now
 static int override_X;
 static int log_stderr;
 static bool log_syslog;
 
-static std::ostream ();
-static void finishDebug();
 static void parseOptions(char const *);
 
+/// minimum level required by the current debugs() call
+static int Level() { return Current ? Current->level : 1; }
+/// maximum level currently allowed
+static int SectionLevel() { return Current ? Current->sectionLevel : 1; }

[squid-dev] [PATCH] Anticipated connection closure disables chunking

[Eduard Bagdasaryan]

Hello,

This patch allows chunking the last HTTP response on a connection.
Squid should avoid signaling the message end by connection closure
because it hurts message integrity and sometimes performance. Squid
now chunks if:

  1. the response has a body;
  2. the client claims HTTP/1.1 support; and
  3. Squid is not going to send a Content-Length header.

Squid used to exclude to-be-closed connections from chunking
because chunking support was added (trunk r10781) specifically to
optimize persistent connection reuse and closing connections were
incorrectly excluded as a non-interesting/out-of-scope case.

Regards,
Eduard.
Allow chunking the last HTTP response on a connection.

Squid should avoid signaling the message end by connection closure
because it hurts message integrity and sometimes performance. Squid
now chunks if:

  1. the response has a body;
  2. the client claims HTTP/1.1 support; and
  3. Squid is not going to send a Content-Length header.

AFAICT, Squid used to exclude to-be-closed connections from chunking
because chunking support was added (trunk r10781) specifically to 
optimize persistent connection reuse and closing connections were 
incorrectly excluded as a non-interesting/out-of-scope case. And/or
perhaps we did not realize the dangers of signaling the message end
by connection closure.

=== modified file 'src/client_side_reply.cc'
--- src/client_side_reply.cc	2016-03-11 18:00:51 +
+++ src/client_side_reply.cc	2016-05-04 21:35:29 +
@@ -1507,63 +1507,61 @@
 } else if (request->flags.proxyKeepalive && shutting_down) {
 debugs(88, 3, "clientBuildReplyHeader: Shutting down, don't keep-alive.");
 request->flags.proxyKeepalive = false;
 } else if (request->flags.connectionAuth && !reply->keep_alive) {
 debugs(33, 2, "clientBuildReplyHeader: Connection oriented auth but server side non-persistent");
 request->flags.proxyKeepalive = false;
 } else if (reply->bodySize(request->method) < 0 && !maySendChunkedReply) {
 debugs(88, 3, "clientBuildReplyHeader: can't keep-alive, unknown body size" );
 request->flags.proxyKeepalive = false;
 } else if (fdUsageHigh()&& !request->flags.mustKeepalive) {
 debugs(88, 3, "clientBuildReplyHeader: Not many unused FDs, can't keep-alive");
 request->flags.proxyKeepalive = false;
 } else if (request->flags.sslBumped && !reply->persistent()) {
 // We do not really have to close, but we pretend we are a tunnel.
 debugs(88, 3, "clientBuildReplyHeader: bumped reply forces close");
 request->flags.proxyKeepalive = false;
 } else if (request->pinnedConnection() && !reply->persistent()) {
 // The peer wants to close the pinned connection
 debugs(88, 3, "pinned reply forces close");
 request->flags.proxyKeepalive = false;
 } else if (http->getConn()) {
 ConnStateData * conn = http->getConn();
 if (!Comm::IsConnOpen(conn->port->listenConn)) {
 // The listening port closed because of a reconfigure
 debugs(88, 3, "listening port closed");
 request->flags.proxyKeepalive = false;
 }
 }
 
 // Decide if we send chunked reply
-if (maySendChunkedReply &&
-request->flags.proxyKeepalive &&
-reply->bodySize(request->method) < 0) {
+if (maySendChunkedReply && reply->bodySize(request->method) < 0) {
 debugs(88, 3, "clientBuildReplyHeader: chunked reply");
 request->flags.chunkedReply = true;
 hdr->putStr(Http::HdrType::TRANSFER_ENCODING, "chunked");
 }
 
 /* Append VIA */
 if (Config.onoff.via) {
 LOCAL_ARRAY(char, bbuf, MAX_URL + 32);
 String strVia;
 hdr->getList(Http::HdrType::VIA, );
 snprintf(bbuf, MAX_URL + 32, "%d.%d %s",
  reply->sline.version.major,
  reply->sline.version.minor,
  ThisCache);
 strListAdd(, bbuf, ',');
 hdr->delById(Http::HdrType::VIA);
 hdr->putStr(Http::HdrType::VIA, strVia.termedBuf());
 }
 /* Signal keep-alive or close explicitly */
 hdr->putStr(Http::HdrType::CONNECTION, request->flags.proxyKeepalive ? "keep-alive" : "close");
 
 #if ADD_X_REQUEST_URI
 /*
  * Knowing the URI of the request is useful when debugging persistent
  * connections in a client; we cannot guarantee the order of http headers,
  * but X-Request-URI is likely to be the very last header to ease use from a
  * debugger [hdr->entries.count-1].
  */
 hdr->putStr(Http::HdrType::X_REQUEST_URI,
 http->memOjbect()->url ? http->memObject()->url : http->uri);

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev