date:20140430

Re: How long is a domain or url can be?

2014-04-30 Thread Henrik Nordström

ons 2014-04-30 klockan 06:22 +1200 skrev Amos Jeffries:

 HTTP defines no limit.
  - squid defines MAX_URL of 8KB, along with a header line limit of 64KB
 total, and a helper line limit of 32KB total.

Unless it has been fixed the UFS based stores also have an implicit
limit on cached entries somewhat less than 4KB (whole meta header need
to fit in first 4KB). Entries failing this gets cached but can never get
hit.

 DNS defines X.Y.Z segments as being no longer than 255 bytes *each*.

For Internet host names the limits are 63 octets per label, and 255
octects in total including dot delimiters.

Regards
Henrik

Re: [PATCH] logformat annotation fixes

2014-04-30 Thread Tsantilas Christos

patch applied to trunk as revno: 13387

On 04/24/2014 05:12 PM, Tsantilas Christos wrote:
 Currently note values printed with %note formating code,  which
 contain non alphanumeric characters, were quoted and quotes were then
 escaped, resulting in bizarre logged rendition of empty or simple values
 (often received from various helpers):
   %22-%22
   %22Default_Google%22
   %22pg13,US%22

 This patch:
 - does not use quotes to print annotations

 - Allow system admin to define a separator to use for logged
 annotations. The %note logformat accepts the following argument:
  [name][:separator]
 The separator can be one of the ',' ';' or ':'. By default, multiple
 note values are separated with , and multiple notes are separated with
 \r\n. When logging named notes with %{name}note, the explicitly
 configured separator is used between note values. When logging all notes
 with %note, the explicitly configured separator is used between
 individual notes. There is currently no way to specify both value and
 notes separators when logging all notes with %note.

 - Makes the Format::Token::data a struct (now is a union) and initialize
 Format::Token::data data members in Format::Token::Token constructor.

 This is a Measurement Factory project

Build failed in Jenkins: 3.HEAD-amd64-FreeBSD-10 #57

2014-04-30 Thread noc

See http://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10/57/changes

Changes:

[Christos Tsantilas] Logformat annotation fixes

Currently note values printed with %note formating code, which contain non
alphanumeric characters, were quoted and quotes were then escaped, resulting
in bizarre logged rendition of empty or simple values (often received from
various helpers):
%22-%22
%22Default_Google%22
%22pg13,US%22

This patch:
- does not use quotes to print annotations

- allow system admin to define a separator to use for logged
annotations. The %note logformat accepts the following argument:
[name][:separator]
The separator can be one of the ',' ';' or ':'.
By default, multiple note values are separated with , and multiple
notes are separated with \r\n. When logging named notes with
%{name}note, the explicitly configured separator is used between note
values. When logging all notes with %note, the explicitly configured
separator is used between individual notes. There is currently no way to
specify both value and notes separators when logging all notes with %note.

- makes the Format::Token::data a struct (now is a union) and initialize
Format::Token::data data members in Format::Token::Token constructor.

This is a Measurement Factory project

--
[...truncated 3547 lines...]
Making all in NCSA
--- basic_ncsa_auth.o ---
--- crypt_md5.o ---
--- basic_ncsa_auth.o ---
/usr/local/bin/ccache g++ -DHAVE_CONFIG_H  -I../../../.. -I../../../../include 
-I../../../../lib  -I../../../../src -I../../../include  -I/usr/local/include 
-I/usr/include  -I/usr/include -I../../../../libltdl 
-I../../../../helpers/basic_auth/NCSA   -I/usr/include  -I/usr/include -Wall 
-Wpointer-arith -Wwrite-strings -Wcomments -Wshadow -Werror -pipe -D_REENTRANT 
-g -O2 -I/usr/local/include -MT basic_ncsa_auth.o -MD -MP -MF 
.deps/basic_ncsa_auth.Tpo -c -o basic_ncsa_auth.o 
../../../../helpers/basic_auth/NCSA/basic_ncsa_auth.cc
--- crypt_md5.o ---
/usr/local/bin/ccache g++ -DHAVE_CONFIG_H  -I../../../.. -I../../../../include 
-I../../../../lib  -I../../../../src -I../../../include  -I/usr/local/include 
-I/usr/include  -I/usr/include -I../../../../libltdl 
-I../../../../helpers/basic_auth/NCSA   -I/usr/include  -I/usr/include -Wall 
-Wpointer-arith -Wwrite-strings -Wcomments -Wshadow -Werror -pipe -D_REENTRANT 
-g -O2 -I/usr/local/include -MT crypt_md5.o -MD -MP -MF .deps/crypt_md5.Tpo -c 
-o crypt_md5.o ../../../../helpers/basic_auth/NCSA/crypt_md5.cc
mv -f .deps/crypt_md5.Tpo .deps/crypt_md5.Po
--- basic_ncsa_auth.o ---
mv -f .deps/basic_ncsa_auth.Tpo .deps/basic_ncsa_auth.Po
--- basic_ncsa_auth ---
/bin/sh ../../../libtool  --tag=CXX--mode=link /usr/local/bin/ccache g++ 
-Wall -Wpointer-arith -Wwrite-strings -Wcomments -Wshadow -Werror -pipe 
-D_REENTRANT  -g -O2 -I/usr/local/include  -g -L/usr/local/lib 
-Wl,-R/usr/local/lib -pthread -o basic_ncsa_auth basic_ncsa_auth.o  crypt_md5.o 
../../../lib/libmisccontainers.la  ../../../lib/libmiscencoding.la  
../../../compat/libcompat-squid.la   -lnettle  -lcrypt-lm 
libtool: link: /usr/local/bin/ccache g++ -Wall -Wpointer-arith -Wwrite-strings 
-Wcomments -Wshadow -Werror -pipe -D_REENTRANT -g -O2 -I/usr/local/include -g 
-Wl,-R/usr/local/lib -pthread -o basic_ncsa_auth basic_ncsa_auth.o crypt_md5.o  
-L/usr/local/lib ../../../lib/.libs/libmisccontainers.a 
../../../lib/.libs/libmiscencoding.a ../../../compat/.libs/libcompat-squid.a 
-lnettle -lcrypt -lm -pthread
Making all in PAM
--- basic_pam_auth.o ---
/usr/local/bin/ccache g++ -DHAVE_CONFIG_H  -I../../../.. -I../../../../include 
-I../../../../lib  -I../../../../src -I../../../include  -I/usr/local/include 
-I/usr/include  -I/usr/include -I../../../../libltdl   -I/usr/include  
-I/usr/include -Wall -Wpointer-arith -Wwrite-strings -Wcomments -Wshadow 
-Werror -pipe -D_REENTRANT -g -O2 -I/usr/local/include -MT basic_pam_auth.o -MD 
-MP -MF .deps/basic_pam_auth.Tpo -c -o basic_pam_auth.o 
../../../../helpers/basic_auth/PAM/basic_pam_auth.cc
mv -f .deps/basic_pam_auth.Tpo .deps/basic_pam_auth.Po
--- basic_pam_auth ---
/bin/sh ../../../libtool  --tag=CXX--mode=link /usr/local/bin/ccache g++ 
-Wall -Wpointer-arith -Wwrite-strings -Wcomments -Wshadow -Werror -pipe 
-D_REENTRANT  -g -O2 -I/usr/local/include  -g -L/usr/local/lib 
-Wl,-R/usr/local/lib -pthread -o basic_pam_auth basic_pam_auth.o 
../../../lib/libmiscencoding.la  ../../../compat/libcompat-squid.la   -lpam  
-lm 
libtool: link: /usr/local/bin/ccache g++ -Wall -Wpointer-arith -Wwrite-strings 
-Wcomments -Wshadow -Werror -pipe -D_REENTRANT -g -O2 -I/usr/local/include -g 
-Wl,-R/usr/local/lib -pthread -o basic_pam_auth basic_pam_auth.o  
-L/usr/local/lib ../../../lib/.libs/libmiscencoding.a 
../../../compat/.libs/libcompat-squid.a -lpam -lm -pthread
Making all in POP3
--- basic_pop3_auth ---
sed -e 's,[@]PERL[@],/usr/bin/perl,g' 
../../../../helpers/basic_auth/POP3/basic_pop3_auth.pl.in basic_pop3_auth || 
(/bin/rm -f -f

Build failed in Jenkins: 3.HEAD-amd64-FreeBSD-10-clang #57

2014-04-30 Thread noc

See http://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/57/changes

Changes:

[Christos Tsantilas] Logformat annotation fixes

Currently note values printed with %note formating code, which contain non
alphanumeric characters, were quoted and quotes were then escaped, resulting
in bizarre logged rendition of empty or simple values (often received from
various helpers):
%22-%22
%22Default_Google%22
%22pg13,US%22

This patch:
- does not use quotes to print annotations

- allow system admin to define a separator to use for logged
annotations. The %note logformat accepts the following argument:
[name][:separator]
The separator can be one of the ',' ';' or ':'.
By default, multiple note values are separated with , and multiple
notes are separated with \r\n. When logging named notes with
%{name}note, the explicitly configured separator is used between note
values. When logging all notes with %note, the explicitly configured
separator is used between individual notes. There is currently no way to
specify both value and notes separators when logging all notes with %note.

- makes the Format::Token::data a struct (now is a union) and initialize
Format::Token::data data members in Format::Token::Token constructor.

This is a Measurement Factory project

--
[...truncated 5757 lines...]
--- Server.o ---
depbase=`echo Server.o | sed 's|[^/]*$|.deps/|;s|\.o$||'`; ccache clang++ 
-DHAVE_CONFIG_H 
-DDEFAULT_CONFIG_FILE=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/etc/squid.conf\;
  
-DDEFAULT_SQUID_DATA_DIR=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/share\;
  
-DDEFAULT_SQUID_CONFIG_DIR=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/etc\;
  -I../.. -I../../include -I../../lib  -I../../src -I../include  
-I/usr/local/include -I/usr/include  -I/usr/include -I../../libltdl  -I../src 
-I../../libltdl -I/usr/include  -I/usr/include   -I/usr/include  -I/usr/include 
-Werror -Qunused-arguments  -D_REENTRANT -g -O2 -I/usr/local/include -MT 
Server.o -MD -MP -MF $depbase.Tpo -c -o Server.o ../../src/Server.cc  mv -f 
$depbase.Tpo $depbase.Po
--- SwapDir.o ---
depbase=`echo SwapDir.o | sed 's|[^/]*$|.deps/|;s|\.o$||'`; ccache clang++ 
-DHAVE_CONFIG_H 
-DDEFAULT_CONFIG_FILE=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/etc/squid.conf\;
  
-DDEFAULT_SQUID_DATA_DIR=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/share\;
  
-DDEFAULT_SQUID_CONFIG_DIR=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/etc\;
  -I../.. -I../../include -I../../lib  -I../../src -I../include  
-I/usr/local/include -I/usr/include  -I/usr/include -I../../libltdl  -I../src 
-I../../libltdl -I/usr/include  -I/usr/include   -I/usr/include  -I/usr/include 
-Werror -Qunused-arguments  -D_REENTRANT -g -O2 -I/usr/local/include -MT 
SwapDir.o -MD -MP -MF $depbase.Tpo -c -o SwapDir.o ../../src/SwapDir.cc  mv 
-f $depbase.Tpo $depbase.Po
--- Transients.o ---
depbase=`echo Transients.o | sed 's|[^/]*$|.deps/|;s|\.o$||'`; ccache clang++ 
-DHAVE_CONFIG_H 
-DDEFAULT_CONFIG_FILE=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/etc/squid.conf\;
  
-DDEFAULT_SQUID_DATA_DIR=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/share\;
  
-DDEFAULT_SQUID_CONFIG_DIR=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/etc\;
  -I../.. -I../../include -I../../lib  -I../../src -I../include  
-I/usr/local/include -I/usr/include  -I/usr/include -I../../libltdl  -I../src 
-I../../libltdl -I/usr/include  -I/usr/include   -I/usr/include  -I/usr/include 
-Werror -Qunused-arguments  -D_REENTRANT -g -O2 -I/usr/local/include -MT 
Transients.o -MD -MP -MF $depbase.Tpo -c -o Transients.o 
../../src/Transients.cc  mv -f $depbase.Tpo $depbase.Po
--- MemStore.o ---
depbase=`echo MemStore.o | sed 's|[^/]*$|.deps/|;s|\.o$||'`; ccache clang++ 
-DHAVE_CONFIG_H 
-DDEFAULT_CONFIG_FILE=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/etc/squid.conf\;
  
-DDEFAULT_SQUID_DATA_DIR=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/share\;
  
-DDEFAULT_SQUID_CONFIG_DIR=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/etc\;
  -I../.. -I../../include -I../../lib  -I../../src -I../include  
-I/usr/local/include -I/usr/include  -I/usr/include -I../../libltdl  -I../src 
-I../../libltdl -I/usr/include

Build failed in Jenkins: 3.HEAD-amd64-OpenBSD-5.4 #60

2014-04-30 Thread noc

See http://build.squid-cache.org/job/3.HEAD-amd64-OpenBSD-5.4/60/

--
Started by upstream project 3.HEAD-amd64-centos-6 build number 319
originally caused by:
 Started by an SCM change
Building remotely on ypg-openbsd-54 (gcc farm amd64-openbsd 5.4 openbsd-5.4 
openbsd amd64-openbsd-5.4 amd64) in workspace 
http://build.squid-cache.org/job/3.HEAD-amd64-OpenBSD-5.4/ws/
Cleaning workspace...
$ bzr branch http://bzr.squid-cache.org/bzr/squid3/trunk/ 
http://build.squid-cache.org/job/3.HEAD-amd64-OpenBSD-5.4/ws/
bzr: ERROR: Connection error: Couldn't resolve host 'bzr.squid-cache.org' 
[Errno -5] no address associated with name
ERROR: Failed to branch http://bzr.squid-cache.org/bzr/squid3/trunk/
Retrying after 10 seconds
Cleaning workspace...
$ bzr branch http://bzr.squid-cache.org/bzr/squid3/trunk/ 
http://build.squid-cache.org/job/3.HEAD-amd64-OpenBSD-5.4/ws/
bzr: ERROR: Connection error: Couldn't resolve host 'bzr.squid-cache.org' 
[Errno -5] no address associated with name
ERROR: Failed to branch http://bzr.squid-cache.org/bzr/squid3/trunk/
Retrying after 10 seconds
Cleaning workspace...
$ bzr branch http://bzr.squid-cache.org/bzr/squid3/trunk/ 
http://build.squid-cache.org/job/3.HEAD-amd64-OpenBSD-5.4/ws/
bzr: ERROR: Connection error: Couldn't resolve host 'bzr.squid-cache.org' 
[Errno -5] no address associated with name
ERROR: Failed to branch http://bzr.squid-cache.org/bzr/squid3/trunk/

Build failed in Jenkins: 3.HEAD-amd64-FreeBSD-10 #58

2014-04-30 Thread noc

See http://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10/58/changes

Changes:

[Christos Tsantilas] author: Alex Rousskov rouss...@measurement-factory.com
cache_peer standby=N implementation.

The feature focus is to instantly provide a ready-to-use connection to a
cooperating cache peer, virtually at all times. This is useful when connection
establishment is too slow and/or when infrequent peer use prevents Squid from
combating slow connection establishment with the regular idle connection pool.

The feature is similar to Squid2 idle=N feature, but there are key differences:

* Standby connections are available virtually at all times, while Squid2 unused
idle connections are available only for a short time after a peer request.

* All N standby connections are not opened at once, reducing the chance of
the feature being mistaken for a DoS attack on a peer.

* More consistent support for peers with multiple IP addresses (peer IPs are
cycled through, just like during regular Squid request forwarding).

Besides, idle is a poor choice of adjective for an unused connection pool
name because the same term is used for used persistent connections, which have
somewhat different properties, are stored in a different pool, may need
distinct set of tuning options, etc. It is better to use a dedicated term for
the new feature.

The relationship between the max-conn limit and standby/idle connections is a
complex one. After several rewrites and tests, Squid now obeys max-conn limit
when opening new standby connections and accounts for standby connections when
checking whether to allow peer use. This often works OK, but leads to standby
guarantee violations when non-standby connections approach the limit. The
alternative design where standby code ignores max-conn works better, but is
really difficult to explain and advocate because an admin expects max-conn to
cover all connections and because of the idle connections accounting and
maintenance bugs. We may come back to this when the idle connections code is
fixed.

Fixed max-conn documentation and XXXed a peerHTTPOkay() bug (now in
peerHasConnAvailable()) that results in max-conn limit preventing the use of a
peer with idle persistent connections.

Decided to use standby connections for non-retriable requests. Avoiding
standby connections for POSTs and such would violate the main purpose of the
feature: providing an instant ready-to-use connection. A user does not care
whether it is waiting too long for a GET or POST request. Actually, a user may
care more when their POST requests are delayed (because canceling and
retrying them is often scary from the user point of view). The idea behind
standby connections is that the admin is responsible for avoiding race
conditions by properly configuring the peering Squids. If such proper
configuration is not possible or the consequences of rare races (e.g., due to
peer shutdown) are more severe than the consequences of slow requests, the
admin should not use standby=N. This choice may become configurable in the
future.

TODO: Teach peer probing code to push successful probing connections into the
standby pool (when enabled). Should be done as a followup project because of
the differences in standby and probe connection opening code, especially when
SSL peers are supported. Will require some discussion.

A standby pool is using a full-blown PconnPool object for storage instead of
the smaller IdleConnList, like the ICAP code does. The primary reasons for
this design were:

* A peer may have multiple addresses and those addresses may change. PconnPool
has code to deal with multiple addresses while IdleConnList does not. I do not
think this difference is really used in this implementation, but I did not
want to face an unknown limitation. Note that ICAP does not support multiple
ICAP server addresses.

* PconnPool has reporting (and cache manager integration) code that we should
eventually improve and report standby-specific stats. When this happens,
PconnPool will probably become abstract and spawn two kids, one for pconn and
one for standby pools.

Seemingly unrelated changes triggered by standby=N addition:

* Removed PconnPool from fde.h. We used to create immortal PconnPool objects.
Now, standby pools are destroyed when their peer is destroyed. Sharing raw
pointers to such pools is too dangerous. We could use smart pointers, but
PconnPools do not really belong to such a low-level object like fde IMO.

* Added FwdState::closeServerConnection() to encapsulate server connection
closing code, including the new noteUses() maintenance. Also updated
FwdState::serverClosed() to do the same maintenance.

* Close all connections in IdleConnList upon deletion. The old code did
not care because we never deleted PconnPools (although I am not sure
there were no bugs related to ICAP service pools which use IdleConnList
directly and do get destroyed).

* Fixed PconnPool::dumpHash(). It was listing the first entry twice because
the code

Build failed in Jenkins: 3.HEAD-amd64-OpenBSD-5.4 #61

2014-04-30 Thread noc

See http://build.squid-cache.org/job/3.HEAD-amd64-OpenBSD-5.4/61/

--
Started by upstream project 3.HEAD-amd64-centos-6 build number 320
originally caused by:
 Started by an SCM change
Building remotely on ypg-openbsd-54 (gcc farm amd64-openbsd 5.4 openbsd-5.4 
openbsd amd64-openbsd-5.4 amd64) in workspace 
http://build.squid-cache.org/job/3.HEAD-amd64-OpenBSD-5.4/ws/
Cleaning workspace...
$ bzr branch http://bzr.squid-cache.org/bzr/squid3/trunk/ 
http://build.squid-cache.org/job/3.HEAD-amd64-OpenBSD-5.4/ws/
bzr: ERROR: Connection error: Couldn't resolve host 'bzr.squid-cache.org' 
[Errno -5] no address associated with name
ERROR: Failed to branch http://bzr.squid-cache.org/bzr/squid3/trunk/
Retrying after 10 seconds
Cleaning workspace...
$ bzr branch http://bzr.squid-cache.org/bzr/squid3/trunk/ 
http://build.squid-cache.org/job/3.HEAD-amd64-OpenBSD-5.4/ws/
bzr: ERROR: Connection error: Couldn't resolve host 'bzr.squid-cache.org' 
[Errno -5] no address associated with name
ERROR: Failed to branch http://bzr.squid-cache.org/bzr/squid3/trunk/
Retrying after 10 seconds
Cleaning workspace...
$ bzr branch http://bzr.squid-cache.org/bzr/squid3/trunk/ 
http://build.squid-cache.org/job/3.HEAD-amd64-OpenBSD-5.4/ws/
bzr: ERROR: Connection error: Couldn't resolve host 'bzr.squid-cache.org' 
[Errno -5] no address associated with name
ERROR: Failed to branch http://bzr.squid-cache.org/bzr/squid3/trunk/

Build failed in Jenkins: 3.HEAD-amd64-centos-6-clang #203

2014-04-30 Thread noc

See http://build.squid-cache.org/job/3.HEAD-amd64-centos-6-clang/203/changes

Changes:

[Christos Tsantilas] author: Alex Rousskov rouss...@measurement-factory.com
cache_peer standby=N implementation.

The feature focus is to instantly provide a ready-to-use connection to a
cooperating cache peer, virtually at all times. This is useful when connection
establishment is too slow and/or when infrequent peer use prevents Squid from
combating slow connection establishment with the regular idle connection pool.

The feature is similar to Squid2 idle=N feature, but there are key differences:

* Standby connections are available virtually at all times, while Squid2 unused
idle connections are available only for a short time after a peer request.

* All N standby connections are not opened at once, reducing the chance of
the feature being mistaken for a DoS attack on a peer.

* More consistent support for peers with multiple IP addresses (peer IPs are
cycled through, just like during regular Squid request forwarding).

Besides, idle is a poor choice of adjective for an unused connection pool
name because the same term is used for used persistent connections, which have
somewhat different properties, are stored in a different pool, may need
distinct set of tuning options, etc. It is better to use a dedicated term for
the new feature.

The relationship between the max-conn limit and standby/idle connections is a
complex one. After several rewrites and tests, Squid now obeys max-conn limit
when opening new standby connections and accounts for standby connections when
checking whether to allow peer use. This often works OK, but leads to standby
guarantee violations when non-standby connections approach the limit. The
alternative design where standby code ignores max-conn works better, but is
really difficult to explain and advocate because an admin expects max-conn to
cover all connections and because of the idle connections accounting and
maintenance bugs. We may come back to this when the idle connections code is
fixed.

Fixed max-conn documentation and XXXed a peerHTTPOkay() bug (now in
peerHasConnAvailable()) that results in max-conn limit preventing the use of a
peer with idle persistent connections.

Decided to use standby connections for non-retriable requests. Avoiding
standby connections for POSTs and such would violate the main purpose of the
feature: providing an instant ready-to-use connection. A user does not care
whether it is waiting too long for a GET or POST request. Actually, a user may
care more when their POST requests are delayed (because canceling and
retrying them is often scary from the user point of view). The idea behind
standby connections is that the admin is responsible for avoiding race
conditions by properly configuring the peering Squids. If such proper
configuration is not possible or the consequences of rare races (e.g., due to
peer shutdown) are more severe than the consequences of slow requests, the
admin should not use standby=N. This choice may become configurable in the
future.

TODO: Teach peer probing code to push successful probing connections into the
standby pool (when enabled). Should be done as a followup project because of
the differences in standby and probe connection opening code, especially when
SSL peers are supported. Will require some discussion.

A standby pool is using a full-blown PconnPool object for storage instead of
the smaller IdleConnList, like the ICAP code does. The primary reasons for
this design were:

* A peer may have multiple addresses and those addresses may change. PconnPool
has code to deal with multiple addresses while IdleConnList does not. I do not
think this difference is really used in this implementation, but I did not
want to face an unknown limitation. Note that ICAP does not support multiple
ICAP server addresses.

* PconnPool has reporting (and cache manager integration) code that we should
eventually improve and report standby-specific stats. When this happens,
PconnPool will probably become abstract and spawn two kids, one for pconn and
one for standby pools.

Seemingly unrelated changes triggered by standby=N addition:

* Removed PconnPool from fde.h. We used to create immortal PconnPool objects.
Now, standby pools are destroyed when their peer is destroyed. Sharing raw
pointers to such pools is too dangerous. We could use smart pointers, but
PconnPools do not really belong to such a low-level object like fde IMO.

* Added FwdState::closeServerConnection() to encapsulate server connection
closing code, including the new noteUses() maintenance. Also updated
FwdState::serverClosed() to do the same maintenance.

* Close all connections in IdleConnList upon deletion. The old code did
not care because we never deleted PconnPools (although I am not sure
there were no bugs related to ICAP service pools which use IdleConnList
directly and do get destroyed).

* Fixed PconnPool::dumpHash(). It was listing the first entry twice because
the code

Build failed in Jenkins: 3.HEAD-amd64-FreeBSD-10 #59

2014-04-30 Thread noc

See http://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10/59/changes

Changes:

[Christos Tsantilas] Ssl::PeerConnector class part 2

Move operator (std::ostream , const Ssl::PeerConnectorAnswer) under the
Ssl namespace to make clang compiler happy.

--
[...truncated 3562 lines...]
/usr/local/bin/ccache g++ -DHAVE_CONFIG_H  -I../../../.. -I../../../../include 
-I../../../../lib  -I../../../../src -I../../../include  -I/usr/local/include 
-I/usr/include  -I/usr/include -I../../../../libltdl 
-I../../../../helpers/basic_auth/NCSA   -I/usr/include  -I/usr/include -Wall 
-Wpointer-arith -Wwrite-strings -Wcomments -Wshadow -Werror -pipe -D_REENTRANT 
-g -O2 -I/usr/local/include -MT crypt_md5.o -MD -MP -MF .deps/crypt_md5.Tpo -c 
-o crypt_md5.o ../../../../helpers/basic_auth/NCSA/crypt_md5.cc
mv -f .deps/crypt_md5.Tpo .deps/crypt_md5.Po
--- basic_ncsa_auth.o ---
/usr/local/bin/ccache g++ -DHAVE_CONFIG_H  -I../../../.. -I../../../../include 
-I../../../../lib  -I../../../../src -I../../../include  -I/usr/local/include 
-I/usr/include  -I/usr/include -I../../../../libltdl 
-I../../../../helpers/basic_auth/NCSA   -I/usr/include  -I/usr/include -Wall 
-Wpointer-arith -Wwrite-strings -Wcomments -Wshadow -Werror -pipe -D_REENTRANT 
-g -O2 -I/usr/local/include -MT basic_ncsa_auth.o -MD -MP -MF 
.deps/basic_ncsa_auth.Tpo -c -o basic_ncsa_auth.o 
../../../../helpers/basic_auth/NCSA/basic_ncsa_auth.cc
mv -f .deps/basic_ncsa_auth.Tpo .deps/basic_ncsa_auth.Po
--- basic_ncsa_auth ---
/bin/sh ../../../libtool  --tag=CXX--mode=link /usr/local/bin/ccache g++ 
-Wall -Wpointer-arith -Wwrite-strings -Wcomments -Wshadow -Werror -pipe 
-D_REENTRANT  -g -O2 -I/usr/local/include  -g -L/usr/local/lib 
-Wl,-R/usr/local/lib -pthread -o basic_ncsa_auth basic_ncsa_auth.o  crypt_md5.o 
../../../lib/libmisccontainers.la  ../../../lib/libmiscencoding.la  
../../../compat/libcompat-squid.la   -lnettle  -lcrypt-lm 
libtool: link: /usr/local/bin/ccache g++ -Wall -Wpointer-arith -Wwrite-strings 
-Wcomments -Wshadow -Werror -pipe -D_REENTRANT -g -O2 -I/usr/local/include -g 
-Wl,-R/usr/local/lib -pthread -o basic_ncsa_auth basic_ncsa_auth.o crypt_md5.o  
-L/usr/local/lib ../../../lib/.libs/libmisccontainers.a 
../../../lib/.libs/libmiscencoding.a ../../../compat/.libs/libcompat-squid.a 
-lnettle -lcrypt -lm -pthread
Making all in PAM
--- basic_pam_auth.o ---
/usr/local/bin/ccache g++ -DHAVE_CONFIG_H  -I../../../.. -I../../../../include 
-I../../../../lib  -I../../../../src -I../../../include  -I/usr/local/include 
-I/usr/include  -I/usr/include -I../../../../libltdl   -I/usr/include  
-I/usr/include -Wall -Wpointer-arith -Wwrite-strings -Wcomments -Wshadow 
-Werror -pipe -D_REENTRANT -g -O2 -I/usr/local/include -MT basic_pam_auth.o -MD 
-MP -MF .deps/basic_pam_auth.Tpo -c -o basic_pam_auth.o 
../../../../helpers/basic_auth/PAM/basic_pam_auth.cc
mv -f .deps/basic_pam_auth.Tpo .deps/basic_pam_auth.Po
--- basic_pam_auth ---
/bin/sh ../../../libtool  --tag=CXX--mode=link /usr/local/bin/ccache g++ 
-Wall -Wpointer-arith -Wwrite-strings -Wcomments -Wshadow -Werror -pipe 
-D_REENTRANT  -g -O2 -I/usr/local/include  -g -L/usr/local/lib 
-Wl,-R/usr/local/lib -pthread -o basic_pam_auth basic_pam_auth.o 
../../../lib/libmiscencoding.la  ../../../compat/libcompat-squid.la   -lpam  
-lm 
libtool: link: /usr/local/bin/ccache g++ -Wall -Wpointer-arith -Wwrite-strings 
-Wcomments -Wshadow -Werror -pipe -D_REENTRANT -g -O2 -I/usr/local/include -g 
-Wl,-R/usr/local/lib -pthread -o basic_pam_auth basic_pam_auth.o  
-L/usr/local/lib ../../../lib/.libs/libmiscencoding.a 
../../../compat/.libs/libcompat-squid.a -lpam -lm -pthread
Making all in POP3
--- basic_pop3_auth ---
sed -e 's,[@]PERL[@],/usr/bin/perl,g' 
../../../../helpers/basic_auth/POP3/basic_pop3_auth.pl.in basic_pop3_auth || 
(/bin/rm -f -f basic_pop3_auth ; exit 1)
Making all in RADIUS
--- basic_radius_auth.o ---
--- radius-util.o ---
--- basic_radius_auth.o ---
/usr/local/bin/ccache g++ -DHAVE_CONFIG_H  -I../../../.. -I../../../../include 
-I../../../../lib  -I../../../../src -I../../../include  -I/usr/local/include 
-I/usr/include  -I/usr/include -I../../../../libltdl 
-I../../../../helpers/basic_auth/RADIUS   -I/usr/include  -I/usr/include -Wall 
-Wpointer-arith -Wwrite-strings -Wcomments -Wshadow -Werror -pipe -D_REENTRANT 
-g -O2 -I/usr/local/include -MT basic_radius_auth.o -MD -MP -MF 
.deps/basic_radius_auth.Tpo -c -o basic_radius_auth.o 
../../../../helpers/basic_auth/RADIUS/basic_radius_auth.cc
--- radius-util.o ---
/usr/local/bin/ccache g++ -DHAVE_CONFIG_H  -I../../../.. -I../../../../include 
-I../../../../lib  -I../../../../src -I../../../include  -I/usr/local/include 
-I/usr/include  -I/usr/include -I../../../../libltdl 
-I../../../../helpers/basic_auth/RADIUS   -I/usr/include  -I/usr/include -Wall 
-Wpointer-arith -Wwrite-strings -Wcomments -Wshadow -Werror -pipe -D_REENTRANT 
-g -O2 -I/usr/local/include -MT radius-util.o -MD -MP -MF

Build failed in Jenkins: 3.HEAD-amd64-FreeBSD-10-clang #59

2014-04-30 Thread noc

See http://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/59/changes

Changes:

[Christos Tsantilas] Ssl::PeerConnector class part 2

Move operator (std::ostream , const Ssl::PeerConnectorAnswer) under the
Ssl namespace to make clang compiler happy.

--
[...truncated 5987 lines...]
--- url.o ---
depbase=`echo url.o | sed 's|[^/]*$|.deps/|;s|\.o$||'`; ccache clang++ 
-DHAVE_CONFIG_H 
-DDEFAULT_CONFIG_FILE=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/etc/squid.conf\;
  
-DDEFAULT_SQUID_DATA_DIR=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/share\;
  
-DDEFAULT_SQUID_CONFIG_DIR=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/etc\;
  -I../.. -I../../include -I../../lib  -I../../src -I../include  
-I/usr/local/include -I/usr/include  -I/usr/include -I../../libltdl  -I../src 
-I../../libltdl -I/usr/include  -I/usr/include   -I/usr/include  -I/usr/include 
-Werror -Qunused-arguments  -D_REENTRANT -g -O2 -I/usr/local/include -MT url.o 
-MD -MP -MF $depbase.Tpo -c -o url.o ../../src/url.cc  mv -f $depbase.Tpo 
$depbase.Po
--- urn.o ---
depbase=`echo urn.o | sed 's|[^/]*$|.deps/|;s|\.o$||'`; ccache clang++ 
-DHAVE_CONFIG_H 
-DDEFAULT_CONFIG_FILE=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/etc/squid.conf\;
  
-DDEFAULT_SQUID_DATA_DIR=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/share\;
  
-DDEFAULT_SQUID_CONFIG_DIR=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/etc\;
  -I../.. -I../../include -I../../lib  -I../../src -I../include  
-I/usr/local/include -I/usr/include  -I/usr/include -I../../libltdl  -I../src 
-I../../libltdl -I/usr/include  -I/usr/include   -I/usr/include  -I/usr/include 
-Werror -Qunused-arguments  -D_REENTRANT -g -O2 -I/usr/local/include -MT urn.o 
-MD -MP -MF $depbase.Tpo -c -o urn.o ../../src/urn.cc  mv -f $depbase.Tpo 
$depbase.Po
--- wccp.o ---
depbase=`echo wccp.o | sed 's|[^/]*$|.deps/|;s|\.o$||'`; ccache clang++ 
-DHAVE_CONFIG_H 
-DDEFAULT_CONFIG_FILE=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/etc/squid.conf\;
  
-DDEFAULT_SQUID_DATA_DIR=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/share\;
  
-DDEFAULT_SQUID_CONFIG_DIR=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/etc\;
  -I../.. -I../../include -I../../lib  -I../../src -I../include  
-I/usr/local/include -I/usr/include  -I/usr/include -I../../libltdl  -I../src 
-I../../libltdl -I/usr/include  -I/usr/include   -I/usr/include  -I/usr/include 
-Werror -Qunused-arguments  -D_REENTRANT -g -O2 -I/usr/local/include -MT wccp.o 
-MD -MP -MF $depbase.Tpo -c -o wccp.o ../../src/wccp.cc  mv -f $depbase.Tpo 
$depbase.Po
--- wccp2.o ---
depbase=`echo wccp2.o | sed 's|[^/]*$|.deps/|;s|\.o$||'`; ccache clang++ 
-DHAVE_CONFIG_H 
-DDEFAULT_CONFIG_FILE=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/etc/squid.conf\;
  
-DDEFAULT_SQUID_DATA_DIR=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/share\;
  
-DDEFAULT_SQUID_CONFIG_DIR=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/etc\;
  -I../.. -I../../include -I../../lib  -I../../src -I../include  
-I/usr/local/include -I/usr/include  -I/usr/include -I../../libltdl  -I../src 
-I../../libltdl -I/usr/include  -I/usr/include   -I/usr/include  -I/usr/include 
-Werror -Qunused-arguments  -D_REENTRANT -g -O2 -I/usr/local/include -MT 
wccp2.o -MD -MP -MF $depbase.Tpo -c -o wccp2.o ../../src/wccp2.cc  mv -f 
$depbase.Tpo $depbase.Po
--- whois.o ---
depbase=`echo whois.o | sed 's|[^/]*$|.deps/|;s|\.o$||'`; ccache clang++ 
-DHAVE_CONFIG_H 
-DDEFAULT_CONFIG_FILE=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/etc/squid.conf\;
  
-DDEFAULT_SQUID_DATA_DIR=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/share\;
  
-DDEFAULT_SQUID_CONFIG_DIR=\/usrhttp://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-10-clang/ws/btlayer-00-default/squid-3.HEAD-BZR/_inst/etc\;
  -I../.. -I../../include -I../../lib  -I../../src -I../include  
-I/usr/local/include -I/usr/include  -I/usr/include -I../../libltdl  -I../src 
-I../../libltdl -I/usr/include  -I/usr/include   -I/usr/include  -I/usr/include 
-Werror -Qunused-arguments  -D_REENTRANT -g -O2 -I/usr/local/include -MT 
whois.o

Build failed in Jenkins: 3.HEAD-amd64-OpenBSD-5.4 #62

2014-04-30 Thread noc

See http://build.squid-cache.org/job/3.HEAD-amd64-OpenBSD-5.4/62/

--
Started by upstream project 3.HEAD-amd64-centos-6 build number 321
originally caused by:
 Started by an SCM change
Building remotely on ypg-openbsd-54 (gcc farm amd64-openbsd 5.4 openbsd-5.4 
openbsd amd64-openbsd-5.4 amd64) in workspace 
http://build.squid-cache.org/job/3.HEAD-amd64-OpenBSD-5.4/ws/
Cleaning workspace...
$ bzr branch http://bzr.squid-cache.org/bzr/squid3/trunk/ 
http://build.squid-cache.org/job/3.HEAD-amd64-OpenBSD-5.4/ws/
bzr: ERROR: Connection error: Couldn't resolve host 'bzr.squid-cache.org' 
[Errno -5] no address associated with name
ERROR: Failed to branch http://bzr.squid-cache.org/bzr/squid3/trunk/
Retrying after 10 seconds
Cleaning workspace...
$ bzr branch http://bzr.squid-cache.org/bzr/squid3/trunk/ 
http://build.squid-cache.org/job/3.HEAD-amd64-OpenBSD-5.4/ws/
bzr: ERROR: Connection error: Couldn't resolve host 'bzr.squid-cache.org' 
[Errno -5] no address associated with name
ERROR: Failed to branch http://bzr.squid-cache.org/bzr/squid3/trunk/
Retrying after 10 seconds
Cleaning workspace...
$ bzr branch http://bzr.squid-cache.org/bzr/squid3/trunk/ 
http://build.squid-cache.org/job/3.HEAD-amd64-OpenBSD-5.4/ws/
bzr: ERROR: Connection error: Couldn't resolve host 'bzr.squid-cache.org' 
[Errno -5] no address associated with name
ERROR: Failed to branch http://bzr.squid-cache.org/bzr/squid3/trunk/

Jenkins build is back to normal : 3.HEAD-amd64-centos-6-clang #204

2014-04-30 Thread noc

See http://build.squid-cache.org/job/3.HEAD-amd64-centos-6-clang/204/changes

Re: How long is a domain or url can be?

2014-04-30 Thread Eliezer Croitoru


On 04/30/2014 11:52 AM, Henrik Nordström wrote:

Unless it has been fixed the UFS based stores also have an implicit
limit on cached entries somewhat less than 4KB (whole meta header need
to fit in first 4KB). Entries failing this gets cached but can never get
hit.

Then StoreID helps a bit with that..
Now it's understood why some urls with the ? in them do not cache well 
sometimes :P



DNS defines X.Y.Z segments as being no longer than 255 bytes*each*.

For Internet host names the limits are 63 octets per label, and 255
octects in total including dot delimiters.


This is indeed what I have been reading in the RFC and it makes the 
regex for domain simpler to define.
From what I have seen 2-3KB of request size was the high limit of the 
size that have been used.
I assume that this is what is happening now in the current data sizes 
over the network.
Every once in a while the data size goes up and the url should also 
since they will be used by bigger sizes hash algorithms.

It was started in smaller and then crc16 crc32 mdX md5 sha1 sha512...etc..

So for now a url blacklist should be at-least 4KB with size but I think 
when jumping\doubling 4KB it's not such a big jump to 8KB.
The main issue I was thinking was between using one field of the DB with 
X size or other one which has indexes.


For now I have used mysql TEXT which doesn't have indexes but only the 
first query takes more then 0.00 ms.


I have tried couple key-value DB's and other DB's but it seems like all 
of them are having some kind of a step which is the slowest and then it 
run's fast.


I have mysql Compared to key-vaule and the main differences are the 
on-disk size of the DB which is important if there is a plan to filter 
many many specific urls and not based only on patterns.


Amos:(or anyone else) since you patched squidguard, maybe you do have 
basic understanding with it's lookup algorithm?
I started reading the code of SquidGuard but then all of a sudden I lost 
my way in it and things got a bit complicated (for me) to understand how 
they do it.(hints..)


Thanks,
Eliezer


Regards
Henrik

Re: /bzr/squid3/trunk/ r13384: Bug 1961: pt1: URL handling redesign

2014-04-30 Thread Alex Rousskov

On 28/04/2014 5:35 a.m., Tsantilas Christos wrote:
 Unfortunately this is not build with ecap.


On 04/27/2014 07:57 PM, Amos Jeffries wrote:
 What the eCAP field needs to be set to depends on its definition:


libecap::FirstLine::protocol() is meant for things like

* the HTTP-Version part of HTTP messages (in RFC 2616 terminology) or
* the ICAP-Version part of ICAP messages (in RFC 3507 terminology).

It is not related to the URI. In fact, libecap::FirstLine does not know
about the message URI! Only its libecap::RequestLine child knows that
requests have URIs.

HttpMsg::http_ver in recent trunk is described as being meant for the
same thing. Thus, there is a perfect match between the two concepts.


Now, we need to understand how the actual code maps to the above design,
and what code changes are needed to fix the trunk build broken by
r13384. My understanding is that

1. If Squid trunk does not set HttpMsg::http_ver [correctly] when
parsing any requests or responses, we should fix the corresponding Squid
code. This will ensure that eCAP adapters are getting the right
information about virgin messages. This item requires an investigation
and may not require any code changes.

2. If Squid trunk does not use HttpMsg::http_ver [correctly] when
formatting any requests or responses, we should fix the corresponding
Squid code. This will ensure that eCAP adapters can adapt virgin
messages as needed. Note that Squid does not support post-cache
vectoring points so Squid should still set its own http_ver in any
outgoing messages. This item requires an investigation and may not
require any code changes.

3. Squid eCAP implementation should be changed to read and write
HttpMsg::http_ver. This affects two
Adaptation::Ecap::FirstLineRep::protocol methods and possibly other
code. This item requires code changes.

There is no need to change eCAP APIs. Some existing adapters might be
inconvenienced by this Squid change, but those adapters are probably
using the wrong interface anyway, and no such cases are known to exist.

If the above is accurate, Amos, would you like to work on any of the
above three items?


Thank you,

Alex.

Re: How long is a domain or url can be?

2014-04-30 Thread Amos Jeffries

On 1/05/2014 6:12 a.m., Eliezer Croitoru wrote:
 On 04/30/2014 11:52 AM, Henrik Nordström wrote:
 Unless it has been fixed the UFS based stores also have an implicit
 limit on cached entries somewhat less than 4KB (whole meta header need
 to fit in first 4KB). Entries failing this gets cached but can never get
 hit.
 Then StoreID helps a bit with that..
 Now it's understood why some urls with the ? in them do not cache well
 sometimes :P
 
 DNS defines X.Y.Z segments as being no longer than 255 bytes*each*.
 For Internet host names the limits are 63 octets per label, and 255
 octects in total including dot delimiters.
 
 This is indeed what I have been reading in the RFC and it makes the
 regex for domain simpler to define.
 From what I have seen 2-3KB of request size was the high limit of the
 size that have been used.
 I assume that this is what is happening now in the current data sizes
 over the network.
 Every once in a while the data size goes up and the url should also
 since they will be used by bigger sizes hash algorithms.
 It was started in smaller and then crc16 crc32 mdX md5 sha1 sha512...etc..
 
 So for now a url blacklist should be at-least 4KB with size but I think
 when jumping\doubling 4KB it's not such a big jump to 8KB.
 The main issue I was thinking was between using one field of the DB with
 X size or other one which has indexes.
 
 For now I have used mysql TEXT which doesn't have indexes but only the
 first query takes more then 0.00 ms.
 
 I have tried couple key-value DB's and other DB's but it seems like all
 of them are having some kind of a step which is the slowest and then it
 run's fast.

At a guess that would probably loading the table data into memory or
constructing some form of cache for the results. If you imagine it the
same way Squid operates: the first request is a MISS and has to actually
get to the origin and do all its processing, second and later requests
can be fast HITs.

 
 I have mysql Compared to key-vaule and the main differences are the
 on-disk size of the DB which is important if there is a plan to filter
 many many specific urls and not based only on patterns.
 
 Amos:(or anyone else) since you patched squidguard, maybe you do have
 basic understanding with it's lookup algorithm?
 I started reading the code of SquidGuard but then all of a sudden I lost
 my way in it and things got a bit complicated (for me) to understand how
 they do it.(hints..)

I only looked at it hard enough to find the reply line syntax and make
it return the URL alone with tah patching. Will give it a look over and
see what I can find later today.

BTW: If you go for optimizing MySQL database be wary of SOUNDEX(). It is
great for textual comparisons and indexing, but only if you are storing
American English words. For any other input it is a brilliant way to
screw up without noticing.

Amos

Re: /bzr/squid3/trunk/ r13384: Bug 1961: pt1: URL handling redesign

2014-04-30 Thread Amos Jeffries

On 1/05/2014 6:33 a.m., Alex Rousskov wrote:
 On 28/04/2014 5:35 a.m., Tsantilas Christos wrote:
 Unfortunately this is not build with ecap.
 
 
 On 04/27/2014 07:57 PM, Amos Jeffries wrote:
 What the eCAP field needs to be set to depends on its definition:
 
 
 libecap::FirstLine::protocol() is meant for things like
 
 * the HTTP-Version part of HTTP messages (in RFC 2616 terminology) or
 * the ICAP-Version part of ICAP messages (in RFC 3507 terminology).
 
 It is not related to the URI. In fact, libecap::FirstLine does not know
 about the message URI! Only its libecap::RequestLine child knows that
 requests have URIs.
 
 HttpMsg::http_ver in recent trunk is described as being meant for the
 same thing. Thus, there is a perfect match between the two concepts.
 
 
 Now, we need to understand how the actual code maps to the above design,
 and what code changes are needed to fix the trunk build broken by
 r13384. My understanding is that
 
 1. If Squid trunk does not set HttpMsg::http_ver [correctly] when
 parsing any requests or responses, we should fix the corresponding Squid
 code. This will ensure that eCAP adapters are getting the right
 information about virgin messages. This item requires an investigation
 and may not require any code changes.
 
 2. If Squid trunk does not use HttpMsg::http_ver [correctly] when
 formatting any requests or responses, we should fix the corresponding
 Squid code. This will ensure that eCAP adapters can adapt virgin
 messages as needed. Note that Squid does not support post-cache
 vectoring points so Squid should still set its own http_ver in any
 outgoing messages. This item requires an investigation and may not
 require any code changes.
 
 3. Squid eCAP implementation should be changed to read and write
 HttpMsg::http_ver. This affects two
 Adaptation::Ecap::FirstLineRep::protocol methods and possibly other
 code. This item requires code changes.
 
 There is no need to change eCAP APIs. Some existing adapters might be
 inconvenienced by this Squid change, but those adapters are probably
 using the wrong interface anyway, and no such cases are known to exist.
 
 If the above is accurate, Amos, would you like to work on any of the
 above three items?


There will also need to be review of internally generated requests,
including the URL rewriter generated clone request. They use Squid
default version on output but what the state of it is up to then I don't
know.


I have kind of being doing #1 already in my parser-ng work and happy to
take that on and then #2 unless someone else gets it done first.

The state I'm aware of right now:

* HttpRequest and HttpReply default constructors set it to HTTP/0.0.

* client request parser is setting it for all valid requests. Though
parser bugs set it to 0.9 on invalid 1.x requests sometimes.

* server reply parsing uncertain.

* some questionable code actions in the ICAP output parsing on first lines.

Amos

Re: How long is a domain or url can be?

Re: [PATCH] logformat annotation fixes

Build failed in Jenkins: 3.HEAD-amd64-FreeBSD-10 #57

Build failed in Jenkins: 3.HEAD-amd64-FreeBSD-10-clang #57

Build failed in Jenkins: 3.HEAD-amd64-OpenBSD-5.4 #60

Build failed in Jenkins: 3.HEAD-amd64-FreeBSD-10 #58

Build failed in Jenkins: 3.HEAD-amd64-OpenBSD-5.4 #61

Build failed in Jenkins: 3.HEAD-amd64-centos-6-clang #203

Build failed in Jenkins: 3.HEAD-amd64-FreeBSD-10 #59

Build failed in Jenkins: 3.HEAD-amd64-FreeBSD-10-clang #59

Build failed in Jenkins: 3.HEAD-amd64-OpenBSD-5.4 #62

Jenkins build is back to normal : 3.HEAD-amd64-centos-6-clang #204

Re: How long is a domain or url can be?

Re: /bzr/squid3/trunk/ r13384: Bug 1961: pt1: URL handling redesign

Re: How long is a domain or url can be?

Re: /bzr/squid3/trunk/ r13384: Bug 1961: pt1: URL handling redesign

16 matches

Site Navigation

Mail list logo

Footer information