Re: squid-smp

[Sachin Malave]

On Tue, Oct 27, 2009 at 8:09 AM, Amos Jeffries  wrote:
> Alex Rousskov wrote:
>>
>> Hello,
>>
>>    I will be working on SMP support in the coming months. I caught up
>> on this and the previous SMP-related squid-dev thread and it looks like
>>  the approach I currently favor has been discussed (again) and did not
>> cause any violent objections, although some ideas were much more
>> ambitious. I am not sure how we can reach consensus on this topic, but I
>> will through in some specifics in hope to better identify competing
>> approaches.
>
> Okay. Before you get buried in this work might I request that you look at
> the stale-while-revalidate / stale-if-error code that was nearly finished?
> http://bugs.squid-cache.org/show_bug.cgi?id=2255
>
> Yahoo! named those as two of their requirements before it would be possible
> for them to assist with performance testing Squid-3. They might not be
> adverse to helping test SMP support if the other requirements are available.
>
> Collapsed forwarding is also a requirement, but I suspect it is too close to
> the request handling and needs a re-designed code architecture to fit with
> whatever SMP threading model is taken anyways.
>
>>
>> My short-term focus would be on the following three areas:
>>
>> A) Identifying a few "large", "rarely-interacting" threads that would
>> work reasonably well on an 8-core 2-CPU machine with 8 http_ports. This
>> should take the lessons learned from existing SMP designs into account,
>> with Squid specifics in mind. Henrik, Amos, and Adrian started
>> discussing this already.
>>
>> B) Making commonly used primitives thread-safe (mostly not in terms of
>> locking their shared state but in terms of not using static/shared data
>> that needs locking). Many posts on this subject, starting with Roberts
>> advice to desynchronize.
>>
>> C) Posting performance benchmarking results for single- and
>> multi-instance Squids on mutli-core systems as a baseline.
>>
>>
>> My mid-term focus will probably be on sharing http_port, memory cache,
>> disk cache and possibly loggin/stats among a "few large threads".
>>
>>
>> My overall goal is to at least approach the performance of a
>> multi-instance caching Squid on 8-core hardware.
>>
>>
>> I am not excited by the "one thread per message", "one thread per
>> AsyncJob", or similar "many tiny threads" designs because, IMO, they
>> would require too much rewriting to be implemented properly. This may
>> need to be re-evaluated as the world moves towards 1000-core systems,
>> but a lot of improvements necessary for the "few large threads" design
>> will not be wasted anyway.
>>
>> I hope that by focusing on a "few large threads" design and fixing
>> primitives we can gain "enough" SMP benefits in a few months of active
>> development. If you think there is a better way to get SMP benefits in
>> the foreseeable future, please post.
>>
>> Thank you,
>>
>> Alex.
>
>
> --
> Please be using
>  Current Stable Squid 2.7.STABLE7 or 3.0.STABLE19
>  Current Beta Squid 3.1.0.14
>

Thats very good Alex, I am also working on the same, have not explored
everything yet, But with all you guys hopefully we will complete this
project as soon as possible, Issues in my mind will be discussed soon.



-- 
Mr. S. H. Malave
Computer Science & Engineering Department,
Walchand College of Engineering,Sangli.
sachinmal...@wce.org.in

Re: test-suite and OS specific options

[Amos Jeffries]

On Wed, 28 Oct 2009 02:31:48 +0100, Henrik Nordstrom
 wrote:
> ons 2009-10-28 klockan 14:24 +1300 skrev Amos Jeffries:
> 
>> In the particular linux-tproxy2 case it is one of the many options that
>> auto-detect and quietly warn before disabling the feature and
continuing
>> with the build. _even if configured explicitly_ (yeah nasty).
> 
> Ugh.. and quite obviously the disable does not work properly..
> 
> I'll give it a stab tomorrow.
> 
>> PS. There is a planned level of 'maybe' which I have not yet got around
>> to
>> adding. It will allow testing of hard-fail options in future and accept
>> certain explicit fail messages (ie "wrong OS type" or "dependencies
>> missing") as an okay pass result.
> 
> What about making TPROXY, netfilter etc default enabled if found
> available?

+1. I think the new NatLookup multi-plexing code is ready for mainstream,
but its not exactly widely tested yet.

> 
> Makes the minimum test have lots of --disable-xxx and --without-xxx, and
> the maximus test is basically no configure options at all I guess except
> for experimental features.

Yes.

If we can keep up the nesting of dependencies under feature enable
wrappers the --without can be omitted in the presence of a --disable. Which
keeps it slightly simpler.

> 
> and to guarantee proper coverage over time, maybe have some nodes
> running tests with specific --enable-xxx and --with-xxx options, making
> the build fail on those hosts if the requirements have gone missing for
> some reason..

The "maybe" and "nodeps" layers should take care of that part when done.

Amos

Re: /bzr/squid3/trunk/ r10049: Use libcap functions instead of raw kernel interface

[Amos Jeffries]

On Wed, 28 Oct 2009 02:04:04 +0100, Henrik Nordstrom
 wrote:
> ons 2009-10-28 klockan 12:04 +1300 skrev Amos Jeffries:
> 
>>  * That should have been --with-cap2 right?  same as all the others
>>  ('lib'
>> being redundant and libcap2 being the official name for it as differs
>> from
>> libcap and libcap-ng).
> 
> I disagree there.
> 
> The official name of libcap-x.y is libcap, not cap or cap2. Source level
> API of libcap-1.x and libcap-2.x is the same, except that 2.x provides
> more functions and is not binary compatible with 1.x applications. It's
> a linear line of development from the same author.
> 
> "all the others" is a very broad statement. We only have two other
> --with- and it's --with-dl and --with-openssl.
> 
> Official name of openssl is openssl, not libssl as Debian likes to call
> it..
> 
> and dl do not really have a name of it's own, and is only known as -ldl
> to most. And can probably go away now that we have libtool adding
> similar tests..

Hmm. Okay.

Checking... fubar.  Not sure where I got that idea.
I've fully buggered the docs on libexpat and libxml2. Fixing.


Amos

Re: test-suite and OS specific options

[Henrik Nordstrom]

ons 2009-10-28 klockan 14:24 +1300 skrev Amos Jeffries:

> In the particular linux-tproxy2 case it is one of the many options that
> auto-detect and quietly warn before disabling the feature and continuing
> with the build. _even if configured explicitly_ (yeah nasty).

Ugh.. and quite obviously the disable does not work properly..

I'll give it a stab tomorrow.

> PS. There is a planned level of 'maybe' which I have not yet got around to
> adding. It will allow testing of hard-fail options in future and accept
> certain explicit fail messages (ie "wrong OS type" or "dependencies
> missing") as an okay pass result.

What about making TPROXY, netfilter etc default enabled if found
available?

Makes the minimum test have lots of --disable-xxx and --without-xxx, and
the maximus test is basically no configure options at all I guess except
for experimental features.

and to guarantee proper coverage over time, maybe have some nodes
running tests with specific --enable-xxx and --with-xxx options, making
the build fail on those hosts if the requirements have gone missing for
some reason..

Regards
Henrik

Re: test-suite and OS specific options

[Amos Jeffries]

On Wed, 28 Oct 2009 00:14:19 +0100, Henrik Nordstrom
 wrote:
> After the libcap change the testsuide now fails on CentOS, this time due
> to layer-02-maximus enablind linux-tproxy but as libcap isn't available
> it's not really enabled but still compiled. A little confusing, as far
> as I know I did not change the logics on when to enable what of Linux
> tproxy..
> 
> So to the question, is it really sane that the test suite blindly
> enables OS specific options? I mean I would not expect the build to work
> if one uses --enable-linux-netfilter on an BSD box for example, or
> --enable-linux-tproxy2 on a box where Linux TPROXY2 is not available or
> alternatively with now required libcap missing..
> 
> Regards
> Henrik

The options were and some continue to be a bit of a mess. Under maximus
the test bed enables all the ones known to not hard-fail for one reason or
another on any OS regardless of their name.

In the particular linux-tproxy2 case it is one of the many options that
auto-detect and quietly warn before disabling the feature and continuing
with the build. _even if configured explicitly_ (yeah nasty).

The TestBed is currently testing that either this quiet-fail behavior
works and prevents build, or the build works. Neither is permitted to fail.

What we see happening now on CentOS is the configure tests soft-failing
but the build happens anyway. Genuine breakage incorrectly causing
LINUX_TPROXY2 to be defined when missing stuff and get through to a broken
build situation.


PS. There is a planned level of 'maybe' which I have not yet got around to
adding. It will allow testing of hard-fail options in future and accept
certain explicit fail messages (ie "wrong OS type" or "dependencies
missing") as an okay pass result.
Until we can get that build level going properly I'm reluctant to
hard-fail things. Since we can't then test them regularly.

Amos

Build failed in Hudson: 3.HEAD-amd64-CentOS-5.3 #162

[noc]

See 

Changes:

[Henrik Nordstrom ] Remove obsolete --disable-caps 
configure option

[Automatic source maintenance ] SourceFormat 
Enforcement

--
[...truncated 20127 lines...]
ranlib .libs/libstate.a
creating libstate.la
(cd .libs && rm -f libstate.la && ln -s ../libstate.la libstate.la)
ar cru .libs/libacls.a .libs/IntRange.o .libs/RegexData.o .libs/StringData.o 
.libs/Time.o .libs/TimeData.o .libs/Asn.o .libs/Browser.o 
.libs/DestinationDomain.o .libs/DestinationIp.o .libs/DomainData.o 
.libs/ExtUser.o .libs/HierCodeData.o .libs/HierCode.o .libs/HttpHeaderData.o 
.libs/HttpRepHeader.o .libs/HttpReqHeader.o .libs/HttpStatus.o .libs/Ip.o 
.libs/MaxConnection.o .libs/Method.o .libs/MethodData.o .libs/MyIp.o 
.libs/MyPort.o .libs/MyPortName.o .libs/PeerName.o .libs/Protocol.o 
.libs/ProtocolData.o .libs/Referer.o .libs/ReplyMimeType.o 
.libs/RequestMimeType.o .libs/SourceDomain.o .libs/SourceIp.o .libs/Tag.o 
.libs/Url.o .libs/UrlPath.o .libs/UrlPort.o .libs/UserData.o .libs/Gadgets.o 
.libs/CertificateData.o .libs/Certificate.o .libs/SslError.o 
.libs/SslErrorData.o .libs/Arp.o
ranlib .libs/libacls.a
creating libacls.la
(cd .libs && rm -f libacls.la && ln -s ../libacls.la libacls.la)
make[4]: Leaving directory 
`
Making all in fs
make[4]: Entering directory 
`
if /bin/sh ../../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H  
-I../../.. -I../../../include -I../../../src -I../../include -I../../../src/fs  
-I/usr/include/libxml2  -I/usr/include/libxml2 -Werror -Wall -Wpointer-arith 
-Wwrite-strings -Wcomments  -D_REENTRANT -D_FILE_OFFSET_BITS=64 -g -O2 -MT 
StoreFSufs.lo -MD -MP -MF ".deps/StoreFSufs.Tpo" -c -o StoreFSufs.lo `test -f 
'ufs/StoreFSufs.cc' || echo '../../../src/fs/'`ufs/StoreFSufs.cc; \
then mv -f ".deps/StoreFSufs.Tpo" ".deps/StoreFSufs.Plo"; else rm -f 
".deps/StoreFSufs.Tpo"; exit 1; fi
if /bin/sh ../../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H  
-I../../.. -I../../../include -I../../../src -I../../include -I../../../src/fs  
-I/usr/include/libxml2  -I/usr/include/libxml2 -Werror -Wall -Wpointer-arith 
-Wwrite-strings -Wcomments  -D_REENTRANT -D_FILE_OFFSET_BITS=64 -g -O2 -MT 
store_dir_ufs.lo -MD -MP -MF ".deps/store_dir_ufs.Tpo" -c -o store_dir_ufs.lo 
`test -f 'ufs/store_dir_ufs.cc' || echo 
'../../../src/fs/'`ufs/store_dir_ufs.cc; \
then mv -f ".deps/store_dir_ufs.Tpo" ".deps/store_dir_ufs.Plo"; else rm 
-f ".deps/store_dir_ufs.Tpo"; exit 1; fi
mkdir .libs
 g++ -DHAVE_CONFIG_H -I../../.. -I../../../include -I../../../src 
-I../../include -I../../../src/fs -I/usr/include/libxml2 -I/usr/include/libxml2 
-Werror -Wall -Wpointer-arith -Wwrite-strings -Wcomments -D_REENTRANT 
-D_FILE_OFFSET_BITS=64 -g -O2 -MT StoreFSufs.lo -MD -MP -MF 
.deps/StoreFSufs.Tpo -c ../../../src/fs/ufs/StoreFSufs.cc  -fPIC -DPIC -o 
.libs/StoreFSufs.o
 g++ -DHAVE_CONFIG_H -I../../.. -I../../../include -I../../../src 
-I../../include -I../../../src/fs -I/usr/include/libxml2 -I/usr/include/libxml2 
-Werror -Wall -Wpointer-arith -Wwrite-strings -Wcomments -D_REENTRANT 
-D_FILE_OFFSET_BITS=64 -g -O2 -MT store_dir_ufs.lo -MD -MP -MF 
.deps/store_dir_ufs.Tpo -c ../../../src/fs/ufs/store_dir_ufs.cc  -fPIC -DPIC -o 
.libs/store_dir_ufs.o
 g++ -DHAVE_CONFIG_H -I../../.. -I../../../include -I../../../src 
-I../../include -I../../../src/fs -I/usr/include/libxml2 -I/usr/include/libxml2 
-Werror -Wall -Wpointer-arith -Wwrite-strings -Wcomments -D_REENTRANT 
-D_FILE_OFFSET_BITS=64 -g -O2 -MT StoreFSufs.lo -MD -MP -MF 
.deps/StoreFSufs.Tpo -c ../../../src/fs/ufs/StoreFSufs.cc  -fPIC -DPIC -o 
StoreFSufs.o >/dev/null 2>&1
 g++ -DHAVE_CONFIG_H -I../../.. -I../../../include -I../../../src 
-I../../include -I../../../src/fs -I/usr/include/libxml2 -I/usr/include/libxml2 
-Werror -Wall -Wpointer-arith -Wwrite-strings -Wcomments -D_REENTRANT 
-D_FILE_OFFSET_BITS=64 -g -O2 -MT store_dir_ufs.lo -MD -MP -MF 
.deps/store_dir_ufs.Tpo -c ../../../src/fs/ufs/store_dir_ufs.cc  -fPIC -DPIC -o 
store_dir_ufs.o >/dev/null 2>&1
if /bin/sh ../../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H  
-I../../.. -I../../../include -I../../../src -I../../include -I../../../src/fs  
-I/usr/include/libxml2  -I/usr/include/libxml2 -Werror -Wall -Wpointer-arith 
-Wwrite-strings -Wcomments  -D_REENTRANT -D_FILE_OFFSET_BITS=64 -g -O2 -MT 
store_io_ufs.lo -MD -MP -MF ".deps/store_io_ufs.Tpo" -c -o store_io_ufs.lo 
`test -f 'ufs/store_io_ufs.cc' || echo '../../../src/fs/'`ufs/store_io_ufs.cc; \
then mv -f ".deps/store_io_ufs.Tpo" ".deps/store_io_ufs.Plo"; else rm 
-f ".deps/store_io_ufs.Tpo"; exit 1; fi
if /bin/sh ../../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H  
-I../../.. -I../

Re: /bzr/squid3/trunk/ r10049: Use libcap functions instead of raw kernel interface

[Henrik Nordstrom]

ons 2009-10-28 klockan 12:04 +1300 skrev Amos Jeffries:

>  * That should have been --with-cap2 right?  same as all the others ('lib'
> being redundant and libcap2 being the official name for it as differs from
> libcap and libcap-ng).

I disagree there.

The official name of libcap-x.y is libcap, not cap or cap2. Source level
API of libcap-1.x and libcap-2.x is the same, except that 2.x provides
more functions and is not binary compatible with 1.x applications. It's
a linear line of development from the same author.

"all the others" is a very broad statement. We only have two other
--with- and it's --with-dl and --with-openssl.

Official name of openssl is openssl, not libssl as Debian likes to call
it..

and dl do not really have a name of it's own, and is only known as -ldl
to most. And can probably go away now that we have libtool adding
similar tests..

>  * "--disable-caps" can drop entirely.  It was only used by Gentoo so far

Ok. Gone.

Regards
Henrik

Build failed in Hudson: 3.HEAD-amd64-CentOS-5.3 #160

[noc]

See 

Changes:

[Henrik Nordstrom ] Forward-port of 
STORE_META_OBJSIZE swapout from squid-2

[Henrik Nordstrom ] Minor quoting error in TPROXY 
autoconf foo

[Henrik Nordstrom ] Use libcap functions instead of 
raw kernel interface

Rework the Linux capabilities code to use the more stable libcap interface
instead or raw kernel syscalls.

Take two, making sure there is a sufficiently recent libcap-2.09 or later.

[Henrik Nordstrom ] Correct use_libcap internal 
variable setting when auto detected

[Henrik Nordstrom ] Cosmetic cleanup of 
--without-libcap message

[Henrik Nordstrom ] The old caps option was 
--disable-caps, not --without-caps...

[Henrik Nordstrom ] Properly rename --without-caps 
to --without-libcap, but keep the old as a backward compatibility option

[Henrik Nordstrom ] Don't try to write autoconf 
snippets when having a fewer..

[Henrik Nordstrom ] Correct AC_CACHE_CHECK use 
regarding sys/capability.h brokeness test

[Henrik Nordstrom ] Attempt to get the 
--with-libcap logics correct

[Henrik Nordstrom ] Activate libcap changes again

[Henrik Nordstrom ] Merged from trunk

--
[...truncated 20141 lines...]
ar cru .libs/libacls.a .libs/IntRange.o .libs/RegexData.o .libs/StringData.o 
.libs/Time.o .libs/TimeData.o .libs/Asn.o .libs/Browser.o 
.libs/DestinationDomain.o .libs/DestinationIp.o .libs/DomainData.o 
.libs/ExtUser.o .libs/HierCodeData.o .libs/HierCode.o .libs/HttpHeaderData.o 
.libs/HttpRepHeader.o .libs/HttpReqHeader.o .libs/HttpStatus.o .libs/Ip.o 
.libs/MaxConnection.o .libs/Method.o .libs/MethodData.o .libs/MyIp.o 
.libs/MyPort.o .libs/MyPortName.o .libs/PeerName.o .libs/Protocol.o 
.libs/ProtocolData.o .libs/Referer.o .libs/ReplyMimeType.o 
.libs/RequestMimeType.o .libs/SourceDomain.o .libs/SourceIp.o .libs/Tag.o 
.libs/Url.o .libs/UrlPath.o .libs/UrlPort.o .libs/UserData.o .libs/Gadgets.o 
.libs/CertificateData.o .libs/Certificate.o .libs/SslError.o 
.libs/SslErrorData.o .libs/Arp.o
ranlib .libs/libacls.a
creating libacls.la
(cd .libs && rm -f libacls.la && ln -s ../libacls.la libacls.la)
make[4]: Leaving directory 
`
Making all in fs
make[4]: Entering directory 
`
if /bin/sh ../../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H  
-I../../.. -I../../../include -I../../../src -I../../include -I../../../src/fs  
-I/usr/include/libxml2  -I/usr/include/libxml2 -Werror -Wall -Wpointer-arith 
-Wwrite-strings -Wcomments  -D_REENTRANT -D_FILE_OFFSET_BITS=64 -g -O2 -MT 
StoreFSufs.lo -MD -MP -MF ".deps/StoreFSufs.Tpo" -c -o StoreFSufs.lo `test -f 
'ufs/StoreFSufs.cc' || echo '../../../src/fs/'`ufs/StoreFSufs.cc; \
then mv -f ".deps/StoreFSufs.Tpo" ".deps/StoreFSufs.Plo"; else rm -f 
".deps/StoreFSufs.Tpo"; exit 1; fi
if /bin/sh ../../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H  
-I../../.. -I../../../include -I../../../src -I../../include -I../../../src/fs  
-I/usr/include/libxml2  -I/usr/include/libxml2 -Werror -Wall -Wpointer-arith 
-Wwrite-strings -Wcomments  -D_REENTRANT -D_FILE_OFFSET_BITS=64 -g -O2 -MT 
store_dir_ufs.lo -MD -MP -MF ".deps/store_dir_ufs.Tpo" -c -o store_dir_ufs.lo 
`test -f 'ufs/store_dir_ufs.cc' || echo 
'../../../src/fs/'`ufs/store_dir_ufs.cc; \
then mv -f ".deps/store_dir_ufs.Tpo" ".deps/store_dir_ufs.Plo"; else rm 
-f ".deps/store_dir_ufs.Tpo"; exit 1; fi
mkdir .libs
 g++ -DHAVE_CONFIG_H -I../../.. -I../../../include -I../../../src 
-I../../include -I../../../src/fs -I/usr/include/libxml2 -I/usr/include/libxml2 
-Werror -Wall -Wpointer-arith -Wwrite-strings -Wcomments -D_REENTRANT 
-D_FILE_OFFSET_BITS=64 -g -O2 -MT StoreFSufs.lo -MD -MP -MF 
.deps/StoreFSufs.Tpo -c ../../../src/fs/ufs/StoreFSufs.cc  -fPIC -DPIC -o 
.libs/StoreFSufs.o
 g++ -DHAVE_CONFIG_H -I../../.. -I../../../include -I../../../src 
-I../../include -I../../../src/fs -I/usr/include/libxml2 -I/usr/include/libxml2 
-Werror -Wall -Wpointer-arith -Wwrite-strings -Wcomments -D_REENTRANT 
-D_FILE_OFFSET_BITS=64 -g -O2 -MT store_dir_ufs.lo -MD -MP -MF 
.deps/store_dir_ufs.Tpo -c ../../../src/fs/ufs/store_dir_ufs.cc  -fPIC -DPIC -o 
.libs/store_dir_ufs.o
 g++ -DHAVE_CONFIG_H -I../../.. -I../../../include -I../../../src 
-I../../include -I../../../src/fs -I/usr/include/libxml2 -I/usr/include/libxml2 
-Werror -Wall -Wpointer-arith -Wwrite-strings -Wcomments -D_REENTRANT 
-D_FILE_OFFSET_BITS=64 -g -O2 -MT StoreFSufs.lo -MD -MP -MF 
.deps/StoreFSufs.Tpo -c ../../../src/fs/ufs/StoreFSufs.cc  -fPIC -DPIC -o 
StoreFSufs.o >/dev/null 2>&1
 g++ -DHAVE_CONFIG_H -I../../.. -I../../../include -I../../../src 
-I../../include -I../../../src/fs -I/usr/include/libxml2 -I/usr/include/libxml2 
-Werror -Wall -Wpointer-arith -Wwrite-strings -Wcomments -D_REEN

Re: compute swap_file_sz before packing it

[Henrik Nordstrom]

tis 2009-10-27 klockan 17:04 -0600 skrev Alex Rousskov:
> On 10/27/2009 04:07 PM, Henrik Nordstrom wrote:
> 
> >> Thank you for porting this.
> > 
> > Was already done, just not sent yet.
> 
> Will you commit your changes to trunk?
> 

Done.

Regards
Henrik

test-suite and OS specific options

[Henrik Nordstrom]

After the libcap change the testsuide now fails on CentOS, this time due
to layer-02-maximus enablind linux-tproxy but as libcap isn't available
it's not really enabled but still compiled. A little confusing, as far
as I know I did not change the logics on when to enable what of Linux
tproxy..

So to the question, is it really sane that the test suite blindly
enables OS specific options? I mean I would not expect the build to work
if one uses --enable-linux-netfilter on an BSD box for example, or
--enable-linux-tproxy2 on a box where Linux TPROXY2 is not available or
alternatively with now required libcap missing..

Regards
Henrik

Re: /bzr/squid3/trunk/ r10049: Use libcap functions instead of raw kernel interface

[Amos Jeffries]

On Tue, 27 Oct 2009 22:55:58 +0100, Henrik Nordstrom
 wrote:
> 
> revno: 10049 [merge]
> committer: Henrik Nordstrom 
> branch nick: trunk
> timestamp: Tue 2009-10-27 22:55:58 +0100
> message:
>   Use libcap functions instead of raw kernel interface
>   
>   Rework the Linux capabilities code to use the more stable libcap
>   interface
>   instead or raw kernel syscalls.
>   
>   Take two, making sure there is a sufficiently recent libcap-2.09 or
>   later.
> modified:
>   configure.in
>   src/tools.cc

Okay, sorry to drop this on you after the fact but, umm...

 * That should have been --with-cap2 right?  same as all the others ('lib'
being redundant and libcap2 being the official name for it as differs from
libcap and libcap-ng).

 * "--disable-caps" can drop entirely.  It was only used by Gentoo so far
as I know and on precisely one beta release so far. I've earlier with Alin
that its okay to drop. It was only --enable to begin with due to Gentoo
(sponsors) having a number of packages where libraries are included with
--enable/--disable despite the autoconf guidelines.

Amos

Re: compute swap_file_sz before packing it

[Alex Rousskov]

On 10/27/2009 04:07 PM, Henrik Nordstrom wrote:

>> Thank you for porting this.
> 
> Was already done, just not sent yet.

Will you commit your changes to trunk?

Thank you,

Alex.

Re: [MERGE] Use libcap instead of direct linux capability syscalls

[Amos Jeffries]

On Tue, 27 Oct 2009 23:09:01 +0100, Henrik Nordstrom
 wrote:
> ons 2009-10-28 klockan 10:32 +1300 skrev Amos Jeffries:
> 
>> > The configure test was broken however, always reporting failure...
>> 
>> Strange. That was the change the Gentoo people are all enjoying at the
>> moment.
> 
> Well, I think most are silently happy with the workaround enabled even
> if not strictly needed.
> 
> Regards
> Henrik

Yeah most will have the old versions of stuff which worked since the hack
was created.

It's those few bug reporters which noticed the missing le32 definition
that have me scratching my head now. If the LIBCAP_BROKEN was always
failing  (==1), how did it fix things for them? simply re-ordering the
include into compat?

Meh. Oh well, it works for now. We can ignore until some bright day when
the erase button is pressed a few times :)

Amos

Build failed in Hudson: 3.HEAD-amd64-CentOS-5.3 #159

[noc]

See 

Changes:

[Henrik Nordstrom ] Use libcap functions instead of 
raw kernel interface

Rework the Linux capabilities code to use the more stable libcap interface
instead or raw kernel syscalls.

Take two, making sure there is a sufficiently recent libcap-2.09 or later.

[Henrik Nordstrom ] Correct use_libcap internal 
variable setting when auto detected

[Henrik Nordstrom ] Cosmetic cleanup of 
--without-libcap message

[Henrik Nordstrom ] The old caps option was 
--disable-caps, not --without-caps...

[Henrik Nordstrom ] Properly rename --without-caps 
to --without-libcap, but keep the old as a backward compatibility option

[Henrik Nordstrom ] Don't try to write autoconf 
snippets when having a fewer..

[Henrik Nordstrom ] Correct AC_CACHE_CHECK use 
regarding sys/capability.h brokeness test

[Henrik Nordstrom ] Attempt to get the 
--with-libcap logics correct

[Henrik Nordstrom ] Activate libcap changes again

[Henrik Nordstrom ] Merged from trunk

[Henrik Nordstrom ] Correct squidclient 64-bit POST 
fix that should work on 32-bit LFS systems as well

--
[...truncated 20134 lines...]
creating libstate.la
(cd .libs && rm -f libstate.la && ln -s ../libstate.la libstate.la)
 g++ -DHAVE_CONFIG_H -I../../.. -I../../../include -I../../../src 
-I../../include -I/usr/include/libxml2 -I/usr/include/libxml2 -Werror -Wall 
-Wpointer-arith -Wwrite-strings -Wcomments -D_REENTRANT -D_FILE_OFFSET_BITS=64 
-g -O2 -MT Arp.lo -MD -MP -MF .deps/Arp.Tpo -c ../../../src/acl/Arp.cc  -fPIC 
-DPIC -o Arp.o >/dev/null 2>&1
/bin/sh ../../libtool --tag=CXX --mode=link g++ -I/usr/include/libxml2 -Werror 
-Wall -Wpointer-arith -Wwrite-strings -Wcomments  -D_REENTRANT 
-D_FILE_OFFSET_BITS=64 -g -O2  -g -o libacls.la   IntRange.lo RegexData.lo 
StringData.lo Time.lo TimeData.lo Asn.lo Browser.lo DestinationDomain.lo 
DestinationIp.lo DomainData.lo ExtUser.lo HierCodeData.lo HierCode.lo 
HttpHeaderData.lo HttpRepHeader.lo HttpReqHeader.lo HttpStatus.lo Ip.lo 
MaxConnection.lo Method.lo MethodData.lo MyIp.lo MyPort.lo MyPortName.lo 
PeerName.lo Protocol.lo ProtocolData.lo Referer.lo ReplyMimeType.lo 
RequestMimeType.lo SourceDomain.lo SourceIp.lo Tag.lo Url.lo UrlPath.lo 
UrlPort.lo UserData.lo Gadgets.lo CertificateData.lo Certificate.lo SslError.lo 
SslErrorData.lo Arp.lo  
ar cru .libs/libacls.a .libs/IntRange.o .libs/RegexData.o .libs/StringData.o 
.libs/Time.o .libs/TimeData.o .libs/Asn.o .libs/Browser.o 
.libs/DestinationDomain.o .libs/DestinationIp.o .libs/DomainData.o 
.libs/ExtUser.o .libs/HierCodeData.o .libs/HierCode.o .libs/HttpHeaderData.o 
.libs/HttpRepHeader.o .libs/HttpReqHeader.o .libs/HttpStatus.o .libs/Ip.o 
.libs/MaxConnection.o .libs/Method.o .libs/MethodData.o .libs/MyIp.o 
.libs/MyPort.o .libs/MyPortName.o .libs/PeerName.o .libs/Protocol.o 
.libs/ProtocolData.o .libs/Referer.o .libs/ReplyMimeType.o 
.libs/RequestMimeType.o .libs/SourceDomain.o .libs/SourceIp.o .libs/Tag.o 
.libs/Url.o .libs/UrlPath.o .libs/UrlPort.o .libs/UserData.o .libs/Gadgets.o 
.libs/CertificateData.o .libs/Certificate.o .libs/SslError.o 
.libs/SslErrorData.o .libs/Arp.o
ranlib .libs/libacls.a
creating libacls.la
(cd .libs && rm -f libacls.la && ln -s ../libacls.la libacls.la)
make[4]: Leaving directory 
`
Making all in fs
make[4]: Entering directory 
`
if /bin/sh ../../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H  
-I../../.. -I../../../include -I../../../src -I../../include -I../../../src/fs  
-I/usr/include/libxml2  -I/usr/include/libxml2 -Werror -Wall -Wpointer-arith 
-Wwrite-strings -Wcomments  -D_REENTRANT -D_FILE_OFFSET_BITS=64 -g -O2 -MT 
StoreFSufs.lo -MD -MP -MF ".deps/StoreFSufs.Tpo" -c -o StoreFSufs.lo `test -f 
'ufs/StoreFSufs.cc' || echo '../../../src/fs/'`ufs/StoreFSufs.cc; \
then mv -f ".deps/StoreFSufs.Tpo" ".deps/StoreFSufs.Plo"; else rm -f 
".deps/StoreFSufs.Tpo"; exit 1; fi
if /bin/sh ../../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H  
-I../../.. -I../../../include -I../../../src -I../../include -I../../../src/fs  
-I/usr/include/libxml2  -I/usr/include/libxml2 -Werror -Wall -Wpointer-arith 
-Wwrite-strings -Wcomments  -D_REENTRANT -D_FILE_OFFSET_BITS=64 -g -O2 -MT 
store_dir_ufs.lo -MD -MP -MF ".deps/store_dir_ufs.Tpo" -c -o store_dir_ufs.lo 
`test -f 'ufs/store_dir_ufs.cc' || echo 
'../../../src/fs/'`ufs/store_dir_ufs.cc; \
then mv -f ".deps/store_dir_ufs.Tpo" ".deps/store_dir_ufs.Plo"; else rm 
-f ".deps/store_dir_ufs.Tpo"; exit 1; fi
mkdir .libs
 g++ -DHAVE_CONFIG_H -I../../.. -I../../../include -I../../../src 
-I../../include -I../../../src/fs -I/usr/include/libxml2 -I/usr/include/libxml2 
-Werror -Wall -Wpointer-arith -Wwrite-strings -Wcomments -D_REE

Re: [MERGE] Use libcap instead of direct linux capability syscalls

[Henrik Nordstrom]

ons 2009-10-28 klockan 10:32 +1300 skrev Amos Jeffries:

> > The configure test was broken however, always reporting failure...
> 
> Strange. That was the change the Gentoo people are all enjoying at the
> moment.

Well, I think most are silently happy with the workaround enabled even
if not strictly needed.

Regards
Henrik

Re: compute swap_file_sz before packing it

[Henrik Nordstrom]

tis 2009-10-27 klockan 15:51 -0600 skrev Alex Rousskov:

> To compute StoreEntry::swap_file_sz, I will add up the ported
> STORE_META_OBJSIZE value and the swap_hdr_len set by StoreMetaUnpacker.
> Would you compute it differently?

Sounds right to me.

> What should I do if STORE_META_OBJSIZE is not known? Does this question
> itself imply that each store that wants to rebuild an index has to store
> the final object size somewhere or update the STORE_META_OBJSIZE value?

Exactly. But see my previous response some seconds ago.

As you already noticed the ufs family uses the filesystem file size meta
information to rebuild swap_file_sz.

COSS in Squid-2 uses STORE_META_OBJSIZE + swap_hdr_sz.

> Thank you for porting this.

Was already done, just not sent yet.

> What happens to STORE_META_OBJSIZE if the object size is not yet known
> at the time when Squid start swapping content to disk?

Then there is no STORE_META_OBJSIZE.

But stores with a max size limit won't get such swapouts.

> The whole patch is not needed if we start relying on STORE_META_OBJSIZE,
> I guess.

Probably, was more a note to illustrate the issues that field battles
with..

Regards
Henrik

Re: compute swap_file_sz before packing it

[Henrik Nordstrom]

tis 2009-10-27 klockan 15:23 -0600 skrev Alex Rousskov:

> Is not that always the case? Even if a store refuses to accept objects
> larger than X KB, that does not mean that all objects will be X KB in
> size, regardless of any reasonable X value. Or did you mean something
> else by "unbounded"?

There is two kinds of swapouts related to this

a) Size-bounded, where objects are known to be of a certain size.

b) Size not known at start of swapout. Impossible to record the size in
headers.

When there is at least one cache_dir with a size restriction we buffer
objects of type 'b' before swapout in case it's small enough to actaully
fit in the cache_dir policy even if size initially unknown.

> Technically, core does not know the true content size for some responses
> until the response has been received, but I do not remember whether we
> allow such responses to be cachable.

We do. It's a quite common response form.

Regards
Henrik

Re: [MERGE] Use libcap instead of direct linux capability syscalls

[Henrik Nordstrom]

tis 2009-10-27 klockan 14:46 +1300 skrev Amos Jeffries:

> I would like the patch to be updated with said function test logic to
> auto-disable libcap2, modulo a fatal death on the --with-* case.  Then for
> it to go back in :)

Done.

Regards
Henrik

Re: compute swap_file_sz before packing it

[Alex Rousskov]

On 10/27/2009 02:41 PM, Henrik Nordstrom wrote:
> The object size field in STORE_META_STD should be ignored. Got broken
> many years ago (1997 or so), and should be recalculated by using
> STORE_META_OBJSIZE or alternatively the on-disk object size.

Rock code can ignore the swap_file_sz field stored in STORE_META_STD but
the core code is still using StoreEntry::swap_file_sz so I have to set
that field to the right value when building an index.

To compute StoreEntry::swap_file_sz, I will add up the ported
STORE_META_OBJSIZE value and the swap_hdr_len set by StoreMetaUnpacker.
Would you compute it differently?

What should I do if STORE_META_OBJSIZE is not known? Does this question
itself imply that each store that wants to rebuild an index has to store
the final object size somewhere or update the STORE_META_OBJSIZE value?

>> Moreover, STORE_META_OBJSIZE has the following comment attached to its
>> declaration: "not implemented, squid26 compatibility" and appears to be
>> unused...
> 
> Right.. should be forward ported. [attached]

Thank you for porting this.

>> Neither approach works for Rock store because Rock store does not have a
>> swap state file like COSS and does not use individual files like UFS.
>> That is why it has to rely on the "file" size information supplied by
>> the core. Perhaps there is a better way of getting that information, but
>> I do not know it.
> 
> STORE_META_OBJSIZE is the object size (if known) not including TLV
> headers, and is generally what you need to know in order to access the
> object.

What happens to STORE_META_OBJSIZE if the object size is not yet known
at the time when Squid start swapping content to disk?

> Long term objects should be split in TLV + HTTP Headers (probably part
> of TLV) + Content, but that's another topic.. 


> Actual file storage size is more a business of the cache_dir than the
> core..

I agree but the current core code does not :-).


>>> +// so that storeSwapMetaBuild/Pack can pack corrent swap_file_sz
>>> +swap_file_sz = objectLen() + mem_obj->swap_hdr_sz;
>>> +
> 
> objectLen() MAY be -1 here...

The whole patch is not needed if we start relying on STORE_META_OBJSIZE,
I guess.

Thank you,

Alex.

Re: [MERGE] Use libcap instead of direct linux capability syscalls

[Amos Jeffries]

On Tue, 27 Oct 2009 15:08:24 +0100, Henrik Nordstrom
 wrote:
> ons 2009-10-28 klockan 01:35 +1300 skrev Amos Jeffries:
> 
>> Not entirely sure what version. I think the Gentoo 2.15 or 2.16 is 
>> fixed. My Ubuntu 2.11 is broken by the test results of last build I
ran.
> 
> Ok.
> 
> The configure test was broken however, always reporting failure...

Strange. That was the change the Gentoo people are all enjoying at the
moment.

Amos

Re: compute swap_file_sz before packing it

[Alex Rousskov]

On 10/27/2009 02:46 PM, Henrik Nordstrom wrote:
> tis 2009-10-27 klockan 21:41 +0100 skrev Henrik Nordstrom:
> 
>> Actual file storage size is more a business of the cache_dir than the
>> core..
> 
> Forgot to mention.. nothing stops a cache_dir implementation from
> storing this attribute somehow associated with the data if one likes to,
> but for cache_dirs taking "unbounded" object sizes the information is
> not known until the object is completed.

Is not that always the case? Even if a store refuses to accept objects
larger than X KB, that does not mean that all objects will be X KB in
size, regardless of any reasonable X value. Or did you mean something
else by "unbounded"?

Technically, core does not know the true content size for some responses
until the response has been received, but I do not remember whether we
allow such responses to be cachable.

Thanks,

Alex.

Re: compute swap_file_sz before packing it

[Henrik Nordstrom]

tis 2009-10-27 klockan 21:41 +0100 skrev Henrik Nordstrom:

> Actual file storage size is more a business of the cache_dir than the
> core..

Forgot to mention.. nothing stops a cache_dir implementation from
storing this attribute somehow associated with the data if one likes to,
but for cache_dirs taking "unbounded" object sizes the information is
not known until the object is completed.

Regards
Henrik

Re: compute swap_file_sz before packing it

[Henrik Nordstrom]

tis 2009-10-27 klockan 13:43 -0600 skrev Alex Rousskov:

> Hi Henrik,
> 
> Can you explain what you mean by "just ignore" above? It is kind of
> difficult for me to ignore the only code that seems to supply the info
> Rock store needs. Do you mean we should ultimately remove STORE_META_STD
> from Squid, replacing all its current uses with STORE_META_OBJSIZE?

The object size field in STORE_META_STD should be ignored. Got broken
many years ago (1997 or so), and should be recalculated by using
STORE_META_OBJSIZE or alternatively the on-disk object size.

> Moreover, STORE_META_OBJSIZE has the following comment attached to its
> declaration: "not implemented, squid26 compatibility" and appears to be
> unused...

Right.. should be forward ported. [attached]

> Neither approach works for Rock store because Rock store does not have a
> swap state file like COSS and does not use individual files like UFS.
> That is why it has to rely on the "file" size information supplied by
> the core. Perhaps there is a better way of getting that information, but
> I do not know it.

STORE_META_OBJSIZE is the object size (if known) not including TLV
headers, and is generally what you need to know in order to access the
object.

Long term objects should be split in TLV + HTTP Headers (probably part
of TLV) + Content, but that's another topic.. 

Actual file storage size is more a business of the cache_dir than the
core..

> > +// so that storeSwapMetaBuild/Pack can pack corrent swap_file_sz
> > +swap_file_sz = objectLen() + mem_obj->swap_hdr_sz;
> > +

objectLen() MAY be -1 here...

Regards
Henrik
=== modified file 'src/StoreMeta.h'
--- src/StoreMeta.h	2009-01-21 03:47:47 +
+++ src/StoreMeta.h	2009-08-27 12:35:36 +
@@ -127,10 +127,6 @@
  */
 STORE_META_STD_LFS,
 
-/**
- \deprecated
- * Object size, not implemented, squid26 compatibility
- */
 STORE_META_OBJSIZE,
 
 STORE_META_STOREURL,	/* the store url, if different to the normal URL */

=== modified file 'src/store_swapmeta.cc'
--- src/store_swapmeta.cc	2009-01-21 03:47:47 +
+++ src/store_swapmeta.cc	2009-08-27 12:35:36 +
@@ -61,6 +61,7 @@
 tlv **T = &TLV;
 const char *url;
 const char *vary;
+const int64_t objsize = e->objectLen();
 assert(e->mem_obj != NULL);
 assert(e->swap_status == SWAPOUT_WRITING);
 url = e->url();
@@ -88,6 +89,17 @@
 return NULL;
 }
 
+
+if (objsize >= 0) {
+	T = StoreMeta::Add(T, t);
+	t = StoreMeta::Factory(STORE_META_OBJSIZE, sizeof(objsize), &objsize);
+
+	if (!t) {
+	storeSwapTLVFree(TLV);
+	return NULL;
+	}
+}
+
 T = StoreMeta::Add(T, t);
 vary = e->mem_obj->vary_headers;
 

Re: compute swap_file_sz before packing it

[Alex Rousskov]

On 09/14/2009 04:49 PM, Robert Collins wrote:
> Just a small meta point: The new function you're adding looks like it
> should be a method to me.

It should be. It is not because it is very dependent on the internal
storeSwapMetaBuild() algorithm, and I wanted to keep the two functions
close to each other. I added XXX to reflect the dependency.

I do not think the dependency should be broken: we need to predict the
packed header size and do not want to spend CPU cycles on packing twice
or make the code even worse by adding "if (really_pack)" statements to
storeSwapMetaBuild() and friends.

Thank you,

Alex.

Re: compute swap_file_sz before packing it

[Alex Rousskov]

On 09/14/2009 03:26 PM, Henrik Nordstrom wrote:
> mån 2009-09-14 klockan 02:04 -0600 skrev Alex Rousskov:
> 
>> Apparently, we are packing StoreEntry.swap_file_sz into the stored 
>> entry header before we compute its value. This happens because to 
>> compute swap_file_sz, we need to know swap_hdr_sz, the size of the 
>> stored entry header. To compute swap_hdr_sz, we need to pack the 
>> header, but we cannot pack the header because swap_file_sz field has 
>> not been computed yet. Chicken and egg, hidden behind a bunch of void* 
>> memcopies.
> 
> Hmm... are we really packing that one? I thougt we only packed the objet
> size in STORE_META_OBJSIZE which is excluding the TLV header.
> 
> Right... it's in STORE_META_STD. Just ignore that one as a design error.
> Use STORE_META_OBJSIZE instead, and calculate the "file size" as object
> size + header size if needed.

Hi Henrik,

Can you explain what you mean by "just ignore" above? It is kind of
difficult for me to ignore the only code that seems to supply the info
Rock store needs. Do you mean we should ultimately remove STORE_META_STD
from Squid, replacing all its current uses with STORE_META_OBJSIZE?

In my case, the core part of Squid code is using STORE_META_STD as a
part of stored "file" header and Rock code needs to rebuild the index by
loading that header from each "file". There is no swap state. Are you
suggesting that the core Squid code should not be using STORE_META_STD
for stored object headers and should be using STORE_META_OBJSIZE instead?

Moreover, STORE_META_OBJSIZE has the following comment attached to its
declaration: "not implemented, squid26 compatibility" and appears to be
unused...


>> The attached patch is a "work in progress" hack to facilitate the 
>> discussion. The patch computes the future swap_hdr_sz without packing 
>> the header, sets swap_file_sz, and then packs the header. It is not 
>> very efficient but appears to work.
> 
> Do we need that? COSS rebuildig is quite content with using
> STORE_META_OBJSIZE.

Squid3 COSS cannot rebuild from the store db, only from swap state. If
swap state is gone, COSS db is gone. See storeCossDirRebuild. COSS is
using StoreSwapLogData::swap_file_sz in swap state and that field is not
broken. Broken STORE_META_STD is in all the store db entries but COSS
does not use it so it does not know that it is broken.

UFS can rebuild from store dir, but it uses fstat() call to determine
the file size and ignores broken swap_file_sz.

Neither approach works for Rock store because Rock store does not have a
swap state file like COSS and does not use individual files like UFS.
That is why it has to rely on the "file" size information supplied by
the core. Perhaps there is a better way of getting that information, but
I do not know it.


Please clarify what your suggestion for dealing with this bug is.

Thank you,

Alex.


> === modified file 'src/store.cc'
> --- src/store.cc  2009-09-02 05:42:24 +
> +++ src/store.cc  2009-09-13 11:06:37 +
> @@ -1801,12 +1801,18 @@
>  char const *
>  StoreEntry::getSerialisedMetaData()
>  {
> +const size_t swap_hdr_sz0 = storeSwapMetaSize(this);
> +assert (swap_hdr_sz0 >= 0);
> +mem_obj->swap_hdr_sz = (size_t) swap_hdr_sz0;
> +// now we can use swap_hdr_sz to calculate swap_file_sz
> +// so that storeSwapMetaBuild/Pack can pack corrent swap_file_sz
> +swap_file_sz = objectLen() + mem_obj->swap_hdr_sz;
> +
>  StoreMeta *tlv_list = storeSwapMetaBuild(this);
>  int swap_hdr_sz;
>  char *result = storeSwapMetaPack(tlv_list, &swap_hdr_sz);
> +assert(static_cast(swap_hdr_sz0) == swap_hdr_sz);
>  storeSwapTLVFree(tlv_list);
> -assert (swap_hdr_sz >= 0);
> -mem_obj->swap_hdr_sz = (size_t) swap_hdr_sz;
>  return result;
>  }
>  
> 
> === modified file 'src/store_swapmeta.cc'
> --- src/store_swapmeta.cc 2009-02-01 10:09:23 +
> +++ src/store_swapmeta.cc 2009-09-13 11:12:46 +
> @@ -52,6 +52,30 @@
>  }
>  
>  /*
> + * Calculate TLV list size for a StoreEntry
> + * XXX: Must match the actual storeSwapMetaBuild result size
> + */
> +size_t
> +storeSwapMetaSize(const StoreEntry * e)
> +{
> + size_t size = 0;
> +++size; // STORE_META_OK
> +size += sizeof(int); // size of header to follow
> +
> + const size_t pfx = sizeof(char) + sizeof(int); // in the start of list 
> entries
> +
> +size += pfx + SQUID_MD5_DIGEST_LENGTH;
> +size += pfx + STORE_HDR_METASIZE;
> +size += pfx + strlen(e->url()) + 1;
> +
> +if (const char *vary = e->mem_obj->vary_headers)
> +size += pfx + strlen(vary) + 1;
> +
> +debugs(20, 3, "storeSwapMetaSize(" << e->url() << "): " << size);
> +return size;
> +}
> +
> +/*
>   * Build a TLV list for a StoreEntry
>   */
>  tlv *
> 
> === modified file 'src/store_swapout.cc'
> --- src/store_swapout.cc  2009-09-02 05:53:27 +
> +++ src/store_swapout.cc  2009-09-13 09:24:16 +
> @@ -345,7 +345,8 @@
>  debugs(20, 3, "

Re: [MERGE] Use libcap instead of direct linux capability syscalls

[Henrik Nordstrom]

ons 2009-10-28 klockan 01:35 +1300 skrev Amos Jeffries:

> Not entirely sure what version. I think the Gentoo 2.15 or 2.16 is 
> fixed. My Ubuntu 2.11 is broken by the test results of last build I ran.

Ok.

The configure test was broken however, always reporting failure...

> I think we need to keep the magic voodoo for a while longer.

It's still there.

Regards
Henrik

Re: [MERGE] Use libcap instead of direct linux capability syscalls

[Amos Jeffries]

Henrik Nordstrom wrote:

tis 2009-10-27 klockan 14:46 +1300 skrev Amos Jeffries:


I would like the patch to be updated with said function test logic to
auto-disable libcap2, modulo a fatal death on the --with-* case.  Then for
it to go back in :)


OK. I'll give it a shot. Will make TPROXY require libcap-2.09 or later.

At what version was libcap fixed for the issue we test in the magic
sys/capability.h test case? Comment says libcap2 fixed, so I guess we no
longer need this?

Regards
Henrik



Not entirely sure what version. I think the Gentoo 2.15 or 2.16 is 
fixed. My Ubuntu 2.11 is broken by the test results of last build I ran.


I think we need to keep the magic voodoo for a while longer.

Amos
--
Please be using
  Current Stable Squid 2.7.STABLE7 or 3.0.STABLE19
  Current Beta Squid 3.1.0.14

Re: squid-smp

[Amos Jeffries]

Alex Rousskov wrote:

Hello,

I will be working on SMP support in the coming months. I caught up
on this and the previous SMP-related squid-dev thread and it looks like
 the approach I currently favor has been discussed (again) and did not
cause any violent objections, although some ideas were much more
ambitious. I am not sure how we can reach consensus on this topic, but I
will through in some specifics in hope to better identify competing
approaches.


Okay. Before you get buried in this work might I request that you look 
at the stale-while-revalidate / stale-if-error code that was nearly 
finished?

http://bugs.squid-cache.org/show_bug.cgi?id=2255

Yahoo! named those as two of their requirements before it would be 
possible for them to assist with performance testing Squid-3. They might 
not be adverse to helping test SMP support if the other requirements are 
available.


Collapsed forwarding is also a requirement, but I suspect it is too 
close to the request handling and needs a re-designed code architecture 
to fit with whatever SMP threading model is taken anyways.




My short-term focus would be on the following three areas:

A) Identifying a few "large", "rarely-interacting" threads that would
work reasonably well on an 8-core 2-CPU machine with 8 http_ports. This
should take the lessons learned from existing SMP designs into account,
with Squid specifics in mind. Henrik, Amos, and Adrian started
discussing this already.

B) Making commonly used primitives thread-safe (mostly not in terms of
locking their shared state but in terms of not using static/shared data
that needs locking). Many posts on this subject, starting with Roberts
advice to desynchronize.

C) Posting performance benchmarking results for single- and
multi-instance Squids on mutli-core systems as a baseline.


My mid-term focus will probably be on sharing http_port, memory cache,
disk cache and possibly loggin/stats among a "few large threads".


My overall goal is to at least approach the performance of a
multi-instance caching Squid on 8-core hardware.


I am not excited by the "one thread per message", "one thread per
AsyncJob", or similar "many tiny threads" designs because, IMO, they
would require too much rewriting to be implemented properly. This may
need to be re-evaluated as the world moves towards 1000-core systems,
but a lot of improvements necessary for the "few large threads" design
will not be wasted anyway.

I hope that by focusing on a "few large threads" design and fixing
primitives we can gain "enough" SMP benefits in a few months of active
development. If you think there is a better way to get SMP benefits in
the foreseeable future, please post.

Thank you,

Alex.



--
Please be using
  Current Stable Squid 2.7.STABLE7 or 3.0.STABLE19
  Current Beta Squid 3.1.0.14

Re: squid-smp

[Alex Rousskov]

Hello,

I will be working on SMP support in the coming months. I caught up
on this and the previous SMP-related squid-dev thread and it looks like
 the approach I currently favor has been discussed (again) and did not
cause any violent objections, although some ideas were much more
ambitious. I am not sure how we can reach consensus on this topic, but I
will through in some specifics in hope to better identify competing
approaches.

My short-term focus would be on the following three areas:

A) Identifying a few "large", "rarely-interacting" threads that would
work reasonably well on an 8-core 2-CPU machine with 8 http_ports. This
should take the lessons learned from existing SMP designs into account,
with Squid specifics in mind. Henrik, Amos, and Adrian started
discussing this already.

B) Making commonly used primitives thread-safe (mostly not in terms of
locking their shared state but in terms of not using static/shared data
that needs locking). Many posts on this subject, starting with Roberts
advice to desynchronize.

C) Posting performance benchmarking results for single- and
multi-instance Squids on mutli-core systems as a baseline.


My mid-term focus will probably be on sharing http_port, memory cache,
disk cache and possibly loggin/stats among a "few large threads".


My overall goal is to at least approach the performance of a
multi-instance caching Squid on 8-core hardware.


I am not excited by the "one thread per message", "one thread per
AsyncJob", or similar "many tiny threads" designs because, IMO, they
would require too much rewriting to be implemented properly. This may
need to be re-evaluated as the world moves towards 1000-core systems,
but a lot of improvements necessary for the "few large threads" design
will not be wasted anyway.

I hope that by focusing on a "few large threads" design and fixing
primitives we can gain "enough" SMP benefits in a few months of active
development. If you think there is a better way to get SMP benefits in
the foreseeable future, please post.

Thank you,

Alex.

Re: [MERGE] Use libcap instead of direct linux capability syscalls

[Henrik Nordstrom]

tis 2009-10-27 klockan 14:46 +1300 skrev Amos Jeffries:

> I would like the patch to be updated with said function test logic to
> auto-disable libcap2, modulo a fatal death on the --with-* case.  Then for
> it to go back in :)

OK. I'll give it a shot. Will make TPROXY require libcap-2.09 or later.

At what version was libcap fixed for the issue we test in the magic
sys/capability.h test case? Comment says libcap2 fixed, so I guess we no
longer need this?

Regards
Henrik

store_client::readBody and writing to memory

[Alex Rousskov]

Hello,

Recent memory cache enhancements in trunk added a piece of code that
copies disk object data into the memory cache. The code has a comment
that essentially says that Squid code parts has their own idea what the
right "offset" is and we have to adjust when going from the store feud
to the memory feud:

/* Copy read data back into memory.
 * but first we need to adjust offset.. some parts of the code
 * counts offset including headers, some parts count offset as
 * withing the body.. copyInto is including headers, but the mem
 * cache expects offset without headers (using negative for headers)
 * eventually not storing packed headers in memory at all.
 */

Following the above comment, the code decrements for-memory offset by
the reply header size (copyInto.offset - rep->hdr_sz). The same code
does not decrement the buffer length (len) and does not increment the
buffer pointer (copyInto.data) to exclude parsed headers.

I suspect that everything works correctly because the memory code does
the necessary length and buffer pointer adjustments itself, when needed,
under the assumption that the headers are always in the first written
buffer. Is that the right explanation for why we adjust the offset but
not the buffer pointer or the buffer length?

If the memory code assumes the headers are there in the first written
chunk, would it be better to adjust the "offset" in the memory code as
well then, so that all the adjustments are done in one place?

Is it possible to define exactly what that offset parameter means for
the MemObject::write() code? Or does the meaning change depending on
MemObject state?

Thank you,

Alex.