Re: Helper children defaults

[Kinkie]

On Thu, Nov 26, 2009 at 5:35 AM, Amos Jeffries  wrote:
> I'm making the helper children configurable for on-demand startup so a
> minimal set can be started and allowed to grow up to a max as determined by
> traffic load.
> Growth is triggered by helpers dying or requests needing to be queued when
> all helpers are full.

[...]

Please also take a look at the helper-demux feature branch, which
approaches this with a slightly different perspective.


-- 
/kinkie

Build failed in Hudson: 3.HEAD-sparc-opensolaris-SunStudioCc #20

[noc]

See 


Changes:

[Henrik Nordstrom ] Bug #2155: Assertion failures 
on malformed Content-Range response headers

A number of conditions with malformed Content-Range headers were
not trapped properly triggering odd conditions in the code.

[Henrik Nordstrom ] Remove libTrie C unit test from 
the Makefile.

Automake do not like having rules for unreferenced binaries.

--
[...truncated 21297 lines...]
Making dvi in repl
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in auth
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in ip
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in icmp
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in ident
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in log
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in adaptation
make[3]: Entering directory 
`
Making dvi in icap
make[4]: Entering directory 
`
make[4]: Nothing to be done for `dvi'.
make[4]: Leaving directory 
`
make[4]: Entering directory 
`
make[4]: Nothing to be done for `dvi-am'.
make[4]: Leaving directory 
`
make[3]: Leaving directory 
`
Making dvi in esi
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi-am'.
make[3]: Leaving directory 
`
make[2]: Leaving directory 
`
Making dvi in icons
make[2]: Entering directory 
`
make[2]: Nothing to be done for `dvi'.
make[2]: Leaving directory 
`

Helper children defaults

[Amos Jeffries]

I'm making the helper children configurable for on-demand startup so a 
minimal set can be started and allowed to grow up to a max as determined 
by traffic load.
Growth is triggered by helpers dying or requests needing to be queued 
when all helpers are full.


We presently have a default of 5 helpers of each type used started 
synchronously on startup and reconfigure.


Are there any objections to making the 3.2 helpers use this on-demand 
style by default:


 * start 1 child synchronously on start/reconfigure
 * new child helpers as needed in bunches of 2
 * maximum running kept capped at 5.
?

This affects helpers for auth_param, url_rewrite, and external_acl_type.

Amos

Re: your suggestion for range_offset_limit

[Amos Jeffries]

Matthew Morgan wrote:

On Wed, Nov 25, 2009 at 7:09 PM, Amos Jeffries  wrote:

Matthew Morgan wrote:

Sorry it's taking me so long to get this done, but I do have a question.

You suggested making getRangeOffsetLimit a member of HttpReply.  There are
two places where this method currently needs to be called: one is
CheckQuickAbort2() in store_client.cc.  This one will be easy, as I can just
do entry->getReply()->getRangeOffsetLimit().

The other is HttpStateData::decideIfWeDoRanges in http.cc.  Here, all we
have access to is an HttpRequest object.  I looked through the source to see
if I could find where a request owned or had access to a reply, but I don't
see anything like that.  If getRangeOffsetLimit were a member of HttpReply,
what do you suggest doing here?  I could make a static version of the
method, but that wouldn't allow caching the result.

Ah. I see. Quite right.

After a bit more though I find my original request a bit weird.

Yes it should be a _Request_ member and do its caching there. You can go
ahead with that now while we discuss whether to do a slight tweak on top of
the basic feature.


[cc'ing squid-dev so others can provide input]

I'm not certain of the behavior we want here if we do open the ACLs to reply
details. Some discussion is in order.

Simple way would be to not cache the lookup the first time when reply
details are not provided.

It would mean making it return potentially two different values across the
transaction.

 1) based on only request detail to
 and other on request+reply details. decide if a range request to possible.
and then
2) based on additional reply details to see if the abort could be done.

No problem if the reply details cause an increase in the limit. But if they
restrict it we enter grounds of potentially making a request then canceling
it and being unable to store the results.


Or, taking the maximum of the two across two calls? so it can only increase.
 would be slightly trickier involving a flag a well to short-circuit the
reply lookups instead of just a magic cache value.

Am I seriously over-thinking things today?


Amos


Here's a question, too: is this feature going to benefit anyone?  I
realized later that it will not solve my problem, because all the
traffic that was getting force downloaded ended up being from windows
updates.  The urls showing up in netstat and such were just weird
because the windows update traffic was actually coming from limelight.
 My ultimate solution was to write a script that reads access.log,
checks for windows update urls that are not cached, and manually
download them one at a time after hours.

If there is anyone at all who would benefit from this I would still be
*more* than glad to code it (as I said, it would be my first real open
source contribution...very exciting), but I just wondered if anyone
will actually use it.


I believe people will find more control here useful.

Windows update service packs are a big reason, but there are also 
similar range issues with Adobe Reader online PDFs, google maps/earth, 
and flash videos when paused/resumed. Potentially other stuff, but I 
have not heard of problems.


This will allow anyone to fine tune the places where ranges are 
permitted or forced to fully cache. Avoiding the problems a blanket 
limit adds.




As to which approach would be better, I don't know enough about that
data path to really suggest.  When I initially made my changes, I just
replaced each reference to Config.range_offset_limit or whatever.
Today I went back and read some more of the code, but I'm still
figuring it out.  How often would the limit change based on the
request vs. the reply?


Just the once. On first time being checked for the reply.
And most likely on the case of testing for a reply mime type. The other 
useful info I can think of are all request data.


You can ignore if you like. I'm just worrying over a borderline case.
Someone else can code a fix if they find it a problem or need to do mime 
checks.


Amos
--
Please be using
  Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20
  Current Beta Squid 3.1.0.15

Build failed in Hudson: 3.HEAD-i386-opensolaris-SunStudioCc #41

[noc]

See 


Changes:

[Henrik Nordstrom ] Bug #2155: Assertion failures 
on malformed Content-Range response headers

A number of conditions with malformed Content-Range headers were
not trapped properly triggering odd conditions in the code.

[Henrik Nordstrom ] Remove libTrie C unit test from 
the Makefile.

Automake do not like having rules for unreferenced binaries.

--
[...truncated 21297 lines...]
Making dvi in repl
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in auth
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in ip
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in icmp
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in ident
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in log
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in adaptation
make[3]: Entering directory 
`
Making dvi in icap
make[4]: Entering directory 
`
make[4]: Nothing to be done for `dvi'.
make[4]: Leaving directory 
`
make[4]: Entering directory 
`
make[4]: Nothing to be done for `dvi-am'.
make[4]: Leaving directory 
`
make[3]: Leaving directory 
`
Making dvi in esi
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi-am'.
make[3]: Leaving directory 
`
make[2]: Leaving directory 
`
Making dvi in icons
make[2]: Entering directory 
`
make[2]: Nothing to be done for `dvi'.
make[2]: Leaving directory 
`

Re: your suggestion for range_offset_limit

[Matthew Morgan]

On Wed, Nov 25, 2009 at 7:09 PM, Amos Jeffries  wrote:
> Matthew Morgan wrote:
>>
>> Sorry it's taking me so long to get this done, but I do have a question.
>>
>> You suggested making getRangeOffsetLimit a member of HttpReply.  There are
>> two places where this method currently needs to be called: one is
>> CheckQuickAbort2() in store_client.cc.  This one will be easy, as I can just
>> do entry->getReply()->getRangeOffsetLimit().
>>
>> The other is HttpStateData::decideIfWeDoRanges in http.cc.  Here, all we
>> have access to is an HttpRequest object.  I looked through the source to see
>> if I could find where a request owned or had access to a reply, but I don't
>> see anything like that.  If getRangeOffsetLimit were a member of HttpReply,
>> what do you suggest doing here?  I could make a static version of the
>> method, but that wouldn't allow caching the result.
>
> Ah. I see. Quite right.
>
> After a bit more though I find my original request a bit weird.
>
> Yes it should be a _Request_ member and do its caching there. You can go
> ahead with that now while we discuss whether to do a slight tweak on top of
> the basic feature.
>
>
> [cc'ing squid-dev so others can provide input]
>
> I'm not certain of the behavior we want here if we do open the ACLs to reply
> details. Some discussion is in order.
>
> Simple way would be to not cache the lookup the first time when reply
> details are not provided.
>
> It would mean making it return potentially two different values across the
> transaction.
>
>  1) based on only request detail to
>  and other on request+reply details. decide if a range request to possible.
> and then
> 2) based on additional reply details to see if the abort could be done.
>
> No problem if the reply details cause an increase in the limit. But if they
> restrict it we enter grounds of potentially making a request then canceling
> it and being unable to store the results.
>
>
> Or, taking the maximum of the two across two calls? so it can only increase.
>  would be slightly trickier involving a flag a well to short-circuit the
> reply lookups instead of just a magic cache value.
>
> Am I seriously over-thinking things today?
>
>
> Amos
> --
> Please be using
>  Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20
>  Current Beta Squid 3.1.0.15
>

Here's a question, too: is this feature going to benefit anyone?  I
realized later that it will not solve my problem, because all the
traffic that was getting force downloaded ended up being from windows
updates.  The urls showing up in netstat and such were just weird
because the windows update traffic was actually coming from limelight.
 My ultimate solution was to write a script that reads access.log,
checks for windows update urls that are not cached, and manually
download them one at a time after hours.

If there is anyone at all who would benefit from this I would still be
*more* than glad to code it (as I said, it would be my first real open
source contribution...very exciting), but I just wondered if anyone
will actually use it.

As to which approach would be better, I don't know enough about that
data path to really suggest.  When I initially made my changes, I just
replaced each reference to Config.range_offset_limit or whatever.
Today I went back and read some more of the code, but I'm still
figuring it out.  How often would the limit change based on the
request vs. the reply?

Build failed in Hudson: 3.HEAD-sparc-opensolaris-SunStudioCc #19

[noc]

See 


Changes:

[Henrik Nordstrom ] Remove libTrie C unit test from 
the Makefile.

Automake do not like having rules for unreferenced binaries.

[Francesco Chemolli ] Fixed some variable shadowing.

--
[...truncated 21297 lines...]
Making dvi in repl
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in auth
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in ip
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in icmp
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in ident
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in log
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in adaptation
make[3]: Entering directory 
`
Making dvi in icap
make[4]: Entering directory 
`
make[4]: Nothing to be done for `dvi'.
make[4]: Leaving directory 
`
make[4]: Entering directory 
`
make[4]: Nothing to be done for `dvi-am'.
make[4]: Leaving directory 
`
make[3]: Leaving directory 
`
Making dvi in esi
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi-am'.
make[3]: Leaving directory 
`
make[2]: Leaving directory 
`
Making dvi in icons
make[2]: Entering directory 
`
make[2]: Nothing to be done for `dvi'.
make[2]: Leaving directory 
`
Making dvi in errors
make[2]: Entering directory 
`

Build failed in Hudson: 3.HEAD-i386-opensolaris-SunStudioCc #40

[noc]

See 


Changes:

[Henrik Nordstrom ] Remove libTrie C unit test from 
the Makefile.

Automake do not like having rules for unreferenced binaries.

[Francesco Chemolli ] Fixed some variable shadowing.

--
[...truncated 21297 lines...]
Making dvi in repl
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in auth
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in ip
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in icmp
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in ident
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in log
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in adaptation
make[3]: Entering directory 
`
Making dvi in icap
make[4]: Entering directory 
`
make[4]: Nothing to be done for `dvi'.
make[4]: Leaving directory 
`
make[4]: Entering directory 
`
make[4]: Nothing to be done for `dvi-am'.
make[4]: Leaving directory 
`
make[3]: Leaving directory 
`
Making dvi in esi
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi-am'.
make[3]: Leaving directory 
`
make[2]: Leaving directory 
`
Making dvi in icons
make[2]: Entering directory 
`
make[2]: Nothing to be done for `dvi'.
make[2]: Leaving directory 
`
Making dvi in errors
make[2]: Entering directory 
`

Re: your suggestion for range_offset_limit

[Amos Jeffries]

Matthew Morgan wrote:

Sorry it's taking me so long to get this done, but I do have a question.

You suggested making getRangeOffsetLimit a member of HttpReply.  There 
are two places where this method currently needs to be called: one is 
CheckQuickAbort2() in store_client.cc.  This one will be easy, as I can 
just do entry->getReply()->getRangeOffsetLimit().


The other is HttpStateData::decideIfWeDoRanges in http.cc.  Here, all we 
have access to is an HttpRequest object.  I looked through the source to 
see if I could find where a request owned or had access to a reply, but 
I don't see anything like that.  If getRangeOffsetLimit were a member of 
HttpReply, what do you suggest doing here?  I could make a static 
version of the method, but that wouldn't allow caching the result.


Ah. I see. Quite right.

After a bit more though I find my original request a bit weird.

Yes it should be a _Request_ member and do its caching there. You can go 
ahead with that now while we discuss whether to do a slight tweak on top 
of the basic feature.



[cc'ing squid-dev so others can provide input]

I'm not certain of the behavior we want here if we do open the ACLs to 
reply details. Some discussion is in order.


Simple way would be to not cache the lookup the first time when reply 
details are not provided.


It would mean making it return potentially two different values across 
the transaction.


 1) based on only request detail to
 and other on request+reply details. decide if a range request to possible.
and then
2) based on additional reply details to see if the abort could be done.

No problem if the reply details cause an increase in the limit. But if 
they restrict it we enter grounds of potentially making a request then 
canceling it and being unable to store the results.



Or, taking the maximum of the two across two calls? so it can only increase.
 would be slightly trickier involving a flag a well to short-circuit 
the reply lookups instead of just a magic cache value.


Am I seriously over-thinking things today?


Amos
--
Please be using
  Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20
  Current Beta Squid 3.1.0.15

Build failed in Hudson: 3.HEAD-sparc-opensolaris-SunStudioCc #18

[noc]

See 


Changes:

[Francesco Chemolli ] Fixed some variable shadowing.

[Amos Jeffries ] Fix detection of obsolete 'all' patterns

A little bit overly enthusiastic on the additions...
* 0.0.0.0 is a /32 not /0
* 0->N is 'all', not 0->0

--
[...truncated 21303 lines...]
Making dvi in repl
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in auth
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in ip
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in icmp
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in ident
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in log
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in adaptation
make[3]: Entering directory 
`
Making dvi in icap
make[4]: Entering directory 
`
make[4]: Nothing to be done for `dvi'.
make[4]: Leaving directory 
`
make[4]: Entering directory 
`
make[4]: Nothing to be done for `dvi-am'.
make[4]: Leaving directory 
`
make[3]: Leaving directory 
`
Making dvi in esi
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi-am'.
make[3]: Leaving directory 
`
make[2]: Leaving directory 
`
Making dvi in icons
make[2]: Entering directory 
`
make[2]: Nothing to be done for `dvi'.
make[2]: Leaving directory 
`
Making dvi in errors
ma

Build failed in Hudson: 3.HEAD-i386-opensolaris-SunStudioCc #39

[noc]

See 


Changes:

[Francesco Chemolli ] Fixed some variable shadowing.

[Amos Jeffries ] Fix detection of obsolete 'all' patterns

A little bit overly enthusiastic on the additions...
* 0.0.0.0 is a /32 not /0
* 0->N is 'all', not 0->0

--
[...truncated 21303 lines...]
Making dvi in repl
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in auth
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in ip
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in icmp
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in ident
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in log
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
Making dvi in adaptation
make[3]: Entering directory 
`
Making dvi in icap
make[4]: Entering directory 
`
make[4]: Nothing to be done for `dvi'.
make[4]: Leaving directory 
`
make[4]: Entering directory 
`
make[4]: Nothing to be done for `dvi-am'.
make[4]: Leaving directory 
`
make[3]: Leaving directory 
`
Making dvi in esi
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi'.
make[3]: Leaving directory 
`
make[3]: Entering directory 
`
make[3]: Nothing to be done for `dvi-am'.
make[3]: Leaving directory 
`
make[2]: Leaving directory 
`
Making dvi in icons
make[2]: Entering directory 
`
make[2]: Nothing to be done for `dvi'.
make[2]: Leaving directory 
`
Making dvi in errors
make[2]: Entering directory 

R: obsoleting nextstep?

[Guido Serassio]

Hi,

> -Messaggio originale-
> Da: Kinkie [mailto:gkin...@gmail.com]
> Inviato: mercoledì 25 novembre 2009 12.50
> A: Squid Developers
> Oggetto: RFC: obsoleting nextstep?
> 
> Hi all,
>just like SunOS: NextStep's last version (3.3) was released in
> 1995, which means 15 years before the expected release date of 3.2 .
> How about dropping support for it?
> 

+1

And there are some other candidate:
- *m88k*, I think the Motorola 88000 systems are no more available ...
- alpha-dec-osf4.*, is the very old DEC OSF 4.x, while latest is HP True64 5.x 
(alpha-dec-osf5.*)
- alpha-dec-osf3 .

And to be investigated:
- powerpc-ibm-aix4.1.*
- *-pc-sco3.2*
- *-sony-newsos[56]*
 
Regards

Guido


Guido Serassio
Acme Consulting S.r.l.
Microsoft Gold Certified Partner
VMware Professional Partner
Via Lucia Savarino, 110098 - Rivoli (TO) - ITALY
Tel. : +39.011.9530135   Fax. : +39.011.9781115
Email: guido.seras...@acmeconsulting.it
WWW: http://www.acmeconsulting.it

Re: RFC: obsoleting nextstep?

[Kinkie]

On Wed, Nov 25, 2009 at 1:52 PM, Henrik Nordstrom
 wrote:
> ons 2009-11-25 klockan 12:49 +0100 skrev Kinkie:
>> Hi all,
>>    just like SunOS: NextStep's last version (3.3) was released in
>> 1995, which means 15 years before the expected release date of 3.2 .
>> How about dropping support for it?
>
> +1 I think. Or just ignore it..

There's messy bits in configure.in about it. I'd like to remove them.
Plus some not-so-messy bits scattered in the code.


-- 
/kinkie

Re: RFC: obsoleting nextstep?

[Henrik Nordstrom]

ons 2009-11-25 klockan 12:49 +0100 skrev Kinkie:
> Hi all,
>just like SunOS: NextStep's last version (3.3) was released in
> 1995, which means 15 years before the expected release date of 3.2 .
> How about dropping support for it?

+1 I think. Or just ignore it..

Regards
Henrik

RFC: obsoleting nextstep?

[Kinkie]

Hi all,
   just like SunOS: NextStep's last version (3.3) was released in
1995, which means 15 years before the expected release date of 3.2 .
How about dropping support for it?



-- 
/kinkie