Re: [squid-dev] Compiling from repository checkout

[Kinkie]

Hi Mika,
  sorry for the delayed answer.

> Thanks, got it. One more question. Do you know why bootstrap.sh verifies
> that glibtool is installed in OS X. Cause I tried this
>
> brew install libtool
> ./bootstrap.sh
> brew uninstall libtool
> ./configure && make
>
> And squid builds (and runs) just fine. But If I do
>
> brew uninstall libtool
> ./bootstrap.sh
>
> Then bootstrapping fails saying glibtool is not installed.

What MacOS calls 'libtool' is an entirely different beast than GNU
libtool. Just try 'man libtool' to see what I mean.
So unless you have glibtool installed, well, it is't.

> The reason I am asking is that if I don’t uninstall glibtool, the final
> binary will have dependency to dynamic glibtool lib.  The glibtool is
> installed to Homebrew default path (/usr/local/..) and doesn’t necessarily
> exists in other OS X installations. So if I want to give the binary I build
> to my coworker it may not work in his machine unless he installs glibtool
> first through homebrew.

If you want to have a fully portable binary I suspect that the only
solution is to link it statically, sorry.
And yes, that's unavoidable, sadly: libtool is a required component to
build squid.


-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] Compiling from repository checkout

[Kinkie]

On Mon, Dec 18, 2017 at 7:34 AM, mika ristimaki
 wrote:
> Hi all,

Hello Mika!

> I noticed that the source code distributions
> (http://www.squid-cache.org/Versions/v4/) and the repository
> (https://github.com/squid-cache/squid) differ in a sense that the tar.gz
> packages include the configure script and Makefiles.
>
>
> I am trying to replicate it, and I it looks to me that the configure
> scripts, etc., are created by running ./bootstrap.sh. But looking at the
> bootstrap.sh it has this
>
>
> # On MAC OS X, GNU libtool is named 'glibtool':
>
> if [ `uname -s 2>/dev/null` = 'Darwin' ]
>
> then
>
>   LIBTOOL_BIN="glibtool"
>
> else
>
>   LIBTOOL_BIN="libtool”
>
> fi
>
>
> So it is platform dependant.

Yes.

> So my question is how platform independent configure scripts are created if
> bootstrap.sh is platform dependant? Or what I am missing here?

Exactly through bootstrap.sh :)
The full build process for Squid-from-source is
$ bootstrap.sh && ./configure && make

This is because bootstrap builds auto-generated code (e.g. configure)
which makes no sense to have in our source repository.
At the same time, building from source requires additional tools (e.g.
autoconf, automake, libtoolize) which may not be present on an user's
build system, so these generated bits are included in the distribution
tarball.


-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] Stuck jenkins jobs

[Kinkie]

I believe so, let’s see in practice

On Thu, 30 Nov 2017 at 16:01, Alex Rousskov <
rouss...@measurement-factory.com> wrote:

> On 11/30/2017 03:12 AM, Kinkie wrote:
>
> >   I've added the build-timeout plugin to jenkins; it'll abort
> > pull-request validation jobs that take "several times their usual
> > completion times". Let's see if it helps unstick the queue.
>
> Thank you. Will these automated aborts be logged with the right actor
> and sufficient detail to understand what happened?
>
> Alex.
>
-- 
@mobile
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] Stuck jenkins jobs

[Kinkie]

Hi,
  I've added the build-timeout plugin to jenkins; it'll abort
pull-request validation jobs that take "several times their usual
completion times". Let's see if it helps unstick the queue.

-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] State of things and stuff

[Kinkie]

Ok thanks!
Do we have a map of the missing bits? Maybe I could help there

On Thu, 23 Nov 2017 at 13:10, Amos Jeffries  wrote:

> As some of you know since the git transition I've had a few issues with
> converting the maintainer scripts which has been the roadblock on our
> regular release schedule.
>
> The biggest of those issues are now resolved, and the remainder can go
> back to being a manual process for now.
>
> So I am planning to do a push on the outstanding backports over the next
> few days. That should give us next week for testing before a 4.0.22
> release on 2nd or 3rd October.
>
> At this stage I am not planning on any more v3.5 releases, but that is
> still flexible so it is not formally deprecated (yet).
>
> Amos
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev
>
-- 
@mobile
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] wiki.squid-cache.org has an expired certificate

[Kinkie]

Thanks for letting us know.
Now fixed.

On Wed, Nov 8, 2017 at 9:52 AM, Marcus Kool  wrote:
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] Build farm updaes?

[Kinkie]

Yes, I agree.

Eduard, let's align on this. Possibly real time (chat or
Skype/WhatsApp/whatever)

Thanks!

On Sun, 23 Jul 2017 at 23:04, Alex Rousskov <
rouss...@measurement-factory.com> wrote:

> On 07/23/2017 10:21 AM, Kinkie wrote:
>
> > is it worth investing time in freshening up the current build farm
> > setup or is preferable to abandon it altogether in favor of a newly
> > built one?
>
> I would be surprised if we should abandon the current build farm as a
> whole, but perhaps you have some reason to believe it should be done?
>
> Please coordinate with Eduard on this. My short-term expectation is that
> the old build farm nodes will remain 95+% the same while the Jenkin's
> configuration will change to integrate with Github. I am worried that if
> both of you modify things independently, it would be difficult to bring
> everything back under one roof.
>
>
> > If the former, I can start investing some time in that, removing
> > obsolete nodes and adding newer ones, etc.
>
> Removing obsolete/broken nodes is fine, I guess. See above regarding
> adding new ones.
>
>
> Thank you,
>
> Alex.
>
-- 
@mobile
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] Build farm updaes?

[Kinkie]

Hi all,
  is it worth investing time in freshening up the current build farm
setup or is preferable to abandon it altogether in favor of a newly
built one?
  If the former, I can start investing some time in that, removing
obsolete nodes and adding newer ones, etc.

-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [RFC] Disable Github issue tracker

[Kinkie]

I also agree to close Issues for the time being. We can always reopen
them later on if it'll be deemed useful.

On Fri, Jul 21, 2017 at 5:37 PM, Alex Rousskov
 wrote:
> On 07/21/2017 10:15 AM, Amos Jeffries wrote:
>> Alex would you like to draw up a formal announcement email to go out to
>> people not on squid-dev about the change having been done?
>>  I'm thinking squid-announce/squid-users and the blog.
>
> I can, but I do not recommend announcing anything and posting README.md
> until we decide on the Github Issues status. Both of those documents
> (and user actions) would be affected by this important migration-related
> decision.
>
> So far, only three people voiced their opinions:
>
>   + For immediate Issues closure: Amos, Alex.
>   - Against immediate Issues closure: Eliezer.
>
> While the current tally is technically enough for me to put my foot down
> and disable Github Issues, it would be nice to have at least one more
> supporting opinion for a more meaningful "consensus" declaration.
>
> Alex.
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] What should we do about these *wrong* wiki articles?

[Kinkie]

Everyone's invited to improve the contents in any way they see
reasonable. Please, just go ahead :)

On Fri, Jul 21, 2017 at 10:17 AM, Eliezer Croitoru  wrote:
> Hey List,
>
> I have seen that these articles aren't up-to-date and are misleading admins.
> The first step to my opinion would be to add a warning at the top of the
> articles that these are  obsolete and should not be used.
> Then fix the article content and redirect toward PBR\FBF\Other routing to
> the squid box example and eventually removing these examples from the wiki.
>
> http://wiki.squid-cache.org/ConfigExamples/Intercept/LinuxDnat?highlight=%28
> masquerade%29
> http://wiki.squid-cache.org/ConfigExamples/Intercept/LinuxRedirect?highlight
> =%28masquerade%29
>
> What do you think?
>
> Eliezer
>
> 
> Eliezer Croitoru
> Linux System Administrator
> Mobile: +972-5-28704261
> Email: elie...@ngtech.co.il
>
>
>
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] Bzr to git migration schedule

[Kinkie]

I'll try to look into it over the weekend.

On Fri, Jul 21, 2017 at 7:30 AM, Eliezer Croitoru <elie...@ngtech.co.il> wrote:
> NOC NOC, anybody there?
>
> Please take a minute to look at the subject.
>
> Thanks,
> Eilezer
> 
> Eliezer Croitoru
> Linux System Administrator
> Mobile: +972-5-28704261
> Email: elie...@ngtech.co.il
>
>
>
> -Original Message-
> From: Alex Rousskov [mailto:rouss...@measurement-factory.com]
> Sent: Friday, July 21, 2017 08:02
> To: Eliezer Croitoru <elie...@ngtech.co.il>; 'Kinkie' <gkin...@gmail.com>; 
> squid-dev@lists.squid-cache.org
> Subject: Re: [squid-dev] Bzr to git migration schedule
>
> On 07/20/2017 02:52 PM, Eliezer Croitoru wrote:
>> And... I must say that I am experiencing something weird with the wiki in 
>> the last month or more.
>> I mean that for example I received the message in the top left side of the 
>> wiki:
>> "Thank you for your changes. Your attention to detail is appreciated.
>> Only minutes after I already seen the page exists in another tab.
>> Should I open a new bug in the bugzilla?
>
> I have already whined to NOC about this problem. They were going to look at 
> it soon. If you do not see improvements for another week or so, it may be a 
> good idea to file a bug report with Bugzilla.
>
> Alex.
>
>
>> From: Kinkie [mailto:gkin...@gmail.com]
>> Sent: Thursday, July 20, 2017 10:59
>> To: Amos Jeffries <squ...@treenet.co.nz>; Eliezer Croitoru
>> <elie...@ngtech.co.il>; squid-dev@lists.squid-cache.org
>> Subject: Re: [squid-dev] Bzr to git migration schedule
>>
>> I believe we are talking about updating the pages that reference bzr
>> with pages that reference the current repositories and best practices
>>
>> On Thu, 20 Jul 2017 at 08:27, Eliezer Croitoru <mailto:elie...@ngtech.co.il> 
>> wrote:
>> I didn't knew we are trying to "re-write" the wiki.
>> To what scale are we talking?
>> Mover everything from the http://wiki.squid-cache.org to github or just 
>> specific articles?
>>
>> Eliezer
>>
>> 
>> Eliezer Croitoru
>> Linux System Administrator
>> Mobile: +972-5-28704261
>> Email: mailto:elie...@ngtech.co.il
>>
>>
>>
>> -Original Message-
>> From: squid-dev
>> [mailto:mailto:squid-dev-boun...@lists.squid-cache.org] On Behalf Of
>> Amos Jeffries
>> Sent: Tuesday, July 18, 2017 11:59
>> To: mailto:squid-dev@lists.squid-cache.org
>> Subject: Re: [squid-dev] Bzr to git migration schedule
>>
>> On 18/07/17 16:40, Alex Rousskov wrote:
>>> On 07/15/2017 10:43 PM, Alex Rousskov wrote:
>>>> On 07/11/2017 10:20 PM, Alex Rousskov wrote:
>>>>
>>>>>2017-07-11: No more new tags in the official bzr repo.
>>>>>2017-07-13: No more new commits(*) in the official bzr repo.
>>>>>2017-07-14: Migration starts.
>>>>>2017-07-15: Anticipated optimistic migration end.
>>>>>2017-07-18: Anticipated pessimistic migration end.
>>>>>
>>>>> All times are noon UTC.
>>>
>>>
>>>> The migration steps are done. According to the automated tests, all
>>>> bzr and git commits match (except for bzr commits containing empty
>>>> directories, as expected). However, I suggest _not_ declaring the
>>>> migration over yet and keeping both the old bzr and the new git
>>>> repository intact for 24 hours in case somebody notices problems
>>>> with the new official git repository at
>>>>
>>>>  https://github.com/squid-cache/squid
>>>
>>> I think we should declare the migration over, and the official
>>> repository to be at the above Github URL.
>>
>> +1.
>>
>>>
>>> I saw no problem reports about the new repository. We found and fixed
>>> one bug in the migration tool when migrating some of the Factory
>>> branches, but that bug did not affect the official repository which
>>> has a simpler structure. We could still go back to bzr in the face of
>>> disaster, but I hope that would not be necessary.
>>>
>>> Needless to say, this git migration is just the first step towards
>>> decent Project QA. The next steps are updating various maintenance
>>> scripts (thank you, Amos, for working on that) and pre-commit build
>>> tests of pull requests (Eduard is leading that effort). The basic
>>> build test enabled on Github should detect many serious problems
>>> today, but I hope to see a lot more powerful Jenkins-driven tests in the 
>>> coming weeks.
>>>
>>
>> We also need someone to lead the wiki re-writing, most of the
>> DeveloperResources mentioning the repository and how-to for certain
>> development activities.
>>
>>
>> Amos
>> ___
>> squid-dev mailing list
>> mailto:squid-dev@lists.squid-cache.org
>> http://lists.squid-cache.org/listinfo/squid-dev
>>
>> ___
>> squid-dev mailing list
>> mailto:squid-dev@lists.squid-cache.org
>> http://lists.squid-cache.org/listinfo/squid-dev
>>
>
>



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] Bzr to git migration schedule

[Kinkie]

I believe we are talking about updating the pages that reference bzr with
pages that reference the current repositories and best practices

On Thu, 20 Jul 2017 at 08:27, Eliezer Croitoru  wrote:

> I didn't knew we are trying to "re-write" the wiki.
> To what scale are we talking?
> Mover everything from the wiki.squid-cache.org to github or just specific
> articles?
>
> Eliezer
>
> 
> Eliezer Croitoru
> Linux System Administrator
> Mobile: +972-5-28704261
> Email: elie...@ngtech.co.il
>
>
>
> -Original Message-
> From: squid-dev [mailto:squid-dev-boun...@lists.squid-cache.org] On
> Behalf Of Amos Jeffries
> Sent: Tuesday, July 18, 2017 11:59
> To: squid-dev@lists.squid-cache.org
> Subject: Re: [squid-dev] Bzr to git migration schedule
>
> On 18/07/17 16:40, Alex Rousskov wrote:
> > On 07/15/2017 10:43 PM, Alex Rousskov wrote:
> >> On 07/11/2017 10:20 PM, Alex Rousskov wrote:
> >>
> >>>2017-07-11: No more new tags in the official bzr repo.
> >>>2017-07-13: No more new commits(*) in the official bzr repo.
> >>>2017-07-14: Migration starts.
> >>>2017-07-15: Anticipated optimistic migration end.
> >>>2017-07-18: Anticipated pessimistic migration end.
> >>>
> >>> All times are noon UTC.
> >
> >
> >> The migration steps are done. According to the automated tests, all bzr
> >> and git commits match (except for bzr commits containing empty
> >> directories, as expected). However, I suggest _not_ declaring the
> >> migration over yet and keeping both the old bzr and the new git
> >> repository intact for 24 hours in case somebody notices problems with
> >> the new official git repository at
> >>
> >>  https://github.com/squid-cache/squid
> >
> > I think we should declare the migration over, and the official
> > repository to be at the above Github URL.
>
> +1.
>
> >
> > I saw no problem reports about the new repository. We found and fixed
> > one bug in the migration tool when migrating some of the Factory
> > branches, but that bug did not affect the official repository which has
> > a simpler structure. We could still go back to bzr in the face of
> > disaster, but I hope that would not be necessary.
> >
> > Needless to say, this git migration is just the first step towards
> > decent Project QA. The next steps are updating various maintenance
> > scripts (thank you, Amos, for working on that) and pre-commit build
> > tests of pull requests (Eduard is leading that effort). The basic build
> > test enabled on Github should detect many serious problems today, but I
> > hope to see a lot more powerful Jenkins-driven tests in the coming weeks.
> >
>
> We also need someone to lead the wiki re-writing, most of the
> DeveloperResources mentioning the repository and how-to for certain
> development activities.
>
>
> Amos
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev
>
-- 
@mobile
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] Bzr to git migration schedule

[Kinkie]

On Tue, Jul 18, 2017 at 9:36 PM, Alex Rousskov
 wrote:
> On 07/18/2017 02:59 AM, Amos Jeffries wrote:
>> We also need someone to lead the wiki re-writing, most of the
>> DeveloperResources mentioning the repository and how-to for certain
>> development activities.
>
> Agreed.
>
> BTW, I trust it is OK to delete bzr/launchpad-specific instructions when
> they are replaced with git/github-specific ones, but please correct me
> if I am wrong. The wiki history will preserve bzr instructions for
> posterity, of course.

I agree on all.

-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] fix escaping errors in manual pages causing information loss

[Kinkie]

Hi Ingo,
  I've merged your changes as Squid-5 revno 15135.
Thanks for your contribution!

On Thu, May 11, 2017 at 6:22 PM, Ingo Schwarze  wrote:
> Hi,
>
> while auditing manual pages in the OpenBSD ports tree, i noticed
> serious escaping errors in four of the squid manuals that cause
> information loss, in most cases loss of important information in
> the SYNOPSIS.
>
> The following patch against bazaar trunk fixes those issues.
>
> Stuart Henderson, who is maintaining the OpenBSD port of squid,
> advised that sending the patch to this list would be appropriate.
>
>
> The reasons why the escaping is wrong are:
>
>  1) In the roff language, \" introduces a comment,
> hence the rest of the input line is discarded by
> all formatters.
>
>  2) In the roff language, \\ does *not* produce a printable
> backslash but merely delays expansion of the escape sequence
> (which is practically never what you want, unless you are
> programming a new macro package).  Consequently, "\\n'" ends
> up being interpreted as an access to the number register with
> the name "'" (yes, roff does allow identifiers consisting of
> special characters in some contexts), which doesn't exist and
> hence yields the output string "0".
>
> Thanks for developing squid!
>
> Yours,
>   Ingo
>
> P.S.
> I'm not attempting to fix all manual page syntax errors that cause
> misformatting (that would cause a very serious amount of work), but
> only those that cause information loss.
>
> --
> Ingo Schwarze 
> http://www.openbsd.org/   
> http://mdocml.bsd.lv/ 
>
>
> === modified file 'src/acl/external/file_userip/ext_file_userip_acl.8'
> --- src/acl/external/file_userip/ext_file_userip_acl.8  2017-01-01 00:14:42 
> +
> +++ src/acl/external/file_userip/ext_file_userip_acl.8  2017-05-11 16:49:06 
> +
> @@ -68,7 +68,7 @@
>  .B ALL
>  and
>  .B NONE
> -, which mean \"any user on this IP address may authenticate\" or \"no user 
> on this IP address may authenticate\".
> +, which mean \(dqany user on this IP address may authenticate\(dq or \(dqno 
> user on this IP address may authenticate\(dq.
>  .
>  .SH AUTHOR
>  This program was written by
>
> === modified file 'src/auth/basic/LDAP/basic_ldap_auth.8'
> --- src/auth/basic/LDAP/basic_ldap_auth.8   2017-03-31 08:57:46 +
> +++ src/auth/basic/LDAP/basic_ldap_auth.8   2017-05-11 16:44:30 +
> @@ -5,9 +5,9 @@
>  .
>  .SH SYNOPSIS
>  .if !'po4a'hide' .B basic_ldap_auth
> -.if !'po4a'hide' .B \-b\ \"
> +.if !'po4a'hide' .B \-b\ \(dq
>  base DN
> -.if !'po4a'hide' .B \"\ [\-u
> +.if !'po4a'hide' .B \(dq\ [\-u
>  attribute
>  .if !'po4a'hide' .B ]\ [
>  options
> @@ -20,11 +20,11 @@
>  .if !'po4a'hide' .B ]...
>  .br
>  .if !'po4a'hide' .B basic_ldap_auth
> -.if !'po4a'hide' .B \-b\ \"
> +.if !'po4a'hide' .B \-b\ \(dq
>  base DN
> -.if !'po4a'hide' .B \"\ \-f\ \"
> +.if !'po4a'hide' .B \(dq\ \-f\ \(dq
>  LDAP search filter
> -.if !'po4a'hide' .B \"\ [
> +.if !'po4a'hide' .B \(dq\ [
>  options
>  .if !'po4a'hide' .B ]\ [
>  LDAP server name
> @@ -74,7 +74,7 @@
>  The search filter can contain up to 15 occurrences of
>  .B %s
>  which will be replaced by the username, as in
> -.B "\"uid\=%s\""
> +.B "\(dquid\=%s\(dq"
>  for RFC2037 directories. For a detailed description of LDAP search
>  filter syntax see RFC2254.
>  .br
>
> === modified file 'src/auth/basic/RADIUS/basic_radius_auth.8'
> --- src/auth/basic/RADIUS/basic_radius_auth.8   2017-01-01 00:14:42 +
> +++ src/auth/basic/RADIUS/basic_radius_auth.8   2017-05-11 16:47:05 +
> @@ -9,9 +9,9 @@
>  config file
>  .br
>  .if !'po4a'hide' .B basic_radius_auth
> -.if !'po4a'hide' .B "\-h \""
> +.if !'po4a'hide' .B "\-h \(dq"
>  server name
> -.if !'po4a'hide' .B "\" [\-p "
> +.if !'po4a'hide' .B "\(dq [\-p "
>  port
>  .if !'po4a'hide' .B "] [\-i "
>  identifier
>
> === modified file 'tools/squidclient/squidclient.1'
> --- tools/squidclient/squidclient.1 2017-01-01 00:14:42 +
> +++ tools/squidclient/squidclient.1 2017-05-11 16:51:31 +
> @@ -86,7 +86,7 @@
>  .if !'po4a'hide' .TP
>  .if !'po4a'hide' .B "\-H 'string'"
>  Extra headers to send. Use
> -.B '\\n'
> +.B '\en'
>  for new lines.
>  .
>  .if !'po4a'hide' .TP
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] SBuf header names

[Kinkie]

What's with the comment

// TODO: this is probably incorrect, for now it should match w_space

?
Apart from this, +1 from me.

On Sun, Apr 23, 2017 at 9:56 PM, Amos Jeffries  wrote:
> This patch replaces String with SBuf for storage of header names in
> HttpHeaderEntry.
>
> In the process we gain SBuf methods for delByName and hasNamed. Deprecating
> the char*  versions.
>
>
>
>
> Amos
>
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev
>



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] configure.ac cleanup of BerkleyDB related checks

[Kinkie]

+1.

On Mon, Mar 27, 2017 at 2:57 PM, Amos Jeffries  wrote:
> While looking into the last remaining bits of bug 4610 I have found that
> most of what we were doing for libdb / -ldb is not necessary any longer.
>
> Most of the logic seems to be hangovers from when session helper was
> using the BerkleyDB v1.85 compatibility interface. Some of it is
> possibly still necessary for the time_quota helper, but that helper has
> not been using it so far and needs an upgrade to match what happened to
> session helper.
>
> Changes:
>
> * The helpers needing -ldb will not be built unless the library and
> headers are available. So we can drop the Makefile LIB_DB substitutions
> and always just link -ldb explicitly to these helpers.
>
> NP: Anyone who needs small minimal binaries, can build with the
> --as-needed linker flag, or without these helpers. This change has no
> effect on other helpers or the main squid binary.
>
> * Since we no longer need to check if -ldb is necessary, we can drop the
> configure.ac and acinclude logic detecting that.
>
> * Remove unused AC_CHECK_DECL(dbopen, ...)
>  - resolves one "FIXME"
>
> * Fix the time_quota helper check to only scan db.h header file contents
> if that file is existing, and if the db_185.h file is not being used
> instead.
>
> * Fix the session helper check to only try compiling with the db.h
> header if that header actually exists.
>
> * De-duplicate the library header file detection shared by configure.ac
> and the helpers required.m4 files (after the above two changes).
>
> * Remove unused DBLIB variable from configure.ac.
>
> Amos
>
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev
>



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Case-insensitive URI schemes

[Kinkie]

> I'm thinking the quick-and-dirty way is to just lowercase the 'proto'
> variable in url.cc urlParse() function. Doing that in the for-loop where
> it is copied from 'src' would be easiest.
>  - it breaks the case preservation on unknown schemes a litte bit. But
> since they are supposed to be insensitive anyway the harm is minimal.

I think it'd be a worthy experiment.


-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] remove USE_CHUNKEDMEMPOOLS

[Kinkie]

Please go ahead when it's convenient for you.

On Wed, Jan 11, 2017 at 4:00 PM, Amos Jeffries  wrote:
> On 12/01/2017 4:46 a.m., Alex Rousskov wrote:
>> On 01/11/2017 08:29 AM, Amos Jeffries wrote:
>>> On 1/01/2017 6:37 p.m., Amos Jeffries wrote:
 The USE_CHUNKEDMEMPOOLS build-time setting is not very useful and adds
 extra complexity to the build system. Even when set it does not always
 enable chunked pools. The environment variable MEMPOOLS can easily be
 used to enable or disable chunked pools as needed.

>>>
>>> If there are no objections I would like to commit this shortly.
>>
>> None from me, but I did not study the patch closely. FWIW, adjusting the
>> commit message to explicitly say whether this change alters any
>> defaults, may help those assessing the change impact on their setups.
>>
>>
>> Thank you,
>>
>> Alex.
>>
>
> It does not change any defaults or behaviour for the vast majority of
> Squid installs. But anyone who was building custom with
> CXXFLAGS="-DUSE_CHUNKEDMEMPOOLS=1" will now have to use the run-time
> environment variable.
>
> Amos
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] remove ip/Qos.cci

[Kinkie]

Looks good, except that

+Qos::Config() {STUB}

Surrounding the STUB macro in brackets is not the common style
(although it is harmless and could actually be a good idea)

+1, possibly with this change

On Sat, Jan 7, 2017 at 11:46 AM, Amos Jeffries  wrote:
> This patch removes ip/Qos.cci file moving its content to ip/QosConfig.cc.
>
> Also, move the stub file to src/tests/stub_libip.cc and update to use
> tests/STUB.h interface.
>
> Amos
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev
>



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Compilation fix for v5 r14973

[Kinkie]

+1; please apply.

Thanks!

On Sat, Dec 10, 2016 at 1:05 PM, Eduard Bagdasaryan
 wrote:
> Hello,
>
> Attached a typo fix for r14973 which caused Jenkins errors.
>
> Regards,
> Eduard.
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev
>



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Rework acl/RegexData optimization to use SBufList

[Kinkie]

Ok, done.
Thanks

On Wed, Nov 30, 2016 at 2:02 PM, Kinkie <gkin...@gmail.com> wrote:
> I'll do that tonight, but after the rever of std::regex which it clashes with.
>
> On Wed, Nov 30, 2016 at 12:31 PM, Amos Jeffries <squ...@treenet.co.nz> wrote:
>> On 15/11/2016 5:34 a.m., Kinkie wrote:
>>> Hi,
>>>   the attached patch fixes the issue with missing ACL entries
>>> (verified) and streamlines the code a bit.
>>>
>>> Performance-wise it improves a bit, parsing the same 1M-entry ACL in
>>> 19.4 seconds (17.8 seconds in userland).
>>>
>>
>> +1. Seems to be okay.
>>
>> If there are no objections this can go in asap.
>>
>> Amos
>>
>
>
>
> --
> Francesco



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] ServerBump class cleanup

[Kinkie]

Looks reasonable to me. +1

On Wed, Nov 30, 2016 at 1:34 PM, Amos Jeffries  wrote:
> This patch is a general cleanup of coding styles and current code
> requirements for the ServerBump class.
>
> Is there any other general cleanup that should be done to this class
> while I'm at it ?
>
> Amos
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev
>



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Rework acl/RegexData optimization to use SBufList

[Kinkie]

I'll do that tonight, but after the rever of std::regex which it clashes with.

On Wed, Nov 30, 2016 at 12:31 PM, Amos Jeffries <squ...@treenet.co.nz> wrote:
> On 15/11/2016 5:34 a.m., Kinkie wrote:
>> Hi,
>>   the attached patch fixes the issue with missing ACL entries
>> (verified) and streamlines the code a bit.
>>
>> Performance-wise it improves a bit, parsing the same 1M-entry ACL in
>> 19.4 seconds (17.8 seconds in userland).
>>
>
> +1. Seems to be okay.
>
> If there are no objections this can go in asap.
>
> Amos
>



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Rework acl/RegexData optimization to use SBufList

[Kinkie]

> For the record, I do not plan on reviewing this patch further. I did
> what I could, but there is just too much information missing for me to
> reconstruct the exact intent and evaluate the results. IMHO, I should
> not be parsing large raw dumps to figure out what the patch does or does
> not do. I am sure you and Kinkie do not need me to finish this.

Thanks for helping out so far.


-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Rework acl/RegexData optimization to use SBufList

[Kinkie]

Hi,
  the attached patch fixes the issue with missing ACL entries
(verified) and streamlines the code a bit.

Performance-wise it improves a bit, parsing the same 1M-entry ACL in
19.4 seconds (17.8 seconds in userland).



On Mon, Nov 14, 2016 at 11:04 AM, Kinkie <gkin...@gmail.com> wrote:
> On Mon, Nov 14, 2016 at 5:59 AM, Amos Jeffries <squ...@treenet.co.nz> wrote:
>> On 14/11/2016 6:30 p.m., Alex Rousskov wrote:
>>> On 11/13/2016 05:11 PM, Kinkie wrote:
>>>
>>>> the attached patch moves away from hand-rolling a c-string onto
>>>> joining a SBufList for optimizing regexes in RegexData.cc.
>>>
>>>> You can find attached as a test case the output of squidclient
>>>> mgr:config taken on trunk and on the submitted code. It is slightly
>>>> different in that the new code allows for a longer optimized regex.
>>>
>>> I see the following before/after difference:
>>>
>>>   -acl bigacl dstdom_regex ... (pattern84) (pattern86) ...
>>>   +acl bigacl dstdom_regex ... (pattern84)|(pattern85) ...
>>>
>>> Does changing space into "|" result in one internal regex instead of
>>> two?
>>
>> It does / should.
>>
>>> And that is the optimization you have implemented?
>>
>> No, that was implemented long ago by Marcus. We got a ~20% speed
>> increase from that. More complexity in the individual regex test seemed
>> to counteract the fewer tests done.
>>
>>
>>> How is that
>>> related to SBufs (i.e., why could not old c-strings accommodate longer
>>> regexes)?
>>
>> FYI (to kinkie): The 100 patterns per aggregation was an arbitrary
>> limit, but the total string length is limited by the ConfigParser buffers.
>>
>> The plain-text length of the aggregated regex string needs to fit within
>> one squid.conf line length. With space for the directive name, label,
>> type and flags being included in the limit.
>
> that's BUFSIZ, or 4k. We currently have 1K + separator, name and flags.
> If anything, we could be more aggressive in having longer regexes,
> modulo OOM in regcomp.
>
>>> And why is there no pattern85 in the "before" test?!
>>>
>>>
>>>> The code is expected to cause a small performance regression on
>>>> parse and reconfigure due to extra data copies. The regression is
>>>> negligible: the attached test cases perform "squid -k parse" in 60
>>>> msec in trunk and 62 on the branch on a warm cache on my i5 macbook
>>>> air.
>>>
>>> A 3% performance regression is not necessarily negligible. Have you
>>> tested with more ACL lines (not larger individual ACL lines) that take a
>>> few seconds to load (as opposed to 60ms) ? If not, please do unless that
>>> test would be irrelevant somehow.
>>>
>>> And what is the expected performance improvement from having fewer
>>> longer regexes?
>>
>> I assume you mean performance of normal request processing through the
>> regex ACL. Agreed, that need to be checked/compared to current speeds.
>
> Time needed for squid -k parse on my Mac 1.3GHz i5 macbook air, for a
> 1M-entry random ACL, 13Mb-big:
> new: 20.8s
> trunk: 18.3s
>
> Hopefully in time we'll move the config parser to SBuf..
>
> --
> Francesco



-- 
Francesco


wordlist-sbuflist-aclregexdata-v2.patch
Description: Binary data
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Rework acl/RegexData optimization to use SBufList

[Kinkie]

>> The code managing case-insensitivity flags is IMO quite complicated
>> and not really intuitive: it switches between case-insensitive and
>> case-sensitive each time it meets a -i flag.
>
> It does?! Wow! The documentation says that +i switches back to
> case-sensitive rather than repeating -i values. If Squid does something
> else here, that would be a pretty big bug IMO!

Double-checked. No it doesn't, it uses -i and +i as documented, I was
remembering wrong details.

Kinkie
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Rework acl/RegexData optimization to use SBufList

[Kinkie]

On Mon, Nov 14, 2016 at 5:59 AM, Amos Jeffries <squ...@treenet.co.nz> wrote:
> On 14/11/2016 6:30 p.m., Alex Rousskov wrote:
>> On 11/13/2016 05:11 PM, Kinkie wrote:
>>
>>> the attached patch moves away from hand-rolling a c-string onto
>>> joining a SBufList for optimizing regexes in RegexData.cc.
>>
>>> You can find attached as a test case the output of squidclient
>>> mgr:config taken on trunk and on the submitted code. It is slightly
>>> different in that the new code allows for a longer optimized regex.
>>
>> I see the following before/after difference:
>>
>>   -acl bigacl dstdom_regex ... (pattern84) (pattern86) ...
>>   +acl bigacl dstdom_regex ... (pattern84)|(pattern85) ...
>>
>> Does changing space into "|" result in one internal regex instead of
>> two?
>
> It does / should.
>
>> And that is the optimization you have implemented?
>
> No, that was implemented long ago by Marcus. We got a ~20% speed
> increase from that. More complexity in the individual regex test seemed
> to counteract the fewer tests done.
>
>
>> How is that
>> related to SBufs (i.e., why could not old c-strings accommodate longer
>> regexes)?
>
> FYI (to kinkie): The 100 patterns per aggregation was an arbitrary
> limit, but the total string length is limited by the ConfigParser buffers.
>
> The plain-text length of the aggregated regex string needs to fit within
> one squid.conf line length. With space for the directive name, label,
> type and flags being included in the limit.

that's BUFSIZ, or 4k. We currently have 1K + separator, name and flags.
If anything, we could be more aggressive in having longer regexes,
modulo OOM in regcomp.

>> And why is there no pattern85 in the "before" test?!
>>
>>
>>> The code is expected to cause a small performance regression on
>>> parse and reconfigure due to extra data copies. The regression is
>>> negligible: the attached test cases perform "squid -k parse" in 60
>>> msec in trunk and 62 on the branch on a warm cache on my i5 macbook
>>> air.
>>
>> A 3% performance regression is not necessarily negligible. Have you
>> tested with more ACL lines (not larger individual ACL lines) that take a
>> few seconds to load (as opposed to 60ms) ? If not, please do unless that
>> test would be irrelevant somehow.
>>
>> And what is the expected performance improvement from having fewer
>> longer regexes?
>
> I assume you mean performance of normal request processing through the
> regex ACL. Agreed, that need to be checked/compared to current speeds.

Time needed for squid -k parse on my Mac 1.3GHz i5 macbook air, for a
1M-entry random ACL, 13Mb-big:
new: 20.8s
trunk: 18.3s

Hopefully in time we'll move the config parser to SBuf..

-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Rework acl/RegexData optimization to use SBufList

[Kinkie]

On Mon, Nov 14, 2016 at 5:36 AM, Alex Rousskov
 wrote:
> On 11/13/2016 10:15 PM, Amos Jeffries wrote:
>
>> I think we should accumulate into two SBufList instead, one for -i and
>> one for +i instead of switching back and forth with potentially lots of
>> little patterns.
>
> Would not forcing regexes into two different groups change the regex
> evaluation order and, hence, make it more difficult for some admins to
> optimize their regexes by placing frequently matching regexes earlier?
>
> I am still not quite sure what problem we are trying to solve exactly
> here, but allowing admins to control exact regex evaluation order seems
> important [for some admins].

Then these admins should not optimize, as the compiled regex' FSM
guarantees the same results but not necessarily evaluation order.
And IMO mixing insensitive and sensitive regexes is bad practice which
I would advise against in the documentation.


-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Rework acl/RegexData optimization to use SBufList

[Kinkie]

On Mon, Nov 14, 2016 at 5:30 AM, Alex Rousskov
<rouss...@measurement-factory.com> wrote:
> On 11/13/2016 05:11 PM, Kinkie wrote:
>
>> the attached patch moves away from hand-rolling a c-string onto
>> joining a SBufList for optimizing regexes in RegexData.cc.
>
>> You can find attached as a test case the output of squidclient
>> mgr:config taken on trunk and on the submitted code. It is slightly
>> different in that the new code allows for a longer optimized regex.
>
> I see the following before/after difference:
>
>   -acl bigacl dstdom_regex ... (pattern84) (pattern86) ...
>   +acl bigacl dstdom_regex ... (pattern84)|(pattern85) ...
>
> Does changing space into "|" result in one internal regex instead of
> two? And that is the optimization you have implemented? How is that
> related to SBufs (i.e., why could not old c-strings accommodate longer
> regexes)?

If we feed too big a regex to the regex compiler it will crash with
OOM; 1024 bytes is probably on the conservative side, but it's hard to
set a predictable limit so I've mostly ketp it. This said, I've
slightly raised the limit by not counting the separators in the total
length, and this explains the difference: we use a few more patterns
per optimized regex.

>
> And why is there no pattern85 in the "before" test?!

Now that is interesting. The code is apparently buggy, and will forget
the last (or first) acl for each optimized ACL. I suspect we have a
bug in trunk, and have had for some time.

>> The code is expected to cause a small performance regression on
>> parse and reconfigure due to extra data copies. The regression is
>> negligible: the attached test cases perform "squid -k parse" in 60
>> msec in trunk and 62 on the branch on a warm cache on my i5 macbook
>> air.
>
> A 3% performance regression is not necessarily negligible. Have you
> tested with more ACL lines (not larger individual ACL lines) that take a
> few seconds to load (as opposed to 60ms) ? If not, please do unless that
> test would be irrelevant somehow.

I've done this test. Whatever performance regression this is
introducing is more than offset by the big benefit of dumping wordlist
in fovour of sbuflist.
I've tested on a 1M-entries, 13MB-long acl, where entries are randomly
generated:

Time for "squid -k parse" was  23 seconds on my macbook air.

> And what is the expected performance improvement from having fewer
> longer regexes?

Amos already answered this one.

>> [RFC]

Thanks for the pointers on this; I'll check them and move to a
separate thread as you suggest.

  Kinkie
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] [PATCH] Rework acl/RegexData optimization to use SBufList

[Kinkie]

_UNTRUSTED 
X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY X509_V_ERR_INVALID_CA 
X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT X509_V_ERR_UNABLE_TO_VERIFY_LEAF_SIGNATURE 
X509_V_ERR_SELF_SIGNED_CERT_IN_CHAIN 
acl ssl::certDomainMismatch ssl_error  SQUID_X509_V_ERR_DOMAIN_MISMATCH 
acl ssl::certNotYetValid ssl_error  X509_V_ERR_CERT_NOT_YET_VALID 
acl ssl::certHasExpired ssl_error  X509_V_ERR_CERT_HAS_EXPIRED 
follow_x_forwarded_for deny all 
 
acl_uses_indirect_client on
delay_pool_uses_indirect_client on
log_uses_indirect_client on
http_access allow bigacl 
 http_access allow manager 
 http_access deny all 
 http_access deny !Safe_ports 
 http_access deny CONNECT !SSL_ports 
 http_access allow localhost manager 
 http_access deny manager 
 http_access allow localnet 
 http_access allow localhost 
 http_access deny all 
 
http_port [::]:3128 name=3128 connection-auth=on tls-disable
https_port [::]:3128 name=3128 connection-auth=on tls-disable
ùÕ,—host_verify_strict off
client_dst_passthru on
tls_outgoing_options
ssl_unclean_shutdown off
sslproxy_session_ttl 300
sslproxy_session_cache_size 2097152 bytes
sslproxy_cert_sign signUntrusted (sslproxy_cert_sign signUntrusted line) 

sslproxy_cert_sign signSelf (sslproxy_cert_sign signSelf line) 

sslproxy_cert_sign signTrusted (sslproxy_cert_sign signTrusted line) 


sslcrtvalidator_children 32 startup=5 idle=1 concurrency=1
dead_peer_timeout 10 seconds
forward_max_tries 25
cache_mem 268435456 bytes
maximum_object_size_in_memory 524288 bytes
memory_cache_shared off
memory_cache_mode always
memory_replacement_policy lru
cache_replacement_policy lru
minimum_object_size 0 bytes
maximum_object_size 4194304 bytes
store_dir_select_algorithm least-load
max_open_disk_fds 0
cache_swap_low 90
cache_swap_high 95
access_log daemon:/Users/kinkie/tmp/squid/var/logs/access.log 
logformat=squid(access_log daemon:/Users/kinkie/tmp/squid/var/logs/access.log 

logfile_daemon /Users/kinkie/tmp/squid/libexec/log_file_daemon
logfile_rotate 10
mime_table /Users/kinkie/tmp/squid/etc/mime.conf
log_mime_hdrs off
pid_filename /Users/kinkie/tmp/squid/var/run/squid.pid
client_netmask :::::::
strip_query_terms on
buffered_logs off
netdb_filename stdio:/Users/kinkie/tmp/squid/var/cache/squid/netdb.state
cache_log /Users/kinkie/tmp/squid/var/logs/cache.log
debug_options ALL,1
coredump_dir /Users/kinkie/tmp/squid/var/cache/squid
ftp_user Squid@
ftp_passive on
ftp_epsv_all off
ftp_eprt on
ftp_sanitycheck on
ftp_telnet_protocol on
diskd_program /Users/kinkie/tmp/squid/libexec/diskd
unlinkd_program /Users/kinkie/tmp/squid/libexec/unlinkd
pinger_program /Users/kinkie/tmp/squid/libexec/pinger
pinger_enable on

url_rewrite_children 20 startup=0 idle=1 concurrency=0
url_rewrite_host_header on
url_rewrite_bypass off
url_rewrite_extras %>a/%>A %un %>rm myip=%la myport=%lp
url_rewrite_timeout 0 seconds
 on_timeout=bypass
store_id_extras %>a/%>A %un %>rm myip=%la myport=%lp

store_id_children 20 startup=0 idle=1 concurrency=0
store_id_bypass on
max_stale 604800 seconds
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
quick_abort_min 16 KB
quick_abort_max 16 KB
quick_abort_pct 95
read_ahead_gap 16384 bytes
negative_ttl 0 seconds
positive_dns_ttl 21600 seconds
negative_dns_ttl 60 seconds
minimum_expiry_time 60 seconds
store_avg_object_size 13312 bytes
store_objects_per_bucket 20
request_header_max_size 65536 bytes
reply_header_max_size 65536 bytes
request_body_max_size 0 bytes
client_request_buffer_max_size 524288 bytes
adaptation_uses_indirect_client on
via on
vary_ignore_expire off
request_entities off
relaxed_header_parser on
collapsed_forwarding off
collapsed_forwarding_shared_entries_limit 16384
forward_timeout 240 seconds
connect_timeout 60 seconds
peer_connect_timeout 30 seconds
read_timeout 900 seconds
write_timeout 900 seconds
request_timeout 300 seconds
request_start_timeout 300 seconds
client_idle_pconn_timeout 120 seconds
ftp_client_idle_timeout 1800 seconds
client_lifetime 86400 seconds
pconn_lifetime 0 seconds
half_closed_clients off
server_idle_pconn_timeout 60 seconds
ident_timeout 10 seconds
shutdown_lifetime 1 seconds
cache_mgr webmaster
mail_program mail
cache_effective_user nobody
httpd_suppress_version_string off
umask 23
announce_period 31536000 seconds
announce_host tracker.ircache.net
announce_port 3131
httpd_accel_surrogate_id kair
http_accel_surrogate_remote off
esi_parser custom
delay_pools 0
delay_initial_bucket_level 50
client_delay_initial_bucket_level 50
wccp_router ::
wccp_version 4
wccp2_rebuild_wait on
wccp2_forwarding_method gre
wccp2_return_method gre
wccp2_assignment_method hash
wccp2_service standard 0
wccp2_weight 1
wccp_address 0.0.0.0
wccp2_address 0.0.0.0
client_persistent_connections on
server_persistent_connections on
persistent_connection_after_error on
detect_broken_pconn off
digest_generation on
digest_bits_per_entry 5
digest

Re: [squid-dev] [PATCH] Extend SBufContainerJoin to have prefix and suffix arguments

[Kinkie]

On Fri, Nov 11, 2016 at 5:02 AM, Amos Jeffries <squ...@treenet.co.nz> wrote:
> On 11/11/2016 9:28 a.m., Kinkie wrote:
>>
>> v4 attached.
>>
>
> Does it have to take begin() and end() iterators explicitly?
>  can we not have it take the container itself and use a for(auto  :
> container) loop ?

No it doesn't but it's sometimes convenient.
My use case is the regex-compressing code. It can be reduced to (pseudocode)

join(regex-so-far, wordsList., separator=")|(", prefix="(", suffix=")")

granted, it can be done with explicit append calls but then you can't
inline it (e.g in streams).

The code used to take the whole container, but there's a use case for
iterating over a slice only.

-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Extend SBufContainerJoin to have prefix and suffix arguments

[Kinkie]

> Tests and code analysis show that JoinContainerIntoSBuf() allocates when
> no allocation is needed. I have also shown that it is possible to
> implement JoinContainerIntoSBuf() so that there is no extra allocation.
> AFAICT, going forward, you choices are:
>
> * state that the extra allocation is not your fault, leaving it as is,
>   and/or
> * fix JoinContainerIntoSBuf() to avoid the extra allocation.
>
> When you added the "dest" parameter, you have started moving in the
> latter direction.

I'm completing that by using the reseve() based approach

>> At the same time, it's not a matter of API: the extended
>> reserve() has the same behavior as the "problem" is elsewhere.
>
> Whether SBuf::reserve() has the same behavior as reserveSpace() depends
> on reserve() configuration parameters.
>
> JoinContainerIntoSBuf() should configure reserve() in such a way that
> reserve() does _not_ have the same behavior as reserveSpace(). I showed
> how to do that in JoinContainerIntoSBuf4() in my earlier patch. The
> patch attached now has an even simpler JoinContainerIntoSBuf5()
> implementation because I fixed reserve() to not require the .ideal
> configuration parameter to be set (trunk r14917).

Aha! I was confused as I had branched off earlier. Merging.

>
>> What we could do is:
>> - define a reserveMinSpace() which does not shrink storage
>
> Having two methods called reserveMinSpace() and reserveSpace(), one of
> which does not shrink and the other one does, makes no sense to me
> because the method names do not reflect that difference at all and
> because reserve() is always primarily about "minimum" space anyway.
>
> More importantly, reserveSpace() guarantees single ownership. IMO,
> shrinking is essentially a side effect of that guarantee.

Yes.

> I suspect we do not need a "provide single-ownership space, shrinking
> the existing single-owned MemBlob if there is one" method, but if you
> have some existing use cases for it, please share them. If there are no
> such cases, reserveSpace()/reserveCapacity() should be optimized to
> avoid unnecessary re-allocation when their MemBlob is already not
> shared, but that is a different story, unrelated to JoinContainerIntoSBuf!

I agree. And that's why I think they could be simply convenience
methods for reserve().

>> - reimplement [...] reserveSpace() as convenience wrapper around reserve()
>
> Sure, but that is outside this patch scope.

Yes.


>> - while we're at it, give SBufReservationRequirements a convenience 
>> constructor
>
> With minSpace as the only parameter? Sure. Please do not forget
> "explicit". Again, that seems to be outside your patch scope.

Yes.

> I do not recommend providing more than one constructor parameter though
> because constructors with multiple same- or similar-type parameters are
> a recipe for mismatching parameters disasters.

Let's optimize the common case, we'll see if there is a need for more later on.

v4 attached.

-- 
Francesco


sbufcontainerjoin-v4.patch
Description: Binary data
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Extend SBufContainerJoin to have prefix and suffix arguments

[Kinkie]

>> +dest.reserveSpace(prefix.length() + totalContainerSize + 
>> suffix.length());
>
> Please note that v4 still allocates memory according to my last
> experiment. See JoinContainerIntoSBuf3() which mimics your patch v4. You
> may claim that the unnecessary allocation is not the fault of this
> patch, and you could be right, but I was hoping this observation will
> inspire you to change

I don't follow. The space requirement in JoinContainerIntoSBuf is not
advisory. The accumulate at the beginning of the function is meant to
calculate exactly how much space is needed, to copy (if needed) only
once and early.

> * either your patch (mimicking my JoinContainerIntoSBuf4() which does
> not result in extra allocation) reserveSpace()

The traces you provided do not reallocate there because the underlying
MemBlob has grown enough in previous calls to have enough free space
to cover the projected needs (it was resized last at
JoinContainerIntoSBuf3).

In other words, the test data is dirtied as the tests progress.
Follow with me:
2016/11/04 11:47:21.841| 24,8| SBuf.cc(42) SBuf: SBuf2698 created
2016/11/04 11:47:21.841| 24,8| SBuf.cc(908) cow: SBuf2698 new size:1024
2016/11/04 11:47:21.841| 24,8| SBuf.cc(878) reAlloc: SBuf2698 new size: 1024

But then:

2016/11/04 11:47:21.841| 1,2| AccessLogEntry.cc(47)
JoinContainerIntoSBuf2: started
2016/11/04 11:47:21.841| 1,2| AccessLogEntry.cc(48)
JoinContainerIntoSBuf2: did not create an empty buf
2016/11/04 11:47:21.841| 24,8| SBuf.cc(908) cow: SBuf2698 new size:124
2016/11/04 11:47:21.841| 24,8| SBuf.cc(878) reAlloc: SBuf2698 new size: 124

This is correct behavior: the code advises that it only needs that
much storage and the SBuf shrinks. It's passed by reference, so from
now on 'buffer' has 124 bytes of storage instead of the pre-allocated
1024.

As a demonstration, swapping order of tests 3 and 4 results in:
[...]
2016/11/06 07:05:47.860 kid1| 1,2| main.cc(457) JoinContainerIntoSBuf4: started
2016/11/06 07:05:47.860 kid1| 24,8| SBuf.cc(130) reserve: SBuf3657
was: 0+81+47=128
2016/11/06 07:05:47.860 kid1| 24,8| SBuf.cc(912) cow: SBuf3657 new size:181
2016/11/06 07:05:47.860 kid1| 24,8| SBuf.cc(882) reAlloc: SBuf3657 new size: 181
2016/11/06 07:05:47.860 kid1| 24,8| MemBlob.cc(101) memAlloc: blob2420
memAlloc: requested=181, received=512
2016/11/06 07:05:47.860 kid1| 24,7| SBuf.cc(891) reAlloc: SBuf3657 new
store capacity: 512
2016/11/06 07:05:47.860 kid1| 24,7| SBuf.cc(146) reserve: SBuf3657
now: 0+81+431=512
2016/11/06 07:05:47.860 kid1| 1,2| main.cc(462)
JoinContainerIntoSBuf4: reserved space
2016/11/06 07:05:47.860 kid1| 24,7| SBuf.cc(154) rawSpace: reserving 7
for SBuf3657
2016/11/06 07:05:47.860 kid1| 24,7| SBuf.cc(161) rawSpace: SBuf3657 not growing


I don't think however that we should change reserveSpace() behavior
not to ever shrink; there may be use cases where it's better not to do
it in fact. At the same time, it's not a matter of API: the extended
reserve() has the same behavior as the "problem" is elsewhere.

>
> * or reserveSpace() (after making sure that all callers are going to be
> OK with that change).
>
> The latter would be better if it is possible. If it is not possible,
> then we may need to change reserveSpace() documentation so that folks do
> not use that method like you did.

What we could do is:
- define a reserveMinSpace() which does not shrink storage
- reimplement both reserveSpace() and reserveMinSpace() as convenience
wrappers around reserve() which is undoubtably more powerful
- while we're at it, give SBufReservationRequirements a convenience constructor

What do you think?
Thanks.

-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Extend SBufContainerJoin to have prefix and suffix arguments

[Kinkie]

On Fri, Nov 4, 2016 at 6:12 PM, Alex Rousskov
<rouss...@measurement-factory.com> wrote:
> On 11/04/2016 01:12 AM, Kinkie wrote:
>> On Thu, Nov 3, 2016 at 10:55 PM, Alex Rousskov
>> <rouss...@measurement-factory.com> wrote:
>>> On 11/03/2016 03:19 PM, Kinkie wrote:
>>>> On Tue, Nov 1, 2016 at 8:47 PM, Alex Rousskov wrote:
>>>>> On 11/01/2016 02:02 PM, Kinkie wrote:
>>>>>>   the attached patch extends SBufContainerJoin to have prefix and
>>>>>> suffix arguments.
>>>
>>>>> I recommend reworking this by adding a dest parameter:
>>>>> SBuf (SBuf , ...);
>
>> attached v3 does this.
>
>>  SBuf
>> +JoinContainerIntoSBuf(SBuf , ...
>
> Please return dest, not a copy of it. There is no reason to create and
> destroy a new SBuf object here IMO. SBuf creation and destruction are
> not as expensive as underlying storage copying, but they are still a
> measurable expense without justification AFAICT.

Doh, right. I had actually thought about this after sending the patch.

> There are certainly many cases where using references is a bad idea but
> returning a fed-by-reference "dest" is not one of them AFAICT. In fact,
> it is nearly the norm in "chainable" calls such as those that append
> something.

I agree that this case fits perfectly.

>> +/// variant of JoinContainerIntoSBuf not needing a SBuf lvalue
>
> I recommend a more informative description than retelling of the
> function declaration code. For example:
>
> /// convenience JoinContainerIntoSBuf() wrapper with an extra memory
> allocation

The extra memory allocation is a side effect; I agree it is a
convenience, but the convenience is in not needing a SBuf lvalue. This
makes it useable e.g. in ostreams

>
>> +dest.reserveSpace(sz + prefix.length() + suffix.length());
>
> I would reorder the operands to make the code more self-documenting,
> especially when using an anonymous variable name like "sz":
>
>   dest.reserveSpace(prefix.length() + sz + suffix.length());

Ok. I'm also renaming sz for extra clarity.

>
>
>> I am not generally worried as it seems you are about copying SBufs as
>> long as the underlying storage is not reallocated needlessly.
>
> Unfortunately, all three patch versions reallocate storage needlessly
> (for various reasons). You can study the attached debugging from the
> attached patch to see what is actually going on in various
> JoinContainerIntoSBuf implementations. It may surprise and inspire you
> to make the code better. It did surprise and inspire me (but I wish I
> was not doing this legwork!).

Sorry about that causing extra work for you.

Updated patch attached. I'm struggling a bit with the convenience
wrapper documentation. Note that the wrapper cannot return a ref,
intentionally so. I hope RVO will optimize that cost away as much as
possible.

-- 
Francesco


sbufcontainerjoin-v4.patch
Description: Binary data
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Extend SBufContainerJoin to have prefix and suffix arguments

[Kinkie]

On Thu, Nov 3, 2016 at 10:55 PM, Alex Rousskov
<rouss...@measurement-factory.com> wrote:
> On 11/03/2016 03:19 PM, Kinkie wrote:
>> On Tue, Nov 1, 2016 at 8:47 PM, Alex Rousskov wrote:
>>> On 11/01/2016 02:02 PM, Kinkie wrote:
>>>>   the attached patch extends SBufContainerJoin to have prefix and
>>>> suffix arguments.
>
>>> I recommend reworking this by adding a dest parameter:
>>> SBuf (SBuf , ...);
>
>
>> Can simply return the modified SBuf.
>
>>  SBuf
>> -SBufContainerJoin(const Container , const SBuf& separator)
>> +JoinContainerIntoSBuf(SBuf dest, const ContainerIterator ,
>
> This implementation does not avoid copies in a general case: When I
> already have an SBuf with some content, I have to feed my writeable SBuf
> to ContainerToSBuf() to avoid allocating and copying.
>
> AFAICT, the reasonable implementation options are:
>
>   1. Your old simpler implementation without "dest".
>   2. A more efficient implementation with writeable "dest".

attached v3 does this.

> The latest implementation has the complexity of #2 but lacks its
> efficiency. I do not think it is a good API.

I am not generally worried as it seems you are about copying SBufs as
long as the underlying storage is not reallocated needlessly. But it's
very simple to express one call in terms of the other, so I'm doing
just that. I am convinced that the end result of our discussion is
better than what we had started with, and I'm grateful for that.


-- 
Francesco


sbufcontainerjoin-v3.patch
Description: Binary data
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Extend SBufContainerJoin to have prefix and suffix arguments

[Kinkie]

On Tue, Nov 1, 2016 at 8:47 PM, Alex Rousskov
<rouss...@measurement-factory.com> wrote:
> On 11/01/2016 02:02 PM, Kinkie wrote:
>>   the attached patch extends SBufContainerJoin to have prefix and
>> suffix arguments. This can support a use-case which I found in the
>> current ACLRegexData work I'm following, where we need to transform
>> {"foo", "bar", "gazonk"}
>> into
>> (foo)|(bar)|(gazonk)
>>
>> It can be done with the current version of the SBufContainerJoin
>> function, but it requires one more copy.
>> There are probably further cases of this pattern elsewhere, all hand-rolled.
>
> I agree that the use case is legitimate, but the implementation does not
> solve the underlying cause -- extra copies.

It removes the need of one copy in a common use case, but it can do better.
It also has the drawback of iterating over a full container; reworking
to add a target SBuf and to use begin-end iterators instead of a
container.

> I recommend reworking this by adding a dest parameter:
>
> /// copies all container items (delimited by separator) into dest
> ...
> /// \returns dest
> SBuf (SBuf , ...);
>
> and using that function to implement the SBufContainerJoin() or a
> similarly named convenience wrapper that does not require a dest
> parameter and that returns a copy of an internally used SBuf.

Can simply return the modified SBuf. All the convenience without the
bloat. Also taking the argument by copy; will cow if needed (passing
by reference makes for awkward client code as it'd have to be an
lvalue).
So the common pattern would be:

auto out = JoinContainerIntoSBuf(SBuf(), container.begin(),
container.end(), SBuf("separator"))

> In either case, please do not forget to document that the new prefix and
> suffix are always added, even if the container is empty.

Done.

>>  // sz can be zero in two cases: ... or all items are zero-length.
>
> FWIW, this is a lie. Zero-length items do not result in zero sz unless
> the separator itself is also surprisingly empty. Zero-length items is
> not a special case worth mentioning IMO. The zero-size check should be
> replaced with a direct (begin != end) check to avoid complexity and
> confusion -- it is only needed to avoid dereferencing begin() AFAICT.

Ok.

> HTH,

Sure does :)
Updated patch attached.


-- 
Francesco


sbufcontainerjoin-v2.patch
Description: Binary data
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] [PATCH] Extend SBufContainerJoin to have prefix and suffix arguments

[Kinkie]

Hi all,
  the attached patch extends SBufContainerJoin to have prefix and
suffix arguments. This can support a use-case which I found in the
current ACLRegexData work I'm following, where we need to transform
{"foo", "bar", "gazonk"}
into
(foo)|(bar)|(gazonk)

It can be done with the current version of the SBufContainerJoin
function, but it requires one more copy.
There are probably further cases of this pattern elsewhere, all hand-rolled.

-- 
Francesco


sbuflist-prefixsuffix-v1.patch
Description: Binary data
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] Heads up with bzr and lp: urls

[Kinkie]

Hi all,
  I just found out that launchpad suffers from a bug where bzr will
hang if trying to negotiate an ecdsa key.
Workaround here: https://bugs.launchpad.net/lazr.sshserver/+bug/830679


-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Refactor wordlist to SBufList in acl/RegexData

[Kinkie]

On Thu, Oct 27, 2016 at 5:10 AM, Alex Rousskov
<rouss...@measurement-factory.com> wrote:
> On 10/26/2016 05:18 PM, Kinkie wrote:
>>>> the attached patch refactors the use of wordlist to SBufList in
>>>> acl/RegexData.cc
>
>> -while (wl != NULL) {
>> +for (SBuf i : sl) {
>
> If possible, please avoid creating new SBufs by declaring "i" to be a
> constant reference to SBuf. It is probably possible unless you modify
> "i" -- I cannot tell for sure by looking at all the "i"s in the patch.

SBuf::c_str() is unfortunately non-const which limits applicability.

> Renaming "i" to something longer and more informative like
> "patternOrOption" or even "slowPatternOrOption" would be useful for
> those who try to understand what is going on and/or have to find that
> variable in a text full of other "i"s.

Done, renamed to configurationLineWord.


>> +for (auto i : sl) {
>
> Same here, on all counts. Auto does not automatically adds "&" or "const".

I know, but this can't be changed due to c_str() below.

> What is the time difference in configuring, for example, one million of
> regexes using patched and unpatched code?

The results were so surprising I had to double-check correctness which
is confirmed.
For a 10-lines regex acl, "squid -k parse" time went DOWN from 36
seconds to 0.7 on my Intel Core i5 Macbook air.
By checking SBuf stats in cachemgr, the calling code is actually quite
bad for SBuf allocations; SBufs are apparently mostly used to hold
temporary c-strings and are immediately freed. I suspect however
calling code some O(n^2) or worse behavior in wordlist, hence the big
win.

> I have not reviewed the patch closely. I have no objections.

Thanks for the insights you gave.

Merging the updated code.

-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Refactor wordlist to SBufList in acl/RegexData

[Kinkie]

On Wed, Oct 26, 2016 at 10:40 PM, Amos Jeffries <squ...@treenet.co.nz> wrote:
> On 27/10/2016 9:16 a.m., Kinkie wrote:
>> Hi all,
>>the attached patch refactors the use of wordlist to SBufList in
>> acl/RegexData.cc; the "test.txt" attachment shows the output of
>> "./src/squid -X 2>&1 | grep 'RegexData' >/tmp/test_plan.txt" when
>> applied to a default configuration file with these additions:
>> acl t1 dstdom_regex pattern1 pattern2
>> acl t2 dstdom_regex -i pattern3 pattern4
>> acl t3 dstdom_regex +i pattern5 pattern6
>>
>
> Audit:
>
> * can you please take advantage of the change to get rid of the
> "compileOptimisedREs: compileOptimisedREs:" in the logs

Cleaned logs up.

> * please use emplace_back instead of sl.push_back(SBuf(clean));
>  - line ~234

Ok. Fixed in all instances.

Updated patch attached. If noone objects before then, I'll merge in ~8hrs.


Francesco Chemolli


wordlist-sbuflist-aclregexdata.patch
Description: Binary data
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] [PATCH] Refactor wordlist to SBufList in acl/RegexData

[Kinkie]

Hi all,
   the attached patch refactors the use of wordlist to SBufList in
acl/RegexData.cc; the "test.txt" attachment shows the output of
"./src/squid -X 2>&1 | grep 'RegexData' >/tmp/test_plan.txt" when
applied to a default configuration file with these additions:
acl t1 dstdom_regex pattern1 pattern2
acl t2 dstdom_regex -i pattern3 pattern4
acl t3 dstdom_regex +i pattern5 pattern6



-- 
Francesco Chemolli
2016/10/26 21:07:27.313| 28,2| RegexData.cc(228) parse: new Regex line or file
2016/10/26 21:07:27.313| 28,3| RegexData.cc(237) parse: buffering RE '-i'
2016/10/26 21:07:27.313| 28,3| RegexData.cc(237) parse: buffering RE 
'^cache_object://'
2016/10/26 21:07:27.313| 28,3| RegexData.cc(237) parse: buffering RE '+i'
2016/10/26 21:07:27.314| 28,3| RegexData.cc(237) parse: buffering RE 
'^https?://[^/]+/squid-internal-mgr/'
2016/10/26 21:07:27.314| 28,2| RegexData.cc(151) compileOptimisedREs: 
compileOptimisedREs: -i
2016/10/26 21:07:27.314| 28,2| RegexData.cc(169) compileOptimisedREs: 
compileOptimisedREs: adding RE '^cache_object://'
2016/10/26 21:07:27.314| 28,2| RegexData.cc(162) compileOptimisedREs: 
compileOptimisedREs: +i
2016/10/26 21:07:27.314| 28,2| RegexData.cc(118) compileRE: compiled 
'(^cache_object://)' with flags 7
2016/10/26 21:07:27.315| 28,2| RegexData.cc(169) compileOptimisedREs: 
compileOptimisedREs: adding RE '^https?://[^/]+/squid-internal-mgr/'
2016/10/26 21:07:27.315| 28,2| RegexData.cc(118) compileRE: compiled 
'(^https?://[^/]+/squid-internal-mgr/)' with flags 5
2016/10/26 21:07:27.315| 28,2| RegexData.cc(197) compileOptimisedREs: 
compileOptimisedREs: 2 REs are optimised into one RE.
2016/10/26 21:07:27.337| 28,2| RegexData.cc(228) parse: new Regex line or file
2016/10/26 21:07:27.337| 28,3| RegexData.cc(237) parse: buffering RE 'pattern1'
2016/10/26 21:07:27.337| 28,3| RegexData.cc(237) parse: buffering RE 'pattern2'
2016/10/26 21:07:27.337| 28,2| RegexData.cc(169) compileOptimisedREs: 
compileOptimisedREs: adding RE 'pattern1'
2016/10/26 21:07:27.338| 28,2| RegexData.cc(169) compileOptimisedREs: 
compileOptimisedREs: adding RE 'pattern2'
2016/10/26 21:07:27.338| 28,2| RegexData.cc(118) compileRE: compiled 
'(pattern1)|(pattern2)' with flags 5
2016/10/26 21:07:27.338| 28,2| RegexData.cc(197) compileOptimisedREs: 
compileOptimisedREs: 2 REs are optimised into one RE.
2016/10/26 21:07:27.338| 28,2| RegexData.cc(228) parse: new Regex line or file
2016/10/26 21:07:27.338| 28,3| RegexData.cc(237) parse: buffering RE '-i'
2016/10/26 21:07:27.338| 28,3| RegexData.cc(237) parse: buffering RE 'pattern3'
2016/10/26 21:07:27.338| 28,3| RegexData.cc(237) parse: buffering RE 'pattern4'
2016/10/26 21:07:27.338| 28,2| RegexData.cc(151) compileOptimisedREs: 
compileOptimisedREs: -i
2016/10/26 21:07:27.338| 28,2| RegexData.cc(169) compileOptimisedREs: 
compileOptimisedREs: adding RE 'pattern3'
2016/10/26 21:07:27.338| 28,2| RegexData.cc(169) compileOptimisedREs: 
compileOptimisedREs: adding RE 'pattern4'
2016/10/26 21:07:27.338| 28,2| RegexData.cc(118) compileRE: compiled 
'(pattern3)|(pattern4)' with flags 7
2016/10/26 21:07:27.338| 28,2| RegexData.cc(197) compileOptimisedREs: 
compileOptimisedREs: 2 REs are optimised into one RE.
2016/10/26 21:07:27.338| 28,2| RegexData.cc(228) parse: new Regex line or file
2016/10/26 21:07:27.338| 28,3| RegexData.cc(237) parse: buffering RE '+i'
2016/10/26 21:07:27.338| 28,3| RegexData.cc(237) parse: buffering RE 'pattern5'
2016/10/26 21:07:27.338| 28,3| RegexData.cc(237) parse: buffering RE 'pattern6'
2016/10/26 21:07:27.339| 28,2| RegexData.cc(160) compileOptimisedREs: 
compileOptimisedREs: optimisation of +i ... +i
2016/10/26 21:07:27.339| 28,2| RegexData.cc(169) compileOptimisedREs: 
compileOptimisedREs: adding RE 'pattern5'
2016/10/26 21:07:27.339| 28,2| RegexData.cc(169) compileOptimisedREs: 
compileOptimisedREs: adding RE 'pattern6'
2016/10/26 21:07:27.339| 28,2| RegexData.cc(118) compileRE: compiled 
'(pattern5)|(pattern6)' with flags 5
2016/10/26 21:07:27.339| 28,2| RegexData.cc(197) compileOptimisedREs: 
compileOptimisedREs: 2 REs are optimised into one RE.


wordlist-sbuflist-aclregexdata.patch
Description: Binary data
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] bzr -> git?

[Kinkie]

Hi all,
  I'm currently trying to use recent advancements in Jenkins to
improve our QA via gated commits to trunk.
  This raises (again) the issue of bazaar versus git. Remaining on
bazaar is getting more and more painful, as tools such as jenkins
focus on git. Asm an example, jenkins pipelines only support git.
 This topic was already touched in the past, and IIRC the output
was "we'll do it, but not now". Can we reevaluate when "now" might be?

Thanks!

-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [RFC] Support concurrent SBuf::c_str() calls

[Kinkie]

On Fri, Sep 30, 2016 at 6:03 PM, Alex Rousskov
 wrote:
> On 09/29/2016 09:19 PM, Amos Jeffries wrote:
>> On 30/09/2016 5:03 a.m., Alex Rousskov wrote:
>>> Should we remove the increment to make concurrent c_str() calls safe?
>
>> The reason it exists remember is to prevent other SBuf sharing that
>> storage MemBuf from thinking they can append to the storage oer top of
>> the terminator.
>
> Yes, I know, but that increment not only prevents problems but _causes_
> them as well: The increment may force the SBuf to realloc, which may
> result in deallocation of the original storage still pointed to by the
> earlier c_str() result.
>
>
>> Given that we have no easy knowledge of how the c_str()
>> is going to be used by the recipient code we cannot guarantee lack of
>> terminator is safe.
>
> Yes, with or without the increment, Squid may crash or misbehave when
> buggy code appends to SBuf while a previous c_str() result is still in
> use. The lack of increment is not safe. The presence of the increment is
> not safe. The problems are different, but they exist with or without the
> increment.
>
>
>> Time to push SBuf usage forward and aggressively remove the need for
>> c_str()?
>
> The need for c_str() will never go away because libraries outside of
> Squid control need c-strings. And, due to the lack of proper QA, any
> "aggressive push" affecting lots of Squid code is a recipe for a
> disaster IMO.
>
> The question is which c_str() implementation is safer? The one that may
> crash Squid when called twice for the same SBuf in the same expression
> or the one that may crash Squid when appending to the same SBuf in the
> same expression? There are already two known cases of the former. There
> are currently no known cases of the latter.
>
>
> Overall, I know of three primary ways to implement c_str():
>
> 1. Always 0-terminate SBuf-used storage. Safest implementation possible,
> but has serious negative performance overheads, especially for tokenizing.

If this is the decision, might as well drop sbuf altogether in favor
of std::string (or possibly std::string + custom mempools-friendly
Allocator).

> 2. Terminate the used storage at the time of c_str().
>
> 2a. Increment the used storage size. Current implementation.
> Leads to problems discussed in this thread.
>
> 2b. Do not increment the used storage size.
> Leads to other problems discussed in this thread.
>
> 3. Allocate and store a new c-string as needed, copying from the
> storage. Safest implementation possible but has serious negative
> performance overheads. These overheads affect different use cases than
> those of #1.

If we are aware of the underlying issue, there's no reason not to save
the temporary to a naked char* and use that, as long as the SBuf is
not touched it'll work fine.

> Needless to say, we can add clever code to reduce risks associated with
> some implementations. For example (these are just rough sketches/ideas):
>
> a) Appending code in #2b can check whether it is about to overwrite the
> 0-terminator added by a previous c_str() call and, if yes, preserve the
> old storage (we can preserve one such storage or maintain a global FIFO
> queue).
>
> b) C-string allocating code in #4 can maintain a global FIFO queue of
> previously allocated but now supposedly unused c-strings to minimize the
> risk of de-allocating a still used c-string.
>
> c) C-string allocating code in #4 can delay allocation until it might be
> needed, similar to how (a) decides that the 0-terminator is in danger.
>
>
> Going forward, do you think we should:
>
> i. Keep using #2a and rewrite the known double-c_str() code.
> ii. Make double-c_str() code safe (i.e., switch to #2b).
> iii. Add code to make the existing #2 implementation safer.
> . Switch to a completely different implementation like #1 or #4.

v. push SBuf as suggested by Amos and document this specific corner
case so to avoid it

-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] Call for interested Squid developers to join our Coverity Scan team

[Kinkie]

Hello,
   The Squid project is benefiting of a kind donation from Coverity,
in the form of free ( as in beer ) access to their Coverity Scan
(https://scan.coverity.com/) static code analysis product to analyze
our code base for defects.
   The tool has highlighted in time several potential (and some real)
bugs; some other developers and I spend some time vetting, analyzing
and fixing those bugs.

We have evaluated the possibility opening up access to the reports to
everyone, but despite that being our wish, this is not technically
feasible.
What can be done is to grant access to parties requesting it; so if
you are interested in seeing, triaging or fixing bugs uncovered by
Scan, please drop me a note so that I may grant you access.
Details at http://wiki.squid-cache.org/CoverityTesting

Thanks!
-- 
Francesco Chemolli
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] dlink question

[Kinkie]

Hi Adam.
That's a very good question. We are in fact slowly refactoring from our
custom data structures to std:: wherever appropriate. If you wish to join
the effort, patches are welcome :)
  Kinkie

On Wednesday, 14 September 2016, Adam Majer <ama...@suse.de> wrote:

> Hello,
>
> This maybe a very naive question, but why is squid 4.x still using
> manually crafted double linked list? (dlink) Why not just use std::list
> instead?
>
> - Adam
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org <javascript:;>
> http://lists.squid-cache.org/listinfo/squid-dev
>


-- 
@mobile
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] Sad performance trend

[Kinkie]

On Tue, Sep 6, 2016 at 6:43 PM, Alex Rousskov <
rouss...@measurement-factory.com> wrote:

> On 09/06/2016 08:27 AM, Amos Jeffries wrote:
> > On 27/08/2016 12:32 p.m., Alex Rousskov wrote:
> >> W1  W2  W3  W4  W5  W6
> >>   v3.1  32% 38% 16% 48% 16+ 9%
> >>   v3.3  23% 31% 14% 42% 15% 8%
> >>   v3.5  11% 16% 12% 36%  7% 6%
> >>   v4.0  11% 15%  9% 30% 14% 5%
>
> > That trend goes all the way back to at least 2.6. Which is a bit weird,
> > since it contradicts the total request-per-second capacity we have been
> > watching in polygraph results.
>
> I do not know what you mean by "total request-per-second capacity", but
> I most likely have not been watching it. Is there a historical results
> table for that measure that I can study?
>
>
> > The speed per request goes down and the number that can be handled rises.
>
> Even if concurrent transactions handling has been improving (and that is
> a big if -- I have no data to back it up), it does not excuse the
> significant worsening of per-transaction performance IMO.
>
>
> > As far as my simple observations have been, the per-request slowdown
> > correlates to the amount of code being added (ie new features) and also
> > to the amount of AsyncCalls involved with the request handling (where
> > more AsyncCall is more latency).
>
> Additional code obviously slows things down, but the existing rate of
> adding new features/async calls does not seem to match the magnitude of
> the measured performance decline, especially for the basic code paths
> tested by these micro tests. For things to be getting worse in such a
> manner, we probably have to make the old code/features worse (in
> addition to adding new features/async calls) and/or adding those new
> features in a terribly inefficient way.
>

Hi,
  As you have obviously spent some effort in this, would it be possible to
share these benchmarks with the larger community, possibly by leveraging
the automations we are already using?

-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] Coding standards

[Kinkie]

Hi Adam!
  You're absolutely right; Squid 4.0 is c++11; squid 3.5 is not getting new
features right now.

On Tue, Aug 23, 2016 at 12:48 PM, Adam Majer  wrote:

> Hello,
>
> What are the coding standards for Squid? More specifically, I would like
> to know if C++11 (or later?) is allowed in Squid 4.0. I'm assuming that
> Squid 3.5 is still on the old C++03.
>
> Thanks,
> Adam
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev
>



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] Retiring unsupported build nodes

[Kinkie]

Hi,
  Ubuntu Utopic and Vivid are now EOL at Ubuntu; I'll be retiring these
nodes and replacing them with a Xenial node instead.
  I'll also be retiring Fedora 21 and replacing it with a Fedora 24 node.

  Opensuse seems to have jumped from release 13.2 to 42.2, I'll be trying
that one out as well. Freebsd-11 is due sometime in september, and openbsd
would benefit from some love and updates as well.

Please excuse any shakey landings.

-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] HTTP meetup in Stockholm

[Kinkie]

Well, there are representatives from all major UA makers. So while I don't
expect them to deviate from their business objectives for us, if we have
any specific grievances for them, we certainly can.

Regarding JSON headers: what I like about it is that while everything you
say is true, more or less can be said about the current state of affairs
which is more or less free-for-all. At least with json we would have an
uniform machine-parseable format.
It could however very well be that our feedback be that it is not ambitious
enough and we would prefer e.g. ASN1+ some schema

On Tue, Jul 12, 2016 at 11:01 AM, Marcus Kool 
wrote:

>
> On 07/12/2016 06:53 AM, Henrik Nordström wrote:
>
>> tis 2016-07-12 klockan 18:34 +1200 skrev Amos Jeffries:
>>
>>> I'm much more in favour of binary formats. The HTTP/2 HPACK design
>>> lends
>>> itself very easily to binary header values (ie sending integers as
>>> interger encoded value). Following PHK's lead on those.
>>>
>>
>> json is very ambiguous with no defined schema or type restrictions.
>> It's up to the receiver to guess type information from format while
>> parsing, which in itself is a mess from security point of view.
>>
>> The beauty of json is that it is trivially extensible with new data,
>> and have all basic data constructs you need for arbitrary data. (name
>> tagged, strings, integers, floats, booleans, arrays, dictionaries and
>> maybe something more). But for the same reason it's also unsuitable for
>> HTTP header information which should be consise, terse and un-ambiguous
>> with little room for syntax errors.
>>
>> Regards
>> Henrik
>>
>
> Extensible json headers seems to lend itself to put a lot of
> application-specific
> stuff in headers instead of in payload. The headers should be used for the
> protocol only.
>
> Squid has had many issues in the past with non-conformity to standards.
> The Squid developers obviously want to stick with the standards and are
> forced by non-conformant apps and servers to support non-conformity.
> Can this workshop be used to address this?
>
> Marcus
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev
>



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] HTTP meetup in Stockholm

[Kinkie]

On Mon, Jul 11, 2016 at 8:53 PM, Henrik Nordström <h...@squid-cache.org>
wrote:

>
> sön 2016-07-10 klockan 11:33 +0100 skrev Kinkie:
> > Hi all,
> >   at the end of the month I will attend the HTTP meetup in Stockholm.
> > Besides having a chance to see Henrik, I'd like to collect your
> > feedback and opinions on the topic that are likely to be touched.
>
> Oh, This have completely bypassed my radar, most likely beacause it has
> been practically turned off for a long while. Any reference to more
> details?
>

Not much really.
https://github.com/HTTPWorkshop/workshop2016/wiki/Arrangements
Short story: 40 people from various projects (I could recognize Google,
Facebook, Mozilla, Varnish, Apple, Akamai, Canon, Microsoft, GSMA,
Cloudflare, and Squid) meet to discuss the status of HTTP and exchange war
stories. 3 days, invite-only. Amos was contacted by mnot and relayed to me.

I the only near time upcoming HTTP activity I can find is in Berlin the
> upcoming weekend as part of IETF96.
>
> Note that I live a bit outside of Stockholm these days.. about 2 hours
> by car or train to Surahammar which is close to Västerås.
>

Nod. I don't think it'll be feasible to have you join the workshop itself,
there may be a possibility for Monday evening's dinner, if you wish I can
ask about it.


-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH Bug 4534 and N-bit fixes for CacheDigest

[Kinkie]

Hi,
   I think it is reasonable to expand the number of objects Squid can
manage. 32-bit systems are almost rare to come by nowadays.

On Sun, Jul 10, 2016 at 11:08 AM, Amos Jeffries 
wrote:

> On 10/07/2016 1:37 a.m., Amos Jeffries wrote:
> > This patch converts the CacheDigest members and method parameters to use
> > explicitly sized data types more appropriate for what details they hold.
> >
> > * 64-bit Digest capacity (entry count)
> > * 32-bit Mask Size (byte count)
> > *  8-bit Bit count per entry
> >
> > Due to various store_digest.cc code still relying on masks not exceeding
> > 2^31-1 worth of memory space we have to still assert that bitCount
> > calculation does not exceed that value.
> >
>
> Additional to this, to fully fix the bug 4534 issue we are going to have
> to decide if it is reasonable to have extremely large Cache Digest masks
> (several tens of GB or memory).
>
> The caches the bug appears on are all several TB of size (by space). But
> even more importantly they are spread over more than 5 cache_dir. So the
> 2^25-ish limit on entries per cache_dir is not keeping the total
> Squid-wide object count within a 32-bit value.
>
> Amos
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev
>



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH Bug 4534 and N-bit fixes for CacheDigest

[Kinkie]

Hi,

+uint32_t mask_size; /* mask size in bytes */
+uint64_t capacity;   /* expected maximum for .count, not a hard
limit */
+int8_t bits_per_entry; /* number of bits allocated for each entry
from capacity */
 int count;  /* number of digested entries */
 int del_count;  /* number of deletions performed so far */

Can we sort data members by descending size? This will allow for more
efficient packing in memory.
Apart from this, LGTM. +1 with the suggested change.

On Sat, Jul 9, 2016 at 2:37 PM, Amos Jeffries  wrote:

> This patch converts the CacheDigest members and method parameters to use
> explicitly sized data types more appropriate for what details they hold.
>
> * 64-bit Digest capacity (entry count)
> * 32-bit Mask Size (byte count)
> *  8-bit Bit count per entry
>
> Due to various store_digest.cc code still relying on masks not exceeding
> 2^31-1 worth of memory space we have to still assert that bitCount
> calculation does not exceed that value.
>
> Amos
>
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev
>
>


-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] HTTP meetup in Stockholm

[Kinkie]

Hi all,
  at the end of the month I will attend the HTTP meetup in Stockholm.
Besides having a chance to see Henrik, I'd like to collect your feedback
and opinions on the topic that are likely to be touched.

Currently there is rather hot discussion on the mailing list on the topics
of JSON headers (https://tools.ietf.org/html/draft-ietf-httpbis-jfv-01),
secondary certificate authentication (
https://tools.ietf.org/html/draft-bishop-httpbis-http2-additional-certs-01),
cache digests (
https://tools.ietf.org/html/draft-ietf-httpbis-cache-digest-00). Other
topics of discussion have tapered off.
What is the project's position I should bring to the table, if requested?

-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] Dealing with RegisteredHeadersHash.gperf

[Kinkie]

On Wed, Jun 22, 2016 at 6:42 AM, Amos Jeffries <squ...@treenet.co.nz> wrote:

> On 22/06/2016 5:13 a.m., Eduard Bagdasaryan wrote:
> > Hello,
> >
> > In my current task I need to change some of header definitions, but it is
> > unclear how to do this correctly. My understanding is that I need to
> > modify http/RegisteredHeadersHash.gperf, and then run
> > "make gperf-files" to generate RegisteredHeadersHash.cci. Is this
> correct?
>
> I believe so, though I've not done it myself yet. Kinkie knows more if
> he replies.
>

Yes, that's it. Remember that - contrary to what we do in many other
similar cases - the .cci file needs to be committed to the repository.


>
> > The http/RegisteredHeadersHash.gperf has a comment that it was
> > auto-generated
> > from RegisteredHeadersHash.gperf. How could it be generated from itself?
>
> Its not. That %{ ...}% is a block of data to be inserted verbatum into
> the output file at the same (top) position. Bit unfortunate that we had
> to do it that way.
>

Yes.
-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] scripts/trace-kid.pl

[Kinkie]

Likewise. +1.

On Tue, Jun 14, 2016 at 2:49 AM, Amos Jeffries  wrote:

> On 14/06/2016 1:22 p.m., Alex Rousskov wrote:
> > Hello,
> >
> > I got tired missing important cache.log lines (or temporary patching
> > other tracing scripts to better handle SMP logs) and added a script to
> > print cache.log lines for a given kid.
> >
> > Any objections to committing this?
> >
>
> None.  +1.
>
> Amos
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev
>



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] What to do with stale related software?

[Kinkie]

Hi,
 done.

On Wed, Jun 1, 2016 at 7:27 AM, Kinkie <gkin...@gmail.com> wrote:

> Ok, I'll check them and readjust/remove as necessary.
> Thanks
>
>
> On Wednesday, 1 June 2016, Amos Jeffries <squ...@treenet.co.nz> wrote:
>
>> On 31/05/2016 4:37 p.m., Kinkie wrote:
>> > Hi all,
>> >   I'm reviewing some 'related software' listings on the website, and
>> > some links are stale.
>> > What should we do with them? I recommend to do a cleanup; opinions?
>> >
>>
>> If it has been a long time and the link still broken, or not popular
>> software. I think we can remove.
>>
>>
>> Every so often I go through and see if I can find a new link to them
>> anywhere.
>>
>> For some that are looking useful but unmaintained we've copied the files
>> to our FTP "pub/squid/contrib/" folder and linked to that. Or like for
>> squidpurge pulled into our repo and taken up maintenance.
>>
>> Amos
>>
>> ___
>> squid-dev mailing list
>> squid-dev@lists.squid-cache.org
>> http://lists.squid-cache.org/listinfo/squid-dev
>>
>
>
> --
> @mobile
>



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] cleanup cppunit detection and use

[Kinkie]

HI!
  have you had a chance to check if the new code works for all the systems
we care about?

Thanks!

On Sun, Jun 12, 2016 at 1:28 AM, Amos Jeffries  wrote:

> The cppunit-config tool has apparently been replaced by pkg-config .pc
> file years ago and is now in the process of being removed from some OS.
> Notably Fedora.
>
> Which means our present way of detecting it for use by "make check" will
> increasingly fail.
>
> This converts configure.ac to using the pkg-config method of detection
> and updates the --with-cppunit-basedir parameter to --without-cppunit
> matching our naming and usage scheme for other similar options. If a
> =PATH is explicitly provided cppunit is assumed to exist at that
> location without configure-time checking.
>
>
> Alex: can you test that at least a bootstrap and "make check" cycle run
> properly with this patch applied on your old system which has been
> having issues recently with ./configure.
>
> Thanks
> Amos
>
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev
>
>


-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] What to do with stale related software?

[Kinkie]

Ok, I'll check them and readjust/remove as necessary.
Thanks

On Wednesday, 1 June 2016, Amos Jeffries <squ...@treenet.co.nz> wrote:

> On 31/05/2016 4:37 p.m., Kinkie wrote:
> > Hi all,
> >   I'm reviewing some 'related software' listings on the website, and
> > some links are stale.
> > What should we do with them? I recommend to do a cleanup; opinions?
> >
>
> If it has been a long time and the link still broken, or not popular
> software. I think we can remove.
>
>
> Every so often I go through and see if I can find a new link to them
> anywhere.
>
> For some that are looking useful but unmaintained we've copied the files
> to our FTP "pub/squid/contrib/" folder and linked to that. Or like for
> squidpurge pulled into our repo and taken up maintenance.
>
> Amos
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org <javascript:;>
> http://lists.squid-cache.org/listinfo/squid-dev
>


-- 
@mobile
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] What to do with stale related software?

[Kinkie]

Hi all,
  I'm reviewing some 'related software' listings on the website, and
some links are stale.
What should we do with them? I recommend to do a cleanup; opinions?

-- 
Kinkie
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Deprecating SMB LanMan helpers

[Kinkie]

+1.

On Thu, May 26, 2016 at 10:03 AM, Amos Jeffries  wrote:
> Bring the SMB LanMan helpers one step closer to removal by dropping them
> from the set of helpers which are auto-detected and built by default
> with Squid.
>
> They are still available for the minority using them. But need to be
> explicitly listed in the ./configure options to be built.
>
> Amos
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev
>



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Drop ie_refresh configuration option

[Kinkie]

+1.

On Thu, May 26, 2016 at 10:16 AM, Amos Jeffries  wrote:
> This option was provided as a hack to workaround problems in MSIE 5.01
> and older.
>
> Since those MSIE versions are long deprecated and no longer even
> registering on the popularity charts for more than 5 years I think its
> time we removed this hack.
>
> Amos
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev
>



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] libmem API cleanup pt2

[Kinkie]

Looks reasonable to me, except that new class names do not comply with
naming conventions

-- 
  Francesco
On Apr 20, 2016 8:00 AM, "Amos Jeffries"  wrote:

> On 12/04/2016 11:31 p.m., Amos Jeffries wrote:
> > This is the first of the followup patches I promised when applying the
> > un-polished bug 4438 patch to 4.0.8.
> >
> >
> > Convert all the objects using the libmem "old API" for as-needed pools
> > to using the MEMPROXY_CLASS() API which is better designed for late
> > initialization.
> >
> > The GetPool(type) function is now guaranteed to produce a
> > pre-initialized pool for all valid pool types. Except in the case where
> > it is recursively being used by Mem::Init() to perform that
> initialization.
> >
>
> This is about to pass the 10-day hold criteria for merge. If there are
> no objections I will apply it in a day or two after todays release has
> settled.
>
> Amos
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev
>
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Replace new/delete operators using modern C++ rules

[Kinkie]

Hi,
  I can check Mac.
We have had a donation of Solaris nodes, though I need to revise and
unrot the setup. I'll do that as soon as possible but don't hold your
breath.

On Sun, Apr 10, 2016 at 5:50 AM, Amos Jeffries  wrote:
> On 7/04/2016 7:05 p.m., Alex Rousskov wrote:
>> Hello,
>>
>> This change was motivated by "Mismatched free()/delete/delete[]"
>> errors reported by valgrind and mused about in Squid source code.
>>
>> I speculate that the old new/delete replacement code was the result of
>> slow accumulation of working hacks to accomodate various environments,
>> as compiler support for the feature evolved. The cumulative result does
>> not actually work well (see the above paragraph), and the replacement
>> functions had the following visible coding problems according to [1,2]:
>>
>> a) Declared with non-standard profiles that included throw specifiers.
>>
>> b) Declared inline. C++ says that the results of inline declarations
>>have unspecified effects. In Squid, they probably necessitated
>>complex compiler-specific "extern inline" workarounds.
>>
>> c) Defined in the header file. C++ says that defining replacements "in
>>any source file" is enough and that multiple replacements per
>>program (which is what a header file definition produces) result in
>>"undefined behavior".
>>
>> d) Declared inconsistently (only 2 out of 4 flavors). Declaring one base
>>flavor should be sufficient, but if we declare more, we should
>>declare all of them.
>>
>> [1] http://en.cppreference.com/w/cpp/memory/new/operator_new
>> [2] http://en.cppreference.com/w/cpp/memory/new/operator_delete
>>
>> The replacements were not provided to clang (trunk r13219), but there
>> was no explanation why. This patch does not change that exclusion.
>>
>>
>> I have no idea whether any of the old hacks are still necessary in some
>> cases. However, I suspect that either we do not care much if the
>> replacements are not enabled on some poorly supported platforms OR we
>> can disable them (or make them work) using much simpler hacks for the
>> platforms we do care about.
>>
>>
>> If the patch is approved, the SquidNew.h file should be removed. The
>> patch does not contain this change, but I hope that removing that file
>> should not be difficult -- it is not even mentioned in Makefile.am files
>> AFAICT.
>>
>> The attached patch does not remove Debug::OutStream class that was
>> created to work around some of the above problems. There is a different
>> pending patch ("Do not hide important/critical messages") which also
>> removes that class completely (for somewhat different reasons):
>> http://lists.squid-cache.org/pipermail/squid-dev/2016-March/005510.html
>>
>
> AFAIK we should still have MacOS and Solaris people around to check that
> this update does not break anything for them. Though the BuildFarm does
> not have nodes to do it ourselves.
>
> In principle this is a great step forward, but I would like confirmation
> about the portability side of things before it actually gets merged.
>
> Amos
>
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] BUg 4466: removal of -k kill command

[Kinkie]

Hi,
  I like it. I am not sure how it fits with our stance not to
deprecate API as much as possible.

So +0, I'd like us to ask feedback from downstream before merging.

On Tue, Mar 29, 2016 at 6:22 PM, Amos Jeffries  wrote:
> The Squid "-k kill" command line option is equivalent to "kill -9" on
> whatever process has its PID in the .pid file.
>
> Since Squid gained multi-process SMP support it has been declining in
> usefulness, and with recent changes to make thee master process be the
> one operating the .pid file this command has effectively become irrelevant.
>
> Since admin have the "kill -9" command available anyway and also have
> better understanding of what will happen when using that command I
> propose removing "squid -k kill" entirely.
>
> The attached patch implements that.
>
> Amos
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev
>



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] implement RFC3986

[Kinkie]

>> Will it be initialized at all? I'd expect that fromHexTable, which is
>> const and POD be simply laid out in the data segment and not require
>> initialization at all.
>
> Are you implying that
>
> (a) fromHexTable is a C++11 constexpr _and_
> (b) constexpr cannot suffer from initialization order problems?
>
> If yes, then you should declare fromHexTable as constexpr so that
> somebody does not accidentally change it to be something else. This will
> make (a) a strong explicit statement rather than an implied and fragile
> implication.

both are list of constants. No calls to any code of any kind.
toHexTable can offer a stronger guarantee than const char *: it's a
const char * const.
It can't be declared constexpr: g++ complains that the standard
doesn't allow that for strings.
fromHexTable is constexpr.

> However, I cannot find any C++ rule that guarantees the behavior you
> expect -- your fromHexTable (even if you add constexpr to it) does not
> seem to match any of the three items at:
>
>   http://en.cppreference.com/w/cpp/language/constant_initialization

To me they seem both cases of (quote)
3) Static or thread-local object (not necessarily of class type), that
is not initialized by a constructor call, if the object is
value-initialized or if every expression in its initializer is a
constant expression.

> Can you point me to some documentation that guarantees your expectation
> will be fulfilled? To avoid misunderstanding: I am not saying your
> expectation is wrong (it certainly sounds reasonable to me). I am only
> saying that I cannot find any confirmation that what you expect is
> actually guaranteed.

This stems from my understading of the meaning of the ".data" section
of ELF files (and similar sections of other binaries), which may be
partial or incomplete.

> [Please avoid "no initialization" terminology because it implies that
> the object is left uninitialized -- what you probably mean is that
> fromHexTable is initialized during C++ "constant initialization" phase.]

Yes.

>> I agree however with simply moving the tables in the .cc file,
>> clearing all doubts.
>
> AFAIK, moving those table definitions to .cc file does not magically
> help FromHex() callers in any way. Moreover, they were already in the
> .cc file in the previous patch.

I suspect you are right. I was under the impression that if code in a
translation unit is called, then that translation unit has been
initialized by then. I now understand that I am wrong in assuming
that.

> Moving FromHex() itself to .cc file does not magically help indirect
> FromHex() callers in any way either.
>
>
> Please forgive me for whining, but it feels like you are trying various
> random combinations and using me as a validation test: Does this work?
> No, then how about this? Or perhaps that? This is a bad approach not
> just because it wastes hours but because I am not a good validator and
> will miss bugs. The correct approach would be to write code that you do
> not just "expect" to work correctly (that's always implied) but can
> _prove_ (to yourself, but using C++ rules) to work correctly as far as
> initialization order is concerned.

Unfortunately initialization rules are quite hard for me to understand yet.

>> +const int16_t fromHexTable[256] = {
>
> AFAICT, this needs to be "static" and should be "constexpr" to (a)
> guarantee constant initialization and (b) minimize the chances of
> somebody changing it to something that will not be initialized at
> "constant initialization" time. Please correct me if I am wrong.

Aren't variables not declared extern marked static by default?
Sure can do but it should be redundant.

> Please check other tables/globals as well.
>
>
>
>>> One full/stand-alone declaration per line please.
>
>> Ok
>
> These are still merged:

Oh, sorry. I restricted myself to declarations, not definitions.
But you are right, I had missed Rfc3986 declarations as well.
Done.

>> +const CharacterSet
>> +Rfc1738::Unsafe("rfc1738:unsafe", "<>\"# %{}|\\^~[]`'"),
>> +Rfc1738::Ctrls("rfc1738:ctrls", {{0x00, 0x1f}, {0x7f,0xff}}),
>> +Rfc1738::Reserved("rfc1738:reserved", ";/?:@=&"),
>> +Rfc1738::UnsafeAndCtrls = Rfc1738::Unsafe + Rfc1738::Ctrls,
>> + Rfc1738::Unescaped = (Rfc1738::UnsafeAndCtrls - 
>> CharacterSet(nullptr,"%") ).rename("rfc1738:unescaped")
>
>
> Please check other declarations as well.
>
>
 +// XXX: SBuf lacking reserve(N)
 +// rv.reserve(s.length()*2); //TODO: optimize arbitrary constant
>>>
>>> AFAICT, SBuf::reserveSpace() should work fine here and in all other
>>> define-SBuf-and-immediately-reserve-space contexts. Am I missing something?
>>
>> API compatiblity; std::string doesn't have reserveSpace but only reserve.
>
> That answer does not compute for me: Why would API compatibility with
> std::string matter when you are not using templates anymore?

This is changed in the current version of the patch in fact.

> Just noticed that this was a private message. Please do not ask 

Re: [squid-dev] [RFC][PATCH] Bug4438 second attempt: give MemBlob its own pools

[Kinkie]

On Tue, Mar 15, 2016 at 9:43 PM, Alex Rousskov
<rouss...@measurement-factory.com> wrote:
> On 03/14/2016 02:11 PM, Kinkie wrote:
>
>>   this second attempt at bug4438 tries a different approach: by giving
>> MemBlob its own pools and having a hard initialization dependency in
>> MemBlob's allocating function, instead of relying on memAllocString.
>
> Forgive me if I missed how this decision was made earlier, but I do not
> understand why we want to duplicate the existing memAllocString()-like
> functionality and split a single set of raw memory pools shared by all
> callers into two isolated sets?

It is an attempt to sidestep initialization issues by having pools in
the same translation unit as their user.
The other effects are IMO generally beneficial but are not the main goal.

> In other words, how did we get from "SIGSEGV in memFreeString" (i.e.,
> bug 4438) to this? If your working (or perhaps already proven!) theory
> is that memAllocString() does something wrong, why not fix
> memAllocString() instead of duplicating it? I must be missing something
> obvious here...

Unfortunately we can't prove what is doing what wrong as we (or at
least I) can't reproduce the bug.

>> Known side-effects:
>> - allows to tune memblob sizes according to actual use patterns, which
>> can be collected via SBuf/MemBlob detailed stats. The currently-set
>> sizes follow some light testing I've done.
>
> Why is that a side effect? Could we not "tune memblob sizes according to
> actual use patterns" before this change?

We do not know the useage patterns of all memAllocString callers; we
do know those of MemBlob via SBufDetailedStats. As an alternative
approach for this kind of optimization, we could instrument
memAllocString to gather that data as well.

> Or do you meant that we can tune two separate sets of pools now? If yes,
> then you should also list "two separate sets of pools instead of one
> shared by all callers" as a side effect -- "more isolated sets" is the
> "cost" associated with this ability to tune more sets...

Nod.

>> - as no MemBlob can be initialized before MemPools, there is no need
>> to baloon statically-initialized MemBlobs to
>> SmallestStringBeforeMemIsInitialized.
>
> If your change guarantees that no MemBlob can be initialized before
> MemPools, why cannot a similar change be done to the original code so
> that no memAllocString()/etc. caller can get its result before MemPools
> are initialized? AFAICT, you have solved the problem that you were
> trying to solve, but I do not understand why you had to solve it in a
> new location (leaving the old problem unsolved?).

I do not yet know if I have solved the problem, unfortunately.
If this attempt is successful, it can be a blueprint for modernizing
memAllocString, or splitting other callers altogether. The file name
(old_api) is to me an indicator that these API are not really
encouraged, and on the way to deprecation if not deprecated already.

Thanks

-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] implement RFC3986

[Kinkie]

> I do not think so. The crux of the issue is that you are defending a
> medium-size template function as a necessary and small evil, while I am
> attacking the principle or direction of accommodating helper needs by
> making Squid code worse.

Hi,
  the attached patch is a modification of what was currently shared,
de-templatized.
It relies on SBuf and provides a convenient std::string-based wrapper,
like you suggest.

>> In order to clarify, when I talk about couplign I mean link-time
>> coupling. This can be directly understood from the list of objects
>> needed for testSBuf: between headers, objects and stubs that's about
>> 30 files (on top of the testSBuf files themselves).
>
> If testSBuf has 30 dependencies, then I doubt testSBuf is a good example
> of how things should be done. Overall, if we want helpers to use Squid
> code, SBuf can be a part of the convenience library that can be linked
> with helpers without 30 "extra" or "stub" files. The number of files in
> that convenience library is pretty much irrelevant.

The introduction of libsbuf that has happened in the meanwhile causes
less dependency issues: it now only requires libsbuf, libmem, libbase,
libmisccontainers and libmiscutil, as _LDADD instead of _SOURCE files.
I'll work on removing the dependencies from misccontainers which is
required for trie by seeing if a std:: container behaves at least
comparably well for mempools.

-- 
Francesco


rfc3986.bundle
Description: Binary data
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] [RFC][PATCH] Bug4438 second attempt: give MemBlob its own pools

[Kinkie]

Hi all,
  this second attempt at bug4438 tries a different approach: by giving
MemBlob its own pools and having a hard initialization dependency in
MemBlob's allocating function, instead of relying on memAllocString.

Known side-effects:
- allows to tune memblob sizes according to actual use patterns, which
can be collected via SBuf/MemBlob detailed stats. The currently-set
sizes follow some light testing I've done.
- it causes a dependency on full libmem in unit tests, since stubbing
memAllocString and friends is no longer sufficient
- as no MemBlob can be initialized before MemPools, there is no need
to baloon statically-initialized MemBlobs to
SmallestStringBeforeMemIsInitialized.

One thing I can't figure out is in cachemgr mem report: it lists the
smallest-sized MemBlob pool twice. I've treble-checked: MemBlob pools
are only initialized once, yet in following the
MemPools::Instance::pools linked-list they appear more than once. I
can't really understand why.

Comments?

Thanks

-- 
Francesco


memblob-self-pools-v1.bundle
Description: Binary data
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] tentative fix for bug 4438

[Kinkie]

On Sun, Mar 6, 2016 at 9:41 PM, Alex Rousskov
<rouss...@measurement-factory.com> wrote:
> On 03/06/2016 11:16 AM, Kinkie wrote:
>
>> As I cannot reproduce the issue it's really hard to do a root cause analysis.
>
> Agreed. However, if you can make SBuf/MemBlob use memory pools before
> those pools are initialized, then we do have a bug worth fixing (even if
> Yuri does not suffer from that specific bug). It may be possible to
> easily reproduce _that_ bug by adding a few static SBuf variables
> directly to squid.h.

old_api should be already doing that by rounding strings' sizes up to
SmallestStringBeforeMemIsInitialized (16kb + 4 bytes), I'm working on
the assumption that for some reason this mechanism is failing either
at startup or at shutdown time.

>> At this point I'm thinking about reinstrumenting MemBlob not to use
>> mem/old_api but to use its own pools.
>
> Why would that make any difference? Whatever pools MemBlob is using,
> they must be initialized prior to use. Thus, it is probably best to fix
> "old_api" initialization than to duplicate that [buggy] code.

.. or to get out from there at all. Having the MemBlob pools be static
members of the MemBlob class will guarantee proper initialization
before even the first MemBlob is instantiated, won't it?

-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] tentative fix for bug 4438

[Kinkie]

On Sat, Mar 5, 2016 at 10:21 PM, Alex Rousskov
<rouss...@measurement-factory.com> wrote:
> On 03/05/2016 04:31 AM, Kinkie wrote:
>> ensuring that mempools are initialized before MemBlob is
>> used.
>
> Please explain how
>
>   * a call to Mem::Init()
>
> at some random time during dynamic initialization[1] would _ensure_ that
> Mem::Init() is called before
>
>   * MemBlob is used.

In fact, it doesn't, and my attempt failed - alternatively I got my
analysis wrong and the issue is not caused by a non-pooled allocation.
It seems I got wrong how initialization works.

> P.P.P.S. Please note that whether your patch helps Yuri is irrelevant
> for the long-term fix. If the bug is a usage/initialization race, then
> it may appear to be fixed when you reshuffle random events, but it will
> happen again unless properly fixed.

As I cannot reproduce the issue it's really hard to do a root cause analysis.
At this point I'm thinking about reinstrumenting MemBlob not to use
mem/old_api but to use its own pools. It's essentially code
duplication, but it should ensure that pools are fully initialized
when used.

-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] [PATCH] tentative fix for bug 4438

[Kinkie]

Hi all,
  attached is a tentative fix for bug 4438.
I can't reproduce the bug, unfortunately; judging from the traces it
seems to me to be a case of mismatched alloc/free calls (alloc through
memallocate, free through mempools).

If that's the case, there's a few angles to take on this problem, and
here is one: ensuring that mempools are initialized before MemBlob is
used. I chose this approach as it's the least invasive, and it's made
possible by libmem not depending on libsbuf; another could be to have
MemBlob not rely on old_api (basically refactoring some old_api into
MemBlob itself), and yet another could be to make mempools handle more
gracefully the case of having to free a memory region they did not
allocate.

please let me know your thoughts; I also kindly ask Yuri if he can try
the fix and see if it fixes the issue he's seeing.

-- 
Francesco
=== modified file 'src/mem/old_api.cc'
--- src/mem/old_api.cc	2016-01-01 00:12:18 +
+++ src/mem/old_api.cc	2016-03-05 11:10:40 +
@@ -396,7 +396,8 @@
 void
 Mem::Init(void)
 {
-int i;
+if (MemIsInitialized)
+return;
 
 /** \par
  * NOTE: Mem::Init() is called before the config file is parsed
@@ -439,7 +440,7 @@
 MemPools[MEM_MD5_DIGEST]->setChunkSize(512 * 1024);
 
 /** Lastly init the string pools. */
-for (i = 0; i < mem_str_pool_count; ++i) {
+for (int i = 0; i < mem_str_pool_count; ++i) {
 StrPools[i].pool = memPoolCreate(StrPoolsAttrs[i].name, StrPoolsAttrs[i].obj_size);
 StrPools[i].pool->zeroBlocks(false);
 

=== modified file 'src/sbuf/MemBlob.cc'
--- src/sbuf/MemBlob.cc	2016-03-01 10:25:13 +
+++ src/sbuf/MemBlob.cc	2016-03-05 11:13:27 +
@@ -14,6 +14,16 @@
 
 #include 
 
+namespace {
+class MemInitializer {
+public:
+MemInitializer() {
+Mem::Init();
+}
+};
+auto mi = MemInitializer();
+} /* anon namespace */
+
 MemBlobStats MemBlob::Stats;
 InstanceIdDefinitions(MemBlob, "blob");
 

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] sbuf/forward.h

[Kinkie]

Hi,
  no comments in almost a day and trivial patch; merged as r14574.

On Tue, Mar 1, 2016 at 5:30 PM, Kinkie <gkin...@gmail.com> wrote:
> Hi,
>   the attached patch defines sbuf/forward.h and makes use of it
> everywhere SBuf had been forward-declared.
>
> --
> Francesco



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] [PATCH] sbuf/forward.h

[Kinkie]

Hi,
  the attached patch defines sbuf/forward.h and makes use of it
everywhere SBuf had been forward-declared.

-- 
Francesco


sbuf-forward.bundle
Description: Binary data
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] libsbuf

[Kinkie]

On Thu, Feb 25, 2016 at 11:56 PM, Alex Rousskov
<rouss...@measurement-factory.com> wrote:
> On 02/25/2016 10:01 AM, Kinkie wrote:
>> Hi all,
>>   please find attached a the bundle implementing libsbuf (merge
>> proposal from lp:~squid/squid/sbuflab).
>>
>> It:
>> - shuffles SBuf core files into src/sbuf and implements sbuf/libsbuf.la
>> - refactors SBuf <-> String adaption functions into a separate header,
>> increasing decoupling
>> - reduces the list of stubs and libraries needed for SBuf unit tests
>> - removes _DEPENDENCIES clauses from affected unit tests' Makefile.am
>
> If possible, please do not call things foo/FooX.h. One foo is enough.

Sure. Preparing a rename patch.

> Is this v4.0-only change? Have you considered _not_ using a library (and
> a whole new directory) for these few SBuf-related files? Why not just
> include them in sources list using a Makefile variable?

My understanding is that the lib approach is the current recommended
best practice. Am I wrong?

> What does libsbuf.la depend on? If it depends on src/mem/ perhaps it
> should go there? Again, this SBuf-dedicated directory feels wrong (too
> narrow-scoped), and MemBlob should not really be there (if it is so
> narrowly scoped; what if I want MemBlob without SBuf)?

The shortest list of dependencies I can think of is in tests_testSBuf_SOURCES:
it requires full libbase, and at least stubs for time, debug, fatal and libmem.
This however raises a general question for all libraries: how do we
effectively document dependencies? Except for libbase, all are
expected to have some.

>> +#include "SBuf.h"
>
> You should #include SBuf.h the same way *everywhere*: sbuf/SBuf.h

Hadn't done so for files in src/sbuf/ . Fixed.

> These are not objections.
Please find attached the related merge bundle.

-- 
Francesco


sbuf-review.bundle
Description: Binary data
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Remove most Makefile.am DEPENDENCIES

[Kinkie]

Committed as r14566.

On Fri, Feb 26, 2016 at 5:50 PM, Amos Jeffries <squ...@treenet.co.nz> wrote:
> On 27/02/2016 12:01 a.m., Kinkie wrote:
>> Hi all,
>>   as part of the "makefile.am DEPENDENCIES considered harmful" push,
>> please find attached a patch removing most DEPENDENCIES in
>> src/Makefile.am .
>> Build-tested.
>
> +!.
>
> Amos
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] [PATCH] Remove most Makefile.am DEPENDENCIES

[Kinkie]

Hi all,
  as part of the "makefile.am DEPENDENCIES considered harmful" push,
please find attached a patch removing most DEPENDENCIES in
src/Makefile.am .
Build-tested.

-- 
Francesco
=== modified file 'src/Makefile.am'
--- src/Makefile.am	2016-02-25 18:01:29 +
+++ src/Makefile.am	2016-02-26 10:48:00 +
@@ -937,9 +937,7 @@ TESTS += $(check_PROGRAMS)
 #tests_testX_LDADD=\
 #	$(SQUID_CPPUNIT_LIBS) \
 #	$(SQUID_CPPUNIT_LA) \
-#	$(COMPAT_LIB) \
-#tests_testX_DEPENDENCIES= $(SQUID_CPPUNIT_LA)
-
+#	$(COMPAT_LIB) 
 
 # - add other component .(h|cc) files needed to link and run tests
 tests_testHttpReply_SOURCES=\
@@ -1046,7 +1044,6 @@ tests_testHttpReply_LDADD=\
 	$(SSLLIB) \
 	$(COMPAT_LIB) \
 	$(XTRA_LIBS)
-tests_testHttpReply_DEPENDENCIES= $(SQUID_CPPUNIT_LA)
 
 tests_testACLMaxUserIP_SOURCES= \
 	cbdata.cc \
@@ -1179,8 +1176,6 @@ tests_testACLMaxUserIP_LDADD= \
 	$(COMPAT_LIB) \
 	$(XTRA_LIBS)
 tests_testACLMaxUserIP_LDFLAGS = $(LIBADD_DL)
-##tests_testACLMaxUserIP_DEPENDENCIES = \
-##	$(SQUID_CPPUNIT_LA)
 
 ## a demonstration test that does nothing but shows the salient points
 ## involved in writing tests.
@@ -1196,13 +1191,12 @@ nodist_tests_testBoilerplate_SOURCES = \
 	$(TESTSOURCES)
 tests_testBoilerplate_LDADD= \
 	$(SQUID_CPPUNIT_LIBS) \
+	$(SQUID_CPPUNIT_LA) \
 	$(SSLLIB) \
 	base/libbase.la \
 	$(COMPAT_LIB) \
 	$(XTRA_LIBS)
 tests_testBoilerplate_LDFLAGS = $(LIBADD_DL)
-tests_testBoilerplate_DEPENDENCIES = \
-	$(SQUID_CPPUNIT_LA)
 
 ## Tests of base/libbase.la objects
 tests_testCharacterSet_SOURCES = \
@@ -1462,9 +1456,6 @@ tests_testCacheManager_LDADD = \
 	$(COMPAT_LIB) \
 	$(XTRA_LIBS)
 tests_testCacheManager_LDFLAGS = $(LIBADD_DL)
-tests_testCacheManager_DEPENDENCIES = \
-	$(REPL_OBJS) \
-	$(SQUID_CPPUNIT_LA)
 
 tests_testDiskIO_SOURCES = \
 	CacheDigest.h \
@@ -1635,7 +1626,6 @@ tests_testDiskIO_LDADD = \
 	$(SSLLIB) \
 	$(COMPAT_LIB) \
 	$(XTRA_LIBS)
-
 tests_testDiskIO_LDFLAGS = $(LIBADD_DL)
 tests_testDiskIO_DEPENDENCIES = \
 	DiskIO/libdiskio.la \
@@ -1898,9 +1888,6 @@ tests_testEvent_LDADD = \
 	$(COMPAT_LIB) \
 	$(XTRA_LIBS)
 tests_testEvent_LDFLAGS = $(LIBADD_DL)
-tests_testEvent_DEPENDENCIES = \
-	$(REPL_OBJS) \
-	$(SQUID_CPPUNIT_LA)
 
 ## Tests of the EventLoop module.
 tests_testEventLoop_SOURCES = \
@@ -2140,9 +2127,6 @@ tests_testEventLoop_LDADD = \
 	$(COMPAT_LIB) \
 	$(XTRA_LIBS)
 tests_testEventLoop_LDFLAGS = $(LIBADD_DL)
-tests_testEventLoop_DEPENDENCIES = \
-	$(REPL_OBJS) \
-	$(SQUID_CPPUNIT_LA)
 
 tests_test_http_range_SOURCES = \
 	AccessLogEntry.cc \
@@ -2378,8 +2362,6 @@ tests_test_http_range_LDADD = \
 	$(COMPAT_LIB) \
 	$(XTRA_LIBS)
 tests_test_http_range_LDFLAGS = $(LIBADD_DL)
-tests_test_http_range_DEPENDENCIES = \
-	$(SQUID_CPPUNIT_LA)
 
 ## Tests of parser/* objects
 tests_testTokenizer_SOURCES = \
@@ -2446,12 +2428,11 @@ tests_testHttp1Parser_LDADD= \
 	sbuf/libsbuf.la \
 	$(top_builddir)/lib/libmiscutil.la \
 	$(SQUID_CPPUNIT_LIBS) \
+	$(SQUID_CPPUNIT_LA) \
 	$(SSLLIB) \
 	$(COMPAT_LIB) \
 	$(XTRA_LIBS)
 tests_testHttp1Parser_LDFLAGS = $(LIBADD_DL)
-tests_testHttp1Parser_DEPENDENCIES = \
-	$(SQUID_CPPUNIT_LA)
 
 ## Tests of the HttpRequest module.
 tests_testHttpRequest_SOURCES = \
@@ -2690,9 +2671,6 @@ tests_testHttpRequest_LDADD = \
 	$(COMPAT_LIB) \
 	$(XTRA_LIBS)
 tests_testHttpRequest_LDFLAGS = $(LIBADD_DL)
-tests_testHttpRequest_DEPENDENCIES = \
-	$(REPL_OBJS) \
-	$(SQUID_CPPUNIT_LA)
 
 ## Tests for icmp/* objects
 # icmp/libicmpcore.la is used by pinger so SHOULD NOT require more dependancies! :-(
@@ -2894,13 +2872,12 @@ tests_testStore_LDADD= \
 	$(NETTLELIB) \
 	$(REGEXLIB) \
 	$(SQUID_CPPUNIT_LIBS) \
+	$(SQUID_CPPUNIT_LA) \
 	$(SSLLIB) \
 	CommCalls.o \
 	$(COMPAT_LIB) \
 	$(XTRA_LIBS)
 tests_testStore_LDFLAGS = $(LIBADD_DL)
-tests_testStore_DEPENDENCIES = \
-	$(SQUID_CPPUNIT_LA)
 
 ## string needs mem.cc.
 ## mem.cc needs ClientInfo.h
@@ -2934,12 +2911,11 @@ tests_testString_LDADD = \
 	$(top_builddir)/lib/libmiscutil.la \
 	$(REGEXLIB) \
 	$(SQUID_CPPUNIT_LIBS) \
+	$(SQUID_CPPUNIT_LA) \
 	$(SSLLIB) \
 	$(COMPAT_LIB) \
 	$(XTRA_LIBS)
 tests_testString_LDFLAGS = $(LIBADD_DL)
-tests_testString_DEPENDENCIES = \
-	$(SQUID_CPPUNIT_LA)
 
 SWAP_TEST_DS =\
 	repl_modules.o \
@@ -3543,9 +3519,6 @@ tests_testURL_LDADD = \
 	$(COMPAT_LIB) \
 	$(XTRA_LIBS)
 tests_testURL_LDFLAGS = $(LIBADD_DL)
-tests_testURL_DEPENDENCIES = \
-	$(REPL_OBJS) \
-	$(SQUID_CPPUNIT_LA)
 
 tests_testSBuf_SOURCES= \
 	tests/testSBuf.h \
@@ -3616,12 +3589,11 @@ tests_testConfigParser_LDADD = \
 	$(top_builddir)/lib/libmiscutil.la \
 	$(REGEXLIB) \
 	$(SQUID_CPPUNIT_LIBS) \
+	$(SQUID_CPPUNIT_LA) \
 	$(SSLLIB) \
 	$(COMPAT_LIB) \
 	$(XTRA_LIBS)
 tests_testConfigParser_LDFLAGS = $(LIBADD_DL)
-tests_testConfigParser_DEPENDENCIES = \
-	$(SQUID_CPPUNIT_LA)
 	
 tests_testStatHist_SOURCES = \
 	tests/stub_cbdata.cc \
@@ -3695,7 +3667,6 @@ tests_testEnumIterator_LDADD = \
 	$(COMPAT_LIB) \
 	$(SQUID_CPPUNIT_LA) \
 	$(XTRA_LIBS)
-tests_testEnumIterator_DEPENDENCIES =
 
 

Re: [squid-dev] [PATCH] libsbuf

[Kinkie]

On Thu, Feb 25, 2016 at 6:50 PM, Amos Jeffries <squ...@treenet.co.nz> wrote:
> On 26/02/2016 6:01 a.m., Kinkie wrote:
>> Hi all,
>>   please find attached a the bundle implementing libsbuf (merge
>> proposal from lp:~squid/squid/sbuflab).
>>
>> It:
>> - shuffles SBuf core files into src/sbuf and implements sbuf/libsbuf.la
>> - refactors SBuf <-> String adaption functions into a separate header,
>> increasing decoupling
>> - reduces the list of stubs and libraries needed for SBuf unit tests
>> - removes _DEPENDENCIES clauses from affected unit tests' Makefile.am
>>
>> It has to be a bundle to carry information about file moves; this
>> means no long-context diff, I'm sorry.
>> Code builds fine in-tree. I'll perform a full test before possibly
>> merging if you approve the patch, of course.
>>
>
> I think all the stats.allocFromString accounting could stay. It would
> just be for std::string now instead of combined std::string and SquidString.
>
> OR, maybe the String to/from counting could be done by the
> SBufStringConvert.h functions instead of the SBuf.

As part of my discussion with Alex we talked about reducing the number
of statistics taken. I believe that number not to be  very relevant
and thus removed it. Other such statistics will follow

> in src/Makefile.am:
> * tests_testSBuf_LDADD needs to include $(SQUID_CPPUNIT_LA)
>  - same for tests_testSBufList_LDADD and tests_testLookupTable_LDADD

Done.

> in src/mk-string-arrays.awk:
> * can you take advantage of the edit to s/namesapce/namespace/ please?

Done

> in src/sbuf/SBufStringConvert.h:
> * use:  #include "sbuf/SBuf.h"

ok

> in src/tests/testSBuf.cc:
> * use: #include "tests/SBufFindTest.h"

Done.


-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] [PATCH] libsbuf

[Kinkie]

Hi all,
  please find attached a the bundle implementing libsbuf (merge
proposal from lp:~squid/squid/sbuflab).

It:
- shuffles SBuf core files into src/sbuf and implements sbuf/libsbuf.la
- refactors SBuf <-> String adaption functions into a separate header,
increasing decoupling
- reduces the list of stubs and libraries needed for SBuf unit tests
- removes _DEPENDENCIES clauses from affected unit tests' Makefile.am

It has to be a bundle to carry information about file moves; this
means no long-context diff, I'm sorry.
Code builds fine in-tree. I'll perform a full test before possibly
merging if you approve the patch, of course.

Thanks

-- 
Francesco


sbuflib.bundle
Description: Binary data
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] implement RFC3986

[Kinkie]

On Sun, Feb 21, 2016 at 4:51 PM, Alex Rousskov
<rouss...@measurement-factory.com> wrote:
> On 02/20/2016 11:27 AM, Kinkie wrote:
>
>> Sorry to bring this topic up again, but honestly I don't understand
>> your position.
>>
>> I believe that the deadlock we currently are in
>
> There is no deadlock. I think the design decision to use templated
> escape functions to accomodate std::string-using helpers was wrong, but
> I am not bold enough to stop you from committing it. Much worse code
> goes in despite my objections or without proper audit.

Unfortunately there is: I am prepared to abandon this effort unless we
reach consensus - at least non-opposition. It'd be a pity, as it's an
useful feature.

>> is caused by the fact
>> that we are discussing about two topics at the same time.
>> One is: should Rfc3986 target both SBuf and std::string, if the
>> performance and complexity penalty in doing so is none or small? I
>> (and I believe Amos) think it's a good, or at least not a bad idea.
>
> Any "Should I do X if X is good?" question is pointless. The question is
> "Should I do X?". I have already outlined why I think doing X is harmful.
>
>
>> The other is: should we use std::string or SBuf or c-strings in
>> helpers? My opinion is that std::string would be preferable, yours is
>> that SBuf is preferable, if any change is performed over the current
>> state of using c-strings. I think change over current state is a good
>> idea, you - I believe - are averse enough to using std::string that
>> would prefer no change.
>
> I think helpers should use whatever works best for them, given the
> available Squid APIs and other factors. The question is not about what
> helpers should use but whether Squid code should bend over backwards
> (e.g., providing a templated function to escape strings and adjust SBuf
> to work with that function) to accommodate a helper. The answer, IMO, is
> "no". It is much better to

> * provide SBuf to helpers that want top performance
>
> and
>
> * provide a trivial std::string-escaping function (that uses
> SBuf-escaping function internally) to helpers that want to use C-strings
> or std::strings.

Ok, I believe here is the crux of the issue. rfc3986 is a library; we
could split it into two versions, one to be used by helpers and one to
be used by squid. The two versions would be remarkably similar and
have a lot of code duplications, or as you propose be one an adapter
of the other.
An adapter exposing a std::string API and using SBuf internally would
be quite inefficient as it'd need to copy data back and forth.
Performance is not a big concern when it comes to helpers, but I argue
that this approach would require more code, be less efficient, and
increase coupling between squid and helpers than the template-based
approach.
In order to clarify, when I talk about couplign I mean link-time
coupling. This can be directly understood from the list of objects
needed for testSBuf: between headers, objects and stubs that's about
30 files (on top of the testSBuf files themselves).

>> Can we try to decouple the two discussions and see if this helps us
>> reach a consensus?
>>
>> My point of view is:
>> - having a more generic API costs us nothing - the code is compact,
>> readable and efficient regardless the genericity, so we should merge
>> Rfc3986.
>
> A single template costs virtually nothing. I speculate that the design
> decision to treat std::string-using helpers as the primary driving
> factor for Squid APIs will cost a lot long-term.

Let's be clear: Squid API are in no way influenced by this. However
Squid has a need (a rfc3986/1738 codec library) which also helpers
have. Using a template leverages the opportunity offered by the very
similar APIs that std::string and SBuf have to increase applicability
of the library.

> This is very similar to assert(string_size < 64K). The cost of that
> single line was nothing when it was written. However, the cost of the
> design decision that Strings can self-limit their size is a growing
> collection of CVEs. IMO, you are opening a Pandora box. The cost of
> lifting the lid is indeed negligible, but that cost is not what I am
> worried about.

I still can't see the long-term harm, while I can see the short-term
harm in not following my idea.
To the cost of repeating myself, we want a SBuf API for squid where
performance and COW matter a lot. I wish a std::string API for helpers
to avoid having to link 30+ squid files and stubs for each helper
(object size is not a concern, it's more about maintaining this long
list of dependencies).

>> - there is no performance or readability benefit in using SBuf in
>> helpers due to the way client code is structured.
&

Re: [squid-dev] [PATCH] implement RFC3986

[Kinkie]

>> What is the big problem with these generic template functions?
>
> I believe the problem has already been discussed. I have no new general
> information to add to that discussion, but would be happy to try to
> answer any specific questions.

Sorry to bring this topic up again, but honestly I don't understand
your position.
Rfc3986 seems to work, offers an API which is arguably cleaner than
the current rfc1738* methods, is at least as fast if not faster, and
comes with unit tests which we are currently lacking (well, live code
has worked for a long time so we may not need them, but still..)

I believe that the deadlock we currently are in is caused by the fact
that we are discussing about two topics at the same time.
One is: should Rfc3986 target both SBuf and std::string, if the
performance and complexity penalty in doing so is none or small? I
(and I believe Amos) think it's a good, or at least not a bad idea.
The other is: should we use std::string or SBuf or c-strings in
helpers? My opinion is that std::string would be preferable, yours is
that SBuf is preferable, if any change is performed over the current
state of using c-strings. I think change over current state is a good
idea, you - I believe - are averse enough to using std::string that
would prefer no change.

Can we try to decouple the two discussions and see if this helps us
reach a consensus?

My point of view is:
- having a more generic API costs us nothing - the code is compact,
readable and efficient regardless the genericity, so we should merge
Rfc3986.
- there is no performance or readability benefit in using SBuf in
helpers due to the way client code is structured. Also, we would need
to stub a lot of functionality it relies on (e.g. mempools,
statistics, cachemgr). This would create new code-paths raising the
risk of problems. Therefore I believe we should use std::string in
helpers, and in particular I will make no attempt at refactoring
helpers to use SBuf, while I have already invested some time in
refactoring to std::string and am willing to work more on that path.

What do we do, and if you disagree with my point of view, why? I will
not spend time coding on a dead-end path, obviously, and while your
preference for SBuf is clear, the root-cause of that preference is not
clear to me yet.

Thanks!

-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] testRock still not working on macOS

[Kinkie]

Hi all,
  the latest work done on Shm to make it work on MacOS seems to be
working, however it is not still possible to do unit tests, because
the anchors IPC segment created by testRock has too long a filename
(it's "/squid-testRock_Store_map_anchors.shm").

Could we maybe work on shortening the shm segments names
(e.g. reducing Store_map to "sm" or getting rid of the underscores)?

-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] SBuf const iterator fixes

[Kinkie]

+1; please merge asap.

On Mon, Feb 8, 2016 at 7:18 PM, Amos Jeffries  wrote:
> The SBufIterator is mostly actually a const iterator. But not quite.
>
> The point of difference from const_iterator is the operator*() API which
> is also different from the normal itertor API in that is returns a char
> instead of a char& or const char &.
>
> The attached patch fixes that API making the iterator equivalent to
> const_iterator for the subset of iterator API that it does present.
>
> This has side effects on the reverse iterator and SBuf begin/end
> methods. Which are also const'ified.
>
> There appears to be no built-time or unit test identified problems from
> this change. IT passes the build farm QA easily.
>
> Are there any objections to fast-tracking this merge? If not I hope to
> apply this in the next day or two.
>
> Amos
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev
>



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] fix for build error mentioned in bso#4403

[Kinkie]

Hi,
  thanks, however the issue was alrady fixed in r14487.

On Sun, Jan 24, 2016 at 11:13 PM, Christian  wrote:
> Hi,
>
> stumbled over this build error mention in bugzilla (#4403) ... and fixed
> it with attached patch.
> Cheers
>
> --
>
> Christian
> 
>- Please do not 'CC' me on list mails.
>   Just reply to the list :)
> 
> Der ultimative shop für Sportbekleidung und Zubehör
>
> http://www.sc24.de
> 
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev
>



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] ftruncate() failures on OS X (Darwin)

[Kinkie]

> If there are no objections, I will commit your patch (with the above
> polishing touches).

+1 from me. Please commit as soon as it's convenient for you.

Thank you guys!


-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [RFC] The situation with helpers/

[Kinkie]

On Tue, Jan 5, 2016 at 6:01 PM, Alex Rousskov
 wrote:
> On 01/04/2016 08:58 PM, Amos Jeffries wrote:
>> The SBuf and CharacterSet patch auditing has brought up the questions of
>> why helpers/ is separate from src/, built first, what cross-dependency
>> there can be, and even why we are bundling C++ helpers to begin with.
>>
>> So lets clarify it formally once and for all.
>>
>> Q1: why is helpers/ built before src/ ?
>
> A1: Primarily for historical and paranoia reasons.
>
>
>> Q2: why is helpers/ separate from src/ ?
>
> A2: Primarily for historical and paranoia reasons.
>
>
>> should we shuffle them back to src/*/helpers/ directories on a
>> per-component basis ?
>
> Yes w.r.t. src/. There is no known good reason to enact artificial hard
> barriers between parts of Squid code. All Squid code should be in src/.

Could or could not. For sure it would help with dependency tracking
where needed, and there are cases where it is needed.

> Not sure whether the helpers/ subdirectory is [always] needed but that
> is a separate question.
>
>
>> * it does somewhat hinder anyone wanting to manually build a single
>> helper. Though I have not heard of anyone doing so in many years.
>
> The move does not preclude anyone from building a helper. The fact that
> a few more things may be built along the way is a minor detail IMO.
> Patches to improve dependency tracking (by migrating away from recursive
> Makefiles??) are welcomed.

Ugh. see src/Makefile.am for the monster file this would yield.

>> * it does mean we should probably discard the goal of having the helpers
>> be demo code examples for people to base their custom helpers on.
>
> The location of the helper code is pretty much irrelevant for the above
> goal. However, if we want to provide demo helpers that are easy to
> understand and customize, then we probably should not use C++ to write
> those demo helpers.

I don't agree. We should use whatever tool makes sense, depending on
the availability of libraries, complexity, volunteer time etc.

> * If we want to provide the smallest set of default helpers for the
> widest possible set of deployment environments, then internal
> dependencies are preferred to the external ones and, hence, C++ is the
> best language choice.
>
> * If we want to provide default production-quality helpers for the most
> demanding deployment environments, then internal dependencies are
> irrelevant, and C++ is the best language choice. Please note that I do
> not know whether it is wise to declare performance the primary goal,
> given the performance deficiencies of the helper interface itself and
> the current Squid Project resources.
> Any open source helper can be examined, customized, and cloned, of
> course. Our decision with regard to programming language and
> dependencies may make it easier or more difficult, but not impossible.
> It is a question of the primary optimization target: ease of
> understanding/customizability versus supported platforms versus performance.

I believe our goals should be, in descending order of importance:
1- cover a reasonable set of needs
2- provide a gallery of examples for administrators who have special
needs not covered by the provided set of helpers. Corollaries of this
objective are:
2a- bundled helpers should aim at being as readable as possible.
Whether this qualifies as "high quality" is up for debate :)
2b- bundled helpers should try to avoid relying on Squid internal code
within reason. We can expect administrators embarking in the task to
be competent programmers, asking them to be also competent Squid
coders is maybe too much to ask.
3- performance is nice to have


About build dependencies from src/ : while refactoring some helpers I
came across a recipe which seems reasonably clean and maintainable,
and I submit it for consideration until a more structured approach
(such as lifting helpers/ into src/) can be chosen:

In each helper's Makefile.am, the following clauses must be added:

IMPORTED_FILES = 
nodist__SOURCES = $(IMPORTED_FILES)
_DEPENDENCIES = $(IMPORTED_FILES)
CLEANFILES += $(IMPORTED_FILES)

$(IMPORTED_FILES): $(top_srcdir)/src/$@
test -d $(dir $@)|| mkdir -p $(dir $@)
cp $(top_srcdir)/src/$@ $@

 must be replaced with the executable's name. The need for
this replacement unfortunately means that this recipe must be copied
to each helper, and can't be factored out.
This has been tested on the full build farm and satisfies our
portability requirements.

-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [RFC] The situation with helpers/

[Kinkie]

On Wed, Jan 13, 2016 at 10:13 PM, Amos Jeffries <squ...@treenet.co.nz> wrote:
> On 14/01/2016 2:12 a.m., Kinkie wrote:
>>
>> About build dependencies from src/ : while refactoring some helpers I
>> came across a recipe which seems reasonably clean and maintainable,
>> and I submit it for consideration until a more structured approach
>> (such as lifting helpers/ into src/) can be chosen:
>>
>> In each helper's Makefile.am, the following clauses must be added:
>>
>> IMPORTED_FILES = > imported into the helper's directory from src>
>> nodist__SOURCES = $(IMPORTED_FILES)
>> _DEPENDENCIES = $(IMPORTED_FILES)
>> CLEANFILES += $(IMPORTED_FILES)
>
> Okay on the above. "No..." on the below bit.
>
>>
>> $(IMPORTED_FILES): $(top_srcdir)/src/$@
>> test -d $(dir $@)|| mkdir -p $(dir $@)
>> cp $(top_srcdir)/src/$@ $@
>>
>
> ... doing it like that will make dozens of .am snippets of code that
> have to be maintained in-sync forever.
>
> Like I said on IRC this part needs to be in a src/ImportedFiles.am that
> gets included wherever its needed. Not copy-pasted everywhere.
> See how we use doc/manuals/Substitute.am if you want an example.

My bad, I thought your suggestion was to factor out the whole recipe,
and that wouldn't work.
This snippet alone will.

> All that the various Makefile.am should contain is the X=Y variable
> definitions followed by a "include src/ImportedFiles.am" statement.
>
>
> And for the love of all thats holy, avoid *_DEPENDENCIES variables.

Won't work without, unfortunately. Without _DEPENDENCIES, the
generated Makefile will copy the .cc files, but not the .h files.


> When DEPENDENCIES is *absent*, automake will take the SOURCES list and
> add dependency for each file listed, plus (compiler version dependent) a
> sub-dependency recursively for each file #include'd by those. It will
> also take the LDADD list and for all local tree objects will add a
> dependency. It is smart enough to filter out flags and system libraries
> nowdays (AFAICT that was the reason DEPENDENCIES was used to begin with
> back in Y2K).
>
> When DEPENDENCIES is present only objects explicitly listed in that
> variable are dependencies. *all* of the existing ones are broken and
> breaking Squid builds.

I hadn't understood htis, however I don't know how to force automake
to copy the .h files without :( Any idea?

>>  must be replaced with the executable's name. The need for
>> this replacement unfortunately means that this recipe must be copied
>> to each helper, and can't be factored out.
>
> The important part to factor away duplication of is:
>
>> $(IMPORTED_FILES): $(top_srcdir)/src/$@
>> test -d $(dir $@)|| mkdir -p $(dir $@)
>> cp $(top_srcdir)/src/$@ $@
>>
>
> which does not contain the "target" bit.

Agreed.

Thanks!

-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [RFC] The situation with helpers/

[Kinkie]

On Tue, Jan 5, 2016 at 4:58 AM, Amos Jeffries  wrote:
> The SBuf and CharacterSet patch auditing has brought up the questions of
> why helpers/ is separate from src/, built first, what cross-dependency
> there can be, and even why we are bundling C++ helpers to begin with.
>
> So lets clarify it formally once and for all.
>
> Q1: why is helpers/ built before src/ ?
>
> * originally most of the helpers were actually C helpers, not C++. We
> had significant build errors some obvious some subtle related to
> compiler include paths pulling in src/ C++ content and dependencies.
>
> * building first enforces that src/ obects do not exist in the build
> directory on clean builds, so cannot be depended on by mistake.
>
> TODAY:
>  a) that C vs C++ distinction non longer exists. If the dependencies are
> re-introduced they will be simply an annoyance with extra LDADD lists to
> be maintained and increased helper binary sizes. Not a build or run-time
> error.
>
>  b) several of the helpers are actually very tightly integrated and even
> built within src/ (pinger, diskd, ssl_crtd etc).
>
>
> Which brings us to ..
>
> Q2: why is helpers/ separate from src/ ?
>   should we shuffle them back to src/*/helpers/ directories on a
> per-component basis ?
>
> 1* the current configure.ac logic would as easily work for scanning
> src/*/helpers/X as for helpers/*/X
>
> 2* the move would simplify the bundle size by a few Makefiles.
>
> 3* the move would simplify the use of our fancy custom logic designs in
> helpers.
>
> 4* it does somewhat hinder anyone wanting to manually build a single
> helper. Though I have not heard of anyone doing so in many years.
>
> 5* it does mean we should probably discard the goal of having the helpers
> be demo code examples for people to base their custom helpers on.

I have no objections to moving helpers/ into src/.
I see little benefits in point 2 above (I've numbered them), and few
downsides in 4, I don't think point 5 applies.
I don't think c vs. c++ makes a difference with dependencies. It's
some "fancy custom logics" which are not available outside helpers. I
don't however believe it'll be prevalent.

In other words, +0 from me.

-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Fix libatomic detection

[Kinkie]

Hi,
  I haven't checked, but it should.
MacOS builds for me right now; probably as a result of the LIBS
side-effect (although I get a "none required" result).
Attached patch fixes, build still failes due to shm not working at
runtime (thus testRock failing), but for me it's a fix candidate.

On Tue, Jan 5, 2016 at 9:06 AM, Amos Jeffries <squ...@treenet.co.nz> wrote:
> On 5/01/2016 7:34 a.m., Kinkie wrote:
>> On Mon, Jan 4, 2016 at 6:47 AM, Amos Jeffries <squ...@treenet.co.nz> wrote:
>>> On 28/12/2015 10:25 p.m., Kinkie wrote:
>>>> Hi,
>>>>
>>>> On Mon, Dec 28, 2015 at 1:32 AM, Amos Jeffries wrote:
>>>>> On 24/12/2015 11:32 a.m., Kinkie wrote:
>>>>>> Hi,
>>>>>>   libatomic detection is currently broken in configure.ac; it will
>>>>>> define -latomic even in case where it wouldn't be required (e.g.
>>>>>> because it's already provided by the compiler).
>>>>>> The attached patch fixes the broken case. Unfortunately I don't know a
>>>>>> system where this library is required, I'm convinced there's room for
>>>>>> further simplification.
>>>>>
>>>>> It was for Clang builds IIRC. At least 3.5 needs it added.
>>>>>
>>>>> Does this work instead for shorter code?
>>>>>   AC_SEARCH_LIBS([__atomic_load_8],[atomic],[
>>>>> ATOMICLIB="$ac_cv_search___atomic_load_8"
>>>>>   ],[])
>>>>
>>>> unfortunately not, as the result could be "none required" or "no",
>>>> too. The former is actually the error case which bought me to the
>>>> topic.
>>>> Documentation states that if a library is needed it should get
>>>> automatically added to LIBS though.
>>>
>>> "Unless the 3rd and 4th parameters are specified". So it will be
>>> automatically added under your patch, but not under the existing/older code.
>>>
>>> Which was an intentional omission from LIBS to avoid build warnings on
>>> some distros about unnecessary dependencies being linked in (such as
>>> <https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=770928>). Since most
>>> of the helper binaries do not use atomics (yet, though we could thread
>>> them in future) the -latomic addition is only needed by squid_LDADD not
>>> the generic LIBS / LDADD.
>>>
>>> Amos
>>>
>>
>> Are you sure? This from the doc I could find:
>> ===
>> the default action-if-found code, adding the library to the LIBS
>> variables, is always executed, even if a parameter is passed
>> ===
>>
>> I have no objection on the intention, of course.
>> If you are right, then the one-liner below should then do the trick.
>> If I am right, we need to go through the
>> SQUID_SAVE_STATE/SQUID_ROLLBACK_STATE dance on top of that.
>>
>> === modified file 'configure.ac'
>> --- configure.ac2016-01-04 14:39:06 +
>> +++ configure.ac2016-01-04 18:25:48 +
>> @@ -458,7 +458,7 @@
>>  fi
>>
>>  ## check for atomics library before anything that might need it
>> -AC_SEARCH_LIBS([__atomic_load_8],[atomic])
>> +AC_SEARCH_LIBS([__atomic_load_8],[atomic],[],[])
>>  if test "x$ac_cv_search___atomic_load_8" = "-latomic"; then
>>ATOMICLIB="-latomic"
>>  fi
>>
>
> Seems I was wrong. I have just checked the two versions and the above
> patch generates exactly the same code as current trunk and sets LIBS. So
> we will have to do the state preservation dance. But I think wait until
> we have evidence of complaints for that.
>
> I also see the test is missing the "x" on the RHS of the conditional.
> Which is a bigger problem for clang where it needs to be set.
>
> PS. this is all fixes bug 4393 right?
>
> Amos
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev



-- 
Francesco
=== modified file 'configure.ac'
--- configure.ac	2016-01-04 14:39:06 +
+++ configure.ac	2016-01-05 08:14:39 +
@@ -458,10 +458,13 @@
 fi
 
 ## check for atomics library before anything that might need it
+# AC_SEARCH_LIBS pollutes LIBS
+SQUID_STATE_SAVE(LIBATOMIC)
 AC_SEARCH_LIBS([__atomic_load_8],[atomic])
-if test "x$ac_cv_search___atomic_load_8" = "-latomic"; then
+if test "x$ac_cv_search___atomic_load_8" = "x-latomic"; then
   ATOMICLIB="-latomic"
 fi
+SQUID_STATE_ROLLBACK(LIBATOMIC)
 AC_SUBST(ATOMICLIB)
 
 AC_SEARCH_LIBS([shm_open], [rt])

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Fix libatomic detection

[Kinkie]

looks good. +1

On Tue, Jan 5, 2016 at 10:01 AM, Amos Jeffries <squ...@treenet.co.nz> wrote:
> On 5/01/2016 9:24 p.m., Kinkie wrote:
>> Hi,
>>   I haven't checked, but it should.
>> MacOS builds for me right now; probably as a result of the LIBS
>> side-effect (although I get a "none required" result).
>> Attached patch fixes, build still failes due to shm not working at
>> runtime (thus testRock failing), but for me it's a fix candidate.
>
> This one matches the way AC_SEARCH_LIBS sets LIBS internally to cut off
> a few more lines of logic.
>
> Amos



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] search for header files in well-known local directories

[Kinkie]

On Mon, Jan 4, 2016 at 6:38 AM, Amos Jeffries  wrote:

> The auto-conf design for using custom directories (such as these /opt
> areas) is that the builder supplies the pth parameter
> (--with-gnutls=/opt/local/gnutls) when such local customized installs
> are to be used.

There is a problem with this: our automated tests do not allow for
setting these parameters, except by going through some hoops.
The code is already in configure.ac, just documented as '--without-gnutls'.

But I believe I found the actual issue, fixed by the attached patch.
There's two problems:
- headers are checked before PKG_CHECK_MODULES
- PKG_CHECK_MODULES sets include-related flags in CFLAGS, but then
AC_CHECK_HEADERS uses CPPFLAGS, not CFLAGS.

The patch addresses both by:
- calling PKG_CHECK_MODULES before AC_CHECK_HEADERS
- extending SQUID_STATE_* macros to also cover CPPFLAGS
- manually setting CPPFLAGS for the benefit of AC_CHECK_HEADERS (and
rolling it back once the check is done).


-- 
Francesco
=== modified file 'acinclude/squid-util.m4'
--- acinclude/squid-util.m4 2016-01-01 00:12:18 +
+++ acinclude/squid-util.m4 2016-01-04 09:48:31 +
@@ -1,88 +1,91 @@
 ## Copyright (C) 1996-2016 The Squid Software Foundation and contributors
 ##
 ## Squid software is distributed under GPLv2+ license and includes
 ## contributions from numerous individuals and organizations.
 ## Please see the COPYING and CONTRIBUTORS files for details.
 ##
 
 dnl save main environment variables to variables to the namespace defined by 
the
 dnl first argument (prefix)
 dnl e.g. SQUID_SAVEFLAGS([foo]) will save CFLAGS to foo_CFLAGS etc.
 dnl Saved variables are:
 dnl CFLAGS, CXXFLAGS, LDFLAGS, LIBS plus any variables specified as
 dnl second argument
 AC_DEFUN([SQUID_STATE_SAVE],[
 # save state, key is $1
 $1_CFLAGS="${CFLAGS}"
 $1_CXXFLAGS="${CXXFLAGS}"
 $1_LDFLAGS="${LDFLAGS}"
 $1_LIBS="${LIBS}"
 $1_CC="${CC}"
 $1_CXX="${CXX}"
+$1_CPPFLAGS="${CPPFLAGS}"
 $1_squid_saved_vars="$2"
 for squid_util_var_tosave in $$1_squid_saved_vars
 do
 squid_util_var_tosave2="$1_${squid_util_var_tosave}"
 eval "${squid_util_var_tosave2}=\"${squid_util_var_tosave}\""
 done
 ])
 
 dnl commit the state changes: deleting the temporary state defined in 
SQUID_STATE_SAVE
 dnl with the same prefix. It's not necessary to specify the extra variables 
passed
 dnl to SQUID_STATE_SAVE again, they will be automatically reclaimed.
 AC_DEFUN([SQUID_STATE_COMMIT],[
 # commit state, key is $1
 unset $1_CFLAGS
 unset $1_CXXFLAGS
 unset $1_LDFLAGS
 unset $1_LIBS
 unset $1_CC
 unset $1_CXX
+unset $1_CPPFLAGS
 for squid_util_var_tosave in $$1_squid_saved_vars
 do
 unset ${squid_util_var_tosave}
 done
 ])
 
 dnl rollback state to the call of SQUID_STATE_SAVE with the same namespace 
argument.
 dnl all temporary state will be cleared, including the custom variables 
specified
 dnl at call time. It's not necessary to explicitly name them, they will be 
automatically
 dnl cleared.
 AC_DEFUN([SQUID_STATE_ROLLBACK],[
 # rollback state, key is $1
 CFLAGS="${$1_CFLAGS}"
 CXXFLAGS="${$1_CXXFLAGS}"
 LDFLAGS="${$1_LDFLAGS}"
 LIBS="${$1_LIBS}"
 CC="${$1_CC}"
 CXX="${$1_CXX}"
+CPPFLAGS="${$1_CPPFLAGS}"
 for squid_util_var_tosave in $$1_squid_saved_vars
 do
 squid_util_var_tosave2="\$$1_${squid_util_var_tosave}"
 eval "$squid_util_var_tosave=\"${squid_util_var_tosave2}\""
 done
 SQUID_STATE_COMMIT($1)
 ])
 
 
 dnl look for modules in the base-directory supplied as argument.
 dnl fill-in the variable pointed-to by the second argument with the
 dnl space-separated list of modules 
 AC_DEFUN([SQUID_LOOK_FOR_MODULES],[
 $2=""
 for dir in $1/*; do
   module="`basename $dir`"
   if test -d "$dir" && test "$module" != CVS; then
   $2="$$2 $module"
   fi
 done
 ])
 
 dnl remove duplicates out of a list.
 dnl dnl argument is the name of a variable to be checked and cleaned up
 AC_DEFUN([SQUID_CLEANUP_MODULES_LIST],[
 squid_cleanup_tmp_outlist=""
 for squid_cleanup_tmp in $$1
 do
   squid_cleanup_tmp_dupe=0
   for squid_cleanup_tmp2 in $squid_cleanup_tmp_outlist

=== modified file 'configure.ac'
--- configure.ac2016-01-03 10:45:27 +
+++ configure.ac2016-01-04 09:48:37 +
@@ -1210,73 +1210,75 @@
 
 dnl Check for libcrypt
 CRYPTLIB=
 dnl Some of our helpers use crypt(3) which may be in libc, or in
 dnl libcrypt (eg FreeBSD)
 AC_CHECK_LIB(crypt, crypt, [CRYPTLIB="-lcrypt"])
 dnl Solaris10 provides MD5 natively through libmd5
 AC_CHECK_LIB(md5, MD5Init, [CRYPTLIB="$CRYPTLIB -lmd5"])
 AC_SUBST(CRYPTLIB)
 
 SSLLIB=""
 
 dnl User may want to disable GnuTLS
 AC_ARG_WITH(gnutls,
   AS_HELP_STRING([--without-gnutls],
  [Do not use GnuTLS for SSL. Default: auto-detect]), [ 
 case "$with_gnutls" in
   yes|no)
 : # Nothing special to do here
 ;;
   *)
 if test ! -d "$withval" ; then
   AC_MSG_ERROR([--with-gnutls path does not point to a directory])
 fi
 LIBGNUTLS_PATH="-L$with_gnutls/lib"
 

Re: [squid-dev] [PATCH] MacOS MIT Kerberos requires libresolv

[Kinkie]

Just for clarity: Squid isn't failing. The issue is in the
kerberos_ldap_group helper.

On Mon, Jan 4, 2016 at 8:58 AM, Kinkie <gkin...@gmail.com> wrote:
> It is, however:
>
> $ pkg-config --libs mit-krb5
> -L/opt/local/lib -lkrb5 -lk5crypto -lcom_err
>
> :(
>
> On Mon, Jan 4, 2016 at 6:10 AM, Amos Jeffries <squ...@treenet.co.nz> wrote:
>> On 4/01/2016 1:01 a.m., Kinkie wrote:
>>> Hi,
>>>   the attached patch improves the build on MacOS, where MIT krb5
>>> libraries require explicit linking of libresolv.
>>>
>>
>> Is pkg-config available and working for krb5 libraries on the MacOS
>> system(s) where this library needs to be checked explicitly?
>>
>> If pkg-config is not available there, the library check should be moved
>> up to the list of library checks between lines 1446 and 1465.
>>
>> Amos
>>
>> ___
>> squid-dev mailing list
>> squid-dev@lists.squid-cache.org
>> http://lists.squid-cache.org/listinfo/squid-dev
>
>
>
> --
> Francesco



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] MacOS MIT Kerberos requires libresolv

[Kinkie]

On Mon, Jan 4, 2016 at 2:29 PM, Markus Moeller <hua...@moeller.plus.com> wrote:
> Hi Kinkie,
>
>   I wonder against which Kerberos library SASL is linked against.  You may
> get strange errors if SASL which is used by ldap is linked against the
> native Kerberos libraries.  So the kerberos_ldap_group helper may not work
> correctly for SASL/GSSAPI based authentication to the ldap server.

Hi,

$ otool -L /usr/lib/libsasl2.2.dylib
/usr/lib/libsasl2.2.dylib:
/usr/lib/libsasl2.2.dylib (compatibility version 3.0.0, current
version 3.15.0)
/usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8,
current version 50.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1197.1.1)

System kerberos seems to be in /usr/lib and
/System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos . I
can't find the SASL libraries; they must be embedded somewhere else :\


-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] search for header files in well-known local directories

[Kinkie]

>> But I believe I found the actual issue, fixed by the attached patch.
>> There's two problems:
>> - headers are checked before PKG_CHECK_MODULES
>> - PKG_CHECK_MODULES sets include-related flags in CFLAGS, but then
>> AC_CHECK_HEADERS uses CPPFLAGS, not CFLAGS.
>
> Arg. Bug in PKG_CHECK_MODULES. It should be setting both.
>
> Maybe we should follow William Pursell's method as outlined in his
> comment to the question
> .

It's messy, need to un-cache values to make it work :(
Let's not fix a problem which is not yet apparent; let's keep the
reference around if it gets bad, but until then let's KIS.

>> The patch addresses both by:
>> - calling PKG_CHECK_MODULES before AC_CHECK_HEADERS
>> - extending SQUID_STATE_* macros to also cover CPPFLAGS
>> - manually setting CPPFLAGS for the benefit of AC_CHECK_HEADERS (and
>> rolling it back once the check is done).
>>
>
> +1 on this new patch.

committed as 14476.

> Also need to check and fix the OpenSSL use of PKG_CHECK_MODULES which is
> probably suffering in the same way.

On MacOS openssl stuff is in /usr/include/openssl and /usr/lib; I have
no way to check.

> The krb5 libraries might be too, but do not use AC_CHECK_HEADERS in the
> same way.

Kerberos librararies are way messy. Kudos to Markus for being able to
navigate through that.

-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] MacOS MIT Kerberos requires libresolv

[Kinkie]

> Could it be that the MIT header is used during configure instead of the MAC
> Kerberos header files ?

Doh! You are right  - I had a stale config.cache

Thanks for pointing it out!

  Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] improve CharacterSet

[Kinkie]

>> This build order change should not be a part of the "improve
>> CharacterSet" patch because it has nothing to do with improving
>> CharacterSet API. Whether helpers are allowed to use src/ is a separate
>> question, implicitly being discussed on the "SBuf API" thread right now.
>
> Aye. Reverting the SUBDIRS change that got applied in rev.14472.
>
> I notice that DIST_SUBDIRS and SUBDIRS have a mismatch with the ordering
> of test-suite and tools directories. That may be causing some build or
> packaging oddities, and IMO should be fixed too if we can agree on best
> way to do that.

Ok.
In the meantime I am applying the same workaround implemented
elsewhere: copying files over.

-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Fix libatomic detection

[Kinkie]

On Mon, Jan 4, 2016 at 6:47 AM, Amos Jeffries <squ...@treenet.co.nz> wrote:
> On 28/12/2015 10:25 p.m., Kinkie wrote:
>> Hi,
>>
>> On Mon, Dec 28, 2015 at 1:32 AM, Amos Jeffries wrote:
>>> On 24/12/2015 11:32 a.m., Kinkie wrote:
>>>> Hi,
>>>>   libatomic detection is currently broken in configure.ac; it will
>>>> define -latomic even in case where it wouldn't be required (e.g.
>>>> because it's already provided by the compiler).
>>>> The attached patch fixes the broken case. Unfortunately I don't know a
>>>> system where this library is required, I'm convinced there's room for
>>>> further simplification.
>>>
>>> It was for Clang builds IIRC. At least 3.5 needs it added.
>>>
>>> Does this work instead for shorter code?
>>>   AC_SEARCH_LIBS([__atomic_load_8],[atomic],[
>>> ATOMICLIB="$ac_cv_search___atomic_load_8"
>>>   ],[])
>>
>> unfortunately not, as the result could be "none required" or "no",
>> too. The former is actually the error case which bought me to the
>> topic.
>> Documentation states that if a library is needed it should get
>> automatically added to LIBS though.
>
> "Unless the 3rd and 4th parameters are specified". So it will be
> automatically added under your patch, but not under the existing/older code.
>
> Which was an intentional omission from LIBS to avoid build warnings on
> some distros about unnecessary dependencies being linked in (such as
> <https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=770928>). Since most
> of the helper binaries do not use atomics (yet, though we could thread
> them in future) the -latomic addition is only needed by squid_LDADD not
> the generic LIBS / LDADD.
>
> Amos
>

Are you sure? This from the doc I could find:
===
the default action-if-found code, adding the library to the LIBS
variables, is always executed, even if a parameter is passed
===

I have no objection on the intention, of course.
If you are right, then the one-liner below should then do the trick.
If I am right, we need to go through the
SQUID_SAVE_STATE/SQUID_ROLLBACK_STATE dance on top of that.

=== modified file 'configure.ac'
--- configure.ac2016-01-04 14:39:06 +
+++ configure.ac2016-01-04 18:25:48 +
@@ -458,7 +458,7 @@
 fi

 ## check for atomics library before anything that might need it
-AC_SEARCH_LIBS([__atomic_load_8],[atomic])
+AC_SEARCH_LIBS([__atomic_load_8],[atomic],[],[])
 if test "x$ac_cv_search___atomic_load_8" = "-latomic"; then
   ATOMICLIB="-latomic"
 fi



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] MacOS MIT Kerberos requires libresolv

[Kinkie]

It is, however:

$ pkg-config --libs mit-krb5
-L/opt/local/lib -lkrb5 -lk5crypto -lcom_err

:(

On Mon, Jan 4, 2016 at 6:10 AM, Amos Jeffries <squ...@treenet.co.nz> wrote:
> On 4/01/2016 1:01 a.m., Kinkie wrote:
>> Hi,
>>   the attached patch improves the build on MacOS, where MIT krb5
>> libraries require explicit linking of libresolv.
>>
>
> Is pkg-config available and working for krb5 libraries on the MacOS
> system(s) where this library needs to be checked explicitly?
>
> If pkg-config is not available there, the library check should be moved
> up to the list of library checks between lines 1446 and 1465.
>
> Amos
>
> ___
> squid-dev mailing list
> squid-dev@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

Re: [squid-dev] [PATCH] Fix libatomic detection

[Kinkie]

10 days without vetoes, committing as revno 14474.

On Mon, Dec 28, 2015 at 10:25 AM, Kinkie <gkin...@gmail.com> wrote:
> Hi,
>
> On Mon, Dec 28, 2015 at 1:32 AM, Amos Jeffries <squ...@treenet.co.nz> wrote:
>> On 24/12/2015 11:32 a.m., Kinkie wrote:
>>> Hi,
>>>   libatomic detection is currently broken in configure.ac; it will
>>> define -latomic even in case where it wouldn't be required (e.g.
>>> because it's already provided by the compiler).
>>> The attached patch fixes the broken case. Unfortunately I don't know a
>>> system where this library is required, I'm convinced there's room for
>>> further simplification.
>>
>> It was for Clang builds IIRC. At least 3.5 needs it added.
>>
>> Does this work instead for shorter code?
>>   AC_SEARCH_LIBS([__atomic_load_8],[atomic],[
>> ATOMICLIB="$ac_cv_search___atomic_load_8"
>>   ],[])
>
> unfortunately not, as the result could be "none required" or "no",
> too. The former is actually the error case which bought me to the
> topic.
> Documentation states that if a library is needed it should get
> automatically added to LIBS though.
>
>
> --
> Francesco



-- 
Francesco
___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] [PATCH] search for header files in well-known local directories

[Kinkie]

Hi,
  the attached patch for configure.ac tries to locate local header
files in system-local directories.
It fixes the build on MacOS/MacPorts which places GnuTLS headers in
/opt/local/include.
The search is not performed if the builder predefines CPPFLAGS.

-- 
Francesco
=== modified file 'configure.ac'
--- configure.ac2016-01-03 10:45:27 +
+++ configure.ac2016-01-03 10:58:12 +
@@ -31,11 +31,21 @@
 PRESET_CXXFLAGS="$CXXFLAGS"
 PRESET_LDFLAGS="$LDFLAGS"
 
-dnl Set default LDFLAGS
+# Set default LDFLAGS
 if test "x$LDFLAGS" = "x" ; then
   LDFLAGS="-g"
 fi
 
+# Look for system-local headers
+if test "x$CPPFLAGS" = "x"; then
+  for d in /usr/local /opt/local /opt
+  do
+if test -d $d/include; then
+  CPPFLAGS="$CPPFLAGS -I$d/include"
+fi
+  done
+fi
+
 # Check for GNU cc
 AC_PROG_CC
 AM_PROG_CC_C_O

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev

[squid-dev] [PATCH] MacOS MIT Kerberos requires libresolv

[Kinkie]

Hi,
  the attached patch improves the build on MacOS, where MIT krb5
libraries require explicit linking of libresolv.

-- 
Francesco
=== modified file 'configure.ac'
--- configure.ac	2016-01-03 10:45:27 +
+++ configure.ac	2016-01-03 11:59:49 +
@@ -1478,6 +1478,7 @@
   AC_DEFINE(USE_MIT_KRB5,1,[MIT Kerberos support is available])
   KRB5_FLAVOUR="MIT" 
 fi
+AC_CHECK_LIB(resolv, [res_9_search], [LIB_KRB5_LIBS="-lresolv $LIB_KRB5_LIBS"],[])
 KRB5LIBS="$LIB_KRB5_PATH $LIB_KRB5_LIBS $KRB5LIBS"
 KRB5INCS="$LIB_KRB5_CFLAGS"
 

___
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev