Re: apreq release

[Brian J. France]

This is just merging the perl stuff into mod_perl, right?

Not merging mod_apreq2 and all the request cache/re-play bucket, POST reading, 
file uploading, etc stuff, right?

I really don't want to have to include mod_perl so my C modules can read POST 
data and handle file uploads.

Cheers,

Brian


> On Nov 16, 2016, at 4:42 AM, Issac Goldstand  wrote:
> 
> Given that the C was (finally) merged into httpd years ago, and given
> that there are no proposed code changes, I'd say that's not such a bad
> idea...
> 
> I've become a bit rusty in Perl (and even with apreq) over the years,
> but IIRC, all of the Perl glue is in
> http://svn.apache.org/repos/asf/httpd/apreq/trunk/glue/perl/
> 
> I'll take a crack at seeing if I can fold it into mod_perl despite the rust.
> 
> 
> On 11/15/2016 2:45 PM, Andres Thomas Stivalet wrote:
>> Good news!! No idea why apreq hasn't just been merged into mod_perl
>> after all these years.
>> 
>> A+++
>> 
>> On Nov 15, 2016 3:27 AM, "Issac Goldstand" > > wrote:
>> 
>>Hi all,
>> 
>>Someone (finally) noticed that apreq's test suite isn't compatible with
>>Apache 2.4 and requested a change.  Given that we haven't released an
>>updated apreq in nearly 6 years, I'm inclined to make/test the changes
>>to the test suite and immediately go to a release cycle.
>> 
>>Does anyone want time to add anything else to libapreq-2.14 before I
>>start tarring and voting (in the next few days, I hope)?
>> 
>> 
>>-
>>To unsubscribe, e-mail: dev-unsubscr...@perl.apache.org
>>
>>For additional commands, e-mail: dev-h...@perl.apache.org
>>
>> 
> 

Re: apreq release

[Brian J. France]

This is just merging the perl stuff into mod_perl, right?

Not merging mod_apreq2 and all the request cache/re-play bucket, POST reading, 
file uploading, etc stuff, right?

I really don't want to have to include mod_perl so my C modules can read POST 
data and handle file uploads.

Cheers,

Brian


> On Nov 16, 2016, at 4:42 AM, Issac Goldstand  wrote:
> 
> Given that the C was (finally) merged into httpd years ago, and given
> that there are no proposed code changes, I'd say that's not such a bad
> idea...
> 
> I've become a bit rusty in Perl (and even with apreq) over the years,
> but IIRC, all of the Perl glue is in
> http://svn.apache.org/repos/asf/httpd/apreq/trunk/glue/perl/
> 
> I'll take a crack at seeing if I can fold it into mod_perl despite the rust.
> 
> 
> On 11/15/2016 2:45 PM, Andres Thomas Stivalet wrote:
>> Good news!! No idea why apreq hasn't just been merged into mod_perl
>> after all these years.
>> 
>> A+++
>> 
>> On Nov 15, 2016 3:27 AM, "Issac Goldstand" > > wrote:
>> 
>>Hi all,
>> 
>>Someone (finally) noticed that apreq's test suite isn't compatible with
>>Apache 2.4 and requested a change.  Given that we haven't released an
>>updated apreq in nearly 6 years, I'm inclined to make/test the changes
>>to the test suite and immediately go to a release cycle.
>> 
>>Does anyone want time to add anything else to libapreq-2.14 before I
>>start tarring and voting (in the next few days, I hope)?
>> 
>> 
>>-
>>To unsubscribe, e-mail: dev-unsubscr...@perl.apache.org
>>
>>For additional commands, e-mail: dev-h...@perl.apache.org
>>
>> 
> 

Re: "httpd -X" segfaults with 2.4.17

[Brian J. France]

This hasn't made it into the 2.4.x branch yet, what is the status of getting 
this in the .19 release?

http://svn.apache.org/viewvc?view=revision=1711479

Thanks,

Brian


> On Oct 16, 2015, at 8:06 AM, Yann Ylavic  wrote:
> 
> Hi Jan,
> 
> On Fri, Oct 16, 2015 at 1:58 PM, Jan Kaluža  wrote:
>> Hi,
>> 
>> httpd 2.4.17 segfaults when used with prefork MPM (and probably also with
>> other MPMs) and -X option since r1705492.
>> 
>> The crash happens in the following call in prefork.c (and probably also
>> worker.c and so on):
>> 
>>ap_mpm_pod_check(my_bucket->pod)
>> 
>> pod is NULL and later dereferenced.
>> 
>> The pod is NULL because of the following:
>> 
>>if (!one_process && /* no POD in one_process mode */
>>(rv = ap_mpm_pod_open(pconf, _buckets[i].pod))) {
> 
> This is a bad copy-and-paste from the worker/event logic, which I'm to
> blame for...
> I think we should simply remove the !one_process check here.
> 
> Regards,
> Yann.
> 

Re: Looking ahead to 2.4.13 / 2.2.30

[Brian J. France]

While you are in mod_dav, could you review these patches and see if it makes 
sense to add them?

httpd-2.2.x : http://www.brianfrance.com/software/apache/dav/mod_dav_fs.diff.22
httpd-2.4.x : http://www.brianfrance.com/software/apache/dav/mod_dav_fs.diff.24

We have been running these for a while at work, just haven't had free time to 
send them.

Thanks!

Brian


 On May 1, 2015, at 3:29 PM, Ben Reser b...@reser.org wrote:
 
 On 4/30/15 2:52 PM, William A Rowe Jr wrote:
 It seems that we have 2 groups of good things to come out of ApacheCon,
 some immediate fixes for things like BSD project efforts, some pretty
 straightforward defects that have been resolved... and then there's a bunch
 of energy about enhancements and the h2 universe.
 
 In the short term, WDYT about giving the trees a week to settle, grab the
 low hanging fruit and move forward for 2.4.13 and 2.2.30 end of this coming
 week?  Guessing Jim's up for RM'ing 2.4.13, and I'm happy to TR 2.2.30 
 in tandem.
 
 Concerns / observations / thoughts?
 
 I have a mod_dav fix that really ought to be in the next 2.4 release.  I'll 
 get
 it committed and nominated sometime this weekend.
 

Re: apr_socket_opt_set always sets TCP_DEFER_ACCEPT to 1 (was @apr)

[Brian J. France]

On Apr 30, 2014, at 6:54 AM, Yann Ylavic ylavic@gmail.com wrote:

 -- Forwarded message --
 From: Yann Ylavic ylavic@gmail.com
 Date: Wed, Apr 30, 2014 at 11:59 AM
 Subject: Re: apr_socket_opt_set always sets TCP_DEFER_ACCEPT to 1
 To: APR Developer List d...@apr.apache.org
 
 On Tue, Apr 29, 2014 at 5:41 PM, Jim Jagielski j...@jagunet.com wrote:
 
 On Apr 22, 2014, at 9:43 AM, Brian J. France br...@brianfrance.com wrote:
 
 For some reason I completely missed that.  APR is fine, but httpd needs 
 updated as it is hard coded to 1:
 
 rv = apr_socket_opt_set(s, APR_TCP_DEFER_ACCEPT, 1);
 
 Will move to the httpd list.
 
 
 Did I miss this on dev@httpd?
 
 I seems that httpd trunk and 2.4.x use apr_socket_opt_set(s,
 APR_TCP_DEFER_ACCEPT, 30); whereas 2.2.x uses a value of 1.
 
 This come from PR
 http://issues.apache.org/bugzilla/show_bug.cgi?id=41270, commit
 http://svn.apache.org/viewvc?view=revisionrevision=501364, which
 since then has been forked to 2.4.x, and never merged in 2.2.x.
 
 Linux (code and mostly comment) is quite clear on the setsockopt() value.
 In master/net/ipv4/tcp.c::do_tcp_setsockopt() :
case TCP_DEFER_ACCEPT:
/* Translate value in seconds to number of retransmits */
icsk-icsk_accept_queue.rskq_defer_accept =
secs_to_retrans(val, TCP_TIMEOUT_INIT / HZ,
TCP_RTO_MAX / HZ);
break;
 
 One second is too low imo (cf. PR above).


Yes, 1 is way to low, even 60 doesn't help us with the RHEL 6 kernel bug.

I am trying to get time to work on a patch that would allow something like this:

  AcceptFilter data:60

for 2.2.x.  That is the only way I can see to allow tweaking the time and 
staying binary compatible with 2.2.x.

Original thought was to change AcceptFilter to a AP_INIT_TAKE23 and add a 
accf_opt_map table, but that would change the conf struct size and break binary 
compatibility for 2.2.x.

Brian

Re: apreq_parse_generic example

[Brian J. France]

Finally got some working code. Registered the parser for the content type:

  apreq_register_parser(application/json, apreq_parse_generic);

in a pre_init hook, then normal apreq_handle_apache2 and apreq_body calls, then 
a apreq_parser_get call to get the parser and:

  dummy = *(apreq_param_t **)parser-ctx;
  apr_brigade_pflatten(dummy-upload, val, vlen, r-pool);

to get the string of the body in val/vlen.  Is there a better way?

Thanks,

Brian

On Nov 9, 2013, at 9:32 AM, Joseph Schaefer joe_schae...@yahoo.com wrote:

 The generic parser stores the POST data in the upload
 brigade of a  param named “_dummy_” but it’s not in
 the output table, its in the parser’s ctx pointer.
 The struct is declared in parser.c so it’s technically
 private but the param is in the first slot and that’s
 meant to be public.
 
 On Nov 8, 2013, at 1:02 PM, Joseph Schaefer joe_schae...@yahoo.com wrote:
 
 Been a while since I thought about this, but
 apreq is really only suited for table-based
 data structures.  JSON is a little too generic
 for that unless you know the data is restricted
 to only key-value pairs.
 
 On Nov 8, 2013, at 12:45 PM, Brian J. France br...@brianfrance.com wrote:
 
 Does anybody have example code of how to use apreq_parse_generic in an 
 httpd module hook (non-handler)?
 
 Need to use apreq2 and mod_apreq2 to read json data and not having any luck.
 
 Thanks,
 
 Brian
 
 - Please keep me looped into any reply, not subscribed.
 
 
 

apreq_parse_generic example

[Brian J. France]

Does anybody have example code of how to use apreq_parse_generic in an httpd 
module hook (non-handler)?

Need to use apreq2 and mod_apreq2 to read json data and not having any luck.

Thanks,

Brian

apreq_parse_generic example

[Brian J. France]

Does anybody have example code of how to use apreq_parse_generic in an httpd 
module hook (non-handler)?

Need to use apreq2 and mod_apreq2 to read json data and not having any luck.

Thanks,

Brian

 - Please keep me looped into any reply, not subscribed.

Re: apreq_parse_generic example

[Brian J. France]

Right, which is why I thought the apreq_parse_generic would handle that and I 
can pull the full string out and handle it myself.

What I really need is a way for a hook to read the post data and the content 
handler to also be able to read it as well.  mod_apreq allowed this for types 
it knows about (application/x-www-form-urlencoded, multipart/form-data, etc), 
but not it doesn't know about like application/json.

Is there another way with apreq2/mod_apreq or ap_setup_client_block, 
ap_should_client_block, ap_get_client_block that both hooks get the data?

Brian


On Nov 8, 2013, at 1:02 PM, Joseph Schaefer joe_schae...@yahoo.com wrote:

 Been a while since I thought about this, but
 apreq is really only suited for table-based
 data structures.  JSON is a little too generic
 for that unless you know the data is restricted
 to only key-value pairs.
 
 On Nov 8, 2013, at 12:45 PM, Brian J. France br...@brianfrance.com wrote:
 
 Does anybody have example code of how to use apreq_parse_generic in an httpd 
 module hook (non-handler)?
 
 Need to use apreq2 and mod_apreq2 to read json data and not having any luck.
 
 Thanks,
 
 Brian
 
 - Please keep me looped into any reply, not subscribed.
 
 

Re: [DISCUSS] CMS site migration

[Brian J. France]

Do you have details on the on the new CMS, format, conversions, etc?  We us the 
httpd current format at work for our internal modules and might want to 
transition to the CMS as well.

Thanks,

Brian

On May 6, 2012, at 5:39 PM, Joe Schaefer wrote:

 Over on docs@ one of the recent conversations was
 around moving the site documentation to the CMS,
 starting first with the httpd site as a testbed.
 After several hours of hacking on the site that
 has now been accomplished, so we'd please like everyone
 to review and comment on the httpd staging site now
 available at 
 
 http://httpd.staging.apache.org/
 
 which is perfectly compatible with the CMS's bookmarklet.
 There are a few remaining syntax/style issues that need
 addressing, but otherwise the content has been successfully
 migrated from xdoc to markdown.
 
 The sooner we can push this work into production the
 less hassle it will be to keep the xdoc and content
 trees in sync using two separate build systems.
 
 After a few days have passed if there are no outstanding
 issues remaining I plan to ask for a VOTE to finish the
 migration of httpd-site to the CMS.  Thanks in advance
 for your consideration!
 
 

Re: mod_dav_fs does not check for return value on stream_close

[Brian J. France]

Yes, you are correct.  Looks like my merge from work code to httpd svn didn't 
fully work.

We have been running this patch for a week or so with no issues.

Brian


On Apr 4, 2012, at 6:24 PM, Graham Leggett wrote:

 On 15 Mar 2012, at 3:56 PM, Brian J. France wrote:
 
 Could somebody review the patch below for 2.2, 2.4, and trunk?
 
 A better error message could be sent, but I am more worried about how the 
 return will effect the code after it.
 
 I am thinking the file needs to be removed either via a apr_file_remove call 
 or:
 
 apr_pool_cleanup_kill(stream-p, stream, tmpfile_cleanup);
 
 call, but I don't know the code well enough to know which is right and 
 2.4/trunk adds even more complexity compared to 2.2.x.
 
 Thoughts/Comments?
 
 Thanks,
 
 Brian
 
 - I am still getting more details why close is failing, but for some reason 
 it is happening enough to cause issues. (responding 200, but no file)
 
 -
 
 2.2:
 
 Index: modules/dav/fs/repos.c
 ===
 --- modules/dav/fs/repos.c   (revision 1300964)
 +++ modules/dav/fs/repos.c   (working copy)
 @@ -881,6 +881,10 @@
 {
apr_file_close(stream-f);
 
 Would be above not be
 
 status = apr_file_close(stream-f);
 
 
 +if (status != APR_SUCCESS) {
 +return dav_new_error(stream-p, MAP_IO2HTTP(status), 0, There was a 
 problem closing the stream);
 +}
 +
if (!commit) {
if (apr_file_remove(stream-pathname, stream-p) != APR_SUCCESS) {
/* ### use a better description? */
 
 
 2.24 and trunk:
 
 
 Index: modules/dav/fs/repos.c
 ===
 --- modules/dav/fs/repos.c   (revision 1300964)
 +++ modules/dav/fs/repos.c   (working copy)
 @@ -970,6 +970,10 @@
 
apr_file_close(stream-f);
 
 Same with this one.
 
 
 +if (status != APR_SUCCESS) {
 +return dav_new_error(stream-p, MAP_IO2HTTP(status), 0, There was a 
 problem closing the stream);
 +}
 +
if (!commit) {
if (stream-temppath) {
apr_pool_cleanup_run(stream-p, stream, tmpfile_cleanup);
 
 
 
 Regards,
 Graham
 --
 

mod_dav_fs does not check for return value on stream_close

[Brian J. France]

Could somebody review the patch below for 2.2, 2.4, and trunk?

A better error message could be sent, but I am more worried about how the 
return will effect the code after it.

I am thinking the file needs to be removed either via a apr_file_remove call or:

  apr_pool_cleanup_kill(stream-p, stream, tmpfile_cleanup);

call, but I don't know the code well enough to know which is right and 
2.4/trunk adds even more complexity compared to 2.2.x.

Thoughts/Comments?

Thanks,

Brian

 - I am still getting more details why close is failing, but for some reason it 
is happening enough to cause issues. (responding 200, but no file)

-

2.2:

Index: modules/dav/fs/repos.c
===
--- modules/dav/fs/repos.c  (revision 1300964)
+++ modules/dav/fs/repos.c  (working copy)
@@ -881,6 +881,10 @@
 {
 apr_file_close(stream-f);
 
+if (status != APR_SUCCESS) {
+   return dav_new_error(stream-p, MAP_IO2HTTP(status), 0, There was a 
problem closing the stream);
+}
+
 if (!commit) {
 if (apr_file_remove(stream-pathname, stream-p) != APR_SUCCESS) {
 /* ### use a better description? */


2.24 and trunk:


Index: modules/dav/fs/repos.c
===
--- modules/dav/fs/repos.c  (revision 1300964)
+++ modules/dav/fs/repos.c  (working copy)
@@ -970,6 +970,10 @@
 
 apr_file_close(stream-f);
 
+if (status != APR_SUCCESS) {
+   return dav_new_error(stream-p, MAP_IO2HTTP(status), 0, There was a 
problem closing the stream);
+}
+
 if (!commit) {
 if (stream-temppath) {
 apr_pool_cleanup_run(stream-p, stream, tmpfile_cleanup);

Re: WebDAV and ACL (RFC3744), status?

[Brian J. France]

I had started breaking up the patches from mod_dav_acl into smaller chunks and 
getting them imported into the trunk.

My goal was to get a mod_dav_acl like module added.  I say like because 
mod_dav_acl currently requires xfs and stores the auth information in the xfs 
attributes and I wanted to create a more authn type module.  Something that 
could have a flat file, dbm, dbd, etc type plugins.

After mod_dav_acl was done I wanted to get mod_caldav and mod_cardav imported 
as well, but free time dried up and I never finished.

Brian


On Feb 1, 2012, at 10:38 AM, Andreas wrote:

 Good evening.
 
 Where can I find out if httpd/mod_dav has support for ACL's?
 
 After digging in the mailinglist, there seem to have been some activity
 about the topic in 2007 and 2009 but no patches seem to be applied.
 
 I checked today on 2.3beta, there is no --enable-dav-acl option yet
 (unless enabled by default?).
 
 I could not find any bugzilla issue tracking the patches either, so now
 I ask here as a last resort if anyone knows status on it. :)
 
 
 Regards
 -- 
 Andreas
 
  ... Mental backup in progress - Do Not Disturb!
 

Re: httpd should be able to reopen log files without a restart.

[Brian J. France]

For some reason the docs for trunk haven't updated.

Here are the details for what is in apr (which is a little different than my 
patch for work):

apr_file_open() has two new flags:

  APR_FOPEN_ROTATING - Do file file rotation checking
  APR_FOPEN_MANUAL_ROTATE  - Enable Manual rotation

Two new functions where added (no docs yet):

  APR_DECLARE(apr_status_t) apr_file_rotating_check(apr_file_t *thefile);
  APR_DECLARE(apr_status_t) apr_file_rotating_manual_check(apr_file_t *thefile, 
apr_time_t time);

Code was added in apr_file_write() and apr_file_writev() to check to see if the 
file needs rotated (the log file in this case) before writing data to it.  If 
rotating is not enabled or set to manual it doesn't do anything, but if it is 
rotating then it checks to see if the now - lastcheck  than the timeout (60 
seconds default) and if it has then it stats the file and compares inode and 
device to make sure they are still the same.  If it finds they are different, 
then it re-opens the file, saves the new inode and device.

If manual check is enabled, then you have to call apr_file_rotating_check() or 
apr_file_rotating_manual_check() if you want a  different timeout.

So if we would added a option for adding APR_FOPEN_ROTATING to the ErrorLog, 
CustomLog, etc directives then we could get httpd to allow rotating logs (or 
may be adding a global setting to enable it by default).

Brian





On Oct 11, 2011, at 10:10 AM, crocket wrote:

 How exactly does it work? Will it be documented?
 
 On Tue, Oct 11, 2011 at 10:28 AM, Brian J. France br...@brianfrance.com 
 wrote:
 apr trunk has APR_FOPEN_ROTATING support (stat checking and re-open), which 
 is based on a patch we use at work.  This allows us to move the log file, 
 wait 90 seconds and then compress the moved log file, all without touch the 
 server as it will stat the log ever 60 and re-open it if needed before 
 writing a log line.
 
 Of course it is in trunk, so may take a bit to get it into a release.
 
 Brian
 
 
 On Oct 10, 2011, at 9:16 PM, crocket wrote:
 
 When some softwares(including stunnel) receive SIGUSR1, they reopen a
 log file without re-reading configuration files or restarting.
 Since httpd uses SIGUSR1 for graceful restart, it could use SIGUSR2
 purely for reopening log files.
 
 Why do people need this signal?
 
 A graceful restart doesn't kill active children processes, and those
 processes still write on an old log file.
 logrotate can be configured to compress old log files, in which case
 httpd would try to write onto a compressed log file.
 Since httpd can't log to a file, it would show abnormal behaviors or
 even abort downloads from httpd.
 
 So SIGUSR2 can come in handy.
 
 
 

Re: httpd should be able to reopen log files without a restart.

[Brian J. France]

apr trunk has APR_FOPEN_ROTATING support (stat checking and re-open), which is 
based on a patch we use at work.  This allows us to move the log file, wait 90 
seconds and then compress the moved log file, all without touch the server as 
it will stat the log ever 60 and re-open it if needed before writing a log line.

Of course it is in trunk, so may take a bit to get it into a release.

Brian


On Oct 10, 2011, at 9:16 PM, crocket wrote:

 When some softwares(including stunnel) receive SIGUSR1, they reopen a
 log file without re-reading configuration files or restarting.
 Since httpd uses SIGUSR1 for graceful restart, it could use SIGUSR2
 purely for reopening log files.
 
 Why do people need this signal?
 
 A graceful restart doesn't kill active children processes, and those
 processes still write on an old log file.
 logrotate can be configured to compress old log files, in which case
 httpd would try to write onto a compressed log file.
 Since httpd can't log to a file, it would show abnormal behaviors or
 even abort downloads from httpd.
 
 So SIGUSR2 can come in handy.

Filter Scoreboard Entries

[Brian J. France]

$DAY job requires filtering the scoreboard for things like username, passwords, 
credit card numbers (don't ask) and other sensitive information.  Currently I 
just added a patch to our build that adds a option function that I set in our 
filtering module.  While at ApacheCon I talked with Paul and we came up with 
this design:

http://www.brianfrance.com/software/apache/filter_scoreboard_entry.diff

It creates a small struct that has the pool and values to be filtered, if 
things changes after the hook_run call it then copies things back into the 
scoreboard.

Any reason to pass the request_rec instead of the pool?

Comments?

Brian

Re: Log file rotation patch

[Brian J. France]

On Nov 5, 2010, at 12:13 PM, Dirk-Willem van Gulik wrote:

 
 On 4 Nov 2010, at 21:26, Brian J. France wrote:
 
 With the current patch, see link below, it changes the syntax to ErrorLog to 
 this:
 
 ErrorLog file-path|syslog[:facility] [rotating[:interval]]
 
 Nice!
 
 There is one security issue that people may have a problem with in that the 
 directory path for the log file has to be writeable by the User that apache 
 drops privilege to.  This is because all the children will need to re-open 
 the log file and the first one will create it.
 
 That is a pretty big eek. Wondering if we need a logging child - but then one 
 would end up with the rotatelog utility again :)


Not that it changes anything much, but my description was wrong.  The directory 
path needs to be readable  by User up to the log directory and then the log 
directory needs to be writable by the User.

Brian

Log file rotation patch

[Brian J. France]

While at ApacheCon I am working on getting a patch for log file rotation and 
would like to get some feed back.

With the current patch, see link below, it changes the syntax to ErrorLog to 
this:

  ErrorLog file-path|syslog[:facility] [rotating[:interval]]

examples:

  ErrorLog logs/error_log rotating

  ErrorLog logs/error_log rotating:30

This enables rotation check to be preformed on the error log and allows 
changing of the check interval from the default 60 seconds.

The patch is using functionality from apr trunk and adds a new set function for 
the interval.

I still need to work out the CustomLog and TransferLog, I am thinking like this:

  CustomLog file|pipe format|nickname [env=[!]environment-variable] 
[rotating[:interval]]

There is one security issue that people may have a problem with in that the 
directory path for the log file has to be writeable by the User that apache 
drops privilege to.  This is because all the children will need to re-open the 
log file and the first one will create it.

This is all based on http-trunk and apr-trunk and will need to be split up, 
just looking for feed back.

If this does get accept, what are the chances of getting it in to 2.4 and the 
rotation code back ported to apr-1.5?

Brian

http://www.brianfrance.com/software/apache/rotate.diff

Re: HTTP trailers?

[Brian J. France]

For those following this thread, here is a update.

You can build a module that is able to insert a trailer by adding a filter and 
ap_hook_create_request call.

You have to have a ap_hook_create_request call that runs after http_core.c and 
loops over the r-output_filters and removes ap_content_length_filter_handle.

Then you have a AP_FTYPE_TRANSCODE+1 filter (chunk filter is AP_FTYPE_TRANSCODE 
and you want to be after) that loops over the brigade looking for a bucket that 
is:

  ASCII_ZERO ASCII_CRLF ASCII_CRLF

followed by a APR_BUCKET_IS_EOS bucket.

Remove it and replaces it with:

  ASCII_ZERO ASCII_CRLF tailer string ASCII_CRLF


and you have chunked encoding with a trailer.

Brian


On Apr 22, 2010, at 10:39 PM, Brian J. France wrote:
 On Apr 22, 2010, at 8:40 PM, Mark Nottingham wrote:
 I couldn't find any obvious way to set HTTP trailers in Apache 2.x without 
 taking over all response processing (a la nph).
 
 Did I miss something?
 
 
 I started hacking on this at work, but got the point where I can't insert a 
 filter in the right spot.
 
 The problem I ran into is that if the Content-Length header is set (which 
 ap_content_length_filter_handle sets), then the chunking filter doesn't kick 
 in.  Something about wanting to use sendfile so don't chunk.
 
 I can't get a filter inserted after ap_content_length_filter_handle, but 
 before ap_http_header_filter (which adds the chunking filter).
 
 My plan was to insert a filter that removes the Content-Length header so the 
 response will be chunked and add another filter after the chunking filter 
 that would search for ASCII_ZERO ASCII_CRLF ASCII_CRLF and insert the trailer 
 data before the last ASCII_CRLF.  This was just a proof of concept to see if 
 it would work and output data in the right format.
 
 If it all worked my plan was add a trailers hook and call the hook in 
 chunk_filter.c in place of the /* trailers */ comment.
 
 Brian
 
 After typing this up and working through it again, I think I can add a filter 
 AP_FTYPE_TRANSCODE+1 for the after chunking filter and AP_FTYPE_PROTOCOL+1 
 and get them to be in the right order.  Will test that out now.
 

Re: HTTP trailers?

[Brian J. France]

On Apr 23, 2010, at 10:08 AM, William A. Rowe Jr. wrote:

 On 4/23/2010 9:03 AM, Brian J. France wrote:
 
 You can build a module that is able to insert a trailer by adding a filter 
 and ap_hook_create_request call.
 
 But doesn't this defeat the purpose of using a modular server
 architecture?  It seems this should be a facility of the core HTTP
 filter, if anyone wants to offer the patch for 2.3.


I agree, my module was more of a proof of concept that I can do it and then get 
some other server to able able to use it.

Not sure what the best solution would be because multiple things need to 
happen.  First part is you have to force chunk encoding either by removing 
content_length filter or tweaking the code to not add it if doing a trailer 
(which you might not know until it is time to insert a tailer).

Then you have to tweak modules/http/chunk_filter.c to allow others to insert a 
trailer, like adding a ap_hook_http_trailer or a optional function for 
inserting it.  I don't know if multiple modules should be allowed to add a 
trailer, if you do how to you join them since a trailer is nothing but a string 
ending with ASCII_CRLF (just strcat?).  Should we just grab 
r-notes['http_trailer'] and let modules just add/set/append values?

I think there is a bigger design discussion that should happen, but I might 
have a patch down the road as a starter if all goes well at work.

Brian

Re: HTTP trailers?

[Brian J. France]

On Apr 22, 2010, at 8:40 PM, Mark Nottingham wrote:
 I couldn't find any obvious way to set HTTP trailers in Apache 2.x without 
 taking over all response processing (a la nph).
 
 Did I miss something?


I started hacking on this at work, but got the point where I can't insert a 
filter in the right spot.

The problem I ran into is that if the Content-Length header is set (which 
ap_content_length_filter_handle sets), then the chunking filter doesn't kick 
in.  Something about wanting to use sendfile so don't chunk.

I can't get a filter inserted after ap_content_length_filter_handle, but before 
ap_http_header_filter (which adds the chunking filter).

My plan was to insert a filter that removes the Content-Length header so the 
response will be chunked and add another filter after the chunking filter that 
would search for ASCII_ZERO ASCII_CRLF ASCII_CRLF and insert the trailer data 
before the last ASCII_CRLF.  This was just a proof of concept to see if it 
would work and output data in the right format.

If it all worked my plan was add a trailers hook and call the hook in 
chunk_filter.c in place of the /* trailers */ comment.

Brian

After typing this up and working through it again, I think I can add a filter 
AP_FTYPE_TRANSCODE+1 for the after chunking filter and AP_FTYPE_PROTOCOL+1 and 
get them to be in the right order.  Will test that out now.

Re: AW: ACL changes in mod_dav

[Brian J. France]

On Feb 23, 2010, at 11:02 PM, markus.l...@dlr.de markus.l...@dlr.de wrote:
 I think this is a little misunderstanding. Yes I mean the WebDAV ACL spec 
 features, but I we don't have implemented this into mod_dav. We implemented 
 it into our own module (Catacomb) and therefore we need to extent mod_dav to 
 handle the ACP.


What kind of extensions did you add to mod_dav?  These are all in TRUNK right 
now, but no in 2.2 branch:

DAV Option Patch to allow adding thing to the DAV or Allow headers:

  http://www.mail-archive.com/dev@httpd.apache.org/msg45113.html


DAV Resource Type Patch to allow additional resource type responses:

  http://www.mail-archive.com/dev@httpd.apache.org/msg45253.html


DAV Provider Patch add hooks for dav modules to get access to the filename:

  http://www.mail-archive.com/dev@httpd.apache.org/msg45288.html


Was this the kind of things you need to add to mod_dav to get your module 
working to do acl support?

I am sure you also had to add hooks or something to do the acl stuff in 
mod_dav, but that was the last part of mod_dav_acl[1] patch that I am splitting 
up once I get the last supporting patch in (still needs a little more work):

DAV ETag Patch:
  http://www.mail-archive.com/dev@httpd.apache.org/msg47239.html

Brian

[1] http://sourceforge.net/projects/moddavacl/





 
 - Markus
 
 
 
 -Ursprüngliche Nachricht-
 Von: Greg Stein [mailto:gst...@gmail.com]
 Gesendet: Sonntag, 21. Februar 2010 21:05
 An: dev@httpd.apache.org
 Betreff: Re: ACL changes in mod_dav
 
 This is pretty cool. I'm assuming you're referring to the WebDAV ACL
 spec features?
 
 Every time that I started to look into the issue, I ran into one basic
 issue: how to notify the multiple processes that the ACLs around a
 particular namespace have changed. How did you handle that?
 
 Cheers,
 -g
 
 On Sat, Feb 20, 2010 at 23:23,  markus.l...@dlr.de wrote:
 Hi,
 
 I have added ACL features to the mod_dav module. Could you tell me the
 correct way to get this changes reviewed and into to official
 mod_dav-source?
 
 Thanks,
 Markus
 

DAV ETag Patch

[Brian J. France]

Hello,

 It has been a while since my last patch, but this is the last supporting patch 
to mod_dav and httpd that is required before adding acl hooks into mod_dav for 
mod_dav_acl[1]

http://www.brianfrance.com/software/apache/dav/dav-etag.diff

This patch adds a new directive for mod_dav: DAVETagResponse

It cleans up some code in mod_dav and also removes the one second last-modified 
check in the httpd etag function.  I talked with Bill Rowe and Paul Querna 
about this at the goole hack day last month and they didn't see a problem with 
it, but I wanted to make sure I mentioned it here.

Brian

[1] http://sourceforge.net/projects/moddavacl/

Re: ACL changes in mod_dav

[Brian J. France]

On Feb 20, 2010, at 11:23 PM, markus.l...@dlr.de markus.l...@dlr.de wrote:
 I have added ACL features to the mod_dav module. Could you tell me the
 correct way to get this changes reviewed and into to official
 mod_dav-source?


Did you use any of the public available mod_dav_acl[1] code?  

I have been splitting up the mod_dav_acl patch into little patches and 
submitting them.  I sent the last supporting patch today and the last thing 
that is missing from trunk is the acl hooks structure from the patch and of 
course the mod_dav_acl module it self.  All this code is in trunk and could be 
back ported down the road to 2.2 as it has all been done in a binary compatible 
way.

Brian

[1] http://sourceforge.net/projects/moddavacl/

Re: reopening of logs without restarting

[Brian J. France]

On Jan 22, 2010, at 1:43 PM, Mikhail T. wrote:
 Hello!
 
 Some of our web-servers take a while to restart (because some custom
 modules need to login to database backends, etc.) This makes it
 undesirable for us to use the SIGUSR1 (for graceful restart) and we
 currently log to stdin of an easier to restart command-line utility.
 
 How hard would it be to implement a separate signal-handler, which
 would -- upon receiving, say, SIGUSR2 -- reopen the log-files without
 performing a full restart of each worker? That would provide for a
 possibility to log straight into a file and rotate that once in a while
 without a full restart of the httpds.
 
 Currently there are ap_run_open_logs and worker_open_logs. Can my
 hypothetical signal-handler simply go through the list of opened
 descriptors, set them to new values and close the old?
 
 Or is this a hairy task, that some have tried, but nobody succeeded
 in implementing?


Funny you should bring this up, at work we have been look at out patches to apr 
and httpd that we can give back to the up stream public source and one of those 
patches is to apr to allow hupless log rotation in httpd 2.x.

The patch adds APR_FOPEN_ROTATING to the flags when opening a file (which we 
patch httpd to add it), then open will save extra data after opening (flags, 
perm, apr_finfo_t, last check and timeout).  If APR_FOPEN_ROTATING is set then 
before every write call it will check to see if it needs to re-stat the file to 
see if anything has change (device/inode) and if it has it will re-open the 
file automatically.

This allows us to move log files to a new name, wait X seconds (our default 
timeout is 60, so we wait 90 to be safe) and then we can do what ever we want 
to the log file because all children will have either re-opened it or will 
re-opened it before the next write call.

Is that what you are looking for?

I was going to bring up the patch during the Monday/Tuesday hack days along 
with a few others.

Brian

Re: reopening of logs without restarting

[Brian J. France]

On Jan 22, 2010, at 3:15 PM, Mikhail Teterin wrote:

 22.01.2010 14:50, Brian J. France ???(??):
 I was going to bring up the patch during the Monday/Tuesday hack days along 
 with a few others.
   
 Can you send me the patch directly /today/? Thanks!


Here is the patch:

http://www.brianfrance.com/software/apache/apr/rotating.diff

A few things to note:

- We #ifdef YAHOO the changes so you have to define it or remove those lines
- APR_FOPEN_ROTATING == 0x08000, we had to keep binary compatibility in our 
code, so we had to re-adjust APR_FOPEN_SPARSE
- I never got around to write get/set functions for timeout, which defaults to 
60
- we have a patch for apr_time_now() to use a faster gettimeofday call that 
doesn't do a system call every time (cpu ticks)

Re: httpd meetup/hackathon, January?

[Brian J. France]

On Dec 8, 2009, at 7:49 PM, Paul Querna wrote:
 On Mon, Dec 7, 2009 at 11:50 AM, Paul Querna p...@querna.org wrote:
 Hi Everyone,
 
 Over on TraffiicServer, there is rough talk of doing some kind of
 meetup/hackathon the week of January 27th 2010, in Silicon Valley.
 
 Alternative is the week of January 13th -- would this earlier date
 change the ability of anyone to attend?
 
 I think we will try to firm up the date by like Thursday so people can
 figure out travel


I had already booked my trip out to CA before the TS guys pinged me about this 
and it is for the week of the 11th (13th being that Thursday), so I count me in 
for that week.

Brian

Re: svn commit: r823703 - in /httpd/httpd/trunk: CHANGES modules/dav/fs/repos.c modules/dav/main/mod_dav.h

[Brian J. France]

Ya, I have been trying to get somebody to commit this fixup patch:

http://www.brianfrance.com/software/apache/dav/fixup.diff

It fixes comments of the new function pointers (so they match others),  
moves the function to static to remove compile time warnings and fixes  
the order in the structure definition.


Can anybody with commit access please review and commit?

Brian


On Oct 27, 2009, at 11:58 AM, Guenter Knauf wrote:


Hi,
minf...@apache.org schrieb:

Author: minfrin
Date: Fri Oct  9 21:41:31 2009
New Revision: 823703
URL: http://svn.apache.org/viewvc?rev=823703view=rev
Log:
mod_dav: Provide a mechanism to obtain the request_rec and pathname
from the dav_resource.
Submitted by: Jari Urpalainen jari.urpalainen nokia.com,
 Brian France brian brianfrance.com
Modified:
   httpd/httpd/trunk/CHANGES
   httpd/httpd/trunk/modules/dav/fs/repos.c
   httpd/httpd/trunk/modules/dav/main/mod_dav.h
Modified: httpd/httpd/trunk/modules/dav/fs/repos.c
URL: 
http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/dav/fs/repos.c?rev=823703r1=823702r2=823703view=diff
= 
= 
= 
= 
= 
= 
= 
= 
= 
=

--- httpd/httpd/trunk/modules/dav/fs/repos.c (original)
+++ httpd/httpd/trunk/modules/dav/fs/repos.c Fri Oct  9 21:41:31 2009
@@ -46,6 +46,7 @@
apr_pool_t *pool;/* memory storage pool associated with  
request */

const char *pathname;   /* full pathname to resource */
apr_finfo_t finfo;   /* filesystem info */
+request_rec *r;
};
 /* private context for doing a filesystem walk */
@@ -210,6 +211,11 @@
**
** PRIVATE REPOSITORY FUNCTIONS
*/
+request_rec *dav_fs_get_request_rec(const dav_resource *resource)
+{
+return resource-info-r;
+}
+
apr_pool_t *dav_fs_pool(const dav_resource *resource)
{
return resource-info-pool;
@@ -648,6 +654,7 @@
/* Create private resource context descriptor */
ctx = apr_pcalloc(r-pool, sizeof(*ctx));
ctx-finfo = r-finfo;
+ctx-r = r;
 /* ### this should go away */
ctx-pool = r-pool;
@@ -1816,6 +1823,9 @@
dav_fs_remove_resource,
dav_fs_walk,
dav_fs_getetag,
+dav_fs_get_request_rec,
+dav_fs_pathname,
+NULL
};
 static dav_prop_insert dav_fs_insert_prop(const dav_resource  
*resource,

here seems to be a problem with the order:
Compiling repos.c
### mwccnlm Compiler:
#File: repos.c
# 
#1827:  dav_fs_pathname,
#   Error:  ^
#   illegal implicit conversion from 'char * (const struct  
dav_resource *)' to

#   'struct request_rec * (*)(const struct dav_resource *)'

Errors caused tool to abort.

From what we have in mod_dav.h
   /* Get the entity tag for a resource */
   const char * (*getetag)(const dav_resource *resource);

   /*
   ** If a provider needs a context to associate with this hooks  
structure,
   ** then this field may be used. In most cases, it will just be  
NULL.

   */
   void *ctx;

   /* return request record */
   request_rec * (*get_request_rec)(const dav_resource *resource);

   /* return path */
   const char * (*get_pathname)(const dav_resource *resource);


it seems to me that it should be:
Index: repos.c
===
--- repos.c (revision 830029)
+++ repos.c (working copy)
@@ -1823,9 +1823,9 @@
dav_fs_remove_resource,
dav_fs_walk,
dav_fs_getetag,
+NULL,
dav_fs_get_request_rec,
-dav_fs_pathname,
-NULL
+dav_fs_pathname
};


Gün.

Re: svn commit: r823703 - in /httpd/httpd/trunk: CHANGES modules/dav/fs/repos.c modules/dav/main/mod_dav.h

[Brian J. France]

On Oct 16, 2009, at 8:01 AM, Graham Leggett wrote:

Brian J. France wrote:

mod_dav_acl would use the filename to validate the acls.  Like I  
said, I
don't know if get_pathname is needed or we should just use r- 
filename

and make sure a mod_dav_fs_db module updated it.


As Joe points out, an ACL could refer to something that wasn't a file,
such as a subversion repository, or something similar.

It would be better if ACLs could be applied to any URI, not just URIs
that map to files.

mod_dav_acl mapping to files only seriously limits the usefulness of  
the

module.



My goal is to create a mod_dav_acl that requires acl providers to do  
the real work.  Like a mod_dav_acl_fs to do file based acls so it  
would need a filename, mod_dav_acl_db could do db acls based on uri,  
mod_dav_acl_svn could do uri or svn fs based acls.


I think there is one more patch that is not truly acl related that is  
required for all of this (etags stuff) before the acl patch.


Anybody want to get together to talk about design for mod_dav_acl at  
ApacheCon?


Brian

Re: svn commit: r823703 - in /httpd/httpd/trunk: CHANGES modules/dav/fs/repos.c modules/dav/main/mod_dav.h

[Brian J. France]

On Oct 10, 2009, at 4:04 AM, Ruediger Pluem wrote:

On 10/09/2009 11:41 PM, minf...@apache.org wrote:

Author: minfrin
Date: Fri Oct  9 21:41:31 2009
New Revision: 823703

URL: http://svn.apache.org/viewvc?rev=823703view=rev
Log:
mod_dav: Provide a mechanism to obtain the request_rec and pathname
from the dav_resource.
Submitted by: Jari Urpalainen jari.urpalainen nokia.com,
 Brian France brian brianfrance.com

Modified:
   httpd/httpd/trunk/CHANGES
   httpd/httpd/trunk/modules/dav/fs/repos.c
   httpd/httpd/trunk/modules/dav/main/mod_dav.h




Modified: httpd/httpd/trunk/modules/dav/fs/repos.c
URL: 
http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/dav/fs/repos.c?rev=823703r1=823702r2=823703view=diff
=
=
=
=
=
=
=
=
=
=
--- httpd/httpd/trunk/modules/dav/fs/repos.c (original)
+++ httpd/httpd/trunk/modules/dav/fs/repos.c Fri Oct  9 21:41:31 2009



@@ -1816,6 +1823,9 @@
dav_fs_remove_resource,
dav_fs_walk,
dav_fs_getetag,
+dav_fs_get_request_rec,
+dav_fs_pathname,
+NULL


This creates the following warning:

repos.c:1827: warning: initialization from incompatible pointer type

I assume the order of the arguments is wrong and needs to be

NULL,
dav_fs_get_request_rec,
dav_fs_pathname

instead. See below in the snipped from mod_dav.h:



Modified: httpd/httpd/trunk/modules/dav/main/mod_dav.h
URL: 
http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/dav/main/mod_dav.h?rev=823703r1=823702r2=823703view=diff
=
=
=
=
=
=
=
=
=
=
--- httpd/httpd/trunk/modules/dav/main/mod_dav.h (original)
+++ httpd/httpd/trunk/modules/dav/main/mod_dav.h Fri Oct  9  
21:41:31 2009

@@ -1940,6 +1940,12 @@
** then this field may be used. In most cases, it will just be  
NULL.

*/
void *ctx;
+
+/* return request record */
+request_rec * (*get_request_rec)(const dav_resource *resource);
+
+/* return path */
+const char * (*get_pathname)(const dav_resource *resource);
};



Below is a patch to fix the bad struct declaration, sorry about that.   
I missed it on the binary compatibility conversion.


Brian

Index: modules/dav/fs/repos.c
===
--- modules/dav/fs/repos.c  (revision 824480)
+++ modules/dav/fs/repos.c  (working copy)
@@ -1823,9 +1823,9 @@
 dav_fs_remove_resource,
 dav_fs_walk,
 dav_fs_getetag,
+NULL,
 dav_fs_get_request_rec,
-dav_fs_pathname,
-NULL
+dav_fs_pathname
 };

 static dav_prop_insert dav_fs_insert_prop(const dav_resource  
*resource,

Re: svn commit: r823703 - in /httpd/httpd/trunk: CHANGES modules/dav/fs/repos.c modules/dav/main/mod_dav.h

[Brian J. France]

On Oct 12, 2009, at 3:58 AM, Joe Orton wrote:


On Sat, Oct 10, 2009 at 10:04:57AM +0200, Ruediger Pluem wrote:

On 10/09/2009 11:41 PM, minf...@apache.org wrote:

=
=
=
=
=
=
=
=
=
=

--- httpd/httpd/trunk/modules/dav/fs/repos.c (original)
+++ httpd/httpd/trunk/modules/dav/fs/repos.c Fri Oct  9 21:41:31  
2009



@@ -1816,6 +1823,9 @@
dav_fs_remove_resource,
dav_fs_walk,
dav_fs_getetag,
+dav_fs_get_request_rec,
+dav_fs_pathname,
+NULL


This creates the following warning:

repos.c:1827: warning: initialization from incompatible pointer type


Plus the dav_fs_get_request_rec prototype needs to be moved up.

/local/asf/httpd-trunk/modules/dav/fs/repos.c:214: warning: no  
previous

prototype for ‘dav_fs_get_request_rec’



I don't know why that warning is happening. dav_fs_get_request_rec is  
defined on line 214 and used in the struct on line 1827.


Brian

Re: svn commit: r823703 - in /httpd/httpd/trunk: CHANGES modules/dav/fs/repos.c modules/dav/main/mod_dav.h

[Brian J. France]

On Oct 12, 2009, at 3:57 AM, Joe Orton wrote:


On Fri, Oct 09, 2009 at 09:41:32PM -, Graham Leggett wrote:

--- httpd/httpd/trunk/modules/dav/main/mod_dav.h (original)
+++ httpd/httpd/trunk/modules/dav/main/mod_dav.h Fri Oct  9  
21:41:31 2009

@@ -1940,6 +1940,12 @@
** then this field may be used. In most cases, it will just be  
NULL.

*/
void *ctx;
+
+/* return request record */
+request_rec * (*get_request_rec)(const dav_resource *resource);
+
+/* return path */
+const char * (*get_pathname)(const dav_resource *resource);
};



What is a pathname in this context? A URI path?  A filesystem path?
If the latter, what is get_pathname supposed to do for a non-fs-backed
repository provider?



That I don't know, it could use bogus paths.  I haven't gone down the  
path of creating a new mod_dav_fs module, so I don't know exactly how  
it would work.




(Also - compare and contrast how all the rest of the comments in this
structure are descriptive phrases and start with capital letters and
everything)


Patch below fixes up the comments for the new functions.

Index: modules/dav/main/mod_dav.h
===
--- modules/dav/main/mod_dav.h  (revision 824480)
+++ modules/dav/main/mod_dav.h  (working copy)
@@ -1941,10 +1941,10 @@
 */
 void *ctx;

-/* return request record */
+/* Get the request rec for a resource */
 request_rec * (*get_request_rec)(const dav_resource *resource);

-/* return path */
+/* Get the pathname for a resource */
 const char * (*get_pathname)(const dav_resource *resource);
 };

Re: svn commit: r823703 - in /httpd/httpd/trunk: CHANGES modules/dav/fs/repos.c modules/dav/main/mod_dav.h

[Brian J. France]

On Oct 12, 2009, at 4:38 PM, Joe Orton wrote:


On Mon, Oct 12, 2009 at 04:23:59PM -0400, Brian J. France wrote:

On Oct 12, 2009, at 3:57 AM, Joe Orton wrote:

On Fri, Oct 09, 2009 at 09:41:32PM -, Graham Leggett wrote:

--- httpd/httpd/trunk/modules/dav/main/mod_dav.h (original)
+++ httpd/httpd/trunk/modules/dav/main/mod_dav.h Fri Oct  9  
21:41:31

+
+/* return request record */
+request_rec * (*get_request_rec)(const dav_resource  
*resource);

+
+/* return path */
+const char * (*get_pathname)(const dav_resource *resource);
};



What is a pathname in this context? A URI path?  A filesystem  
path?
If the latter, what is get_pathname supposed to do for a non-fs- 
backed

repository provider?



That I don't know, it could use bogus paths.  I haven't gone down the
path of creating a new mod_dav_fs module, so I don't know exactly  
how it

would work.


Well, there needs to be some API contract specified so that repos
backends can implement it.

So: why does the resource abstraction need to be extended to return  
the

filesystem path?  What will mod_dav use it for?



mod_dav_acl would use the filename to validate the acls.  Like I said,  
I don't know if get_pathname is needed or we should just use r- 
filename and make sure a mod_dav_fs_db module updated it.


Brian

Re: svn commit: r823703 - in /httpd/httpd/trunk: CHANGES modules/dav/fs/repos.c modules/dav/main/mod_dav.h

[Brian J. France]

On Oct 12, 2009, at 4:39 PM, Joe Orton wrote:


On Mon, Oct 12, 2009 at 04:17:00PM -0400, Brian J. France wrote:

On Oct 12, 2009, at 3:58 AM, Joe Orton wrote:

On Sat, Oct 10, 2009 at 10:04:57AM +0200, Ruediger Pluem wrote:

This creates the following warning:

repos.c:1827: warning: initialization from incompatible pointer  
type


Plus the dav_fs_get_request_rec prototype needs to be moved up.

/local/asf/httpd-trunk/modules/dav/fs/repos.c:214: warning: no
previous
prototype for ‘dav_fs_get_request_rec’


I don't know why that warning is happening. dav_fs_get_request_rec is
defined on line 214 and used in the struct on line 1827.


Sorry, my mistake, it's not a positioning problem - the function needs
to be made static.


Updated patch:

Index: modules/dav/fs/repos.c
===
--- modules/dav/fs/repos.c  (revision 824480)
+++ modules/dav/fs/repos.c  (working copy)
@@ -211,7 +211,7 @@
 **
 ** PRIVATE REPOSITORY FUNCTIONS
 */
-request_rec *dav_fs_get_request_rec(const dav_resource *resource)
+status request_rec *dav_fs_get_request_rec(const dav_resource  
*resource)

 {
 return resource-info-r;
 }
@@ -1823,9 +1823,9 @@
 dav_fs_remove_resource,
 dav_fs_walk,
 dav_fs_getetag,
+NULL,
 dav_fs_get_request_rec,
-dav_fs_pathname,
-NULL
+dav_fs_pathname
 };

 static dav_prop_insert dav_fs_insert_prop(const dav_resource  
*resource,

Re: svn commit: r823703 - in /httpd/httpd/trunk: CHANGES modules/dav/fs/repos.c modules/dav/main/mod_dav.h

[Brian J. France]

On Oct 12, 2009, at 5:15 PM, Brian J. France wrote:



On Oct 12, 2009, at 4:39 PM, Joe Orton wrote:


On Mon, Oct 12, 2009 at 04:17:00PM -0400, Brian J. France wrote:

On Oct 12, 2009, at 3:58 AM, Joe Orton wrote:

On Sat, Oct 10, 2009 at 10:04:57AM +0200, Ruediger Pluem wrote:

This creates the following warning:

repos.c:1827: warning: initialization from incompatible pointer  
type


Plus the dav_fs_get_request_rec prototype needs to be moved up.

/local/asf/httpd-trunk/modules/dav/fs/repos.c:214: warning: no
previous
prototype for ‘dav_fs_get_request_rec’


I don't know why that warning is happening. dav_fs_get_request_rec  
is

defined on line 214 and used in the struct on line 1827.


Sorry, my mistake, it's not a positioning problem - the function  
needs

to be made static.


Updated patch:



Scratch that, here is a patch:

Index: modules/dav/fs/repos.c
===
--- modules/dav/fs/repos.c  (revision 824480)
+++ modules/dav/fs/repos.c  (working copy)
@@ -211,7 +211,7 @@
 **
 ** PRIVATE REPOSITORY FUNCTIONS
 */
-request_rec *dav_fs_get_request_rec(const dav_resource *resource)
+static request_rec *dav_fs_get_request_rec(const dav_resource  
*resource)

 {
 return resource-info-r;
 }
@@ -1823,9 +1823,9 @@
 dav_fs_remove_resource,
 dav_fs_walk,
 dav_fs_getetag,
+NULL,
 dav_fs_get_request_rec,
-dav_fs_pathname,
-NULL
+dav_fs_pathname
 };

 static dav_prop_insert dav_fs_insert_prop(const dav_resource  
*resource,

Re: DAV Provider Patch

[Brian J. France]

On Oct 7, 2009, at 5:37 PM, Graham Leggett wrote:


Brian J. France wrote:


Sorry for the delay in response to this, life got in the way.

I have updated the patch here:

http://www.brianfrance.com/software/apache/dav/dav-provider-3.diff

This patch doesn't break binary compatibility (adds the functions  
to the

end of the struct) and adds both get_request_rec and get_pathname.
While in most cases you can pull pathname from the request_rec, how
would you get the pathname from a mod_dav_fs_db type module?  Should
mod_dav_fs_db update r-filename or should we keep the get_pathname
function in the provider struct?

Either way works for me, just happen to have a discussion at work  
about

writing a custom mod_dav_fs module and thought of this patch case.


Just a quick check - am I right in understanding that the get_pathname
function below is an oversight?

Index: modules/dav/main/mod_dav.h
===
--- modules/dav/main/mod_dav.h  (revision 822497)
+++ modules/dav/main/mod_dav.h  (working copy)
@@ -1940,6 +1940,12 @@
** then this field may be used. In most cases, it will just be  
NULL.

*/
void *ctx;
+
+/* return request record */
+request_rec * (*get_request_rec)(const dav_resource *resource);
+
+/* return path */
+const char * (*get_pathname)(const dav_resource *resource);
};


Depends.

Should a mod_dav_fs type module (like mod_dav_fs_database) update r- 
filename so other modules like mod_dav_acl could use the filename  
from the request_rec.
Or should mod_dav_acl use a hook function to get the pathname because  
r-filename would not be set correctly since that is a path on disk in  
the case of mod_dav_fs_database?


My patch (version 3) left the get_pathname hook with the assumption  
that r-filename should not be used and instead a hook should be used.


Brian

Re: DAV Provider Patch

[Brian J. France]

Sorry for the delay in response to this, life got in the way.

I have updated the patch here:

http://www.brianfrance.com/software/apache/dav/dav-provider-3.diff

This patch doesn't break binary compatibility (adds the functions to  
the end of the struct) and adds both get_request_rec and  
get_pathname.  While in most cases you can pull pathname from the  
request_rec, how would you get the pathname from a mod_dav_fs_db type  
module?  Should mod_dav_fs_db update r-filename or should we keep the  
get_pathname function in the provider struct?


Either way works for me, just happen to have a discussion at work  
about writing a custom mod_dav_fs module and thought of this patch case.


Thoughts?

Brian

On Sep 21, 2009, at 11:34 AM, Plüm, Rüdiger, VF-Group wrote:

-Original Message-
From: Dan Poirier
Sent: Montag, 21. September 2009 17:28
To: dev@httpd.apache.org
Subject: Re: DAV Provider Patch

Brian J. France br...@brianfrance.com writes:


On Sep 21, 2009, at 10:15 AM, Graham Leggett wrote:


Brian J. France wrote:


I believe this is the first patch that will break binary
compatibility
because it adds a function pointer to the middle of the struct.  I
believe binary compatibility could be retained if we add

the function

pointers to the end of the struct instead of in the

middle.  Moving

the
function could be part of the back porting patch to 2.2, but leave
it as
is in 2.3+.


Would it be possible to move the function pointers to the

end of the

struct for httpd-trunk as well? Breaking binary

compatibility is to be

avoided if it can be, even on trunk.



Sure.  Which method would be preferred?  Having two hooks

or just one

and use the request_rec to get the filename?


As long as the filename can be gotten trivially from the request rec,
I'd say keep things simple and just add the one hook.


+1.

Regards

Rüdiger

Re: DAV Provider Patch

[Brian J. France]

On Oct 6, 2009, at 7:40 PM, Nick Kew wrote:



On 6 Oct 2009, at 22:15, Brian J. France wrote:


Sorry for the delay in response to this, life got in the way.

I have updated the patch here:

http://www.brianfrance.com/software/apache/dav/dav-provider-3.diff

This patch doesn't break binary compatibility (adds the functions  
to the end of the struct) and adds both get_request_rec and  
get_pathname.  While in most cases you can pull pathname from the  
request_rec, how would you get the pathname from a mod_dav_fs_db  
type module?  Should mod_dav_fs_db update r-filename or should we  
keep the get_pathname function in the provider struct?


Either way works for me, just happen to have a discussion at work  
about writing a custom mod_dav_fs module and thought of this patch  
case.


My recollection of hacking at mod_dav is that I wanted to make some  
similar
changes, but I was in two minds whether patching the API like this  
was the
right solution (certainly getting the request_rec), or whether it  
wanted a

deeper-level redesign.

Is this API change sufficient for your app?  And if not, how much
more is there to come?



There are two more patches left.  One is for handling ETags and will  
patch both dav and http_etag.c to handle usec timestamps in the  
ETags.  The second one is the main ACL support patch to the dav stuff.


After that, all the rest of mod_dav_acl, mod_caldav and mod_carddav  
will work with out any patches to httpd/dav.


Brian

DAV Provider Patch

[Brian J. France]

Next up is the dav providers patch.  Currently there is no way for dav  
modules to get access to the filename or the the request_rec.  A dav  
module would need to check the filename to see if needs to enable acls  
or other options.  A dav module would also need the request_rec for  
checking options via the r-per_dir_config (like mod_dav_acl needs to  
know what the principle path and location are set to).  This can be  
done one of two ways.


This patch adds two hooks functions to the dav_hooks_repository  
struct, get_request_rec and get_pathname:


  http://www.brianfrance.com/software/apache/dav/dav-provider-1.diff

it also patches mod_dav_fs to support these two functions.  The other  
option would be to only add the get_request_rec to the struct like this:


  http://www.brianfrance.com/software/apache/dav/dav-provider-2.diff

and have a module just use r-filename to get the filename instead of  
adding another function.


I believe this is the first patch that will break binary compatibility  
because it adds a function pointer to the middle of the struct.  I  
believe binary compatibility could be retained if we add the function  
pointers to the end of the struct instead of in the middle.  Moving  
the function could be part of the back porting patch to 2.2, but leave  
it as is in 2.3+.


Both of these patches also fix an issue where the definition of  
dav_hooks_repository_fs in modules/dav/fs/repos.c doesn't have a NULL  
value for the ctx (broken in HEAD and 2.2.x) at the end of the struct.


The patches are based on Jari's httpd-2.2.8-ju.patch patch.

Comments, questions, problems?

Brian

Re: DAV Provider Patch

[Brian J. France]

On Sep 21, 2009, at 10:15 AM, Graham Leggett wrote:


Brian J. France wrote:

I believe this is the first patch that will break binary  
compatibility

because it adds a function pointer to the middle of the struct.  I
believe binary compatibility could be retained if we add the function
pointers to the end of the struct instead of in the middle.  Moving  
the
function could be part of the back porting patch to 2.2, but leave  
it as

is in 2.3+.


Would it be possible to move the function pointers to the end of the
struct for httpd-trunk as well? Breaking binary compatibility is to be
avoided if it can be, even on trunk.



Sure.  Which method would be preferred?  Having two hooks or just one  
and use the request_rec to get the filename?


Brian

DAV Resource Type Patch

[Brian J. France]

Next up is the dav resource provider patch.

Currently there is no way to add additional resource type responses to  
mod_dav.  We are stuck with hard coded list from  
DAV_PROPID_resourcetype:


  
http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/dav/main/std_liveprop.c?annotate=420983#l58

but with this patch:

  http://www.brianfrance.com/software/apache/dav/dav-resource-type-provider.diff

other modules can become providers and add resource types to the  
response.  This is needed because mod_dav_acl needs to add principal,  
mod_caldav needs to add calendar and mod_carddav needs to add  
addressbook resource types.


This patch (which should *not* be committed) extends the previous  
example mod_dav_acl to add the principal resource type:


  
http://www.brianfrance.com/software/apache/dav/dav-resource-type-provider-acl.diff

Once patched and installed you can test it by adding the following to  
extra/httpd-dav.conf in the uploads Directory block:


AuthType Digest
AuthName DAV-upload

AuthUserFile path to/user.passwd
AuthDigestProvider file

creating a user.passwd file like this:

  % htdigest -c path to/user.passwd DAV-upload admin

starting apache and then doing a request like this:

% curl -v --digest --data-binary @resourcetype.txt --user admin \
-H Content-type: text/xml; charset=\utf-8\ \
-H Depth: 0 \
-X 'PROPFIND' \
http://localhost/uploads/index.html

where a file called resourcetype.txt is in the same directory with the  
following xml in it:


?xml version=1.0 encoding=utf-8 ?
D:propfind xmlns:D=DAV:
   D:prop
  D:resourcetype/
   /D:prop
 /D:propfind

and you should see a response that looks like this:

?xml version=1.0 encoding=utf-8?
D:multistatus xmlns:D=DAV: xmlns:ns0=DAV:
D:response xmlns:lp1=DAV:
D:href/uploads/index.html/D:href
D:propstat
D:prop
lp1:resourcetypeD:principal//lp1:resourcetype
/D:prop
D:statusHTTP/1.1 200 OK/D:status
/D:propstat
/D:response
/D:multistatus

The patches are based on Jari's httpd-2.2.8-ju.patch patch.  I have  
changed dav_hooks_resource to dav_resource_type_provider, based on  
previous comments on the options patch and also remove the ctx as it  
was not being used.


Comments, questions, problems?

Brian

DAV Option Patch

[Brian J. France]

I would like to get some form of mod_dav_acl[1] added to httpd.  My  
end goal with all of this is to get a mod_caldav and mod_cardav  
accepted down the line or at least be able to build the module with  
out hacking the core httpd source.


I am going to start by splitting up the Jari's mod_dav_acl patches to  
httpd into small patches with detailed explanation on why they are  
needed and first up is the DAV options patch.


Currently there is no way for other dav modules to add things do the  
DAV or Allow headers of a OPTIONS request, only the mod_dav module can  
output those items.  This patch:


  http://www.brianfrance.com/software/apache/dav/dav-options.diff

allows other mod_dav modules to add options to those headers.  The  
next patch is only an example usage of the above patch and should not  
be added to the svn tree.  This patch will add a mod_dav_acl module to  
the build:


  http://www.brianfrance.com/software/apache/dav/dav-options-acl.diff

The patch will require you to run ./buildconf to regenerate configure  
and:


--enable-dav --enable-dav-acl

to enable it.  This example skeleton acl module using the options  
patch will add access-control to the DAV header and REPORT and ACL to  
the Allow header:


  DAV: 1,2,access-control
  Allow:  
OPTIONS 
,GET 
,HEAD 
,POST,DELETE,TRACE,PROPFIND,PROPPATCH,COPY,MOVE,LOCK,UNLOCK,REPORT,ACL


This is the first step in getting a working mod_dav_acl module and  
with that allow a mod_caldav module to do something like this:


  DAV: 1,2,access-control,calendar-access
  Allow:  
OPTIONS 
,GET 
,HEAD 
,POST 
,DELETE 
,TRACE,PROPFIND,PROPPATCH,COPY,MOVE,LOCK,UNLOCK,REPORT,ACL,MKCALENDAR


and a mod_carddav module to do something like:

  DAV: 1,2,access-control,addressbook,extended-mkcol
  Allow:  
OPTIONS 
,GET 
,HEAD 
,POST,DELETE,TRACE,PROPFIND,PROPPATCH,COPY,MOVE,LOCK,UNLOCK,REPORT,ACL


There is one draw back to this patch in that there could be duplicated  
values in the headers.  Both mod_dav_acl and mod_caldav want to add  
the REPORT in the Allow header, so it would show up twice in the  
list.  I am not sure if this is a major problem, but wanted to make a  
note of it.


Thoughts, comments, questions?

These patches are based off of 2.2.13, but if you would rather have  
patch against a branch or head, please let me know.


Brian

[1] http://sourceforge.net/projects/moddavacl/

Re: DAV Option Patch

[Brian J. France]

On Sep 14, 2009, at 10:23 AM, Graham Leggett wrote:

Brian J. France wrote:


These patches are based off of 2.2.13, but if you would rather have
patch against a branch or head, please let me know.


Creating patches off head is the place to start, and once the patches
have been committed to head, the next step is for people to vote on
backporting the patches to v2.2, which as you already have patches for
v2.2 should be straightfoward.



I have updated the dav-options patch against head:

  http://www.brianfrance.com/software/apache/dav/dav-options.diff

The dav-options-acl.diff patch will apply to head because it just  
creates new files, but remember it shouldn't be committed as it is  
just an example.



Breaking up the patches as you have mentioned above makes reviewing  
the
patches easier, and in turn that makes it easier to gets votes for  
backport.


If you break up the patches, I'll take a look and get them into trunk.



Great!

Thanks,

Brian

Re: DAV Option Patch

[Brian J. France]

On Sep 14, 2009, at 12:04 PM, Dan Poirier wrote:
One suggestion - using hook in the names could be confusing since  
this
isn't using the built-in hook mechanism.  Since it is using the  
provider

mechanism, maybe names like dav_options_provider would be clearer.




I have updated the patches:

  http://www.brianfrance.com/software/apache/dav/dav-option-provider.diff

and the example module (which shouldn't be commited):

  http://www.brianfrance.com/software/apache/dav/dav-option-provider-acl.diff

Let me know what you think.

Thanks!

Brian

Re: DAV Option Patch

[Brian J. France]

On Sep 14, 2009, at 04:32 PM, Graham Leggett wrote:


Brian J. France wrote:


I have updated the patches:

 http://www.brianfrance.com/software/apache/dav/dav-option-provider.diff


Committed in r814832, can you verify that I have attributed the patch
correctly in CHANGES?


Thanks!

Can you credit Jari Urpalainen (jari.urpalainen nokia.com) as well as  
these are all based on his patches for his mod_dav_acl.


More patches tomorrow.

Cheers,

Brian

Re: DAV Option Patch

[Brian J. France]

On Sep 14, 2009, at 05:23 PM, Graham Leggett wrote:


Brian J. France wrote:


Can you credit Jari Urpalainen (jari.urpalainen nokia.com) as well as
these are all based on his patches for his mod_dav_acl.


Done in 814860.


More patches tomorrow.


I see the example module that you asked not to commit is ASF licensed,
is this true



That example module is my testing module and is under the ASF licensed.



also of the patches you've have submitted so far?



While Jari's mod_dav_acl is licensed under LGPL, can the patches to  
httpd be licensed that way?


What would we need to do to get them added if Jari's patches (or even  
mod_dav_acl) would fall under LGPL?


I know he is would like to see them added to httpd.

Brian

Re: DAV Option Patch

[Brian J. France]

On Sep 14, 2009, at 05:02 PM, Joe Orton wrote:


On Mon, Sep 14, 2009 at 10:11:24AM -0400, Brian J. France wrote:
I would like to get some form of mod_dav_acl[1] added to httpd.  My  
end
goal with all of this is to get a mod_caldav and mod_cardav  
accepted down

the line or at least be able to build the module with out hacking the
core httpd source.

I am going to start by splitting up the Jari's mod_dav_acl patches to
httpd into small patches with detailed explanation on why they are
needed and first up is the DAV options patch.

Currently there is no way for other dav modules to add things do  
the DAV

or Allow headers of a OPTIONS request, only the mod_dav module can
output those items.  This patch:


It's not clear to me - is Jari the original author of this code, or
yourself?  mod_dav_acl seems to be licensed under the LGPL still.



Jari is the original author of mod_dav_acl, which requires patches to  
httpd to work.  I need the same functionality added to httpd to get a  
mod_dav_acl type module working, so I have split up his patch into  
smaller pieces.  Can a patch be under a different license than the  
original code?


My plan is to create a mod_dav_acl module that would require providers  
to plugin (mod_dav_acl_fs, mod_dav_acl_dbd, etc).


Brian

Re: incubator proposal for (what was once) Inktomi Traffic Server

[Brian J. France]

On Jun 16, 2009, at 10:02 AM, howard chen wrote:
On Mon, Jun 15, 2009 at 12:25 PM, Roy T. Fieldingfield...@gbiv.com  
wrote:

I think this is an interesting opportunity to compare
different implementations and share code where desirable.
I haven't seen anyone comment on the proposal yet.



Just out of curiosity, Flickr has been a die hard users of Squid.

Did your team ever convinced them to use YTS?



While I don't work on the YTS team, I am in the same group and deal a  
lot with them.


I believe Flickr partially uses YTS already for serving all movies,  
but I could be wrong.


The YTS team is working with Flickr on evaluating YTS as an  
alternative to squid, but I don't know how it is going.


Brian

Re: incubator proposal for (what was once) Inktomi Traffic Server

[Brian J. France]

On Jun 15, 2009, at 12:39 PM, Pranav Desai wrote:

On Sun, Jun 14, 2009 at 9:25 PM, Roy T. Fieldingfield...@gbiv.com  
wrote:

I think this is an interesting opportunity to compare
different implementations and share code where desirable.
I haven't seen anyone comment on the proposal yet.

Roy


Begin forwarded message:


From: Leif Hedstrom l...@yahoo-inc.com
Date: June 12, 2009 9:17:59 AM PDT
To: gene...@incubator.apache.org
Subject: [Proposal] Traffic Server


Good morning,

We would like to submit the Traffic Server proposal to the  
incubator. Our

draft is available at

  http://wiki.apache.org/incubator/TrafficServerProposal



This sounds awesome. Can it act as a forward proxy as well ? I guess
we will find out once the source is released.



Yes, YTS can be used as a forward proxy.



The 35000 rps is for reverse proxy or forward proxy ?



That is for reverse proxy mode and 35k is conservative.

Brian

  - Not on the YTS team, but in the same group

Re: hook before receiving HTTP request

[Brian J. France]

On Mar 16, 2009, at 9:13 AM, Andrej van der Zee wrote:

I am looking for a hook function that I can call to initialize some
structures before it accepts a connections. Similar like
ap_log_transaction() is called after the HTTP reply is sent to the
client, I need a hook that is called before a HTTP request is
received. Is it there?


http://httpd.apache.org/docs/2.2/developer/modules.html

ap_hook_pre_connection
do any setup required just before processing, but after accepting

ap_hook_create_request
??

(Assuming) ap_hook_create_request will be called when creating a  
request_rec * for each request on a connection (multiple request on a  
connection with KeepAlive enabled).


Is that what you are looking for?

Brian

Re: Static compilation of a module

[Brian J. France]

On Jul 31, 2008, at 11:16 AM, lusob wrote:

Ok, I'm going to try it, thanks!!


At work we have moved away from using static libraries to shared  
libraries because of duplicate symbols issues.


If your client loads your module as well as some module that uses  
apreq (say mod_perl or something else) there will be duplicate symbols  
loaded.  Say a bug is found in libapreq and they upgrade their version  
of the shared library, they may or may not pick up the fix depending  
on how modules are loaded.  If you module is loaded first, it will use  
your apreq symbols instead of a shared version loaded later.  If your  
module is loaded second, they you will be using the shared library  
version and not yours (and your module will use the shared version as  
well).


Granted most of time we were running into duplicate symbols with  
binary incompatibles and I would hope apreq wouldn't have this  
problem, but it still something to keep in mind.


Brian







Sorin Manolache wrote:


On Thu, Jul 31, 2008 at 16:26, lusob [EMAIL PROTECTED]  
wrote:


I'm developing a apache module. This module use the libapreq  
library, for

a
most simple instalation I would rather install only my module to the
customer, I would prefer that the customer doesn't have to install  
the

libapreq.
Are there any way to do it? If I perform a static compilation del  
module

(mymod.a) how can I load this static module in apache?
Thanks in advance


I have done a similar thing. Do not compile your module statically,
i.e. do not create mymod.a.

Create a dynamic shared object module, but link libapreq.a  
_statically_ to

it.

g++ -shared -nostdlib /usr/lib/crti.o
/usr/lib/gcc/i486-linux-gnu/4.2.4/crtbeginS.o mymod.o
-Wl,--whole-archive libapreq.a -L/usr/lib/gcc/i486-linux-gnu/4.2.4
-L/usr/lib -lstdc++ -lm -lc -lgcc_s
/usr/lib/gcc/i486-linux-gnu/4.2.4/crtendS.o /usr/lib/crtn.o
-Wl,--as-needed -Wl,-soname -Wl,mod_my.so.1 -o mod_my.so.1.0.0

Thus, all the code of libapreq that is needed by mod_my.so will be
present in the mod_my.so binary. So the client does not have to have
libapreq.a on his/her machine. However, your mod_my.so will still  
be a

dynamic shared object that will be loaded by apache upon startup.

S




--
View this message in context: 
http://www.nabble.com/Static-compilation-of-a-module-tp18755821p18756871.html
Sent from the Apache HTTP Server - Module Writers mailing list  
archive at Nabble.com.

Re: ap_custom_response content type 'text/xml'

[Brian J. France]

On Jun 27, 2008, at 4:00 AM, Kiffin Gish wrote:


Brian,

This is great, thanks alot!

I've 'almost' got it working correctly, the only problem being that  
the

Content-Type is not getting truncated correctly, e.g. instead of:

Content-Type: text/xml\n
\n
?xml version='1.0' encoding=UTF-8 standalone=no ?\n


I'm getting this:

Content-Type: text/xml?xml version='1.0' encoding=UTF-8
standalone=no ?\n

Any tips would be greatly appreciated.



In my example it looks for \n for the end of the line and to calculate  
the split it add +1 to the value (to grab the \r).


I think you can change +1 to -1 and it should keep the \n\r at the end  
of the header.


Brian








On Thu, 2008-06-26 at 10:29 -0400, Brian J. France wrote:

On Jun 26, 2008, at 9:48 AM, Kiffin Gish wrote:

Thanks Brian, I've tried what you suggested but it doesn't seem to
work.
Could you be more specific? Here's what I've tried:

void register_hooks(apr_pool_t *p)
{
   ap_hook_translate_name(hook_translate_name, NULL, NULL,
APR_HOOK_REALLY_FIRST);
   ap_hook_pre_connection(hook_pre_connection, NULL, NULL,
APR_HOOK_MIDDLE);
   ap_register_output_filter(wms-error-http-header,
wmserror_ofilter,
   NULL, AP_FTYPE_CONNECTION) ;
}

static int hook_translate_name (request_rec *r)
{
   apr_table_setn(r-notes, MY_NOTE, .);
   ap_custom_response(r, HTTP_INTERNAL_SERVER_ERROR, xml);
   return HTTP_INTERNAL_SERVER_ERROR;
}

static int hook_pre_connection(conn_rec *c, void *csd)
{
   ap_add_output_filter(wms-error-httpd-header, NULL, NULL, c);
   return OK;
}

int wmserror_ofilter(ap_filter_t* f, apr_bucket_brigade* bb)
{
   const char *t = apr_table_get(f-r-notes, MY_NOTE);
   if (t != NULL) { ap_set_content_type(f-r, text/xml); }
   return ap_pass_brigade(f-next, bb) ;
}



My guess would be you can't use ap_set_content_type in your filter,
you will have to examine the buckets, find Conetent-Type and change  
it

to text/xml.

Below is a quick hack I created from code we use to remove the server
header and insert the host comment.  It may not work perfectly out of
the box, but by adding some log lines you should be able to get it
working.

This also assume the headers will fit in a 8k brigade.

Brian



/* Because strnstr is not on RHEL4 */
static char *__strnstr(const char *big, const char *little, size_t  
len)

{
size_t little_len = strlen(little);
size_t i;
for (i = 0; i = len - little_len; i++) {
if (memcmp(big + i, little, little_len) == 0) {
return (char *)(big + i);
}
}

return 0;
}

static apr_status_t wmserror_output_filter(ap_filter_t *f,
apr_bucket_brigade *in)
{
  if (f-r != NULL  apr_table_get(f-r-notes, MY_NOTE) !=  
NULL) {


apr_bucket *b;

for (b = APR_BRIGADE_FIRST(in); b != APR_BRIGADE_SENTINEL(in) 
done == 0; b = APR_BUCKET_NEXT(b)) {
  const char *buf;
  size_t bytes ;

  if (!(APR_BUCKET_IS_METADATA(b))) {
if (apr_bucket_read(b, buf, bytes, APR_BLOCK_READ) ==
APR_SUCCESS) {
  char *ct_header = __strnstr(buf, Content-Type: , bytes);
  char *end;
  if (cl_header != NULL  (end = strstr(ct_header, \n)) !=
NULL) {
apr_bucket *newb = NULL;
apr_bucket *new_ct;

/* split off buffer at the ct header */
apr_bucket_split(b, ct_header +  14 - buf);

/* skip to the ct value bucket */
b = APR_BUCKET_NEXT(b);

/* split off after value */
apr_bucket_split(b, end - ct_header + 14 + 1);

/* skip to the next one */
newb = APR_BUCKET_NEXT(b);

/* remove it */
APR_BUCKET_REMOVE(b);

/* nuke it */
apr_bucket_destroy(b);

b = newb;

/* I think this is the right function */
new_ct = apr_bucket_immortal_create(text/xml,
sizeof(text/xml), f-c-bucket_alloc);

APR_BUCKET_INSERT_BEFORE(b, new_ct);

apr_table_unset(f-r-notes, MY_NOTE);
break;
  }
}
  }
}
  }

  /* send the data up the stack */
  return ap_pass_brigade(f-next,in);
}



static void wmserror_insert_output_filter(request_rec *r)
{
ap_add_output_filter(WMSERROR_OUTPUT_FILTER, NULL, r, r-

connection);

}

static void register_hooks(apr_pool_t *p)
{
ap_register_output_filter(WMSERROR_OUTPUT_FILTER,
wmserror_output_filter, NULL, AP_FTYPE_PROTOCOL);

ap_hook_insert_filter(wmserror_insert_output_filter, NULL, NULL,
APR_HOOK_REALLY_LAST);
}







On Thu, 2008-06-26 at 08:35 -0400, Brian J. France wrote:

On Jun 26, 2008, at 3:33 AM, Kiffin Gish wrote:

In order to reply with my own xml error, I want to use
ap_custom_response(r, HTTP_INTERNAL_SERVER_ERROR, xml);

However, default content type is text/html. If I try to change

it by

using ap_set_content_type(r, text/xml), this has no effect.

Is there anyone out there who can help me?



Ran into the same thing with apache 1.3.  We have a patch that

adds a
custom hook

Re: ap_custom_response content type 'text/xml'

[Brian J. France]

On Jun 27, 2008, at 9:28 AM, Kiffin Gish wrote:

I tried your suggestion but it doesn't help. To be more specific, I  
want

to convert:

HTTP/1.1 500 Internal Server Error
Date: Fri, 27 Jun 2008 13:25:56 GMT
Server: Apache/2.2.3 (Unix)
Content-Length: 1066
Connection: close
Content-Type: text/html; charset=iso-8859-1

?xml version='1.0' encoding=UTF-8 standalone=no ?
...

to:

HTTP/1.1 500 Internal Server Error
Date: Fri, 27 Jun 2008 13:25:56 GMT
Server: Apache/2.2.3 (Unix)
Content-Length: 1066
Connection: close
Content-Type: text/xml; charset=iso-8859-1

?xml version='1.0' encoding=UTF-8 standalone=no ?
...

Thanks in advance for your help.



Looks like you need to look for ; and do the -1 or make you insert  
string text/xml; charset=iso-8859-1 and still do the -1.


Biran







On Fri, 2008-06-27 at 09:02 -0400, Brian J. France wrote:
 On Jun 27, 2008, at 4:00 AM, Kiffin Gish wrote:

  Brian,
 
  This is great, thanks alot!
 
  I've 'almost' got it working correctly, the only problem being  
that

  the
  Content-Type is not getting truncated correctly, e.g. instead of:
 
  Content-Type: text/xml\n
  \n
  ?xml version='1.0' encoding=UTF-8 standalone=no ?\n
 
 
  I'm getting this:
 
  Content-Type: text/xml?xml version='1.0' encoding=UTF-8
  standalone=no ?\n
 
  Any tips would be greatly appreciated.


 In my example it looks for \n for the end of the line and to  
calculate

 the split it add +1 to the value (to grab the \r).

 I think you can change +1 to -1 and it should keep the \n\r at the  
end

 of the header.

 Brian




 
 
 
  On Thu, 2008-06-26 at 10:29 -0400, Brian J. France wrote:
  On Jun 26, 2008, at 9:48 AM, Kiffin Gish wrote:
  Thanks Brian, I've tried what you suggested but it doesn't  
seem to

  work.
  Could you be more specific? Here's what I've tried:
 
  void register_hooks(apr_pool_t *p)
  {
 ap_hook_translate_name(hook_translate_name, NULL, NULL,
  APR_HOOK_REALLY_FIRST);
 ap_hook_pre_connection(hook_pre_connection, NULL, NULL,
  APR_HOOK_MIDDLE);
 ap_register_output_filter(wms-error-http-header,
  wmserror_ofilter,
 NULL, AP_FTYPE_CONNECTION) ;
  }
 
  static int hook_translate_name (request_rec *r)
  {
 apr_table_setn(r-notes, MY_NOTE, .);
 ap_custom_response(r, HTTP_INTERNAL_SERVER_ERROR, xml);
 return HTTP_INTERNAL_SERVER_ERROR;
  }
 
  static int hook_pre_connection(conn_rec *c, void *csd)
  {
 ap_add_output_filter(wms-error-httpd-header, NULL, NULL,  
c);

 return OK;
  }
 
  int wmserror_ofilter(ap_filter_t* f, apr_bucket_brigade* bb)
  {
 const char *t = apr_table_get(f-r-notes, MY_NOTE);
 if (t != NULL) { ap_set_content_type(f-r, text/xml); }
 return ap_pass_brigade(f-next, bb) ;
  }
 
 
  My guess would be you can't use ap_set_content_type in your  
filter,
  you will have to examine the buckets, find Conetent-Type and  
change

  it
  to text/xml.
 
  Below is a quick hack I created from code we use to remove the  
server
  header and insert the host comment.  It may not work perfectly  
out of
  the box, but by adding some log lines you should be able to get  
it

  working.
 
  This also assume the headers will fit in a 8k brigade.
 
  Brian
 
 
 
  /* Because strnstr is not on RHEL4 */
  static char *__strnstr(const char *big, const char *little,  
size_t

  len)
  {
  size_t little_len = strlen(little);
  size_t i;
  for (i = 0; i = len - little_len; i++) {
  if (memcmp(big + i, little, little_len) == 0) {
  return (char *)(big + i);
  }
  }
 
  return 0;
  }
 
  static apr_status_t wmserror_output_filter(ap_filter_t *f,
  apr_bucket_brigade *in)
  {
if (f-r != NULL  apr_table_get(f-r-notes, MY_NOTE) !=
  NULL) {
 
  apr_bucket *b;
 
  for (b = APR_BRIGADE_FIRST(in); b !=  
APR_BRIGADE_SENTINEL(in) 

  done == 0; b = APR_BUCKET_NEXT(b)) {
const char *buf;
size_t bytes ;
 
if (!(APR_BUCKET_IS_METADATA(b))) {
  if (apr_bucket_read(b, buf, bytes, APR_BLOCK_READ) ==
  APR_SUCCESS) {
char *ct_header = __strnstr(buf, Content-Type: ,  
bytes);

char *end;
if (cl_header != NULL  (end = strstr(ct_header,  
\n)) !=

  NULL) {
  apr_bucket *newb = NULL;
  apr_bucket *new_ct;
 
  /* split off buffer at the ct header */
  apr_bucket_split(b, ct_header +  14 - buf);
 
  /* skip to the ct value bucket */
  b = APR_BUCKET_NEXT(b);
 
  /* split off after value */
  apr_bucket_split(b, end - ct_header + 14 + 1);
 
  /* skip to the next one */
  newb = APR_BUCKET_NEXT(b);
 
  /* remove it */
  APR_BUCKET_REMOVE(b);
 
  /* nuke it */
  apr_bucket_destroy(b);
 
  b = newb;
 
  /* I think this is the right function */
  new_ct = apr_bucket_immortal_create(text/xml,
  sizeof(text/xml), f-c

Re: ap_custom_response content type 'text/xml'

[Brian J. France]

On Jun 27, 2008, at 11:04 AM, Kiffin Gish wrote:

Sorry to keep bugging you, but where does this -1 belong?


In my example there was this line:

/* split off after value */
apr_bucket_split(b, end - ct_header + 14 + 1);

+ 14 is to skip the Content-Type: 
+ 1 is to skip the \r

This is where you want to change that to -1 instead to keep the \n (or  
the ; if you scan for ; instead of ;).


Brian

Re: ap_custom_response content type 'text/xml'

[Brian J. France]

On Jun 26, 2008, at 3:33 AM, Kiffin Gish wrote:

In order to reply with my own xml error, I want to use
ap_custom_response(r, HTTP_INTERNAL_SERVER_ERROR, xml);

However, default content type is text/html. If I try to change it by
using ap_set_content_type(r, text/xml), this has no effect.

Is there anyone out there who can help me?



Ran into the same thing with apache 1.3.  We have a patch that adds a  
custom hook that is called before headers are sent and we can re-set  
it back to text/xml there (remember this is 1.3).


You could call ap_custom_response, set a flag in r-notes, have a  
output filter in your module that checks r-notes and if the flag set  
it scans for Content-type and resets it to text/xml.


Brian

Re: ap_custom_response content type 'text/xml'

[Brian J. France]

On Jun 26, 2008, at 9:48 AM, Kiffin Gish wrote:
Thanks Brian, I've tried what you suggested but it doesn't seem to  
work.

Could you be more specific? Here's what I've tried:

void register_hooks(apr_pool_t *p)
{
ap_hook_translate_name(hook_translate_name, NULL, NULL,
APR_HOOK_REALLY_FIRST);
ap_hook_pre_connection(hook_pre_connection, NULL, NULL,
APR_HOOK_MIDDLE);
ap_register_output_filter(wms-error-http-header,  
wmserror_ofilter,

NULL, AP_FTYPE_CONNECTION) ;
}

static int hook_translate_name (request_rec *r)
{
apr_table_setn(r-notes, MY_NOTE, .);
ap_custom_response(r, HTTP_INTERNAL_SERVER_ERROR, xml);
return HTTP_INTERNAL_SERVER_ERROR;
}

static int hook_pre_connection(conn_rec *c, void *csd)
{
ap_add_output_filter(wms-error-httpd-header, NULL, NULL, c);
return OK;
}

int wmserror_ofilter(ap_filter_t* f, apr_bucket_brigade* bb)
{
const char *t = apr_table_get(f-r-notes, MY_NOTE);
if (t != NULL) { ap_set_content_type(f-r, text/xml); }
return ap_pass_brigade(f-next, bb) ;
}



My guess would be you can't use ap_set_content_type in your filter,  
you will have to examine the buckets, find Conetent-Type and change it  
to text/xml.


Below is a quick hack I created from code we use to remove the server  
header and insert the host comment.  It may not work perfectly out of  
the box, but by adding some log lines you should be able to get it  
working.


This also assume the headers will fit in a 8k brigade.

Brian



/* Because strnstr is not on RHEL4 */
static char *__strnstr(const char *big, const char *little, size_t len)
{
size_t little_len = strlen(little);
size_t i;
for (i = 0; i = len - little_len; i++) {
if (memcmp(big + i, little, little_len) == 0) {
return (char *)(big + i);
}
}

return 0;
}

static apr_status_t wmserror_output_filter(ap_filter_t *f,  
apr_bucket_brigade *in)

{
  if (f-r != NULL  apr_table_get(f-r-notes, MY_NOTE) != NULL) {

apr_bucket *b;

for (b = APR_BRIGADE_FIRST(in); b != APR_BRIGADE_SENTINEL(in)   
done == 0; b = APR_BUCKET_NEXT(b)) {

  const char *buf;
  size_t bytes ;

  if (!(APR_BUCKET_IS_METADATA(b))) {
if (apr_bucket_read(b, buf, bytes, APR_BLOCK_READ) ==  
APR_SUCCESS) {

  char *ct_header = __strnstr(buf, Content-Type: , bytes);
  char *end;
  if (cl_header != NULL  (end = strstr(ct_header, \n)) !=  
NULL) {

apr_bucket *newb = NULL;
apr_bucket *new_ct;

/* split off buffer at the ct header */
apr_bucket_split(b, ct_header +  14 - buf);

/* skip to the ct value bucket */
b = APR_BUCKET_NEXT(b);

/* split off after value */
apr_bucket_split(b, end - ct_header + 14 + 1);

/* skip to the next one */
newb = APR_BUCKET_NEXT(b);

/* remove it */
APR_BUCKET_REMOVE(b);

/* nuke it */
apr_bucket_destroy(b);

b = newb;

/* I think this is the right function */
new_ct = apr_bucket_immortal_create(text/xml,  
sizeof(text/xml), f-c-bucket_alloc);


APR_BUCKET_INSERT_BEFORE(b, new_ct);

apr_table_unset(f-r-notes, MY_NOTE);
break;
  }
}
  }
}
  }

  /* send the data up the stack */
  return ap_pass_brigade(f-next,in);
}



static void wmserror_insert_output_filter(request_rec *r)
{
ap_add_output_filter(WMSERROR_OUTPUT_FILTER, NULL, r, r- 
connection);

}

static void register_hooks(apr_pool_t *p)
{
ap_register_output_filter(WMSERROR_OUTPUT_FILTER,  
wmserror_output_filter, NULL, AP_FTYPE_PROTOCOL);


ap_hook_insert_filter(wmserror_insert_output_filter, NULL, NULL,  
APR_HOOK_REALLY_LAST);

}







On Thu, 2008-06-26 at 08:35 -0400, Brian J. France wrote:
 On Jun 26, 2008, at 3:33 AM, Kiffin Gish wrote:
  In order to reply with my own xml error, I want to use
  ap_custom_response(r, HTTP_INTERNAL_SERVER_ERROR, xml);
 
  However, default content type is text/html. If I try to change  
it by

  using ap_set_content_type(r, text/xml), this has no effect.
 
  Is there anyone out there who can help me?
 

 Ran into the same thing with apache 1.3.  We have a patch that  
adds a

 custom hook that is called before headers are sent and we can re-set
 it back to text/xml there (remember this is 1.3).

 You could call ap_custom_response, set a flag in r-notes, have a
 output filter in your module that checks r-notes and if the flag  
set

 it scans for Content-type and resets it to text/xml.

 Brian

--
Kiffin Gish | Desktop  Services Development | TomTom | [EMAIL PROTECTED] 
 | +31 (0) 6 15529214 mobile | +31 (0) 20 757 5000 office

Re: buildconf and apr source

[Brian J. France]

On Jun 13, 2008, at 9:14 AM, Nick Kew wrote:

If you try building from svn, buildconf complains of no
apr/apr-util source.

Why does buildconf actually need the sources?
Shouldn't an installed version be sufficient?



Glad to see I am not the only one confused by this.  I have been  
working on a new build at work and added new patches that require ./ 
buildconf to be run, couldn't figure out why it needed the source  
either since I have apr/apr-util installed already.


Brian

mod_deflate Vary header tweak

[Brian J. France]

I would like to propose a change to mod_deflate that would still send  
the Vary header if the request is flagged with no-gzip or gzip-only- 
text/html.


I think the checks for Content-Range and Content-Encoding should be  
done, then the Vary header should be set and then the checks for no- 
gzip and gzip-only-text/html should be done.


Thoughts?

Brian


Index: mod_deflate.c
===
--- mod_deflate.c   (revision 652071)
+++ mod_deflate.c   (working copy)
@@ -422,34 +422,12 @@
 return ap_pass_brigade(f-next, bb);
 }

-/* some browsers might have problems, so set no-gzip
- * (with browsermatch) for them
- */
-if (apr_table_get(r-subprocess_env, no-gzip)) {
-ap_remove_output_filter(f);
-return ap_pass_brigade(f-next, bb);
-}
-
 /* We can't operate on Content-Ranges */
 if (apr_table_get(r-headers_out, Content-Range) != NULL) {
 ap_remove_output_filter(f);
 return ap_pass_brigade(f-next, bb);
 }

-/* Some browsers might have problems with content types
- * other than text/html, so set gzip-only-text/html
- * (with browsermatch) for them
- */
-if (r-content_type == NULL
- || strncmp(r-content_type, text/html, 9)) {
-const char *env_value = apr_table_get(r-subprocess_env,
-  gzip-only-text/ 
html);

-if ( env_value  (strcmp(env_value,1) == 0) ) {
-ap_remove_output_filter(f);
-return ap_pass_brigade(f-next, bb);
-}
-}
-
 /* Let's see what our current Content-Encoding is.
  * If it's already encoded, don't compress again.
  * (We could, but let's not.)
@@ -500,6 +478,28 @@
  */
 apr_table_mergen(r-headers_out, Vary, Accept-Encoding);

+/* some browsers might have problems, so set no-gzip
+ * (with browsermatch) for them
+ */
+if (apr_table_get(r-subprocess_env, no-gzip)) {
+ap_remove_output_filter(f);
+return ap_pass_brigade(f-next, bb);
+}
+
+/* Some browsers might have problems with content types
+ * other than text/html, so set gzip-only-text/html
+ * (with browsermatch) for them
+ */
+if (r-content_type == NULL
+ || strncmp(r-content_type, text/html, 9)) {
+const char *env_value = apr_table_get(r-subprocess_env,
+  gzip-only-text/ 
html);

+if ( env_value  (strcmp(env_value,1) == 0) ) {
+ap_remove_output_filter(f);
+return ap_pass_brigade(f-next, bb);
+}
+}
+
 /* force-gzip will just force it out regardless if the browser
  * can actually do anything with it.
  */

internal_internal_redirect and ap_run_create_request bug?

[Brian J. France]

Could somebody else review when ap_run_create_request is called in  
both internal_internal_redirect (modules/http/http_request.c) and  
ap_read_request (server/protocol.c)?


I think a few more things need to be setup before calling  
ap_run_create_request in internal_internal_redirect.


What I am having problems with is that r-subprocess_env is NULL when  
calling ap_run_create_request in internal_internal_redirect, but is  
valid for ap_read_request.


This means a module that wants to hook ap_run_create_request to make  
sure something is always added to the subprocess_env, can't because  
sometimes (internal_internal_redirect) it is NULL.


I think other things need to be setup as well like notes, headers_in,  
headers_out, etc.


Thoughts?

Brian

Re: Passing parameters to PHP

[Brian J. France]

On Aug 15, 2007, at 3:46 PM, Sam Carleton wrote:

Is there any way for my apache module to pass parameters in such a way
that my php code can get at them?


Stick the data in r-subprocess_env and PHP can access them via  
$_SERVER (or apache_getenv if $_SERVER is disabled).


Or you can put them in r-notes and use apache_note for access.

Brian

-X and SIGTERM

[Brian J. France]

Here is a patch that will allow SIGTERM to work with -X

http://www.brianfrance.com/software/apache/httpd.signal.diff

Without the patch running httpd on the command line with -X and  
trying to stop it can only be done by backgrounding it and then kill  
-9'ing it (ctrl-c doesn't work).


Brian

Re: Patch for implementing ap_document_root as a hook

[Brian J. France]

On Apr 26, 2007, at 8:35 AM, Jim Jagielski wrote:
We currently hack the doc root in the post read hook in 1.3, would  
like to be able to do it with out hacking the core and screwing  
around with internal structs at runtime.


VERY doubtful that 1.3 will be updated to do this.


I don't need it in 1.3, but I would like to have a clean way to do it  
in 2.x that doesn't include replacing data in the internal structs at  
runtime and putting it back at the end of the request.


Brian

Re: Patch for implementing ap_document_root as a hook

[Brian J. France]

On Apr 23, 2007, at 10:32 AM, Jakob Goldbach wrote:

-1 on the face of things.  The map_to_storage hook was added to  
accomplish

what you desire.


I thought map_to_storage was made to do per-dir configuration. Not
path-translation.

The problem is not really doing the translation. I can always provide
my own translate handler in my module.

But in the current API I cannot to set my env. variables at will. They
will be overwritten by ap_add_common_vars which returns
carved-in-stone docroot from ap_document_root.


We need this same functionality (would like to back port to 2.2 if  
possible).


We currently hack the doc root in the post read hook in 1.3, would  
like to be able to do it with out hacking the core and screwing  
around with internal structs at runtime.


Brian





My only other option is to patch every single module which calls
add_common_vars, that is,  cgi,cgid, fastcgi, includes,...

I thought a document_root hook was more elegant. Or maybe a
add_common_vars hook? [I would be happy to supply it]


Unfortunately it is not coupled to the DOCUMENT_ROOT
variable, but I'd look at remedying this over building on  
ap_document_root,

which should simply go away, IMHO.


What's so bad about ap_document_root?  I know the source says 'dont
use' because it won't be right with mod_userdir etc. But with a hook
it would be possible to get right.

/Jakob

Re: mod_authn_dbd and apr_password_validate

[Brian J. France]

On Jan 6, 2007, at 1:48 PM, Patrick Welche wrote:

/*
 * Validate a plaintext password against a smashed one.  Uses either
 * crypt() (if available) or apr_md5_encode() or apr_sha1_base64(),  
depending
 * upon the format of the smashed input password.  Returns  
APR_SUCCESS if
 * they match, or APR_EMISMATCH if they don't.  If the platform  
doesn't
 * support crypt, then the default check is against a clear text  
string.

 */
APU_DECLARE(apr_status_t) apr_password_validate(const char *passwd,
const char *hash)


but here, it looks as though we have to validate a plaintext  
password against

a plaintext password.

What am I missing?


That mod_authn_dbd is assuming your database password is not plain  
text, but smashed (crypt, md5, sh1).  I ran into the same issue and  
created this:


  http://www.brianfrance.com/software/apache/mod_authn_dbd.c.diff

Apply the patch and add this to your config:

  AuthDBDPlainTextPasswords on

and you should be set.

Brian

Re: [PATCH 40026] ServerTokens Off

[Brian J. France]

On Aug 2, 2006, at 3:57 AM, William A. Rowe, Jr. wrote:


Sebastian Nohn wrote:


please take the time to read it before voting against the proposal :)


I am all for this patch (I know my vote means nothing)!

I've read your comments, agree it's 17 bytes (that you can just as  
well remove,

as you point out, by hand.)


I have written a protocol output filter that removes the server  
header, but would much rather have a config directive.


I'm curious - do IE, Firefox or other common clients use the server  
name tag
as a clue for fixups around aberrant behavior or to enable optimal  
behavior?


We (Yahoo!) have run for years without sending the Server header and  
have not had any problem.  I think it is more likely a case of the  
server detecting the browser and tweaking the output to get around  
browser bugs.


Brian

Re: Re[2]: apache 2.2 crashes at the start time in mod_dbd.c then preparing AuthDBDUserPWQuery

[Brian J. France]

I think this is the same issue I had:

http://www.mail-archive.com/dev@httpd.apache.org/msg31299.html

Not sure if there was a fix committed into the code tree or not.

Brian


On Jul 20, 2006, at 3:52 PM, Anton Golubev wrote:


Hello Nick,

Here is here the minimal complete config, which crashes the server:

DBDriver mysql
DBDParams dbname=Users user=auther passreplace=IrjkfN3
DBDMin 1
DBDKeep 2
DBDMax 10
DBDExptime 60

ServerRoot /usr/local/apache

Listen 80

User nobody
Group nobody

ServerAdmin [EMAIL PROTECTED]
NameVirtualHost 85.142.33.11

VirtualHost 85.142.33.11
DocumentRoot /home/ivc2/public_html

Directory /home/ivc2/public_html

AuthType Basic
AuthName 'ADMIN ZONE'
AuthBasicProvider dbd

AuthDBDUserPWQuery select PASS from Users where LOGIN= %s

/Directory

/VirtualHost


Another backtrace:

(gdb) run -f /usr/local/apache/conf/httpd-mini.conf
Starting program: /usr/local/apache2_2/bin/httpd -f /usr/local/ 
apache/conf/httpd-mini.conf

[Thread debugging using libthread_db enabled]
[New Thread -1208047936 (LWP 449)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1208047936 (LWP 449)]
ap_dbd_prepare (s=0x0, query=0x9aa64f8 select PASS from Users  
where LOGIN= %s, label=0x9aa64f8 select PASS from Users where  
LOGIN= %s) at mod_dbd.c:149

149 prepared-next = svr-prepared;
(gdb) bt
#0  ap_dbd_prepare (s=0x0, query=0x9aa64f8 select PASS from Users  
where LOGIN= %s, label=0x9aa64f8 select PASS from Users where  
LOGIN= %s)

at mod_dbd.c:149
#1  0x08080415 in authn_dbd_prepare (cmd=0xbfef6c70, cfg=0x9aa64f0,  
query=0x9aa64f8 select PASS from Users where LOGIN= %s) at  
mod_authn_dbd.c:70
#2  0x0807880b in invoke_cmd (cmd=0x80d30a0, parms=0xbfef6c70,  
mconfig=0x9aa64f0, args=0x9a937a0 ) at config.c:873
#3  0x08079075 in ap_walk_config (current=0x9a93758,  
parms=0xbfef6c70, section_vector=0x9aa5fb0) at config.c:1141
#4  0x08070d09 in dirsection (cmd=0xbfef6c70, mconfig=0x9aa5a88,  
arg=0x9aa6256 ) at core.c:1865
#5  0x0807880b in invoke_cmd (cmd=0x80d0a80, parms=0xbfef6c70,  
mconfig=0x9aa5a88, args=0x9a93670 /home/ivc2/public_html) at  
config.c:873
#6  0x08079075 in ap_walk_config (current=0x9a93650,  
parms=0xbfef6c70, section_vector=0x9aa56f0) at config.c:1141
#7  0x08071553 in virtualhost_section (cmd=0xbfef6c70,  
dummy=0x9a8efe8, arg=0x9a935e8 85.142.33.11) at core.c:2206
#8  0x0807880b in invoke_cmd (cmd=0x80d0ab0, parms=0xbfef6c70,  
mconfig=0x9a8efe8, args=0x9a935e8 85.142.33.11) at config.c:873
#9  0x08079075 in ap_walk_config (current=0x9a935c8,  
parms=0xbfef6c70, section_vector=0x9a8df00) at config.c:1141
#10 0x08079ace in ap_process_config_tree (s=0x9a4e570,  
conftree=0x9a932b8, p=0x9a430a8, ptemp=0x9a8b240) at config.c:1743

#11 0x08067eeb in main (argc=3, argv=0xbfef6dd4) at main.c:621
(gdb)


I'm wondering how could I catch it with so many people luckily using
this feature around? Otherwise I would be proud to be the first  
one! :)



Greets,
Anton Golubev
ENGECON
St. Petersburg
Russia

mod_deflate patch

[Brian J. France]

This was talked about a few weeks back but I don't think anything  
ever came of it.  The patch below would allow mod_deflate to compress  
internal redirects while still skipping sub requests.


I have been running this on my personal server for a few weeks now  
with no issues.


Let the voting begin.

Brian

Index: modules/filters/mod_deflate.c
===
--- modules/filters/mod_deflate.c   (revision 394996)
+++ modules/filters/mod_deflate.c   (working copy)
@@ -240,7 +240,7 @@
 const char *encoding;
 /* only work on main request/no subrequests */
-if (!ap_is_initial_req(r)) {
+if (r-main != NULL) {
 ap_remove_output_filter(f);
 return ap_pass_brigade(f-next, bb);
 }

Re: mod_deflate patch

[Brian J. France]

On Apr 18, 2006, at 2:52 PM, William A. Rowe, Jr. wrote:

@@ -240,7 +240,7 @@
 const char *encoding;
 /* only work on main request/no subrequests */
-if (!ap_is_initial_req(r)) {
+if (r-main != NULL) {
 ap_remove_output_filter(f);


Actually, explain to me how this code successfully leaves the http  
protocol

layer output_filter in the filter chain for subrequest components?


Using ap_is_initial_req:

AP_DECLARE(int) ap_is_initial_req(request_rec *r)
{
return (r-main == NULL)   /* otherwise, this is a sub- 
request */
(r-prev == NULL);   /* otherwise, this is an internal  
redirect */

}

it will remove the filter for both sub-request and internal  
redirects.  The patch just removes the filter if it is a sub request.



  I'd think
this code (original, and even the patched flavor) could break the  
filter stack
by yanking the deflate filter out from the middle of servicing a  
request, e.g.

when a subrequest is included midstream.


The patched block of code is only called when f-ctx is NULL and  
hasn't been setup yet by mod_deflate.  I would assume when a sub  
request would get added the ctx for its ap_filter_t struct would be  
NULL and f-r-main would be the top request so the deflate filter  
would be removed.



It seems this should be a conditional add-filter, never a  
conditional remove
filter event.  add-filter on the top level request, noop on nested  
requests.


Not sure I have the expertise to comment on that.

Brian

mod_deflate and internal redirects

[Brian J. France]

I was digging into why mod_deflate wasn't compressing some pages and  
found that any internal redirects (rewriterule) don't get compress.   
That is because of this code:


/* only work on main request/no subrequests */
if (!ap_is_initial_req(r)) {
ap_remove_output_filter(f);
return ap_pass_brigade(f-next, bb);
}

The problem is ap_is_initial_req is:

  AP_DECLARE(int) ap_is_initial_req(request_rec *r)
  {
  return (r-main == NULL)   /* otherwise, this is a sub- 
request */
  (r-prev == NULL);   /* otherwise, this is an  
internal redirect */

  }

r-main will be NULL, but r-prev is valid for internal redirects.   
This was talked about on IRC and Brian Akins mentioned creating a  
ap_is_subrequest call:


  #define ap_is_subrequest(r) (r-main != NULL)

and that mod_deflate could use it instead of the ap_is_initial_req  
call so it could compress internal redirects.


Thoughts?

Brian

mod_dbd crash

[Brian J. France]

In a virtual host block if you forget to add DBDriver or DBDParams in  
the main section, but add a AuthBasicProvider dbd and  
AuthDBDUserPWQuery in a Location block it will core dump while  
processing AuthDBDUserPWQuery (see below).


This is with 2.2.0 and trunk (core dump from trunk).  The problem is  
when ap_dbd_prepare is called to process AuthDBDUserPWQuery, the svr  
config pointer will be NULL and not checked before being used.


A quick fix is to have ap_dbd_prepare print a warning and abort, but  
I think a better fix it to have ap_dbd_prepare return a status value  
(or maybe a string pointer or something) that mod_authn_dbd and other  
modules could check and raise an error.


Here is a patch to mod_dbd.c and mod_authn_dbd.c (not sure if  
anything else uses that function) that returns a status code:


  http://www.brianfrance.com/software/apache/2006.03.17.diff

Brian


142 DBD_DECLARE_NONSTD(void) ap_dbd_prepare(server_rec *s, const char  
*query,

143 const char *label)
144 {
145 svr_cfg *svr = ap_get_module_config(s-module_config,  
dbd_module);
146 dbd_prepared *prepared = apr_pcalloc(s-process-pool, sizeof 
(dbd_prepared));

147 prepared-label = label;
148 prepared-query = query;
149 prepared-next = svr-prepared;
150 svr-prepared = prepared;
151 }

sudo gdb ./httpd
(gdb) run -t
Program received signal SIGSEGV, Segmentation fault.
ap_dbd_prepare (s=0x57c2a8, query=0x659f80 query, label=0x65a038  
authn_dbd_6) at mod_dbd.c:149

149 prepared-next = svr-prepared;
(gdb) bt
#0  ap_dbd_prepare (s=0x57c2a8, query=0x659f80 query,  
label=0x65a038 authn_dbd_6) at mod_dbd.c:149
#1  0x000801832f7e in authn_dbd_prepare (cmd=0x7fffe990,  
cfg=0x659f48,

query=0x659f80 query) at mod_authn_dbd.c:73
#2  0x00433b03 in invoke_cmd (cmd=0x801933800,  
parms=0x7fffe990, mconfig=0x659f48, args=0x603325 ) at config.c: 
768
#3  0x0043448b in ap_walk_config (current=0x603230,  
parms=0x7fffe990, section_vector=0x659758) at config.c:1141
#4  0x0042b06c in urlsection (cmd=0x7fffe990,  
mconfig=0x57c290, arg=0x659c33 ) at core.c:1942
#5  0x00433b03 in invoke_cmd (cmd=0x44ffe8,  
parms=0x7fffe990, mconfig=0x657c28, args=0x603078 \/\) at  
config.c:768
#6  0x0043448b in ap_walk_config (current=0x603038,  
parms=0x7fffe990, section_vector=0x657650) at config.c:1141
#7  0x0042b7e0 in virtualhost_section (cmd=0x7fffe990,  
dummy=0x57c290, arg=0x657028 \hostm:443\) at core.c:2206
#8  0x00433b03 in invoke_cmd (cmd=0x450010,  
parms=0x7fffe990, mconfig=0x5b2570, args=0x6025f8 \host:443 
\) at config.c:768
#9  0x0043448b in ap_walk_config (current=0x6025b8,  
parms=0x7fffe990, section_vector=0x5b2150) at config.c:1141
#10 0x004352e2 in ap_process_config_tree (s=0x57c2a8,  
conftree=0x65a038, p=0x57d028, ptemp=0x0) at config.c:1743
#11 0x0042064d in main (argc=2, argv=0x7fffeb28) at  
main.c:616

(gdb) p svr
$1 = (svr_cfg *) 0x0
(gdb)

Re: mod_dbd crash

[Brian J. France]

On Mar 17, 2006, at 11:42 AM, Nick Kew wrote:

On Friday 17 March 2006 16:03, Brian J. France wrote:

Here is a patch to mod_dbd.c and mod_authn_dbd.c (not sure if
anything else uses that function) that returns a status code:

   http://www.brianfrance.com/software/apache/2006.03.17.diff


Thanks.  This patch is a change to the API/ABI, but since it's just a
void -- apr_status_t return value, it won't break anything.


Please tweak the patch as needed, this was just a 5 min hack to show  
what I was thinking.  Paul pointed out on IRC it should return an  
defined error instead of -1, so may be apr_status_t isn't the best fix.



Not sure if that's allowed within 2.2.x.


In that case may be add some code to print an error and then abort,  
since it is going to crash anyways.


Brian

Re: [mod_smtpd] patch need for the SIZE extension

[Brian J. France]

On Feb 8, 2006, at 12:42 AM, Rian Hunter wrote:

On Feb 7, 2006, at 8:15 PM, Brian J. France wrote:
  Before I started converting my other modules to the new code I  
figured I would start with writing a new module to handle the SIZE  
extension.  I needed to apply the following patch (link below) to  
the mod_smtpd code to get access at the max data size.


+0, Well this is an interesting point. The max data size should be  
settable and gettable from extending modules. In a way max_data is  
a lot like the list of extensions: should it be set on every  
connection or upon server initialization (ie should it's scope be  
per connection or per mod_smtpd instance). I chose that the list of  
extensions might want to be per-connection since modules may not  
want to offer all clients all extensions, just the same way modules  
may want to enforce different max_data sizes for different clients.  
Where do you think these variables belong?


One thing I do know is that max_data doesn't belong in mod_smtpd's  
configuration structure and rather either in some other per  
mod_smtpd instance structure accessible to extending modules or  
just in scr (the per connection structure). Right? I think that  
would be cleaner and more modular. We should talk about this some  
more before I apply this patch.


+1, I like the idea of the storing the setting in the per connection  
instance.  Leaving the default in the core and copying it to an per  
connection struct would allow modules to tweak the setting per  
connection.


One other addition to the smtpd_session_rec I would like to see would  
be something like r-notes or r-subprocess_env.  Some way for  
modules to set flags that would change the behavior of another  
module.  As an example I might want to add a new modules that changes  
the max data size based on either connection ip or mail from  
address.  This module would want to set a flag telling the size  
module to not show the numerical size limit in the ehlo response as  
it might change based on the the mail from address.


  I hooked the mail from hook, check for a valid SIZE in  
mail_parameters and check to make sure it is not over the limit.   
If it is over the limit I can use smtpd_respond_oneline to send  
the 552 message exceeds fixed maximum message size line back to  
the client, but what should the function return to make it force a  
QUIT or REST command as anything but SMTPD_DONE sends more stuff  
to the client.


Should I just return SMTPD_DONE and set scr-should_disconnect?   
Could we tweak it to support two different settings, one would  
only allow QUIT only and the other would allow QUIT and REST (to  
start over).


+1, Since quit and rset are both handled by mod_smtpd there should  
probably be another variable called scr-only_quit_or_rset, also  
because I think there are other times when the client should only  
issue a quit or rset. More discussion follows though:


My basic design strategy has been that mod_smtpd should do as  
little as possible or what will be practical to a large amount of  
extending modules. If mod_smtpd sets a variable in a structure it  
should use it. If an extending module sets a variable in a  
structure it should use it. Having an only_quit_or_rset variable  
in the scr structure with mod_smtpd consciously checking for it,  
but never changing the value itself sort of violates that strategy.  
That doesn't mean I'm against it because like I already mentioned I  
think a lot of modules will need this sort of thing.


Have you thought about hooking into all the commands, and then  
sending a 503 Only QUIT and RSET from your SIZE module until a  
rset is received?


I only added the scr-should_disconnect logic since it's what  
should happen when a simple module wants to SMTPD_DENY some  
connections. I didn't want every module that wanted to implement  
connection policy to have to deal with being hooked into all the  
commands, and just be able to return SMTPD_DENY.


The scr-should_disconnect situation I just explained (where I  
didn't want every modules to have to deal with hooking all the  
commands for a redundant task) could be applied to any potential  
module that wants to disallow any of the built in commands with  
instead a bit-field that specifies which mod_smtpd core commands  
are currently allowed. I think I like this idea the best, what do  
you think?


+1, if I understand you currently.  Changing scr-should_disconnect  
to scr-allows_commands which is a bit-field of what commands that  
are allowed.  Then smtp_protocol.c handles denying command instead of  
having a module hook every command and doing it, right?


Brian

Re: [mod_smtpd] patch need for the SIZE extension

[Brian J. France]

On Feb 8, 2006, at 9:28 PM, Rian Hunter wrote:

On Feb 8, 2006, at 10:56 AM, Brian J. France wrote:
+1, I like the idea of the storing the setting in the per  
connection instance.  Leaving the default in the core and copying  
it to an per connection struct would allow modules to tweak the  
setting per connection.


One other addition to the smtpd_session_rec I would like to see  
would be something like r-notes or r-subprocess_env.  Some way  
for modules to set flags that would change the behavior of another  
module.  As an example I might want to add a new modules that  
changes the max data size based on either connection ip or mail  
from address.  This module would want to set a flag telling the  
size module to not show the numerical size limit in the ehlo  
response as it might change based on the the mail from address.


Something like r-notes does exist, check scr-session_config.


session_config is a ap_conf_vector_t, which I think is more like r- 
per_dir_config or r-request_config.


r-notes and r-subprocess_env is a apr_table_t.

Brian

[mod_smtpd] patch need for the SIZE extension

[Brian J. France]

Hi Rian,

  Before I started converting my other modules to the new code I  
figured I would start with writing a new module to handle the SIZE  
extension.  I needed to apply the following patch (link below) to the  
mod_smtpd code to get access at the max data size.


  I hooked the mail from hook, check for a valid SIZE in  
mail_parameters and check to make sure it is not over the limit.  If  
it is over the limit I can use smtpd_respond_oneline to send the 552  
message exceeds fixed maximum message size line back to the client,  
but what should the function return to make it force a QUIT or REST  
command as anything but SMTPD_DONE sends more stuff to the client.


Should I just return SMTPD_DONE and set scr-should_disconnect?   
Could we tweak it to support two different settings, one would only  
allow QUIT only and the other would allow QUIT and REST (to start over).


Thanks,

Brian

http://www.brianfrance.com/software/apache/size.diff

Re: mod_smtpd changes

[Brian J. France]

On Dec 31, 2005, at 1:23 PM, Rian Hunter wrote:

Any comments, ideas and criticisms are highly welcomed! Thanks!


Any changes on how recipients and queue/deliver is handled?

I started implementing my proposal from a while back and got stuck on  
how to handle errors.  If a transaction has multiple valid recipients  
and each one is handle by different queue modules, but only one of  
queue modules returned a failed, is the whole thing a failure or  
should the other messages be delivered and what should be returned  
for a status for the DATA command?


I am thinking that if all recipients are valid and can be handle by a  
queue module then DATA will always return OK/200 what ever success  
is.  If a queue module then finds an error happens while trying to  
really deliver the message, a bounce need to be generated.  This  
means we should have a common way to generate a bounce message  
instead of having every module come up with its own.


Brian

Re: svn commit: r307031 - in /httpd/httpd/branches/2.2.x: CHANGES include/ap_mmn.h include/http_core.h modules/http/http_core.c server/core.c server/core_filters.c server/protocol.c

[Brian J. France]

Will this patch allow the removal of this code from mod_ftp  
ftp_ssl_init function:


/* This is handled in the NET_TIME filter, which unfortunately
 * ignores the timeout for the purpose of AP_MODE_INIT.
 * Fix a timeout so the core read filter will behave.
 */
client_socket = ap_get_module_config(cdata-conn_config,
 core_module);
apr_socket_timeout_set(client_socket, cdata-base_server-timeout);

http://svn.apache.org/viewcvs.cgi/incubator/mod_ftp/trunk/src/ 
ftp_connection.c?view=markup


Thanks,

Brian

BTW, did subversion lose the annotate web feature?

On Oct 7, 2005, at 1:41 AM, [EMAIL PROTECTED] wrote:

Author: wrowe
Date: Thu Oct  6 23:41:23 2005
New Revision: 307031

URL: http://svn.apache.org/viewcvs?rev=307031view=rev
Log:
Backport Revision: 306495

  NET_TIME, as a standalone feature, was a horrid idea, so eliminate  
it.


  The core filter will NOT operate correctly across platforms
  (even between Linux/Solaris) without setting up the conn-timeout,
  so always apply the timeout when establishing the core filter.

  The keep-alive-timeout is entirely an HTTP-ism, and needs to
  move to the http protocol handler.  This isn't triggered in the
  event mpm, but the event mpm introspects s-keep_alive_timeout
  directly adding it to the pollset, so this is a non-sequitor.

  Finally, once the headers are read, the named virtual host may
  have a different (more/less permissive) timeout for the remainder
  of the request body.  This http-centric patch picks up that subtle
  detail and can switch to a named-vhost timeout.


Modified:
httpd/httpd/branches/2.2.x/CHANGES
httpd/httpd/branches/2.2.x/include/ap_mmn.h
httpd/httpd/branches/2.2.x/include/http_core.h
httpd/httpd/branches/2.2.x/modules/http/http_core.c
httpd/httpd/branches/2.2.x/server/core.c
httpd/httpd/branches/2.2.x/server/core_filters.c
httpd/httpd/branches/2.2.x/server/protocol.c

Modified: httpd/httpd/branches/2.2.x/CHANGES
URL:  
http://svn.apache.org/viewcvs/httpd/httpd/branches/2.2.x/CHANGES? 
rev=307031r1=307030r2=307031view=diff
=== 
===

--- httpd/httpd/branches/2.2.x/CHANGES [utf-8] (original)
+++ httpd/httpd/branches/2.2.x/CHANGES [utf-8] Thu Oct  6 23:41:23 2005
@@ -1,6 +1,12 @@
 -*- coding:  
utf-8 -*-

 Changes with Apache 2.1.9

+  *) Elimiated the NET_TIME filter, restructuring the timeout logic.
+ This provides a working mod_echo on all platforms, and ensures  
any

+ custom protocol module is at least given an initial timeout value
+ based on the VirtualHost  context's Timeout directive.
+ [William Rowe]
+
   *) mod_proxy: Run the request_status hook also if there are no free  
workers

  or all workers are in error state.
  [Ruediger Pluem, Brian Akins brian.akins turner.com]

Modified: httpd/httpd/branches/2.2.x/include/ap_mmn.h
URL:  
http://svn.apache.org/viewcvs/httpd/httpd/branches/2.2.x/include/ 
ap_mmn.h?rev=307031r1=307030r2=307031view=diff
=== 
===

--- httpd/httpd/branches/2.2.x/include/ap_mmn.h (original)
+++ httpd/httpd/branches/2.2.x/include/ap_mmn.h Thu Oct  6 23:41:23  
2005

@@ -105,14 +105,15 @@
  * 20050701.1 (2.1.7-dev) trace_enable member added to core  
server_config

  * 20050708.0 (2.1.7-dev) Bump MODULE_MAGIC_COOKIE to AP22!
  * 20050708.1 (2.1.7-dev) add proxy request_status hook (minor)
-  */
+ * 20051006.0 (2.1.8-dev) NET_TIME filter eliminated
+ */

 #define MODULE_MAGIC_COOKIE 0x41503232UL /* AP22 */

 #ifndef MODULE_MAGIC_NUMBER_MAJOR
-#define MODULE_MAGIC_NUMBER_MAJOR 20050708
+#define MODULE_MAGIC_NUMBER_MAJOR 20051006
 #endif
-#define MODULE_MAGIC_NUMBER_MINOR 1 /* 0...n */
+#define MODULE_MAGIC_NUMBER_MINOR 0 /* 0...n */

 /**
  * Determine if the server's current MODULE_MAGIC_NUMBER is at least a

Modified: httpd/httpd/branches/2.2.x/include/http_core.h
URL:  
http://svn.apache.org/viewcvs/httpd/httpd/branches/2.2.x/include/ 
http_core.h?rev=307031r1=307030r2=307031view=diff
=== 
===

--- httpd/httpd/branches/2.2.x/include/http_core.h (original)
+++ httpd/httpd/branches/2.2.x/include/http_core.h Thu Oct  6 23:41:23  
2005

@@ -600,9 +600,6 @@
 AP_CORE_DECLARE_NONSTD(const char *) ap_limit_section(cmd_parms *cmd,  
void *dummy, const char *arg);


 /* Core filters; not exported. */
-int ap_net_time_filter(ap_filter_t *f, apr_bucket_brigade *b,
-   ap_input_mode_t mode, apr_read_type_e block,
-   apr_off_t readbytes);
 int ap_core_input_filter(ap_filter_t *f, apr_bucket_brigade *b,
  ap_input_mode_t mode, apr_read_type_e block,
  apr_off_t readbytes);
@@ -641,7 +638,6 @@
 extern 

mod_smtpd: handling rctp addresses

[Brian J. France]

I would like to propose a change on how rcpt to addresses are  
validated and messages are handled (expanded) and how queue modules  
know it needs to process this message.


I would like to change rcpt_to in smtpd_trans_rec to apr_table_t.  The  
key for the table will be the address sent to rcpt to command and the  
value will be a pointer to a apr_array_header_t list of pointers to the  
following structure:


typedef struct smtpd_rctp_rec {
apr_pool_t *p;
const char *mode;
int validated;
const char *address;
apr_table_t info;
const char *expanded;
}

mode: what queue handler should handle this message.
Example would be local, relay, relay_to, forward, postfix.

validated: if the rctp has been validated and ready to process

address: what address to send to

info: extra data is stored that the queue handler might needed

expanded: log of how the address was expanded

mod_smtpd would create a hook for expanding rctp address which other  
modules will use to enable different ways to validate rcpt address and  
setting what queue module should process a message.  It would take a  
smtpd_rctp_rec * and a apr_array_header_t * of things to add to add and  
return decline, ok or delete.


HANDLER_DECLARE(rcpt) will create a smtpd_rctp_rec and run the  
expanding hook and handle the return value and also the things to be  
add.


Here are a few example of how I think it would work. This will be using  
some modules that do not exists yet and some that do:


mod_smtpd_user_pwd - handle validating local users and setting info on  
them


mod_smtpd_alias - enables aliases (forwarding email)

mod_smtpd_queue_local - handles the local mode
Handles deliver of mail to local users

mod_smtpd_queue_relay - handles the relay and forward modes
This module would hook the expanding hook using
APR_HOOK_LAST since it will allow forwarding and
relaying for authenticated user, but wants to make
sure all other modules have a chance to change the
address if needed.

mod_smtpd_queue_smtp - handles the relay_to mode
Sends all mail to another server.
This module would hook the expanding hook using
APR_HOOK_FIRST since it want so to set every
smtpd_rctp_rec mode to replay_to and doesn't
care what other modules have to say.

Examples:

 



[EMAIL PROTECTED] - user

expand hook:
{ mode: NULL
  address: [EMAIL PROTECTED]
  validate: 0
  info: { }
  expanded: NULL
}

mod_smtpd_user_pwd handles request and returns ok.
It changes the struct to look like this:

{ mode: local
  address: [EMAIL PROTECTED]
  info: { mail_box - username }
  validate: 1
  expanded: [EMAIL PROTECTED] - local user username
}

mod_smtpd_queue_local would handle queuing

 



auth'ed user relaying mail to [EMAIL PROTECTED]

expand hook:
{ mode: NULL
  address: [EMAIL PROTECTED]
  validate: 0
  info: { }
  expanded: NULL
}

mod_smtpd_auth will handle this and returns ok.
It changes the struct to look like this:

{ mode: relay
  address: [EMAIL PROTECTED]
  info: { }
  validate: 1
  expanded: authenticated user user relaying [EMAIL PROTECTED]
}

mod_smtpd_queue_relay would handle queuing

 



[EMAIL PROTECTED] - [EMAIL PROTECTED]

expand hook:
{ mode: NULL
  address: [EMAIL PROTECTED]
  info: { }
  validate: 0
  expanded: NULL
}

mod_smtpd_alias would handle the request and return delete.
It adds a new entry to the list that looks like this:

{ mode: forward
  address: [EMAIL PROTECTED]
  info: { }
  validate: 0
  expanded : [EMAIL PROTECTED] - [EMAIL PROTECTED]
}

expand hook call on the new entry:

{ mode: forward
  address: [EMAIL PROTECTED]
  info: { }
  validate: 0
  expanded : [EMAIL PROTECTED] - [EMAIL PROTECTED]
}

mod_smtpd_forward would handle the request and return ok.
It would set the struct like this:

{ mode: forward
  address: [EMAIL PROTECTED]
  info: { }
  validate: 1
  expanded : [EMAIL PROTECTED] - [EMAIL PROTECTED]
}

mod_smtpd_queue_relay would handle queuing

 



[EMAIL PROTECTED]
- [EMAIL PROTECTED]
- [EMAIL 

new/update mod_smtpd modules

[Brian J. France]

Here are some new or update mod_smtpd modules:

http://www.brianfrance.com/software/apache/

mod_smtpd_auth.tar.gz:
mod_smtpd_auth - core auth module
mod_smtpd_authm_plain - auth method plain
mod_smtpd_authm_login - auth method login
mod_smtpd_authm_crammd5 - auth method crammd5
mod_smtpd_authm_digestmd5 - auth method digestmd5
mod_smtpd_authv_dbd - auth verify/lookup via dbd

	I have rewritten this module to use the unrecognized_command hook.  It 
uses defines a hook for methods and a hook for lookups and all method 
code was split off into their own module.


mod_smtpd_access.tar.gz:
mod_smtpd_access - core access module - connection, greeting, from, to
mod_smtpd_accessl_dbd - lookup module via dbd

	After finishing the auth rewrite I like the way it worked so I did the 
same for the access module.  mod_smtpd_access now has a lookup hook 
that accessl modules can use to plugin.  I will be adding a relay look 
up once we have some way of setting relaying is allowed.


Warning: apreq has problems with multi-line headers and will core dump 
when it gets them (has this been fixed?)


mod_smtpd_clamd.tar.gz
	This is a module that sends the mail to clamd on the data_post hook.  
It only works with TCP connections to clamd because apr doesn't support 
unix domain sockets.  Currently this module will bounce mail with 
virus, but there is a option to not reject them.  More features to come 
like adding headers, rewriting subject and anything else I can come up 
with.


mod_smtpd_spamd.tar.gz
	This is a module that sends the mail to spamd on the data_post hook.  
Currently this module will bounce mail that is spam if it over the spam 
threshold + reject_level. More features to come like adding headers, 
rewriting subject lines, etc.


mod_smtpd_load.tar.gz
	This module just check the load on the server and can temporarily deny 
connections.


Any news on the mod_ftp import?  I can't seem to get at the needed 
ssl_init_ssl_connection from mod_ssl.  I have even tried to dlsym it 
from the mod_ssl handle and I just get back NULL (of course this is on 
my powerbook which doesn't break on dlopen so dlsym might not work 
either).  There must be a different way to upgrade a socket to ssl.


Brian

Re: dbd connections tied to conn_rec

[Brian J. France]

Thanks for the work!

I am having problems with it on the second connection, first connection 
works with multiple calls to ap_dbd_cacquire.


I think it might have to do with allocating a dbd from the connection 
pool when it needs to be from the server pool for persist connections.  
non-persist connection can use the connection pool.


I don't have time to dig into right now, but will try this weekend.

Cheers,

Brian

On Sep 30, 2005, at 6:21 PM, Nick Kew wrote:

mod_dbd currently supports modules requiring a connection of an 
arbitrary
duration (explicit ap_dbd_open and ap_dbd_close), or tied to the 
request_rec
(ap_dbd_acquire, which releases the connection back to the reslist on 
request

pool cleanup).

Brian France's dbd modules for smtpd want the option that's missing 
from
there: a dbd handle tied to the conn_rec.  Today I got around to 
hacking
that - patch attached.  Review invited - I'll commit to trunk if noone 
shouts.


--
Nick Kew
dbd-patch

mod_smtpd_auth and mod_smtpd_auth_dbd

[Brian J. France]

Here are two modules I worked on last week while on a trip and ready 
for some discussion.


mod_smtpd_auth
mod_smtpd_auth_dbd

http://www.brianfrance.com/software/apache/

mod_smtpd.patch will be needed which adds the auth hooks and info into 
the smtp rec.  This patch also does a dns look up of the remote_addr 
because some auth methods need it and I figured it would be a good idea 
to do it anyways.


mod_smtpd_auth currently has support for PLAIN, LOGIN, CRAM-MD5 and 
DIGEST-MD5, but after finishing it I think whole auth stuff needs a 
little rethinking/reworking.


The way I did the mod_smtpd_auth module was to add the command AUTH to 
mod_smtpd and then mod_smtpd_auth hooks that and has a big if/else if 
block for all the auth methods.  I think a better way to do it is to 
have the auth stuff in the mod_smtpd server define two hooks, one for 
methods and one for validating/getting passwords.  Then in 
smtp_protocol the command auth hook will just get the function to call 
from the method hook list (just like command hooks are done) and run 
that function.  The big if/else if block that is currently in 
mod_smtpd_auth will be turned into multiple modules that will use the 
passwords hooks get get validate/get passwords.  This means 
mod_smtpd_auth will then go away and be replaced with two types of 
modules, method (mod_smtpd_authm_*) and validate/getting 
(mod_smtpd_authv_*) modules:


mod_smtpd_authm_plain
mod_smtpd_authm_login
mod_smtpd_authm_cram_md5
mod_smtpd_authm_digest_md5

mod_smtpd_authv_pwd
mod_smtpd_authv_dbd
mod_smtpd_authv_dba

The other thing that will need to change would be the AUTH ehlo 
response.  It would need pull the method names from the method hook 
list, which I don't know if it is possible or not.


I think somebody with a little more hook experience could use the 
current code and make the changes pretty quick, but I don't know enough 
about the hook stuff to get it all setup with out getting a big 
headache.


There are a few issue that might be a problem with this setup.  If 
somebody wants to enable/disable auth methods based on server or 
virtual hosts could be a problem, but I think that can be handled by 
having the method modules delay hooking the method hook until the 
connect phases to make sure they are enabled in that server.  Not sure 
if this is a problem or not, but if you look at the smtpd_protocol auth 
function, it is pretty plain  compared to the other ones.  This goes 
back to my original request to be able to set the response code from 
the module (mod_smtpd_auth sets 6 different codes).  Not sure how I did 
this was the best way, but it got it working until we talk about this 
again.


On a side note, has there been some bugs fixed recently in apreq for 
header parsing?  I am trying to do final testing of mod_smtpd_clamd and 
mod_smtpd_spamd, but I am failing on a assert in the parser header 
function when I use real emails (vs real simple stuff).  Need to svn 
update and try it again when I get a chance, but figure I would ask.


Brian

Re: mod_smtpd_access_dbd code review, mod_smtpd_auth plan

[Brian J. France]

On Sep 14, 2005, at 12:27 PM, Nick Kew wrote:
But I see you're using ap_dbd_open and ap_dbd_close over the lifetime 
of
a connection.  This looks like a good reason to update mod_dbd to 
support
a connection-lifetime variant on ap_dbd_acquire (which has 
request-lifetime).


Ya, I wanted to use the acquire call, but couldn't because of the 
request_rec * requirement.  The connection-lifetime sounds good and 
would also be useful for the mod_smtpd_auth_dbd as well.



Prod me if I don't get around to it.

BTW, have you made any changes to the mod_authn_dbd.c in that package?


Doh!  That shouldn't have been included. :)  I was using it as an 
example on how to get the dbd, get a prepared statement from it, do a 
query and test for the results using the apr_dbd functions.


Brian

mod_dbd

[Brian J. France]

Any reason why this patch couldn't be applied to allow the option of 
building mod_dbd?


The title might need a little tweaking.

Thanks,

Brian
  - with no commit access

Index: modules/experimental/config.m4
===
--- modules/experimental/config.m4  (revision 279774)
+++ modules/experimental/config.m4  (working copy)
@@ -13,5 +13,6 @@
 APACHE_MODULE(case_filter, example uppercase conversion filter, , , no)
 APACHE_MODULE(case_filter_in, example uppercase conversion input 
filter, , , no)

 APACHE_MODULE(filter, smart filtering module, , , no)
+APACHE_MODULE(dbd, Apache DBD Framework, , , no)

 APACHE_MODPATH_FINISH

Re: mod_smtpd filter support

[Brian J. France]

I needed the following patch to get one of my modules to build:

--- /usr/local/asf/include/mod_smtpd.h.orig Mon Aug 29 16:03:40 2005
+++ /usr/local/asf/include/mod_smtpd.h  Mon Aug 29 16:03:55 2005
@@ -20,6 +20,7 @@
 #include apr_pools.h
 #include apr_hash.h
 #include apr_file_io.h
+#include util_filter.h
 #include httpd.h

 #ifdef __cplusplus


Otherwise I get an error like this:
In file included from mod_smtpd_load.c:23:
/usr/local/asf/include/mod_smtpd.h:110: error: parse error before 
ap_filter_t
/usr/local/asf/include/mod_smtpd.h:110: warning: no semicolon at end of 
struct or union


Brian

On Aug 29, 2005, at 1:17 PM, Rian A Hunter wrote:
I just checked in support for input filters and header parsing in 
mod_smtpd.
This currently means little since there is no documentation on how to 
use
mod_smtpd or many example plugins. In the next few days (once I 
receive power)
I will have a couple of example plugins checked in (regular expression 
vrfy,
postfix queuer) and some minor documentation (maybe a tutorial + 
tutorial

plugin).

mod_smtpd now also depends on libapreq2 for rfc822 header parsing 
(configure

does not yet check if this is installed, patches welcome!!).

This mostly means that mod_smtpd is very close to completion. I expect 
some
bug-fixes and I plan on adding a one-recipient/one-transaction feature 
and a
message body reading abstraction, but other than that it seems to be 
in its

final working state. Features include:

- Hooks on every important SMTP event, with pre-done logic to handle 
denies and

disconnections.
- DATA command input filter support per transaction.
- RFC822 Header parsing via libapreq2
- Exported IO functions.

Have Fun!
-rian

mod_smtpd module review

[Brian J. France]

This past week I have finished up a few modules and ready for review.

http://www.brianfrance.com/software/apache/mod_smtpd_load.tar.gz

mod_smtpd_load:
  This module allows rejecting connection (temporarily) based on server 
load
  It is not very cross platform (any os with getloadavg), but I am sure 
we

  can work on that.

I have finished mod_smtpd_access_dbi, but after talking with Paul on IRC
I need to convert it to use mod_dbd instead of mod_dbi_pool.  Working 
on that
now and will post another message when done.  I could build a tar ball 
of this
module if anybody is interested as the the flow will not change, just 
how the

db connection is handled.

mod_smtpd_access_dbi:
This module is similar to sendmails access file.  It allows
checking of the ip/hostname/from/to items via a database to see if
things should be rejected.  It uses mod_dbi_pool and libdbi.
Thanks to Paul for mod_dbi_pool and code examples from mod_vhost_dbi.

A few things I have found while writing these modules. How about 
changing smtpd_return_data from this:


typedef struct smtpd_return_data {
apr_pool_t *p;
/* list of messages */
apr_array_header_t *msgs;
} smtpd_return_data;

to something like this:

typedef struct smtpd_return_data {
apr_pool_t *p;
int code;
/* list of messages */
apr_array_header_t *user_msgs;
apr_array_header_t *log_msgs;
} smtpd_return_data;

While doing the mod_smtpd_load module I found when I want to deny a 
connection I can set what message the user will get, but I also want to 
log a different message instead of the default Connection Denied 
(current I log my own and the default gets logged).  Of course this 
might be another thread of how and what do we plan on logging.


I would also like to set the error code, because looking over rfc0821 I 
think it should return 452 or may be that needs to be a default for 
smtpd_run_connect soft errors (552 for hard errors).  Should we allow 
the module to set the error code?


In mod_smtpd_access_dbi I found it strange that I get a string that 
looks like this:  [EMAIL PROTECTED] instead of [EMAIL PROTECTED].  
For the mail/rcpt hooks should we send a struct that has the full line 
sent, the data from the full line and the parsed email address?  I have 
some code that duplicates the string and then remove spaces and  from 
the beginning and  from the end, but that seems like it should be done 
before my function is called.  Another problem I can see is when we get 
into things like the size options:


MAIL FROM:[EMAIL PROTECTED] SIZE=50

Do we want every module to have to parse the email address (removing  
) and the SIZE?


Brian

Re: mod_smtpd overhaul

[Brian J. France]

On Aug 23, 2005, at 12:18 AM, Jem Berkes wrote:

I noticed a couple posts about examples, there is now one as I have
committed all the RBL stuff I wrote. See:

https://svn.apache.org/repos/asf/httpd/mod_smtpd/trunk/mod_smtpd_rbl/

This hooks into mod_smtpd in two places and returns various data (e.g. 
if
the client IP is blacklisted then mod_smtpd is told to deny mail). I 
hope

it serves as a good example, it seems to work quite nicely to give
mod_smtpd all the DNSBL/RHSBL features in a modular fashion.


Now that we have the means to start writing modules for smtpd should we 
create a modules directory?  Something like:


/repos/asf/httpd/mod_smtpd/trunk/modules/access
mod_smtpd_rbl : RBL (DNSBL/RHSBL) support to mod_smtpd
 mod_smtp_access_db : Add sendmail access.db type support
 mod_smtp_access_dbi : Add sendmail access.db type support via SQL

/repos/asf/httpd/mod_smtpd/trunk/modules/auth
mod_smtpd_auth_plain
mod_smtpd_auth_login
mod_smtpd_auth_cram_md5
mod_smtpd_auth_digest_md5

or should this be a skeleton auth modules that needs plugins:

mod_smtpd_auth : supports plain, login, cram_md5, digest_md5
mod_smtpd_auth_pwd : add /etc/passwd auth support for plain and login
mod_smtpd_auth_db : add db auth support for all auth
mod_smtpd_auth_dbi : add SQL auth support for all auth

/repos/asf/httpd/mod_smtpd/trunk/modules/misc
mod_smtpd_load : Allow temp error if load to high on the server

How hard is adding STARTTLS support going to be?  Is this something 
that is going to require more internal changes?


Is there a way to add more options to the EHLO response (an example)?

Brian

Re: mod_smtpd overhaul

[Brian J. France]

Trying to build build a smtpd module, but having a problem.  I added a 
line like this:


  smtpd_hook_connect( smtpd_access_dbi_connect, NULL, NULL, 
APR_HOOK_MIDDLE );


in the register_hooks function, but when I load the module I get 
undefined function _smtpd_hook_connect.


Is this the right way or is there an example module I could compare 
with?


Brian

On Aug 22, 2005, at 12:59 PM, Rian Hunter wrote:

I just have committed mod_smtpd with changes that free it from
request_rec. There are now two major structures in mod_smtpd:
smtpd_conn_rec, and smtpd_trans_rec. These are distinguished by how
often these get cleared.

Also mod_smtpd no longer uses the ap_getline and ap_rprintf functions
for I/O. There are three new functions for I/O called smtpd_getline,
smtpd_respond_oneline and smtpd_respond_multiline. This should allow 
the

use of filters in mod_smtpd although I may have implemented this wrong.
Comments are requested.

Joe:
For the data command I still call ap_get_brigade with AP_MODE_GETLINE,
just because SMTP is a line oriented protocol was there any explicit
reason you wanted it to be called with AP_MODE_READBYTES?

You can check it out from
https://svn.apache.org/repos/asf/httpd/mod_smtpd/trunk/.
-rian