mod_autoindex string pluggability
Hello Everyone,
I'm scratching an itch to make mod_autoindex output what I want, and
would love to know what, if anything would make the changes merge-able.
In its simplest form, I'd like apache to be able to give me an index in
JSON format - previously, I've parsed the html in javascript, but
somehow I think I can do better.
While I was reading the code (today) it occurred to me that there are a
lot of if statements throughout, which could be eliminated by moving
(obscuring) the output strings to a switchable container (right now I'm
using arrays of strings for my simplicity - I don't know this codebase
any better than you know me :)
so here is the kind of thing I started to do (partial diff, its all
really kind of dull - I've extracted the HTML/XHTML strings into another
similarly replaceable array):
#define STRING_INDEX_START 0
#define STRING_INDEX_END 1
const char *table_index_string[] = {
table\n tr,
/table\n
};
const char *table_index_string_stylesheet[] = {
table id=\indexlist\\n tr class=\indexhead\,
/table\n
};
const char *fancy_index_string[] = {
pre,
/pre\n
};
const char *default_index_string[] = {
ul,
/ul\n
};
/* set the default string set (choose alternatives based on user conf
options) */
const char **index_string = default_index_string;
@@ -1873,23 +1872,14 @@ static void output_directories(struct ent **ar,
int n,
}
}
if (autoindex_opts TABLE_INDEXING) {
-ap_rvputs(r, breakrow, /table\n, NULL);
+ap_rputs(breakrow, r);
}
else if (autoindex_opts FANCY_INDEXING) {
if (!(autoindex_opts SUPPRESS_RULES)) {
-ap_rputs(hr, r);
-if (autoindex_opts EMIT_XHTML) {
-ap_rputs( /, r);
-}
-ap_rputs(/pre\n, r);
-}
-else {
-ap_rputs(/pre\n, r);
+ap_rputs(output_string[STRING_HR], r);
}
}
-else {
-ap_rputs(/ul\n, r);
-}
+ap_rputs(index_string[STRING_INDEX_END], r);
}
Cheers
Sven
Re: mod_proxy, oooled backend connections and the keep-alive race condition
One could do an 'OPTIONS *' request. But I am not sure if that is any better than proxy-initial-not-pooled in terms of performance. I don't see why an OPTIONS request should not encounter problems where a GET request will. After all, the problem is on the transport layer, not on the application layer. Am I missing something ? Or a filter could just send the first bytes of the request (less than the request line) and then do a filter flush. If that fails, repeating the request on a different connection would be ok even for non- idempotent requests, because we would know that the backend has not received the full request yet. I don't know how many errors this would catch in practice, though. This is pretty much what I stated earlier. So I assume (re)opening a connection and having the whole process of sending the request transition to that (re)new(ed) connection is possible for an input filter. I was not sure about it. Going to look into it once I have time.. On Sat, Aug 3, 2013 at 7:26 PM, Stefan Fritsch s...@sfritsch.de wrote: Am Freitag, 2. August 2013, 11:21:56 schrieb Eric Covener: I think this does not work for GET requests or request without a request body. Just re-read spec, you are right -- we are abusing this in a module as a sort of extended handshake even w/ no body, but not against heterogenous backends. One could do an 'OPTIONS *' request. But I am not sure if that is any better than proxy-initial-not-pooled in terms of performance. Or a filter could just send the first bytes of the request (less than the request line) and then do a filter flush. If that fails, repeating the request on a different connection would be ok even for non- idempotent requests, because we would know that the backend has not received the full request yet. I don't know how many errors this would catch in practice, though.
Re: r1470679, async write completion, non blocking writes, and mod_ssl
On Mon, 5 Aug 2013, Graham Leggett wrote: Are you seeing a specific problem? Well, when I download a large file over a slow link, the request does not enter write completion state but rather the worker thread is still hogged for (nearly) the entire download. The way openssl's async behaviour works, is that is the middle of a read, openssl might need to write, and in the middle of a write, openssl might need to read. Openssl tells you this through the codes SSL_ERROR_WANT_READ and SSL_ERROR_WANT_WRITE. You receive these codes, re-run the select or poll or whatever with the sense that openssl has asked for, and you're done. It is that simple. AFAICT, that part is ok. But it doesn't solve the problem that one thread is hogged for every ssl connection. Sure, httpd might do all sorts of buffering of reads and writes, but at the end of the day it won't matter, because openssl will never attempt to write-during read or read-during-write without asking you whether it is ok first. I can see a potential problem if the core decided to buffer a write, but in theory that could be solved by mod_ssl simply sending a flush bucket down the stack whenever the sense changes. I don't see why the core needs to care that the data is a file, or encrypted, or whatever, the core should just do what the core does. Let me recap the way async works in mpm event: There is no way to interrupt and resume the handler, so the handler always must run to completion. The only way async write completion can work is if the data produced by the handler is buffered and is later sent bit by bit in an async way. However, this cannot be done unconditionally or the memory usage would grow to infinity. Therefore the core output filter has some heuristics on when to buffer data for async write completion and when to do blocking writes instead. In general, if there is more than 64k of data in memory, this is written to the network in a blocking way. Only if the buckets are backed by files (and therefore do not use significant memory before being sent to the network), more than 64k of request data is put into async write completion. This is where mod_ssl comes in. It will read the file buckets, encrypt them, and then we have encrypted data in memory that will cause the core output filters to do blocking writes. The blocking writes happen while the handler does ap_pass_brigade() and therefore before the worker thread which executes the handler is freed. An ideal solution would put the buffering/decision for blocking/non-blocking into ap_pass_brigade(). This way other filters like deflate could also be called asynchronously. But I am not too optimistic that this can be achieved without API changes.
Re: mod_proxy, oooled backend connections and the keep-alive race condition
On Mon, Aug 5, 2013 at 2:49 AM, Thomas Eckert thomas.r.w.eck...@gmail.com wrote: One could do an 'OPTIONS *' request. But I am not sure if that is any better than proxy-initial-not-pooled in terms of performance. I don't see why an OPTIONS request should not encounter problems where a GET request will. After all, the problem is on the transport layer, not on the application layer. Am I missing something ? Could have problems at either level (e.g. MaxKeepalives)
[PATCH] fix timeout in mod_ssl's trace2 log output
timeout is nowadays a duration, not an absolute value. Fix the trace2
output to print it as such, instead of subtracting time(NULL) and
resulting in lines such as:
[Mon Aug 05 03:51:07.369625 2013] [ssl:trace2] [pid 7916:tid 140003006699264]
ssl_engine_kernel.c(1698): Inter-Process Session Cache: request=SET status=OK
id=e53934c8b21017256ebe00327be265ccbb82bbaaef52adb900a884a07ad32eac
timeout=-1375674367s (session caching)
--- modules/ssl/ssl_engine_kernel.c (revision 1510425)
+++ modules/ssl/ssl_engine_kernel.c (working copy)
@@ -1687,7 +1687,7 @@
if (timeout) {
apr_snprintf(timeout_str, sizeof(timeout_str),
- timeout=%lds , (timeout - time(NULL)));
+ timeout=%lds , timeout);
}
ap_log_error(APLOG_MARK, APLOG_TRACE2, 0, s,
[PATCH] mod_socache_memcache: don't ignore expiry
The memcache socache backend currently completely ignores the expiry
value, presumably due to historical limitations of aprutil that don't
apply anymore.
The current behavior is to always send 0 as the expiry value, which in
the memcached protocol translates as never. This could have security
repercussions when memcache is used as a backing store for
SSLSessionCache, especially since SSLSessionCacheTimeout is ignored
silently. The session keys would presumably be expired by memcached as
the cache gets full but due to the LRU nature of memcached, an attacker
could request it often and thus keeping it hot in the cache and never
expired.
Fixing this is trivial by just propagating the expiry time to memcached.
From my limited testing (intercepting memcached writes over the wire
dumping memcached contents) the current time + SSLSessionCacheTimeout
seems to be correctly sent with this patch.
--- modules/cache/mod_socache_memcache.c(revision 1510425)
+++ modules/cache/mod_socache_memcache.c(working copy)
@@ -205,9 +205,10 @@
return APR_EINVAL;
}
-/* In APR-util - unclear what 'timeout' is, as it was not implemented */
-rv = apr_memcache_set(ctx-mc, buf, (char*)ucaData, nData, 0, 0);
+rv = apr_memcache_set(ctx-mc, buf, (char*)ucaData, nData,
+ apr_time_sec(expiry), 0);
+
if (rv != APR_SUCCESS) {
ap_log_error(APLOG_MARK, APLOG_CRIT, rv, s, APLOGNO(00790)
scache_mc: error setting key '%s'
Re: mod_autoindex string pluggability
How about implementing XHTML → JSON as a filter? Either with existing modules
or with something dedicated to autoindex.
TimOn 05/08/2013 7:26 Sven Dowideit wrote:
Hello Everyone,
I'm scratching an itch to make mod_autoindex output what I want, and
would love to know what, if anything would make the changes merge-able.
In its simplest form, I'd like apache to be able to give me an index in
JSON format - previously, I've parsed the html in javascript, but
somehow I think I can do better.
While I was reading the code (today) it occurred to me that there are a
lot of if statements throughout, which could be eliminated by moving
(obscuring) the output strings to a switchable container (right now I'm
using arrays of strings for my simplicity - I don't know this codebase
any better than you know me :)
so here is the kind of thing I started to do (partial diff, its all
really kind of dull - I've extracted the HTML/XHTML strings into another
similarly replaceable array):
#define STRING_INDEX_START 0
#define STRING_INDEX_END 1
const char *table_index_string[] = {
table\n tr,
/table\n
};
const char *table_index_string_stylesheet[] = {
table id=\indexlist\\n tr class=\indexhead\,
/table\n
};
const char *fancy_index_string[] = {
pre,
/pre\n
};
const char *default_index_string[] = {
ul,
/ul\n
};
/* set the default string set (choose alternatives based on user conf
options) */
const char **index_string = default_index_string;
@@ -1873,23 +1872,14 @@ static void output_directories(struct ent **ar,
int n,
}
}
if (autoindex_opts TABLE_INDEXING) {
-ap_rvputs(r, breakrow, /table\n, NULL);
+ap_rputs(breakrow, r);
}
else if (autoindex_opts FANCY_INDEXING) {
if (!(autoindex_opts SUPPRESS_RULES)) {
-ap_rputs(hr, r);
-if (autoindex_opts EMIT_XHTML) {
-ap_rputs( /, r);
-}
-ap_rputs(/pre\n, r);
-}
-else {
-ap_rputs(/pre\n, r);
+ap_rputs(output_string[STRING_HR], r);
}
}
-else {
-ap_rputs(/ul\n, r);
-}
+ap_rputs(index_string[STRING_INDEX_END], r);
}
Cheers
Sven
Re: [PATCH] fix timeout in mod_ssl's trace2 log output
On Mon, Aug 5, 2013 at 7:19 AM, Faidon Liambotis parav...@debian.orgwrote:
timeout is nowadays a duration, not an absolute value. Fix the trace2
output to print it as such, instead of subtracting time(NULL) and
resulting in lines such as:
[Mon Aug 05 03:51:07.369625 2013] [ssl:trace2] [pid 7916:tid
140003006699264] ssl_engine_kernel.c(1698): Inter-Process Session Cache:
request=SET status=OK
id=e53934c8b21017256ebe00327be265ccbb82bbaaef52adb900a884a07ad32eac
timeout=-1375674367s (session caching)
--- modules/ssl/ssl_engine_kernel.c (revision 1510425)
+++ modules/ssl/ssl_engine_kernel.c (working copy)
@@ -1687,7 +1687,7 @@
if (timeout) {
apr_snprintf(timeout_str, sizeof(timeout_str),
- timeout=%lds , (timeout - time(NULL)));
+ timeout=%lds , timeout);
}
ap_log_error(APLOG_MARK, APLOG_TRACE2, 0, s,
Someone else committed this to trunk yesterday (
http://svn.apache.org/viewvc?view=revisionrevision=r1510098), and it now
has 2 of 3 necessary votes to appear in the next 2.4.x release.
--
Born in Roswell... married an alien...
http://emptyhammock.com/
Re: r1470679, async write completion, non blocking writes, and mod_ssl
On Aug 5, 2013, at 4:00 AM, Stefan Fritsch s...@sfritsch.de wrote: An ideal solution would put the buffering/decision for blocking/non-blocking into ap_pass_brigade(). This way other filters like deflate could also be called asynchronously. But I am not too optimistic that this can be achieved without API changes. Agreed, but from what I can see the proposed patch does add some benefit, allows for future improvement, adds no overhead and no bugs. As such, even though it doesn't completely solve the whole issue, it is valuable enough to be folded into 2.4. (imo 'natch)
Re: mod_proxy, oooled backend connections and the keep-alive race condition
On 05.08.2013 13:18, Eric Covener wrote: On Mon, Aug 5, 2013 at 2:49 AM, Thomas Eckert thomas.r.w.eck...@gmail.com wrote: One could do an 'OPTIONS *' request. But I am not sure if that is any better than proxy-initial-not-pooled in terms of performance. I don't see why an OPTIONS request should not encounter problems where a GET request will. After all, the problem is on the transport layer, not on the application layer. Am I missing something ? Could have problems at either level (e.g. MaxKeepalives) I think what people are trying to say: another request in front of each request might increase the relative frequency (per real request) with which the problem occurs. Regards, Rainer
Re: mod_proxy, oooled backend connections and the keep-alive race condition
On Aug 5, 2013, at 10:13 AM, Rainer Jung rainer.j...@kippdata.de wrote: On 05.08.2013 13:18, Eric Covener wrote: On Mon, Aug 5, 2013 at 2:49 AM, Thomas Eckert thomas.r.w.eck...@gmail.com wrote: One could do an 'OPTIONS *' request. But I am not sure if that is any better than proxy-initial-not-pooled in terms of performance. I don't see why an OPTIONS request should not encounter problems where a GET request will. After all, the problem is on the transport layer, not on the application layer. Am I missing something ? Could have problems at either level (e.g. MaxKeepalives) I think what people are trying to say: another request in front of each request might increase the relative frequency (per real request) with which the problem occurs. +1
[PATCH 55360] Potential buffer overflows in support/ab
Hello all, A comment section in support/ab.c lists the following known problems: /* * BUGS: * * - uses strcpy/etc. * - has various other poor buffer attacks related to the lazy parsing of * response headers from the server * - doesn't implement much of HTTP/1.x, only accepts certain forms of * responses * - (performance problem) heavy use of strstr shows up top in profile * only an issue for loopback usage */ I was able to duplicate segmentation faults through the T and X command line options. I submitted a patch to fix potential buffer overflows through these options. - https://issues.apache.org/bugzilla/show_bug.cgi?id=55360 The patch also removes 2 unreferenced fixed length buffers. support/ab.c also contains 3 additional fixed length buffers that could potentially overflow: - servername, buffer and _request Fixing these problems will require a deeper understanding of the code. Please, consider the submitted patch for adoption. Thanks, Mike Rumph
[RFC] mod_authnz_fcgi
I have been working on an AA (just 2 for now ;) ) over FastCGI interface for httpd 2.4+ which uses the provider APIs. The source still needs some TLC (especially some parts which originated with mod_proxy_fcgi but still need drastic restructure to suit the more limited requirements of auth), but if anyone would like to look around in the doc or code, and/or is just bored out of their skull, you can find the the pieces at http://emptyhammock.com/hypermanual/mod/mod_authnz_fcgi.html http://emptyhammock.com/downloads/ As part of that I've created a handful of utility routines, such as the buffer logging ones I mentioned yesterday as well as some FastCGI protocol functions which are essentially the same as some private functions in mod_proxy_fcgi. The utility routines still need more work, after which I hope to include them, along with mod_authnz_fcgi, in trunk. -- Born in Roswell... married an alien... http://emptyhammock.com/
Re: [PATCH 55360] Potential buffer overflows in support/ab
On Mon, Aug 5, 2013 at 2:11 PM, Mike Rumph mike.ru...@oracle.com wrote: Hello all, A comment section in support/ab.c lists the following known problems: /* * BUGS: * * - uses strcpy/etc. * - has various other poor buffer attacks related to the lazy parsing of * response headers from the server * - doesn't implement much of HTTP/1.x, only accepts certain forms of * responses * - (performance problem) heavy use of strstr shows up top in profile * only an issue for loopback usage */ I was able to duplicate segmentation faults through the T and X command line options. I submitted a patch to fix potential buffer overflows through these options. - https://issues.apache.org/**bugzilla/show_bug.cgi?id=55360https://issues.apache.org/bugzilla/show_bug.cgi?id=55360 The patch also removes 2 unreferenced fixed length buffers. support/ab.c also contains 3 additional fixed length buffers that could potentially overflow: - servername, buffer and _request Fixing these problems will require a deeper understanding of the code. Please, consider the submitted patch for adoption. The patch looks fine in an initial glance. I anticipate committing it today after eyeballing it a bit more. (Or else I'll speak up.) Thanks, Jeff Thanks, Mike Rumph -- Born in Roswell... married an alien... http://emptyhammock.com/
Re: [PATCH 55360] Potential buffer overflows in support/ab
On Mon, Aug 5, 2013 at 4:10 PM, Jeff Trawick traw...@gmail.com wrote: On Mon, Aug 5, 2013 at 2:11 PM, Mike Rumph mike.ru...@oracle.com wrote: Hello all, A comment section in support/ab.c lists the following known problems: /* * BUGS: * * - uses strcpy/etc. * - has various other poor buffer attacks related to the lazy parsing of * response headers from the server * - doesn't implement much of HTTP/1.x, only accepts certain forms of * responses * - (performance problem) heavy use of strstr shows up top in profile * only an issue for loopback usage */ I was able to duplicate segmentation faults through the T and X command line options. I submitted a patch to fix potential buffer overflows through these options. - https://issues.apache.org/**bugzilla/show_bug.cgi?id=55360https://issues.apache.org/bugzilla/show_bug.cgi?id=55360 The patch also removes 2 unreferenced fixed length buffers. support/ab.c also contains 3 additional fixed length buffers that could potentially overflow: - servername, buffer and _request Fixing these problems will require a deeper understanding of the code. Please, consider the submitted patch for adoption. The patch looks fine in an initial glance. I anticipate committing it today after eyeballing it a bit more. (Or else I'll speak up.) This is now in trunk as r1510707; I'll nominate for inclusion in 2.4.next shortly. Thanks, Jeff Thanks, Mike Rumph -- Born in Roswell... married an alien... http://emptyhammock.com/ -- Born in Roswell... married an alien... http://emptyhammock.com/
Re: r1470679, async write completion, non blocking writes, and mod_ssl
Am Montag, 5. August 2013, 09:57:16 schrieb Jim Jagielski: On Aug 5, 2013, at 4:00 AM, Stefan Fritsch s...@sfritsch.de wrote: An ideal solution would put the buffering/decision for blocking/non-blocking into ap_pass_brigade(). This way other filters like deflate could also be called asynchronously. But I am not too optimistic that this can be achieved without API changes. Agreed, but from what I can see the proposed patch does add some benefit, allows for future improvement, adds no overhead and no bugs. As such, even though it doesn't completely solve the whole issue, it is valuable enough to be folded into 2.4. (imo 'natch) It gives us a new API that we need to keep. But if you and Graham both think that this is a good tradeoff, then that's fine with me. But I want to test one more thing before I vote +1. Hopefully I will have some time for that next week-end. The wording of CHANGES and the docs need to be adjusted, though.
Re: [RFC] http_log functions to log buffers
I have not studied logging in httpd. The only logs I have ever looked at are the error_logs and access_logs. These look like something different. For systems security I like to use syslog as a place to collect data. If apr already supports, please excuse my ignorance and ignore this. If not, please take my feedback to be: would be very nice to be able to (also) direct this to syslog mechanism. Michael On Mon, Aug 5, 2013 at 9:32 AM, Jeff Trawick traw...@gmail.com wrote: Any thoughts on the API below? For mod_ssl as an example, at least a couple of additions would be needed to replace ssl_io_data_dump(): 1. a processing flag that converted the printable form to EBCDIC in an EBCDIC environment 2. the ap_log_csdata() variation This doesn't currently implement the optimization to check the configured log level before calling the function. /** * Processing flags for ap_log_data() et al * * AP_LOG_DATA_DEFAULT - default formatting * AP_LOG_DATA_SHOW_OFFSET - prefix each line with hex offset from the start * of the buffer */ #define AP_LOG_DATA_DEFAULT 0 #define AP_LOG_DATA_SHOW_OFFSET 1 /** * ap_log_data() - log buffers which are not related to a particular request * or connection. * @param file The file in which this function is called * @param line The line number on which this function is called * @param module_index The module_index of the module logging this buffer * @param level The log level * @param s The server on which we are logging * @param label A label for the buffer, to be logged preceding the buffer * @param data The buffer to be logged * @param len The length of the buffer * @param flags Special processing flags like AP_LOG_DATA_SHOW_OFFSET * @note Use APLOG_MARK to fill out file, line, and module_index * @note If a request_rec is available, use that with ap_log_rerror() * in preference to calling this function. Otherwise, if a conn_rec is * available, use that with ap_log_cerror() in preference to calling * this function. */ AP_DECLARE(void) ap_log_data(const char *file, int line, int module_index, int level, const server_rec *s, const char *label, const char *data, apr_size_t len, unsigned int flags); /** * ap_log_rdata() - log buffers which are related to a particular request. * @param file The file in which this function is called * @param line The line number on which this function is called * @param module_index The module_index of the module logging this buffer * @param level The log level * @param r The request which we are logging for * @param label A label for the buffer, to be logged preceding the buffer * @param data The buffer to be logged * @param len The length of the buffer * @param flags Special processing flags like AP_LOG_DATA_SHOW_OFFSET * @note Use APLOG_MARK to fill out file, line, and module_index * @note If a request_rec is available, use that with ap_log_rerror() * in preference to calling this function. Otherwise, if a conn_rec is * available, use that with ap_log_cerror() in preference to calling * this function. */ AP_DECLARE(void) ap_log_rdata(const char *file, int line, int module_index, int level, const request_rec *r, const char *label, const char *data, apr_size_t len, unsigned int flags); /** * ap_log_cdata() - log buffers which are related to a particular connection. * @param file The file in which this function is called * @param line The line number on which this function is called * @param module_index The module_index of the module logging this buffer * @param level The log level * @param c The connection which we are logging for * @param label A label for the buffer, to be logged preceding the buffer * @param data The buffer to be logged * @param len The length of the buffer * @param flags Special processing flags like AP_LOG_DATA_SHOW_OFFSET * @note Use APLOG_MARK to fill out file, line, and module_index * @note If a request_rec is available, use that with ap_log_rerror() * in preference to calling this function. Otherwise, if a conn_rec is * available, use that with ap_log_cerror() in preference to calling * this function. */ AP_DECLARE(void) ap_log_cdata(const char *file, int line, int module_index, int level, const conn_rec *c, const char *label, const char *data, apr_size_t len, unsigned int flags); Sample output with AP_LOG_DATA_SHOW_OFFSET and non-default ErrorLogFormat: [authnz_fcgi:trace1] mod_authnz_fcgi.c(127): FastCGI data sent (8 bytes) [authnz_fcgi:trace1] mod_authnz_fcgi.c(127): : 0104000103a8 [authnz_fcgi:trace1] mod_authnz_fcgi.c(127): FastCGI data sent (936 bytes) [authnz_fcgi:trace1] mod_authnz_fcgi.c(127): : ..UNIQUE_IDUf76O 0918554e495155455f4944556637364f
