RE: [PATCH] mod_log_forensic security considerations
-Original Message- From: Daniel Ruggeri Sent: Freitag, 8. Juni 2012 00:16 To: dev@httpd.apache.org Subject: Re: [PATCH] mod_log_forensic security considerations On 6/7/2012 3:11 PM, Stefan Fritsch wrote: On Thursday 07 June 2012, Eric Covener wrote: On Wed, Jun 6, 2012 at 9:15 PM, Jeff Trawick wrote: On Wed, Jun 6, 2012 at 3:49 PM, Joe Schaefer joe_schae...@yahoo.com wrote: Session cookies sometimes pose a security risk as well. Yeah. That could be any cookie though although there are a few very common defaults :( My guess is that cookie values are more useful for debugging crashes than Authorization headers, but that it should still be opt-in. Thoughts, anyone? +1 to separate knob to opt-in to Cookie logging. I share Williams concern that this makes mod_forensic potentially less useful. Maybe making the forensic log mode 600 by default would be a better idea? Agreed as well. This module isn't enabled by default and is most likely to be enabled by a user that knows what they are trying to accomplish. To me, a clear and concise security warning in the documentation should be all that is needed. IMO, having unadulterated logging capability is what makes mod_dumpio/mod_log_forensic some of the most useful modules for troubleshooting in a proxy/crashing scenario (respectively). +1 Regards Rüdiger
Re: [PATCH] mod_log_forensic security considerations
On 06/08/2012 12:13 PM, Graham Leggett wrote: On 08 Jun 2012, at 12:16 AM, Daniel Ruggeri wrote: I share Williams concern that this makes mod_forensic potentially less useful. Maybe making the forensic log mode 600 by default would be a better idea? Agreed as well. This module isn't enabled by default and is most likely to be enabled by a user that knows what they are trying to accomplish. To me, a clear and concise security warning in the documentation should be all that is needed. IMO, having unadulterated logging capability is what makes mod_dumpio/mod_log_forensic some of the most useful modules for troubleshooting in a proxy/crashing scenario (respectively). +1. Regards, Graham -- +1 to that. We already have the same kind of warnings in place for people setting up proxies, I see no reason why we can't do the same to mod_log_forensic. The module is, as the name says, for forensic logging, so it should be expected that as much as possible is logged by default, and any special considerations should be something you could change, but it shouldn't be the default behaviour to not include this and that because it may be potentially unsafe. We got bit by it, yes, but that was because we made the logs available to people, and that's what we should warn about if anything. With regards, Daniel.
Re: [PATCH] mod_log_forensic security considerations
- Original Message - From: Daniel Gruno rum...@cord.dk To: dev@httpd.apache.org Cc: Sent: Friday, June 8, 2012 6:24 AM Subject: Re: [PATCH] mod_log_forensic security considerations On 06/08/2012 12:13 PM, Graham Leggett wrote: On 08 Jun 2012, at 12:16 AM, Daniel Ruggeri wrote: I share Williams concern that this makes mod_forensic potentially less useful. Maybe making the forensic log mode 600 by default would be a better idea? Agreed as well. This module isn't enabled by default and is most likely to be enabled by a user that knows what they are trying to accomplish. To me, a clear and concise security warning in the documentation should be all that is needed. IMO, having unadulterated logging capability is what makes mod_dumpio/mod_log_forensic some of the most useful modules for troubleshooting in a proxy/crashing scenario (respectively). +1. Regards, Graham -- +1 to that. We already have the same kind of warnings in place for people setting up proxies, I see no reason why we can't do the same to mod_log_forensic. The module is, as the name says, for forensic logging, so it should be expected that as much as possible is logged by default, and any special considerations should be something you could change, but it shouldn't be the default behaviour to not include this and that because it may be potentially unsafe. We got bit by it, yes, but that was because we made the logs available to people, and that's what we should warn about if anything. Well not quite, we'd still have had a problem with storing and archiving those logs even if we hadn't made them available to committers, because they violate our password retention policies. With regards, Daniel.
Re: [PATCH] mod_log_forensic security considerations
On 06/08/2012 05:45 PM, Joe Schaefer wrote: Well not quite, we'd still have had a problem with storing and archiving those logs even if we hadn't made them available to committers, because they violate our password retention policies. My point was, that it should fall upon us to add a filter if we want to archive our logs with certain forensic details omitted, and not be a default assumption that people want forensic logs but not this, this and that stored. Thus, I'd be more in favor of either piping it through a filter or adding something like ForensicLogFilter option [,option...] With regards, Daniel.
Re: [PATCH] mod_log_forensic security considerations
On 08 Jun 2012, at 5:45 PM, Joe Schaefer wrote: Well not quite, we'd still have had a problem with storing and archiving those logs even if we hadn't made them available to committers, because they violate our password retention policies. Can you clarify if possible what purpose you were trying to solve by enabling the forensic logs? Forensic logging is to answer the question what is going wrong, and shouldn't be enabled under normal operational circumstances unless there is something genuinely going wrong, at which point you record what you need and then switch it off again. A forensic log that has had a whole lot of filters applied to it is counterproductive, because the forensic log no longer tells you exactly what is going on, and when you're troubleshooting you need to know precisely that. Regards, Graham --
Re: [PATCH] mod_log_forensic security considerations
For several years Graham those logs were rather valuable in tracking down segfaulting svn requests. Security releases were made as a result of some of those reports to the Subversion project. - Original Message - From: Graham Leggett minf...@sharp.fm To: dev@httpd.apache.org Cc: Sent: Friday, June 8, 2012 12:51 PM Subject: Re: [PATCH] mod_log_forensic security considerations On 08 Jun 2012, at 5:45 PM, Joe Schaefer wrote: Well not quite, we'd still have had a problem with storing and archiving those logs even if we hadn't made them available to committers, because they violate our password retention policies. Can you clarify if possible what purpose you were trying to solve by enabling the forensic logs? Forensic logging is to answer the question what is going wrong, and shouldn't be enabled under normal operational circumstances unless there is something genuinely going wrong, at which point you record what you need and then switch it off again. A forensic log that has had a whole lot of filters applied to it is counterproductive, because the forensic log no longer tells you exactly what is going on, and when you're troubleshooting you need to know precisely that. Regards, Graham --
Re: [PATCH] mod_log_forensic security considerations
On Jun 8, 2012, at 11:51 AM, Graham Leggett wrote: On 08 Jun 2012, at 5:45 PM, Joe Schaefer wrote: Well not quite, we'd still have had a problem with storing and archiving those logs even if we hadn't made them available to committers, because they violate our password retention policies. Can you clarify if possible what purpose you were trying to solve by enabling the forensic logs? Forensic logging is to answer the question what is going wrong, and shouldn't be enabled under normal operational circumstances unless there is something genuinely going wrong, at which point you record what you need and then switch it off again. A forensic log that has had a whole lot of filters applied to it is counterproductive, because the forensic log no longer tells you exactly what is going on, and when you're troubleshooting you need to know precisely that. In my situation, we have them enabled so that when an issue arises, we have one more tool at our disposal to identify a root cause. When I get an alert that there is something wrong with one of our sites, it is usually too late to enable forensic logging at that point. Something has already happened. We need to mitigate and get everything back up to normal. The question is usually not what IS going wrong?, but rather what WENT wrong?, because it is often a short-lived event. Having the forensic logs available has proven extremely helpful in this scenario. Might the full, unfiltered forensic data be valuable? Yes, but I don't believe the security risk is worth it in my situation. The rare case where an Authorization header might be truly useful for debugging or RCA is vastly overshadowed by the usefulness of the rest of the request information stored in the forensic log. The key to the forensic log, obviously, is that we have some information about an incoming request before it is completed. We can't get this information from any of the standard or custom logs, and we don't have any control over the format. Perhaps, just like we have LogFormat and now ErrorLogFormat, we should have ForensicLogFormat? If we did, then everyone could have what they want/need, whether full or partial forensic data. - Jim
Re: [PATCH] mod_log_forensic security considerations
On 6/8/2012 12:52 PM, Jim Riggs wrote: Having the forensic logs available has proven extremely helpful in this scenario. Might the full, unfiltered forensic data be valuable? Yes, but I don't believe the security risk is worth it in my situation. The rare case where an Authorization header might be truly useful for debugging or RCA is vastly overshadowed by the usefulness of the rest of the request information stored in the forensic log. I'd think this use case represents the minority - seems to me that since the module already supports writing to a pipe, a simple run through sed or perl -p -e should be enough for those who would like to run this in production all the time. If a code change is really what the community thinks is needed, it should become an optional parameter disabled by default. Should we run down that path, it becomes an exercise in figuring out how we give the administrator the option of disabling certain headers from being printed with the flexibility for that administrator to define a match that can suit every need (headers of various names and cookies of various names being the more recently discussed items). -- Daniel Ruggeri
Re: [PATCH] mod_log_forensic security considerations
On 08 Jun 2012, at 7:22 PM, Joe Schaefer wrote: For several years Graham those logs were rather valuable in tracking down segfaulting svn requests. Security releases were made as a result of some of those reports to the Subversion project. I'm sure they were, that's exactly what the forensic logs were designed for. I think what went wrong was that they weren't turned off when they were no longer needed, and they weren't properly secured. I think it would be useful to have a generic log encryption hook or provider, which any/all logging modules can call to encrypt data that is about to be written to a file and/or pipe. This way it can be safe to keep logging turned on and available to those who need it, while being suitably protected at the same time. Something like this maybe: # apply key in /path/to/key to files matching /var/log/httpd/secure/* LogEncrypt /path/to/key /var/log/httpd/secure/* Regards, Graham --
Re: [PATCH] mod_log_forensic security considerations
On 6/8/2012 10:55 AM, Daniel Gruno wrote: On 06/08/2012 05:45 PM, Joe Schaefer wrote: Well not quite, we'd still have had a problem with storing and archiving those logs even if we hadn't made them available to committers, because they violate our password retention policies. My point was, that it should fall upon us to add a filter if we want to archive our logs with certain forensic details omitted, and not be a default assumption that people want forensic logs but not this, this and that stored. Thus, I'd be more in favor of either piping it through a filter or adding something like ForensicLogFilter option [,option...] Should the support/check_forensic script (which fails to install into bin/, fwiw) be given the magic to purge all obvious credential by default, and only retain them in the result set if we --disclose-everything (or some similar semantic?) It might help reinforce how delicate this data is. OTOH, I have the same fundemental objection to this patch as I do to applying crazy logic to purge crash dumps of any potential p/w leakage. At any given time, the socket traffic and the httpd process itself are privy to all sorts of security details not available to mere mortals. Anyone with root can intercept this information. People need to grok the responsibilities of root, and grok the dangers of core files and forensic crash and traffic analysis. We can document the heck out of these features, and their dangerous side effects, but treating mod_log_forensic as 'just a logging tool' is foolish.
Re: [PATCH] mod_log_forensic security considerations
On Thu, Jun 7, 2012 at 2:18 PM, William A. Rowe Jr. wr...@rowe-clan.net wrote: On 6/6/2012 2:46 PM, Jeff Trawick wrote: On Tue, May 29, 2012 at 1:36 PM, Daniel Shahaf d...@daniel.shahaf.name wrote: Perhaps it would be a useful feature to allow excluding those headers from being logged, too. IMO they shouldn't be logged by default (if at all). Proxy-Authorization also needs to be handled. (Anything else? My search query foo is particularly bad today.) ANY parsing which occurs within mod_log_forensic is going to open that module itself to suspicion and potential un-captured exploitative header values. My own theory; provide pipe log redirection and write a filter to do whatever you like to corrupt the pure data received from the client. Otherwise, you have other issues like proxy connect scheme://user:pass@backend/ or session tokens in URL's or cookies to contend with. There is no way to make forensic logging 'safe for general consumption' and that is the message we have to broadcast loudly. The list of concerns can't be fixed with with code and we need to put more hints in the documentation. Logging obvious passwords by default is just plain stupid though. A forensic logging pipe could easily kill off all matched requests before they were ever logged to disk, resulting in only unmatched pairs. The parent process which spawned the logger shouldn't be crashing, so the logging should 'just work'. Writing only the unmatched requests is equivalent to what you get with mod_whatkilledus, though using a much different implementation. -- Born in Roswell... married an alien... http://emptyhammock.com/
Re: [PATCH] mod_log_forensic security considerations
On 6/7/2012 1:56 PM, Jeff Trawick wrote: On Thu, Jun 7, 2012 at 2:18 PM, William A. Rowe Jr. wr...@rowe-clan.net wrote: On 6/6/2012 2:46 PM, Jeff Trawick wrote: On Tue, May 29, 2012 at 1:36 PM, Daniel Shahaf d...@daniel.shahaf.name wrote: Perhaps it would be a useful feature to allow excluding those headers from being logged, too. IMO they shouldn't be logged by default (if at all). Proxy-Authorization also needs to be handled. (Anything else? My search query foo is particularly bad today.) ANY parsing which occurs within mod_log_forensic is going to open that module itself to suspicion and potential un-captured exploitative header values. My own theory; provide pipe log redirection and write a filter to do whatever you like to corrupt the pure data received from the client. Otherwise, you have other issues like proxy connect scheme://user:pass@backend/ or session tokens in URL's or cookies to contend with. There is no way to make forensic logging 'safe for general consumption' and that is the message we have to broadcast loudly. The list of concerns can't be fixed with with code and we need to put more hints in the documentation. Logging obvious passwords by default is just plain stupid though. And when that user and pass is fed into mod_authnz_ldap and causes the crash? There's nothing stupid about it, IMHO. A forensic logging pipe could easily kill off all matched requests before they were ever logged to disk, resulting in only unmatched pairs. The parent process which spawned the logger shouldn't be crashing, so the logging should 'just work'. Writing only the unmatched requests is equivalent to what you get with mod_whatkilledus, though using a much different implementation. The point to mod_log_forensic is for deep forensic studies of server corruption, at least that's the only point I've ever taken away from that module. It certainly -can- do other things, but mod_log_config is better suited for most of those applications.
Re: [PATCH] mod_log_forensic security considerations
On Thursday 07 June 2012, Eric Covener wrote: On Wed, Jun 6, 2012 at 9:15 PM, Jeff Trawick traw...@gmail.com wrote: On Wed, Jun 6, 2012 at 3:49 PM, Joe Schaefer joe_schae...@yahoo.com wrote: Session cookies sometimes pose a security risk as well. Yeah. That could be any cookie though although there are a few very common defaults :( My guess is that cookie values are more useful for debugging crashes than Authorization headers, but that it should still be opt-in. Thoughts, anyone? +1 to separate knob to opt-in to Cookie logging. I share Williams concern that this makes mod_forensic potentially less useful. Maybe making the forensic log mode 600 by default would be a better idea?
Re: [PATCH] mod_log_forensic security considerations
On Thu, Jun 7, 2012 at 4:11 PM, Stefan Fritsch s...@sfritsch.de wrote: On Thursday 07 June 2012, Eric Covener wrote: On Wed, Jun 6, 2012 at 9:15 PM, Jeff Trawick traw...@gmail.com wrote: On Wed, Jun 6, 2012 at 3:49 PM, Joe Schaefer joe_schae...@yahoo.com wrote: Session cookies sometimes pose a security risk as well. Yeah. That could be any cookie though although there are a few very common defaults :( My guess is that cookie values are more useful for debugging crashes than Authorization headers, but that it should still be opt-in. Thoughts, anyone? +1 to separate knob to opt-in to Cookie logging. I share Williams concern that this makes mod_forensic potentially less useful. Maybe making the forensic log mode 600 by default would be a better idea? A more appropriate mode is fine, but if a crash really occurs and the log file gets passed around/uploaded to vendor ftp servers/etc. for debugging the mode won't mean anything. -- Born in Roswell... married an alien... http://emptyhammock.com/
Re: [PATCH] mod_log_forensic security considerations
On Jun 7, 2012, at 3:11 PM, Stefan Fritsch wrote: I share Williams concern that this makes mod_forensic potentially less useful. Maybe making the forensic log mode 600 by default would be a better idea? I have to agree with Jeff. I would rather have a more difficult or even impossible time debugging a crash than have a security hole that relies solely on file permissions. Maybe it should be a toggle in mod_forensic for debugging purposes (defaulting to hiding Authorization). The problem with just changing the file permissions is that sensitive data is still stored in the files. Even if the files are owned by root, anyone with root access would have access to others' usernames and passwords. I don't want to have that access to others' credentials, nor do I want them to have access to mine. I applied Jeff's patch as soon as it came across, wiped out all of our archived forensic logs, and had all of our affected users reset their passwords. Thanks, Jeff! - Jim
Re: [PATCH] mod_log_forensic security considerations
On 6/7/2012 3:11 PM, Stefan Fritsch wrote: On Thursday 07 June 2012, Eric Covener wrote: On Wed, Jun 6, 2012 at 9:15 PM, Jeff Trawick traw...@gmail.com wrote: On Wed, Jun 6, 2012 at 3:49 PM, Joe Schaefer joe_schae...@yahoo.com wrote: Session cookies sometimes pose a security risk as well. Yeah. That could be any cookie though although there are a few very common defaults :( My guess is that cookie values are more useful for debugging crashes than Authorization headers, but that it should still be opt-in. Thoughts, anyone? +1 to separate knob to opt-in to Cookie logging. I share Williams concern that this makes mod_forensic potentially less useful. Maybe making the forensic log mode 600 by default would be a better idea? Agreed as well. This module isn't enabled by default and is most likely to be enabled by a user that knows what they are trying to accomplish. To me, a clear and concise security warning in the documentation should be all that is needed. IMO, having unadulterated logging capability is what makes mod_dumpio/mod_log_forensic some of the most useful modules for troubleshooting in a proxy/crashing scenario (respectively). -- Daniel Ruggeri
Re: [PATCH] mod_log_forensic security considerations
On Tue, May 29, 2012 at 1:36 PM, Daniel Shahaf d...@daniel.shahaf.name wrote:
https://blogs.apache.org/infra/entry/apache_org_incident_report_for
Infra got bit by mod_log_forensic logs including Authorization headers
and being world-readable, so in an effort to save someone else from
repeating this mistake how about adding it to the Security
considerations section of the documentation:
[[[
Index: docs/manual/mod/mod_log_forensic.xml
===
--- docs/manual/mod/mod_log_forensic.xml (revision 1342688)
+++ docs/manual/mod/mod_log_forensic.xml (working copy)
@@ -93,6 +93,10 @@
document for details on why your security could be compromised
if the directory where logfiles are stored is writable by
anyone other than the user that starts the server./p
+ pThe logfiles may contain sensitive data such as the contents of
+ codeAuthorization:/code headers (which can contain passwords), so
+ they should not be readable by anyone except the user that starts the
+ server./p
/section
directivesynopsis
]]]
Perhaps it would be a useful feature to allow excluding those headers
from being logged, too.
IMO they shouldn't be logged by default (if at all).
Proxy-Authorization also needs to be handled. (Anything else? My
search query foo is particularly bad today.)
Attached is a potential code fix... I guess a directive could be
added to allow them to be logged, but when would that be needed? (A.
When the request crashes due to the exact value of one of these
headers and the module author needs it for debugging.)
--
Born in Roswell... married an alien...
http://emptyhammock.com/
Index: modules/loggers/mod_log_forensic.c
===
--- modules/loggers/mod_log_forensic.c (revision 1347068)
+++ modules/loggers/mod_log_forensic.c (working copy)
@@ -156,10 +156,21 @@
return n;
}
+static const char *sanitize(const char *key, const char *value)
+{
+if (!strcasecmp(key, Authorization) || !strcasecmp(key,
Proxy-Authorization)) {
+value = X;
+}
+
+return value;
+}
+
static int count_headers(void *h_, const char *key, const char *value)
{
hlog *h = h_;
+value = sanitize(key, value);
+
h-count += count_string(key)+count_string(value)+2;
return 1;
@@ -169,6 +180,8 @@
{
hlog *h = h_;
+value = sanitize(key, value);
+
/* note that we don't have to check h-pos here, coz its been done
for us by log_escape */
*h-pos++ = '|';
Re: [PATCH] mod_log_forensic security considerations
Session cookies sometimes pose a security risk as well. - Original Message - From: Jeff Trawick traw...@gmail.com To: d...@httpd.apache.org; dev@httpd.apache.org Cc: Sent: Wednesday, June 6, 2012 3:46 PM Subject: Re: [PATCH] mod_log_forensic security considerations On Tue, May 29, 2012 at 1:36 PM, Daniel Shahaf d...@daniel.shahaf.name wrote: https://blogs.apache.org/infra/entry/apache_org_incident_report_for Infra got bit by mod_log_forensic logs including Authorization headers and being world-readable, so in an effort to save someone else from repeating this mistake how about adding it to the Security considerations section of the documentation: [[[ Index: docs/manual/mod/mod_log_forensic.xml === --- docs/manual/mod/mod_log_forensic.xml (revision 1342688) +++ docs/manual/mod/mod_log_forensic.xml (working copy) @@ -93,6 +93,10 @@ document for details on why your security could be compromised if the directory where logfiles are stored is writable by anyone other than the user that starts the server./p + pThe logfiles may contain sensitive data such as the contents of + codeAuthorization:/code headers (which can contain passwords), so + they should not be readable by anyone except the user that starts the + server./p /section directivesynopsis ]]] Perhaps it would be a useful feature to allow excluding those headers from being logged, too. IMO they shouldn't be logged by default (if at all). Proxy-Authorization also needs to be handled. (Anything else? My search query foo is particularly bad today.) Attached is a potential code fix... I guess a directive could be added to allow them to be logged, but when would that be needed? (A. When the request crashes due to the exact value of one of these headers and the module author needs it for debugging.) -- Born in Roswell... married an alien... http://emptyhammock.com/ - To unsubscribe, e-mail: docs-unsubscr...@httpd.apache.org For additional commands, e-mail: docs-h...@httpd.apache.org
Re: [PATCH] mod_log_forensic security considerations
On Wed, Jun 6, 2012 at 3:49 PM, Joe Schaefer joe_schae...@yahoo.com wrote: Session cookies sometimes pose a security risk as well. Yeah. That could be any cookie though although there are a few very common defaults :( My guess is that cookie values are more useful for debugging crashes than Authorization headers, but that it should still be opt-in. Thoughts, anyone? - Original Message - From: Jeff Trawick traw...@gmail.com To: d...@httpd.apache.org; dev@httpd.apache.org Cc: Sent: Wednesday, June 6, 2012 3:46 PM Subject: Re: [PATCH] mod_log_forensic security considerations On Tue, May 29, 2012 at 1:36 PM, Daniel Shahaf d...@daniel.shahaf.name wrote: https://blogs.apache.org/infra/entry/apache_org_incident_report_for Infra got bit by mod_log_forensic logs including Authorization headers and being world-readable, so in an effort to save someone else from repeating this mistake how about adding it to the Security considerations section of the documentation: [[[ Index: docs/manual/mod/mod_log_forensic.xml === --- docs/manual/mod/mod_log_forensic.xml (revision 1342688) +++ docs/manual/mod/mod_log_forensic.xml (working copy) @@ -93,6 +93,10 @@ document for details on why your security could be compromised if the directory where logfiles are stored is writable by anyone other than the user that starts the server./p + pThe logfiles may contain sensitive data such as the contents of + codeAuthorization:/code headers (which can contain passwords), so + they should not be readable by anyone except the user that starts the + server./p /section directivesynopsis ]]] Perhaps it would be a useful feature to allow excluding those headers from being logged, too. IMO they shouldn't be logged by default (if at all). Proxy-Authorization also needs to be handled. (Anything else? My search query foo is particularly bad today.) Attached is a potential code fix... I guess a directive could be added to allow them to be logged, but when would that be needed? (A. When the request crashes due to the exact value of one of these headers and the module author needs it for debugging.) -- Born in Roswell... married an alien... http://emptyhammock.com/ - To unsubscribe, e-mail: docs-unsubscr...@httpd.apache.org For additional commands, e-mail: docs-h...@httpd.apache.org - To unsubscribe, e-mail: docs-unsubscr...@httpd.apache.org For additional commands, e-mail: docs-h...@httpd.apache.org -- Born in Roswell... married an alien... http://emptyhammock.com/
Re: [PATCH] mod_log_forensic security considerations
On Wed, Jun 6, 2012 at 9:15 PM, Jeff Trawick traw...@gmail.com wrote: On Wed, Jun 6, 2012 at 3:49 PM, Joe Schaefer joe_schae...@yahoo.com wrote: Session cookies sometimes pose a security risk as well. Yeah. That could be any cookie though although there are a few very common defaults :( My guess is that cookie values are more useful for debugging crashes than Authorization headers, but that it should still be opt-in. Thoughts, anyone? +1 to separate knob to opt-in to Cookie logging.
