Re: [openstack-dev] [ironic][security] what is OK to put in DEBUG logs?

[Dan Prince]

On Mon, 2015-11-23 at 11:43 -0500, Ruby Loo wrote:
> On 20 November 2015 at 18:32, Ben Nemec 
> wrote:
> > On 11/19/2015 06:00 AM, Lucas Alvares Gomes wrote:
> > > Hi,
> > >
> > >> Also keep in mind that DEBUG logging, while still should have
> > some masking
> > >> of data, since it is explicitly called out (or should be) as not
> > safe for
> > >> production, can contain some " sensitive" data. Credentials
> > should still be
> > >> scrubbed, but I would say the swift temp URL is something that
> > may line up
> > >> with this more flexible level of filtering logs.
> > >>
> > >> Now, if the service (and I don't think ironic suffers from this
> > issue) is
> > >> only really runnable with debug on (because there is no useful
> > information
> > >> otherwise) then I would aim to fix that before putting even
> > potentially
> > >> sensitive data in DEBUG.
> > >>
> > >> The simple choice is if there is even a question, don't log it
> > (or log it in
> > >> a way that obscures the data but still shows unique use).
> > >>
> > >
> > > I agree with Morgan's statement here.
> > >
> > > And just throwing an idea in the wind here, we could make use of
> > the
> > > python logging filters to create a filter for sensitive
> > information.
> > > We probably need one already to avoid having to do things like
> > [1] in
> > > the code.
> > 
> > We actually have a thing to do that:
> > https://github.com/openstack/oslo.utils/blob/master/oslo_utils/stru
> > tils.py#L215
> > 
> > You might need to add a new key to the list of things to mask, but
> > I
> > think it should be able to handle masking the log message for you. 
> > I
> > don't know whether configdrive is a globally sensitive key, but if
> > not
> > then we probably need to revisit the question of whether to allow
> > extending the key list dynamically in the consuming application
> > instead
> > of having only the one hard-coded list.  More context here:
> > https://bugs.launchpad.net/oslo.utils/+bug/1407811
> > 
> > 
> The problem is that the developer WANTS to see the temporary URL, so
> masking it defeats the whole purpose.


Exactly. The sad thing here is that anyone who tries to use Ironic IPA
w/ Temp URLs is could to get their configuration wrong initially... and
they may very well end up adding the same log statement for the temp
URL (I was just trying to be generally helpful to the next person that
tries this). In my case... the actual URL was the only thing that if
logged would be helpful. Next best thing is just read the python code
yourself I guess...

> 
> The only thing I was thinking of (besides the developer modifying
> their code to spit it out) is to have some additional level of
> debugging that doesn't filter anything sensitive and/or add a config
> in ironic to log 'only-in-non-production-environments-shows-
> sensitive-info' with some big warning...

I thought this was what DEBUG was for :). It isn't like the upstream
projects enable by default or anything.

I'm of the belief that you shouldn't have to enable DEBUG in production
and I feel like OpenStack "installers" (my own company included) get it
wrong when they started trending towards defaulting their installations
to enable DEBUG by default.

So yeah. If there is no channel safe for "developer only logs" (DEBUG)
I guess we might should bail on this one.

> 
> My take on this (thanks everyone) is that it is a potential security
> risk to expose the URL in the logging, so we shouldn't allow it
> without some other way (besides expecting/documenting INFO level or
> higher) to prevent it from showing up in production.
> 
> --ruby
> _
> _
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubs
> cribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][security] what is OK to put in DEBUG logs?

[Ruby Loo]

On 20 November 2015 at 18:32, Ben Nemec  wrote:

> On 11/19/2015 06:00 AM, Lucas Alvares Gomes wrote:
> > Hi,
> >
> >> Also keep in mind that DEBUG logging, while still should have some
> masking
> >> of data, since it is explicitly called out (or should be) as not safe
> for
> >> production, can contain some " sensitive" data. Credentials should
> still be
> >> scrubbed, but I would say the swift temp URL is something that may line
> up
> >> with this more flexible level of filtering logs.
> >>
> >> Now, if the service (and I don't think ironic suffers from this issue)
> is
> >> only really runnable with debug on (because there is no useful
> information
> >> otherwise) then I would aim to fix that before putting even potentially
> >> sensitive data in DEBUG.
> >>
> >> The simple choice is if there is even a question, don't log it (or log
> it in
> >> a way that obscures the data but still shows unique use).
> >>
> >
> > I agree with Morgan's statement here.
> >
> > And just throwing an idea in the wind here, we could make use of the
> > python logging filters to create a filter for sensitive information.
> > We probably need one already to avoid having to do things like [1] in
> > the code.
>
> We actually have a thing to do that:
>
> https://github.com/openstack/oslo.utils/blob/master/oslo_utils/strutils.py#L215
>
> You might need to add a new key to the list of things to mask, but I
> think it should be able to handle masking the log message for you.  I
> don't know whether configdrive is a globally sensitive key, but if not
> then we probably need to revisit the question of whether to allow
> extending the key list dynamically in the consuming application instead
> of having only the one hard-coded list.  More context here:
> https://bugs.launchpad.net/oslo.utils/+bug/1407811
>
>
The problem is that the developer WANTS to see the temporary URL, so
masking it defeats the whole purpose.

The only thing I was thinking of (besides the developer modifying their
code to spit it out) is to have some additional level of debugging that
doesn't filter anything sensitive and/or add a config in ironic to log
'only-in-non-production-environments-shows-sensitive-info' with some big
warning...

My take on this (thanks everyone) is that it is a potential security risk
to expose the URL in the logging, so we shouldn't allow it without some
other way (besides expecting/documenting INFO level or higher) to prevent
it from showing up in production.

--ruby
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][security] what is OK to put in DEBUG logs?

[Ben Nemec]

On 11/19/2015 06:00 AM, Lucas Alvares Gomes wrote:
> Hi,
> 
>> Also keep in mind that DEBUG logging, while still should have some masking
>> of data, since it is explicitly called out (or should be) as not safe for
>> production, can contain some " sensitive" data. Credentials should still be
>> scrubbed, but I would say the swift temp URL is something that may line up
>> with this more flexible level of filtering logs.
>>
>> Now, if the service (and I don't think ironic suffers from this issue) is
>> only really runnable with debug on (because there is no useful information
>> otherwise) then I would aim to fix that before putting even potentially
>> sensitive data in DEBUG.
>>
>> The simple choice is if there is even a question, don't log it (or log it in
>> a way that obscures the data but still shows unique use).
>>
> 
> I agree with Morgan's statement here.
> 
> And just throwing an idea in the wind here, we could make use of the
> python logging filters to create a filter for sensitive information.
> We probably need one already to avoid having to do things like [1] in
> the code.

We actually have a thing to do that:
https://github.com/openstack/oslo.utils/blob/master/oslo_utils/strutils.py#L215

You might need to add a new key to the list of things to mask, but I
think it should be able to handle masking the log message for you.  I
don't know whether configdrive is a globally sensitive key, but if not
then we probably need to revisit the question of whether to allow
extending the key list dynamically in the consuming application instead
of having only the one hard-coded list.  More context here:
https://bugs.launchpad.net/oslo.utils/+bug/1407811

> 
> [1] 
> https://github.com/openstack/ironic/blob/812ed66ccabfcb1c1862951ea95a68b9d93b1672/ironic/drivers/modules/iscsi_deploy.py#L275-L284
> 
> Cheers,
> Lucas
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][security] what is OK to put in DEBUG logs?

[Lucas Alvares Gomes]

Hi,

> Also keep in mind that DEBUG logging, while still should have some masking
> of data, since it is explicitly called out (or should be) as not safe for
> production, can contain some " sensitive" data. Credentials should still be
> scrubbed, but I would say the swift temp URL is something that may line up
> with this more flexible level of filtering logs.
>
> Now, if the service (and I don't think ironic suffers from this issue) is
> only really runnable with debug on (because there is no useful information
> otherwise) then I would aim to fix that before putting even potentially
> sensitive data in DEBUG.
>
> The simple choice is if there is even a question, don't log it (or log it in
> a way that obscures the data but still shows unique use).
>

I agree with Morgan's statement here.

And just throwing an idea in the wind here, we could make use of the
python logging filters to create a filter for sensitive information.
We probably need one already to avoid having to do things like [1] in
the code.

[1] 
https://github.com/openstack/ironic/blob/812ed66ccabfcb1c1862951ea95a68b9d93b1672/ironic/drivers/modules/iscsi_deploy.py#L275-L284

Cheers,
Lucas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ironic][security] what is OK to put in DEBUG logs?

[Ruby Loo]

Hi,

I think we all agree that it isn't OK to log credentials (like passwords)
in DEBUG logs. However, what about other information that might be
sensitive? A patch was recently submitted to log (in debug) the SWIFT
temporary URL [1]. I agree that it would be useful for debugging, but since
that temporary URL could be used (by someone that has access to the logs
but no admin access to ironic/glance) eg for fetching private images, is it
OK?

Even though we say that debug shouldn't be used in production, we can't
enforce what folks choose to do. And we know of at least one company that
runs their production environment with the debug setting. Which isn't to
say we shouldn't put things in debug, but I think it would be useful to
have some guidelines as to what we can safely expose or not.

I took a quick look at the security web page [2] but nothing jumped out at
me wrt this issue.

Thoughts?

--ruby

[1] https://review.openstack.org/#/c/243141/
[2] https://security.openstack.org
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][security] what is OK to put in DEBUG logs?

[Devananda van der Veen]

On Wed, Nov 18, 2015 at 9:48 AM, Ruby Loo  wrote:

> Hi,
>
> I think we all agree that it isn't OK to log credentials (like passwords)
> in DEBUG logs. However, what about other information that might be
> sensitive? A patch was recently submitted to log (in debug) the SWIFT
> temporary URL [1]. I agree that it would be useful for debugging, but since
> that temporary URL could be used (by someone that has access to the logs
> but no admin access to ironic/glance) eg for fetching private images, is it
> OK?
>
> Even though we say that debug shouldn't be used in production, we can't
> enforce what folks choose to do. And we know of at least one company that
> runs their production environment with the debug setting. Which isn't to
> say we shouldn't put things in debug, but I think it would be useful to
> have some guidelines as to what we can safely expose or not.
>
> I took a quick look at the security web page [2] but nothing jumped out at
> me wrt this issue.
>
> Thoughts?
>
> --ruby
>
> [1] https://review.openstack.org/#/c/243141/
> [2] https://security.openstack.org
>
>
In this context, the URL is a time-limited access code being used in place
of a password or keystone auth token to allow an unprivileged client
temporary access to a specific privileged resource, without granting that
client access to any other resources. In some cases, that resource might be
a public Glance image and so one might say, "oh, it's not _that_
sensitive". However, the same module being affected by [1] is also used by
the iLO driver to upload a temporary image containing sensitive
instance-specific data.

I agree that it's not the same risk as exposing a password, but I still
consider this an access token, and therefore don't think it should be
written to log files, even at DEBUG.

-Deva
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][security] what is OK to put in DEBUG logs?

[Morgan Fainberg]

On Nov 18, 2015 13:52, "Devananda van der Veen" 
wrote:
>
>
> On Wed, Nov 18, 2015 at 9:48 AM, Ruby Loo  wrote:
>>
>> Hi,
>>
>> I think we all agree that it isn't OK to log credentials (like
passwords) in DEBUG logs. However, what about other information that might
be sensitive? A patch was recently submitted to log (in debug) the SWIFT
temporary URL [1]. I agree that it would be useful for debugging, but since
that temporary URL could be used (by someone that has access to the logs
but no admin access to ironic/glance) eg for fetching private images, is it
OK?
>>
>> Even though we say that debug shouldn't be used in production, we can't
enforce what folks choose to do. And we know of at least one company that
runs their production environment with the debug setting. Which isn't to
say we shouldn't put things in debug, but I think it would be useful to
have some guidelines as to what we can safely expose or not.
>>
>> I took a quick look at the security web page [2] but nothing jumped out
at me wrt this issue.
>>
>> Thoughts?
>>
>> --ruby
>>
>> [1] https://review.openstack.org/#/c/243141/
>> [2] https://security.openstack.org
>>
>
> In this context, the URL is a time-limited access code being used in
place of a password or keystone auth token to allow an unprivileged client
temporary access to a specific privileged resource, without granting that
client access to any other resources. In some cases, that resource might be
a public Glance image and so one might say, "oh, it's not _that_
sensitive". However, the same module being affected by [1] is also used by
the iLO driver to upload a temporary image containing sensitive
instance-specific data.
>
> I agree that it's not the same risk as exposing a password, but I still
consider this an access token, and therefore don't think it should be
written to log files, even at DEBUG.
>
> -Deva
>
>

Also keep in mind that DEBUG logging, while still should have some masking
of data, since it is explicitly called out (or should be) as not safe for
production, can contain some " sensitive" data. Credentials should still be
scrubbed, but I would say the swift temp URL is something that may line up
with this more flexible level of filtering logs.

Now, if the service (and I don't think ironic suffers from this issue) is
only really runnable with debug on (because there is no useful information
otherwise) then I would aim to fix that before putting even potentially
sensitive data in DEBUG.

The simple choice is if there is even a question, don't log it (or log it
in a way that obscures the data but still shows unique use).

Just $0.02 on this.
--Morgan
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev