[jira] [Commented] (LIBCLOUD-835) Malformed auth token causes fatal exception in Google Storage driver

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LIBCLOUD-835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400637#comment-15400637
 ] 

ASF subversion and git services commented on LIBCLOUD-835:
--

Commit 78df34cf8db8706440ee594c571d80de8613433e in libcloud's branch 
refs/heads/trunk from [~paul.tiplady]
[ https://git-wip-us.apache.org/repos/asf?p=libcloud.git;h=78df34c ]

Fix caching of Google auth tokens

_write_token_to_file was not zeroing the file before writing
a new token, causing corruption.

FIXES: LIBCLOUD-835

Closes #844

Signed-off-by: Tomaz Muraus 


> Malformed auth token causes fatal exception in Google Storage driver
> 
>
> Key: LIBCLOUD-835
> URL: https://issues.apache.org/jira/browse/LIBCLOUD-835
> Project: Libcloud
>  Issue Type: Bug
>Reporter: Paul Tiplady
>Priority: Critical
>
> One of my Django instances has started hitting a libcloud error which is 
> causing a fatal exception, bringing down the instance.
> It looks like libcloud is writing invalid JSON into the auth token, which 
> then causes a JSON parse error when it is subsequently read back in.
> Here's the token that's written:
> {code}
> $ cat /root/.google_libcloud_auth.
> {"access_token": "", "token_type": "Bearer", "expire_time": 
> "2016-07-12T16:45:09Z", "expires_in": 3559}09Z", "expires_in": 3537}
> {code}
> Note the two "expires_in" keys, one with a nonsense value of `3559}09Z"`
> Environment:
> Python 3.4.4 
> apache-libcloud==1.0.0



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (LIBCLOUD-835) Malformed auth token causes fatal exception in Google Storage driver

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LIBCLOUD-835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400638#comment-15400638
 ] 

ASF GitHub Bot commented on LIBCLOUD-835:
-

Github user asfgit closed the pull request at:

https://github.com/apache/libcloud/pull/844


> Malformed auth token causes fatal exception in Google Storage driver
> 
>
> Key: LIBCLOUD-835
> URL: https://issues.apache.org/jira/browse/LIBCLOUD-835
> Project: Libcloud
>  Issue Type: Bug
>Reporter: Paul Tiplady
>Priority: Critical
>
> One of my Django instances has started hitting a libcloud error which is 
> causing a fatal exception, bringing down the instance.
> It looks like libcloud is writing invalid JSON into the auth token, which 
> then causes a JSON parse error when it is subsequently read back in.
> Here's the token that's written:
> {code}
> $ cat /root/.google_libcloud_auth.
> {"access_token": "", "token_type": "Bearer", "expire_time": 
> "2016-07-12T16:45:09Z", "expires_in": 3559}09Z", "expires_in": 3537}
> {code}
> Note the two "expires_in" keys, one with a nonsense value of `3559}09Z"`
> Environment:
> Python 3.4.4 
> apache-libcloud==1.0.0



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (LIBCLOUD-835) Malformed auth token causes fatal exception in Google Storage driver

[Eric Johnson (JIRA)]

[ 
https://issues.apache.org/jira/browse/LIBCLOUD-835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392352#comment-15392352
 ] 

Eric Johnson commented on LIBCLOUD-835:
---

Thanks Paul - Pointed Tom to this too.

> Malformed auth token causes fatal exception in Google Storage driver
> 
>
> Key: LIBCLOUD-835
> URL: https://issues.apache.org/jira/browse/LIBCLOUD-835
> Project: Libcloud
>  Issue Type: Bug
>Reporter: Paul Tiplady
>Priority: Critical
>
> One of my Django instances has started hitting a libcloud error which is 
> causing a fatal exception, bringing down the instance.
> It looks like libcloud is writing invalid JSON into the auth token, which 
> then causes a JSON parse error when it is subsequently read back in.
> Here's the token that's written:
> {code}
> $ cat /root/.google_libcloud_auth.
> {"access_token": "", "token_type": "Bearer", "expire_time": 
> "2016-07-12T16:45:09Z", "expires_in": 3559}09Z", "expires_in": 3537}
> {code}
> Note the two "expires_in" keys, one with a nonsense value of `3559}09Z"`
> Environment:
> Python 3.4.4 
> apache-libcloud==1.0.0



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (LIBCLOUD-835) Malformed auth token causes fatal exception in Google Storage driver

[Paul Tiplady (JIRA)]

[ 
https://issues.apache.org/jira/browse/LIBCLOUD-835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392345#comment-15392345
 ] 

Paul Tiplady commented on LIBCLOUD-835:
---

Fix is here: https://github.com/apache/libcloud/pull/844

I have verified that this resolves my issue. Had a quick stab at tests, but I 
don't have time to push this across the finish line today, so happy for someone 
else to take over if you're interested in getting this bugfix in sooner than I 
can.

> Malformed auth token causes fatal exception in Google Storage driver
> 
>
> Key: LIBCLOUD-835
> URL: https://issues.apache.org/jira/browse/LIBCLOUD-835
> Project: Libcloud
>  Issue Type: Bug
>Reporter: Paul Tiplady
>Priority: Critical
>
> One of my Django instances has started hitting a libcloud error which is 
> causing a fatal exception, bringing down the instance.
> It looks like libcloud is writing invalid JSON into the auth token, which 
> then causes a JSON parse error when it is subsequently read back in.
> Here's the token that's written:
> {code}
> $ cat /root/.google_libcloud_auth.
> {"access_token": "", "token_type": "Bearer", "expire_time": 
> "2016-07-12T16:45:09Z", "expires_in": 3559}09Z", "expires_in": 3537}
> {code}
> Note the two "expires_in" keys, one with a nonsense value of `3559}09Z"`
> Environment:
> Python 3.4.4 
> apache-libcloud==1.0.0



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (LIBCLOUD-835) Malformed auth token causes fatal exception in Google Storage driver

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LIBCLOUD-835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392342#comment-15392342
 ] 

ASF GitHub Bot commented on LIBCLOUD-835:
-

GitHub user paultiplady opened a pull request:

https://github.com/apache/libcloud/pull/844

[LIBCLOUD-835] Fix caching of Google auth tokens

## Fix corruption bug in Google auth token caching

### Description

The `GoogleOAuth2Credential. _write_token_to_file()` method writes a copy 
of the latest OAuth token to disk. Prior to this fix, the token was being 
written to disk without truncating the file first, which is fine in the case 
where the new token has the same number of characters (or more) as the old one. 
However, in some situations Google OAuth returns a shorter token string, which 
was causing the library to crash when loading the corrupted token.

### Status

Fixed, needs tests.

### Checklist (tick everything that applies)

- [x] [Code 
linting](http://libcloud.readthedocs.org/en/latest/development.html#code-style-guide)
 (required, can be done after the PR checks)
- [ ] Documentation
- [ ] [Tests](http://libcloud.readthedocs.org/en/latest/testing.html)
- [ ] 
[ICLA](http://libcloud.readthedocs.org/en/latest/development.html#contributing-bigger-changes)
 (required for bigger changes)

_write_token_to_file was not zeroing the file before writing
a new token, causing corruption.

FIXES: LIBCLOUD-835

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/qwil/libcloud 
LIBCLOUD-835_google-token-corruption

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/libcloud/pull/844.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #844


commit 9d05463aa2faa4733ac0129c2797ee9d043e58f9
Author: Paul Tiplady 
Date:   2016-07-22T18:32:27Z

[LIBCLOUD-835] Fix caching of Google auth tokens

_write_token_to_file was not zeroing the file before writing
a new token, causing corruption.

FIXES: LIBCLOUD-835




> Malformed auth token causes fatal exception in Google Storage driver
> 
>
> Key: LIBCLOUD-835
> URL: https://issues.apache.org/jira/browse/LIBCLOUD-835
> Project: Libcloud
>  Issue Type: Bug
>Reporter: Paul Tiplady
>Priority: Critical
>
> One of my Django instances has started hitting a libcloud error which is 
> causing a fatal exception, bringing down the instance.
> It looks like libcloud is writing invalid JSON into the auth token, which 
> then causes a JSON parse error when it is subsequently read back in.
> Here's the token that's written:
> {code}
> $ cat /root/.google_libcloud_auth.
> {"access_token": "", "token_type": "Bearer", "expire_time": 
> "2016-07-12T16:45:09Z", "expires_in": 3559}09Z", "expires_in": 3537}
> {code}
> Note the two "expires_in" keys, one with a nonsense value of `3559}09Z"`
> Environment:
> Python 3.4.4 
> apache-libcloud==1.0.0



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (LIBCLOUD-835) Malformed auth token causes fatal exception in Google Storage driver

[Paul Tiplady (JIRA)]

[ 
https://issues.apache.org/jira/browse/LIBCLOUD-835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392279#comment-15392279
 ] 

Paul Tiplady commented on LIBCLOUD-835:
---

Ping [~kami], [~ec_johnson2000] -- think this bug is high priority.

> Malformed auth token causes fatal exception in Google Storage driver
> 
>
> Key: LIBCLOUD-835
> URL: https://issues.apache.org/jira/browse/LIBCLOUD-835
> Project: Libcloud
>  Issue Type: Bug
>Reporter: Paul Tiplady
>Priority: Critical
>
> One of my Django instances has started hitting a libcloud error which is 
> causing a fatal exception, bringing down the instance.
> It looks like libcloud is writing invalid JSON into the auth token, which 
> then causes a JSON parse error when it is subsequently read back in.
> Here's the token that's written:
> {code}
> $ cat /root/.google_libcloud_auth.
> {"access_token": "", "token_type": "Bearer", "expire_time": 
> "2016-07-12T16:45:09Z", "expires_in": 3559}09Z", "expires_in": 3537}
> {code}
> Note the two "expires_in" keys, one with a nonsense value of `3559}09Z"`
> Environment:
> Python 3.4.4 
> apache-libcloud==1.0.0



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (LIBCLOUD-835) Malformed auth token causes fatal exception in Google Storage driver

[Paul Tiplady (JIRA)]

[ 
https://issues.apache.org/jira/browse/LIBCLOUD-835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390017#comment-15390017
 ] 

Paul Tiplady commented on LIBCLOUD-835:
---

I neglected to update, that this issue didn't actually clear when I upgraded to 
1.1.0.

I added some debug logging to the token generation code, and I have figured out 
what the problem is.

Here's a sequence of token writes (W) and reads (R):

{code}
W: {"token_type": "Bearer", "expires_in": 3599, "expire_time": 
"2016-07-22T07:12:46Z", "access_token": "ya29."}
W: {"token_type": "Bearer", "expires_in": 3294, "expire_time": 
"2016-07-22T18:17:05Z", "access_token": "ya29."}
R: {"token_type": "Bearer", "expires_in": 3294, "expire_time": 
"2016-07-22T18:17:05Z", "access_token": "ya29.<82 chars from new token"}<23 
chars from original token>"}
{code}

The write_token code is not clearing the old token before writing the new one, 
resulting in corruption.

The offending code, in libcloud/common/google.py:

{code}
with os.fdopen(os.open(filename, os.O_CREAT | os.O_WRONLY,
   int('600', 8)), 'w') as f:
f.write(data)
{code}

In the case where the file exists, O_CREAT is a no-op, so we're just opening 
the existing file and writing our bytes into it, without first clearing it. 
Need to set O_TRUNC as well, to get > semantics instead of >>.

I don't know what causes different token lengths to be returned by the Google 
APIs; since this issue appeared spontaneously, and only on one of my projects, 
it may be that the Google auth APIs changed recently, thus triggering this 
latent bug. Also note that I'm using auth type GoogleAuthType.GCE, which might 
scope the issue more tightly -- but the affected code is used by all Google 
auth types, so this in principle could be hit on any of them.

Until it's understood under what circumstances the Google APIs return different 
token lengths, I think it's safest to assume that this issue can break 
everybody using this library with Google Cloud Storage, so this looks like an 
exceptionally critical bug; I propose raising to Blocker level and making an 
immediate bugfix release with a fix, and backporting to all supported versions.

I'm going to knock together a quick fix on my fork of Libcloud, since it's not 
much code. I'll push a fix out without UTs onto my staging environment, and 
look at tests later. Happy to contribute this fix back. Will you want a test 
for this fix, even though it's a one-liner?

> Malformed auth token causes fatal exception in Google Storage driver
> 
>
> Key: LIBCLOUD-835
> URL: https://issues.apache.org/jira/browse/LIBCLOUD-835
> Project: Libcloud
>  Issue Type: Bug
>Reporter: Paul Tiplady
>Priority: Critical
>
> One of my Django instances has started hitting a libcloud error which is 
> causing a fatal exception, bringing down the instance.
> It looks like libcloud is writing invalid JSON into the auth token, which 
> then causes a JSON parse error when it is subsequently read back in.
> Here's the token that's written:
> {code}
> $ cat /root/.google_libcloud_auth.
> {"access_token": "", "token_type": "Bearer", "expire_time": 
> "2016-07-12T16:45:09Z", "expires_in": 3559}09Z", "expires_in": 3537}
> {code}
> Note the two "expires_in" keys, one with a nonsense value of `3559}09Z"`
> Environment:
> Python 3.4.4 
> apache-libcloud==1.0.0



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (LIBCLOUD-835) Malformed auth token causes fatal exception in Google Storage driver

[Paul Tiplady (JIRA)]

[ 
https://issues.apache.org/jira/browse/LIBCLOUD-835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375471#comment-15375471
 ] 

Paul Tiplady commented on LIBCLOUD-835:
---

I upgraded from 1.0.0 to 1.1.0, and the error has not resurfaced (yet).

> Malformed auth token causes fatal exception in Google Storage driver
> 
>
> Key: LIBCLOUD-835
> URL: https://issues.apache.org/jira/browse/LIBCLOUD-835
> Project: Libcloud
>  Issue Type: Bug
>Reporter: Paul Tiplady
>Priority: Critical
>
> One of my Django instances has started hitting a libcloud error which is 
> causing a fatal exception, bringing down the instance.
> It looks like libcloud is writing invalid JSON into the auth token, which 
> then causes a JSON parse error when it is subsequently read back in.
> Here's the token that's written:
> {code}
> $ cat /root/.google_libcloud_auth.
> {"access_token": "", "token_type": "Bearer", "expire_time": 
> "2016-07-12T16:45:09Z", "expires_in": 3559}09Z", "expires_in": 3537}
> {code}
> Note the two "expires_in" keys, one with a nonsense value of `3559}09Z"`
> Environment:
> Python 3.4.4 
> apache-libcloud==1.0.0



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (LIBCLOUD-835) Malformed auth token causes fatal exception in Google Storage driver

[Tomaz Muraus (JIRA)]

[ 
https://issues.apache.org/jira/browse/LIBCLOUD-835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373323#comment-15373323
 ] 

Tomaz Muraus commented on LIBCLOUD-835:
---

Thanks for the report, we will look into it.

/cc [~erjohnso]

> Malformed auth token causes fatal exception in Google Storage driver
> 
>
> Key: LIBCLOUD-835
> URL: https://issues.apache.org/jira/browse/LIBCLOUD-835
> Project: Libcloud
>  Issue Type: Bug
>Reporter: Paul Tiplady
>Priority: Critical
>
> One of my Django instances has started hitting a libcloud error which is 
> causing a fatal exception, bringing down the instance.
> It looks like libcloud is writing invalid JSON into the auth token, which 
> then causes a JSON parse error when it is subsequently read back in.
> Here's the token that's written:
> {code}
> $ cat /root/.google_libcloud_auth.
> {"access_token": "", "token_type": "Bearer", "expire_time": 
> "2016-07-12T16:45:09Z", "expires_in": 3559}09Z", "expires_in": 3537}
> {code}
> Note the two "expires_in" keys, one with a nonsense value of `3559}09Z"`
> Environment:
> Python 3.4.4 
> apache-libcloud==1.0.0



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)