On Tue, Sep 26, 2017 at 8:14 AM, Michael Merickel <[email protected]> wrote:
>> What is the Origin header and shouldn't Pyramid/WebOb set it automatically
>> if it's becoming more important?
>
> 1) You can read the RFC about it or anything on google.
> 2) The Origin header is set by the client, not the server.
>
> If the origin matches the current domain (usually set by the host header)
> then the request is trusted by default and you do not need to modify
> anything. You have your app setup in such a way that your requests *look*
> like they are originating from another server instead of the domain hosting
> the content. Just configure your webtest requests such that the origin and
> host match and you'll be fine.
>
>> I still feel like there's a missing piece, something that needs to be
>> documented so that others don't fall into this same trap.
>
> If you would like to contribute some documentation on this once you figure
> it out I'm more than happy to review. It probably belongs in the testing
> chapter about how to use webtest. If you think webtest should set the
> origin/host the same by default then perhaps you could open an issue over
> there.
OK, I've got something that works; I'll make a cookbook page for
testing with HTTPS-only cookies. I'll try to explain it precisely now
although it's a long journey through webtest.app, webob.request,
pyramid.csrf, and session cookies. To recap the situation:
1. The login form is CSRF-protected. (IT requirement.)
2. The app uses SessionAuthenticationPolicy and
pyramid_redis_sessions. Cookies are HTTPS-only (IT requirement.), so
setting 'redis.sessions.cookie_secure = true'. This tells the client
to send the cookie only with HTTPS requests (not HTTP).
3. All this is working fine in production.
4. The functional test logs in via a form. It uses WebTest. There is
no web server, just WebTest calling the WSGI app.
5. The test ran fine before implementing HTTPS cookies and CSRF protection.
6. HTTPS cookies broke the test, because WebTest emulates an HTTP
environment by default, so it won't transmit the cookie that indicates
the session containing the authenticated status. The login submit
fails.
7.I made the request "HTTPS" with an environ var:
``extra_environ = {"wsgi.url_scheme": "https"}``.
``apptest = webtest.testApp(app, extra_environ=extra_environ}``.
That fixed #6.
8. I implemented CSRF checking, and now the login failed because the
"origin" (the client's domain) was different was different from what
the "server" expected.
9. Troubleshooting and this list revealed two workarounds: turn off
HTTPS-only cookies and revert to HTTP, or set the de facto host as a
trusted host. (Setting 'pyramid.csrf_trusted_origins = localhost:80'.)
10. However, I didn't want to make my test configuration so different
from the real one and lose some aspect of testing. Instead I wanted to
understand the underlying problem and fix it.
11. The problem is that in the form submission request, the referer
domain was different from the current domain. I don't have access to
the request; it's internal to WebTest. I put print statements in
'pyramid.csrf' to determine what the request attributes were and what
the mismatch was. The actual domain was 'localhost' while the referer
had 'localhost:80'. It wasn't clear what code was calculating the
referer value or what it based it on. It didn't make sense because
didn't it know that 'localhost' is the same as 'localhost:80'?
12. Enlightenment came when I realized the referer was
"https://localhost:80";, which is nonsense. The default HTTPS port is
443, not 80.
13. I tried setting extra_environ "HTTP_PORT" : "443" but that didn't
work: the referer still had "https://localhost:80";.
14. I tracked down the code to the
'webob.request.BaseRequest.host_url' property. It looks first in
envvar 'HTTP_HOST' and falls back to 'SERVER_NAME' and 'SERVER_PORT'.
15. So I changed my code thus:
``extra_environ = {"HTTP_HOST": "443", "wsgi.url_scheme": "https"}
apptest.webtest.TestApp(app, extra_environ=extra_environ)''
Presto, it works!
I'm not sure if 'webtest' should be smarter, or if so exactly how. But
at least this solves this underlying problem.
Is there anything else I should do to more completely mimic an HTTPS request?
> - Michael
>
>
> On Tue, Sep 26, 2017 at 12:52 AM, Mike Orr <[email protected]> wrote:
>>
>> On Mon, Sep 25, 2017 at 10:40 PM, Mike Orr <[email protected]> wrote:
>> > On Mon, Sep 25, 2017 at 9:00 PM, Mike Orr <[email protected]> wrote:
>> >> On Mon, Sep 25, 2017 at 5:47 PM, Michael Merickel <[email protected]>
>> >> wrote:
>> >>>> So what's the best way forward?
>> >>>
>> >>> I think you covered your options pretty well.
>> >>>
>> >>> 1) Set wsgi.url_scheme to "http" as origin checks are only done on
>> >>> https.
>> >>> 2) Set the pyramid.csrf_trusted_origins as you are doing now.
>> >>> 3) Disable csrf checking for your tests.
>> >>>
>> >>> I think it's just a helpful reminder that you would be wise to think
>> >>> about
>> >>> the origin header more these days as it's required by CORS requests
>> >>> and, of
>> >>> course, cross origin requests are the attack vector CSRF is helping to
>> >>> protect.
>> >>
>> >> It sounds like it needs documentation then. What is the Origin header
>> >> and shouldn't Pyramid/WebOb set it automatically if it's becoming more
>> >> important?
>> >>
>> >> #1 and #3 would make the test environment different from the real
>> >> environment. #2 raises the question of what is WebTest's Origin
>> >> header, what should it be, why are they different, and does something
>> >> need to be changed in the library?
>> >
>> > I guess the solution is #1, to roll back the HTTPS, because there is
>> > no HTTPS because there's no network server. That in turn will require
>> > a configuration that doesn't make the cookies HTTPS-only.
>>
>> (The browser got in a mood and did "Send" too quickly.)
>>
>> I still feel like there's a missing piece, something that needs to be
>> documented so that others don't fall into this same trap. The only
>> reason I set HTTPS-only and CSRF is our IT department asked us to do
>> this wherever feasible, and since it didn't make much difference
>> either way I went along with it. So presumably other people in other
>> organizations will be doing the same thing, and have the same thing
>> happen in their tests.
>>
>>
>> --
>> Mike Orr <[email protected]>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "pylons-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To post to this group, send email to [email protected].
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/pylons-discuss/CAH9f%3Duo%2BAYpm6-oS-vPLFg0Ek3gz1G2JDHcW2m0MmdH_X8zp4g%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pylons-discuss/CAKdhhwFWJkrGFxOTkAyMDB61JkCm0%3D%3DwWONQ5V9U60D%3DAqV%3DXQ%40mail.gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.
--
Mike Orr <[email protected]>
--
You received this message because you are subscribed to the Google Groups
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/pylons-discuss/CAH9f%3Duqvpy-3pQ3rCVSi0K4roB6kkt8VqmxAPo20YYKtgeOgvw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
