Re: About Understanding of source code
I would suggest you to start by looking into the contrib apps. This would give you great insights onto writing Django apps. Now that you are already in the source code, you can stumble into the rest of it or even the core. This way has been particularly helpful to me. Also, you might want to look into this great talk by James Bennett, titled "Django in Depth": http://www.youtube.com/watch?v=t_ziKY1ayCo -- Rohan Jain On Mon, Jan 7, 2013 at 10:27 AM, Mayur Patil <ram.nath241...@gmail.com>wrote: > Hello there, > > I want to understand how to get deep insight into Django code? > > Thank You. > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/django-developers/-/CcvuNeLYrJsJ. > To post to this group, send email to django-developers@googlegroups.com. > To unsubscribe from this group, send email to > django-developers+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-developers?hl=en. > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: GSoC Check-in: Security Enhancements
Hi, Ref Merging Centralized Tokenization. I found out that some tests for sessions are failing with it. Stupidly, I didn't have sessions in my test subset for tokenization. Anyway, I'll try to get it fixed and write some tests for tokens in next few days. -- Thanks Rohan On 10:50 +0100 / 21 Aug, Andrew Godwin wrote: > Thanks for your work during the GSOC, Rohan - don't worry about not > achieving everything, it looks like there's still some useful code there! > > Hopefully we can get some of the code merged, especially centralised > tokenisation if it's so near completion, as it looks like a nice bit of > cleanup code! > > Andrew > > On Mon, Aug 20, 2012 at 2:49 PM, Rohan Jain <crod...@gmail.com> wrote: > > > Hi, > > > > Today is the 'pencils down' date for this GSoC project. Past 4 months > > have been a great learning experience from me. Just being in the > > context of security side of the web has been really beneficial. Moving > > around in a very well written code base is also delightful. > > Meanwhile, I did get to work on multiple sections of the code base. > > But compared to the expectations at the initial phase of the program, > > I myself am a little disappointed about the output I was able to > > generate out of the time - specially regarding CSRF. > > This can be attributed to my being unable to give consistent attention > > to the project through the duration and to solicit feedback. Also, the > > project comprised of multiple discreet tasks, so not enough > > consideration was there for each task. > > > > This is a overview of various complete/incomplete topics I worked on. > > > > Centralized Tokenization: > > This was about integrating a central tokenization api throughout the > > django project. I tracked this at [centralized-tokenization][0] branch > > of my fork on github. This is near completion, and also has a > > documentation[1] of the API with basic usage examples. > > > > Improvements to sessions: > > My first task, related to some improvements to the sessions framework. > > Includes signing and session cleanup based on the engine. Tracked at > > [sessions-improvements][2] branch. > > > > CSRF Enhancements: > > It occupied most of my time through the project. I tried multiple ways > > to solve the issues under this. Tracked at [csrf-enhancements][3]. > > First up, I started with using the origin header for doing a CSRF > > check. One implementation with some tests is at > > [csrf-origin-checking][4]. Recently I started on the idea of > > implementing moduler checkers for each kind of CSRF check, but > > haven't got anything useful out of it yet. While progressing, it > > seemed like I was virtually writing a middleware per checker, so now I > > have moved on to attempt on CSRF cookie store. Basically something > > which should seamlessly switch Session/Cookie for storage based on the > > availability of session. > > > > I'll try to hang around the django bug tracker and submit some > > patches. > > > > -- > > Thanks > > Rohan Jain > > > > [0]: https://github.com/crodjer/django/tree/centralized-tokenization > > [1]: > > https://github.com/crodjer/django/blob/centralized-tokenization/docs/topics/tokenization.txt > > [2]: https://github.com/crodjer/django/tree/sessions-improvements > > [3]: https://github.com/crodjer/django/tree/csrf-enhancements > > [4]: https://github.com/crodjer/django/tree/csrf-origin-checking > > [5]: https://github.com/crodjer/django/tree/csrf-moduler-checkers > > > > On 01:57 +0530 / 7 Aug, Rohan Jain wrote: > > > Hi, > > > > > > Sorry for the delay in getting back. I was meanwhile working on > > > centralized tokenization for few days, while still trying to figure > > > something better for CSRF. > > > > > > On 03:52 -0400 / 25 Jul, Alex Ogier wrote: > > > > On Tue, Jul 24, 2012 at 11:37 PM, Rohan Jain <crod...@gmail.com> > > wrote: > > > > > > > > > > I had one more idea, "Pluggable CSRF checkers". > > > > > > > > > > Currently, the CSRF middleware has two kinds of checks, referer (for > > > > > https) and secret validation token (common). These with origin header > > > > > based checker (if we add it) come in conditional blocks, making > > > > > switching them difficult. So what I propose to do is decouple their > > > > > logic from CSRF middleware and each of them provide a checker. It > > goes > > > > > like this: > > > > &g
Re: GSoC Check-in: Security Enhancements
Hi, Thanks for the reply. On 10:50 +0100 / 21 Aug, Andrew Godwin wrote: > Thanks for your work during the GSOC, Rohan - don't worry about not > achieving everything, it looks like there's still some useful code there! > > Hopefully we can get some of the code merged, especially centralised > tokenisation if it's so near completion, as it looks like a nice bit of > cleanup code! Hoping that we can merge and ship it by 1.5. There is a possibility of running into issues still, I'll try to fix those as they come up in future. -- Rohan > > Andrew > > On Mon, Aug 20, 2012 at 2:49 PM, Rohan Jain <crod...@gmail.com> wrote: > > > Hi, > > > > Today is the 'pencils down' date for this GSoC project. Past 4 months > > have been a great learning experience from me. Just being in the > > context of security side of the web has been really beneficial. Moving > > around in a very well written code base is also delightful. > > Meanwhile, I did get to work on multiple sections of the code base. > > But compared to the expectations at the initial phase of the program, > > I myself am a little disappointed about the output I was able to > > generate out of the time - specially regarding CSRF. > > This can be attributed to my being unable to give consistent attention > > to the project through the duration and to solicit feedback. Also, the > > project comprised of multiple discreet tasks, so not enough > > consideration was there for each task. > > > > This is a overview of various complete/incomplete topics I worked on. > > > > Centralized Tokenization: > > This was about integrating a central tokenization api throughout the > > django project. I tracked this at [centralized-tokenization][0] branch > > of my fork on github. This is near completion, and also has a > > documentation[1] of the API with basic usage examples. > > > > Improvements to sessions: > > My first task, related to some improvements to the sessions framework. > > Includes signing and session cleanup based on the engine. Tracked at > > [sessions-improvements][2] branch. > > > > CSRF Enhancements: > > It occupied most of my time through the project. I tried multiple ways > > to solve the issues under this. Tracked at [csrf-enhancements][3]. > > First up, I started with using the origin header for doing a CSRF > > check. One implementation with some tests is at > > [csrf-origin-checking][4]. Recently I started on the idea of > > implementing moduler checkers for each kind of CSRF check, but > > haven't got anything useful out of it yet. While progressing, it > > seemed like I was virtually writing a middleware per checker, so now I > > have moved on to attempt on CSRF cookie store. Basically something > > which should seamlessly switch Session/Cookie for storage based on the > > availability of session. > > > > I'll try to hang around the django bug tracker and submit some > > patches. > > > > -- > > Thanks > > Rohan Jain > > > > [0]: https://github.com/crodjer/django/tree/centralized-tokenization > > [1]: > > https://github.com/crodjer/django/blob/centralized-tokenization/docs/topics/tokenization.txt > > [2]: https://github.com/crodjer/django/tree/sessions-improvements > > [3]: https://github.com/crodjer/django/tree/csrf-enhancements > > [4]: https://github.com/crodjer/django/tree/csrf-origin-checking > > [5]: https://github.com/crodjer/django/tree/csrf-moduler-checkers > > > > On 01:57 +0530 / 7 Aug, Rohan Jain wrote: > > > Hi, > > > > > > Sorry for the delay in getting back. I was meanwhile working on > > > centralized tokenization for few days, while still trying to figure > > > something better for CSRF. > > > > > > On 03:52 -0400 / 25 Jul, Alex Ogier wrote: > > > > On Tue, Jul 24, 2012 at 11:37 PM, Rohan Jain <crod...@gmail.com> > > wrote: > > > > > > > > > > I had one more idea, "Pluggable CSRF checkers". > > > > > > > > > > Currently, the CSRF middleware has two kinds of checks, referer (for > > > > > https) and secret validation token (common). These with origin header > > > > > based checker (if we add it) come in conditional blocks, making > > > > > switching them difficult. So what I propose to do is decouple their > > > > > logic from CSRF middleware and each of them provide a checker. It > > goes > > > > > like this: > > > > > > > > > > A setting for configuring global CSRF checkers: > >
Re: GSoC Check-in: Security Enhancements
Hi, Today is the 'pencils down' date for this GSoC project. Past 4 months have been a great learning experience from me. Just being in the context of security side of the web has been really beneficial. Moving around in a very well written code base is also delightful. Meanwhile, I did get to work on multiple sections of the code base. But compared to the expectations at the initial phase of the program, I myself am a little disappointed about the output I was able to generate out of the time - specially regarding CSRF. This can be attributed to my being unable to give consistent attention to the project through the duration and to solicit feedback. Also, the project comprised of multiple discreet tasks, so not enough consideration was there for each task. This is a overview of various complete/incomplete topics I worked on. Centralized Tokenization: This was about integrating a central tokenization api throughout the django project. I tracked this at [centralized-tokenization][0] branch of my fork on github. This is near completion, and also has a documentation[1] of the API with basic usage examples. Improvements to sessions: My first task, related to some improvements to the sessions framework. Includes signing and session cleanup based on the engine. Tracked at [sessions-improvements][2] branch. CSRF Enhancements: It occupied most of my time through the project. I tried multiple ways to solve the issues under this. Tracked at [csrf-enhancements][3]. First up, I started with using the origin header for doing a CSRF check. One implementation with some tests is at [csrf-origin-checking][4]. Recently I started on the idea of implementing moduler checkers for each kind of CSRF check, but haven't got anything useful out of it yet. While progressing, it seemed like I was virtually writing a middleware per checker, so now I have moved on to attempt on CSRF cookie store. Basically something which should seamlessly switch Session/Cookie for storage based on the availability of session. I'll try to hang around the django bug tracker and submit some patches. -- Thanks Rohan Jain [0]: https://github.com/crodjer/django/tree/centralized-tokenization [1]: https://github.com/crodjer/django/blob/centralized-tokenization/docs/topics/tokenization.txt [2]: https://github.com/crodjer/django/tree/sessions-improvements [3]: https://github.com/crodjer/django/tree/csrf-enhancements [4]: https://github.com/crodjer/django/tree/csrf-origin-checking [5]: https://github.com/crodjer/django/tree/csrf-moduler-checkers On 01:57 +0530 / 7 Aug, Rohan Jain wrote: > Hi, > > Sorry for the delay in getting back. I was meanwhile working on > centralized tokenization for few days, while still trying to figure > something better for CSRF. > > On 03:52 -0400 / 25 Jul, Alex Ogier wrote: > > On Tue, Jul 24, 2012 at 11:37 PM, Rohan Jain <crod...@gmail.com> wrote: > > > > > > I had one more idea, "Pluggable CSRF checkers". > > > > > > Currently, the CSRF middleware has two kinds of checks, referer (for > > > https) and secret validation token (common). These with origin header > > > based checker (if we add it) come in conditional blocks, making > > > switching them difficult. So what I propose to do is decouple their > > > logic from CSRF middleware and each of them provide a checker. It goes > > > like this: > > > > > > A setting for configuring global CSRF checkers: > > > > > > CSRF_CHECKERS = { > > > 'django.middleware.csrf.checkers.OriginChecker', > > > # This one can be strict for https and lax for http > > > 'django.middleware.csrf.checkers.RefererChecker', > > > # contrib.sessions could provide a csrf checker maintained > > > # with sessions. This stores the token in session data. > > > 'django.contrib.sessions.csrf_checkers.SessionChecker' > > > } > > > > > > > I don't think this is a good idea. If you enumerate security features > > in settings.py, then later additions won't be picked up by default. If > > Django add a new CSRF checking mechanism, we want everybody to take > > advantage of it with no modifications. > > > > Ordinarily I agree with you, explicit is better than implicit. > > However, in the case of security features, I think this is inverted: > > Django sites should be implicitly enrolled in all security mechanisms > > if possible, and should be able to explicitly opt out if necessary. > > Almost everyone should be using every single protection Django offers > > on all their requests, and therefore it should be verbose and > > discouraged to turn off these protections. > > > > Hmm, that is a valid point. I can drop the configurable CSRF
Re: GSoC Check-in: Security Enhancements
Hi, Sorry for the delay in getting back. I was meanwhile working on centralized tokenization for few days, while still trying to figure something better for CSRF. On 03:52 -0400 / 25 Jul, Alex Ogier wrote: > On Tue, Jul 24, 2012 at 11:37 PM, Rohan Jain <crod...@gmail.com> wrote: > > > > I had one more idea, "Pluggable CSRF checkers". > > > > Currently, the CSRF middleware has two kinds of checks, referer (for > > https) and secret validation token (common). These with origin header > > based checker (if we add it) come in conditional blocks, making > > switching them difficult. So what I propose to do is decouple their > > logic from CSRF middleware and each of them provide a checker. It goes > > like this: > > > > A setting for configuring global CSRF checkers: > > > > CSRF_CHECKERS = { > > 'django.middleware.csrf.checkers.OriginChecker', > > # This one can be strict for https and lax for http > > 'django.middleware.csrf.checkers.RefererChecker', > > # contrib.sessions could provide a csrf checker maintained > > # with sessions. This stores the token in session data. > > 'django.contrib.sessions.csrf_checkers.SessionChecker' > > } > > > > I don't think this is a good idea. If you enumerate security features > in settings.py, then later additions won't be picked up by default. If > Django add a new CSRF checking mechanism, we want everybody to take > advantage of it with no modifications. > > Ordinarily I agree with you, explicit is better than implicit. > However, in the case of security features, I think this is inverted: > Django sites should be implicitly enrolled in all security mechanisms > if possible, and should be able to explicitly opt out if necessary. > Almost everyone should be using every single protection Django offers > on all their requests, and therefore it should be verbose and > discouraged to turn off these protections. > Hmm, that is a valid point. I can drop the configurable CSRF settings. But still a modular CSRF checkers might be useful, in which the checkers are selected dynamically. When sessions app is present, use sessions checker instead of cookies based CSRF token store. Also we can have switches which incorporate existing behaviour based on https/http connection and also origin header checking based on its presence. I will do a prototype implementation for this to polish and clarify the idea. ## Centralized Tokenization: A functioning implementation is up at the [centralize-tokenization][0] branch on my fork. I have written an [initial documentation][1] for this too. This is basically a merge with my cleanups and customizations over the work done in period of [djangocon 2011][2]. [0]: https://github.com/crodjer/django/commits/centralized-tokenization [1]: https://gist.github.com/2203174#file_tokenization.mkd [2]: https://github.com/yarko/django/commits/djangocon2011-sec -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: GSoC Check-in: Security Enhancements
On 19:46 +0100 / 23 Jul, Luke Plant wrote: > On 23/07/12 14:24, Rohan Jain wrote: > > With this, attacker won't be able to directly set arbitrary tokens on > > other sub domains through cookies, they will need a signature of the > > token with the form which is to be verified against the cookie. > > Plus it also puts a limit on the duration a token stays valid on the > > server side. > > > > Yes, still with this, someone can spoof the whole pair using a separate > > legitimate session. So really it doesn't make it completely secure, > > just makes it little difficult for the attacker. > > So, to make it clear: > > Attacker controls evil.example.com, and wants to attack example.com. By > appropriate setting of a cookie, and by providing a matching token in > the form, they can forge a request to example.com > > With your proposed change, they are in no way hampered. Using HMAC to do > sign the value, and a timestamp, as you mention, they can simply > regularly directly contact example.com and pull in a valid token/cookie > value pair, which they can use since there is no correlation to the > session of the person being attacked. I'm pretty sure this can be done > entirely in Javascript too. > > Yes, this change would make it "a little difficult". But the value of > "little" is very small. It's like suggesting we add ROT 13 encryption to > one of the values - sure it makes it a "little" more difficult, but it > does not *materially* affect the feasibility of the attack. The > resources they need are identical: the ability to read the Django source > code, and the subdomain control that they have in both cases. > > On the other hand, this adds significant complication to our code, which > is the last thing you need for security related code. I'm -1 on this > change. I did highlight all these things before. > > > > If it weren't for the possibility of attacker injecting cookies from > > other subdomains, I think CSRF token should be a fine check for > > CSRF. > > > > That is why I am siding on adding referer checking in case of non > > https scheme requests too. > > I really don't think we can consider this - for HTTP, proxies can and do > strip the referer header. Quoting from Barth, Jackon and Mitchell: > > <<< > Over HTTP, sites cannot afford > to block requests that lack a Referer header because > they would cease to be compatible with the sizable > percentage (roughly 3–11%) of users > >> > > This makes strict referer checking a non-started, and lax referer > checking (only check it if it is present) has known flaws. Thanks for this quote. I didn't know about this paper and the fact that the web behaves differentially with respect to referer checking in http from https earlier. So, yes relying on it isn't the best idea. I had one more idea, "Pluggable CSRF checkers". Currently, the CSRF middleware has two kinds of checks, referer (for https) and secret validation token (common). These with origin header based checker (if we add it) come in conditional blocks, making switching them difficult. So what I propose to do is decouple their logic from CSRF middleware and each of them provide a checker. It goes like this: A setting for configuring global CSRF checkers: CSRF_CHECKERS = { 'django.middleware.csrf.checkers.OriginChecker', # This one can be strict for https and lax for http 'django.middleware.csrf.checkers.RefererChecker', # contrib.sessions could provide a csrf checker maintained # with sessions. This stores the token in session data. 'django.contrib.sessions.csrf_checkers.SessionChecker' } Views can have their own custom checkers: from django.view.decorators.csrf import csrf_checkers @csrf_checkers( 'django.middleware.csrf.checkers.OriginChecker', 'django.middleware.csrf.checkers.TokenChecker', ) def foo(request): '''This view needs token and origin for CSRF checks. For some reason session can't be used here.''' @csrf_checkers( 'django.middleware.csrf.checkers.OriginChecker', 'django.middleware.csrf.checkers.SignedTokenChecker', ) def bar(request): '''This view for some reason needs time signed token and origin for CSRF checks.''' A base checker class: class BaseCSRFChecker(object): def __init__(request, *args, **kwargs): self.request = request def _accept(self, request): request.csrf_processing_done = True return None def _reject(self, request, reason): return _get_failure_view()(request, reason=reason) def _check(strict=True): raise NotImplementedError ("Checkers m
Re: GSoC Check-in: Security Enhancements
On 11:06 +0100 / 23 Jul, Luke Plant wrote: > On 23/07/12 08:07, Rohan Jain wrote: > > ###CSRF Cookies (Time signed): > > > > - A random token generated by the server stored in the browser cookies. For > >verification, every non get request will need to provide a signed > > version of > >the same token. This can then be verified on the browser side. > > - This can be implemented by adding signing to the existing csrf token > > system, > >using the signing framework. > > - A conventional method of CSRF checks, all the major frameworks have > > similar > >systems. > > - Signing takes care of the side effects due to cross domain behaviour of > >cookies. > > - Cons: > > > > - Relies on the browser cookies system, which introduces insecurities. > > - Can be broken easily by having a parallel legitimate session, which > > gives > > a valid token, signature pair. This generator can then be used in MITM > > attacks. > > I don't understand the motivation for this part. Could you describe the > attack scenarios where this method improves our security with respect to > the current system? With this, attacker won't be able to directly set arbitrary tokens on other sub domains through cookies, they will need a signature of the token with the form which is to be verified against the cookie. Plus it also puts a limit on the duration a token stays valid on the server side. Yes, still with this, someone can spoof the whole pair using a separate legitimate session. So really it doesn't make it completely secure, just makes it little difficult for the attacker. If it weren't for the possibility of attacker injecting cookies from other subdomains, I think CSRF token should be a fine check for CSRF. That is why I am siding on adding referer checking in case of non https scheme requests too. -- Thanks Rohan -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: GSoC Check-in: Security Enhancements
Hi, Centralized Tokenization: I have merged the work already done for centralized-tokenization in last djangocon at [yarko/django][0]. Since it there have been significant amount of changes since then, merging and resolving conflicts took a little more time then expected. As of now the tests are passing. I am yet to do some minor work to move remaining components to the token system. I'll post it up by tonight or tomorrow and then document how the system works. CSRF: Clubbing the information regarding CSRF, I have started writing down an overview about it in one place, listing the pros and cons of the methods for CSRF checking. Its in my same soc proposal gist: <https://gist.github.com/2203174#file_csrf.mkd> Also added the content of the gist in the end of this mail. -- Thanks Rohan [0]: https://github.com/yarko/django [1]: https://github.com/crodjer/django/commits/centralized-tokenization ## Current state Django currently uses two methods for protection against CSRF: - In case of http non SSL requests, a conventional token system is used. - In case of https requests, we do referer checking. Both these system introduce a difference in behaviour. Using cookie-token system provides a side effect feature which lets you access the CSRF token across multiple subdomains. But the other, referer checking looks for the request to come from the same host and hence, doesn't allow for cookie like cross subdomain behaviour. The fact that there is a `CSRF_COOKIE_DOMAIN` setting shows that django supports the cross domain capabilities in the CSRF system. So users might utilize this while building multiple sites. When they change to https scheme, they will see a new surprising behaviour. ## The modified system The working tree is at [crodjer/django](https://github.com/crodjer/django/tree/csrf-enhancements) over github. ###Origin and Referer Checking, Permitted domains. - Origin header, a fairly new introduction, has CSRF as one of the target. Unlike the referer header, it sends back only the domain and the path information, so the people don't need to disable these through plugins like the referer header. - With origin header absent, referer header can be used as a fallback. - Strict referer checking is already there in https scheme which is used in all major deployments. We can bring it to the non secure scheme also. - With this method a permitted domains functionality is possible, which lets explicit specification of domains which are trusted and can bypass CSRF. A prototype implementation is there at crodjer/django, csrf-enhancements branch. - Cons: - Since origin checking is still a new concept, it is not implemented yet in major browsers. - Referer headers spoofing is also heard of in some old versions of flash. - Plugins which disable referer header are popular. A client which doesn't provide origin header and with one such plugin, will fail against this kind of checking. ###CSRF Cookies (Time signed): - A random token generated by the server stored in the browser cookies. For verification, every non get request will need to provide a signed version of the same token. This can then be verified on the browser side. - This can be implemented by adding signing to the existing csrf token system, using the signing framework. - A conventional method of CSRF checks, all the major frameworks have similar systems. - Signing takes care of the side effects due to cross domain behaviour of cookies. - Cons: - Relies on the browser cookies system, which introduces insecurities. - Can be broken easily by having a parallel legitimate session, which gives a valid token, signature pair. This generator can then be used in MITM attacks. Given these two methods, I am slightly inclined at Origin/Referer checking, which is already being completely relied upon in case of the secure requests. Or we could loose the possibility of permitted domains functionality and implement both kinds of checks. On 02:16 +0530 / 10 Jul, Rohan Jain wrote: > Hi, > > This check-in is a little delayed. Meanwhile, I continued working on > CSRF and started a on centralized tokenization. > > As I had posted, I implemented the permitted domains system, as > `CSRF_PERMITTED_DOMAINS`. I did some minor cleanups and tests, the > behaviour is mostly the same. Since with the origin header peresent, > referer checking can be neglected, now the system bypasses referer > checking when origin header is not none. > > The new permitted domains system is practically invalidated because of > the existing csrf cookie and token functionality. So for this, I > created a separate branch on my fork, which goes by > [purge-cookies][0]. In this I'll try to remove the CSRF cookie > functionality without affecting the interface which CSRF checking > system provides here. This interface could generate
Re: GSoC Check-in: Security Enhancements
Hi, This check-in is a little delayed. Meanwhile, I continued working on CSRF and started a on centralized tokenization. As I had posted, I implemented the permitted domains system, as `CSRF_PERMITTED_DOMAINS`. I did some minor cleanups and tests, the behaviour is mostly the same. Since with the origin header peresent, referer checking can be neglected, now the system bypasses referer checking when origin header is not none. The new permitted domains system is practically invalidated because of the existing csrf cookie and token functionality. So for this, I created a separate branch on my fork, which goes by [purge-cookies][0]. In this I'll try to remove the CSRF cookie functionality without affecting the interface which CSRF checking system provides here. This interface could generate deprecation warnings about CSRF cookie setting and csrf middleware token. Later we can remove this interface completely too. I would also like to get some suggestions if completely shifting towards header checking will be a good idea. From all I could find out from other frameworks out there, tokens and cookies are used to save a state for CSRF checks generally. Header checks are hardly used anywhere. Is it because of some flaw with how the headers system work? Given that referer checking is already being used for request which use the https scheme, I am assuming this is okay to do and will only decrease the vulnerabilites which exists from cookies system. If there are some reasons for using a cookie, we could at least start siging them. Probably have the post form send a signed (and timed?) hash apart from the csrf token for verification. Because otherwise it is easy to break the system if the attacker has access to some other subdomain, maybe on a shared hosting. Awaiting some feedback on CSRF, I have now started looking into integrating a centeralized tokenization system with django. Work for this already exists, done in djangocon 2011 and exists at [yarko/django][1] on github. The commits were on django-old, so it can't be merged directly. I cherry picked commits by commits and ran the merge conflicts with my best guesses. The result of the merge (broken) exists at [centralized-tokenization][2] branch of my fork. I'll have to look into this one more and see if I get it to work, as it does at yarko/django. Also, my first change sets regarding contrib.sessions are still pending some feedback, at [pull-78].[3]. -- Thanks Rohan Jain [0]: https://github.com/crodjer/django/tree/purge-cookies [1]: https://github.com/yarko/django [2]: https://github.com/crodjer/django/commits/centralized-tokenization [3]: https://github.com/django/django/pull/78 On Tue, Jun 12, 2012 at 1:32 PM, Rohan Jain <crod...@gmail.com> wrote: > > I have done some work on CSRF revolving around origin header checking. > The origin header is fairly new and is not yet implemented in a > uniform fashion in the major browsers, so it cannot be solely relied > upon for CSRF checks. Instead we check if the header exists and use it > only for rejection of requests. > > If an `HTTP_ORIGIN` header is sent by the browser, it checks if it > matches to be from a valid origin. In case it is not the request is > rejected right away. Otherwise it proceeds for further CSRF checks > > Because of the way browser cookies behave, I had to do some unpleasant > tweaks to origin header checking such that `COOKIE_DOMAIN_SETTING` is > followed. From the commit: > > if not ((good_origin.startswith('.') > and good_origin.count('.') is origin.count('.') > and origin.endswith(good_origin)) > or origin[origin.find('://')+3:] == good_origin): > > All this is because using cookies open the possibility for allowing > cross subdomain requests (a side effect?). I had a chat with Paul for > breaking (fixing) this behaviour. The plan is to: > > - Make the strict referer checking, currently implemented only for >HTTPS, default for http requests too. The http scheme which is more >popular definitely deserves better checking. > - Make the origin checking work similar to the way referer checking >is done. Also, will it be safe to bypass referer checking in case >of origin header being present? > - Implement a PERMITTTED_DOMAINS setting which lets administrators >explicitly mention the domains which are permitted to make the >otherwise restricted requests. This gives more control to the >administrators. > - The PERMITTED_DOMAINS setting, a list, accepts patterns. Regex >cannot be used here because characters like `.`, `-` will need >escaping. So I decided to settle on the simpler unix style globs. >The pattern matching is done through the `fnmatch` library >function, documented [here][fnmatch-docs]. One can use this setting >like this: > > PERMITTED_
Re: GSoC Check-in: Security Enhancements
I have done some work on CSRF revolving around origin header checking. The origin header is fairly new and is not yet implemented in a uniform fashion in the major browsers, so it cannot be solely relied upon for CSRF checks. Instead we check if the header exists and use it only for rejection of requests. If an `HTTP_ORIGIN` header is sent by the browser, it checks if it matches to be from a valid origin. In case it is not the request is rejected right away. Otherwise it proceeds for further CSRF checks Because of the way browser cookies behave, I had to do some unpleasant tweaks to origin header checking such that `COOKIE_DOMAIN_SETTING` is followed. From the commit: if not ((good_origin.startswith('.') and good_origin.count('.') is origin.count('.') and origin.endswith(good_origin)) or origin[origin.find('://')+3:] == good_origin): All this is because using cookies open the possibility for allowing cross subdomain requests (a side effect?). I had a chat with Paul for breaking (fixing) this behaviour. The plan is to: - Make the strict referer checking, currently implemented only for HTTPS, default for http requests too. The http scheme which is more popular definitely deserves better checking. - Make the origin checking work similar to the way referer checking is done. Also, will it be safe to bypass referer checking in case of origin header being present? - Implement a PERMITTTED_DOMAINS setting which lets administrators explicitly mention the domains which are permitted to make the otherwise restricted requests. This gives more control to the administrators. - The PERMITTED_DOMAINS setting, a list, accepts patterns. Regex cannot be used here because characters like `.`, `-` will need escaping. So I decided to settle on the simpler unix style globs. The pattern matching is done through the `fnmatch` library function, documented [here][fnmatch-docs]. One can use this setting like this: PERMITTED_DOMAINS = [ 'www.example.com', # Exactly the domain `www.example.com` '*.supertrusteddomain.com', # All subdomains to `supertrusteddomain.com` '?.services.example.com', # Single letter subdomains to `.services.example.com` ] I have done an initial implementation of these, changes in [pull request #95][pull-95]. I'll now proceed to clean these up, writing better tests and documentation for these. Also with these, we can completely get rid of the cookie based CSRF check system. -- Thanks Rohan Jain [fnmatch-docs]: http://docs.python.org/library/fnmatch.html [pull-95]: https://github.com/django/django/pull/95 On 20:10 +0530 / 21 May, Rohan Jain wrote: > Hi, > > Since my last check in I worked on improvements to > contrib.sessions: > > - Introduction of signing framework > - Session expiry checks (for Ticket [#18194][0] > - And some other trivial patches. > > The tests (existing and the one which I added) are passing. > > These changes are in my [Pull Request #78][1] over github. > Paul, could you please review it to see if the patches are usable. > > Next, I'll make the changes which may be required in documentation > because of the above. > Today is official start date of the GSoC project, so I'll now start > concentrating more on the project now. > > Rohan Jain > > [0]: https://code.djangoproject.com/ticket/18194 > [1]: https://github.com/django/django/pull/78 > > On Mon, May 7, 2012 at 12:21 PM, Rohan Jain <crod...@gmail.com> wrote: > > Hi, > > > > Last week I looked into the Ticket [#18194][0]: > > > > - Trivial attempts to handle the issue. > > - Wrote a minor initial patch. > > - The test fails for Cache and Cookie backend. > > > > Also, I looked at the talks from Paul regarding advanced security > > topics at py/django cons. Realised that why I should not attempt > > anything related to encryption in my project. > > > > There is high academic pressure currently, so I am not able to give > > enough time to these. I think the situation will be better this > > weekend onwards. > > > > I'll try to work on: > > > > - Write tests which emulate the problem in #18194 well, and then work > > on the final fix. > > - Start looking into resources useful for my project, like [The > > Tangled Web][1]. > > > > Rohan Jain > > > > > > [0]: https://code.djangoproject.com/ticket/18194 > > [1]: > > http://www.amazon.com/The-Tangled-Web-Securing-Applications/dp/1593273886 -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Improvements to contrib.sessions (Pull #78)
Hi all, Recently, I have been working on some patches for contrib.sessions. These include server side sessions expiry check (#18194) and some other trivial changes. These changes are in the pull request [#78][pull-78] The expiry checks is be done in the base backend, i.e. inherited by every backend, utilizing the timed signer from signing framework. This would mean that only session data which was signed within a duration of `settings.SESSION_COOKIE_AGE` will be valid. The format in which session data is stored is changed with this, so existing sessions are invalidated and reset. To prevent loss of existing sessions, I have provided a compatibility setting which when set to true, will run the legacy decoding mechanism on a failed session decode with new one. This does make the system susceptible to all attacks possible with the previous mechanism. I have moved the session cleanup command logic to individual backend. Cleanup for sessions data will be done on the basis of current backend setting, instead of just for the database backend. Also, the session key character set is now `a-z0-9`. -- Rohan [pull-78]: https://github.com/django/django/pull/78 -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: GSoC Check-in: Security Enhancements
Hi, Since my last check in I worked on improvements to contrib.sessions: - Introduction of signing framework - Session expiry checks (for Ticket [#18194][0] - And some other trivial patches. The tests (existing and the one which I added) are passing. These changes are in my [Pull Request #78][1] over github. Paul, could you please review it to see if the patches are usable. Next, I'll make the changes which may be required in documentation because of the above. Today is official start date of the GSoC project, so I'll now start concentrating more on the project now. Rohan Jain [0]: https://code.djangoproject.com/ticket/18194 [1]: https://github.com/django/django/pull/78 On Mon, May 7, 2012 at 12:21 PM, Rohan Jain <crod...@gmail.com> wrote: > Hi, > > Last week I looked into the Ticket [#18194][0]: > > - Trivial attempts to handle the issue. > - Wrote a minor initial patch. > - The test fails for Cache and Cookie backend. > > Also, I looked at the talks from Paul regarding advanced security > topics at py/django cons. Realised that why I should not attempt > anything related to encryption in my project. > > There is high academic pressure currently, so I am not able to give > enough time to these. I think the situation will be better this > weekend onwards. > > I'll try to work on: > > - Write tests which emulate the problem in #18194 well, and then work > on the final fix. > - Start looking into resources useful for my project, like [The > Tangled Web][1]. > > Rohan Jain > > > [0]: https://code.djangoproject.com/ticket/18194 > [1]: http://www.amazon.com/The-Tangled-Web-Securing-Applications/dp/1593273886 -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: GSoC Check-in: Security Enhancements
Hi, Last week I looked into the Ticket [#18194][0]: - Trivial attempts to handle the issue. - Wrote a minor initial patch. - The test fails for Cache and Cookie backend. Also, I looked at the talks from Paul regarding advanced security topics at py/django cons. Realised that why I should not attempt anything related to encryption in my project. There is high academic pressure currently, so I am not able to give enough time to these. I think the situation will be better this weekend onwards. I'll try to work on: - Write tests which emulate the problem in #18194 well, and then work on the final fix. - Start looking into resources useful for my project, like [The Tangled Web][1]. Rohan Jain [0]: https://code.djangoproject.com/ticket/18194 [1]: http://www.amazon.com/The-Tangled-Web-Securing-Applications/dp/1593273886 On Fri, Apr 27, 2012 at 6:54 PM, Rohan Jain <crod...@gmail.com> wrote: > Hi, > > I am Rohan Jain, a student from Indian Institute of Technology, > Kharagpur. I'll be doing a Google Summer of Code project with django > this year under the title "Security Enhancements". As the title > suggests, it has something to do with Security Enhancements: like > improvements in CSRF protection and tokenization. > > I have made some small updates to the proposal with the feedback it > got. It is under VC over here: http://gist.github.com/2203174 > There isn't a direct way to diff gists, so here are the changes I did > if somebody has already read the proposal: > > - The origin check will be an additional step to ensure a valid > request and not standalone. The conventional checks will still > exist. > > - Add some issues Luke pointed out about signing and using sessions. > > - Add info about my github fork and branches. > > What I will be doing the following week: > > - I haven't done any major contribution to django yet apart from a > tiny ticket some time ago. So, I'll be working on an ticket next > few weeks. It is related to filesystem backend of contrib.sessions, > was raised some time ago: > https://code.djangoproject.com/ticket/18194 > > - Cleanup and organize the proposal a bit more (Probably start > tracking it as the CSRF protection page - > https://code.djangoproject.com/wiki/CsrfProtection) > > (I have also appended the current proposal below in this post) > > -- > Rohan > > > Proposal > > > #Abstract > > Django is a reasonably secure framework. It provides an API and > development patterns which transparently take care of the common web > security issues. But still there are security features which need > attention. I propose to work and improved CSRF checking without any > compromises and on integration of existing work on centralized token > system. If time permits I will also attempt on integration of > django-secure. > > #Description > ##CSRF Improvements > > Cross-Origin Resource Sharing (CORS): > W3C has a working draft regarding [CORS][w3c-cors-draft], which opens > up the possibility for allowing client-side request cross-origin > requests. This directly triggers in mind the capability to develop > API which can be exposed directly to the web browser. This would let > us get rid of proxies and other hacks used to achieve this. > Currently all the major browsers support this: Chrome (all versions), > Firefox (> 3.0), IE (> 7.0), Safari (> 3.2), Opera (> 12.0). Firefox > and Chrome send the origin header for both AJAX and standard from POST > requests. Introduced it here as some further parts of the post refer > to this. > > ###Origin checking > > With CORS around need for using CSRF token can be dropped, at least in > some browsers. [Ticket #16859][orig-check-ticket], is an attempt for > that. But this was rejected because of neglecting the case for > presence of `CSRF_COOKE_DOMAIN` (Refer to the closing comment on the > ticket for details). So to handle this we need to simulate checking of > CSRF cookie domain as web browsers do it. Maybe: > > ```python > > reqest.META.get('HTTP_ORIGIN').endswith(settings.CSRF_COOKIE_DOMAIN) > > ``` > > In case the server receives an origin header in the request, it will > be used for an initial checking and then all the conventional checks > will be done. The general security will automatically be improved with > the increased market share of newer browsers which support Origin > Header. > > As the closing comment points it out, we can't do this with secure > requests. They need to be essentially checked against the referrer or > origin, at least for now. We can not be sure that some untrusted or > insecure subdomain has not already set the cookie or cookie domain. > To deal with this, we have to consider ht
GSoC Check-in: Security Enhancements
Hi, I am Rohan Jain, a student from Indian Institute of Technology, Kharagpur. I'll be doing a Google Summer of Code project with django this year under the title "Security Enhancements". As the title suggests, it has something to do with Security Enhancements: like improvements in CSRF protection and tokenization. I have made some small updates to the proposal with the feedback it got. It is under VC over here: http://gist.github.com/2203174 There isn't a direct way to diff gists, so here are the changes I did if somebody has already read the proposal: - The origin check will be an additional step to ensure a valid request and not standalone. The conventional checks will still exist. - Add some issues Luke pointed out about signing and using sessions. - Add info about my github fork and branches. What I will be doing the following week: - I haven't done any major contribution to django yet apart from a tiny ticket some time ago. So, I'll be working on an ticket next few weeks. It is related to filesystem backend of contrib.sessions, was raised some time ago: https://code.djangoproject.com/ticket/18194 - Cleanup and organize the proposal a bit more (Probably start tracking it as the CSRF protection page - https://code.djangoproject.com/wiki/CsrfProtection) (I have also appended the current proposal below in this post) -- Rohan Proposal #Abstract Django is a reasonably secure framework. It provides an API and development patterns which transparently take care of the common web security issues. But still there are security features which need attention. I propose to work and improved CSRF checking without any compromises and on integration of existing work on centralized token system. If time permits I will also attempt on integration of django-secure. #Description ##CSRF Improvements Cross-Origin Resource Sharing (CORS): W3C has a working draft regarding [CORS][w3c-cors-draft], which opens up the possibility for allowing client-side request cross-origin requests. This directly triggers in mind the capability to develop API which can be exposed directly to the web browser. This would let us get rid of proxies and other hacks used to achieve this. Currently all the major browsers support this: Chrome (all versions), Firefox (> 3.0), IE (> 7.0), Safari (> 3.2), Opera (> 12.0). Firefox and Chrome send the origin header for both AJAX and standard from POST requests. Introduced it here as some further parts of the post refer to this. ###Origin checking With CORS around need for using CSRF token can be dropped, at least in some browsers. [Ticket #16859][orig-check-ticket], is an attempt for that. But this was rejected because of neglecting the case for presence of `CSRF_COOKE_DOMAIN` (Refer to the closing comment on the ticket for details). So to handle this we need to simulate checking of CSRF cookie domain as web browsers do it. Maybe: ```python reqest.META.get('HTTP_ORIGIN').endswith(settings.CSRF_COOKIE_DOMAIN) ``` In case the server receives an origin header in the request, it will be used for an initial checking and then all the conventional checks will be done. The general security will automatically be improved with the increased market share of newer browsers which support Origin Header. As the closing comment points it out, we can't do this with secure requests. They need to be essentially checked against the referrer or origin, at least for now. We can not be sure that some untrusted or insecure subdomain has not already set the cookie or cookie domain. To deal with this, we have to consider https separately as it is being done now. So it will be something like: ```python def process_view(self, request, ): # Same initial setup if request.method not in ('GET', 'HEAD', 'OPTIONS', 'TRACE'): host = request.get_host() origin = reqest.META.get('HTTP_ORIGIN', "") cookie_domain = settings.CSRF_COOKIE_DOMAIN if request.is_secure(): good_referer = 'https://%s/' % host referer = origin or request.META.get('HTTP_REFERER') # Do the same origin checks here # We are insecure, so care less # A better way for this check can be used if needed elif origin.endswith(cookie_domain): # Safe, continue conventional checking # Do the conventional checks here ``` If the above were to be implemented, the setting `CSRF_COOKIE_DOMAIN` should be deprecated for something like `CSRF_ALLOWED_DOMAIN` which makes more sense. ###Multiple Allowed Domains (was Better CORS Support) Since, already introducing Origin checking, we can go one step further and try to provide better support for CORS for browsers supporting it. A tuple/list setting, which specifies allowed domains will be provided. Using this the various access control allowance response headers will be set when the request origin is from amongst the allow
Re: GSoC 2012: Security Enhancements
On 16:03 +0100 / 18 Apr, Luke Plant wrote: > On 15/04/12 05:23, Rohan Jain wrote: > > On 22:50 +0100 / 13 Apr, Luke Plant wrote: > >> The reason for the strict referer checking under HTTPS is set out here: > >> > >> https://code.djangoproject.com/wiki/CsrfProtection > >> > >> Particularly, it is to fix the 'CSRF + MITM' attack that is possible > >> under HTTPS. The key elements are set out in the following scenario > >> (although it is not the only variation): > >> > >> - a client connects to a site via HTTP: http://example.com/ > >> - in the response, an active MITM attacker sets a cookie over > >>HTTP for example.com > >> - this cookie will be used by the client when it connects over HTTPS > >>to the same domain (this is the fundamental problem, but we can't > >>fix it - it's what browsers do). > >> - the MITM also inserts a POST form in the HTTP response. > >>The form has a CSRF token that matches the cookie that was > >>set by the attacker. > >>The forms targets https://example.com/ and is automatically > >>submitted by javascript. > >> > >> Without strict referer checking, the POST request will succeed, even > >> though it is forged. > >> > >> Signing the cookie or token value does no good at all, because the > >> attacker can retrieve a valid cookie/token pair directly from > >> https://example.com/. > > > > I am a bit confused about this. How can an attacker extract the token > > out of the signed cookie without the private key? There is a > > possibility that the attacker deletes and sets the cookie right out, > > but since that case won't be correctly signed server can handle it > > securely. > > They don't need to extract the token, they just need to replay an > existing good token/cookie pair, which they can get directly from the > server any time they want. > > (Also, signing does nothing to hide the token. Are you actually talking > about encryption here? But even if you encrypt it, it doesn't help - the > only thing that matters is that the CSRF form token 'matches' the CSRF > cookie, whatever your definition of 'match', and that can be achieved by > getting the pair from the server.) Sorry, my bad. I didn't realize swaying towards encryption. Encryption is something the SoC Ideas page suggests against for CSRF. I am no crypto expert so shouldn't waste time over this I guess. -- Rohan -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: GSoC 2012: Security Enhancements
I hosted a simple app which responds with the request details for testing purposes: https://request-mirror.herokuapp.com/ (source: https://github.com/crodjer/request-mirror) On 12:05 -0700 / 18 Apr, Paul McMillan wrote: > There seems to be some confusion about CORS (a hairy draft spec that > is not fully implemented in any browser, and not appropriate for > inclusion in Django at this time) and the "Origin" header (aka Web > Origin, rfc6454). > > http://tools.ietf.org/html/rfc6454 > https://wiki.mozilla.org/Security/Origin > http://www.w3.org/TR/access-control/#origin-request-header > > The Origin header defined in rfc6454 is compatible with the CORS > origin header, but does not require full CORS support from the browser > or the server to be useful. > > In my tests with Firefox and Chrome, both sent the origin header > properly when making potentially state-changing cross-origin requests > (both AJAX and standard form POST requests). > How did you test the form POST requests with firefox? For me, with a normal POST, Iceweasel (firefox for debian) 10 does not seem to be sending any origin headers but Chrome (18) is. MDN says that this should work firefox >= 3.5 (https://developer.mozilla.org/en/http_access_control#The_HTTP_request_headers) and the Origin header should be present: https://wiki.mozilla.org/Security/Origin#Origin_header_format I tried testing these cross domain (between the secure remote and local, between local aliases) through HTTP Post, but the browser never complained or did a Preflighted request (HTTP OPTIONS request) or sent back the origin header. But it apparently isn't with mine (Hopefully I am doing something wrong here) > My suggestion here is to include optional support for the Origin > header as follows: > - if present and null, fail the CSRF check > - if present and not null, use in alongside the Referer header > - if absent, keep current behavior > > As a general rule, if a browser sends an origin header, that value is > more reliable (harder for malicious sites to manipulate, less often > stripped by firewalls, less often disabled by users) than the referer > header. This addition won't improve CSRF protection for older > browsers, but it also won't break anything for them. For users with > newer browsers, it should prevent CSRF even in cases when the CSRF > token is stolen due to misconfiguration or user error. > > -Paul I second this. The selective origin checking, though not completely, will improve CSRF protection for some clients. We can then be sure about an automatic increase in security with the shift towards such clients. -- Rohan -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: GSoC 2012: Security Enhancements
On 22:50 +0100 / 13 Apr, Luke Plant wrote: > Hi Rohan, > > Sorry for the slow reply on this one, I've had a busy time recently. > Please see my comments on some parts of this proposal. No worries about this. > > On 31/03/12 19:10, Rohan Jain wrote: > > Hi, > > > > I am Rohan Jain, a 4th (final) year B.Tech undergraduate Student from > > Indian Institute of Technology, Kharagpur. I have been using django > > since over a year and generally look into the code base to find about > > various implementations. I have made attempts to make some minor > > contributions and if selected this would be my first major one. > > > > More about Me: <http://www.rohanjain.in/about/> IRC, Github: crodjer > > > > I am interested in contributing some security enhancements to django > > as my Summer of Code project. Below is the 1st draft of my proposal > > regarding this. A pretty version of this is available at: > > https://gist.github.com/2203174 > > > > > > #Abstract > > > > Django is a reasonably secure framework. It provides an API and > > development patterns which transparently take care of the common web > > security issues. But still there are security features which need > > attention. I propose to work on integration of existing work on > > centralized token system and improved CSRF checking without any > > compromises. If time permits I will also attempt on integration of > > django-secure. > > > > #Description ##Centralized tokenization There are multiple places in > > django which use some or other kinds of tokens: > > > > - contirb.auth (random password, password reset) - formtools - > > session (backends) - cache - csrf - etags > > > > Token generation is pretty common around the framework. So, instead > > of each application having its own token system, and hence needs to > > be maintained separately. There should be centralized token system, > > which provides an abstract API for everyone to loose. In fact, I have > > seen that some apps use `User.objects.make_random_password` from > > contrib.auth, which they can be sure of being maintained in the > > future for random generation. To me this looks kind of weird. In last > > djangocon, a lot of work regarding this was done over [Yarko's > > Fork][yarko-fork]. > > > > I had a discussion with Yarko Tymciurak regarding this. The work is > > nearly ready for a merge, only some tasks left. In the initial > > period my SoC I can work over these to insure that the already done > > significant work gets in django and is updated for 1.5. > > > > - Porting more stuff to the new system (README.sec in [yarko's > > fork][yarko-fork]) - Testing - See if the current coverage of the > > tests is enough, write them if not. - Compatibility issues - API > > Documentation > > > > I will study the changes done at djangocon and then attempt the > > tasks mentioned above. > > > > ##CSRF Improvements > > > > Cross-Origin Resource Sharing (CORS): W3C has a working draft > > regarding [CORS][w3c-cors-draft], which opens up the possibility for > > allowing client-side request cross-origin requests. This directly > > triggers in mind the capability to develop API which can be exposed > > directly to the web browser. This would let us get rid of proxies and > > other hacks used to achieve this. Currently all the major browsers > > support this: Chrome (all versions), Firefox (> 3.0), IE (> 7.0), > > Safari (> 3.2), Opera (> 12.0). Introduced it here as some further > > parts of the post refer to this. > > > > ###Origin checking > > > > With CORS around need for using CSRF token can be dropped, at least > > in some browsers. [Ticket #16859][orig-check-ticket], is an attempt > > for that. But this was rejected because of neglecting the case for > > presence of `CSRF_COOKE_DOMAIN` (Refer to the closing comment on the > > ticket for details). So to handle this we need to simulate checking > > of CSRF cookie domain as web browsers do it. Maybe: > > > > ```python > > reqest.META.get('HTTP_ORIGIN').endswith(settings.CSRF_COOKIE_DOMAIN) > > ``` > > I'm very cautious about making the logic here more complex. It can be > done, but every additional code path increases the possibility of a > security hole. .. Yes, this will touch critical code of the framework and cannot afford to expose any possible vulnerabilities. That is why I am planning of a thorough security level testing and of course any patch relating this (or anything else in this se
Re: GSoC 2012: Security Enhancements
Hi Russel, That is a good news for me. I have added a timeline and posted it over melange. Public Gist for the same: https://gist.github.com/2203174 -- Rohan On 16:14 +0800 / 6 Apr, Russell Keith-Magee wrote: > > On 06/04/2012, at 3:54 PM, Rohan Jain wrote: > > > Hi Russel, > > > > Thanks for the reply. > > > > On 14:42 +0800 / 6 Apr, Russell Keith-Magee wrote: > >> > >> Hi Rohan, > >> > >> Apologies for the lack of response. Anyone who has put effort into writing > >> up a proposal certainly deserves a response of some kind, so we've dropped > >> the ball here. > >> > >> In our defence, here's a couple of the reasons why your proposal probably > >> hasn't got a wild response: > >> > >> * You've picked a project on your own, rather than one that is on the list > >> of suggested projects. Part of the GSoC process is mentoring, and you need > >> to have a mentor that can spare the time, and has the technical skills to > >> review your work. Most of the projects on the suggested list already have > >> champions inside the community, so if you pick one of them, you're likely > >> to get a response. If you pick your own project, you also have to find > >> someone to get sufficiently enthused about it. > > > > I had picked the project from the list of suggested projects > > (https://code.djangoproject.com/wiki/SummerOfCode2012#SecurityEnhancements). > > In fact, I have tried to take on each of the issues mentioned there, > > point by point. I did talk to Paul over the IRC about this. > > Ah. Now I look a little stupid :-) > > That project has evidently been added to the list since I last looked at the > list. Apologies for not keeping up to date. > > >> Regarding your project proposal itself: I can't really address the > >> technical merits, because I don't have any expertise on CORS, or the > >> subtleties of the CSRF changes your proposing. What you've proposed > >> certainly sounds interesting on the surface, but I'd really want to see > >> someone like Paul McMillan comment on the technical specifics. Ideally, > >> Paul would also mentor the project, since he's Django's resident security > >> expert, and he'd need to sign off on anything that was bound for trunk. > > > > I had some discussions with Paul over this earlier, but since a week > > days I haven't been able to find him online. > > It's good that you've discussed this with Paul -- he's certainly the man to > speak to. > > >> So - apologies for not responding sooner. Unfortunately, I suspect that > >> while your project probably has merit, the community isn't in a position > >> to support your ambition at the moment. If Paul wants to swoop in at the > >> last minute and prove me wrong, I'd be a very happy man -- I hate seeing > >> someone enthusiastic get turned away -- but absent of that, it's only fair > >> that we be honest to you about your chances. > > > > No worries. I am at a blame too for my own activity. I myself have > > learnt a lot of interesting stuff from this. I would anyway be glad to > > give something back to django. > > The fact that this project *is* in fact on our suggested projects list means > I probably have to alter my advice. > > If you're still interested in doing this project, it's worth submitting it. > You've clearly put in a bunch of effort already, and it would be a pity for > that to go to waste. As an organization, we have a certain amount of > discretion as to the projects we pick, and if someone is clearly working with > the community to improve API or sort out fine details, we may be able to > overlook small problems with an "official" application. However, the > "official" application still needs to be submitted in order for the GSoC > process to go ahead. What you've got here looks solid enough to demonstrate > you've thought about the problem and have an good idea of how you're going to > solve it. > > That said, ultimate selection of the project will probably have more to do > with Paul's availability and enthusiasm to mentor than the technical aspects > of your proposal. > > All I would suggest is: > > * Add a detailed work breakdown (granularity ~1 week) > > * Push the "merging" activities to the status of "stuff I'll do if the > schedule allows at the end of the project". That removes the emphasis from > the merging activity, making it look less suspect from the GSoC process > point-of-view. > > * Su
Re: GSoC 2012: Security Enhancements
Hi Russel, Thanks for the reply. On 14:42 +0800 / 6 Apr, Russell Keith-Magee wrote: > > Hi Rohan, > > Apologies for the lack of response. Anyone who has put effort into writing up > a proposal certainly deserves a response of some kind, so we've dropped the > ball here. > > In our defence, here's a couple of the reasons why your proposal probably > hasn't got a wild response: > > * You've picked a project on your own, rather than one that is on the list > of suggested projects. Part of the GSoC process is mentoring, and you need to > have a mentor that can spare the time, and has the technical skills to review > your work. Most of the projects on the suggested list already have champions > inside the community, so if you pick one of them, you're likely to get a > response. If you pick your own project, you also have to find someone to get > sufficiently enthused about it. I had picked the project from the list of suggested projects (https://code.djangoproject.com/wiki/SummerOfCode2012#SecurityEnhancements). In fact, I have tried to take on each of the issues mentioned there, point by point. I did talk to Paul over the IRC about this. > > * You've picked a very gnarly problem. Security issues are the very model of > an 'anti-bikeshed'. If you read the original discussion about Bikesheds [1], > it's all about how everyone gives their opinion on "simple" topics, but > everyone leaves hard problems alone. In Django's context -- *everyone* has an > opinion about contrib.auth.User because it seems like a simple problem. > However, security is all about subtle issues and expert knoweledge. > Therefore, your pool of experts is much smaller. > > [1] http://bikeshed.com/ I recently generated some interest in this field, so picked up what I would like to work on the best. For being in field of web development, I believe it is necessary to have a good understanding of security. > > Regarding your project proposal itself: I can't really address the technical > merits, because I don't have any expertise on CORS, or the subtleties of the > CSRF changes your proposing. What you've proposed certainly sounds > interesting on the surface, but I'd really want to see someone like Paul > McMillan comment on the technical specifics. Ideally, Paul would also mentor > the project, since he's Django's resident security expert, and he'd need to > sign off on anything that was bound for trunk. I had some discussions with Paul over this earlier, but since a week days I haven't been able to find him online. > > What I can do is point at the things that look like problems from a GSoC > perspective. In particular, of the three sections to your project plan, two > of them (unified tokenization and django-secure) involve merging existing > projects into trunk. This is problematic, because one of the conditions of > GSoC is that the student writes the bulk of the code. While integrating these > two code pools may well be very valuable contributions to Django, they're not > good from the perspective of a GSoC project. Yes, these two will be more calling out to my `git merge` skills. I was assuming the most of my time would have spanned over CSRF improvements, while using some functionality from centralized token generation. > > So - apologies for not responding sooner. Unfortunately, I suspect that while > your project probably has merit, the community isn't in a position to support > your ambition at the moment. If Paul wants to swoop in at the last minute and > prove me wrong, I'd be a very happy man -- I hate seeing someone enthusiastic > get turned away -- but absent of that, it's only fair that we be honest to > you about your chances. No worries. I am at a blame too for my own activity. I myself have learnt a lot of interesting stuff from this. I would anyway be glad to give something back to django. -- Rohan > > Yours, > Russ Magee %-) > > On 06/04/2012, at 2:09 PM, Rohan Jain wrote: > > > Hi again, > > > > I really couldn't understand the response this post has got. It > > deserved at least a little feedback, positive or negative. I guess I > > wont be submitting this over melange. > > > > Still, I have put some effort and research in the proposal. So if > > possible I would like to know if it had anything of value. Maybe some > > one could work over that, even me if I get the time. > > > > -- > > Rohan > > > > > > On 23:40 +0530 / 31 Mar, Rohan Jain wrote: > >> Hi, > >> > >> I am Rohan Jain, a 4th (final) year B.Tech undergraduate Student > >> from Indian Institute of Technology, Kharagpur. I have been using > >> django since over a year and genera
Re: GSoC 2012: Security Enhancements
Hi again, I really couldn't understand the response this post has got. It deserved at least a little feedback, positive or negative. I guess I wont be submitting this over melange. Still, I have put some effort and research in the proposal. So if possible I would like to know if it had anything of value. Maybe some one could work over that, even me if I get the time. -- Rohan On 23:40 +0530 / 31 Mar, Rohan Jain wrote: > Hi, > > I am Rohan Jain, a 4th (final) year B.Tech undergraduate Student > from Indian Institute of Technology, Kharagpur. I have been using > django since over a year and generally look into the code base to find > about various implementations. I have made attempts to make some minor > contributions and if selected this would be my first major one. > > More about Me: <http://www.rohanjain.in/about/> > IRC, Github: crodjer > > I am interested in contributing some security enhancements to django > as my Summer of Code project. Below is the 1st draft of my proposal > regarding this. A pretty version of this is available at: > https://gist.github.com/2203174 > > > #Abstract > > Django is a reasonably secure framework. It provides an API and > development patterns which transparently take care of the common web > security issues. But still there are security features which need > attention. I propose to work on integration of existing work on > centralized token system and improved CSRF checking without any > compromises. If time permits I will also attempt on integration of > django-secure. > > #Description > ##Centralized tokenization > There are multiple places in django which use some or other kinds of > tokens: > > - contirb.auth (random password, password reset) > - formtools > - session (backends) > - cache > - csrf > - etags > > Token generation is pretty common around the framework. So, instead > of each application having its own token system, and hence needs to be > maintained separately. There should be centralized token system, which > provides an abstract API for everyone to loose. In fact, I have seen > that some apps use `User.objects.make_random_password` from > contrib.auth, which they can be sure of being maintained in the future > for random generation. To me this looks kind of weird. > In last djangocon, a lot of work regarding this was done over [Yarko's > Fork][yarko-fork]. > > I had a discussion with Yarko Tymciurak regarding this. The work is > nearly ready for a merge, only some tasks left. In the initial period > my SoC I can work over these to insure that the already done > significant work gets in django and is updated for 1.5. > > - Porting more stuff to the new system (README.sec in >[yarko's fork][yarko-fork]) > - Testing - See if the current coverage of the tests is enough, write >them if not. > - Compatibility issues > - API Documentation > > I will study the changes done at djangocon and then attempt the tasks > mentioned above. > > ##CSRF Improvements > > Cross-Origin Resource Sharing (CORS): > W3C has a working draft regarding [CORS][w3c-cors-draft], which opens > up the possibility for allowing client-side request cross-origin > requests. This directly triggers in mind the capability to develop > API which can be exposed directly to the web browser. This would let > us get rid of proxies and other hacks used to achieve this. > Currently all the major browsers support this: Chrome (all versions), > Firefox (> 3.0), IE (> 7.0), Safari (> 3.2), Opera (> 12.0). > Introduced it here as some further parts of the post refer to this. > > ###Origin checking > > With CORS around need for using CSRF token can be dropped, at least in > some browsers. [Ticket #16859][orig-check-ticket], is an attempt for > that. But this was rejected because of neglecting the case for > presence of `CSRF_COOKE_DOMAIN` (Refer to the closing comment on the > ticket for details). So to handle this we need to simulate checking of > CSRF cookie domain as web browsers do it. Maybe: > > ```python > reqest.META.get('HTTP_ORIGIN').endswith(settings.CSRF_COOKIE_DOMAIN) > ``` > > As the closing comment points it out, we can't do this with secure > requests. They need to be essentially checked against the referrer or > origin, at least for now. We can not be sure that some untrusted or > insecure subdomain has not already set the cookie or cookie domain. > d > To deal with this, we have to consider https separately as it is > being done now. So it will be something like: > > > ```python > def process_view(self, request, ): > > # Same initial setup > > if request.method not in ('GET', 'HEAD', 'OPTIONS', 'TRACE')
GSoC 2012: Security Enhancements
Hi, I am Rohan Jain, a 4th (final) year B.Tech undergraduate Student from Indian Institute of Technology, Kharagpur. I have been using django since over a year and generally look into the code base to find about various implementations. I have made attempts to make some minor contributions and if selected this would be my first major one. More about Me: <http://www.rohanjain.in/about/> IRC, Github: crodjer I am interested in contributing some security enhancements to django as my Summer of Code project. Below is the 1st draft of my proposal regarding this. A pretty version of this is available at: https://gist.github.com/2203174 #Abstract Django is a reasonably secure framework. It provides an API and development patterns which transparently take care of the common web security issues. But still there are security features which need attention. I propose to work on integration of existing work on centralized token system and improved CSRF checking without any compromises. If time permits I will also attempt on integration of django-secure. #Description ##Centralized tokenization There are multiple places in django which use some or other kinds of tokens: - contirb.auth (random password, password reset) - formtools - session (backends) - cache - csrf - etags Token generation is pretty common around the framework. So, instead of each application having its own token system, and hence needs to be maintained separately. There should be centralized token system, which provides an abstract API for everyone to loose. In fact, I have seen that some apps use `User.objects.make_random_password` from contrib.auth, which they can be sure of being maintained in the future for random generation. To me this looks kind of weird. In last djangocon, a lot of work regarding this was done over [Yarko's Fork][yarko-fork]. I had a discussion with Yarko Tymciurak regarding this. The work is nearly ready for a merge, only some tasks left. In the initial period my SoC I can work over these to insure that the already done significant work gets in django and is updated for 1.5. - Porting more stuff to the new system (README.sec in [yarko's fork][yarko-fork]) - Testing - See if the current coverage of the tests is enough, write them if not. - Compatibility issues - API Documentation I will study the changes done at djangocon and then attempt the tasks mentioned above. ##CSRF Improvements Cross-Origin Resource Sharing (CORS): W3C has a working draft regarding [CORS][w3c-cors-draft], which opens up the possibility for allowing client-side request cross-origin requests. This directly triggers in mind the capability to develop API which can be exposed directly to the web browser. This would let us get rid of proxies and other hacks used to achieve this. Currently all the major browsers support this: Chrome (all versions), Firefox (> 3.0), IE (> 7.0), Safari (> 3.2), Opera (> 12.0). Introduced it here as some further parts of the post refer to this. ###Origin checking With CORS around need for using CSRF token can be dropped, at least in some browsers. [Ticket #16859][orig-check-ticket], is an attempt for that. But this was rejected because of neglecting the case for presence of `CSRF_COOKE_DOMAIN` (Refer to the closing comment on the ticket for details). So to handle this we need to simulate checking of CSRF cookie domain as web browsers do it. Maybe: ```python reqest.META.get('HTTP_ORIGIN').endswith(settings.CSRF_COOKIE_DOMAIN) ``` As the closing comment points it out, we can't do this with secure requests. They need to be essentially checked against the referrer or origin, at least for now. We can not be sure that some untrusted or insecure subdomain has not already set the cookie or cookie domain. d To deal with this, we have to consider https separately as it is being done now. So it will be something like: ```python def process_view(self, request, ): # Same initial setup if request.method not in ('GET', 'HEAD', 'OPTIONS', 'TRACE'): host = request.get_host() origin = reqest.META.get('HTTP_ORIGIN', "") cookie_domain = settings.CSRF_COOKIE_DOMAIN if request.is_secure(): good_referer = 'https://%s/' % host referer = origin or request.META.get('HTTP_REFERER') # Do the same origin checks here # We are insecure, so care less # A better way for this check can be used if needed elif origin.endswith(cookie_domain): # Safe, accept request # Some unsupported browser # Do the conventional checks here ``` If the above were to be implemented, the setting `CSRF_COOKIE_DOMAIN` should be deprecated for something like `CSRF_ALLOWED_DOMAIN` which makes more sense. I would also suggest making CSRF cookie as http only. There doesn't seem a reason currently why the cookies would be needed to be accessed in browser. ###Less restrictive secure reque
Enhanced auth.User
These are some auth settings and models I propose to account for the generally raised questions for auth.User flexibility. Settings: - AUTH_USER_EMAIL_UNIQUE If the email should be unique for users. It is a rare case when a website would have users sharing emails. It is more likely other way round that a user has multiple emails. - AUTH_USERNAME_MIN_LENGTH/AUTH_USERNAME_MAX_LENGTH Min and max lengths of the username field. - AUTH_USE_OLD_USER_MODLE If the old style (fn-ln based) user model should be used. Maybe a deprication warning can be provided when this is set to True. Models: - BaseUser Model ~ Current User Model, abstract There will be a base model with all the same fields as current user model but not the first_name and last_name fields. - UNIQUE_EMAIL = getattr(settings,'AUTH_USER_EMAIL_UNIQUE', False) email = EmailField(blank=not UNIQUE_EMAIL, unique=UNIQUE_EMAIL) email column unique/no unique according to AUTH_USER_EMAIL_UNIQUE setting. - A method can be defined which could generate random username while registering if required, like in case of logging in with email, openid, oauth etc. - Provision for authentication through email id can easily be provided if UNIQUE_EMAIL is set. - EnhancedUser, abstract Inherits from BaseUser. Have a field `name` instead of old `first_name`, `last_name`. A single field for name is preferable as it would be more comfortable to users and would also take care of case when the User model doesn't represents an individual but may be a team, an organization etc. Also a text field `description/about` could be provided as it is generally present in basic user profiles everywhere. This would prevent necessity to use a separate Profile model. These fields would result in some minor changes in places like auth.models and auth.admin. The first_name and last_name fields are generally accessed through get_full_name method which could be redefined to return `name`. - OldUser, abstract This with BaseUser will generate the current User model with `first_name`, `last_name` fields and the get_full_name method. In case of an old project, with large database, upgrading to newer django versions. - User This would inherit from EnhancedUser or OldUser according to the AUTH_USE_OLD_USER_MODLE setting and will be the model exposed to other apps, hence there should not be any compatibility issues. Now as far as old projects upgradation to new User model is concerned, that could be done with the help of application 'south'. A migration scripts can be written: 1. Create a column `name`. 2. Create datamigration script to copy `first_name` + `last_name` to `name`. 3. Drop `first_name` and `last_name` column. About Me: I am a 3rd year Engineering undergraduate student from the Department of Agricultural and Food Engineering, Indian Institue of Technology, Kharagpur. I have been using django for a year now and recently started working with the SVN version, hacking the codes of contrib apps and trying to learn the best practices. Website: http://www.rohanjain.in GitHub: http://github.com/crodjer -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: One Django instance, hundreds of websites
I am also trying to achieve something highly similar to this but in a dilemma, for how to proceed. I have written a post about this: http://www.rohanjain.in/blog/hosting-multiple-sites-with-same-django-project/. Is there any existing big project following a similar concept? -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.