Re: Django Oracle backend vs. numbers

[Shai Berger]

Hi Ian, Jacob, Anssi and all,

Thanks for the quick and positive responses.

On Wednesday 29 August 2012, Ian Kelly wrote:
>https://github.com/ikelly/django/commit/086f502a1c2acb9db27531f7df78cb9df9ab83bd
> 
> Shai, if you would be willing to try this patch, I'd like to hear your
> results.  Please bear in mind that this is currently only a POC, so it
> could be buggy -- I've only run it through the basic tests.
> 

This looks very promising, I will try it next week. On a first look, there is 
just one thing about it that I don't like, and it is related to Jacob's 
comments --

On Wed, Aug 29, 2012 at 10:34 AM, Jacob Kaplan-Moss  
wrote:
> I'm generally adverse to introducing more settings -- settings creep
> is a real problem -- so a couple of alternate ideas you might consider
> in producing a patch:
> 
> 1. Just pick one correct behavior, probably for each type. My guess
> would be that ints and floats should get the fast behavior, and
> decimals the slow-but-correct one. If you're using a decimal field
> it's because you care about precision, so correctness makes more sense
> there IMO.
> 
This would definitely be preferrable to a setting, but it doesn't cover all 
cases -- we have some raw queries where expressions are being selected. The 
default number type in this case -- in current code as well as Ian's patch -- 
is to return the data as a string and convert it to int or Decimal. Our code, 
which was written against other database systems, tries to use the result in 
arithmetic with floats, and this fails (you can't add Decimals to floats).

I could, of course, prevent the exception by converting the value to float in 
my code, but that feels like the wrong solution -- I know, in my case, that 
float is enough, and I'd much rather tell the backend "give me floats" than 
have 
it go through float->str->Decimal->float.

> 2. If you must introduce a setting, do it in DATABASES[OPTIONS]
> instead of a top-level ORACLE_FAST_NUMBERS or whatever.

That is indeed one good option. The alternative I'm considering -- less 
convenient for me, but perhaps more flexible generally -- is, since we're 
dealing with raw SQL anyway, to put some API for this on the cursor; perhaps a 
property, so that setting it would be a no-op on other backends.

As per Anssi's comment, I have no backwards-compatibility requirements with 
respect to cx_Oracle, but we are still running Django 1.3. Trying to call the 
performance improvement a "bugfix" so it can be backported, while changing 
cx_Oracle requirements, would be pushing my luck, right?

Thanks again,

Shai.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Python 3: should we apply unicode_literals everywhere?

[Felipe Prenholato]

Here at PDG (Brazil) we are migrating our software to Djang 1.4 and already
using unicode_literals. I can count in my fingers places that I needed to
use 'b' for byte code string (most on settings.py).

In my experience, maintain byte code strings isn't that hard and we should
than go to option 2.

Felipe 'chronos' Prenholato.
Linux User nº 405489
Home page: http://devwithpassion.com | http://chronosbox.org/blog
GitHub: http://github.com/chronossc/ | Twitter: http://twitter.com/chronossc


2012/8/24 Vinay Sajip 

> I would also prefer Option 2, as the places where str('...') are needed
> are not all that many.
>
> Regards,
>
> Vinay Sajip
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/django-developers/-/WLtnInRyKyAJ.
>
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: GSoC Check-in: Security Enhancements

[Rohan Jain]

Hi,

Ref Merging Centralized Tokenization.
I found out that some tests for sessions are failing with it. Stupidly,
I didn't have sessions in my test subset for tokenization. Anyway,
I'll try to get it fixed and write some tests for tokens in next few
days.

--
Thanks
Rohan

On 10:50 +0100 / 21 Aug, Andrew Godwin wrote:
> Thanks for your work during the GSOC, Rohan - don't worry about not
> achieving everything, it looks like there's still some useful code there!
> 
> Hopefully we can get some of the code merged, especially centralised
> tokenisation if it's so near completion, as it looks like a nice bit of
> cleanup code!
> 
> Andrew
> 
> On Mon, Aug 20, 2012 at 2:49 PM, Rohan Jain  wrote:
> 
> > Hi,
> >
> > Today is the 'pencils down' date for this GSoC project. Past 4 months
> > have been a great learning experience from me. Just being in the
> > context of security side of the web has been really beneficial. Moving
> > around in a very well written code base is also delightful.
> > Meanwhile, I did get to work on multiple sections of the code base.
> > But compared to the expectations at the initial phase of the program,
> > I myself am a little disappointed about the output I was able to
> > generate out of the time - specially regarding CSRF.
> > This can be attributed to my being unable to give consistent attention
> > to the project through the duration and to solicit feedback. Also, the
> > project comprised of multiple discreet tasks, so not enough
> > consideration was there for each task.
> >
> > This is a overview of various complete/incomplete topics I worked on.
> >
> > Centralized Tokenization:
> > This was about integrating a central tokenization api throughout the
> > django project. I tracked this at [centralized-tokenization][0] branch
> > of my fork on github. This is near completion, and also has a
> > documentation[1] of the API with basic usage examples.
> >
> > Improvements to sessions:
> > My first task, related to some improvements to the sessions framework.
> > Includes signing and session cleanup based on the engine. Tracked at
> > [sessions-improvements][2] branch.
> >
> > CSRF Enhancements:
> > It occupied most of my time through the project. I tried multiple ways
> > to solve the issues under this. Tracked at [csrf-enhancements][3].
> > First up, I started with using the origin header for doing a CSRF
> > check. One implementation with some tests is at
> > [csrf-origin-checking][4]. Recently I started on the idea of
> > implementing moduler checkers for each kind of CSRF check, but
> > haven't got anything useful out of it yet. While progressing, it
> > seemed like I was virtually writing a middleware per checker, so now I
> > have moved on to attempt on CSRF cookie store. Basically something
> > which should seamlessly switch Session/Cookie for storage based on the
> > availability of session.
> >
> > I'll try to hang around the django bug tracker and submit some
> > patches.
> >
> > --
> > Thanks
> > Rohan Jain
> >
> > [0]: https://github.com/crodjer/django/tree/centralized-tokenization
> > [1]:
> > https://github.com/crodjer/django/blob/centralized-tokenization/docs/topics/tokenization.txt
> > [2]: https://github.com/crodjer/django/tree/sessions-improvements
> > [3]: https://github.com/crodjer/django/tree/csrf-enhancements
> > [4]: https://github.com/crodjer/django/tree/csrf-origin-checking
> > [5]: https://github.com/crodjer/django/tree/csrf-moduler-checkers
> >
> > On 01:57 +0530 /  7 Aug, Rohan Jain wrote:
> > > Hi,
> > >
> > > Sorry for the delay in getting back. I was meanwhile working on
> > > centralized tokenization for few days, while still trying to figure
> > > something better for CSRF.
> > >
> > > On 03:52 -0400 / 25 Jul, Alex Ogier wrote:
> > > > On Tue, Jul 24, 2012 at 11:37 PM, Rohan Jain 
> > wrote:
> > > > >
> > > > > I had one more idea, "Pluggable CSRF checkers".
> > > > >
> > > > > Currently, the CSRF middleware has two kinds of checks, referer (for
> > > > > https) and secret validation token (common). These with origin header
> > > > > based checker (if we add it) come in conditional blocks, making
> > > > > switching them difficult. So what I propose to do is decouple their
> > > > > logic from CSRF middleware and each of them provide a checker. It
> > goes
> > > > > like this:
> > > > >
> > > > > A setting for configuring global CSRF checkers:
> > > > >
> > > > > CSRF_CHECKERS = {
> > > > > 'django.middleware.csrf.checkers.OriginChecker',
> > > > > # This one can be strict for https and lax for http
> > > > > 'django.middleware.csrf.checkers.RefererChecker',
> > > > > # contrib.sessions could provide a csrf checker maintained
> > > > > # with sessions. This stores the token in session data.
> > > > > 'django.contrib.sessions.csrf_checkers.SessionChecker'
> > > > > }
> > > > >
> > > >
> > > > I don't think this is a good idea. If you enumerate security 

RawQuerySet as subquery when used with an __in filter

[Alex Hill]

Hi, and thanks for making Django.

At the moment, when passing a RawQuerySet to an __in filter, the raw query 
is evaluated and the result passed as a parameter to the filter's SQL. It 
would be nice to have the option of using the raw SQL in a subquery 
instead, for performance reasons.

I think in this case the rationale for accepting 
https://code.djangoproject.com/ticket/14733 applies. Raw queries already 
require the developer to know what they're doing - this just makes them a 
little more useful.

I can see two ways of implementing this. The first is simple: just require 
that the query only select one column, and implement _as_sql on RawQuerySet 
to simply return the SQL and params. A flag passed to raw() could let a 
developer explicitly indicate that the query should be used as a subquery 
instead of evaluated.

The second way is more complicated but nicer: wrap the raw SQL in another 
SELECT, selecting the Model's primary key. The advantage of this method is 
that the raw query can select many columns, and therefore remain useful in 
other contexts, while still working as expected with __in. This could then 
be the default behaviour, with the current behaviour achieved by wrapping 
the query in list() as is the case for regular queries.

I much prefer the second approach, but I don't think I'm familiar enough 
with the ORM to implement it without a little guidance from someone more 
knowledgeable.

I'd love to hear some feedback on this idea!

Cheers,
Alex

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/zfVpJLrs7XsJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Yak shaving the test framework on the way to pluggable user models (#3011)

[Roger Barnes]

On Wednesday, August 29, 2012 6:04:09 PM UTC+10, Alex Ogier wrote:
>
> On Wed, Aug 29, 2012 at 3:44 AM, Russell Keith-Magee 
>  wrote: 
> > 
> > I suppose you could see it as a semantic nuance. However, to my mind, 
> > there is a different. A skipped test is something that could -- or 
> > even *should* be run -- but can't due to missing some optional 
> > prerequisite. In this case, we're talking about tests that can't ever 
> > be run.  To my mind, it doesn't make sense to have those tests present 
> > but "skipped". 
>
> I'm not sure I see the difference between a configuration that makes a 
> test unnecessary and a missing optional dependency that makes a test 
> unnecessary. 
>

I recently had a similar need to flag/mark/annotate tests for the purpose 
of separation of unit, smoke, functional etc.

My solution involved a kind of hacked up test runner (and django-jenkins) 
and an annotation for flagging tests.

In this case, the distinction between skipped and not run is an important 
one, IMHO.  If I have a test marked "unit" and "flaky" and another test 
marked "functional", I want my "unit, not-flaky" run to _skip_ the first 
(matches "unit", but also matches a "not" condition), but _ignore_ the 
second (no match).  That way my unit test counts don't show a whole load of 
irrelevant skipped tests. In other words, they're simply not candidates.

Some more details of my (work in progress) approach to test separation are 
midway through the following presentation (slides 24-27)...
http://www.slideshare.net/mindsocket/towards-continuous-deployment-with-django

Since that talk a whole 2 weeks ago, I've become aware of django-nose and 
I'm looking at the possibilities there now.

I hope this provides some useful context to the discussion, it may only be 
of relevance to determining how many yaks to shave.

- Roger Barnes

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/-Z5tKkOTGPYJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Django Oracle backend vs. numbers

[Anssi Kääriäinen]

On 29 elo, 22:18, Ian Kelly  wrote:
> There's another option that also hasn't been implemented, because it
> would mean dropping support for cx_Oracle versions older than 5.0.1.
> Perhaps the time has come to do that (I note that the oldest binary
> still listed at the cx_Oracle website is 5.1).  This would involve
> removing the _rowhandler function and the numbers_as_strings setting
> and replacing it with an _outputtypehandler that would only be called
> once per column instead of on every single returned value.  I've
> written a POC patch for this and so far, performance seems to be much
> improved, on par with using cx_Oracle without Django.

5.0.1 seems to be released somewhere around 03/2009. [http://
sourceforge.net/projects/cx-oracle/files/]. +1 for requiring 5.0.1.

We could probably detect the cx_oracle version and based on that use
either the _rowhandler approach or the _outputtypehandler approach.
This seems like overkill to me. We do not have backwards compatibility
requirements for old library versions, right?

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.