Re: 'MySQL server has gone away' with pool_recycle=1 , what's wrong here?

2007-10-31 Thread Atilla

Thanks Mike, it was indeed just wrong settings in the config file. I
should pay more attention when looking through the documentation - I
thought that there's something specific. It properly echoes back the
queries now.

It still doesn't explain why I would get Mysql has gone away though
- I've MysqlDB 1.2.2 installed and my server timeout has been
increased to values that should never expire the connection. I'll
monitor it today and see if it'll continue doing this.

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---



xml-rpc, authentication and sessions

2007-10-31 Thread oscar tackstrom

Hi,

I'm developing a combined chat-bot/question answering system and plan
to use xml-rpc with pylons to serve web clients. Since other parts of
the project are already implemented in pylons (administration and
database connections), I would rather not switch to Twisted for this.

The server needs to maintain a state for each authenticated session,
and all communication should be secure. Is there a simple way to use
standard authentication and session management (AuthKit?) together
with xml-rpc in pylons?

Sorry if I'm ignorant and the solution should be obvious, I'm
completely new to Pylons and haven't found any information on how to
combine xml-rpc with authenticated session management in the docs.

Best,
Oscar Täckström


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---



Re: Reducing pylons app memory usage?

2007-10-31 Thread Marcin Kasperski


Just a (post-mortem?) note. After having very good experience from
migration (on some non-pylons site) from moinmoin hosted on apache
with fastcgi to moinmoin on twisted proxied behind nginx (I did not
make any formal measures, but it just feels that the app runs faster)
I opted for similar path on my pylons site. So I run nginx (to serve
static resources and to reverse-proxy pylons) and paster (to serve
pylons content).

I am extremely happy with nginx memory usage (4MB virtual, 700kB
resident).

I am still not very happy with paster (~100MB virtual, 22MB resident),
and hoping to search for some improvements here. But at least it is
just one such process...

Unfortunately this:

 [server:main]
 threadpool_nworkers = 5

did not work for me. Bare entering of this into the ini file results
in the following error:

$ paster serve runtime/development.ini
Starting server in PID 16404.
Traceback (most recent call last):
  File /usr/bin/paster, line 7, in ?
sys.exit(
  File /usr/lib/python2.4/site-packages/PasteScript-1.3.6-py2.4.egg/paste/scri
pt/command.py, line 78, in run
invoke(command, command_name, options, args[1:])
  File /usr/lib/python2.4/site-packages/PasteScript-1.3.6-py2.4.egg/paste/scri
pt/command.py, line 117, in invoke
exit_code = runner.run(args)
  File /usr/lib/python2.4/site-packages/PasteScript-1.3.6-py2.4.egg/paste/scri
pt/command.py, line 212, in run
result = self.command()
  File /usr/lib/python2.4/site-packages/PasteScript-1.3.6-py2.4.egg/paste/scri
pt/serve.py, line 232, in command
server(app)
  File /usr/lib/python2.4/site-packages/paste/deploy/loadwsgi.py, line 139, i
n server_wrapper
wsgi_app, context.global_conf,
  File /usr/lib/python2.4/site-packages/paste/deploy/util/fixtypeerror.py, li
ne 57, in fix_call
val = callable(*args, **kw)
  File /usr/lib/python2.4/site-packages/Paste-1.4-py2.4.egg/paste/httpserver.p 
y, line 1303, in server_runner
serve(wsgi_app, **kwargs)
  File /usr/lib/python2.4/site-packages/Paste-1.4-py2.4.egg/paste/httpserver.p 
y, line 1253, in serve
threadpool_options=threadpool_options)
  File /usr/lib/python2.4/site-packages/Paste-1.4-py2.4.egg/paste/httpserver.p 
y, line 1108, in __init__
ThreadPoolMixIn.__init__(self, nworkers, daemon_threads,
TypeError: __init__() got multiple values for keyword argument 'nworkers'


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---



Re: Reducing pylons app memory usage?

2007-10-31 Thread Ben Bangert

On Oct 31, 2007, at 8:39 AM, Marcin Kasperski wrote:


I am still not very happy with paster (~100MB virtual, 22MB resident),
and hoping to search for some improvements here. But at least it is
just one such process...


Can you ask your ISP if the virtual is counting against some limit?  
The reason I ask is that virtual doesn't necessarily even reflect  
swap (which I can believe a picky virtual provider might care about).  
Virtual can encompass shared libraries, paging files, existing  
resident, etc. So 100MB virtual is *not* 100 MB of swap. I continue  
to be baffled why there is such a problem with a number that itself  
varies wildly depending on architecture (vs the resident size, which  
*is* consistent across architectures).


For example, according to this detailed explanation of unix memory  
management:

http://www.dataexpedition.com/~sbnoble/Tips/memory.html

It would appear that the virtual number you see could even be  
referencing files on the filesystem (not in swap or ram). This is  
generally why the most meaningful numbers you *should* be looking at,  
are the resident memory, and shared memory (useful with something  
like Postgres to see how much ram each individual process is taking).


Can you elaborate on how you believe your hosting provider is  
counting the virtual number vs swap files and why you believe that  
virtual does mean swap? Please keep in mind, the 'virtual' memory on  
unix is *not* the same as 'virtual' memory on Windows (where it does  
generally mean swap).


Cheers,
Ben


smime.p7s
Description: S/MIME cryptographic signature


Re: Reducing pylons app memory usage?

2007-10-31 Thread Marcin Kasperski

 [server:main]
 threadpool_nworkers = 5

 did not work for me.


Resolved. One should use 

  threadpool_workers = 5

(without 'n'). Then it works. Some observations in the next post.


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---



Re: Reducing pylons app memory usage?

2007-10-31 Thread Marcin Kasperski

Ben Bangert [EMAIL PROTECTED] writes:

 On Oct 31, 2007, at 8:39 AM, Marcin Kasperski wrote:

 I am still not very happy with paster (~100MB virtual, 22MB resident),
 and hoping to search for some improvements here. But at least it is
 just one such process...

 Can you ask your ISP if the virtual is counting against some limit?

Well, I recently migrated some app to another host and the problem
does not hurt me that much, so I won't trouble him... But I am almost
sure that indeed, it is RAM and swap what are measured, I can't even
imagine how it could be otherwise.


 The reason I ask is that virtual doesn't necessarily even reflect
 swap (which I can believe a picky virtual provider might care
 about). Virtual can encompass shared libraries, paging files,
 existing resident, etc. So 100MB virtual is *not* 100 MB of swap.

This is true. But this is also relatively easy to measure, it suffices
to compare virtual size of the running app with virtual size of python
which just loaded all the libraries. The latter is fairly well
approximated by paster shell.

The latter allocates about 27MB of virtual on my host (but almost 20MB
of resident).


BTW, in the meantime I resolved the workers problem and indeed it
seems that it is all about per-thread storage. See by yourself
(measures after starting and serving one request):


* threadpool_workers = 5

virtual 66084, resident 18956

* threadpool_workers = 10

virtual 108104, resident 18900

* threadpool_workers = 20

virtual 189060, resident 19132


So it looks like it is all about thread-local storage. Maybe just
the thread stack size...


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---



Re: Reducing pylons app memory usage?

2007-10-31 Thread Ben Bangert

On Oct 31, 2007, at 9:41 AM, Marcin Kasperski wrote:


This is true. But this is also relatively easy to measure, it suffices
to compare virtual size of the running app with virtual size of python
which just loaded all the libraries. The latter is fairly well
approximated by paster shell.

The latter allocates about 27MB of virtual on my host (but almost 20MB
of resident).


To try and illustrate how useless the virtual number is that you're  
so worried about. On my OSX unix box, according to the OS, its using  
9.6GB of Virtual Memory. Now, I know where the real swap file it  
uses, is actually stored, and while top says 9.6GB of virtual, the  
swap file is 64megs


For another test, to really try and gauge what is occurring, I opened  
up some swap counters, to watch swap files page in, and page out.  
Then I started increasing the threadpool workers to see whether  
swapping was occurring.


At 15 threadpool:
VM: 65MB, Res: 17.13MB

At 65 threadpool:
VM: 90.65MB, Res: 18.13MB

So yes, increasing threadpool workers is definitely making virtual  
memory increase. Should we all be worried and stress out over it?  
Well, I was watching paging and swap activity the entire time, and  
there was *Zero* pager activity. Zero pageins, Zero pageouts, Zero  
increase in swap.


Let me re-iterate this again, the increase in virtual memory did  
*not* cause any increase in swap.


Can we please stop worrying over the virtual numbers now?

If you have some sort of hard data about real swap activity, I'd love  
to see it, but posting virtual numbers is pointless and does not  
represent swap usage. If there's some other unix expert that can  
shine more light on what the heck this virtual number does mean,  
that'd be great.


In the meantime, I'm considering the virtual memory issue a moot point.

Cheers,
Ben

smime.p7s
Description: S/MIME cryptographic signature


Re: Reducing pylons app memory usage?

2007-10-31 Thread Graham Dumpleton

On Nov 1, 6:08 am, Ben Bangert [EMAIL PROTECTED] wrote:
 On Oct 31, 2007, at 9:41 AM, Marcin Kasperski wrote:

  This is true. But this is also relatively easy to measure, it suffices
  to compare virtual size of the running app with virtual size of python
  which just loaded all the libraries. The latter is fairly well
  approximated by paster shell.

  The latter allocates about 27MB of virtual on my host (but almost 20MB
  of resident).

 To try and illustrate how useless the virtual number is that you're
 so worried about. On my OSX unix box, according to the OS, its using
 9.6GB of Virtual Memory. Now, I know where the real swap file it
 uses, is actually stored, and while top says 9.6GB of virtual, the
 swap file is 64megs

 For another test, to really try and gauge what is occurring, I opened
 up some swap counters, to watch swap files page in, and page out.
 Then I started increasing the threadpool workers to see whether
 swapping was occurring.

 At 15 threadpool:
 VM: 65MB, Res: 17.13MB

 At 65 threadpool:
 VM: 90.65MB, Res: 18.13MB

 So yes, increasing threadpool workers is definitely making virtual
 memory increase. Should we all be worried and stress out over it?
 Well, I was watching paging and swap activity the entire time, and
 there was *Zero* pager activity. Zero pageins, Zero pageouts, Zero
 increase in swap.

 Let me re-iterate this again, the increase in virtual memory did
 *not* cause any increase in swap.

 Can we please stop worrying over the virtual numbers now?

 If you have some sort of hard data about real swap activity, I'd love
 to see it, but posting virtual numbers is pointless and does not
 represent swap usage. If there's some other unix expert that can
 shine more light on what the heck this virtual number does mean,
 that'd be great.

 In the meantime, I'm considering the virtual memory issue a moot point.

We are talking about VPS systems, not a full OS running directly on
hardware. As a result, the amount of virtual memory may not be a moot
point.

As an example of configuration values for one VPS system:

Memory Limit - maximum virtual memory size a VPS/context can allocate;
Used memory - virtual memory size used by a running VPS/context;
Max total memory - maximum virtual memory usage by VPS/context;
Context RSS limit - maximum resident memory size a VPS/context can
allocate. If limit is exceeded, VPS starts to use the host's SWAP;
Context RSS - resident memory size used by a running VPS/context;
Max RSS memory - maximum resident memory usage by VPS/context;
Disk Limit - maximum disk space that can be used by VPS (calculated
for the entire VPS file tree); Used disk memory - disk space used by a
VPS file tree;
Files limit - maximum number of files that can be switched to a VPS/
context;
Used files - number of files used in a VPS/context;
TCP sockets limit - limit on the number of established connections in
a virtual server
Established sockets - number of established connections in a virtual
server

In respect of these limits it states:

when summary virtual memory size used by VPS/context exceedes
Memory Limit value, processes can't allocate the required memory and
fail to run

and:

it is recommended to set Context RSS Limit to one third of Memory
Limit

In other words the VPS configuration allows the provider to control
both Context RSS Limit and Memory Limit. Feasibly, if a provider is
setting the Memory Limit to be too low a value in relation to the
Context RSS Limit, then how a particular OS allocates virtual memory
can be a problem as it will reach the defined Memory Limit before it
reaches the Context RSS Limit.

This is why how the provider has set up the VPS is an issue. It would
appear that some providers may be setting quite restrictive limits,
not perhaps really understanding how processes make use of memory.

FWIW, this issue has come up before on the Apache httpd developers
list:

  http://marc.info/?l=apache-httpd-devm=119296272027150w=2

It was in relation to this that the Linux default per thread stack
size came up as a contributer.

How valid all this is still needs some investigation, but it is
probably too early to dismiss as a non issue.

Graham


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---



Re: model definition spread across several files

2007-10-31 Thread Ron Bickers

On Oct 31, 4:03 pm, Evert Rol [EMAIL PROTECTED] wrote:
 I have a quick question on what to do with the metadata object when  
 my model definition is spread across several files. It's all in the  
 same database, all different tables, but for my own logic, I'm  
 defining parts in different model files. I import the classes then in  
 the __init__ file, which is imported in the app (controllers) itself.  
 I'd like to keep the mapping per file, though. Other than that, I'm  
 following the 'SA 0.4 for people in hurry' by Mike.
 So, is that possible, and what do I do with the metdata object? I  
 would guess that only one call to MetaData() should be enough, or  
 does it result in the same object anyway? If it's only one call, how  
 do I get it into the other files?

Here's what I do:

Create metadata.py in model with the following (or whatever else you
need):

from pylons import config
from sqlalchemy import Column, MetaData, Table, ForeignKey, types
from sqlalchemy.orm import mapper, relation, backref
from sqlalchemy.orm import scoped_session, sessionmaker

Session = scoped_session(sessionmaker(autoflush=True,
transactional=True,
 
bind=config['pylons.g'].sa_engine))
metadata = MetaData()

Then I put 'from metadata import *' at the top of each of the model
files and 'from metadata import metadata, Session' at the top of
__init__.py

 As an small additional complication, some classes and tables are  
 imported from one model file into another, for mapping relations. I'm  
 now stuck at the point of creating the tables (using 'paster setup-
 app'), which complains about a non-existant table from which it needs  
 a foreign key (table is defined in a different file, but imported for  
 the relation. My initial guess is, that this is where the metadata  
 object comes into play).

I think I ran into this problem a while ago that I resolved, but I
can't recall the details.  I'd have to see some sample code.

--
Ron


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---



Re: Reducing pylons app memory usage?

2007-10-31 Thread Ben Bangert

On Oct 31, 2007, at 4:44 PM, Graham Dumpleton wrote:


What figures do you get for the above if you run:

  ulimit -s 512

in your shell prior to running the web application?

You will need to exit the shell when done to get it back to default
value.

If the VPS Memory Limit (virtual memory size) is found to be an issue
for some providers of VPS systems, for Python 2.5 at least, the simple
solution would be to allow a configuration option something like:

  threadpool_workers_stackize = 524288

when this is present, then the web server could call:

  import thread
  thread.stack_size(value)

This way the user could change it if they needed to because of an
overly restrictive VPS configuration.


Thanks Graham! I've looked over that Apache thread, the clarifying  
post was:

http://marc.info/?l=apache-httpd-devm=119296450928812w=2

I did some more testing, with base Python, then paster shell (which  
loads a Pylons app but *no* threads), then paster serve (which has  
threads). Sure enough, all the virtual consumption seems to be due to  
the thread size on linux being rather hefty.


On a linux system, Pylons itself consumes about 15mb of Virtual (not  
too unreasonable I hope?). The massive increase in Virtual is due to  
paster serve spawning 10 threads (at 8mb virtual each), plus one  
watch thread that monitors the other threads if they get stuck. Since  
linux is allocating 8mb in the stack per thread, this is where that  
extra 80mb of Virtual comes from.


You can run Pylons with FastCGI, directly loading the Pylons app  
(thus no threadpools). This should work great with your nginx setup  
as well, and mitigate the Virtual issue from the thread size on  
linux. Your virtual should then hover around 20mb instead of the  
100mb you're seeing.


Cheers,
Ben


smime.p7s
Description: S/MIME cryptographic signature


Re: Reducing pylons app memory usage?

2007-10-31 Thread Ben Bangert

On Oct 31, 2007, at 5:05 PM, Ben Bangert wrote:

You can run Pylons with FastCGI, directly loading the Pylons app  
(thus no threadpools). This should work great with your nginx setup  
as well, and mitigate the Virtual issue from the thread size on  
linux. Your virtual should then hover around 20mb instead of the  
100mb you're seeing.


Actually, Ian clarified that the threadpool can be turned off. Add  
the following to the [server] section:

use_threadpool = false

On my linux system, that cut the Virtual from 100Mb to 18Mb.

Cheers,
Ben

smime.p7s
Description: S/MIME cryptographic signature


Re: using use_unicode in mysql:// uri causes url_for crash in pylons

2007-10-31 Thread SamDonaldson

On my dev machine running pastehttpserver, I noticed that it only
works if I use the use_unicode=1 param on my db uri, even after
passing in the convert_unicode and encoding params to create_engine
(note that I'm calling my own create_engine).

Thanks,

Saureen

On Oct 30, 10:25 pm, Mike Orr [EMAIL PROTECTED] wrote:
 On 10/30/07, SamDonaldson [EMAIL PROTECTED] wrote:





  Hello,

  I ran into another problem on production, where, if I add the
  use_unicode parameter on the mysql:// URI, I get back unicode but I
  see that I get a crash in a particular case with the following stack:

  Module pylons.helpers:65 in redirect_to
and cookies extracted from it and added into the redirect
  issued.and cookies extracted from it and added into the redirect
  issued.
  response = kargs.pop('_response', None)
  found = httpexceptions.HTTPFound(url_for(*args, **kargs))
  if response:
  if str(response.status_code).startswith('3'):  found =
  httpexceptions.HTTPFound(url_for(*args, **kargs))
  Module routes.util:200 in url_for
  
  if not isinstance(url, str) and url is not None:
  raise Exception(url_for can only return a string or None,
  got 
   unicode instead: %s % url)  raise
  Exception(url_for can only return a string or None, got 
  Exception: url_for can only return a string or None, got unicode
  instead:

  There's an explicity check on 'str' to see if it's an instance even
  thought it's a unicode object.  How do I get this to work with
  Unicode? Note, I don't get this problem on my dev environment.  On
  production I'm running SCGI + Lighthttpd.

 That's really strange, both this and your other error.  Of course, any
 time str and unicode are used together in an expression, the str gets
 promoted to unicode. So one place to start would be to look back ward
 thorough the code and look at the intermediate variables: where was a
 unicode value first introduced? In your routes definitions? Do they
 show up in the route map result? What does url_for return in paster
 shell?

 scgi is of course a different environment than pastehttpserver.
 Perhaps pastehttpserver is more forgiving about converting unicode to
 str while scgi just raises an exception.  'use_unicode' is
 disrecommended now; the engine option 'convert_unicode' is recommended
 instead.

 --
 Mike Orr [EMAIL PROTECTED]


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---



Re: using use_unicode in mysql:// uri causes url_for crash in pylons

2007-10-31 Thread Mike Orr

use_unicode and convert_unicode should not be used together.
use_unicode makes MySQLdb convert the database values to Unicode.
convert_unicode does the same thing in SQLAlchemy..  encoding twice
will lead to exceptions or wrong values if there are any non-ASCII
characters. If users paste text into forms from Word, they'll contain
curly quotes, bullet characters, 1/2 ligatures, degree symbols,
accented letters, etc.

But you are getting errors that don't make any sense, so who knows
what's happening on your server.  There should be no relationship
between SQLAlchemy settings and url_for.

On 10/31/07, SamDonaldson [EMAIL PROTECTED] wrote:

 On my dev machine running pastehttpserver, I noticed that it only
 works if I use the use_unicode=1 param on my db uri, even after
 passing in the convert_unicode and encoding params to create_engine
 (note that I'm calling my own create_engine).

 Thanks,

 Saureen

 On Oct 30, 10:25 pm, Mike Orr [EMAIL PROTECTED] wrote:
  On 10/30/07, SamDonaldson [EMAIL PROTECTED] wrote:
 
 
 
 
 
   Hello,
 
   I ran into another problem on production, where, if I add the
   use_unicode parameter on the mysql:// URI, I get back unicode but I
   see that I get a crash in a particular case with the following stack:
 
   Module pylons.helpers:65 in redirect_to
 and cookies extracted from it and added into the redirect
   issued.and cookies extracted from it and added into the redirect
   issued.
   response = kargs.pop('_response', None)
   found = httpexceptions.HTTPFound(url_for(*args, **kargs))
   if response:
   if str(response.status_code).startswith('3'):  found =
   httpexceptions.HTTPFound(url_for(*args, **kargs))
   Module routes.util:200 in url_for
   
   if not isinstance(url, str) and url is not None:
   raise Exception(url_for can only return a string or None,
   got 
unicode instead: %s % url)  raise
   Exception(url_for can only return a string or None, got 
   Exception: url_for can only return a string or None, got unicode
   instead:
 
   There's an explicity check on 'str' to see if it's an instance even
   thought it's a unicode object.  How do I get this to work with
   Unicode? Note, I don't get this problem on my dev environment.  On
   production I'm running SCGI + Lighthttpd.
 
  That's really strange, both this and your other error.  Of course, any
  time str and unicode are used together in an expression, the str gets
  promoted to unicode. So one place to start would be to look back ward
  thorough the code and look at the intermediate variables: where was a
  unicode value first introduced? In your routes definitions? Do they
  show up in the route map result? What does url_for return in paster
  shell?
 
  scgi is of course a different environment than pastehttpserver.
  Perhaps pastehttpserver is more forgiving about converting unicode to
  str while scgi just raises an exception.  'use_unicode' is
  disrecommended now; the engine option 'convert_unicode' is recommended
  instead.
 
  --
  Mike Orr [EMAIL PROTECTED]


 



-- 
Mike Orr [EMAIL PROTECTED]

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---



Re: using use_unicode in mysql:// uri causes url_for crash in pylons

2007-10-31 Thread SamDonaldson

FYI, I noticed ths post that's very similar:

http://groups.google.com/group/pylons-discuss/browse_thread/thread/42f2728b1f0264f7/59959f5bdd503884?lnk=gstq=FastCGI+Unicode+HTML

I noticed that the url_for implementation for Routes 1.7 is different
from Routes 1.6.8 under util.py.  In 1.7, which my production server
is running, there's an exception that gets thrown and an explicit
check on making sure it's a string and not unicode.

As far as FLUP is concerned, it seems like the write() method is
getting the template as a Unicode string to render and it's crapping
out on that.  Does FLUP SCGI actually understand Unicode.  The above
thread is very similar to what I'm going through.

Sam


On Oct 31, 9:09 pm, Mike Orr [EMAIL PROTECTED] wrote:
 use_unicode and convert_unicode should not be used together.
 use_unicode makes MySQLdb convert the database values to Unicode.
 convert_unicode does the same thing in SQLAlchemy..  encoding twice
 will lead to exceptions or wrong values if there are any non-ASCII
 characters. If users paste text into forms from Word, they'll contain
 curly quotes, bullet characters, 1/2 ligatures, degree symbols,
 accented letters, etc.

 But you are getting errors that don't make any sense, so who knows
 what's happening on your server.  There should be no relationship
 between SQLAlchemy settings and url_for.

 On 10/31/07, SamDonaldson [EMAIL PROTECTED] wrote:





  On my dev machine running pastehttpserver, I noticed that it only
  works if I use the use_unicode=1 param on my db uri, even after
  passing in the convert_unicode and encoding params to create_engine
  (note that I'm calling my own create_engine).

  Thanks,

  Saureen

  On Oct 30, 10:25 pm, Mike Orr [EMAIL PROTECTED] wrote:
   On 10/30/07, SamDonaldson [EMAIL PROTECTED] wrote:

Hello,

I ran into another problem on production, where, if I add the
use_unicode parameter on the mysql:// URI, I get back unicode but I
see that I get a crash in a particular case with the following stack:

Module pylons.helpers:65 in redirect_to
  and cookies extracted from it and added into the redirect
issued.and cookies extracted from it and added into the redirect
issued.
response = kargs.pop('_response', None)
found = httpexceptions.HTTPFound(url_for(*args, **kargs))
if response:
if str(response.status_code).startswith('3'):  found =
httpexceptions.HTTPFound(url_for(*args, **kargs))
Module routes.util:200 in url_for

if not isinstance(url, str) and url is not None:
raise Exception(url_for can only return a string or None,
got 
 unicode instead: %s % url)  raise
Exception(url_for can only return a string or None, got 
Exception: url_for can only return a string or None, got unicode
instead:

There's an explicity check on 'str' to see if it's an instance even
thought it's a unicode object.  How do I get this to work with
Unicode? Note, I don't get this problem on my dev environment.  On
production I'm running SCGI + Lighthttpd.

   That's really strange, both this and your other error.  Of course, any
   time str and unicode are used together in an expression, the str gets
   promoted to unicode. So one place to start would be to look back ward
   thorough the code and look at the intermediate variables: where was a
   unicode value first introduced? In your routes definitions? Do they
   show up in the route map result? What does url_for return in paster
   shell?

   scgi is of course a different environment than pastehttpserver.
   Perhaps pastehttpserver is more forgiving about converting unicode to
   str while scgi just raises an exception.  'use_unicode' is
   disrecommended now; the engine option 'convert_unicode' is recommended
   instead.

   --
   Mike Orr [EMAIL PROTECTED]

 --
 Mike Orr [EMAIL PROTECTED]


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---