Re: [web2py] Problem with Does web2py modify python strings?

[Anthony]

I'm not sure what's wrong -- I cannot reproduce it on an Ubuntu 12.04 
desktop or server.

Anthony

On Friday, September 27, 2013 4:03:00 PM UTC-4, viniciusban wrote:
>
> A new situation. 
>
> Before calling code.interact(): 
> >>> len("á")  # len() == 2 is OK 
> 2 
> >>> "á" # 2 bytes: ok 
> '\xc3\xa1' 
> >>> code.interact() 
> Python 2.7.3 (default, Apr 10 2013, 06:20:15) 
> [GCC 4.6.3] on linux2 
> Type "help", "copyright", "credits" or "license" for more information. 
> (InteractiveConsole) 
> >>> len("á")  # len() == 4 is not OK 
> 4 
> >>> "á" # 4 bytes: strange. 
> '\xc3\x83\xc2\xa1' 
> >>> # here I pressed CTRL-D to return from code.interact() 
> >>> "á" # 2 bytes, ok again. 
> '\xc3\xa1' 
>
>
> Any ideas? 
>
>
>
>
> On Fri, Sep 27, 2013 at 4:53 PM, Vinicius Assef 
> <vinic...@gmail.com<javascript:>> 
> wrote: 
> > Anthony, I think you're in the right direction. 
> > See: 
> > # ---------------- begin 
> > $ python 
> > Python 2.7.3 (default, Apr 10 2013, 06:20:15) 
> > [GCC 4.6.3] on linux2 
> > Type "help", "copyright", "credits" or "license" for more information. 
> >>>> print "á" 
> > á 
> >>>> import code 
> >>>> print "á" 
> > á 
> >>>> code.interact() 
> > Python 2.7.3 (default, Apr 10 2013, 06:20:15) 
> > [GCC 4.6.3] on linux2 
> > Type "help", "copyright", "credits" or "license" for more information. 
> > (InteractiveConsole) 
> >>>> print "á" 
> > á 
> >>>> exit() 
> > # ---------------- end 
> > 
> > The problem occurs after code.interact() call. 
> > 
> > 
> > 
> > 
> > On Fri, Sep 27, 2013 at 4:38 PM, Anthony <abas...@gmail.com<javascript:>> 
> wrote: 
> >> What happens if you run a Python file that includes the following: 
> >> 
> >> import code 
> >> code.interact() 
> >> 
> >> Anthony 
> >> 
> >> 
> >> On Friday, September 27, 2013 3:33:45 PM UTC-4, viniciusban wrote: 
> >>> 
> >>> Anthony, I'm using Ubuntu server 12.04 inside a Vagrant machine with 
> >>> provided by Virtualbox. 
> >>> 
> >>> Michele, my $LANG is exactly en_US.UTF-8, like yours. 
> >>> 
> >>> Would it be a problem with my OS? As I explained in first message, it 
> >>> works with standard Python shell, without web2py environment loaded. 
> >>> 
> >>> The problem occurs only when I use web2py from shell. 
> >>> 
> >>> 
> >>> 
> >>> On Fri, Sep 27, 2013 at 4:08 PM, Michele Comitini 
> >>> <michele....@gmail.com> wrote: 
> >>> > On your terminal you have to look at locales setup. Issue the 
> following: 
> >>> > 
> >>> > $ locales 
> >>> > 
> >>> > You should have utf-8 based locale. For instance: 
> >>> > 
> >>> > LANG=en_US.UTF-8 
> >>> > 
> >>> > On the browser look on the prefereces, see if you have 
> locale/language 
> >>> > settings supporting UTF-8. 
> >>> > If you have UTF-8 both on terminal and browser, then there is a 
> >>> > problem... 
> >>> > 
> >>> > 
> >>> > 2013/9/27 Vinicius Assef <vinic...@gmail.com> 
> >>> >> 
> >>> >> I was making a maintenance in table data through shell. 
> >>> >> 
> >>> >> Despite that, I'm still curious to discover why this behaviour is 
> >>> >> different from natural Python behaviour. 
> >>> >> 
> >>> >> 
> >>> >> On Fri, Sep 27, 2013 at 3:53 PM, Richard Vézina 
> >>> >> <ml.richa...@gmail.com> wrote: 
> >>> >> > Ok, but why do you need to do that in web2py? I remember have 
> read 
> >>> >> > something 
> >>> >> > about encoding in the book that was saying don't need to concern 
> >>> >> > about 
> >>> >> > that 
> >>> >> > in web2py. 
> >>> >> > 
> >>> >> > Richard 
> >>> >> > 
> >>> >> > 
> >>> >> > On Fri, Sep 27, 2013 at 2:46 PM, Vinicius Assef <
> vinic...@gmail.com> 
> >>> >> > wrote: 
> >>> >> >> 
> >>> >> >> On Fri, Sep 27, 2013 at 3:37 PM, Richard Vézina 
> >>> >> >> <ml.richa...@gmail.com> wrote: 
> >>> >> >> > Ha ok, I was miss leading by the linux shell... Could it be 
> that 
> >>> >> >> > you 
> >>> >> >> > need 
> >>> >> >> > u('éà') ?? 
> >>> >> >> 
> >>> >> >> No. the "áéíóú".decode("utf-8") should do that. 
> >>> >> >> And really does, in standard Python shell, outside web2py. 
> >>> >> >> 
> >>> >> >> 
> >>> >> >> > 
> >>> >> >> > 
> >>> >> >> > On Fri, Sep 27, 2013 at 1:47 PM, Vinicius Assef 
> >>> >> >> > <vinic...@gmail.com> 
> >>> >> >> > wrote: 
> >>> >> >> >> 
> >>> >> >> >> It wasn't a bash script, Richard. 
> >>> >> >> >> 
> >>> >> >> >> I made exactly as I described: straight from terminal. 
> >>> >> >> >> 
> >>> >> >> >> On Fri, Sep 27, 2013 at 2:23 PM, Richard Vézina 
> >>> >> >> >> <ml.richa...@gmail.com> wrote: 
> >>> >> >> >> > Could it be the encoding of the file that content your bash 
> >>> >> >> >> > script? 
> >>> >> >> >> > 
> >>> >> >> >> > Richard 
> >>> >> >> >> > 
> >>> >> >> >> > 
> >>> >> >> >> > On Fri, Sep 27, 2013 at 12:23 PM, Vinicius Assef 
> >>> >> >> >> > <vinic...@gmail.com> 
> >>> >> >> >> > wrote: 
> >>> >> >> >> >> 
> >>> >> >> >> >> Hey all. 
> >>> >> >> >> >> 
> >>> >> >> >> >> I'm facing a strange problem concerning special 
> characters. 
> >>> >> >> >> >> 
> >>> >> >> >> >> Using web2py shell, through command line (linux shell), I 
> do 
> >>> >> >> >> >> this: 
> >>> >> >> >> >> 
> >>> >> >> >> >> # ----------------  start 
> >>> >> >> >> >> $ python web2py.py -S welcome 
> >>> >> >> >> >> web2py Web Framework 
> >>> >> >> >> >> Created by Massimo Di Pierro, Copyright 2007-2013 
> >>> >> >> >> >> Version 2.6.4-stable+timestamp.2013.09.22.01.43.37 
> >>> >> >> >> >> Database drivers available: SQLite(sqlite3), 
> MySQL(pymysql), 
> >>> >> >> >> >> PostgreSQL(pg8000), IMAP(imaplib) 
> >>> >> >> >> >> WARNING:web2py:import IPython error; use default python 
> shell 
> >>> >> >> >> >> Python 2.7.3 (default, Apr 10 2013, 06:20:15) 
> >>> >> >> >> >> [GCC 4.6.3] on linux2 
> >>> >> >> >> >> Type "help", "copyright", "credits" or "license" for more 
> >>> >> >> >> >> information. 
> >>> >> >> >> >> (InteractiveConsole) 
> >>> >> >> >> >> >>> print "áéíóú"   # note the strange chars! 
> >>> >> >> >> >> áéíóú 
> >>> >> >> >> >> >>> from unicodedata import normalize 
> >>> >> >> >> >> >>> normalize("NFKD", 
> >>> >> >> >> >> >>> "áéuíóú".decode("utf-8")).encode("ASCII", 
> >>> >> >> >> >> >>> "ignore") 
> >>> >> >> >> >> >>> # again! 
> >>> >> >> >> >> 'AAuAA3Ao' 
> >>> >> >> >> >> # ----------------  end 
> >>> >> >> >> >> 
> >>> >> >> >> >> Note the noisy characters on the last line. 
> >>> >> >> >> >> 
> >>> >> >> >> >> 
> >>> >> >> >> >> Then, at the same machine, I enter into the naked Python 
> shell 
> >>> >> >> >> >> and: 
> >>> >> >> >> >> # ----------------  start 
> >>> >> >> >> >> $ python 
> >>> >> >> >> >> Python 2.7.3 (default, Apr 10 2013, 06:20:15) 
> >>> >> >> >> >> [GCC 4.6.3] on linux2 
> >>> >> >> >> >> Type "help", "copyright", "credits" or "license" for more 
> >>> >> >> >> >> information. 
> >>> >> >> >> >> >>> print "áéíóú" # looks ok 
> >>> >> >> >> >> áéíóú 
> >>> >> >> >> >> >>> from unicodedata import normalize 
> >>> >> >> >> >> >>> normalize("NFKD", 
> >>> >> >> >> >> >>> "áéuíóú".decode("utf-8")).encode("ASCII", 
> >>> >> >> >> >> >>> "ignore") 
> >>> >> >> >> >> >>> # ok again. 
> >>> >> >> >> >> 'aeiou' 
> >>> >> >> >> >> # ----------------  end 
> >>> >> >> >> >> 
> >>> >> >> >> >> Everything works fine. 
> >>> >> >> >> >> 
> >>> >> >> >> >> The same code. 
> >>> >> >> >> >> 
> >>> >> >> >> >> I found this problem because I'm working to strip special 
> >>> >> >> >> >> characters 
> >>> >> >> >> >> off of names, with unicodedata.normalize(). 
> >>> >> >> >> >> 
> >>> >> >> >> >> If it wan't strange enough, the unicodedata.normalize() 
> works 
> >>> >> >> >> >> fine 
> >>> >> >> >> >> from web environment, throug app admin. There the 
> >>> >> >> >> >> normalization 
> >>> >> >> >> >> works 
> >>> >> >> >> >> as expected (the same way as the standard naked Python 
> shell). 
> >>> >> >> >> >> 
> >>> >> >> >> >> What could be the root of this situation? 
> >>> >> >> >> >> 
> >>> >> >> >> >> -- 
> >>> >> >> >> >> Resources: 
> >>> >> >> >> >> - http://web2py.com 
> >>> >> >> >> >> - http://web2py.com/book (Documentation) 
> >>> >> >> >> >> - http://github.com/web2py/web2py (Source code) 
> >>> >> >> >> >> - https://code.google.com/p/web2py/issues/list (Report 
> Issues) 
> >>> >> >> >> >> --- 
> >>> >> >> >> >> You received this message because you are subscribed to 
> the 
> >>> >> >> >> >> Google 
> >>> >> >> >> >> Groups 
> >>> >> >> >> >> "web2py-users" group. 
> >>> >> >> >> >> To unsubscribe from this group and stop receiving emails 
> from 
> >>> >> >> >> >> it, 
> >>> >> >> >> >> send 
> >>> >> >> >> >> an 
> >>> >> >> >> >> email to web2py+un...@googlegroups.com. 
> >>> >> >> >> >> For more options, visit 
> >>> >> >> >> >> https://groups.google.com/groups/opt_out. 
> >>> >> >> >> > 
> >>> >> >> >> > 
> >>> >> >> >> > -- 
> >>> >> >> >> > Resources: 
> >>> >> >> >> > - http://web2py.com 
> >>> >> >> >> > - http://web2py.com/book (Documentation) 
> >>> >> >> >> > - http://github.com/web2py/web2py (Source code) 
> >>> >> >> >> > - https://code.google.com/p/web2py/issues/list (Report 
> Issues) 
> >>> >> >> >> > --- 
> >>> >> >> >> > You received this message because you are subscribed to the 
> >>> >> >> >> > Google 
> >>> >> >> >> > Groups 
> >>> >> >> >> > "web2py-users" group. 
> >>> >> >> >> > To unsubscribe from this group and stop receiving emails 
> from 
> >>> >> >> >> > it, 
> >>> >> >> >> > send 
> >>> >> >> >> > an 
> >>> >> >> >> > email to web2py+un...@googlegroups.com. 
> >>> >> >> >> > For more options, visit 
> >>> >> >> >> > https://groups.google.com/groups/opt_out. 
> >>> >> >> >> 
> >>> >> >> >> -- 
> >>> >> >> >> Resources: 
> >>> >> >> >> - http://web2py.com 
> >>> >> >> >> - http://web2py.com/book (Documentation) 
> >>> >> >> >> - http://github.com/web2py/web2py (Source code) 
> >>> >> >> >> - https://code.google.com/p/web2py/issues/list (Report 
> Issues) 
> >>> >> >> >> --- 
> >>> >> >> >> You received this message because you are subscribed to the 
> >>> >> >> >> Google 
> >>> >> >> >> Groups 
> >>> >> >> >> "web2py-users" group. 
> >>> >> >> >> To unsubscribe from this group and stop receiving emails from 
> it, 
> >>> >> >> >> send 
> >>> >> >> >> an 
> >>> >> >> >> email to web2py+un...@googlegroups.com. 
> >>> >> >> >> For more options, visit 
> https://groups.google.com/groups/opt_out. 
> >>> >> >> > 
> >>> >> >> > 
> >>> >> >> > -- 
> >>> >> >> > Resources: 
> >>> >> >> > - http://web2py.com 
> >>> >> >> > - http://web2py.com/book (Documentation) 
> >>> >> >> > - http://github.com/web2py/web2py (Source code) 
> >>> >> >> > - https://code.google.com/p/web2py/issues/list (Report 
> Issues) 
> >>> >> >> > --- 
> >>> >> >> > You received this message because you are subscribed to the 
> Google 
> >>> >> >> > Groups 
> >>> >> >> > "web2py-users" group. 
> >>> >> >> > To unsubscribe from this group and stop receiving emails from 
> it, 
> >>> >> >> > send 
> >>> >> >> > an 
> >>> >> >> > email to web2py+un...@googlegroups.com. 
> >>> >> >> > For more options, visit 
> https://groups.google.com/groups/opt_out. 
> >>> >> >> 
> >>> >> >> -- 
> >>> >> >> Resources: 
> >>> >> >> - http://web2py.com 
> >>> >> >> - http://web2py.com/book (Documentation) 
> >>> >> >> - http://github.com/web2py/web2py (Source code) 
> >>> >> >> - https://code.google.com/p/web2py/issues/list (Report Issues) 
> >>> >> >> --- 
> >>> >> >> You received this message because you are subscribed to the 
> Google 
> >>> >> >> Groups 
> >>> >> >> "web2py-users" group. 
> >>> >> >> To unsubscribe from this group and stop receiving emails from 
> it, 
> >>> >> >> send 
> >>> >> >> an 
> >>> >> >> email to web2py+un...@googlegroups.com. 
> >>> >> >> For more options, visit https://groups.google.com/groups/opt_out. 
>
> >>> >> > 
> >>> >> > 
> >>> >> > -- 
> >>> >> > Resources: 
> >>> >> > - http://web2py.com 
> >>> >> > - http://web2py.com/book (Documentation) 
> >>> >> > - http://github.com/web2py/web2py (Source code) 
> >>> >> > - https://code.google.com/p/web2py/issues/list (Report Issues) 
> >>> >> > --- 
> >>> >> > You received this message because you are subscribed to the 
> Google 
> >>> >> > Groups 
> >>> >> > "web2py-users" group. 
> >>> >> > To unsubscribe from this group and stop receiving emails from it, 
> >>> >> > send 
> >>> >> > an 
> >>> >> > email to web2py+un...@googlegroups.com. 
> >>> >> > For more options, visit https://groups.google.com/groups/opt_out. 
>
> >>> >> 
> >>> >> -- 
> >>> >> Resources: 
> >>> >> - http://web2py.com 
> >>> >> - http://web2py.com/book (Documentation) 
> >>> >> - http://github.com/web2py/web2py (Source code) 
> >>> >> - https://code.google.com/p/web2py/issues/list (Report Issues) 
> >>> >> --- 
> >>> >> You received this message because you are subscribed to the Google 
> >>> >> Groups 
> >>> >> "web2py-users" group. 
> >>> >> To unsubscribe from this group and stop receiving emails from it, 
> send 
> >>> >> an 
> >>> >> email to web2py+un...@googlegroups.com. 
> >>> >> For more options, visit https://groups.google.com/groups/opt_out. 
> >>> > 
> >>> > 
> >>> > -- 
> >>> > Resources: 
> >>> > - http://web2py.com 
> >>> > - http://web2py.com/book (Documentation) 
> >>> > - http://github.com/web2py/web2py (Source code) 
> >>> > - https://code.google.com/p/web2py/issues/list (Report Issues) 
> >>> > --- 
> >>> > You received this message because you are subscribed to the Google 
> >>> > Groups 
> >>> > "web2py-users" group. 
> >>> > To unsubscribe from this group and stop receiving emails from it, 
> send 
> >>> > an 
> >>> > email to web2py+un...@googlegroups.com. 
> >>> > For more options, visit https://groups.google.com/groups/opt_out. 
> >> 
> >> -- 
> >> Resources: 
> >> - http://web2py.com 
> >> - http://web2py.com/book (Documentation) 
> >> - http://github.com/web2py/web2py (Source code) 
> >> - https://code.google.com/p/web2py/issues/list (Report Issues) 
> >> --- 
> >> You received this message because you are subscribed to the Google 
> Groups 
> >> "web2py-users" group. 
> >> To unsubscribe from this group and stop receiving emails from it, send 
> an 
> >> email to web2py+un...@googlegroups.com <javascript:>. 
> >> For more options, visit https://groups.google.com/groups/opt_out. 
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.