Re: [web2py] Problem with Does web2py modify python strings?

[Vinicius Assef]

A new situation.

Before calling code.interact():
>>> len("á")  # len() == 2 is OK
2
>>> "á" # 2 bytes: ok
'\xc3\xa1'
>>> code.interact()
Python 2.7.3 (default, Apr 10 2013, 06:20:15)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> len("á")  # len() == 4 is not OK
4
>>> "á" # 4 bytes: strange.
'\xc3\x83\xc2\xa1'
>>> # here I pressed CTRL-D to return from code.interact()
>>> "á" # 2 bytes, ok again.
'\xc3\xa1'


Any ideas?




On Fri, Sep 27, 2013 at 4:53 PM, Vinicius Assef <vinicius...@gmail.com> wrote:
> Anthony, I think you're in the right direction.
> See:
> # ---------------- begin
> $ python
> Python 2.7.3 (default, Apr 10 2013, 06:20:15)
> [GCC 4.6.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> print "á"
> á
>>>> import code
>>>> print "á"
> á
>>>> code.interact()
> Python 2.7.3 (default, Apr 10 2013, 06:20:15)
> [GCC 4.6.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> (InteractiveConsole)
>>>> print "á"
> á
>>>> exit()
> # ---------------- end
>
> The problem occurs after code.interact() call.
>
>
>
>
> On Fri, Sep 27, 2013 at 4:38 PM, Anthony <abasta...@gmail.com> wrote:
>> What happens if you run a Python file that includes the following:
>>
>> import code
>> code.interact()
>>
>> Anthony
>>
>>
>> On Friday, September 27, 2013 3:33:45 PM UTC-4, viniciusban wrote:
>>>
>>> Anthony, I'm using Ubuntu server 12.04 inside a Vagrant machine with
>>> provided by Virtualbox.
>>>
>>> Michele, my $LANG is exactly en_US.UTF-8, like yours.
>>>
>>> Would it be a problem with my OS? As I explained in first message, it
>>> works with standard Python shell, without web2py environment loaded.
>>>
>>> The problem occurs only when I use web2py from shell.
>>>
>>>
>>>
>>> On Fri, Sep 27, 2013 at 4:08 PM, Michele Comitini
>>> <michele....@gmail.com> wrote:
>>> > On your terminal you have to look at locales setup. Issue the following:
>>> >
>>> > $ locales
>>> >
>>> > You should have utf-8 based locale. For instance:
>>> >
>>> > LANG=en_US.UTF-8
>>> >
>>> > On the browser look on the prefereces, see if you have locale/language
>>> > settings supporting UTF-8.
>>> > If you have UTF-8 both on terminal and browser, then there is a
>>> > problem...
>>> >
>>> >
>>> > 2013/9/27 Vinicius Assef <vinic...@gmail.com>
>>> >>
>>> >> I was making a maintenance in table data through shell.
>>> >>
>>> >> Despite that, I'm still curious to discover why this behaviour is
>>> >> different from natural Python behaviour.
>>> >>
>>> >>
>>> >> On Fri, Sep 27, 2013 at 3:53 PM, Richard Vézina
>>> >> <ml.richa...@gmail.com> wrote:
>>> >> > Ok, but why do you need to do that in web2py? I remember have read
>>> >> > something
>>> >> > about encoding in the book that was saying don't need to concern
>>> >> > about
>>> >> > that
>>> >> > in web2py.
>>> >> >
>>> >> > Richard
>>> >> >
>>> >> >
>>> >> > On Fri, Sep 27, 2013 at 2:46 PM, Vinicius Assef <vinic...@gmail.com>
>>> >> > wrote:
>>> >> >>
>>> >> >> On Fri, Sep 27, 2013 at 3:37 PM, Richard Vézina
>>> >> >> <ml.richa...@gmail.com> wrote:
>>> >> >> > Ha ok, I was miss leading by the linux shell... Could it be that
>>> >> >> > you
>>> >> >> > need
>>> >> >> > u('éà') ??
>>> >> >>
>>> >> >> No. the "áéíóú".decode("utf-8") should do that.
>>> >> >> And really does, in standard Python shell, outside web2py.
>>> >> >>
>>> >> >>
>>> >> >> >
>>> >> >> >
>>> >> >> > On Fri, Sep 27, 2013 at 1:47 PM, Vinicius Assef
>>> >> >> > <vinic...@gmail.com>
>>> >> >> > wrote:
>>> >> >> >>
>>> >> >> >> It wasn't a bash script, Richard.
>>> >> >> >>
>>> >> >> >> I made exactly as I described: straight from terminal.
>>> >> >> >>
>>> >> >> >> On Fri, Sep 27, 2013 at 2:23 PM, Richard Vézina
>>> >> >> >> <ml.richa...@gmail.com> wrote:
>>> >> >> >> > Could it be the encoding of the file that content your bash
>>> >> >> >> > script?
>>> >> >> >> >
>>> >> >> >> > Richard
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> > On Fri, Sep 27, 2013 at 12:23 PM, Vinicius Assef
>>> >> >> >> > <vinic...@gmail.com>
>>> >> >> >> > wrote:
>>> >> >> >> >>
>>> >> >> >> >> Hey all.
>>> >> >> >> >>
>>> >> >> >> >> I'm facing a strange problem concerning special characters.
>>> >> >> >> >>
>>> >> >> >> >> Using web2py shell, through command line (linux shell), I do
>>> >> >> >> >> this:
>>> >> >> >> >>
>>> >> >> >> >> # ----------------  start
>>> >> >> >> >> $ python web2py.py -S welcome
>>> >> >> >> >> web2py Web Framework
>>> >> >> >> >> Created by Massimo Di Pierro, Copyright 2007-2013
>>> >> >> >> >> Version 2.6.4-stable+timestamp.2013.09.22.01.43.37
>>> >> >> >> >> Database drivers available: SQLite(sqlite3), MySQL(pymysql),
>>> >> >> >> >> PostgreSQL(pg8000), IMAP(imaplib)
>>> >> >> >> >> WARNING:web2py:import IPython error; use default python shell
>>> >> >> >> >> Python 2.7.3 (default, Apr 10 2013, 06:20:15)
>>> >> >> >> >> [GCC 4.6.3] on linux2
>>> >> >> >> >> Type "help", "copyright", "credits" or "license" for more
>>> >> >> >> >> information.
>>> >> >> >> >> (InteractiveConsole)
>>> >> >> >> >> >>> print "áéíóú"   # note the strange chars!
>>> >> >> >> >> áéíóú
>>> >> >> >> >> >>> from unicodedata import normalize
>>> >> >> >> >> >>> normalize("NFKD",
>>> >> >> >> >> >>> "áéuíóú".decode("utf-8")).encode("ASCII",
>>> >> >> >> >> >>> "ignore")
>>> >> >> >> >> >>> # again!
>>> >> >> >> >> 'AAuAA3Ao'
>>> >> >> >> >> # ----------------  end
>>> >> >> >> >>
>>> >> >> >> >> Note the noisy characters on the last line.
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >> Then, at the same machine, I enter into the naked Python shell
>>> >> >> >> >> and:
>>> >> >> >> >> # ----------------  start
>>> >> >> >> >> $ python
>>> >> >> >> >> Python 2.7.3 (default, Apr 10 2013, 06:20:15)
>>> >> >> >> >> [GCC 4.6.3] on linux2
>>> >> >> >> >> Type "help", "copyright", "credits" or "license" for more
>>> >> >> >> >> information.
>>> >> >> >> >> >>> print "áéíóú" # looks ok
>>> >> >> >> >> áéíóú
>>> >> >> >> >> >>> from unicodedata import normalize
>>> >> >> >> >> >>> normalize("NFKD",
>>> >> >> >> >> >>> "áéuíóú".decode("utf-8")).encode("ASCII",
>>> >> >> >> >> >>> "ignore")
>>> >> >> >> >> >>> # ok again.
>>> >> >> >> >> 'aeiou'
>>> >> >> >> >> # ----------------  end
>>> >> >> >> >>
>>> >> >> >> >> Everything works fine.
>>> >> >> >> >>
>>> >> >> >> >> The same code.
>>> >> >> >> >>
>>> >> >> >> >> I found this problem because I'm working to strip special
>>> >> >> >> >> characters
>>> >> >> >> >> off of names, with unicodedata.normalize().
>>> >> >> >> >>
>>> >> >> >> >> If it wan't strange enough, the unicodedata.normalize() works
>>> >> >> >> >> fine
>>> >> >> >> >> from web environment, throug app admin. There the
>>> >> >> >> >> normalization
>>> >> >> >> >> works
>>> >> >> >> >> as expected (the same way as the standard naked Python shell).
>>> >> >> >> >>
>>> >> >> >> >> What could be the root of this situation?
>>> >> >> >> >>
>>> >> >> >> >> --
>>> >> >> >> >> Resources:
>>> >> >> >> >> - http://web2py.com
>>> >> >> >> >> - http://web2py.com/book (Documentation)
>>> >> >> >> >> - http://github.com/web2py/web2py (Source code)
>>> >> >> >> >> - https://code.google.com/p/web2py/issues/list (Report Issues)
>>> >> >> >> >> ---
>>> >> >> >> >> You received this message because you are subscribed to the
>>> >> >> >> >> Google
>>> >> >> >> >> Groups
>>> >> >> >> >> "web2py-users" group.
>>> >> >> >> >> To unsubscribe from this group and stop receiving emails from
>>> >> >> >> >> it,
>>> >> >> >> >> send
>>> >> >> >> >> an
>>> >> >> >> >> email to web2py+un...@googlegroups.com.
>>> >> >> >> >> For more options, visit
>>> >> >> >> >> https://groups.google.com/groups/opt_out.
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> > --
>>> >> >> >> > Resources:
>>> >> >> >> > - http://web2py.com
>>> >> >> >> > - http://web2py.com/book (Documentation)
>>> >> >> >> > - http://github.com/web2py/web2py (Source code)
>>> >> >> >> > - https://code.google.com/p/web2py/issues/list (Report Issues)
>>> >> >> >> > ---
>>> >> >> >> > You received this message because you are subscribed to the
>>> >> >> >> > Google
>>> >> >> >> > Groups
>>> >> >> >> > "web2py-users" group.
>>> >> >> >> > To unsubscribe from this group and stop receiving emails from
>>> >> >> >> > it,
>>> >> >> >> > send
>>> >> >> >> > an
>>> >> >> >> > email to web2py+un...@googlegroups.com.
>>> >> >> >> > For more options, visit
>>> >> >> >> > https://groups.google.com/groups/opt_out.
>>> >> >> >>
>>> >> >> >> --
>>> >> >> >> Resources:
>>> >> >> >> - http://web2py.com
>>> >> >> >> - http://web2py.com/book (Documentation)
>>> >> >> >> - http://github.com/web2py/web2py (Source code)
>>> >> >> >> - https://code.google.com/p/web2py/issues/list (Report Issues)
>>> >> >> >> ---
>>> >> >> >> You received this message because you are subscribed to the
>>> >> >> >> Google
>>> >> >> >> Groups
>>> >> >> >> "web2py-users" group.
>>> >> >> >> To unsubscribe from this group and stop receiving emails from it,
>>> >> >> >> send
>>> >> >> >> an
>>> >> >> >> email to web2py+un...@googlegroups.com.
>>> >> >> >> For more options, visit https://groups.google.com/groups/opt_out.
>>> >> >> >
>>> >> >> >
>>> >> >> > --
>>> >> >> > Resources:
>>> >> >> > - http://web2py.com
>>> >> >> > - http://web2py.com/book (Documentation)
>>> >> >> > - http://github.com/web2py/web2py (Source code)
>>> >> >> > - https://code.google.com/p/web2py/issues/list (Report Issues)
>>> >> >> > ---
>>> >> >> > You received this message because you are subscribed to the Google
>>> >> >> > Groups
>>> >> >> > "web2py-users" group.
>>> >> >> > To unsubscribe from this group and stop receiving emails from it,
>>> >> >> > send
>>> >> >> > an
>>> >> >> > email to web2py+un...@googlegroups.com.
>>> >> >> > For more options, visit https://groups.google.com/groups/opt_out.
>>> >> >>
>>> >> >> --
>>> >> >> Resources:
>>> >> >> - http://web2py.com
>>> >> >> - http://web2py.com/book (Documentation)
>>> >> >> - http://github.com/web2py/web2py (Source code)
>>> >> >> - https://code.google.com/p/web2py/issues/list (Report Issues)
>>> >> >> ---
>>> >> >> You received this message because you are subscribed to the Google
>>> >> >> Groups
>>> >> >> "web2py-users" group.
>>> >> >> To unsubscribe from this group and stop receiving emails from it,
>>> >> >> send
>>> >> >> an
>>> >> >> email to web2py+un...@googlegroups.com.
>>> >> >> For more options, visit https://groups.google.com/groups/opt_out.
>>> >> >
>>> >> >
>>> >> > --
>>> >> > Resources:
>>> >> > - http://web2py.com
>>> >> > - http://web2py.com/book (Documentation)
>>> >> > - http://github.com/web2py/web2py (Source code)
>>> >> > - https://code.google.com/p/web2py/issues/list (Report Issues)
>>> >> > ---
>>> >> > You received this message because you are subscribed to the Google
>>> >> > Groups
>>> >> > "web2py-users" group.
>>> >> > To unsubscribe from this group and stop receiving emails from it,
>>> >> > send
>>> >> > an
>>> >> > email to web2py+un...@googlegroups.com.
>>> >> > For more options, visit https://groups.google.com/groups/opt_out.
>>> >>
>>> >> --
>>> >> Resources:
>>> >> - http://web2py.com
>>> >> - http://web2py.com/book (Documentation)
>>> >> - http://github.com/web2py/web2py (Source code)
>>> >> - https://code.google.com/p/web2py/issues/list (Report Issues)
>>> >> ---
>>> >> You received this message because you are subscribed to the Google
>>> >> Groups
>>> >> "web2py-users" group.
>>> >> To unsubscribe from this group and stop receiving emails from it, send
>>> >> an
>>> >> email to web2py+un...@googlegroups.com.
>>> >> For more options, visit https://groups.google.com/groups/opt_out.
>>> >
>>> >
>>> > --
>>> > Resources:
>>> > - http://web2py.com
>>> > - http://web2py.com/book (Documentation)
>>> > - http://github.com/web2py/web2py (Source code)
>>> > - https://code.google.com/p/web2py/issues/list (Report Issues)
>>> > ---
>>> > You received this message because you are subscribed to the Google
>>> > Groups
>>> > "web2py-users" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send
>>> > an
>>> > email to web2py+un...@googlegroups.com.
>>> > For more options, visit https://groups.google.com/groups/opt_out.
>>
>> --
>> Resources:
>> - http://web2py.com
>> - http://web2py.com/book (Documentation)
>> - http://github.com/web2py/web2py (Source code)
>> - https://code.google.com/p/web2py/issues/list (Report Issues)
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "web2py-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to web2py+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.