[web2py] Re: what is web2py doing internally with unicode?

2011-09-04 Thread Massimo Di Pierro
No. that is why web2py assumes strings of bytes only. If you read a file and store it in a text or blob field you are fine. If you read text from a form then it depends on the html encoding. Also database searches are affected by encoding. For consistency I advice that your bytes are utf8 encoded

[web2py] Re: what is web2py doing internally with unicode?

2011-09-04 Thread weheh
Yes, agreed. However, my question is more about what web2py is doing under the hood. If I do a file upload, is web2py doing any kind of encoding detection on the file contents and then converting to utf8? How about text upload? Furthermore, when I write a file using dbstore, does web2py do a ut

[web2py] Re: what is web2py doing internally with unicode?

2011-09-04 Thread Ray (a.k.a. Iceberg)
Generally speaking, you can't assume a file's encoding, unless you are the creator of that file therefore knows its encoding. Regards, Ray On Sep 5, 7:52 am, weheh wrote: > I have a text string which is read from a file. The file was created > by the db...store mechanism. The original data came

[web2py] Re: what is web2py doing internally with unicode?

2011-09-04 Thread weheh
I have a text string which is read from a file. The file was created by the db...store mechanism. The original data came from either an uploaded file or a text field. I have neither decoded nor encoded anything. Shouldn't I assume the text in the file is utf8? (Please oh please say yes.) On Sep 4,

[web2py] Re: what is web2py doing internally with unicode?

2011-09-04 Thread Massimo Di Pierro
web2py internally expects everything to be bytes representing utf8- encoded strings. On Sep 4, 1:02 pm, weheh wrote: > When I create a form with form=SQLFORM(...), if I have a myfield of > type "text"is the form.vars.myfield value of type unicode? If I then > save the field in a file by doing a d