Hello All,
I'm an absolute newbie with web2py so I'm not sure if the issue lies in
web2py/3.6 or my own mistakes... My apologies in advance.
I encounter this error in a db.my_table.update_or_insert(..)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfe in position 96328:
invalid start byte'
What I'm trying to do:
I have a text file that is the representation of a datamodel in a specific
COTS format.
I want to parse it, list the tables to be created and create them actually.
I know that the file countains non-utf-8 characters, that's why I open it
in binary:
dpm_file = gzip.open(filepath,'rb')
... parse it using binary regex: (I imported the 'regex' module)
regexp = regex.compile(b':columns
(\(((?>[^()]+)|(?1))*\))',regex.DOTALL)
header_classes = regexp.findall(one_format_text)[0]
... and insert the results in db blob fields: (the error occurs here)
db.dpm_header.update_or_insert(db.dpm_header.dpm == my_dpm_id, dpm =
my_dpm_id, header_classes = header_classes, header_objects =
header_objects)
I defined blob fields because I thought that it was the way to write
directly in binary in database.
I really wanted to use binary preciselly to avoid to manage the
encoding/decoding issues.
In addition, this project is an attempt to transport in web2py an older
project that was working perfectly in pure python 3.6.3 (unix and windows)
(except that instead of creating tables in a DB, I created text files in a
filer...).
Is it a issue on web2py/python 3.6? Or my misunderstanding on database
management...
Thank you for your help!
Here is the trace
Traceback (most recent call last):
File "D:\web\zz\web2py\gluon\restricted.py", line 219, in restricted
exec(ccode, environment)
File "D:/web/zz/web2py/applications/dpm_manager/controllers/default.py"
<http://127.0.0.1:8000/admin/default/edit/dpm_manager/controllers/default.py>,
line 175, in <module>
File "D:\web\zz\web2py\gluon\globals.py", line 419, in <lambda>
self._caller = lambda f: f()
File "D:\web\zz\web2py\gluon\tools.py", line 3982, in f
return action(*a, **b)
File "D:/web/zz/web2py/applications/dpm_manager/controllers/default.py"
<http://127.0.0.1:8000/admin/default/edit/dpm_manager/controllers/default.py>,
line 131, in analyze
header_classes = header_classes)
File "D:\web\zz\web2py\gluon\packages\dal\pydal\objects.py", line 811, in
update_or_insert
record = self(_key)
File "D:\web\zz\web2py\gluon\packages\dal\pydal\objects.py", line 565, in
__call__
orderby_on_limitby=False).first()
File "D:\web\zz\web2py\gluon\packages\dal\pydal\objects.py", line 2250, in
select
return adapter.select(self.query, fields, attributes)
File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\sqlite.py", line 82,
in select
return super(SQLite, self).select(query, fields, attributes)
File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\base.py", line 762,
in select
return self._select_aux(sql, fields, attributes, colnames)
File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\base.py", line 741,
in _select_aux
return processor(rows, fields, colnames, cacheable=cacheable)
File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\base.py", line 305,
in parse
for row in rows
File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\base.py", line 302,
in <listcomp>
self._parse(
File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\base.py", line 231,
in _parse
value = self.parse_value(value, fit, ft, blob_decode)
File "D:\web\zz\web2py\gluon\packages\dal\pydal\adapters\base.py", line 198,
in parse_value
return self.parser.parse(value, field_itype, field_type)
File "D:\web\zz\web2py\gluon\packages\dal\pydal\parsers\__init__.py", line
101, in parse
return self.registered[field_itype](value, field_type)
File "D:\web\zz\web2py\gluon\packages\dal\pydal\parsers\__init__.py", line
76, in __call__
return self.call(value, field_type)
File "D:\web\zz\web2py\gluon\packages\dal\pydal\parsers\__init__.py", line
73, in _call
return self.f(self.parser, value)
File "D:\web\zz\web2py\gluon\packages\dal\pydal\parsers\base.py", line 37, in
_blob
return to_native(b64decode(to_bytes(value)))
File "D:\web\zz\web2py\gluon\packages\dal\pydal\_compat.py", line 82, in
to_native
return obj.decode(charset, errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfe in position 96328:
invalid start byte
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
