New submission from Ingo Ruhnke: When a sqlite database contains invalid UTF-8 code in a TEXT column, Python can query that data normally when .text_factory is set appropriately. However when a custom function is created with .create_function() and applied to that column the custom function will receive 'None' as argument instead of the value of the column.
The following example demonstrate the issue: Example: -------- import sqlite3 import sys import os con = sqlite3.connect(":memory:") con.text_factory = os.fsdecode con.create_function("py_identity", 1, lambda x: x) cur = con.cursor() cur.execute("CREATE TABLE foo(bar TEXT)") # insert some invalid UTF-8 into the database cur.execute("INSERT INTO foo(bar) VALUES(cast(? AS TEXT))", [b"\xff"]) # try to call a custom function on the invalid UTF-8 cur.execute("SELECT " " typeof(bar), " " bar, " # this works " py_identity(bar), " # this returns None instead of the content of 'bar' " cast(py_identity(cast(bar as BLOB)) AS TEXT) " # this works around the issue "FROM foo") for row in cur: print(row) Output: ------- ('text', '\udcff', None, '\udcff') Expected: --------- ('text', '\udcff', '\udcff', '\udcff') ---------- components: Library (Lib) messages: 283674 nosy: Ingo Ruhnke priority: normal severity: normal status: open title: Custom functions in sqlite receive None on invalid UTF-8 type: behavior versions: Python 3.6 _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue29021> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com