Hi --
Okay. It looks like sqlite3_win32_mbcs_to_utf8() is being called from a
number of functions, including sqlite3_value_blob, sqlite3_randomness,
sqlite3_extended_errcode, and sqlite3_os_init. From a spot check of the
execution trace it looks like sqlite3_os_init is probably the most frequent
caller. But tons and tons of calls to this function are being made, often
within the scope of a single prepared statement execution.
Here's a pretty typical sample piece of the execution trace:
sqldb::sqlite::Sqlite3PreparedStatement::execute
sqlite3_step
sqlite3_declare_vtab
sqlite3_declare_vtab
sqlite3_backup_step
sqlite3_value_blob
sqlite3_value_blob
sqlite3_win32_mbcs_to_utf8
sqlite3_os_init
sqlite3_win32_mbcs_to_utf8
sqlite3_win32_mbcs_to_utf8
sqlite3_randomness
sqlite3_mutex_leave
sqlite3_win32_mbcs_to_utf8
sqlite3_os_init
sqlite3_strnicmp
sqlite3_extended_errcode
sqlite3_enable_shared_cache
sqlite3_backup_step
sqlite3_value_numeric_type
sqlite3_backup_pagecount
sqlite3_extended_errcode
sqlite3_sql
sqlite3_enable_shared_cache
sqlite3_value_blob
sqlite3_value_blob
sqlite3_value_blob
sqlite3_os_init
sqlite3_win32_mbcs_to_utf8
sqlite3_win32_mbcs_to_utf8
I have no way of verifying if the indentation in the snippet above truly
accurately reflects call chains, but presumably you will be able to determine
this. I have the distinct feeling that something significant fundamental is
not configured correctly. Any help you can provide would be greatly
appreciated.
Also, quick question. Where would be the most appropriate place to issue the
PRAGMA you suggest. Does it operate globally or on a per database basis?
And if the latter, would I want to issue it before or after actually creating
the database schema?
Thanks.
Mike
-----Original Message-----
From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org]
On Behalf Of Richard Hipp
Sent: Wednesday, October 16, 2013 8:07 AM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Trying to figure out how to circumvent
sqlite3_win32_mbcs_to_utf8
On Wed, Oct 16, 2013 at 7:51 AM, Mike Clagett <mike.clag...@mathworks.com>wrote:
> Hi -
>
> We have a C++ (VisualC++) app that is reading from and writing to a sqlite
> database. Profiling reveals that it is spending 883.437 of its 2160.988
> seconds in the sqlite3_win32_mbcs_to_utf8 function.
Wow. That routine should only be called when (1) reporting a low-level I/O
error and (2) creating a temporary filename. And both of those should be rare
occurrences.
1. What does your profiler say is the most frequent caller to
sqlite3_win32_mbcs_to_utf8()?
2. Have you enabled error logging? (http://www.sqlite.org/errlog.html)
3. Have you tried running with PRAGMA temp_store=MEMORY to see if that helps?
4. Please tell us which MathWorks products us SQLite, so that we can add them
to http://www.sqlite.org/famous.html
> We are using std::basic_string<wchar_t,...> as our string type and I
> can only assume that these are being seen by sqlite as mbcs strings.
No. The problem is that SQLite uses UTF8 for filenames and Windows uses MBCS
for filenames and so we have to convert between the two when making Windows
system calls such as opening new files.
> I would like to know a better way of doing this that will eliminate all
> these unnecessary conversions. I believe it may end up being a
> combination of picking the correct string type (although using
> anything but the type we are using may be difficult if it contravenes a
> product-wide
> standard) and setting the defaults properly in sqlite. I have attempted
> the latter by issuing a m_db.executeStatement("PRAGMA encoding =
> \"UTF-16\"", error); just after I create a database. I clearly am not
> doing this effectively as it seems to have no effect on the use of the
> function in question.
>
> Any guidance from older hands would be greatly appreciated.
>
> Thanks.
>
> Michael Clagett
> Principal Software Engineer
> Mathworks, Inc.
> mike.clag...@mathworks.com<mailto:mike.clag...@mathworks.com>
> (508)-647-4307
>
> _______________________________________________
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
--
D. Richard Hipp
d...@sqlite.org
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
