I think there might be a bug in this code in lockBtree:
if( sqlite3pager_pagecount(pBt->pPager)>0 ){ u8 *page1 = pPage1->aData; if( memcmp(page1, zMagicHeader, 16)!=0 ){ goto page1_init_failed; The problem is if you are trying to open a non-sqlite file that is smaller than pageSize. I believe in that case sqlite3pager_pagecount will return 0, which means that the code above that tests the magic header won't execute. At which point SQLite thinks the file is a sqlite database and use it. If the file is larger than pageSize, then it is correctly identified as not-a-database.