Re: [sqlite] Anybody know why mono SQLite uses SqliteConnection but Windows uses SQLiteConnection?
mono install.exe often works. On Fri, Nov 18, 2016 at 9:16 PM, Jim Hendersonwrote: > Thanks for the suggestion, Bill. > > Will this run on a Linux system? The zip file has Install.exe and Test.exe > inside it. > > Jim > > > On 18-Nov-2016 7:44 AM, Drago, William @ CSG - NARDA-MITEQ wrote: > >> Do any of you know why the two different versions of SQLite have different >>> case in the method names? >>> >>> Thanks, >>> Jim >>> >>> Is there a reason why you can't use this one which should have the >> correct case? >> >> https://system.data.sqlite.org/index.html/doc/trunk/www/down >> loads.wiki#sqlite-netFx451-binary-Mono-2013 >> >> -- >> Bill Drago >> Staff Engineer >> L3 Narda-MITEQ >> 435 Moreland Road >> Hauppauge, NY 11788 >> 631-272-5947 / william.dr...@l-3com.com >> >> CONFIDENTIALITY, EXPORT CONTROL AND DISCLAIMER NOTE:This e-mail and any >> attachments are solely for the use of the addressee and may contain >> information that is privileged or confidential. Any disclosure, use or >> distribution of the information contained herein is prohibited. In the >> event this e-mail contains technical data within the definition of the >> International Traffic in Arms Regulations or Export Administration >> Regulations, it is subject to the export control laws of the >> U.S.Government. The recipient should check this e-mail and any attachments >> for the presence of viruses as L-3 does not accept any liability associated >> with the transmission of this e-mail. If you have received this >> communication in error, please notify the sender by reply e-mail and >> immediately delete this message and any attachments. >> ___ >> sqlite-users mailing list >> sqlite-users@mailinglists.sqlite.org >> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users >> > > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Anybody know why mono SQLite uses SqliteConnection but Windows uses SQLiteConnection?
Thanks for the suggestion, Bill. Will this run on a Linux system? The zip file has Install.exe and Test.exe inside it. Jim On 18-Nov-2016 7:44 AM, Drago, William @ CSG - NARDA-MITEQ wrote: Do any of you know why the two different versions of SQLite have different case in the method names? Thanks, Jim Is there a reason why you can't use this one which should have the correct case? https://system.data.sqlite.org/index.html/doc/trunk/www/downloads.wiki#sqlite-netFx451-binary-Mono-2013 -- Bill Drago Staff Engineer L3 Narda-MITEQ 435 Moreland Road Hauppauge, NY 11788 631-272-5947 / william.dr...@l-3com.com CONFIDENTIALITY, EXPORT CONTROL AND DISCLAIMER NOTE:This e-mail and any attachments are solely for the use of the addressee and may contain information that is privileged or confidential. Any disclosure, use or distribution of the information contained herein is prohibited. In the event this e-mail contains technical data within the definition of the International Traffic in Arms Regulations or Export Administration Regulations, it is subject to the export control laws of the U.S.Government. The recipient should check this e-mail and any attachments for the presence of viruses as L-3 does not accept any liability associated with the transmission of this e-mail. If you have received this communication in error, please notify the sender by reply e-mail and immediately delete this message and any attachments. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Changing ID's to UUID
On Fri, 18 Nov 2016 10:22:15 -0800 Jens Alfkewrote: > You?d have to assign a central ?count server? to hand out consecutive > numbers No. You need only enumerate the sources, and accept that the key is the unique counter from the source plus the centrally assigned source identifier. > You can assign ?peer IDs?, but then you need a sure-fire-unique way > to assign them (using the MAC address is not sure-fire-unique) The sure-fire way to assign them is simple enumeration when they connect to the central server to upload their information. If the "peer" will ever need to retrieve the information, it will need to store its assigned id. Else it's just up to central to keep everything straight. > the system has to keep a persistent counter that can?t ever be rolled > back, even in the case of something like restoring an OS backup. That's just not a problem if each system is in control of its own counter. --jkl ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] I keep getting seg faults building my database using python sqlite3
On Fri, 18 Nov 2016 10:56:37 -0800 Roger Binnswrote: > Popen calls fork (it seems like you are doing Unix/Mac, not Windows). > fork() duplicates the process including all open file descriptors. > One or more of those descriptors belong to open SQLite databases and > ancillary files. Good catch, Roger. It's a liability, but I slightly disagree with your characterization. > - Running any Python code (destructors can be called which then run in > the parent and child) Yes, if those destructors affect shared resources. The OP did say the processes on the other side of popen were C programs. > - Not having file descriptors closed so the child process trashes them > (close_fds Popen argument is False in python 2.x but True in python > 3.x). The child process can't "trash" the parent's descriptors. When the child exits, the OS will close its descriptors, that's all. But, yes, if the child process is making some assumption about open descriptors it receives at startup, that could be lead to problems. Especially if it scribbles on the SQLite database. > Also python 2.x subprocess module is broken in many ways. My foray into Unicode in Python convinced me once and for all that Python 3 is the only way to go. But would you care to elaborate on the problems with 2.x subprocess? --jkl ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Hung query question
On Fri, 18 Nov 2016 19:20:06 + David Raymondwrote: > I've got a query that I've tried a few times here that seems like > it's hanging up on something and I'm wondering if it's just some > brain dead thing I'm forgetting or doing wrong. > > I've got a database with a bunch of records, and am trying to > populate a table in another database with coordinate extremes for > each state. Using the CLI I open up the side database, attach the > main one and run > > insert into coordExtremes select State, min(Latitude), max(Latitude), > min(Longitude), max(Longitude) from foo.bar group by State; > > Twice I've left that running overnight and it's still been sitting > there not completed the next day when I came in. I gave up on using > the CLI to do it and wrote a quick Python script to scan through the > whole big table once and keep track of the extremes, and it finished > in all of 15 minutes. So I'm left scratching my head as to what's up. > If anyone can think of something or sees where I'm being an idiot, > please let me know. Is python using bigger cache? Increase cache with pragma in CLI. Set it to 100-500MB for foo db, don't need to waste cache on destiny db (where you insert the data). Change foo with the real origin db name. pragma foo.cache_size=-50 Why don't use R*Tree virtual table? http://www.sqlite.org/rtree.html --- --- Eduardo Morras ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Hung query question
On 18 Nov 2016, at 7:20pm, David Raymondwrote: > insert into coordExtremes select State, min(Latitude), max(Latitude), > min(Longitude), max(Longitude) from foo.bar group by State; Apologies. Correction to my earlier post: Create indexes on table foo for the State, latitude and longitude columns. CREATE INDEX idx_foo_lat ON foo (State, Latitude); CREATE INDEX idx_foo_long ON foo (State, Longitude); ANALYZE; Then try the SELECT again. Simon. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Hung query question
On 18 Nov 2016, at 7:20pm, David Raymondwrote: > insert into coordExtremes select State, min(Latitude), max(Latitude), > min(Longitude), max(Longitude) from foo.bar group by State; Create indexes on table foo for the latitude and longitude columns. CREATE INDEX idx_foo_lat ON foo (Latitude); CREATE INDEX idx_foo_long ON foo (Longitude); ANALYZE; Then try the SELECT again. Simon. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Hung query question
I've got a query that I've tried a few times here that seems like it's hanging up on something and I'm wondering if it's just some brain dead thing I'm forgetting or doing wrong. I've got a database with a bunch of records, and am trying to populate a table in another database with coordinate extremes for each state. Using the CLI I open up the side database, attach the main one and run insert into coordExtremes select State, min(Latitude), max(Latitude), min(Longitude), max(Longitude) from foo.bar group by State; Twice I've left that running overnight and it's still been sitting there not completed the next day when I came in. I gave up on using the CLI to do it and wrote a quick Python script to scan through the whole big table once and keep track of the extremes, and it finished in all of 15 minutes. So I'm left scratching my head as to what's up. If anyone can think of something or sees where I'm being an idiot, please let me know. Below are more details, gonna try it again with the pre-compiled CLI rather than my self-compiled version to make sure it's not just something I managed to screw up in my compilation process. Of course, if it's the same then it won't finish by the time I send this. pragma quick_check; is running at the moment. File is ~33 gigs with ~110 million records. If quick_check finishes with nothing before I leave for the weekend I'll kick off a full integrity_check. This is on 64 bit Windows 7 Enterprise. Both databases are in WAL mode. The main data table is the only table in the main file. (Names have been changed to protect the innocent): create table bar ( id integer primary key, State text collate nocase, Latitude real, Longitude real, someOtherID int, bunch of other fields including between the above ); create index idx_bar_1 on bar (State, field2, field3, field4); create index idx_bar_2 on bar (someOtherID); select * from sqlite_stat1; tbl|idx|stat bar|idx_bar_1|109801252 2152966 3766 2887 2 bar|idx_bar_2|109801252 1 In the side database file what I'm trying to populate is: create table coordExtremes ( State text not null primary key collate nocase check (length(State) = 2), minLat real, maxLat real, minLon real, maxLon real ); sqlite> select sqlite_version(); sqlite_version() 3.15.1 sqlite> select sqlite_source_id(); sqlite_source_id() 2016-11-04 12:08:49 1136863c76576110e710dd5d69ab6bf347c65e36 sqlite> pragma compile_options; compile_option COMPILER=gcc-4.9.3 DEFAULT_MMAP_SIZE=0 ENABLE_COLUMN_METADATA ENABLE_DBSTAT_VTAB ENABLE_MEMORY_MANAGEMENT ENABLE_RTREE LIKE_DOESNT_MATCH_BLOBS MAX_MMAP_SIZE=0 OMIT_SHARED_CACHE SYSTEM_MALLOC THREADSAFE=0 attach database 'theBigOne.sqlite' as foo; explain query plan insert into coordExtremes select State, min(Latitude), max(Latitude), min(Longitude), max(Longitude) from foo.bar group by State; selectid|order|from|detail 0|0|0|SCAN TABLE bar USING INDEX idx_bar_1 explain insert into coordExtremes select State, min(Latitude), max(Latitude), min(Longitude), max(Longitude) from foo.bar group by State; addr opcode p1p2p3p4 p5 comment - - -- - 0 Init 0 78000 Start at 78 1 InitCoroutine 7 56200 2 Noop 1 3 000 3 Integer0 16000 r[16]=0; clear abort flag 4 Integer0 15000 r[15]=0; indicate accumulator empty 5 Null 0 1919 00 r[19..19]=NULL 6 Gosub 1853000 7 OpenRead 0 3 2 16 00 root=3 iDb=2; bar 8 OpenRead 2 4 2 k(5,NOCASE,NOCASE,NOCASE,NOCASE,) 00 root=4 iDb=2; idx_bar_1 9 Rewind 2 38210 00 10 Seek 2 0 000 Move 0 to 2.rowid 11 Column 2 0 20 00 r[20]=bar.State 12 Compare19201 k(1,NOCASE)00 r[19] <-> r[20] 13 Jump 141814 00 14 Move 2019100 r[19]=r[20] 15 Gosub 1744000 output one row 16 IfPos 1655000 if r[16]>0 then r[16]-=0, goto 55; check abort flag 17 Gosub 1853000 reset accumulator 18 Column 0 1421 00 r[21]=bar.Latitude 19 RealAffinity 210 000 20 CollSeq220 0 (BINARY) 00 21 AggStep0 0 219 min(1) 01 accum=r[9] step(r[21]) 22 Column 0 1421 00 r[21]=bar.Latitude 23 RealAffinity 210 0
Re: [sqlite] I keep getting seg faults building my database using python sqlite3
On 18/11/16 08:55, Kevin O'Gorman wrote: >> I am not. All of the python code is a single thread. The closest I come > is a few times where I use subprocess.Popen to create what amounts to a > pipeline, and one place where I start a number of copies of a C program in > parallel, but each is a separate process with its own input and output > files. Ding ding ding. You didn't mention which version of Python. Popen calls fork (it seems like you are doing Unix/Mac, not Windows). fork() duplicates the process including all open file descriptors. One or more of those descriptors belong to open SQLite databases and ancillary files. If the child process does virtually anything, it will result in crashes. Examples of doing things include: - Running any Python code (destructors can be called which then run in the parent and child) - Not having file descriptors closed so the child process trashes them (close_fds Popen argument is False in python 2.x but True in python 3.x). Also python 2.x subprocess module is broken in many ways. There are three methods for addressing this assuming it is the cause. 0: Python 3 is safer if you aren't already using it. The subprocess32 module for Python 2 backports Python 3 subprocess if you are stuck on Python 2 1: Make sure that the process that does SQLite work is a "leaf". ie it doesn't call subprocess or make any child processes. Instead a master process feeds it data on its stdin/out 2: Switch to APSW which has a fork checker built in. This will catch the fork problem no matter how it happens, and whatever is going on in other libraries you may use: https://rogerbinns.github.io/apsw/apsw.html?highlight=fork#apsw.fork_checker Disclaimer: I am the author of APSW Roger signature.asc Description: OpenPGP digital signature ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Changing ID's to UUID
> On Nov 16, 2016, at 5:59 AM, Keith Medcalfwrote: > > What I do not understand is why one would use a UUID (randomly generated > bunch of bytes) as a key in a database. It is long, every use must be > checked for collisions, and inherently far less efficient than the simple > integer sequence it is replacing. Distributed databases such as CouchDB and Couchbase do this. (I work on Couchbase Mobile.) When a database record is to be replicated between any number of peer databases, it obviously becomes infeasible to assign it a simple auto-incremented integer. (You’d have to assign a central ‘count server’ to hand out consecutive numbers, and then your system becomes bottlenecked on that server; it limits the rate at which the entire system can create records, and becomes a single point of failure.) You can assign ‘peer IDs’, but then you need a sure-fire-unique way to assign them (using the MAC address is not sure-fire-unique) and the system has to keep a persistent counter that can’t ever be rolled back, even in the case of something like restoring an OS backup. You can use a timestamp to try to avoid rollback problems, but now you’ve added the well-known problems of using clocks in a distributed system. Worse, you’ve started leaking a significant amount of interesting information about your computer, which in some cases can be a privacy problem. In the face of all that, simply generating a sufficiently long cryptographically-random number is very attractive. > THe problem is an inability to properly determine and assess risk. When > using a sequence the probability of a collision is 0. When using a random > generated number (passing a bunch of random data through a digest function) > has a probability of collision of 100%. Um. Either you’re trolling me, or you don’t have a good understanding of probability. In any case, this thread is pretty far off-topic and we should stop arguing this. I will not respond to any further posts. —Jens ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] I keep getting seg faults building my database using python sqlite3
On 18 Nov 2016, at 6:18pm, James K. Lowdenwrote: > The most likely culprit in my mind is RAM. You're exercising new memory > pretty hard, running a bunch of processes at it at full tilt. Any > defect in the chips or DMA could explain what you're seeing. An easy > test, not necessarily cheap, would be to replace the RAM (or, if > possible, run with some removed). Or borrow someone else's computer. Simon. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] I keep getting seg faults building my database using python sqlite3
On Fri, 18 Nov 2016 08:55:11 -0800 "Kevin O'Gorman"wrote: > All of the python code is a single thread. The closest I come > is a few times where I use subprocess.Popen to create what amounts to > a pipeline, and one place where I start a number of copies of a C > program in parallel, but each is a separate process with its own > input and output files. These C programs have been in use for a > number of months for earlier stages of this project, and I regard > them as quite reliable. None of them uses threads, and they are > mostly very simple filters. As you know, a process started with Popen cannot corrupt the Python process's memory. If you're not doing anything to defeat the GIL, a segfault inside the Python interpreter would be considered a bug. But is it happening in the interpreter, or in SQLite for that matter? ISTM that's what you need to know. To know that, you're going to need to run a debug version of the interpreter under gdb. When it faults, a backtrace will tell you where. That's not definititive proof; memory corruption is often detected far from where it was caused. But if the fault is at a consistent place in SQLite code, for example, you can use a hardware watchpoint to discover what's writing to it. I don't know what more to suggest. I would be surprised if you find a fault in Python, in the Python standard library, or in SQLite. I'm sure it won't be in anything on the other side of a popen call. Are there non-standard libraries or Python modules in use that you haven't mentioned? The most likely culprit in my mind is RAM. You're exercising new memory pretty hard, running a bunch of processes at it at full tilt. Any defect in the chips or DMA could explain what you're seeing. An easy test, not necessarily cheap, would be to replace the RAM (or, if possible, run with some removed). I have two war stories related to rotten I/O hardware, where the device appeared to work for all intents and purposes, but was actually a high-speed bit munger. Those were both over 20 years ago. It will be interesting to hear if that turns out to be your issue. HTH. --jkl ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] I keep getting seg faults building my database using python sqlite3
On Fri, Nov 18, 2016 at 8:38 AM, Roger Binnswrote: > On 17/11/16 19:14, Kevin O'Gorman wrote: > > SO: I need help bifurcating this problem. For instance, how can I tell > if > > the fault lies in SQLite, or in python? Or even in the hardware, given > that > > the time to failure is so variable? > > Are you using threads, threading related settings etc in any way? The > python sqlite3 module (aka pysqlite) is not threadsafe. That doesn't > stop people "working around" it, which can lead to crashes. > > Roger > > > I am not. All of the python code is a single thread. The closest I come is a few times where I use subprocess.Popen to create what amounts to a pipeline, and one place where I start a number of copies of a C program in parallel, but each is a separate process with its own input and output files. These C programs have been in use for a number of months for earlier stages of this project, and I regard them as quite reliable. None of them uses threads, and they are mostly very simple filters. The one that runs in parallel cannot be the culprit, however, because the code has not reached the point where it would come into play. That is the step where the results get "cached" (in flat files) and all of the early results are in those files. This thing is dying before reaching unknown territory where new results are needed. The reason the results exist is that they were generated by previous versions of the software that did not use Python. I am switching because (a) I want a database instead of flat files for speed reasons and (b) the C code was getting too hard to maintain. All of this is a hobby project, and I can share any parts of it that you care to see. -- #define QUESTION ((bb) || (!bb)) /* Shakespeare */ ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] I keep getting seg faults building my database using python sqlite3
On Fri, Nov 18, 2016 at 3:11 AM, Simon Slavinwrote: > Forgot to say ... > > Most of these problems result from attempting to reuse memory you've > already released. Even if the error is happening inside a SQLite routine, > it will be because you passed it a pointer to an SQLite connection which > had already been _close()'d or a statement which had already been > _finalize()'d. > > > Well, the entire process runs under a single connection that is never closed. Although cursors are opened and closed within some of the Python functions, the close is always at the end of the function. And I have no idea how things get finalized under python, or if that's an issue. Anyway, most of the time I use patterns like for row in c1.execute(...) or with dbfile.connect() as conn: and as I understand it, these protect me from most errors of that kind. Moreover, it seems to me that these would all lead to failure at exactly the same point, rather than at three points up to an hour different from each other. I was thinking it more likely to be one of a) random hardware malfunction b) some bug in memory handling that was subject to influence by other activity in the machine) c) some bug in hanling pointers to the python structures (it would have to be a python or sqlite3 bug.) -- #define QUESTION ((bb) || (!bb)) /* Shakespeare */ ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] I keep getting seg faults building my database using python sqlite3
On 17/11/16 19:14, Kevin O'Gorman wrote: > SO: I need help bifurcating this problem. For instance, how can I tell if > the fault lies in SQLite, or in python? Or even in the hardware, given that > the time to failure is so variable? Are you using threads, threading related settings etc in any way? The python sqlite3 module (aka pysqlite) is not threadsafe. That doesn't stop people "working around" it, which can lead to crashes. Roger signature.asc Description: OpenPGP digital signature ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] I keep getting seg faults building my database using python sqlite3
Forgot to say ... Most of these problems result from attempting to reuse memory you've already released. Even if the error is happening inside a SQLite routine, it will be because you passed it a pointer to an SQLite connection which had already been _close()'d or a statement which had already been _finalize()'d. Simon. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] I keep getting seg faults building my database using python sqlite3
On 18 Nov 2016, at 3:14am, Kevin O'Gormanwrote: > SO: I need help bifurcating this problem. For instance, how can I tell if > the fault lies in SQLite, or in python? Or even in the hardware, given that > the time to failure is so variable? Normal recommendation is to use GDB. You can find several tutorials on how to use GDB to spot seg faults on the web. Some other deveopment environments have shells around GBD or tools which will spot misused memory, for example Apple's Xcode environment. Simon. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users