Re: [opendbx] slow insert with sqlite
Hi Alain I've done a small program in Pharo 1.3 with glorp+opendbx that insert 1000 rows in a customer table in a sqlite db. The 1000 insert takes 140 sec (very slow), but the Pharo profiler says that it spend 95% of the time waiting for input. I can't speak for the Pharo implementation but SQLite is known for being problematic when at least two threads trying to write to the same database file. Norbert -- All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 ___ libopendbx-devel mailing list libopendbx-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/libopendbx-devel http://www.linuxnetworks.de/doc/index.php/OpenDBX
Re: [opendbx] slow insert with sqlite
Hi Norbert, Yes they are problems with multi threading and sqlite, my question was about the opendbx implementation and/or the opendbx interface in Pharo. In Pharo there is only one vm thread, but I thought that may be opendbx or the opendbx interface was doing the call in another thread and the vm thread was waiting for completion? Are they documents about the opendbx architecture and or the opendbx interface in Pharo (couldn't find on the web site). Thank you for your answer Regards Alain Norbert Sendetzky norb...@linuxnetworks.de a écrit dans le message de news:4e137b24.7010...@linuxnetworks.de... Hi Alain I've done a small program in Pharo 1.3 with glorp+opendbx that insert 1000 rows in a customer table in a sqlite db. The 1000 insert takes 140 sec (very slow), but the Pharo profiler says that it spend 95% of the time waiting for input. I can't speak for the Pharo implementation but SQLite is known for being problematic when at least two threads trying to write to the same database file. Norbert -- All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 -- All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 ___ libopendbx-devel mailing list libopendbx-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/libopendbx-devel http://www.linuxnetworks.de/doc/index.php/OpenDBX
Re: [opendbx] slow insert with sqlite
On Tue, Jul 5, 2011 at 11:11 PM, Alain Rastoul alr@free.fr wrote: Hi Norbert, Yes they are problems with multi threading and sqlite, my question was about the opendbx implementation and/or the opendbx interface in Pharo. In Pharo there is only one vm thread, yes but I thought that may be opendbx or the opendbx interface was doing the call in another thread and the vm thread was waiting for completion? yes, exactly. But only if the backend and database client library support that. Are they documents about the opendbx architecture and or the opendbx interface in Pharo (couldn't find on the web site). Check the links I've sent you and the emails I have forwarded to squeakdbx mailing list. After you have read Hernik mails I can point you to the code if you want ;) Thank you for your answer Regards Alain Norbert Sendetzky norb...@linuxnetworks.de a écrit dans le message de news:4e137b24.7010...@linuxnetworks.de... Hi Alain I've done a small program in Pharo 1.3 with glorp+opendbx that insert 1000 rows in a customer table in a sqlite db. The 1000 insert takes 140 sec (very slow), but the Pharo profiler says that it spend 95% of the time waiting for input. I can't speak for the Pharo implementation but SQLite is known for being problematic when at least two threads trying to write to the same database file. Norbert -- All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 -- All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 ___ libopendbx-devel mailing list libopendbx-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/libopendbx-devel http://www.linuxnetworks.de/doc/index.php/OpenDBX -- Mariano http://marianopeck.wordpress.com -- All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2___ libopendbx-devel mailing list libopendbx-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/libopendbx-devel http://www.linuxnetworks.de/doc/index.php/OpenDBX
Re: [opendbx] slow insert with sqlite
hi alan. i dont know if sqlite implemetation has a problem.. but in glorp some times do unnecesary read.. by example when you making N times CommitAndContinue.. and because it, you see a lot time spending in InputEventPollingFetcherwait. if you send to me or send to SqueakDbx list you glorp code(ST), may be, I can send to you some advices. or may be, we can find the delay source. Best. pd: please check in you sqlite the pragmas.. by example synchronous ( http://www.sqlite.org/pragma.html#pragma_synchronous) that increment the performance to much. On Tue, Jul 5, 2011 at 6:46 PM, Alain Rastoul alr@free.fr wrote: ** Hi Mariano, I don't want to do multi threading with sqlite because I know it doesn't work. I was curious about the squeakdbx (or opendbx architecture) because of the not so good performance and the time spent in waiting , I do not understand the squeakdbx package vs opendbx package: the doc is mentioning a squeakdbx plugin dll but I have no squeakdbx dll ? You are saying that in that case the external call is counted on the InputEventPollingFetcher wait and not in primitives (?). I will investigate with FFI/SQlite and it should be the same (I've seen some messages about incorrect profiling reports in primitives), I expected much better performance with sqlite , and glorp is very good (5% of the time), I would have expected the contrary. Thanks Cheers Alain Mariano Martinez Peck marianop...@gmail.com a écrit dans le message de news:CAA+-=mvv3zvpcfpm3uwts11y1ugxpcji6pzxypvjpztfsdr...@mail.gmail.com... On Tue, Jul 5, 2011 at 10:50 PM, Alain Rastoul alr@free.fr wrote: Hi, (sorry for sending this mail again, my pc was off for a long time and the message was dated from 2007, people who sort their messages would not see it) I've done a small program in Pharo 1.3 with glorp+opendbx that insert 1000 rows in a customer table in a sqlite db. The 1000 insert takes 140 sec (very slow), but the Pharo profiler says that it spend 95% of the time waiting for input. (in InputEventPollingFetcher waitForInput) I was wondering if the queries are executed in another thread than the vm thread ? Hi Alain. No. Squeak/Pharo's thread architecture is the so called green thread, that is, only ONE OS thread is used. Internally, the language reifies Process, Scheduler, #fork: , etc etc etc. But from the OS point of view there is only one thread for the VM. So.the regular FFI blocks the VM. What does it mean? that while the C function called by FFI is being executed, the WHOLE VM is block. Notihgn can happen at the same time. Imagine the function that retrieves the results and needs to wait for them.TERRIBLE. So...if the backend does not support async quieries, then you are screw and dbx may be slow in Pharo. Nothing to do. However, some backends support async queries, and opendbx let us configure this. This is explained in: http://www.squeakdbx.org/Architecture%20and%20desing?_s=FlIhkPQOOFSlqf8C_k=j-3_7Kw__n18 where it says External call implementation You can see the list of backends that support async queries in here: http://www.squeakdbx.org/documentation/Asynchronous%20queries?_s=FlIhkPQOOFSlqf8C_k=j-3_7Kw__n17 Notice that there is some room for improvements, but we didn't have time so far. Hernik told us some good ideas. But since we didn't need more power so far we couldn't find time to integrate his ideas. I am forwarding now the emails to the mailing list. If you can take a look and provide code, it would be awesome. Basically, it improves how and how much we wait in each side: image and opendbx. Finally, notice that Eliot is working in a multithreared FFI for Cog, but it is not yet available as far as I know. Cheers Mariano (I thought I've seen a document about opendbx architecture but could'nt find it on the site). TIA Alain -- All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 ___ libopendbx-devel mailing list libopendbx-devel-5nwgofrqmnerv+lv9mx5uipxlwaov...@public.gmane.org https://lists.sourceforge.net/lists/listinfo/libopendbx-devel http://www.linuxnetworks.de/doc/index.php/OpenDBX -- Mariano http://marianopeck.wordpress.com -- -- All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense.
