Re: [opendbx] slow insert with sqlite

Alain Rastoul Wed, 06 Jul 2011 10:54:52 -0700

Thank you Diogenes,

I will try your advice about synchronous pragma and send you my code, but I 
have to work for a customer this evening :(
I will subscribe to the squeak dbx mailing list too.
thanks again

Regards,
Alain
"Diogenes Moreira" <diogenes.more...@gmail.com> a écrit dans le message de
news: calwf4u6p3tas-ph9ew3ok5e-5r6c7kwzkeyrvgfcunfzs7n...@mail.gmail.com...
hi alan.

i dont know if sqlite implemetation has a problem.. but in glorp some times
do unnecesary read.. by example when you making "N times" CommitAndContinue..
and because it, you see a lot time spending in InputEventPollingFetcher>>wait.
if you send to me or send to SqueakDbx list you glorp code(ST), may be, I
can send to you some advices. or may be, we can find the delay source.

Best.

pd: please check in you sqlite the pragmas.. by example synchronous
(http://www.sqlite.org/pragma.html#pragma_synchronous) that increment the
performance to much.

On Tue, Jul 5, 2011 at 6:46 PM, Alain Rastoul <alr....@free.fr> wrote:

Hi Mariano,
I don't want to do multi threading with sqlite because I know it doesn't
work.
I was curious about the squeakdbx (or opendbx architecture) because of the
not so good performance and the time spent in waiting , I do not understand the
squeakdbx package vs opendbx package: the doc is mentioning a squeakdbx plugin
dll but I have no squeakdbx dll ?

You are saying that in that case the external call is counted on the
InputEventPollingFetcher>> wait and not in primitives (?).
I will investigate with FFI/SQlite and it should be the same (I've seen
some messages about incorrect profiling reports in primitives),

I expected much better performance with sqlite , and glorp is very good
(5% of the time), I would have expected the contrary.

Thanks

Cheers
Alain
"Mariano Martinez Peck" <marianop...@gmail.com> a écrit dans le message
de
news:CAA+-=mvv3zvpcfpm3uwts11y1ugxpcji6pzxypvjpztfsdrrdq-jsoawuisxosn+bqq9rb...@public.gmane.org...

On Tue, Jul 5, 2011 at 10:50 PM, Alain Rastoul <alr....@free.fr> wrote:

Hi,
(sorry for sending this mail again, my pc was off for a long time and
the
message was dated from 2007, people who sort their messages would not
see
it)

I've done a small program in Pharo 1.3 with glorp+opendbx that insert
1000
rows in a customer table in a sqlite db.
The 1000 insert takes 140 sec (very slow), but the Pharo profiler says
that
it spend 95%
of the time waiting for input.
(in InputEventPollingFetcher>> waitForInput)
I was wondering if the queries are executed in another thread than the
vm
thread ?

Hi Alain. No. Squeak/Pharo's thread architecture is the so called green
thread, that is, only ONE OS thread is used. Internally, the language reifies
Process, Scheduler, #fork: , etc etc etc. But from the OS point of view there
is only one thread for the VM. So.....the regular FFI blocks the VM. What does
it mean? that while the C function called by FFI is being executed, the WHOLE
VM is block. Notihgn can happen at the same time. Imagine the function that
retrieves the results and needs to wait for them.....TERRIBLE. So...if the
backend does not support async quieries, then you are screw and dbx may be slow
in Pharo. Nothing to do.

However, some backends support async queries, and opendbx let us
configure this. This is explained in:

http://www.squeakdbx.org/Architecture%20and%20desing?_s=FlIhkPQOOFSlqf8C&_k=j-3_7Kw_&_n&18
where it says "External call implementation"

You can see the list of backends that support async queries in here:

http://www.squeakdbx.org/documentation/Asynchronous%20queries?_s=FlIhkPQOOFSlqf8C&_k=j-3_7Kw_&_n&17

Notice that there is some room for improvements, but we didn't have time
so far. Hernik told us some good ideas. But since we didn't need more power so
far we couldn't find time to integrate his ideas. I am forwarding now the
emails to the mailing list. If you can take a look and provide code, it would
be awesome. Basically, it improves how and how much we wait in each side: image
and opendbx.

Finally, notice that Eliot is working in a multithreared FFI for Cog, but
it is not yet available as far as I know.

Cheers

Mariano

(I thought I've seen a document about opendbx architecture but could'nt
find
it on the site).

TIA
Alain

------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously
valuable.
Why? It contains a definitive record of application performance,
security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
libopendbx-devel mailing list

libopendbx-devel-5nwgofrqmnerv+lv9mx5uipxlwaovq5f-xmd5yjdbdmrexy1tmh2...@public.gmane.org

https://lists.sourceforge.net/lists/listinfo/libopendbx-devel
http://www.linuxnetworks.de/doc/index.php/OpenDBX

--
Mariano
http://marianopeck.wordpress.com

--------------------------------------------------------------------------

------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
libopendbx-devel mailing list
libopendbx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libopendbx-devel
http://www.linuxnetworks.de/doc/index.php/OpenDBX

------------------------------------------------------------------------------

------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security 
threats, fraudulent activity, and more. Splunk takes this data and makes 
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2

_______________________________________________
libopendbx-devel mailing list
libopendbx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libopendbx-devel
http://www.linuxnetworks.de/doc/index.php/OpenDBX

Re: [opendbx] slow insert with sqlite

Reply via email to