[go-nuts] Re: Concurrent SQL queries with PG
One of my worst case scenarios completed in less then 21 minutes. Very encouraging!! Suppose I opened two connections each running 10 threads, would I approach 10 minutes? I'll give this a try later this week. This is pretty exciting for me since I never seen this problem solved in less than *hours*. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[go-nuts] Re: Concurrent SQL queries with PG
On Monday, 7 November 2016 19:54:35 UTC-5, Mandolyte wrote: > > Thanks for the quick response. and for your book - one of the best I've > ever purchased! > Thanks! Glad it was helpful. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[go-nuts] Re: Concurrent SQL queries with PG
Thanks for the quick response. and for your book - one of the best I've ever purchased! To close the loop, optimal number of go routines is about 10 and the database is actually Greenplum, a massively parallel processing architecture based on an early fork of PG ... YMMV On Monday, November 7, 2016 at 5:36:59 PM UTC-5, adon...@google.com wrote: > > On Monday, 7 November 2016 16:57:29 UTC-5, Mandolyte wrote: >> >> I have what amounts to a recursion problem and I wrote a minimal test >> using go routines. I am able to vary the max number of go routines as a >> parameter on the command line (*). But the times don't vary much whether a >> single go routine is used or 50 are used. I get the correct results, no >> matter how many are used. >> >> Are my expectations valid that I should be able to process faster with >> multiple go routines using the lib/pg driver? >> >> Even a minimal tester for this got to almost 200 lines, but if it helps: >> https://play.golang.org/p/y0cKhF7ujv >> >> Thanks for any advice... >> >> >> (*) The structure of my code are based on the technique used in gopl.io >> in chapter 8, the concurrent filesystem directory traversal. >> > > There's a bug in your code: numthreads is a pointer to an int variable > that is modified by the flag package during the call to flag.Parse, which > is usually the first thing done by main. However, the channel you're using > as a counting semaphore is created during package initialization, before > main and the flag.Parse function are executed, so it always has the default > size of 20. Create the channel after flag.Parse and see if that solves > your problem. > > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[go-nuts] Re: Concurrent SQL queries with PG
On Monday, 7 November 2016 16:57:29 UTC-5, Mandolyte wrote: > > I have what amounts to a recursion problem and I wrote a minimal test > using go routines. I am able to vary the max number of go routines as a > parameter on the command line (*). But the times don't vary much whether a > single go routine is used or 50 are used. I get the correct results, no > matter how many are used. > > Are my expectations valid that I should be able to process faster with > multiple go routines using the lib/pg driver? > > Even a minimal tester for this got to almost 200 lines, but if it helps: > https://play.golang.org/p/y0cKhF7ujv > > Thanks for any advice... > > > (*) The structure of my code are based on the technique used in gopl.io > in chapter 8, the concurrent filesystem directory traversal. > There's a bug in your code: numthreads is a pointer to an int variable that is modified by the flag package during the call to flag.Parse, which is usually the first thing done by main. However, the channel you're using as a counting semaphore is created during package initialization, before main and the flag.Parse function are executed, so it always has the default size of 20. Create the channel after flag.Parse and see if that solves your problem. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
