This message was forwarded from developers-l...@monetdb.org. The MonetDB
mailing lists have moved to monetdb.org. Please subscribe to
developers-l...@monetdb.org, and unsubscribe from this list.
See: http://mail.monetdb.org/mailman/listinfo/developers-list
Send developers-list mailing list submissions to
developers-l...@monetdb.org
To subscribe or unsubscribe via the World Wide Web, visit
http://mail.monetdb.org/mailman/listinfo/developers-list
or, via email, send a message with subject or body 'help' to
developers-list-requ...@monetdb.org
You can reach the person managing the list at
developers-list-ow...@monetdb.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of developers-list digest..."
Today's Topics:
1. Re: MonetDB: Feb2013 - Limit the number of concurrent queries
(Stefan Manegold)
2. Re: MonetDB: Feb2013 - Limit the number of concurrent queries
(Stefan Manegold)
----------------------------------------------------------------------
Message: 1
Date: Fri, 4 Jan 2013 13:06:41 +0100 (CET)
From: Stefan Manegold <stefan.maneg...@cwi.nl>
To: developers-l...@monetdb.org
Cc: checkin-l...@monetdb.org
Subject: Re: MonetDB: Feb2013 - Limit the number of concurrent queries
Message-ID: <952711866.2190532.1357301201236.javamail.r...@cwi.nl>
Content-Type: text/plain; charset=utf-8
> Changeset: e79b2edeba71 for MonetDB
> URL:
> http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=e79b2edeba71
> Modified Files:
> monetdb5/mal/mal.c
> monetdb5/mal/mal.h
> monetdb5/mal/mal_interpreter.c
> Branch: Feb2013
> Log Message:
>
> Limit the number of concurrent queries
> The total throughput for small (and large) queries is
> improved if we limit the number of concurrent queries
> being executed. This patch is geared at SQL, wrapping
> callMAL with a semaphore to limit the level of parallelism
> to number of physical cores.
>
>
> diffs (59 lines):
>
> diff --git a/monetdb5/mal/mal.c b/monetdb5/mal/mal.c
> --- a/monetdb5/mal/mal.c
> +++ b/monetdb5/mal/mal.c
> @@ -195,6 +195,7 @@ MT_Lock mal_remoteLock;
> MT_Lock mal_profileLock ;
> MT_Lock mal_copyLock;
> MT_Lock mal_delayLock;
> +MT_Sema mal_parallelism;
> /*
> * Initialization of the MAL context
> * The compiler directive STRUCT_ALIGNED tells that the
> @@ -233,6 +234,7 @@ int mal_init(void){
> MT_lock_init( &mal_profileLock, "mal_profileLock");
> MT_lock_init( &mal_copyLock, "mal_copyLock");
> MT_lock_init( &mal_delayLock, "mal_delayLock");
> + MT_sema_init( &mal_parallelism, (GDKnr_threads ? GDKnr_threads/2 : 4),
> "mal_parallelism");
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
With GDKnr_threads == 1 (single threaded execution or single-core system),
this results in mal_parallelism == 0,
which in turn results in all SQL tests failing (hanging until timeout) with
`Mtest.py -n1` (single-threaded).
Stefan
>
> tstAligned();
> MCinit();
> diff --git a/monetdb5/mal/mal.h b/monetdb5/mal/mal.h
> --- a/monetdb5/mal/mal.h
> +++ b/monetdb5/mal/mal.h
> @@ -118,6 +118,7 @@ mal_export MT_Lock mal_remoteLock;
> mal_export MT_Lock mal_profileLock ;
> mal_export MT_Lock mal_copyLock ;
> mal_export MT_Lock mal_delayLock ;
> +mal_export MT_Sema mal_parallelism;
>
>
> mal_export int mal_init(void);
> diff --git a/monetdb5/mal/mal_interpreter.c
> b/monetdb5/mal/mal_interpreter.c
> --- a/monetdb5/mal/mal_interpreter.c
> +++ b/monetdb5/mal/mal_interpreter.c
> @@ -479,7 +479,13 @@ callMAL(Client cntxt, MalBlkPtr mb, MalS
> ValPtr lhs;
> InstrPtr pci = getInstrPtr(mb, 0);
> RuntimeProfileRecord runtimeProfile;
> -
> +
> +/*
> + * Control the level of parallelism. The maximum number of
> concurrent MAL plans
> + * is determined by an environment variable. It is initially set
> equal to the
> + * number of cores, which may be too coarse.
> + */
> + MT_sema_down(&mal_parallelism,"mal_parallelism");
> runtimeProfileInit(mb, &runtimeProfile, cntxt->flags & memoryFlag);
> #ifdef DEBUG_CALLMAL
> mnstr_printf(cntxt->fdout, "callMAL\n");
> @@ -518,8 +524,10 @@ callMAL(Client cntxt, MalBlkPtr mb, MalS
> case PATcall:
> case CMDcall:
> default:
> + MT_sema_up(&mal_parallelism,"mal_parallelism");
> throw(MAL, "mal.interpreter", RUNTIME_UNKNOWN_INSTRUCTION);
> }
> + MT_sema_up(&mal_parallelism,"mal_parallelism");
> if (cntxt->qtimeout && time(NULL) - stk->clock.tv_usec >
> cntxt->qtimeout)
> throw(MAL, "mal.interpreter", RUNTIME_QRY_TIMEOUT);
> return ret;
> _______________________________________________
> checkin-list mailing list
> checkin-l...@monetdb.org
> http://mail.monetdb.org/mailman/listinfo/checkin-list
>
------------------------------
Message: 2
Date: Fri, 4 Jan 2013 13:19:54 +0100 (CET)
From: Stefan Manegold <stefan.maneg...@cwi.nl>
To: "Communication channel for developers of the MonetDB suite."
<developers-l...@monetdb.org>
Cc: Martin Kersten <martin.kers...@cwi.nl>
Subject: Re: MonetDB: Feb2013 - Limit the number of concurrent queries
Message-ID: <464989246.2191046.1357301994842.javamail.r...@cwi.nl>
Content-Type: text/plain; charset=utf-8
> > Changeset: e79b2edeba71 for MonetDB
> > URL:
> > http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=e79b2edeba71
> > Modified Files:
> > monetdb5/mal/mal.c
> > monetdb5/mal/mal.h
> > monetdb5/mal/mal_interpreter.c
> > Branch: Feb2013
> > Log Message:
> >
> > Limit the number of concurrent queries
> > The total throughput for small (and large) queries is
> > improved if we limit the number of concurrent queries
> > being executed. This patch is geared at SQL, wrapping
> > callMAL with a semaphore to limit the level of parallelism
> > to number of physical cores.
> >
> >
> > diffs (59 lines):
> >
> > diff --git a/monetdb5/mal/mal.c b/monetdb5/mal/mal.c
> > --- a/monetdb5/mal/mal.c
> > +++ b/monetdb5/mal/mal.c
> > @@ -195,6 +195,7 @@ MT_Lock mal_remoteLock;
> > MT_Lock mal_profileLock ;
> > MT_Lock mal_copyLock;
> > MT_Lock mal_delayLock;
> > +MT_Sema mal_parallelism;
> > /*
> > * Initialization of the MAL context
> > * The compiler directive STRUCT_ALIGNED tells that the
> > @@ -233,6 +234,7 @@ int mal_init(void){
> > MT_lock_init( &mal_profileLock, "mal_profileLock");
> > MT_lock_init( &mal_copyLock, "mal_copyLock");
> > MT_lock_init( &mal_delayLock, "mal_delayLock");
> > + MT_sema_init( &mal_parallelism, (GDKnr_threads ? GDKnr_threads/2 : 4),
> > "mal_parallelism");
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> With GDKnr_threads == 1 (single threaded execution or single-core
> system),
> this results in mal_parallelism == 0,
> which in turn results in all SQL tests failing (hanging until
> timeout) with `Mtest.py -n1` (single-threaded).
Maybe you intended one of (I cannot guess which one):
MT_sema_init( &mal_parallelism, (GDKnr_threads > 8 ? GDKnr_threads/2 :
4), "mal_parallelism");
MT_sema_init( &mal_parallelism, (GDKnr_threads > 4 ? GDKnr_threads/2 :
2), "mal_parallelism");
MT_sema_init( &mal_parallelism, (GDKnr_threads > 2 ? GDKnr_threads/2 :
1), "mal_parallelism");
Stefan
> Stefan
>
> >
> > tstAligned();
> > MCinit();
> > diff --git a/monetdb5/mal/mal.h b/monetdb5/mal/mal.h
> > --- a/monetdb5/mal/mal.h
> > +++ b/monetdb5/mal/mal.h
> > @@ -118,6 +118,7 @@ mal_export MT_Lock mal_remoteLock;
> > mal_export MT_Lock mal_profileLock ;
> > mal_export MT_Lock mal_copyLock ;
> > mal_export MT_Lock mal_delayLock ;
> > +mal_export MT_Sema mal_parallelism;
> >
> >
> > mal_export int mal_init(void);
> > diff --git a/monetdb5/mal/mal_interpreter.c
> > b/monetdb5/mal/mal_interpreter.c
> > --- a/monetdb5/mal/mal_interpreter.c
> > +++ b/monetdb5/mal/mal_interpreter.c
> > @@ -479,7 +479,13 @@ callMAL(Client cntxt, MalBlkPtr mb, MalS
> > ValPtr lhs;
> > InstrPtr pci = getInstrPtr(mb, 0);
> > RuntimeProfileRecord runtimeProfile;
> > -
> > +
> > +/*
> > + * Control the level of parallelism. The maximum number of
> > concurrent MAL plans
> > + * is determined by an environment variable. It is initially set
> > equal to the
> > + * number of cores, which may be too coarse.
> > + */
> > + MT_sema_down(&mal_parallelism,"mal_parallelism");
> > runtimeProfileInit(mb, &runtimeProfile, cntxt->flags &
> > memoryFlag);
> > #ifdef DEBUG_CALLMAL
> > mnstr_printf(cntxt->fdout, "callMAL\n");
> > @@ -518,8 +524,10 @@ callMAL(Client cntxt, MalBlkPtr mb, MalS
> > case PATcall:
> > case CMDcall:
> > default:
> > + MT_sema_up(&mal_parallelism,"mal_parallelism");
> > throw(MAL, "mal.interpreter", RUNTIME_UNKNOWN_INSTRUCTION);
> > }
> > + MT_sema_up(&mal_parallelism,"mal_parallelism");
> > if (cntxt->qtimeout && time(NULL) - stk->clock.tv_usec >
> > cntxt->qtimeout)
> > throw(MAL, "mal.interpreter", RUNTIME_QRY_TIMEOUT);
> > return ret;
> > _______________________________________________
> > checkin-list mailing list
> > checkin-l...@monetdb.org
> > http://mail.monetdb.org/mailman/listinfo/checkin-list
> >
> _______________________________________________
> developers-list mailing list
> developers-l...@monetdb.org
> http://mail.monetdb.org/mailman/listinfo/developers-list
>
------------------------------
_______________________________________________
developers-list mailing list
developers-l...@monetdb.org
http://mail.monetdb.org/mailman/listinfo/developers-list
End of developers-list Digest, Vol 5, Issue 2
*********************************************
------------------------------------------------------------------------------
Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current
with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
MVPs and experts. SALE $99.99 this month only -- learn more at:
http://p.sf.net/sfu/learnmore_122912
_______________________________________________
Monetdb-developers mailing list
Monetdb-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/monetdb-developers
