date:20130717

Re: [HACKERS] [PATCH] pgbench --throttle (submission 7 - with lag measurement)

2013-07-17 Thread Fabien COELHO




To clarify what state this is all in: Fabien's latest
pgbench-throttle-v15.patch is the ready for a committer version.  The
last two revisions are just tweaking the comments at this point, and
his version is more correct than my last one.


Got it. I will take care of this.


Please find attached an updated version which solves conflicts introduced 
by the progress patch.


--
Fabien.diff --git a/contrib/pgbench/pgbench.c b/contrib/pgbench/pgbench.c
index 23ee53c..4111e8c 100644
--- a/contrib/pgbench/pgbench.c
+++ b/contrib/pgbench/pgbench.c
@@ -137,6 +137,12 @@ int			unlogged_tables = 0;
 double		sample_rate = 0.0;
 
 /*
+ * When threads are throttled to a given rate limit, this is the target delay
+ * to reach that rate in usec.  0 is the default and means no throttling.
+ */
+int64		throttle_delay = 0;
+
+/*
  * tablespace selection
  */
 char	   *tablespace = NULL;
@@ -202,11 +208,13 @@ typedef struct
 	int			listen;			/* 0 indicates that an async query has been
  * sent */
 	int			sleeping;		/* 1 indicates that the client is napping */
+	boolthrottling; /* whether nap is for throttling */
 	int64		until;			/* napping until (usec) */
 	Variable   *variables;		/* array of variable definitions */
 	int			nvariables;
 	instr_time	txn_begin;		/* used for measuring transaction latencies */
 	instr_time	stmt_begin;		/* used for measuring statement latencies */
+	bool		is_throttled;	/* whether transaction throttling is done */
 	int			use_file;		/* index in sql_files for this client */
 	bool		prepared[MAX_FILES];
 } CState;
@@ -224,6 +232,9 @@ typedef struct
 	instr_time *exec_elapsed;	/* time spent executing cmds (per Command) */
 	int		   *exec_count;		/* number of cmd executions (per Command) */
 	unsigned short random_state[3];		/* separate randomness for each thread */
+	int64   throttle_trigger; 	/* previous/next throttling (us) */
+	int64   throttle_lag; 		/* total transaction lag behind throttling */
+	int64   throttle_lag_max; 	/* max transaction lag */
 } TState;
 
 #define INVALID_THREAD		((pthread_t) 0)
@@ -232,6 +243,8 @@ typedef struct
 {
 	instr_time	conn_time;
 	int			xacts;
+	int64   throttle_lag;
+	int64   throttle_lag_max;
 } TResult;
 
 /*
@@ -356,6 +369,7 @@ usage(void)
 		 -N, --skip-some-updates  skip updates of pgbench_tellers and pgbench_branches\n
 		 -P, --progress NUM   show thread progress report every NUM seconds\n
 		 -r, --report-latencies   report average latency per command\n
+		 -R, --rate SPEC  target rate in transactions per second\n
 		 -s, --scale=NUM  report this scale factor in output\n
 		 -S, --select-onlyperform SELECT-only transactions\n
 		 -t, --transactions   number of transactions each client runs 
@@ -898,17 +912,62 @@ doCustom(TState *thread, CState *st, instr_time *conn_time, FILE *logfile, AggVa
 {
 	PGresult   *res;
 	Command   **commands;
+	booltrans_needs_throttle = false;
 
 top:
 	commands = sql_files[st-use_file];
 
+	/*
+	 * Handle throttling once per transaction by sleeping.  It is simpler
+	 * to do this here rather than at the end, because so much complicated
+	 * logic happens below when statements finish.
+	 */
+	if (throttle_delay  ! st-is_throttled)
+	{
+		/*
+		 * Use inverse transform sampling to randomly generate a delay, such
+		 * that the series of delays will approximate a Poisson distribution
+		 * centered on the throttle_delay time.
+ *
+ * 1000 implies a 6.9 (-log(1/1000)) to 0.0 (log 1.0) delay multiplier.
+		 *
+		 * If transactions are too slow or a given wait is shorter than
+		 * a transaction, the next transaction will start right away.
+		 */
+		int64 wait = (int64)
+			throttle_delay * -log(getrand(thread, 1, 1000)/1000.0);
+
+		thread-throttle_trigger += wait;
+
+		st-until = thread-throttle_trigger;
+		st-sleeping = 1;
+		st-throttling = true;
+		st-is_throttled = true;
+		if (debug)
+			fprintf(stderr, client %d throttling INT64_FORMAT us\n,
+	st-id, wait);
+	}
+
 	if (st-sleeping)
 	{			/* are we sleeping? */
 		instr_time	now;
+		int64 now_us;
 
 		INSTR_TIME_SET_CURRENT(now);
-		if (st-until = INSTR_TIME_GET_MICROSEC(now))
+		now_us = INSTR_TIME_GET_MICROSEC(now);
+		if (st-until = now_us)
+		{
 			st-sleeping = 0;	/* Done sleeping, go ahead with next command */
+			if (st-throttling)
+			{
+/* Measure lag of throttled transaction relative to target */
+int64 lag = now_us - st-until;
+thread-throttle_lag += lag;
+if (lag  thread-throttle_lag_max)
+	thread-throttle_lag_max = lag;
+st-throttling = false;
+			}
+		}
 		else
 			return true;		/* Still sleeping, nothing to do here */
 	}
@@ -1095,6 +1154,15 @@ top:
 			st-state = 0;
 			st-use_file = (int) getrand(thread, 0, num_files - 1);
 			commands = sql_files[st-use_file];
+			st-is_throttled = false;
+			/*
+			 * No transaction is underway anymore, which means there is nothing
+			 * to

Re: [HACKERS] [PATCH] pgbench --throttle (submission 7 - with lag measurement)

2013-07-17 Thread Tatsuo Ishii

 To clarify what state this is all in: Fabien's latest
 pgbench-throttle-v15.patch is the ready for a committer version.  The
 last two revisions are just tweaking the comments at this point, and
 his version is more correct than my last one.

 Got it. I will take care of this.
 
 Please find attached an updated version which solves conflicts
 introduced by the progress patch.

Thanks, but I already solved the conflict and fixed some minor
indentation issues. Now I have question regarding the function.

./pgbench -p 5433 -S -T 10 -R 1 test
starting vacuum...end.
transaction type: SELECT only
scaling factor: 1
query mode: simple
number of clients: 1
number of threads: 1
duration: 10 s
number of transactions actually processed: 71339
average rate limit lag: 862.534 ms (max 2960.913 ms)
tps = 7133.745911 (including connections establishing)
tps = 7135.130810 (excluding connections establishing)

What does average rate limit lag mean? From the manual:

-R rate
--rate rate

Execute transactions targeting the specified rate instead of
running as fast as possible (the default). The rate is given in
transactions per second. If the targeted rate is above the maximum
possible rate these transactions can execute at, the rate limit
won't have any impact on results. The rate is targeted by starting
transactions along a Poisson-distributed event time line. When a
rate limit is active, the average and maximum transaction lag time
(the delay between the scheduled and actual transaction start
times) are reported in ms. High values indicate that the database
could not handle the scheduled load at some time.

So in my understanding the number shows the delay time before *each*
transaction starts. If my understanding is correct, why

71339 (total transactions) * 862.534 ms = 61532 sec

could exceed 10 seconds, which is the total run time?

Also I noticed small bug.

./pgbench -R 0 test
invalid rate limit: 0

Shouldn't this be treated as if -R is not specified? Actually in the program:

/*
 * When threads are throttled to a given rate limit, this is the target delay
 * to reach that rate in usec.  0 is the default and means no throttling.
 */
int64   throttle_delay = 0;

So it seems treating -R 0 means no throttling makes more sense to me.
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding optionally commit number in PG_VERSION_STR

2013-07-17 Thread Erik Rijkers

On Wed, July 17, 2013 05:27, Tom Lane wrote:
 Michael Paquier michael.paqu...@gmail.com writes:
 It happens that I work occasionally on multiple builds based on


FWIW, I've been doing this for while:


where $project is mostly the patchname, $commit_hash is extracted like this:

commit_hash=$( cd /home/aardvark/pg_stuff/git/master; git log | head -n 1 | cut 
--delimiter=  -f 2 );

version_string=${project}-${db_timestamp}-${commit_hash};

perl -i.original -ne 

s,(PACKAGE_.*[[:digit:]]+\.[[:digit:]]+(?:\.[[:digit:]]+)?(?:devel)?(?:(?:alpha|beta|rc)[[:digit:]]+)?),\\1-${version_string},;
  print; configure


Which will then give:

PostgreSQL 9.4devel-HEAD-20130717_0828-ffcf654547ef38555203e6d716f47b7065a0a87d 
on x86_64-unknown-linux-gnu, compiled by
gcc (GCC) 4.8.1, 64-bit


Btw, in an even more ugly hack I also stick some of that same config data in 
information_schema.sql_packages, with a
feature_id = 100:

select * from information_schema.sql_packages where cast(substring(feature_id 
from E'^PKG([[:digit:]]+)') as integer) = 100
 feature_id |feature_name| is_supported | is_verified_by |  
 comments
++--++--
 PKG100 | project name   | YES  | ej | HEAD
 PKG101 | patched| NO   | ej | NO
 PKG103 | build time | YES  | ej | 2013-07-17 
08:32:03.400521+02
 PKG104 | server_version | YES  | ej |
9.4devel-HEAD-20130717_0828-ffcf654547ef38555203e6d716f47b7065a0a87d
 PKG105 | server_version_num | YES  | ej | 90400
 PKG106 | port   | YES  | ej | 6544
 PKG110 | commit hash| YES  | ej | 
ffcf654547ef38555203e6d716f47b7065a0a87d
 PKG111 | catversion | YES  | ej | 201307161
 PKG112 | control version| YES  | ej | 937
(9 rows)



Needless to say, YMMV ...


Thanks,

Erik Rijkers





-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Cube extension point support // GSoC'13

2013-07-17 Thread Alexander Korotkov

On Fri, Jul 12, 2013 at 3:57 PM, Stas Kelvich stas.kelv...@gmail.comwrote:

Hello.

here is a patch adding to cube extension support for compressed
representation of point cubes. If cube is a point, i.e. has coincident
lower left and upper right corners, than only one corner is stored. First
bit of the cube header indicates whether the cube is point or not. Few
moments:

* Patch preserves binary compatibility with old indices

New representation of points will work in both index and heap. So, we
should speak about just compatibility with old cubes.

* All functions that create cubes from user input, check whether it is a
point or not
* All internal functions that can return cubes takes care of all cases
where a cube might become a point
* Added tests for checking correct point behavior

Also this patch includes adapted Alexander Korotkov's patch with kNN-based
ordering operator, which he wrote for postgresql-9.0beta1 with knngist
patch. More info there
http://www.postgresql.org/message-id/aanlktimhfaq6hcibrnk0tlcqmiyhywhwaq2zd87wb...@mail.gmail.com

I think ordering operator should be extracted into separated patch together
with another ordering operators of your project.

Patch contains some formatting issues. For example, this comment

/* Point can arise in two cases:
1) When argument is point and r == 0
2) When all coordinates was set to their averages */

should contain star sign on the beginning of each line. Also it will be
reflowed by pgindent. Correct formatting for this comment should look like
this:

77 matches

Mail list logo