Re: [HACKERS] 10.0

[Tom Lane]

Jeff Janes  writes:
> There are lots of improvement which get done to in-memory data
> structures that wouldn't require a pg_dump/pg_upgrade, which could in
> principle be ported into prior major versions if we had the resources
> (reviewing, testing, packaging) to do it, with an increase in the
> middle number.  Maybe we will never find the resources to do that, but
> why should that assumption get baked into the numbering scheme?

If we were to do that today, it'd just be an increase in the minor number.
I don't see why we'd need to change that approach.  The real blocking
factors there are about manpower and stability of the resulting code, not
about whether you need some special version numbering to describe it.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] 10.0

[Jeff Janes]

On Sat, May 14, 2016 at 7:51 PM, Greg Sabino Mullane  wrote:
>
> -BEGIN PGP SIGNED MESSAGE-
> Hash: RIPEMD160
>
>
>> Wasn't there some controversy about switching to major.minor versioning
>> this in -advocacy?
>>
>> http://www.postgresql.org/message-id/ee13fd2bb44cb086b457be34e81d5...@biglumber.com
>
> I proposed in that thread that we always increment the first number,
> never increment the second number, and increment the third exactly as we do
> now for bugfix releases.

I like this idea, roughly in line with SemVer.

There are lots of improvement which get done to in-memory data
structures that wouldn't require a pg_dump/pg_upgrade, which could in
principle be ported into prior major versions if we had the resources
(reviewing, testing, packaging) to do it, with an increase in the
middle number.  Maybe we will never find the resources to do that, but
why should that assumption get baked into the numbering scheme?

Cheers,

Jeff


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] 10.0

[Tom Lane]

"Greg Sabino Mullane"  writes:
> I think moving to a two-number format is a mistake: what exactly will 
> PQserverVersion() return in that case?

For, say, 10.2 it would be 12, equivalent to 10.0.2 under old style.

We could redefine it as being major plus four-digit minor, really.
Under the current maintenance scheme we never get anywhere near minor
release 99 before a branch dies ... but having some more breathing room
there would not be a bad thing.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] 10.0

[Greg Sabino Mullane]

-BEGIN PGP SIGNED MESSAGE-
Hash: RIPEMD160


> Wasn't there some controversy about switching to major.minor versioning
> this in -advocacy?
>
> http://www.postgresql.org/message-id/ee13fd2bb44cb086b457be34e81d5...@biglumber.com

I proposed in that thread that we always increment the first number, 
never increment the second number, and increment the third exactly as we do 
now for bugfix releases.

I think moving to a two-number format is a mistake: what exactly will 
PQserverVersion() return in that case? But I understand people have a hard 
time swallowing the "never change the middle number" portion of this idea.

Thus, here's a slight variation on that theme: what if we simply reversed the 
expectations of bumping the first number, and put the onus on people to 
change the *middle* number? Thus, the next release by default will be 10.0.0, 
the one after that will be by default 11.0.0, and so on. We can reserve the 
middle number for "lesser" releases - which may never happen - but at least 
we will have a mechanism to provide for them. So rather than the current spate 
of messages like this:

"This should be called 12.0 because of cool feature X and reason Y"

we would get the rare message like this:

"We don't really have much for this release, maybe it should just be 11.1?"


- -- 
Greg Sabino Mullane g...@turnstep.com
End Point Corporation http://www.endpoint.com/
PGP Key: 0x14964AC8 201605142247
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
-BEGIN PGP SIGNATURE-

iEYEAREDAAYFAlc34/UACgkQvJuQZxSWSsgQLgCeJS9v69R5C3BJxNy2ih1P2Tk8
xngAn0UQoSn6y3iOwMr5aHSKzuBh+3Xn
=wzw4
-END PGP SIGNATURE-




-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Losing memory references - SRF + SPI

[Anderson Carniel]

Thank you very much Joe.

I have followed the crosstab() implementation and understood the idea of
per query memory context. Now, I am using a unique SPI instance (which I
perform several sql queries), process the result, transform my result into
a tuplestore, close the SPI and done. It works perfectly.

I have a curiosity with regard to the tuplestore: is there a problem with
performance if my tuplestore form a big table with million of tuples? Other
question is regarding to SPI: is there a problem to use only  one instance
of SPI (for instance, if multiple users call the same function)?

Thank you again,
Anderson Carniel

2016-05-14 12:19 GMT-03:00 Joe Conway :

> On 05/13/2016 09:35 PM, Anderson Carniel wrote:
> > I am writing a function that returns a set of tuples by using also the
> > PostGIS. Thuis, I am using SRF too. It successfully returns the expected
> > result when it has at most 4 tuples. However, this is not the case when
> > more than 4 tuples have to be returned. When I debug the code, I found
> > that the problem is in my function that transforms a cstring after a
> > SPI_connection. It seems that this cstring is not valid anymore in the
> > moment of this conversion (see my comment below). I know that the SPI
> > uses different contexts when it init and finish its process. But, I
> > don't understand why I have this problem here. Please, note that I tried
> > to copy the values of the whole tuple, but I have the same problem:
> > system crash after the forth call of the function. Also note that I call
> > this function only in the init call of the SRF. Please I would
> > appreciate any suggestion and help.
>
> You probably need to allocate your returned values in a per query memory
> context. Take a look at how it is done in, for example, crosstab() in
> contrib/tablefunc.
>
> HTH,
>
> Joe
>
> --
> Crunchy Data - http://crunchydata.com
> PostgreSQL Support for Secure Enterprises
> Consulting, Training, & Open Source Development
>
>

Re: [HACKERS] 10.0

[Josh berkus]

On 05/13/2016 07:18 PM, Mark Dilger wrote:
> My main concern is that a commitment to never, ever break backwards
> compatibility is a commitment to obsolescence.  It therefore makes sense to
> reserve room in the numbering scheme to be clear and honest about when
> backwards compatibility has been broken.  The major number is the normal
> place to do that.

The problem with that idea is that *minor* backwards compatibility
breakage is much more likely in each-and-every version than major
breakage is at any time in the foreseeable future.  The last major
breakage we really had was version 8.3, which if we'd been going by
compatibility should have been 9.0 (also for other reasons).

And if we adopt the "backwards compatibility" approach, then we'll just
be switching from the argument we're having now to the argument of "is
this enough breakage to rate a .0?  Yes/No?".  Which argument will be
just as long as this one.

So, my vote is now +1 to go to the 2-part numbering scheme.

-- 
--
Josh Berkus
Red Hat OSAS
(any opinions are my own)


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Parallel pg_dump's error reporting doesn't work worth squat

[Tom Lane]

[ getting back to a complaint I made in December ]

I wrote:
> ... the pg_dump process has crashed with a SIGPIPE without printing
> any message whatsoever, and without coming anywhere near finishing the
> dump.

> A bit of investigation says that this is because somebody had the bright
> idea that worker processes could report fatal errors back to the master
> process instead of just printing them to stderr.  So when the workers
> fail to establish connections (because of the password problem cited in
> #13727), they don't tell me about that.  Oh no, they send those errors
> back up to the pipe to the parent, and then die silently.  Meanwhile,
> the parent is trying to send them commands, and since it doesn't protect
> itself against SIGPIPE on the command pipes, it crashes without ever
> reporting anything.  If you aren't paying close attention, you wouldn't
> even realize you didn't get a completed dump.

Attached is a proposed patch for this.  It reverts exit_horribly() to
what it used to be pre-9.3, ie just print the message on stderr and die.
The master now notices child failure by observing EOF on the status pipe.
While that works automatically on Unix, we have to explicitly close the
child side of the pipe on Windows (could someone check that that works?).
I also fixed the parent side to ignore SIGPIPE earlier, so that it won't
just die if it was in the midst of sending to the child.

BTW, after having spent the afternoon staring at it, I will assert with
confidence that pg_dump/parallel.c is an embarrassment to the project.
It is chock-full of misleading, out-of-date, and/or outright wrong
comments, useless or even wrong Asserts, ill-designed APIs, code that
works quite differently in Unix and Windows cases without a shred of
commentary about the fact, etc etc.  I have mostly resisted the temptation
to do cosmetic cleanup in the attached, but this code really needs some
editorial attention.

regards, tom lane

diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index 9167294..f650d3f 100644
*** a/src/bin/pg_dump/parallel.c
--- b/src/bin/pg_dump/parallel.c
*** static ShutdownInformation shutdown_info
*** 77,84 
  static const char *modulename = gettext_noop("parallel archiver");
  
  static ParallelSlot *GetMyPSlot(ParallelState *pstate);
- static void parallel_msg_master(ParallelSlot *slot, const char *modulename,
- 	const char *fmt, va_list ap) pg_attribute_printf(3, 0);
  static void archive_close_connection(int code, void *arg);
  static void ShutdownWorkersHard(ParallelState *pstate);
  static void WaitForTerminatingWorkers(ParallelState *pstate);
--- 77,82 
*** GetMyPSlot(ParallelState *pstate)
*** 163,227 
  }
  
  /*
-  * Fail and die, with a message to stderr.  Parameters as for write_msg.
-  *
-  * This is defined in parallel.c, because in parallel mode, things are more
-  * complicated. If the worker process does exit_horribly(), we forward its
-  * last words to the master process. The master process then does
-  * exit_horribly() with this error message itself and prints it normally.
-  * After printing the message, exit_horribly() on the master will shut down
-  * the remaining worker processes.
-  */
- void
- exit_horribly(const char *modulename, const char *fmt,...)
- {
- 	va_list		ap;
- 	ParallelState *pstate = shutdown_info.pstate;
- 	ParallelSlot *slot;
- 
- 	va_start(ap, fmt);
- 
- 	if (pstate == NULL)
- 	{
- 		/* Not in parallel mode, just write to stderr */
- 		vwrite_msg(modulename, fmt, ap);
- 	}
- 	else
- 	{
- 		slot = GetMyPSlot(pstate);
- 
- 		if (!slot)
- 			/* We're the parent, just write the message out */
- 			vwrite_msg(modulename, fmt, ap);
- 		else
- 			/* If we're a worker process, send the msg to the master process */
- 			parallel_msg_master(slot, modulename, fmt, ap);
- 	}
- 
- 	va_end(ap);
- 
- 	exit_nicely(1);
- }
- 
- /* Sends the error message from the worker to the master process */
- static void
- parallel_msg_master(ParallelSlot *slot, const char *modulename,
- 	const char *fmt, va_list ap)
- {
- 	char		buf[512];
- 	int			pipefd[2];
- 
- 	pipefd[PIPE_READ] = slot->pipeRevRead;
- 	pipefd[PIPE_WRITE] = slot->pipeRevWrite;
- 
- 	strcpy(buf, "ERROR ");
- 	vsnprintf(buf + strlen("ERROR "),
- 			  sizeof(buf) - strlen("ERROR "), fmt, ap);
- 
- 	sendMessageToMaster(pipefd, buf);
- }
- 
- /*
   * A thread-local version of getLocalPQExpBuffer().
   *
   * Non-reentrant but reduces memory leakage. (On Windows the memory leakage
--- 161,166 
*** getThreadLocalPQExpBuffer(void)
*** 271,277 
  
  /*
   * pg_dump and pg_restore register the Archive pointer for the exit handler
!  * (called from exit_horribly). This function mainly exists so that we can
   * keep shutdown_info in file scope only.
   */
  void
--- 210,216 
  
  /*
   * pg_dump and pg_restore register the Archive pointer for the exit handler
!  * (called from exit_nicely). This function mainly 

Re: [HACKERS] exit() behavior on Windows?

[Tom Lane]

I wrote:
> If it's just a quick kill, then there's a totally separate bug here for
> Windows.  What is likely to happen with the current coding is that a
> failing child thread will queue its error message into the communication
> pipe, and then exit(1), thereby killing the whole pg_dump process before
> the master thread can receive and print the error.

Ah, scratch that --- I hadn't noticed the badly-underdocumented #ifdef
in exit_nicely().

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] 10.0

[Michael Banck]

On Fri, May 13, 2016 at 08:55:20PM -0400, David G. Johnston wrote:
> The opinion seems to be that major.0 is some kind of magic incantation in
> the broader world of users...

>From my reading of the thread, while certainly that is the general
definition of a .0, having infrequent .0 releases is not very practical
for PostgreSQL because the major versions are not that different from
each other and all are treated the same development-wise. So it would be
a huge drain on the project to discuss which major version should be a
.0 unless planning towards them steps up significantly.

So I think the (slight) consensus is more that all major versions are
mostly equal and hence only one version number is needed.


Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] 10.0

[Michael Banck]

On Fri, May 13, 2016 at 05:31:00PM -0400, David G. Johnston wrote:
> The underlying premise, for me, of choosing .4 or .5  was that presently we
> discontinue support after 5 years/releases.  A new .0 would come out just
> as we discontinue the previous .0

Well maybe the 5 year support cycle would be nice to encode, but how is
.0 different from .1 then?  You make sound like .0 would be similar to
Ubuntu's LTS which is not the case, unless you want to propose that only
.0 releases get 5 years and not the in-betweens? That'd be a shame.

Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Perf Benchmarking and regression.

[Fabien COELHO]

These raw tps suggest that {backend,bgwriter}_flush_after should better be
zero for this kind of load.Whether it should be the default is unclear yet,
because as Andres pointed out this is one kind of load.


FWIW, I don't think {backend,bgwriter} are the same here. It's primarily
backend that matters.


Indeed, I was a little hasty to put bgwriter together based on this 
report.


I'm a little wary of "bgwriter_flush_after" though, I would not be 
surprised if someone reports some regressions, although probably not with 
a pgbench tpcb kind of load.


--
Fabien.


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] 10.0

[Petr Jelinek]

+1 for going with 10.0 after 9.6 and 11.0 afterwards, etc.

It will hopefully both end these discussions and remove the confusion 
the current versioning scheme has (I too heard way to many times about 
people using postgres8 or postgres9).


For those saying this is version inflation. I don't see the problem, we 
are project with long history, which releases major version every year, 
I don't see why version number shouldn't reflect that.


About changing first digit only when we break compatibility with 
applications for example by removing money type. We do have app breaking 
incompatibilities in almost every major version so in effect we'd have 
to bump the first digit every time anyway if we went with that schema (I 
still remember working on db that was kept on 8.3 till EOL just because 
we changes hashtext implementation, or db which was kept back because of 
removal plgpsql rename clause where nobody wanted to rewrite couple 
thousand function which used it and there are many examples like that).


--
  Petr Jelinek  http://www.2ndQuadrant.com/
  PostgreSQL Development, 24x7 Support, Training & Services


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] 10.0

[Martín Marqués]

El 13/05/16 a las 15:36, Josh berkus escribió:
> On 05/13/2016 11:31 AM, Alvaro Herrera wrote:
>> Josh berkus wrote:
>>  
>>> Anyway, can we come up with a consensus of some minimum changes it will
>>> take to make the next version 10.0?
>>
>> I think the next version should be 10.0 no matter what changes we put
>> in.
>>
> 
> Well, if we adopt 2-part version numbers, it will be.  Maybe that's the
> easiest thing?  Then we never have to have this discussion again, which
> certainly appeals to me ...

Wasn't there some controversy about switching to major.minor versioning
this in -advocacy?

http://www.postgresql.org/message-id/ee13fd2bb44cb086b457be34e81d5...@biglumber.com

IMO, this versioning is pretty good and people understand it well, with
the other will be using postgres 13 by 2020, which isn't far away. ;)

-- 
Martín Marquéshttp://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] 10.0

[Martín Marqués]

El 13/05/16 a las 15:31, Alvaro Herrera escribió:
> Josh berkus wrote:
>  
>> Anyway, can we come up with a consensus of some minimum changes it will
>> take to make the next version 10.0?
> 
> I think the next version should be 10.0 no matter what changes we put
> in.

+1

And another +1 on Tom's opinion on it being too late after beta1 has
been released.

-- 
Martín Marquéshttp://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] exit() behavior on Windows?

[Tom Lane]

Rather belatedly, I've started to look into a fix for pg_dump's
problem with error reporting in parallel mode:
http://www.postgresql.org/message-id/2458.1450894...@sss.pgh.pa.us

One of the issues here is that pg_dump uses threads not subprocesses
to do parallelism on Windows, which means that totally different
reasoning is needed for that case.  I'm looking in particular at the
use of exit().  What I gather from Microsoft's documentation is that
exit() will kill the whole process not just the calling thread ---
can anyone confirm that that's correct?  Are the other threads just
killed asynchronously, or do they get a chance to shut down?

If it's just a quick kill, then there's a totally separate bug here for
Windows.  What is likely to happen with the current coding is that a
failing child thread will queue its error message into the communication
pipe, and then exit(1), thereby killing the whole pg_dump process before
the master thread can receive and print the error.  The visible result
would be about the same as what I described on Linux, ie failure exit
with no error message reported.  But the mechanism is different.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Perf Benchmarking and regression.

[Andres Freund]

On 2016-05-14 18:49:27 +0200, Fabien COELHO wrote:
> 
> Hello,
> 
> > Please find the results for the following 3 scenarios with unpatched master:
> > 
> > 1. Default settings for *_flush_after : TPS = *10677.662356*
> > 2. backend_flush_after=0, rest defaults : TPS = *18452.655936*
> > 3. backend_flush_after=0, bgwriter_flush_after=0,
> > wal_writer_flush_after=0, checkpoint_flush_after=0 : TPS = *18614.479962*
> 
> Thanks for these runs.

Yes!

> These raw tps suggest that {backend,bgwriter}_flush_after should better be
> zero for this kind of load.Whether it should be the default is unclear yet,
> because as Andres pointed out this is one kind of load.

FWIW, I don't think {backend,bgwriter} are the same here. It's primarily
backend that matters.  This is treating the os page cache as an
extension of postgres' buffer cache. That really primarily matters for
backend_, because otherwise backends spend time waiting for IO.


Andres


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Perf Benchmarking and regression.

[Fabien COELHO]

Hello,


Please find the results for the following 3 scenarios with unpatched master:

1. Default settings for *_flush_after : TPS = *10677.662356*
2. backend_flush_after=0, rest defaults : TPS = *18452.655936*
3. backend_flush_after=0, bgwriter_flush_after=0,
wal_writer_flush_after=0, checkpoint_flush_after=0 : TPS = *18614.479962*


Thanks for these runs.

These raw tps suggest that {backend,bgwriter}_flush_after should better be 
zero for this kind of load.Whether it should be the default is unclear 
yet, because as Andres pointed out this is one kind of load.


Note: these options have been added to smooth ios over time and to help 
avoid "io panics" on sync, especially with HDDs without a large BBU cache 
in front. The real benefit is that the performance are much more constant 
over time, and pg is much more responsive.


If you do other runs, it would be nice to report some stats about tps 
variability (eg latency & latency stddev which should be in the report). 
For experiments I did I used to log "-P 1" output (tps every second) and 
to compute stats on these tps (avg, stddev, min, q1, median, q3, max, pc 
of time with tps below a low threshold...), which provides some indication 
of the overall tps distribution.


--
Fabien


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Losing memory references - SRF + SPI

[Joe Conway]

On 05/13/2016 09:35 PM, Anderson Carniel wrote:
> I am writing a function that returns a set of tuples by using also the
> PostGIS. Thuis, I am using SRF too. It successfully returns the expected
> result when it has at most 4 tuples. However, this is not the case when
> more than 4 tuples have to be returned. When I debug the code, I found
> that the problem is in my function that transforms a cstring after a
> SPI_connection. It seems that this cstring is not valid anymore in the
> moment of this conversion (see my comment below). I know that the SPI
> uses different contexts when it init and finish its process. But, I
> don't understand why I have this problem here. Please, note that I tried
> to copy the values of the whole tuple, but I have the same problem:
> system crash after the forth call of the function. Also note that I call
> this function only in the init call of the SRF. Please I would
> appreciate any suggestion and help.

You probably need to allocate your returned values in a per query memory
context. Take a look at how it is done in, for example, crosstab() in
contrib/tablefunc.

HTH,

Joe

-- 
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development



signature.asc
Description: OpenPGP digital signature

Re: [HACKERS] 10.0

[Joshua D. Drake]

On 05/14/2016 07:08 AM, Robert Haas wrote:

On Fri, May 13, 2016 at 8:00 PM, Tom Lane  wrote:

Any project that starts inflating its numbering scheme sends a message to
users of the form, "hey, we've just been taken over by marketing people, and
software quality will go down from now on."


I don't think this is about version number inflation, but actually more
the opposite.  What you're calling the major number is really a marketing
number.  There is not a technical distinction between major releases where
we choose to bump the first number and those where we choose to bump the
second.  It's all about marketing.  So to me, merging those numbers would
be an anti-marketing move.  I think it's a good move: it would be more
honest and transparent about what the numbers mean, not less so.


"Marketing" and "honesty" are not antonyms, and thinking that
marketing is unimportant for the success of the project is not
correct.


+1

JD



--
Command Prompt, Inc.  http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] 10.0

[Robert Haas]

On Fri, May 13, 2016 at 11:39 AM, Tom Lane  wrote:
> Robert Haas  writes:
>> There is a long-running thread on pgsql-hackers on whether 9.6 should
>> instead be called 10.0.
>
> First I've seen it mentioned here.
>
> I think you are just about exactly one week too late to bring this up.
> Once we've shipped a beta, rebranding is way too confusing.

So, not that it makes any difference at this point, but this seems to
fly in the face of what you said before.

http://www.postgresql.org/message-id/24567.1456864...@sss.pgh.pa.us

Unless you thought that beta wasn't going to be out until the fall.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] 10.0

[Robert Haas]

On Fri, May 13, 2016 at 8:00 PM, Tom Lane  wrote:
>> Any project that starts inflating its numbering scheme sends a message to
>> users of the form, "hey, we've just been taken over by marketing people, and
>> software quality will go down from now on."
>
> I don't think this is about version number inflation, but actually more
> the opposite.  What you're calling the major number is really a marketing
> number.  There is not a technical distinction between major releases where
> we choose to bump the first number and those where we choose to bump the
> second.  It's all about marketing.  So to me, merging those numbers would
> be an anti-marketing move.  I think it's a good move: it would be more
> honest and transparent about what the numbers mean, not less so.

"Marketing" and "honesty" are not antonyms, and thinking that
marketing is unimportant for the success of the project is not
correct.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Re: [HACKERS] Re: [HACKERS] Re: [HACKERS] Re: [HACKERS] Windows service is not starting so there’s message in log: FATAL: "could not create shared memory segment “Global/PostgreSQL.851401618

[Robert Haas]

On Mon, May 9, 2016 at 3:17 AM, Michael Paquier
 wrote:
> On Tue, Mar 22, 2016 at 1:56 PM, Amit Kapila  wrote:
>> So as far as I can see there are two ways to resolve this issue, one is to
>> retry generation of dsm name if CreateFileMapping returns EACCES and second
>> is to append data_dir name to dsm name as the same is done for main shared
>> memory, that will avoid the error to occur.  First approach has minor flaw
>> that if CreateFileMapping returns EACCES due to reason other then duplicate
>> dsm name which I am not sure is possible to identify, then we should report
>> error instead try to regenerate the name
>>
>> Robert and or others, can you share your opinion on what is the best way to
>> proceed for this issue.
>
> For my 2c here, the approach using GetSharedMemName to identify the
> origin of a dynamic shared memory segment looks more solid in terms of
> robustness and collision handling. Retrying a segment is never going
> to be completely water-proof.

Why not?  I mean, there are ~2^32 possible segment handles, and not
all that many of them can be in use.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Re: [HACKERS] Re: [HACKERS] Re: [HACKERS] Windows service is not starting so there’s message in log: FATAL: "could not create shared memory segment “Global/PostgreSQL.851401618”: Permission

[Robert Haas]

On Tue, Mar 22, 2016 at 12:56 AM, Amit Kapila  wrote:
>> >> Yes, same random number generation is not the problem. In windows apart
>> >> from EEXIST error, EACCES also needs to be validated and returned for
>> >> new random number generation, instead of throwing an error.
>> >
>> > Doing the same handling for EACCES doesn't seem to be sane because if
>> > EACCES
>> > came for reason other than duplicate dsm name, then we want to report
>> > the
>> > error instead of trying to regenerate the name.  I think here fix should
>> > be
>> > to append data_dir path as we do for main shared memory.
>>
>> Yes, EACCES may be possible other than duplicate dsm name.
>
> So as far as I can see there are two ways to resolve this issue, one is to
> retry generation of dsm name if CreateFileMapping returns EACCES and second
> is to append data_dir name to dsm name as the same is done for main shared
> memory, that will avoid the error to occur.  First approach has minor flaw
> that if CreateFileMapping returns EACCES due to reason other then duplicate
> dsm name which I am not sure is possible to identify, then we should report
> error instead try to regenerate the name
>
> Robert and or others, can you share your opinion on what is the best way to
> proceed for this issue.

I think we should retry on EACCES.  Possibly we should do other things
too, but at least that.  It completely misses the point of retrying on
EEXIST if we don't retry on other error codes that can also be
generated when the segment already exists.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Perf Benchmarking and regression.

[Ashutosh Sharma]

Hi,

Please find the results for the following 3 scenarios with unpatched master:

1. Default settings for *_flush_after : TPS = *10677.662356*
2. backend_flush_after=0, rest defaults : TPS = *18452.655936*
3. backend_flush_after=0, bgwriter_flush_after=0,
wal_writer_flush_after=0, checkpoint_flush_after=0 : TPS = *18614.479962*

With Regards,
Ashutosh Sharma
EnterpriseDB: http://www.enterprisedb.com

On Fri, May 13, 2016 at 7:50 PM, Robert Haas  wrote:

> On Fri, May 13, 2016 at 7:08 AM, Ashutosh Sharma 
> wrote:
> > Following are the performance results for read write test observed with
> > different numbers of "backend_flush_after".
> >
> > 1) backend_flush_after = 256kb (32*8kb), tps = 10841.178815
> > 2) backend_flush_after = 512kb (64*8kb), tps = 11098.702707
> > 3) backend_flush_after = 1MB (128*8kb), tps = 11434.964545
> > 4) backend_flush_after = 2MB (256*8kb), tps = 13477.089417
>
> So even at 2MB we don't come close to recovering all of the lost
> performance.  Can you please test these three scenarios?
>
> 1. Default settings for *_flush_after
> 2. backend_flush_after=0, rest defaults
> 3. backend_flush_after=0, bgwriter_flush_after=0,
> wal_writer_flush_after=0, checkpoint_flush_after=0
>
> --
> Robert Haas
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>

Re: [HACKERS] Just-in-time compiling things

[Konstantin Knizhnik]

On 05/14/2016 12:10 PM, Andreas Seltenreich wrote:

Konstantin Knizhnik writes:


Latest information from ISP RAS guys: them have made good progress
since February: them have rewritten most of methods of Scan, Aggregate
and Join to LLVM API.

Is their work available somewhere?  I'm experimenting in that area as
well, although I'm using libFirm instead of LLVM.  I wonder what their
motivation to rewrite backend code in LLVM IR was, since I am following
the approach of keeping the IR around when compiling the vanilla
postgres C code, possibly inlining it during JIT and then doing
optimizations on this IR.  That way the logic doesn't have to be
duplicated.


The work is not yet completed but finally it will be definitely put to open 
source.
I am going to talk a little bit about this project at PGcon in Ottawa at 
lighting talks, although I do not know details of the project myself.
The main difference of their approach comparing with Vitesse DB is that them 
implement a way of automatic conversion of PostgreSQL operators to LLVM IR.
So instead of rewritting ~2000 operators manually (a lot of work and errors), them implement converter which transform the code of this operators to ... C++ code producing LLVM IR. So manually them need to rewrite only plan nodes. Them already implemented 
most of nodes (SeqScan, Sort, HashJoin,...) which allows to execute all TPC-H queries. Result will be published soon. The larghest advantage is definitely at Q1 - about 4 times. It is worser than Vitesse DB (8 times) and with manually written operators (7 
times). The most probable reason causing such performance penalty is overflow checking: in manually written LLVM code it can be done in more efficient way using correspondent assembler instruction than in code automatically converted from standard C.

But ISP RAS guys are going to fix this problem and improve automatic conversion 
quality.

I include in CC members of ISP RAS team - you can ask them questions directly.





regrads
Andreas



--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] 10.0

[Christoph Berg]

Re: Álvaro Hernández Tortosa 2016-05-14 <5736f966.3040...@8kdata.com>
> Having said that, I believe having a single major number is a more
> effective marketing. Non major-major versions may make the release look like
> a "probably not worth" upgrade. People may hold their breath until a
> major-major upgrade, specially if people support this idea in forums like
> saying: "10.0 will come with amazing features, because version is bumped
> from 9.6".
> 
> So +1 to call 10.0 the next version and 11.0 the one after that.

+1.

Another data point: Debian did exactly this x.y.z -> x.y switch (with
exactly the same major/minor version number semantics re major/minor
upgrades) a few years ago [*], and it was a real relief. "Debian 9"
just sounds so much better than "Debian 9.6".

IMHO getting rid of the problem that people are always saying "I'm
using Postgres 9" when asked about the major version should be enough
reason to change to a x.y scheme.

Christoph

[*] There was a period of two releases where we bumped the x and y was
kept 0, before the y was ultimately dropped.
(3.0 - 3.1 - 4.0 - 5.0 - 6.0 - 7 - 8)


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] 10.0

[Álvaro Hernández Tortosa]

On 14/05/16 02:00, Tom Lane wrote:

[...]

I don't think this is about version number inflation, but actually more
the opposite.  What you're calling the major number is really a marketing
number.  There is not a technical distinction between major releases where
we choose to bump the first number and those where we choose to bump the
second.  It's all about marketing.  So to me, merging those numbers would
be an anti-marketing move.  I think it's a good move: it would be more
honest and transparent about what the numbers mean, not less so.


If having two "major" numbers is a marketing game, and if it works 
in such a way, I'd immediately say let's keep it. Decisions like the one 
debated here should be driven more from what is going to help user 
adoption rather than -hackers personal taste. BTW, none of these 
approaches seem dishonest to me.


Having said that, I believe having a single major number is a more 
effective marketing. Non major-major versions may make the release look 
like a "probably not worth" upgrade. People may hold their breath until 
a major-major upgrade, specially if people support this idea in forums 
like saying: "10.0 will come with amazing features, because version is 
bumped from 9.6".


So +1 to call 10.0 the next version and 11.0 the one after that.

Álvaro

--
Álvaro Hernández Tortosa


---
8Kdata



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Just-in-time compiling things (was: asynchronous and vectorized execution)

[Andreas Seltenreich]

Konstantin Knizhnik writes:

> Latest information from ISP RAS guys: them have made good progress
> since February: them have rewritten most of methods of Scan, Aggregate
> and Join to LLVM API.

Is their work available somewhere?  I'm experimenting in that area as
well, although I'm using libFirm instead of LLVM.  I wonder what their
motivation to rewrite backend code in LLVM IR was, since I am following
the approach of keeping the IR around when compiling the vanilla
postgres C code, possibly inlining it during JIT and then doing
optimizations on this IR.  That way the logic doesn't have to be
duplicated.

regrads
Andreas


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers