Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

On 2017-10-11 08:54:10 -0700, Andres Freund wrote: > On 2017-10-11 10:53:56 +0200, Alvaro Herrera wrote: > > Maybe it'd be a good idea to push 0001 with some user of restrict ahead > > of the rest, just to see how older msvc reacts. > > Can do. Not quite sure which older user yet, but I'm sure I

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

On Wed, Oct 11, 2017 at 2:47 PM, Andres Freund wrote: >> If do nothing, it's unlikely we'd ever get rid of the compat function. > > I think that's ok. Yeah. I mean, it seems similar to what happened with heap_formtuple: the last in-tree users of that function went away in

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

Hi, On 2017-10-11 18:05:32 +0200, Alvaro Herrera wrote: > Well, my concern is to ensure that extension authors take advantage of > the optimized implementation. If we never let them know that we've > rewritten things, most are not going to realize they can make their > extensions faster with a

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

Andres Freund wrote: > On 2017-10-11 10:53:56 +0200, Alvaro Herrera wrote: > > I wonder if it'd be a good idea to nag external users about pq_sendint > > usage (is a #warning possible?). > > Think we'd need some separate infrastructure, i.e. for gcc ending up > with __attribute__((deprecated)).

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

On 2017-10-11 10:53:56 +0200, Alvaro Herrera wrote: > Andres Freund wrote: > > Hi, > > > > On 2017-10-03 13:58:37 -0400, Robert Haas wrote: > > > On Tue, Oct 3, 2017 at 12:23 PM, Andres Freund wrote: > > > > Makes sense? > > > > > > Yes. > > > > Here's an updated version of

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

Andres Freund wrote: > Hi, > > On 2017-10-03 13:58:37 -0400, Robert Haas wrote: > > On Tue, Oct 3, 2017 at 12:23 PM, Andres Freund wrote: > > > Makes sense? > > > > Yes. > > Here's an updated version of this patchset. Maybe it'd be a good idea to push 0001 with some user

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

Hi, On 2017-10-03 13:58:37 -0400, Robert Haas wrote: > On Tue, Oct 3, 2017 at 12:23 PM, Andres Freund wrote: > > Makes sense? > > Yes. Here's an updated version of this patchset. Changes: - renamed pq_send$type_pre to pq_write*type - renamed

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

On Tue, Oct 3, 2017 at 12:23 PM, Andres Freund wrote: > Makes sense? Yes. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

On 2017-10-03 11:06:08 -0400, Robert Haas wrote: > On Tue, Oct 3, 2017 at 3:55 AM, Andres Freund wrote: > > Attached is a revised version of this patchset. > > I don't much like the functions with "_pre" affixed to their names. > It's not at all clear that "pre" means

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

On Tue, Oct 3, 2017 at 3:55 AM, Andres Freund wrote: > Attached is a revised version of this patchset. I don't much like the functions with "_pre" affixed to their names. It's not at all clear that "pre" means "preallocated"; it sounds more like you're doing something ahead

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

Hi, Attached is a revised version of this patchset. I'd like to get some input on two points: 1) Does anybody have a better idea than the static buffer in SendRowDescriptionMessage()? That's not particularly pretty, but there's not really a convenient stringbuffer to use when called from

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

On 2017-09-29 17:56:10 -0400, Tom Lane wrote: > Andres Freund writes: > > Does anybody have an opinion on whether we'll want to convert examples > > like testlibpq3.c (included in libpq.sgml) too? I'm inclined not to, > > because currently using pg_bswap.h requires c.h

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

Andres Freund writes: > Does anybody have an opinion on whether we'll want to convert examples > like testlibpq3.c (included in libpq.sgml) too? I'm inclined not to, > because currently using pg_bswap.h requires c.h presence (just for a few > typedefs and configure data).

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

On 2017-09-28 14:23:45 +0900, Michael Paquier wrote: > On Thu, Sep 28, 2017 at 1:31 PM, Andres Freund wrote: > > On September 27, 2017 9:06:49 PM PDT, Andres Freund > > wrote: > >>On 2017-09-28 00:01:53 -0400, Tom Lane wrote: > >>> Could we please not

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

On Fri, Sep 29, 2017 at 5:02 AM, tushar wrote: > Case 3- TIME=1000 > > PG HEAD =>tps = 1061.031463 (excluding connections establishing) > PG HEAD+patch => tps= 8011.784839(3.30+% vs. head) Going from 1061 tps to 8011 tps is not a 3.3% gain. I assume you garbled

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

On 09/27/2017 10:50 PM, Andres Freund wrote: This'll allow the later patches to allow the compiler to perform the relevant optimizations. It also allows to optimize e.g. pq_sendint64() to avoid having to do multiple byteswaps. After applying all the required patches, able to see some

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

On Thu, Sep 28, 2017 at 1:31 PM, Andres Freund wrote: > On September 27, 2017 9:06:49 PM PDT, Andres Freund > wrote: >>On 2017-09-28 00:01:53 -0400, Tom Lane wrote: >>> Could we please not perpetuate the brain-dead "s" and "l" suffixes >>> on these names?

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

On September 27, 2017 9:06:49 PM PDT, Andres Freund wrote: >On 2017-09-28 00:01:53 -0400, Tom Lane wrote: >> Andres Freund writes: >> > Attached is an extension of the already existing pg_bswap.h that >> > a) adds 16 bit support >> > b) moves everything

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

On 2017-09-28 00:01:53 -0400, Tom Lane wrote: > Andres Freund writes: > > Attached is an extension of the already existing pg_bswap.h that > > a) adds 16 bit support > > b) moves everything to inline functions, removing multiple evaluation > >hazards that were present

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

Andres Freund writes: > Attached is an extension of the already existing pg_bswap.h that > a) adds 16 bit support > b) moves everything to inline functions, removing multiple evaluation >hazards that were present everywhere. > c) adds pg_nto{s,l,ll} and pg_hton{s,l,ll}

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

On Thu, Sep 28, 2017 at 2:20 AM, Andres Freund wrote: > This'll allow the later patches to allow the compiler to perform the > relevant optimizations. It also allows to optimize e.g. pq_sendint64() > to avoid having to do multiple byteswaps. I guess that you could clean up

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

Hi, On 2017-09-13 23:34:18 -0700, Andres Freund wrote: >I'm not yet super sure about the implementation. For one, I'm not >sure this shouldn't instead be stringinfo.h functions, with very very >tiny pqformat.h wrappers. But conversely I think it'd make a lot of >sense for the

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

On Mon, Sep 18, 2017 at 1:38 PM, Mithun Cy wrote: > On Sat, Sep 16, 2017 at 3:03 AM, Andres Freund wrote: > So I think performance gain is visible. We saved a good amount of > execution cycle in SendRowDescriptionMessagewhen(my callgrind report >

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

Hi Thom, On 2017-09-15 22:05:35 +0100, Thom Brown wrote: > Okay, I've retested it using a prepared statement executed 100,000 > times (which selects a single row from a table with 101 columns, each > column is 42-43 characters long, and 2 rows in the table), and I get > the following: > >

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

On 15 September 2017 at 19:23, Andres Freund wrote: > Hi Thom, > > Thanks for taking a whack at this! > > On 2017-09-15 12:16:22 +0100, Thom Brown wrote: >> I've run a fairly basic test with a table with 101 columns, selecting >> a single row from the table and I get the

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

Hi Thom, Thanks for taking a whack at this! On 2017-09-15 12:16:22 +0100, Thom Brown wrote: > I've run a fairly basic test with a table with 101 columns, selecting > a single row from the table and I get the following results: > > > Columns with 1-character names: > > master (80 jobs, 80

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

On 14 September 2017 at 07:34, Andres Freund wrote: > Hi, > > When running workloads that include fast queries with a lot of columns, > SendRowDescriptionMessage(), and the routines it calls, becomes a > bottleneck. Besides syscache lookups (see [1] and [2]) a major cost of >