Re: Query about pg asynchronous processing support

2021-03-26 Thread legrand legrand
Hi, You should search informations about postgres hooks like thoses used in extension pg_stat_statements https://github.com/postgres/postgres/blob/master/contrib/pg_stat_statements/pg_stat_statements.c And about background capabilities as thoses used in extension pg_background

Re: WIP: System Versioned Temporal Table

2021-01-15 Thread legrand legrand
Hello, it seems that Oracle (11R2) doesn't add the Start and End timestamp columns and permit statement like select * from tt union select * from tt AS OF TIMESTAMP (SYSTIMESTAMP - INTERVAL '6' SECOND) minus select * from tt VERSIONS BETWEEN TIMESTAMP (SYSTIMESTAMP - INTERVAL '6' second) and

Re: pg_stat_statements oddity with track = all

2020-12-03 Thread legrand legrand
Hi Julien, > The extra field I've proposed would increase the number of records, as it > needs to be a part of the key. To get an increase in the number of records that means that the same statement would appear at top level AND nested level. This seems a corner case with very low

Re: pg_stat_statements oddity with track = all

2020-12-02 Thread legrand legrand
Hi, a crazy idea: - add a parent_statement_id column that would be NULL for top level queries - build statement_id for nested queries based on the merge of: a/ current_statement_id and parent one or b/ current_statement_id and nested level. this would offer the ability to track counters at

Re: Implementing Incremental View Maintenance

2020-11-11 Thread legrand legrand
Hello Konstantin, I remember testing it with pg_stat_statements (and planning counters enabled). Maybe identifying internal queries associated with this (simple) test case, could help dev team ? Regards PAscal -- Sent from: https://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html

Re: Is it useful to record whether plans are generic or custom?

2020-09-28 Thread legrand legrand
Hi Atsushi, +1: Your proposal is a good answer for time based performance analysis (even if parsing durationor blks are not differentiated) . As it makes pgss number of columns wilder, may be an other solution would be to create a pg_stat_statements_xxx view with the same key as pgss

Re: enable pg_stat_statements to track rows processed by REFRESH MATERIALIZED VIEW

2020-09-25 Thread legrand legrand
oups, sorry so +1 for this fix Regards PAscal -- Sent from: https://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html

Re: enable pg_stat_statements to track rows processed by REFRESH MATERIALIZED VIEW

2020-09-25 Thread legrand legrand
Hi, isn't this already fixed in pg14 https://www.postgresql.org/message-id/e1k0mzg-0002vn...@gemulon.postgresql.org ? Regards PAscal -- Sent from: https://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html

Re: [PATCH] Add features to pg_stat_statements

2020-09-24 Thread legrand legrand
Not limited to 3, Like an history table. Will have to think if data is saved at shutdown. -- Sent from: https://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html

Re: [PATCH] Add features to pg_stat_statements

2020-09-23 Thread legrand legrand
Hi, Both are probably not needed. I would prefer it accessible in a view live Event | date | victims Eviction... Reset... Part reset ... As there are other needs to track reset times. Regards PAscal -- Sent from: https://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html

Re: [PATCH] Add features to pg_stat_statements

2020-09-19 Thread legrand legrand
+1 ! An other way is to log evictions, it provides informations about time and amount : for (i = 0; i < nvictims; i++) { hash_search(pgssp_hash, [i]->key, HASH_REMOVE, NULL); } pfree(entries); /* trace when evicting entries, if appening

RE: Is it useful to record whether plans are generic or custom?

2020-08-17 Thread legrand legrand
I thought it might be preferable to make a GUC to enable or disable this feature, but changing the hash key makes it harder. >> >>> What happens if the server was running with this option enabled and then >>> restarted with the option disabled? Firstly two entries for the same

nested queries vs. pg_stat_activity

2020-08-10 Thread legrand legrand
Hello, An other solution is to expose nested queryid, and to join it with pg_stat_statements. Actual development trying to add queryid to pg_stat_activity isn't helpfull, because it is only exposing top level one. Extension pg_stat_sql_plans (github) propose a function called

RE: Is it useful to record whether plans are generic or custom?

2020-07-30 Thread legrand legrand
>> Main purpose is to decide (1) the user interface and (2) the >> way to get the plan type from pg_stat_statements. >> >> (1) the user interface >> I added a new boolean column 'generic_plan' to both >> pg_stat_statements view and the member of the hash key of >> pg_stat_statements. >> >> This is

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-05-22 Thread legrand legrand
>> If we can store the plan for each statement, e.g., like pg_store_plans >> extension [1] does, rather than such partial information, which would >> be enough for your cases? > That'd definitely address way more use cases. Do you know if some > benchmark were done to see how much overhead such

Re: Is it useful to record whether plans are generic or custom?

2020-05-16 Thread legrand legrand
> To track executed plan types, I think execution layer hooks > are appropriate. > These hooks, however, take QueryDesc as a param and it does > not include cached plan information. It seems that the same QueryDesc entry is reused when executing a generic plan. For exemple marking

Re: Is it useful to record whether plans are generic or custom?

2020-05-13 Thread legrand legrand
Hello, yes this can be usefull, under the condition of differentiating all the counters for a queryid using a generic plan and the one using a custom one. For me one way to do that is adding a generic_plan column to pg_stat_statements key, someting like: - userid, - dbid, - queryid, -

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-04-08 Thread legrand legrand
Fujii Masao-4 wrote > On 2020/04/03 16:26 > [...] >> >> "Note that planning and execution statistics are updated only at their >> respective end phase, and only for successful operations. >> For example the execution counters of a long running query >> will only be updated at the execution end,

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-04-02 Thread legrand legrand
Fujii Masao-4 wrote > On 2020/04/01 18:19, Fujii Masao wrote: > > Finally I pushed the patch! > Many thanks for all involved in this patch! > > As a remaining TODO item, I'm thinking that the document would need to > be improved. For example, previously the query was not stored in pgss > when it

Re: Patch: to pass query string to pg_plan_query()

2020-03-27 Thread legrand legrand
Tom Lane-2 wrote > Fujii Masao > masao.fujii@.nttdata > writes: >> Does anyone object to this patch? I'm thinking to commit it separetely >> at first before committing the planning_counter_in_pg_stat_statements >> patch. > > I took a quick look through

Re: Allow auto_explain to log plans before queries are executed

2020-03-27 Thread legrand legrand
Kyotaro Horiguchi-4 wrote > At Thu, 27 Feb 2020 06:27:24 +0100, Pavel Stehule > pavel.stehule@ > wrote in >> odesílatel Kyotaro Horiguchi > horikyota.ntt@ > >> napsal: > > If we need a live plan dump of a running query, We could do that using > some kind of inter-backend triggering. (I'm

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-03-16 Thread legrand legrand
> I'm instead attaching a v7 which removes the assert in pg_plan_query, and > modify pgss_planner_hook to also ignore queries without a query text, as > this > seems the best option. Ok, it was the second solution, go on ! -- Sent from:

RE: Planning counters in pg_stat_statements (using pgss_store)

2020-03-14 Thread legrand legrand
> I don't know it is useful but there are also codes that avoid an error when > sourceText is NULL. > executor_errposition(EState *estate, int location) > { > ... >/* Can't do anything if source text is not available */ >if (estate == NULL || estate->es_sourceText == NULL) > } or

RE: Planning counters in pg_stat_statements (using pgss_store)

2020-03-14 Thread legrand legrand
imai.yoshik...@fujitsu.com wrote > On Thu, Mar 12, 2020 at 6:37 PM, Julien Rouhaud wrote: >> On Thu, Mar 12, 2020 at 1:11 PM Marco Slot > marco.slot@ > wrote: >> > On Thu, Mar 12, 2020 at 11:31 AM Julien Rouhaud > rjuju123@ > >> wrote: >> > > There's at least the current version of IVM

Patch: to pass query string to pg_plan_query()

2020-03-09 Thread legrand legrand
Hello, This is a call for committers, reviewers and users, regarding "planning counters in pg_stat_statements" patch [1] but not only. Historically, this version of pg_stat_statements with planning counters was performing 3 calls to pgss_store() for non utility statements in: 1 -

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-03-05 Thread legrand legrand
Never mind ... Please consider PG13 shortest path ;o) My one is parse->queryId != UINT64CONST(0) in pgss_planner_hook(). It fixes IVM problem (verified), and keep CTAS equal to pgss without planning counters (verified too). Regards PAscal -- Sent from:

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-03-02 Thread legrand legrand
Julien Rouhaud wrote > On Sun, Mar 1, 2020 at 3:55 PM legrand legrand > > legrand_legrand@ > wrote: >> >> >> I like the idea of adding a check for a non-zero queryId in the new >> >> pgss_planner_hook() (zero queryid shouldn't be reserved for >&g

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-03-01 Thread legrand legrand
>> I like the idea of adding a check for a non-zero queryId in the new >> pgss_planner_hook() (zero queryid shouldn't be reserved for >> utility_statements ?). > Some assert hit later, I can say that it's not always true. For > instance a CREATE TABLE AS won't run parse analysis for the

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-02-28 Thread legrand legrand
Hi Julien, >> But I would have prefered this new feature to work the same way with or >> without track_planning activated ;o( > Definitely, but fixing the issue in pgss (ignoring planner calls when > we don't have a query text) means that pgss won't give an exhaustive > view of activity anymore,

Re: Implementing Incremental View Maintenance

2020-02-28 Thread legrand legrand
>> thank you for patch v14, that fix problems inherited from temporary tables. >> it seems that this ASSERT problem with pgss patch is still present ;o( >> > > Sorry but we are busy on fixing and improving IVM patches. I think fixing > the assertion failure needs non trivial changes to other part

Re: Implementing Incremental View Maintenance

2020-02-27 Thread legrand legrand
> I have tried to use an other patch with yours: > "Planning counters in pg_stat_statements (using pgss_store)" > setting > shared_preload_libraries='pg_stat_statements' > pg_stat_statements.track=all > restarting the cluster and creating the extension > When trying following syntax: > create

Re: Allow auto_explain to log plans before queries are executed

2020-02-27 Thread legrand legrand
Hi, that feature for dumping plans with auto explain is already available in https://github.com/legrandlegrand/pg_stat_sql_plans This is an hybrid extension combining auto_explain and pg_stat_statements, adding a planid and tracking metrics even on error, ..., ... With

Re: Using stat collector for collecting long SQL

2020-02-27 Thread legrand legrand
Hi > There is a often problem with taking source of long SQL strings. The > pg_stat_activity field query is reduced to some short limit and is not too > practical to increase this limit. I thought it was "old story", since that track_activity_query_size can be increased widely:

Re: PoC: custom signal handler for extensions

2020-02-26 Thread legrand legrand
Hello Maksim, reading about https://www.postgresql-archive.org/Allow-auto-explain-to-log-plans-before-queries-are-executed-td6124646.html makes me think (again) about your work and pg_query_state ... Is there a chance to see you restarting working on this patch ? Regards PAscal -- Sent from:

Re: WIP: Aggregation push-down

2020-02-26 Thread legrand legrand
Antonin Houska-2 wrote > Alvaro Herrera > alvherre@ > wrote: > >> This stuff seems very useful. How come it sits unreviewed for so long? > > I think the review is hard for people who are not interested in the > planner > very much. And as for further development, there are a few design >

Re: Implementing Incremental View Maintenance

2020-02-13 Thread legrand legrand
Yugo Nagata wrote > Thank you for your suggestion! I agree that the feature to switch between > normal materialized view and incrementally maintainable view is useful. > We will add this to our ToDo list. Regarding its syntax, > I would not like to add new keyword like NONINCREMENTAL, so how

Re: Implementing Incremental View Maintenance

2020-02-11 Thread legrand legrand
Takuma Hoshiai wrote > Hi, > > Attached is the latest patch (v12) to add support for Incremental > Materialized View Maintenance (IVM). > It is possible to apply to current latest master branch. > > Differences from the previous patch (v11) include: > * support executing REFRESH MATERIALIZED

Re: POC: rational number type (fractions)

2020-02-08 Thread legrand legrand
Hello, It seems you are not the first to be interested in such feature. There was a similar extension used in "incremental view maintenance" testing: https://github.com/nuko-yokohama/pg_fraction didn't tryed it myself. Regards PAscal -- Sent from:

Re: WIP: Aggregation push-down

2020-01-31 Thread legrand legrand
Hello, Thank you for this great feature ! I hope this will be reviewed/validated soon ;o) Just a comment: set enable_agg_pushdown to true; isn't displayed in EXPLAIN (SETTINGS) syntax. The following modification seems to fix that: src/backend/utils/misc/guc.c

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-01-18 Thread legrand legrand
Hi Julien, bot is still unhappy https://travis-ci.org/postgresql-cfbot/postgresql/builds/638701399 portalcmds.c: In function ‘PerformCursorOpen’: portalcmds.c:93:7: error: ‘queryString’ may be used uninitialized in this function [-Werror=maybe-uninitialized] plan = pg_plan_query(query,

Re: postgresql-13devel initDB Running in debug mode.

2020-01-18 Thread legrand legrand
Tom Lane-2 wrote > legrand legrand > legrand_legrand@ > writes: >> after building devel snapshot from 2020-01-17 with msys, >> initDB generates a lot of additional informations when launched: >> [ debug output snipped ] >> Is that the expected behavior, or

postgresql-13devel initDB Running in debug mode.

2020-01-18 Thread legrand legrand
Hello, after building devel snapshot from 2020-01-17 with msys, initDB generates a lot of additional informations when launched: VERSION=13devel PGDATA=../data share_path=C:/msys64/usr/local/pgsql/share PGPATH=C:/msys64/usr/local/pgsql/bin POSTGRES_SUPERUSERNAME=lemoyp

Re: pg13 PGDLLIMPORT list

2020-01-18 Thread legrand legrand
Michael Paquier-2 wrote > On Fri, Jan 17, 2020 at 03:07:48PM -0700, legrand legrand wrote: > [...] > > No objections from me to add both to what's imported. Do you have a > specific use-case in mind for an extension on Windows? Just > wondering.. > -- > Michael >

pg13 PGDLLIMPORT list

2020-01-17 Thread legrand legrand
Hello, would it be possible to add PGDLLIMPORT to permit to build following extensions on windows pg_stat_sql_plans: src/include/pgstat.h extern PGDLLIMPORT bool pgstat_track_activities; pg_background: src/include/storage/proc.h extern PGDLLIMPORT int StatementTimeout; Thanks in advance

Re: Implementing Incremental View Maintenance

2020-01-17 Thread legrand legrand
Hello, It seems that patch v11 doesn't apply any more. Problem with "scanRTEForColumn" maybe because of change: https://git.postgresql.org/pg/commitdiff/b541e9accb28c90656388a3f827ca3a68dd2a308 Regards PAscal -- Sent from: https://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html

Re: WIP: System Versioned Temporal Table

2020-01-05 Thread legrand legrand
Vik Fearing-4 wrote > On 05/01/2020 16:01, legrand legrand wrote: > > > No, that is incorrect.  The standard syntax is: > > >     FROM tablename FOR SYSTEM_TIME AS OF '...' > >     FROM tablename FOR SYSTEM_TIME BETWEEN '...' AND '...' > >     FR

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-01-05 Thread legrand legrand
Julien Rouhaud wrote > On Sun, Jan 5, 2020 at 4:11 PM legrand legrand > > legrand_legrand@ > wrote: >> >> Hi Julien, >> >> I would like to create a link with >> https://www.postgresql.org/message-id/ > 1577490124579-0.post@.nabble >>

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-01-05 Thread legrand legrand
Hi Julien, I would like to create a link with https://www.postgresql.org/message-id/1577490124579-0.p...@n3.nabble.com where we met an ASSET FAILURE because query text was not initialized ... The question raised is: - should query text be always provided or - if not how to deal that case (in

Re: WIP: System Versioned Temporal Table

2020-01-05 Thread legrand legrand
Vik Fearing-4 wrote > On 05/01/2020 11:16, Surafel Temesgen wrote: >> >> >> On Fri, Jan 3, 2020 at 4:22 PM Vik Fearing >> > vik.fearing@ > mailto: > vik.fearing@ > > wrote: >> > > [...] > > You only test FROM-TO and with a really wide interval.  There are no > tests for AS OF and no tests

Incremental View Maintenance: ERROR: out of shared memory

2019-12-28 Thread legrand legrand
Hello here is an unexpected error found while testing IVM v11 patches create table b1 (id integer, x numeric(10,3)); create incremental materialized view mv1 as select id, count(*),sum(x) from b1 group by id; do $$ declare i integer; begin for i in 1..1 loop

Re: Implementing Incremental View Maintenance

2019-12-27 Thread legrand legrand
Hello, Thank you for this patch. I have tried to use an other patch with yours: "Planning counters in pg_stat_statements (using pgss_store)" https://www.postgresql.org/message-id/CAOBaU_Y12bn0tOdN9RMBZn29bfYYH11b2CwKO1RO7dX9fQ3aZA%40mail.gmail.com setting

Re: Implementing Incremental View Maintenance

2019-12-24 Thread legrand legrand
Yugo Nagata wrote > On Mon, 23 Dec 2019 03:41:18 -0700 (MST) > legrand legrand > legrand_legrand@ > wrote: > > [ ...] > >> I would even >> prefer a common "table" shared between all sessions like GLOBAL TEMPORARY >> TABLE (

Re: Implementing Incremental View Maintenance

2019-12-23 Thread legrand legrand
Hello, regarding my initial post: > For each insert into a base table there are 3 statements: > - ANALYZE pg_temp_3.pg_temp_81976 > - WITH updt AS ( UPDATE public.mv1 AS mv SET __ivm_count__ = ... > - DROP TABLE pg_temp_3.pg_temp_81976 For me there where 3 points to discuss: - create/drop

Re: Implementing Incremental View Maintenance

2019-12-22 Thread legrand legrand
Hello, First of all many thanks for this Great feature replacing so many triggers by a so simple syntax ;o) I was wondering about performances and add a look at pg_stat_statements (with track=all) with IVM_v9.patch. For each insert into a base table there are 3 statements: - ANALYZE

Re: Columns correlation and adaptive query optimization

2019-10-14 Thread legrand legrand
Hello Konstantin, What you have proposed regarding join_selectivity and multicolumn statistics is a very good new ! Regarding your auto_explain modification, maybe an "advisor" mode would also be helpfull (with auto_explain_add_statistics_threshold=-1 for exemple). This would allow to track

Re: How to install login_hook in Postgres 10.5

2019-10-05 Thread legrand legrand
pavan95 wrote > Hello Community, > > While I was searching for logon trigger in postgres similar to that of > Oracle, I came across "login_hook", which can be installed as a Postgres > database extension to mimic a logon trigger. > > But I tried to install but failed. Error is that it could not

Re: Hooks for session start and end, take two

2019-09-27 Thread legrand legrand
Michael Paquier-2 wrote > On Thu, Sep 26, 2019 at 09:57:57AM -0700, legrand legrand wrote: >> Does that mean that all processes seen in pg_stat_activity like >> - autovacuum launcher >> - logical replication launcher >> - background writer >> - checkpointer >

Re: Hooks for session start and end, take two

2019-09-26 Thread legrand legrand
Hello, Thank you for the work done on this subject. After starting to play with it, I have a question and a remark: > - previous hook calls were only called for normal backends, which was > incorrect as we define the backend so as we apply no backend-related > filtering for the hook. Does that

Re: Proposal: Better query optimization for "NOT IN" clause

2019-09-23 Thread legrand legrand
Hello, Just for information there are some works regarding how to include this in core, that may interest you ;o) see "NOT IN subquery optimization" https://www.postgresql.org/message-id/flat/1550706289606-0.post%40n3.nabble.com commitfest entry: https://commitfest.postgresql.org/24/2023/ and

Re: [survey] New "Stable" QueryId based on normalized query text

2019-08-12 Thread legrand legrand
my understanding is * pg_stat_statements.track = 'none' or 'top' (default) or 'all' to make queryId optionally computed * a new GUC: pg_stat_statements.queryid_based = 'oids' (default) or 'names' or 'fullnames' to choose the queryid computation algorithm am I rigth ? -- Sent from:

Re: [survey] New "Stable" QueryId based on normalized query text

2019-08-10 Thread legrand legrand
Hi Jim, Its never too later, as nothing has been concluded about that survey ;o) For information, I thought It would be possible to get a more stable QueryId, by hashing relation name or fully qualified names. With the support of Julien Rouhaud, I tested with this kind of code: case

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2019-08-05 Thread legrand legrand
Kyotaro Horiguchi-4 wrote > At Sun, 4 Aug 2019 00:04:01 -0700 (MST), legrand legrand > legrand_legrand@ > wrote in < > 1564902241482-0.post@.nabble >> >> > However having the nested queryid in >> > pg_stat_activity would be convenient to track >>

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2019-08-04 Thread legrand legrand
> However having the nested queryid in > pg_stat_activity would be convenient to track > what is a long stored functions currently doing. +1 And this could permit to get wait event sampling per queryid when pg_stat_statements.track = all Regards PAscal -- Sent from:

Re: How to install login_hook in Postgres 10.5

2019-08-01 Thread legrand legrand
Hello, shouldn't we update associated commitfest entry https://commitfest.postgresql.org/15/1318/ to give it a chance to be included in pg13 ? Regards PAscal -- Sent from: https://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html

Re: Adaptive query optimization

2019-06-12 Thread legrand legrand
>>I tried to create much simpler version of AQO based on auto_explain >>extension. >>This extension provide all necessary infrastructure to analyze >>statements with long execution time. >>I have added two new modes to auto_explain: >>1. Auto generation of multicolumn statistics for variables

Re: Adaptive query optimization

2019-06-11 Thread legrand legrand
Hello, this seems very interesting and make me think about 2 other projets: - https://github.com/trustly/pg_badplan - https://github.com/ossc-db/pg_plan_advsr As I understand all this, there are actually 3 steps: - compare actual / estimated rows - suggests some statistics gathering modification

Re: How to install login_hook in Postgres 10.5

2019-05-13 Thread legrand legrand
Hello, This extension https://github.com/splendiddata/login_hook seems very interesting ! But I didn't test it myself and maybe the best place to ask support is there https://github.com/splendiddata/login_hook/issues For information there is something equivalent in core "[PATCH] A hook for

RE: Re: Logging the feature of SQL-level read/write commits

2019-05-05 Thread legrand legrand
Hi, good point ! wal2Json seems to correspond to your needs, this is first designed for Change Data Capture, taht could generate a (very) big size of logs. You didn't tell us much about your use case ... and maybe, if the number of data modifications is not too big, and the number of tables

Re: Logging the feature of SQL-level read/write commits

2019-05-05 Thread legrand legrand
Hello, may be you can find more informations regarding WAL concepts in Write Ahead Logging — WAL http://www.interdb.jp/pg/pgsql09.html It seems very complicated to change WAL format ... Maybe there are other solutions to answer your need, I found many interesting solutions in postgres archives

RE: minimizing pg_stat_statements performance overhead

2019-04-08 Thread legrand legrand
CF entry created https://commitfest.postgresql.org/23/2092/ Regards PAscal -- Sent from: http://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html

Re: minimizing pg_stat_statements performance overhead

2019-04-03 Thread legrand legrand
Robert Haas wrote > On Tue, Apr 2, 2019 at 5:37 AM Christoph Berg > myon@ > wrote: >> Re: Raymond Martin 2019-04-01 > BN8PR21MB121708579A3782866DF1F745B1550@.outlook > >> > Thanks again Fabien. I am attaching the patch to this email in the hope >> of getting it approved during the next

RE: Planning counters in pg_stat_statements (using pgss_store)

2019-04-02 Thread legrand legrand
Hi, >> >> case avg_tps pct_diff >> 089 278 -- >> 188 745 0,6% >> 288 282 1,1% >> 386 660 2,9% >> >> This means that even in this extrem test case, the worst degradation is less >> than 3% >> (this overhead can be removed using

Re: DWIM mode for psql

2019-04-01 Thread legrand legrand
Andreas Karlsson wrote > On 3/31/19 10:52 PM, Thomas Munro wrote:> Building on the excellent work > begun by commit e529cd4ffa60, I would >> like to propose a do-what-I-mean mode for psql. Please find a POC >> patch attached. It works like this: >> >> postgres=# select datnaam from pg_database

Re: Planning counters in pg_stat_statements (using pgss_store)

2019-04-01 Thread legrand legrand
Hi, I have played with this patch, it works fine. rem the last position of the "new" total_time column is confusing +CREATE VIEW pg_stat_statements AS + SELECT *, total_plan_time + total_exec_time AS total_time +FROM pg_stat_statements(true); I wanted to perform some benchmark between

RE: minimizing pg_stat_statements performance overhead

2019-04-01 Thread legrand legrand
Hi, it seems that your patch is not readable. If you want it to be included in a commitfest, you should add it by yourself in https://commitfest.postgresql.org/ Not sure that there is any room left in pg12 commitfest. Regard PAscal -- Sent from:

RE: minimizing pg_stat_statements performance overhead

2019-03-27 Thread legrand legrand
my test case: drop table a; create table a (); DO $$ DECLARE i int; BEGIN for i in 1..20 loop execute 'alter table a add column a'||i::text||' int'; end loop; END $$; select pg_stat_statements_reset(); set pg_stat_statements.track='none'; DO $$ DECLARE i int; j int; BEGIN for j in 1..20 loop

Re: Planning counters in pg_stat_statements (using pgss_store)

2019-03-27 Thread legrand legrand
>> - trailing whitespaces and comments wider than 80 characters >> not fixed > why? In case it's not clear, I'm talking about the .c file, not the > regression tests. I work on a poor msys install on windows 7, where perl is broken ;o( So no pgindent available. Will fix that

RE: minimizing pg_stat_statements performance overhead

2019-03-27 Thread legrand legrand
Your fix is probably the best one. Maybe this could be considered as a bug and back ported to previous versions ... Regards PAscal -- Sent from: http://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html

Re: minimizing pg_stat_statements performance overhead

2019-03-27 Thread legrand legrand
Hi, the part that hurts in terms or performances is: if (jstate.clocations_count > 0) pgss_store(pstate->p_sourcetext, query->queryId, query->stmt_location, query->stmt_len,

RE: Planning counters in pg_stat_statements (using pgss_store)

2019-03-26 Thread legrand legrand
here is a new version: - "track_planning" GUC added to permit to keep previous behavior unchanged - columns names have been changed / added: total_plan_time, total_exec_time, total_time - trailing whitespaces and comments wider than 80 characters not fixed

Re: Planning counters in pg_stat_statements (using pgss_store)

2019-03-25 Thread legrand legrand
As there are now 3 locking times on pgss hash struct, one day or an other, somebody will ask for a GUC to disable this feature (to be able to run pgss unchanged with only one lock as today). With this GUC, pgss_store should be able to store the query text and accumulated execution duration in

RE: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2019-03-25 Thread legrand legrand
>> Shoudn't you add this to commitfest ? > I added it last week, see https://commitfest.postgresql.org/23/2069/ Oups, sorry for the noise

RE: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2019-03-25 Thread legrand legrand
>> Would it make sense to add it in auto explain ? >> I don't know for explain itself, but maybe ... > I'd think that people interested in getting the queryid in the logs > would configure the log_line_prefix to display it consistently rather > than having it in only a subset of cases, so that's

RE: Planning counters in pg_stat_statements (using pgss_store)

2019-03-23 Thread legrand legrand
> This patch has multiple trailing whitespace, indent and coding style > issues. You should consider running pg_indent before submitting a > patch. I attach the diff after running pgindent if you want more > details about the various issues. fixed > - * Track statement execution times

RE: Planning counters in pg_stat_statements (using pgss_store)

2019-03-22 Thread legrand legrand
Hi, Here is a rebased and corrected version . Columns naming has not been modified, I would propose to change it to: - plans: ok - planning_time --> plan_time - calls: ok - total_time --> exec_time - {min,max,mean,stddev}_time: ok - new total_time (being the sum of plan_time and exec_time)

Re: [survey] New "Stable" QueryId based on normalized query text

2019-03-20 Thread legrand legrand
Julien Rouhaud wrote > On Wed, Mar 20, 2019 at 10:30 PM legrand legrand > > legrand_legrand@ > wrote: >> >> maybe this patch (with a GUC) >> https://www.postgresql.org/message-id/ > 55E51C48.1060102@ >> would be enough for thoses actually using a tex

Re: [survey] New "Stable" QueryId based on normalized query text

2019-03-20 Thread legrand legrand
Julien Rouhaud wrote > On Wed, Mar 20, 2019 at 10:18 PM legrand legrand > > legrand_legrand@ > wrote: >> >> On my personal point of view, I need to get the same Queryid between >> (OLAP) >> environments >> to be able to compare Production, Pre-produ

Re: [survey] New "Stable" QueryId based on normalized query text

2019-03-20 Thread legrand legrand
maybe this patch (with a GUC) https://www.postgresql.org/message-id/55e51c48.1060...@uptime.jp would be enough for thoses actually using a text normalization function. -- Sent from: http://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html

Re: [survey] New "Stable" QueryId based on normalized query text

2019-03-20 Thread legrand legrand
Julien Rouhaud wrote > On Wed, Mar 20, 2019 at 8:39 PM legrand legrand > > legrand_legrand@ > wrote: >> >> Yes, I would like first to understand what are the main needs, > > I don't really see one implementation that suits every need, as > probably not ever

Re: [survey] New "Stable" QueryId based on normalized query text

2019-03-20 Thread legrand legrand
> From "Kyotaro HORIGUCHI-2" >>At Wed, 20 Mar 2019 00:23:30 +, "Tsunakawa, Takayuki" >>> From: legrand legrand [mailto:legrand_legrand@] >>> norm.9: comments aware >> Is this to distinguish queries that have different comments for optimizer

RE: [survey] New "Stable" QueryId based on normalized query text

2019-03-20 Thread legrand legrand
> From: "Tsunakawa, Takayuki" >> From: legrand legrand [mailto:legrand_legrand@] >> There are many projects that use alternate QueryId >> distinct from the famous pg_stat_statements jumbling algorithm. >I'd like to welcome the standard QueryID that DBAs a

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2019-03-20 Thread legrand legrand
Hi Jim, Robert, As this is a distinct subject from adding QueryId to pg_stat_activity, would it be possible to continue the discussion "new QueryId definition" (for postgres open source software) here: https://www.postgresql.org/message-id/1553029215728-0.p...@n3.nabble.com Thanks in advance.

[survey] New "Stable" QueryId based on normalized query text

2019-03-19 Thread legrand legrand
Hello, There are many projects that use alternate QueryId distinct from the famous pg_stat_statements jumbling algorithm. https://github.com/postgrespro/aqo query_hash https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.Optimize.ViewPlans.html sql_hash

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2019-03-19 Thread legrand legrand
Great, thank you Julien ! Would it make sense to add it in auto explain ? I don't know for explain itself, but maybe ... Regards PAscal -- Sent from: http://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2019-03-16 Thread legrand legrand
Hello, This is available in https://github.com/legrandlegrand/pg_stat_sql_plans extension with a specific function pgssp_backend_queryid(pid) that permits to join pg_stat_activity with pg_stat_sql_plans (that is similar to pg_stat_statements) and also permits to collect samples of wait events per

Re: any plan to support shared servers like Oracle in PG?

2019-03-05 Thread legrand legrand
There already are solutions regarding this feature in Postgres using "connection pooler" wording see pgpool: http://www.pgpool.net/mediawiki/index.php/Main_Page pgbouncer: https://pgbouncer.github.io/ there are also discussions to include this as a core feature

Re: Planning counters in pg_stat_statements (using pgss_store)

2019-02-14 Thread legrand legrand
Thank you Sergei for your comments, > Did you register patch in CF app? I did not found entry. created today: https://commitfest.postgresql.org/22/1999/ > Currently we have pg_stat_statements 1.7 version and this patch does not > apply... will rebase and create a 1.8 version > -

Re: Planning counters in pg_stat_statements (using pgss_store)

2019-02-12 Thread legrand legrand
Hi Sergei, Thank you for this review ! >Did you register patch in CF app? I did not found entry. I think it is related to https://commitfest.postgresql.org/16/1373/ but I don't know how to link it with. Yes, there are many things to improve, but before to go deeper, I would like to know if

Re: explain plans with information about (modified) gucs

2019-01-14 Thread legrand legrand
Tomas Vondra-4 wrote > Hello Sergei, > >> This patch correlates with my proposal >> "add session information column to pg_stat_statements" >> https://www.postgresql.org/message-id/3aa097d7-7c47-187b-5913-db8366cd4491%40gmail.com >> They both address the problem to identify the factors that make

Planning counters in pg_stat_statements (using pgss_store)

2018-12-29 Thread legrand legrand
Starting from https://www.postgresql.org/message-id/CAEepm%3D2vORBhWQZ1DJmKXmCVi%2B15Tgrv%2B9brHLanWU7XE_FWxQ%40mail.gmail.com Here is a patch trying to implement what was proposed by Tom Lane: "What we could/should do instead, I think, is have pgss_planner_hook make its own pgss_store() call to

  1   2   >