Re: DROP OWNED BY fails to clean out pg_init_privs grants

2024-05-25 Thread Hannu Krosing
On Sat, May 25, 2024 at 4:48 PM Tom Lane wrote: > > Hannu Krosing writes: > > Having an pg_init_privs entry referencing a non-existing user is > > certainly of no practical use. > > Sure, that's not up for debate. What I think we're discussing > right now is > &

Re: DROP OWNED BY fails to clean out pg_init_privs grants

2024-05-25 Thread Hannu Krosing
On Fri, May 24, 2024 at 10:00 PM Tom Lane wrote: > > Robert Haas writes: > > On Fri, May 24, 2024 at 2:57 PM Tom Lane wrote: > >> Doesn't seem right to me. That will give pg_dump the wrong idea > >> of what the initial privileges actually were, and I don't see how > >> it can construct correct

Re: DROP OWNED BY fails to clean out pg_init_privs grants

2024-05-23 Thread Hannu Krosing
While the 'DROP OWNED BY fails to clean out pg_init_privs grants' issue is now fixed,we have a similar issue with REASSIGN OWNED BY that is still there: Tested on fresh git checkout om May 20th test=# create user privtestuser superuser; CREATE ROLE test=# set role privtestuser; SET test=# create

Re: Function and Procedure with same signature?

2024-03-11 Thread Hannu Krosing
On Thu, Mar 7, 2024 at 5:46 PM Tom Lane wrote: > > Hannu Krosing writes: > > On Sat, Feb 10, 2024 at 12:38 AM Tom Lane wrote: > >> Worth noting perhaps that this is actually required by the SQL > >> standard: per spec, functions and procedures are both &quo

CREATE TABLE creates a composite type corresponding to the table row, which is and is not there

2024-03-07 Thread Hannu Krosing
I could not find any explanation of the following behaviour in docs - Our documentation for CREATE TABLE says: CREATE TABLE also automatically creates a data type that represents the composite type corresponding to one row of the table. Therefore, tables cannot have the same name as any

Re: Function and Procedure with same signature?

2024-03-07 Thread Hannu Krosing
Hi Tom On Sat, Feb 10, 2024 at 12:38 AM Tom Lane wrote: > > "David G. Johnston" writes: > > On Fri, Feb 9, 2024, 12:05 Deepak M wrote: > >> Folks, When tried to create a function with the same signature as > >> procedure it fails. > > > That seems like a good hint you cannot do it.

Re: pgbench - adding pl/pgsql versions of tests

2024-02-02 Thread Hannu Krosing
My justification for adding pl/pgsql tests as part of the immediately available tests is that pl/pgsql itself is always enabled, so having a no-effort way to test its performance benefits would be really helpful. We also should have "tps-b-like as SQL function" to round up the "test what's

Re: pgbench - adding pl/pgsql versions of tests

2024-02-02 Thread Hannu Krosing
Thanks for the update. I will give it another go over the weekend Cheers, Hannu On Thu, Feb 1, 2024 at 7:33 PM vignesh C wrote: > On Fri, 18 Aug 2023 at 23:04, Hannu Krosing wrote: > > > > I will address the comments here over this coming weekend. > > The patch which y

Re: Emitting JSON to file using COPY TO

2023-12-09 Thread Hannu Krosing
> On Sat, Dec 2, 2023 at 4:11 PM Tom Lane wrote: > > Joe Conway writes: > >> I noticed that, with the PoC patch, "json" is the only format that must be > >> quoted. Without quotes, I see a syntax error. In longer term we should move any specific COPY flag names and values out of grammar and

Why are wal_keep_size, max_slot_wal_keep_size requiring server restart?

2023-12-09 Thread Hannu Krosing
Hello fellow Hackers, Does anyone know why we have decided that the wal_keep_size, max_slot_wal_keep_size GUCs "can only be set in the postgresql.conf file or on the server command line." [1]? It does not seem fundamentally needed , as they are "kind of guidance", especially the second one.

Re: Allowing TRUNCATE of FK target when session_replication_role=replica

2023-10-31 Thread Hannu Krosing
wrote: > > On Tue, Oct 31, 2023, at 5:09 AM, Hannu Krosing wrote: > > Currently we do not allow TRUNCATE of a table when any Foreign Keys > point to that table. > > > It is allowed iif you *also* truncate all tables referencing it. > > At the same time we

Allowing TRUNCATE of FK target when session_replication_role=replica

2023-10-31 Thread Hannu Krosing
Hi Currently we do not allow TRUNCATE of a table when any Foreign Keys point to that table. At the same time we do allow one to delete all rows when session_replication_role=replica This causes all kinds of pain when trying to copy in large amounts of data, especially at the start of logical

Re: Initdb-time block size specification

2023-09-05 Thread Hannu Krosing
: > > Hi, > > On 2023-09-05 21:52:18 +0200, Hannu Krosing wrote: > > Something I also asked at this years Unconference - Do we currently > > have Build Farm animals testing with different page sizes ? > > You can check that yourself as easily as anybody else. > > Greetings, > > Andres Freund

Re: Initdb-time block size specification

2023-09-05 Thread Hannu Krosing
Something I also asked at this years Unconference - Do we currently have Build Farm animals testing with different page sizes ? I'd say that testing all sizes from 4KB up (so 4, 8, 16, 32) should be done at least before each release if not continuously. -- Cheers Hannu On Tue, Sep 5, 2023 at

Re: pgbench - adding pl/pgsql versions of tests

2023-08-18 Thread Hannu Krosing
I will address the comments here over this coming weekend. I think that in addition to current "tpc-b like" test we could also have more modern "tpc-c like" and "tpc-h like" tests And why not any other "* -like" from the rest of TPC-*, YCSP, sysbench, ... :) though maybe not as part of

Re: How to build a new grammer for pg?

2023-08-08 Thread Hannu Krosing
I would look at how Babelfish DB did it when adding SQL Server compatibility https://babelfishpg.org/ and https://github.com/babelfish-for-postgresql/ another source to inspect could be https://github.com/IvorySQL/IvorySQL for "oracle compatible PostgreSQL" On Tue, Aug 1, 2023 at 10:07 PM Jonah

Re: incremental-checkopints

2023-07-26 Thread Hannu Krosing
On Wed, Jul 26, 2023 at 9:54 PM Matthias van de Meent wrote: > > Then you ignore the max_wal_size GUC as PostgreSQL so often already > does. At least, it doesn't do what I expect it to do at face value - > limit the size of the WAL directory to the given size. That would require stopping any new

Re: incremental-checkopints

2023-07-26 Thread Hannu Krosing
Starting from increments checkpoint is approaching the problem from the wrong end. What you actually want is Atomic Disk Writes which will allow turning off full_page_writes . Without this you really can not do incremental checkpoints efficiently as checkpoints are currently what is used to

Re: Example Table AM implementation

2023-07-06 Thread Hannu Krosing
Thanks a lot Mark, I will take a look at this and get back to you if I find anything unclear --- Hannu On Tue, Jul 4, 2023 at 10:14 PM Mark Dilger wrote: > > Hackers, > > Over in [1], Hannu Krosing asked me to create and post several Table Access > Methods for testing/example

Including a sample Table Access Method with core code

2023-07-03 Thread Hannu Krosing
At PgCon 2023 in Ottawa we had an Unconference session on Table Access Methods [1] One thing that was briefly mentioned (but is missing from the notes) is need to have a sample API client in contrib/ , both for having a 2nd user for API to make it more likely that non-heap AMs are doable and also

Re: Let's make PostgreSQL multi-threaded

2023-06-15 Thread Hannu Krosing
caches. On Thu, Jun 15, 2023 at 11:04 AM Hannu Krosing wrote: > > On Thu, Jun 15, 2023 at 10:41 AM James Addison wrote: > > > > This is making me wonder about other performance/scalability areas > > that might not have been considered due to focus on the details of t

Re: Let's make PostgreSQL multi-threaded

2023-06-15 Thread Hannu Krosing
On Thu, Jun 15, 2023 at 10:41 AM James Addison wrote: > > This is making me wonder about other performance/scalability areas > that might not have been considered due to focus on the details of the > existing codebase, but I'll save that for another thread and will try > to learn more first. A

Re: Let's make PostgreSQL multi-threaded

2023-06-15 Thread Hannu Krosing
On Thu, Jun 15, 2023 at 9:12 AM Konstantin Knizhnik wrote: > There are three different but related directions of improving current > Postgres: > 1. Replacing processes with threads Here we could likely start with making parallel query multi-threaded. This would also remove the big blocker for

Re: Let's make PostgreSQL multi-threaded

2023-06-14 Thread Hannu Krosing
On Tue, Jun 13, 2023 at 9:55 AM Kyotaro Horiguchi wrote: > > At Tue, 13 Jun 2023 09:55:36 +0300, Konstantin Knizhnik > wrote in > > Postgres backend is "thick" not because of large number of local > > variables. > > It is because of local caches: catalog cache, relation cache, prepared > >

Re: Let's make PostgreSQL multi-threaded

2023-06-10 Thread Hannu Krosing
On Mon, Jun 5, 2023 at 4:52 PM Heikki Linnakangas wrote: > > If there are no major objections, I'm going to update the developer FAQ, > removing the excuses there for why we don't use threads [1]. I think it is not wise to start the wholesale removal of the objections there. But I think it is

Re: Let's make PostgreSQL multi-threaded

2023-06-08 Thread Hannu Krosing
I discovered this thread from a Twitter post "PostgreSQL will finally be rewritten in Rust" :) On Mon, Jun 5, 2023 at 5:18 PM Tom Lane wrote: > > Heikki Linnakangas writes: > > I spoke with some folks at PGCon about making PostgreSQL multi-threaded, > > so that the whole server runs in a

Re: Let's make PostgreSQL multi-threaded

2023-06-08 Thread Hannu Krosing
On Thu, Jun 8, 2023 at 4:56 PM Robert Haas wrote: > > On Thu, Jun 8, 2023 at 8:44 AM Hannu Krosing wrote: > > > That sounds like a bad idea, dynamic shared memory is more expensive > > > to maintain than our static shared memory systems, not in the least > > &

Re: Let's make PostgreSQL multi-threaded

2023-06-08 Thread Hannu Krosing
On Thu, Jun 8, 2023 at 2:15 PM Matthias van de Meent wrote: > > On Thu, 8 Jun 2023 at 11:54, Hannu Krosing wrote: > > > > On Wed, Jun 7, 2023 at 11:37 PM Andres Freund wrote: > > > > > > Hi, > > > > > > On 2023-06-05 13:40:13 -0400, Jo

Re: Use COPY for populating all pgbench tables

2023-06-08 Thread Hannu Krosing
I guess that COPY will still be slower than generating the data server-side ( --init-steps=...G... ) ? What I'd really like to see is providing all the pgbench functions also on the server. Specifically the various random(...) functions - random_exponential(...), random_gaussian(...),

Re: Let's make PostgreSQL multi-threaded

2023-06-08 Thread Hannu Krosing
On Thu, Jun 8, 2023 at 11:54 AM Hannu Krosing wrote: > > On Wed, Jun 7, 2023 at 11:37 PM Andres Freund wrote: > > > > Hi, > > > > On 2023-06-05 13:40:13 -0400, Jonathan S. Katz wrote: > > > 2. While I wouldn't want to necessarily discourage a moonshot effo

Re: Let's make PostgreSQL multi-threaded

2023-06-08 Thread Hannu Krosing
On Thu, Jun 8, 2023 at 12:09 AM Andres Freund wrote: ... > We could e.g. eventually decide that we > don't support parallel query without threading support - which would allow us > to get rid of a very significant amount of code and runtime overhead. Here I was hoping to go in the opposite

Re: Let's make PostgreSQL multi-threaded

2023-06-08 Thread Hannu Krosing
I think I remember that in the early days of development somebody did send a patch-set for making PostgreSQL threaded on Solaris. I don't remember why this did not catch on. On Wed, Jun 7, 2023 at 11:40 PM Thomas Kellerer wrote: > > Tomas Vondra schrieb am 07.06.2023 um 21:20: > > Also, which

Re: Let's make PostgreSQL multi-threaded

2023-06-08 Thread Hannu Krosing
On Wed, Jun 7, 2023 at 11:37 PM Andres Freund wrote: > > Hi, > > On 2023-06-05 13:40:13 -0400, Jonathan S. Katz wrote: > > 2. While I wouldn't want to necessarily discourage a moonshot effort, I > > would ask if developer time could be better spent on tackling some of the > > other problems

Re: Time to move pg_test_timing to measure in nanoseconds

2023-03-26 Thread Hannu Krosing
Sure, will do. On Sun, Mar 26, 2023 at 11:40 PM Andres Freund wrote: > > Hi, > > On 2023-03-26 16:43:21 +0200, Hannu Krosing wrote: > > Currently pg_test_timing utility measures its timing overhead in > > microseconds, giving results like this > > I have a patch

Re: Disable vacuuming to provide data history

2023-03-26 Thread Hannu Krosing
There is also another blocker - our timestamp resolution is 1 microsecond and we are dangerously close to speeds where one could update a row twice in the same microsecond . I have been thinking about this, and what is needed is 1. a nanosecond-resolution "abstime" type - not absolutely

Time to move pg_test_timing to measure in nanoseconds

2023-03-26 Thread Hannu Krosing
Currently pg_test_timing utility measures its timing overhead in microseconds, giving results like this ~$ /usr/lib/postgresql/15/bin/pg_test_timing Testing timing overhead for 3 seconds. Per loop time including overhead: 18.97 ns Histogram of timing durations: < us % of total count

pgbench - adding pl/pgsql versions of tests

2023-01-04 Thread Hannu Krosing
can be disabled fia long flag --no-functions ) I selected Yy as they were unused and can be thought of as "inverted lambda symbol" :) If there are no strong objections, I'll add it to the commitfest as well - Hannu Krosing Google Cloud - We have a long list of planned contributions

Is there a way to use exported snapshots in autocommit mode ?

2022-12-09 Thread Hannu Krosing
Hello hackers, Is there a way to use exported snapshots in autocommit mode ? Either something similar to defaults in default_transaction_deferrable, default_transaction_isolation, default_transaction_read_only Or something that could be set in the function's SET part. Context: I am working on

Is anybody planning to release pglogical for v15 ?

2022-10-13 Thread Hannu Krosing
So, is anybody planning to release pglogical for v15 ? There are still a few things that one can do in pglogical but not in native / built0in replication ... Best Regards Hannu

Re: making relfilenodes 56 bits

2022-07-12 Thread Hannu Krosing
Re: staticAssertStmt(MAX_FORKNUM <= INT8_MAX); Have you really thought through making the ForkNum 8-bit ? For example this would limit a columnar storage with each column stored in it's own fork (which I'd say is not entirely unreasonable) to having just about ~250 columns. And there can easily

Re: Hardening PostgreSQL via (optional) ban on local file system access

2022-07-01 Thread Hannu Krosing
And thanks to Robert and Bruce for bringing up good points about potential pitfalls! I think we do have a good discussion going on here :) --- Hannu On Fri, Jul 1, 2022 at 11:14 AM Hannu Krosing wrote: > > On Thu, Jun 30, 2022 at 7:25 PM Bruce Momjian wrote: > > > > On Thu,

Re: Hardening PostgreSQL via (optional) ban on local file system access

2022-07-01 Thread Hannu Krosing
On Thu, Jun 30, 2022 at 7:25 PM Bruce Momjian wrote: > > On Thu, Jun 30, 2022 at 11:52:20AM -0400, Robert Haas wrote: > > I don't think this would be very convenient in most scenarios, This is the eternal problem with security - more security always includes more inconvenience. Unlocking your

Re: Hardening PostgreSQL via (optional) ban on local file system access

2022-06-29 Thread Hannu Krosing
ble to log in as a user with superuser attribute 2) you must present proof that you can access the underlying file system Cheers, Hannu Krosing On Wed, Jun 29, 2022 at 12:48 PM Laurenz Albe wrote: > > On Wed, 2022-06-29 at 00:05 -0700, Andres Freund wrote: > > On 2022-06-29 08:51:10 +02

Re: Hardening PostgreSQL via (optional) ban on local file system access

2022-06-29 Thread Hannu Krosing
025: $ CREATE EXTENSION ... .. and then clean up the sentinel file after, or just make it valid for N minutes from creation Cheers, Hannu Krosing On Wed, Jun 29, 2022 at 8:51 AM Laurenz Albe wrote: > > On Tue, 2022-06-28 at 16:27 -0700, Andres Freund wrote: > > > Experience shows that 9

Re: Hardening PostgreSQL via (optional) ban on local file system access

2022-06-28 Thread Hannu Krosing
to figuring out the set of roles into which one can decompose superuser access in longer run -- Hannu On Tue, Jun 28, 2022 at 8:30 PM Robert Haas wrote: > > On Mon, Jun 27, 2022 at 5:37 PM Hannu Krosing wrote: > > My current thinking is (based on more insights from Andres) that w

Re: Hardening PostgreSQL via (optional) ban on local file system access

2022-06-27 Thread Hannu Krosing
00:08 +0200, Hannu Krosing wrote: > > Hi Pgsql-Hackers > > > > As part of ongoing work on PostgreSQL security hardening we have > > added a capability to disable all file system access (COPY TO/FROM > > [PROGRAM] , pg_*file*() functions, lo_*() functions > &g

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-06-27 Thread Hannu Krosing
> Another thought: for non-x86 platforms, the SIMD nodes degenerate to > "simple loop", and looping over up to 32 elements is not great > (although possibly okay). We could do binary search, but that has bad > branch prediction. I am not sure that for relevant non-x86 platforms SIMD / vector

Re: Hardening PostgreSQL via (optional) ban on local file system access

2022-06-25 Thread Hannu Krosing
ching user superuser to fix whatever needs fixing as superuser. Cheers Hannu On Sat, Jun 25, 2022 at 10:54 PM Hannu Krosing wrote: > > What are your ideas of applying a change similar to above to actually > being a superuser ? > > That is adding a check for "superuser being curr

Re: Hardening PostgreSQL via (optional) ban on local file system access

2022-06-25 Thread Hannu Krosing
What are your ideas of applying a change similar to above to actually being a superuser ? That is adding a check for "superuser being currently available" to function superuser() in ./src/backend/utils/misc/superuser.c ? It could be as simple as a flag that can be set only at startup for maximum

Re: Hardening PostgreSQL via (optional) ban on local file system access

2022-06-24 Thread Hannu Krosing
n 25, 2022 at 2:08 AM David G. Johnston wrote: > > > > On Friday, June 24, 2022, Gurjeet Singh wrote: >> >> On Fri, Jun 24, 2022 at 4:13 PM Andres Freund wrote: >> > On 2022-06-25 00:08:13 +0200, Hannu Krosing wrote: >> >> > > 3) should this be

Re: Hardening PostgreSQL via (optional) ban on local file system access

2022-06-24 Thread Hannu Krosing
the info in not on that page). On Sat, Jun 25, 2022 at 1:46 AM Gurjeet Singh wrote: > > On Fri, Jun 24, 2022 at 4:13 PM Andres Freund wrote: > > On 2022-06-25 00:08:13 +0200, Hannu Krosing wrote: > > > > 3) should this be back-patched (we can provide batches for all > &

Re: Hardening PostgreSQL via (optional) ban on local file system access

2022-06-24 Thread Hannu Krosing
On Sat, Jun 25, 2022 at 1:23 AM Hannu Krosing wrote: > My impression was that this was largely fixed via disabling the old > direct file calling convention, but then again I did not pay much > attention at that time :) I meant of course direct FUNCTION calling convention (Version

Re: Hardening PostgreSQL via (optional) ban on local file system access

2022-06-24 Thread Hannu Krosing
On Sat, Jun 25, 2022 at 1:13 AM Andres Freund wrote: > > Hi, > > On 2022-06-25 00:08:13 +0200, Hannu Krosing wrote: > > Currently the file system access is controlled via being a SUPREUSER > > or having the pg_read_server_files, pg_write_server_files and > > p

Hardening PostgreSQL via (optional) ban on local file system access

2022-06-24 Thread Hannu Krosing
Hi Pgsql-Hackers As part of ongoing work on PostgreSQL security hardening we have added a capability to disable all file system access (COPY TO/FROM [PROGRAM] , pg_*file*() functions, lo_*() functions accessing files, etc) in a way that can not be re-enabled without already having access to the

Which hook to use when overriding utility commands (COPY ...)

2022-03-19 Thread Hannu Krosing
Hi Pgsql-Hackers Which hook should I use when overriding the COPY command in an extension? I am working on adding new functionalities to COPY (compression, index management, various other transports in addition to stdin and file, other data formats, etc...) and while the aim is to contribute

Re: logical decoding and replication of sequences

2021-09-25 Thread Hannu Krosing
t;sequences are not transactional - so rollbacks should not matter" . Or we may get away with most in-detail sequence tracking on the source if we just keep track of the xmin of the sequence and send the sequence info over at commit if it == current_transaction_id ? ----- Hannu Krosing Google Cloud - We

Re: logical replication restrictions

2021-09-25 Thread Hannu Krosing
These might be addressed in future releases." so there is no exclusivity of being either a restriction or TODO. > [1] - https://wiki.postgresql.org/wiki/Todo > [2] - > https://www.postgresql.org/docs/devel/logical-replication-restrictions.html - Hannu Krosing Google Cloud - We have a long list of planned contributions and we are hiring. Contact me if interested.

Re: prevent immature WAL streaming

2021-09-24 Thread Hannu Krosing
ity, for example 2-out-of-3 where primary is one of the 3 and not necessarily the most durable one? ----- Hannu Krosing Google Cloud - We have a long list of planned contributions and we are hiring. Contact me if interested. On Fri, Sep 24, 2021 at 4:33 AM Alvaro Herrera wrote: > > On 20

Re: WIP: System Versioned Temporal Table

2021-09-20 Thread Hannu Krosing
On Mon, Sep 20, 2021 at 7:09 AM Corey Huinker wrote: > > On Sun, Sep 19, 2021 at 3:12 PM Hannu Krosing wrote: >> >> A side table has the nice additional benefit that we can very easily >> version the *table structure* so when we ALTER TABLE and the table >> struct

Re: WIP: System Versioned Temporal Table

2021-09-19 Thread Hannu Krosing
And we may even want to partition history tables for speed, storage cost or just to drop very ancient history - Hannu Krosing Google Cloud - We have a long list of planned contributions and we are hiring. Contact me if interested. On Sun, Sep 19, 2021 at 8:32 PM Simon Riggs wrote: > >

Re: The Free Space Map: Problems and Opportunities

2021-09-08 Thread Hannu Krosing
On Wed, Sep 8, 2021 at 6:52 AM Peter Geoghegan wrote: > > On Tue, Sep 7, 2021 at 5:25 AM Hannu Krosing wrote: > > Are you speaking of just heap pages here or also index pages ? > > Mostly heap pages, but FWIW I think it could work for index tuples > too, with retail index tu

Re: The Free Space Map: Problems and Opportunities

2021-09-07 Thread Hannu Krosing
On Tue, Sep 7, 2021 at 2:29 AM Peter Geoghegan wrote: > > On Mon, Sep 6, 2021 at 4:33 PM Hannu Krosing wrote: > > When I have been thinking of this type of problem it seems that the > > latest -- and correct :) -- place which should do all kinds of > > cleanup like

Re: The Free Space Map: Problems and Opportunities

2021-09-06 Thread Hannu Krosing
will not improve our feeder mechanism to have back-to-back incoming commands, which can already be done today, but which I have seen seldom used. Cheers, - Hannu Krosing Google Cloud - We have a long list of planned contributions and we are hiring. Contact me if interested. On Mon, Sep 6, 20

Re: Middleware Messages for FE/BE

2021-08-19 Thread Hannu Krosing
middleware component at which point they should be standardised . - Hannu Krosing Google Cloud - We have a long list of planned contributions and we are hiring. Contact me if interested. On Thu, Aug 19, 2021 at 8:29 PM Simon Riggs wrote: > > On Thu, 19 Aug 2021 at 19:04, Jesper Pe

Re: Middleware Messages for FE/BE

2021-08-19 Thread Hannu Krosing
in writing Of course there are (and should be) ways to still use the WALs normally for cases where replica x.x.x.x does not exists, like PITR And making this play nicely with Logical Decoding is another can of worms needing to be solved, but it is a start to becoming "Cloud Native" :) -

Re: Middleware Messages for FE/BE

2021-08-19 Thread Hannu Krosing
One more set of "standard middleware messages" clients/middleware could turn on could be reporting LSNs * various local LSNs for progress of WAL persisting * reporting replication state of some or all replicas ----- Hannu Krosing Google Cloud - We have a long list of planned con

Re: Middleware Messages for FE/BE

2021-08-19 Thread Hannu Krosing
der moving some load away" or "Planning to switch over to replica x.x.x.x, please follow" - Hannu Krosing Google Cloud - We have a long list of planned contributions and we are hiring. Contact me if interested. On Thu, Aug 19, 2021 at 10:33 AM Simon Riggs wrote: > >

pgbench functions as extension

2021-08-18 Thread Hannu Krosing
ed to have exact same behaviour as the ones in bundled pgbench would enable more work to be pushed to database and generally resu;lt in better interoperability for tests. -- [*] https://www.postgresql.org/docs/13/pgbench.html#PGBENCH-BUILTIN-FUNCTIONS ----- Hannu Krosing Google Cloud - We have a

Re: NAMEDATALEN increase because of non-latin languages

2021-08-18 Thread Hannu Krosing
that at least the truncation does not cut utf-8 characters in half ? - Hannu Krosing Google Cloud - We have a long list of planned contributions and we are hiring. Contact me if interested. On Wed, Aug 18, 2021 at 1:33 PM Julien Rouhaud wrote: > > On Wed, Aug 18, 2021 at 7:27 PM John Naylor &g

Is there now an official way to pass column projection to heap AM

2021-08-18 Thread Hannu Krosing
ods As this was already almost a year ago, have there been any developments in the core PostgreSQL code for supporting this ? Or is projection still supported only in FDWs ? -- [*] https://github.com/greenplum-db/postgres/blob/zedstore/src/backend/access/zedstore/README - Hannu Krosin

Re: [PoC] Improve dead tuple storage for lazy vacuum

2021-07-08 Thread Hannu Krosing
ing > like cheap insurance that we should be glad to pay for. If the most expensive operation is sorting a few hundred of tids, then this should be fast enough. My worries were more that after the sorting we can not to dsimple index lookups for them, but each needs to be found via bse

Re: [PoC] Improve dead tuple storage for lazy vacuum

2021-07-08 Thread Hannu Krosing
ck() to do this. It does the same > basic thing already. Also a lot of testing would be needed to figure out which strategy fits best for which distribution of dead tuples, and possibly their relation to the order of tuples to check from indexes . Cheers -- Hannu Krosing Google Cloud - We have a long list of planned contributions and we are hiring. Contact me if interested.

Re: pgbench logging broken by time logic changes

2021-07-08 Thread Hannu Krosing
an it solves :) Cheers - Hannu Krosing Google Cloud - We have a long list of planned contributions and we are hiring. Contact me if interested.

Re: [PoC] Improve dead tuple storage for lazy vacuum

2021-07-08 Thread Hannu Krosing
dead tids into a file makes it trivial (well, almost :) ) to parallelize the initial heap scan, so more resources can be thrown at it if available. Cheers - Hannu Krosing Google Cloud - We have a long list of planned contributions and we are hiring. Contact me if interested. On Thu, Jul 8

Re: .ready and .done files considered harmful

2021-05-06 Thread Hannu Krosing
How are you envisioning the shared-memory signaling should work in the original sample case, where the archiver had been failing for half a year ? Or should we perhaps have a system table for ready-to-archive WAL files to get around limitation sof file system to return just the needed files with

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Hannu Krosing
- Hannu Krosing On Thu, May 6, 2021 at 4:53 AM Jeff Davis wrote: > > On Thu, 2021-05-06 at 03:26 +0200, Hannu Krosing wrote: > > How hard would it be to declare TID as current ItemPointerData with > > some values prohibited (NULL, SpecTokenOff

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Hannu Krosing
ut would this not have the downside that all the secondary indexes will blow up as they now need to have the full table row as the TID ? - Hannu Krosing

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Hannu Krosing
likely should be moved to access/heap/ Doing it this way would leave us with some manageable complexity in mapping from TID to 48-bit integer and/or 3 wanted positions in 2^32 Hannu Krosing On Wed, May 5, 2021 at 8:40 PM Peter Geoghegan wrote: > > On Wed, May 5, 2021 at 11

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-20 Thread Hannu Krosing
It would be really convenient if user-visible serialisations of the query id had something that identifies the computation method. maybe prefix 'N' for internal, 'S' for pg_stat_statements etc. This would immediately show in logs at what point the id calculator was changed On Fri, Mar 19, 2021

Re: shared memory stats: high level design decisions: consistency, dropping

2021-03-20 Thread Hannu Krosing
> But now we could instead schedule stats to be removed at commit time. That's not trivial of course, as we'd need to handle cases where the commit fails after the commit record, but before processing the dropped stats. We likely can not remove them at commit time, but only after the oldest open

Re: shared memory stats: high level design decisions: consistency, dropping

2021-03-19 Thread Hannu Krosing
On Sat, Mar 20, 2021 at 1:21 AM Andres Freund wrote: > > Hi, > > On 2021-03-20 01:16:31 +0100, Hannu Krosing wrote: > > > But now we could instead schedule stats to be removed at commit > > time. That's not trivial of course, as we'd need to handle cases where

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-19 Thread Hannu Krosing
On Fri, Mar 19, 2021 at 2:29 PM Bruce Momjian wrote: > > OK, that makes perfect sense. I think the best solution is to document > that compute_query_id just controls the built-in computation of the > query id, and that extensions can also compute it if this is off, and > pg_stat_activity and

Re: pl/pgsql feature request: shorthand for argument and local variable references

2021-03-19 Thread Hannu Krosing
On Thu, Mar 18, 2021 at 4:23 PM Pavel Stehule wrote: > But we don't support this feature. We are changing just a top scope's label. > So syntax "ALIAS FOR FUNCTION is not good. The user can have false hopes In this case it looks like it should go together with other labels and have <<

Re: pl/pgsql feature request: shorthand for argument and local variable references

2021-03-18 Thread Hannu Krosing
On Thu, Mar 18, 2021 at 3:45 PM Pavel Stehule wrote: > > > > čt 18. 3. 2021 v 15:19 odesílatel Hannu Krosing napsal: ... >> Variation could be >> >> DECLARE >>fnarg ALIAS FOR FUNCTION a_function_with_an_inconveniently_long_name; >> >> so it is

Re: pl/pgsql feature request: shorthand for argument and local variable references

2021-03-18 Thread Hannu Krosing
, 2021 at 3:19 PM Hannu Krosing wrote: > > On Thu, Mar 18, 2021 at 5:27 AM Pavel Stehule wrote: > > > > > > There are few main reasons: > > > > a) compile options are available already - so I don't need invent new > > syntax - #OPTION DUMP,

Re: pl/pgsql feature request: shorthand for argument and local variable references

2021-03-18 Thread Hannu Krosing
On Thu, Mar 18, 2021 at 5:27 AM Pavel Stehule wrote: > > > There are few main reasons: > > a) compile options are available already - so I don't need invent new syntax > - #OPTION DUMP, #PRINT_STRICT ON, #VARIABLE_CONFLICT ERROR Are these documented anywhere ? At least a quick search for

Re: pl/pgsql feature request: shorthand for argument and local variable references

2021-03-17 Thread Hannu Krosing
why are you using yet another special syntax for this ? would it not be better to do something like this: CREATE FUNCTION a_reall_long_and_winding_function_name(i int, out o int) LANGUAGE plpgsql AS $plpgsql$ DECLARE args function_name_alias BEGIN args.o = 2 * args.i END; $plpgsql$; or at

Re: Boundary value check in lazy_tid_reaped()

2021-03-16 Thread Hannu Krosing
iteration for each value. Of course it is possible that this has a very bad RAM access behaviour and is no win at all even if it otherways works. -- Hannu Krosing On Tue, Mar 16, 2021 at 10:08 PM Peter Geoghegan wrote: > On Sun, Mar 14, 2021 at 4:22 PM Thomas Munro > wrote: > > BTW

Re: Extensibility of the PostgreSQL wire protocol

2021-03-04 Thread Hannu Krosing
On Thu, Mar 4, 2021 at 9:55 PM Jan Wieck wrote: > > Another possibility, and this is what is being used by the AWS team > implementing the TDS protocol for Babelfish, is to completely replace > the entire TCOP mainloop function PostgresMain(). I suspect this is the only reasonable way to do it

Re: PROXY protocol support

2021-03-04 Thread Hannu Krosing
The current proposal seems to miss the case of transaction pooling (and statement pooling) where the same established connection multiplexes transactions / statements from multiple remote clients. What we would need for that case would be a functionl pg_set_remote_client_address( be_key,

Re: Extensibility of the PostgreSQL wire protocol

2021-03-03 Thread Hannu Krosing
I have not looked at the actual patch, but does it allow you to set up its own channels to listen to ? For example if I'd want to set up a server to listen to incoming connections over QUIC [1] - a protocol which create a connection over UDP and allows clients to move to new IP addresses (among

Re: We should stop telling users to "vacuum that database in single-user mode"

2021-03-03 Thread Hannu Krosing
On Wed, Mar 3, 2021 at 11:33 AM David Rowley wrote: > > On Wed, 3 Mar 2021 at 21:44, Magnus Hagander wrote: ... > > I think we misunderstand each other. I meant this only as a comment > > about the idea of ignoring the cost limit in single user mode -- that > > is, it's a reason to *want* vacuum

We should stop telling users to "vacuum that database in single-user mode"

2021-03-01 Thread Hannu Krosing
omplete in 15-30 min, after which the database is available for writes again. Cheers, Hannu Krosing

Re: CREATE ROUTINE MAPPING

2018-09-10 Thread Hannu Krosing
Hi Corey Have you looked at pl/proxy ? It does this and then some (sharding) It actually started out as a set of pl/pythonu functions, but then got formalized into a full extension language for defining remote (potentially sharded) function calls Best Regards Hannu Krosng On Fri, 12 Jan

Re: Proposal: http2 wire format

2018-03-29 Thread Hannu Krosing
> > > * room for other resultset formats later. Like Damir, I really want to > add > > protobuf or json serializations of result sets at some point, mainly so > we > > can return "entity graphs" in graph representation rather than left-join > > projection. > > -1. I don't think this belongs in

A Generic Question about Generic type subscripting

2018-01-28 Thread Hannu Krosing
Sorry for being late to the party I started looking at the thread about "Generic type subscripting" and am wondering, why does it take the approach of modifying pg_type and modifying lots of internal functions, when instead it could be defined in a much lighter and less intrusive way as an

Re: AS OF queries

2017-12-27 Thread Hannu Krosing
s=# select * from foo asof timestamp '2017-12-20 15:10'; >  pk | ts     |  val > ++ >   2 | 2017-12-20 14:59:22.933753 | insert >   3 | 2017-12-20 14:59:27.87712  | insert >   1 | 2017-12-20 15:09:17.046047 | upd > (3 rows) > > > Comments and feedback are welcome:) > -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability https://2ndquadrant.com/

Re: Ethiopian calendar year(DATE TYPE) are different from the Gregorian calendar year

2017-12-27 Thread Hannu Krosing
t least Apple seems to have an Ethiopian calendar locale https://developer.apple.com/documentation/foundation/nscalendar -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability https://2ndquadrant.com/