Re: [HACKERS] make async slave to wait for lsn to be replayed

2017-10-31 Thread Ants Aasma
xids. On master that would be just a ShareLock on the transactionid. On standby it would wait for the commit or rollback record for that transaction to be replayed. Robert made a good point that people will still rely on the token being an LSN, but perhaps they will be slightly less angry when we expli

Re: [HACKERS] make async slave to wait for lsn to be replayed

2017-10-26 Thread Ants Aasma
On Mon, Oct 23, 2017 at 12:29 PM, Ivan Kartyshov <i.kartys...@postgrespro.ru> wrote: > Ants Aasma писал 2017-09-26 13:00: >> >> Exposing this interface as WAITLSN will encode that visibility order >> matches LSN order. This removes any chance of fixing for example >

Re: [HACKERS] Discussion on missing optimizations

2017-10-09 Thread Ants Aasma
viewing it. The tautological col = col comparison on is occasionally used as a planner "hint" to correct for row count overestimates. Not a great solution, but PostgreSQL doesn't really have a better way to guide the planner. Those queries will now have to do something else, like col =

Re: [HACKERS] JIT compiling - v4.0

2017-10-04 Thread Ants Aasma
far as I can tell these are not in mainline LLVM. Is there a branch or patchset of LLVM available somewhere that I need to use this? Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] make async slave to wait for lsn to be replayed

2017-09-26 Thread Ants Aasma
ixing for example visibility order of async/vs/sync transactions. Just renaming it so the token is an opaque commit visibility token that just happens to be a LSN would still allow for progress in transaction management. For example, making PostgreSQL distributed will likely want timestamp

Re: [HACKERS] Hooks to track changed pages for backup purposes

2017-09-13 Thread Ants Aasma
it off, using 4 cores for 10 minutes - making the ratio of work from generating WAL to parsing it be about 750:1. Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] emergency outage requiring database restart

2017-08-10 Thread Ants Aasma
On Wed, Jan 18, 2017 at 4:33 PM, Merlin Moncure <mmonc...@gmail.com> wrote: > On Wed, Jan 18, 2017 at 4:11 AM, Ants Aasma <ants.aa...@eesti.ee> wrote: >> On Wed, Jan 4, 2017 at 5:36 PM, Merlin Moncure <mmonc...@gmail.com> wrote: >>> Still getting checksum fail

Re: [HACKERS] Reducing pg_ctl's reaction time

2017-06-28 Thread Ants Aasma
shutdown checkpoint complete at leisure without risk for split brain. Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP: Data at rest encryption

2017-06-14 Thread Ants Aasma
eds to be encrypted. Me neither, but it currently is, and it looks like that's broken in a "silently corrupts your data" way in face of torn writes. Using OFB mode (xor plaintext with pseudorandom stream for cipher) looks like it might help here, if other approaches fail. Regards, Ants

Re: [HACKERS] WIP: Data at rest encryption

2017-06-14 Thread Ants Aasma
got assumptions > like this baked into the system, but I'm guessing there are a bunch. I think we need to require wal_log_hints=on when encryption is enabled. Currently I have not considered tearing on CLOG bits. Other SLRUs probably have similar issues. I need to think a bit about how to solv

Re: [HACKERS] WIP: Data at rest encryption

2017-06-12 Thread Ants Aasma
On Mon, Jun 12, 2017 at 10:38 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Mon, Jun 13, 2016 at 11:07 AM, Peter Eisentraut > <peter.eisentr...@2ndquadrant.com> wrote: >> On 6/7/16 9:56 AM, Ants Aasma wrote: >>> >>> Similar things can be achieved wit

Re: [HACKERS] WIP: Faster Expression Processing v4

2017-03-25 Thread Ants Aasma
e interpreter is about 15% (5% speedup on a workload that spends 1/3 in ExecInterpExpr). My idea of prefetching op->resnull/resvalue to local vars before the indirect jump is somewhere between a tiny benefit and no effect, certainly not worth introducing extra complexity. Clang 3.8 does the corr

Re: [HACKERS] Checksums by default?

2017-02-24 Thread Ants Aasma
On Fri, Feb 24, 2017 at 10:30 PM, Bruce Momjian <br...@momjian.us> wrote: > On Fri, Feb 24, 2017 at 10:09:50PM +0200, Ants Aasma wrote: >> On Fri, Feb 24, 2017 at 9:37 PM, Bruce Momjian <br...@momjian.us> wrote: >> > Oh, that's why we will hopefully event

Re: [HACKERS] Checksums by default?

2017-02-24 Thread Ants Aasma
On Fri, Feb 24, 2017 at 9:49 PM, Jim Nasby wrote: > On 2/24/17 12:30 PM, Tomas Vondra wrote: >> >> In any case, we can't just build x86-64 packages with compile-time >> SSE4.1 checks. > > > Dumb question... since we're already discussing llvm for the executor, would >

Re: [HACKERS] Checksums by default?

2017-02-24 Thread Ants Aasma
orking on swapping out the current algorithm. And I don't really see a reason to, it would introduce a load of headaches for no real gain. Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Checksums by default?

2017-02-24 Thread Ants Aasma
are a GPU built > into CPUs. Both LLVM and GCC are capable of compiling the code that we have to a vectorized loop using SSE4.1 or AVX2 instructions given the proper compilation flags. This is exactly what was giving the speedup in the test I showed in my e-mail. Regards, Ants Aasma -- Sent via pgs

Re: [HACKERS] Checksums by default?

2017-02-24 Thread Ants Aasma
On Fri, Feb 24, 2017 at 7:47 PM, Bruce Momjian <br...@momjian.us> wrote: > On Sat, Jan 21, 2017 at 09:02:25PM +0200, Ants Aasma wrote: >> > It might be worth looking into using the CRC CPU instruction to reduce this >> > overhead, like we do for the WAL checksums

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Ants Aasma
full_papers/bairavasundaram/bairavasundaram.pdf Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Checksums by default?

2017-01-24 Thread Ants Aasma
tion to data corruption is really important for minimizing data loss. Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Checksums by default?

2017-01-21 Thread Ants Aasma
On Sat, Jan 21, 2017 at 10:16 PM, Michael Banck <michael.ba...@credativ.de> wrote: > On Sat, Jan 21, 2017 at 09:02:25PM +0200, Ants Aasma wrote: >> On Sat, Jan 21, 2017 at 6:41 PM, Andreas Karlsson <andr...@proxel.se> wrote: >> > It might be worth looking into

Re: [HACKERS] Checksums by default?

2017-01-21 Thread Ants Aasma
re sitting around a similarly simple tool for turning on checksums while the database is offline. FWIW, based on customers and fellow conference goers I have talked to most would gladly take the performance hit, but not the downtime to turn it on on an existing database. Regards, Ants Aasma -- Sent via

Re: [HACKERS] Checksums by default?

2017-01-21 Thread Ants Aasma
ed CRC. That said the actual checksum calculation was not a big issue for performance. The only way to make it really matter was with a larger than shared buffers smaller than RAM workload with tiny per page execution overhead. My test case was SELECT COUNT(*) on wide rows with a small fill factor. H

Re: [HACKERS] Causal reads take II

2017-01-19 Thread Ants Aasma
On Thu, Jan 19, 2017 at 2:22 PM, Thomas Munro <thomas.mu...@enterprisedb.com> wrote: > On Thu, Jan 19, 2017 at 8:11 PM, Ants Aasma <ants.aa...@eesti.ee> wrote: >> On Tue, Jan 3, 2017 at 3:43 AM, Thomas Munro >> <thomas.mu...@enterprisedb.com> wrote: >>>

Re: [HACKERS] Causal reads take II

2017-01-18 Thread Ants Aasma
ire either keeping some per client state or a global agreement on what snapshots are safe to provide, both of which you tried to avoid for this feature. Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] emergency outage requiring database restart

2017-01-18 Thread Ants Aasma
ctual checksums are in data files surrounding the failed page. If the requested block number contains something completely else, but the page that follows contains the expected checksum value, then it would support this theory. Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] Replication slot xmin is not reset if HS feedback is turned off while standby is shut down

2017-01-13 Thread Ants Aasma
happier if it had a test to go with it. > > You could probably just add to src/test/recover/t/001 which now > contains my additions for hot standby. Do you feel the test in the attached patch is enough or would you like to see anything else covered? Regards, Ants Aasma diff --git a/src/backend

Re: [HACKERS] Replication slot xmin is not reset if HS feedback is turned off while standby is shut down

2017-01-05 Thread Ants Aasma
t into a TAP test in src/test/recovery? Simon has said he would like to commit this fix. I'd personally be happier if it had a test to go with it. You could probably just add to src/test/recover/t/001 which now contains my additions for hot standby. I'm travelling right now, but I should be able to give it a shot next week. Regards, Ants Aasma

Re: [HACKERS] Replication slot xmin is not reset if HS feedback is turned off while standby is shut down

2016-12-21 Thread Ants Aasma
On Wed, Dec 21, 2016 at 2:09 PM, Craig Ringer <cr...@2ndquadrant.com> wrote: > On 21 December 2016 at 15:40, Ants Aasma <ants.aa...@eesti.ee> wrote: > >>> So -1 on this part of the patch, unless there's something I've >>> misunderstood. >> >> Cur

Re: [HACKERS] Replication slot xmin is not reset if HS feedback is turned off while standby is shut down

2016-12-20 Thread Ants Aasma
eplica slots wanting to hold back xmin, where resetting the parents xmin is indeed wrong. Do you know if GetOldestXmin() is safe at this point and we can just remove the HotStandbyActive() check? Otherwise I think the correct approach is to move the check and return inside the hot_standby_feedback c

[HACKERS] Replication slot xmin is not reset if HS feedback is turned off while standby is shut down

2016-12-20 Thread Ants Aasma
. A shell script to reproduce the problem is also attached, adjust the PGPATH variable to your postgres install and run in an empty directory. Regards, Ants Aasma diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index cc3cf7d..31333ec 100644 --- a/src

Re: [HACKERS] emergency outage requiring database restart

2016-10-27 Thread Ants Aasma
re different block numbers were used for writing out and reading in the page. Either the blocknum gets corrupted between calculating the checksum and writing the page out (unlikely given the proximity), or the pages are somehow getting transposed in the storage. Regards, Ants Aasma --

Re: [HACKERS] Pluggable storage

2016-08-18 Thread Ants Aasma
r columns... Hm. What about going even further than [1] in converting the executor to being opcode based and merging projection and qual evaluation to a single pass? Optimizer would then have some leeway about how to order column extraction and qual evaluation. Might even be worth it to special case some fun

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-10 Thread Ants Aasma
sync record to flush and become visible. Simplest solution is to not require CSN == LSN and just assign a CSN value immediately before becoming visible. Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-10 Thread Ants Aasma
rily conflate commit record durability and transaction visibility ordering. Not having them tied together allows for an external source to provide CSN values, allowing for interesting distributed transaction implementations. E.g. using a timestamp as the CSN a'la Google Spanner and the TrueTime API. Re

Re: [HACKERS] multivariate statistics (v19)

2016-08-10 Thread Ants Aasma
cts of Predicates" [1]. While effective it seems to be quite heavy-weight, so would probably need support for tiered optimization. [1] https://courses.cs.washington.edu/courses/cse544/11wi/papers/markl-vldb-2005.pdf Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hacke

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-10 Thread Ants Aasma
e, would take care most of the traffic in most of the cases. Looks like it would be a good trade-off for complexity/performance. * To keep workloads with wildly varying transaction lengths in bounded amount of memory, a significantly more complex hybrid snapshot scheme is needed. It remains to be see

Re: [HACKERS] WIP: Data at rest encryption

2016-06-13 Thread Ants Aasma
On Mon, Jun 13, 2016 at 5:17 AM, Michael Paquier <michael.paqu...@gmail.com> wrote: > On Sun, Jun 12, 2016 at 4:13 PM, Ants Aasma <ants.aa...@gmail.com> wrote: >>> I feel separate file is better to include the key data instead of pg_control >>> file. >>

Re: [HACKERS] WIP: Data at rest encryption

2016-06-12 Thread Ants Aasma
trol > file. I guess that would be more flexible. However I think at least the fact that the database is encrypted should remain in the control file to provide useful error messages for faulty backup procedures. Thanks for your input. Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] WIP: Data at rest encryption

2016-06-07 Thread Ants Aasma
ent cipher primitive. Regards, Ants Aasma diff --git a/contrib/pgcrypto/Makefile b/contrib/pgcrypto/Makefile index 18bad1a..04ce887 100644 --- a/contrib/pgcrypto/Makefile +++ b/contrib/pgcrypto/Makefile @@ -20,7 +20,7 @@ SRCS = pgcrypto.c px.c px-hmac.c px-crypt.c \ mbuf.c pgp.c pgp-armor.c pgp-

Re: [HACKERS] Does people favor to have matrix data type?

2016-05-25 Thread Ants Aasma
n to do similarity matching. However using real[] instead of a hypothetical realvector or realmatrix did not prove to be a huge overhead, so overall I'm on the fence for the usefulness of a special type. Maybe a helper function or two to validate the additional restrictions in a check constraint would be

Re: [HACKERS] Is the unfair lwlock behavior intended?

2016-05-24 Thread Ants Aasma
ved. It would be helpful if you could test if it's still possible to trigger starvation with the new code. Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Is the unfair lwlock behavior intended?

2016-05-24 Thread Ants Aasma
work, ping me to write up some of the insights and ideas I have had while thinking about this topic. Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] asynchronous and vectorized execution

2016-05-11 Thread Ants Aasma
On Wed, May 11, 2016 at 3:52 AM, Andres Freund <and...@anarazel.de> wrote: > On 2016-05-11 03:20:12 +0300, Ants Aasma wrote: >> On Tue, May 10, 2016 at 7:56 PM, Robert Haas <robertmh...@gmail.com> wrote: >> > On Mon, May 9, 2016 at 8:34 PM, David Rowley >> &g

Re: [HACKERS] asynchronous and vectorized execution

2016-05-10 Thread Ants Aasma
roborated by what I have seen found by other VM implementations. Once you get the data into an uniform format where vectorized execution could be used, the CPU execution resources are no longer the bottleneck. Memory bandwidth gets in the way, unless each input value is used in multiple calculat

Re: [HACKERS] Reviewing freeze map code

2016-05-09 Thread Ants Aasma
y the best time to hastily slap on such a feature, but I just wanted the thought to be out there for consideration. Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] what to revert

2016-05-05 Thread Ants Aasma
5. mai 2016 6:14 AM kirjutas kuupäeval "Andres Freund" <and...@anarazel.de>: > > On 2016-05-05 06:08:39 +0300, Ants Aasma wrote: > > On 5 May 2016 1:28 a.m., "Andres Freund" <and...@anarazel.de> wrote: > > > On 2016-05-04 18:22:27 -0400, Robe

Re: [HACKERS] what to revert

2016-05-04 Thread Ants Aasma
ll likely be tied to the time xids are > assigned. That seems perfectly alright, but it'll change behaviour. FWIW moving the maintenance to a clock tick process will not change user visible semantics in any significant way. The change could easily be made in the next release. Regards, Ants Aasma

Re: [HACKERS] what to revert

2016-05-03 Thread Ants Aasma
real world use cases. Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-05-02 Thread Ants Aasma
g like autovacuum launcher this task doesn't seem too bad and the consequence of falling behind is just delayed timing out of old snapshots. As far as I can see this approach would get rid of any scalability issues, but it is a pretty significant change and requires 64bit atomic reads to get rid o

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-04-21 Thread Ants Aasma
On Thu, Apr 21, 2016 at 5:16 PM, Kevin Grittner <kgri...@gmail.com> wrote: > On Wed, Apr 20, 2016 at 8:08 PM, Ants Aasma <ants.aa...@eesti.ee> wrote: > >> However, while checking out if my proof of concept patch actually >> works I hit another issue. I couldn

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-04-20 Thread Ants Aasma
I'm seeing that the old rows do not get cleaned up, and that I'm only seeing the interfering query get cancelled when old_snapshot_threshold = 0. Larger values do not result in cancellation. Am I doing something wrong or is the feature just not working at all? Regards, Ants Aasma diff --git

Re: [HACKERS] raw output from copy

2016-04-12 Thread Ants Aasma
buffers, Redis bulk load, BSON, ASN.1 or whatever else serialisation format du jour. It will still have the same backwards compatibility issues as adding the raw output, but the payoff is greater. Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Using quicksort for every external sort run

2015-12-12 Thread Ants Aasma
network comparison order in an array and use that to drive a simple loop. The code size would be pretty similar to insertion sort and the loop overhead should mostly be hidden by the CPU OoO machinery. Probably won't help much, but would be interesting and simple enough to try out. Can you share you

Re: [HACKERS] W-TinyLfu for cache eviction

2015-12-09 Thread Ants Aasma
ocking solution. I think a global lock would be good enough for a proof of concept that only evaluates cache hit ratios. Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Re: Revisiting pg_stat_statements and IN() (Was: Re: pg_stat_statements fingerprinting logic and ArrayExpr)

2015-11-24 Thread Ants Aasma
Having runtime intelligence in PostgreSQL that would be capable of distinguishing interesting variations from irrelevant doesn't seem like a feasible plan. In my view the best we could do is to aim to have entries roughly correspond to application query invocation points and leave the more complex stati

Re: [HACKERS] parallelism and sorting

2015-11-24 Thread Ants Aasma
eeds to be periodically rebuilt with a newer timestamp constant, and the rebuild would be a lot faster if it could use the existing index to perform an index only scan of 10% of data instead of scanning and sorting the full table. Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-

Re: [HACKERS] Proposal: "Causal reads" mode for load balancing reads without stale data

2015-11-11 Thread Ants Aasma
read request arrives, or pick a host to use based on the token, and then there are "dumb" clients that want to use write side waits. Also, it should be possible to configure which standbys are considered for waiting on. Otherwise a reporting slave will occasionally catch up enough to be considered &

Re: [HACKERS] Allow snapshot too old error, to prevent bloat

2015-02-19 Thread Ants Aasma
. This way the error is deterministic, vacuum on the high churn tables doesn't have to wait for the old transaction delay to expire and we avoid a hard to tune GUC (what do I need to set old_snapshot_threshold to, to reduce bloat while not having normal transactions abort). Regards, Ants Aasma

Re: [HACKERS] Allow snapshot too old error, to prevent bloat

2015-02-19 Thread Ants Aasma
the datastructure in shared memory as I don't have a great idea what to use there. It's somewhat similar to the lock table in that in theory the size is unbounded, but in practice it's expected to be relatively tiny. Regards, Ants Aasma

[HACKERS] Using RTLD_DEEPBIND to handle symbol conflicts in loaded libraries

2014-11-26 Thread Ants Aasma
be a good idea? Are there any similar options for other platforms? Alternatively, does anyone know of linker flags that would give a similar effect? Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de diff --git a/src/backend

Re: [HACKERS] What exactly is our CRC algorithm?

2014-11-21 Thread Ants Aasma
On Fri, Nov 21, 2014 at 12:11 PM, Abhijit Menon-Sen a...@2ndquadrant.com wrote: If anyone has other suggestions, I'm all ears. Do you have a WIP patch I could take a look at and tweak? Maybe there's something about the compilers code generation that could be improved. Regards, Ants Aasma

Re: [HACKERS] Failback to old master

2014-11-13 Thread Ants Aasma
at tli1.x2, wal sender does tliSwitchPoint(tli1) to lookup x1, finds that x1 x2 and gives the error requested starting point %X/%X on timeline %u is not in this server's history. The alignment of x2 on tli2 does not play a role here. Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH

Re: [HACKERS] Failback to old master

2014-11-12 Thread Ants Aasma
integers, not GUID's, but at least it's significantly harder. Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription

Re: [HACKERS] WIP: dynahash replacement for buffer table

2014-10-16 Thread Ants Aasma
On Oct 15, 2014 7:32 PM, Ants Aasma a...@cybertec.at wrote: I'm imagining a bucketized cuckoo hash with 5 item buckets (5-way associativity). This allows us to fit the bucket onto 2 regular sized cache lines and have 8 bytes left over. Buckets would be protected by seqlocks stored in the extra

Re: [HACKERS] WIP: dynahash replacement for buffer table

2014-10-15 Thread Ants Aasma
Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Value locking Wiki page

2014-10-01 Thread Ants Aasma
without a retry loop. The hypothetical ON CONFLICT REPLACE and ON CONFLICT UPDATE-AND-THEN-INSERT modes would also make sense in the unique index case. Not saying that I view this as necessary for the first cut of the feature, just providing an example where it could be useful. Regards, Ants Aasma

Re: [HACKERS] Scaling shared buffer eviction

2014-09-26 Thread Ants Aasma
first and possibly have better branch prediction. Do you have a workload where I could test if this helps alleviate the comparison overhead? Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de diff --git a/src/backend

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-09-24 Thread Ants Aasma
On Tue, Sep 23, 2014 at 8:15 PM, Florian Weimer f...@deneb.enyo.de wrote: * Ants Aasma: CRC has exactly one hardware implementation in general purpose CPU's I'm pretty sure that's not true. Many general purpose CPUs have CRC circuity, and there must be some which also expose them

Re: [HACKERS] Scaling shared buffer eviction

2014-09-12 Thread Ants Aasma
to the pinning backend, not a huge amount of savings. Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-09-12 Thread Ants Aasma
for testing this patch to see if the performance improvement is due to less data being checksummed. Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-09-12 Thread Ants Aasma
lookup tables. If we choose to stay with CRC we must accept that we can only solve the performance issues for Intel CPUs and provide slight alleviation for others. Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-09-04 Thread Ants Aasma
tablespace. Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-09-04 Thread Ants Aasma
and then compress it to a list of buffer IDs that need to be written out. This way most of the time you only need a small array and the large array is only needed for a fraction of a second. Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-08 Thread Ants Aasma
- this would just be a bit of extra arithmetic. I don't think binary search is the main problem here. Objects are usually reasonably sized, while arrays are more likely to be huge. To make matters worse, jsonb - int goes from O(1) to O(n). Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH

Re: [HACKERS] Reporting the commit LSN at commit time

2014-08-07 Thread Ants Aasma
commit. Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] better atomics - v0.5

2014-07-02 Thread Ants Aasma
for Apache Derby. You may find some interesting ideas in there. [1] http://code.google.com/p/derby-nb/source/browse/trunk/derby-nb/ICDE10_conf_full_409.pdf Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent

Re: [HACKERS] gettimeofday is at the end of its usefulness?

2014-05-14 Thread Ants Aasma
, Ants Aasma -- Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal for CSN based snapshots

2014-05-12 Thread Ants Aasma
mostly the same ground work and if turns out to work well enough, simpler is always better. Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org

Re: [HACKERS] Proposal for CSN based snapshots

2014-05-12 Thread Ants Aasma
, SimpleLruReadPage() does SlruReportIOError(), which in turn does ereport(ERROR), while inside a critical section initiated in RecordTransactionCommit(). Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql

Re: [HACKERS] Proposal for CSN based snapshots

2014-05-12 Thread Ants Aasma
if those records are visible or not unless they're very recent transactions which started in that short window while the committing transaction was in the process of committing. I don't believe this is worth the complexity. The contention window is extremely short here. Regards, Ants Aasma

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-16 Thread Ants Aasma
-Pro with a larger number of levels, ranging from cold, hot and scalding to infernal. :) Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-15 Thread Ants Aasma
://www.postgresql.org/message-id/ca+tgmozypeyhwauejvyy9a5andoulcf33wtnprfr9sycw30...@mail.gmail.com Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org

[HACKERS] Missing pfree in logical_heap_rewrite_flush_mappings()

2014-03-26 Thread Ants Aasma
It seems to me that when flushing logical mappings to disk, each mapping file leaks the buffer used to pass the mappings to XLogInsert. Also, it seems consistent to allocate that buffer in the RewriteState memory context. Patch attached. Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH

Re: [HACKERS] Memory ordering issue in LWLockRelease, WakeupWaiters, WALInsertSlotRelease

2014-02-12 Thread Ants Aasma
order) provides that guarantee in this specific case. Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2013-12-15 Thread Ants Aasma
better ideas please let them be known. I'd be a bit inclined to build the terminology around reverse instead of negative --- the latter seems a bit too arithmetic-centric. But that's just MHO. To contribute to the bike shedding, inverse is often used in similar contexts. -- Ants Aasma

Re: [HACKERS] better atomics

2013-11-06 Thread Ants Aasma
the patch looks like. Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref

Re: [HACKERS] better atomics

2013-11-06 Thread Ants Aasma
. If there are data races in the code, they need comments pointing this out and probably memory barriers too. If there are no data races, then the volatile declaration is useless and a pessimization. Currently it's more of a catch all here be dragons declaration for data structures. Regards, Ants

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2013-10-21 Thread Ants Aasma
order of magnitude from less than 1ms to more than 1000s. Or with 31 bins you can cover factor of 2 increments from 100us to over 27h. And the code is almost trivial, just take a log of the duration and calculate the bin number from that and increment the value in the corresponding bin. Regards, Ants

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2013-10-21 Thread Ants Aasma
row takes about 20us on my computer, I picked 100us as a reasonable limit below where the exact speed doesn't matter anymore. Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing

Re: [HACKERS] removing old ports and architectures

2013-10-18 Thread Ants Aasma
microprocessors shows that the extra hardware needed for fast and strong memory ordering guarantees more than pays for itself in performance. Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org

Re: [HACKERS] removing old ports and architectures

2013-10-18 Thread Ants Aasma
On Fri, Oct 18, 2013 at 8:04 PM, Peter Geoghegan p...@heroku.com wrote: On Fri, Oct 18, 2013 at 9:55 AM, Ants Aasma a...@cybertec.at wrote: FWIW, I think that if we approach coding lock free algorithms correctly - i.e. which memory barriers can we avoid while being safe, instead of which

Re: [HACKERS] [PERFORM] Cpu usage 100% on slave. s_lock problem.

2013-10-02 Thread Ants Aasma
On Wed, Oct 2, 2013 at 4:39 PM, Merlin Moncure mmonc...@gmail.com wrote: On Mon, Sep 30, 2013 at 7:51 PM, Ants Aasma a...@cybertec.at wrote: So we need a read barrier somewhere *after* reading the flag in RecoveryInProgress() and reading the shared memory structures, and in theory a full

Re: [HACKERS] [PERFORM] Cpu usage 100% on slave. s_lock problem.

2013-10-02 Thread Ants Aasma
On Wed, Oct 2, 2013 at 10:37 PM, Merlin Moncure mmonc...@gmail.com wrote: On Wed, Oct 2, 2013 at 9:45 AM, Ants Aasma a...@cybertec.at wrote: I haven't reviewed the code in as much detail to say if there is an actual race here, I tend to think there's probably not, but the specific pattern

Re: [HACKERS] Freezing without write I/O

2013-10-01 Thread Ants Aasma
would consider using a naming scheme that accounts for possible future uint64 atomics. Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org

Re: [HACKERS] [PERFORM] Cpu usage 100% on slave. s_lock problem.

2013-10-01 Thread Ants Aasma
disallow it. I wonder if we should just emit a full fence there for platforms with weak memory models. Trying to enforce the control dependency seems quite fragile. Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de

Re: [HACKERS] [PERFORM] Cpu usage 100% on slave. s_lock problem.

2013-09-30 Thread Ants Aasma
://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-vol-3a-part-1-manual.pdf Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Freezing without write I/O

2013-09-30 Thread Ants Aasma
for scads of volatile references all over the place. +1. The commits that you showed fixing issues in this area show quite well why this is a good idea. Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via

Re: [HACKERS] Freezing without write I/O

2013-09-19 Thread Ants Aasma
approach of sprinkling around volatile pointers. Regards, Ants Aasma [1] http://en.cppreference.com/w/c/atomic [2] (long video about atomics) http://channel9.msdn.com/Shows/Going+Deep/Cpp-and-Beyond-2012-Herb-Sutter-atomic-Weapons-1-of-2 -- Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700

Re: [HACKERS] Questions about checksum feature in 9.3

2013-09-16 Thread Ants Aasma
measurable effect. A single core can checksum several gigabytes per second of I/O without vectorization, and about 30GB/s with vectorization. Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql

Re: [HACKERS] Master-slave visibility order

2013-08-29 Thread Ants Aasma
clients be aware of how fresh they want their snapshot to be. If we add this capability then clients aware of this functionality could shift the visibility wait from commit to the start of next transaction that needs to see the changes. Regards, Ants Aasma -- Cybertec Schönig Schönig GmbH

  1   2   3   >