Re: [HACKERS] Pluggable storage

2016-08-18 Thread Ants Aasma
at about going even further than [1] in converting the executor to being opcode based and merging projection and qual evaluation to a single pass? Optimizer would then have some leeway about how to order column extraction and qual evaluation. Might even be worth it to special case some functions as

Re: [HACKERS] JIT compiling - v4.0

2017-10-04 Thread Ants Aasma
n tell these are not in mainline LLVM. Is there a branch or patchset of LLVM available somewhere that I need to use this? Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Discussion on missing optimizations

2017-10-09 Thread Ants Aasma
ccasionally used as a planner "hint" to correct for row count overestimates. Not a great solution, but PostgreSQL doesn't really have a better way to guide the planner. Those queries will now have to do something else, like col = col + 0, which still works. Regards, Ants Aasma --

Re: [HACKERS] make async slave to wait for lsn to be replayed

2017-10-26 Thread Ants Aasma
On Mon, Oct 23, 2017 at 12:29 PM, Ivan Kartyshov wrote: > Ants Aasma писал 2017-09-26 13:00: >> >> Exposing this interface as WAITLSN will encode that visibility order >> matches LSN order. This removes any chance of fixing for example >> visibility order of async

Re: [HACKERS] make async slave to wait for lsn to be replayed

2017-10-31 Thread Ants Aasma
would be just a ShareLock on the transactionid. On standby it would wait for the commit or rollback record for that transaction to be replayed. Robert made a good point that people will still rely on the token being an LSN, but perhaps they will be slightly less angry when we explicitly tel

Re: [HACKERS] emergency outage requiring database restart

2016-10-27 Thread Ants Aasma
ock numbers were used for writing out and reading in the page. Either the blocknum gets corrupted between calculating the checksum and writing the page out (unlikely given the proximity), or the pages are somehow getting transposed in the storage. Regards, Ants Aasma -- Sent via pgsql-hackers mai

Re: [HACKERS] Checksums by default?

2017-02-24 Thread Ants Aasma
On Fri, Feb 24, 2017 at 7:47 PM, Bruce Momjian wrote: > On Sat, Jan 21, 2017 at 09:02:25PM +0200, Ants Aasma wrote: >> > It might be worth looking into using the CRC CPU instruction to reduce this >> > overhead, like we do for the WAL checksums. Since that is a different &g

Re: [HACKERS] Checksums by default?

2017-02-24 Thread Ants Aasma
oth LLVM and GCC are capable of compiling the code that we have to a vectorized loop using SSE4.1 or AVX2 instructions given the proper compilation flags. This is exactly what was giving the speedup in the test I showed in my e-mail. Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (p

Re: [HACKERS] Checksums by default?

2017-02-24 Thread Ants Aasma
apping out the current algorithm. And I don't really see a reason to, it would introduce a load of headaches for no real gain. Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Checksums by default?

2017-02-24 Thread Ants Aasma
On Fri, Feb 24, 2017 at 9:49 PM, Jim Nasby wrote: > On 2/24/17 12:30 PM, Tomas Vondra wrote: >> >> In any case, we can't just build x86-64 packages with compile-time >> SSE4.1 checks. > > > Dumb question... since we're already discussing llvm for the executor, would > that potentially be an option

Re: [HACKERS] Checksums by default?

2017-02-24 Thread Ants Aasma
On Fri, Feb 24, 2017 at 10:30 PM, Bruce Momjian wrote: > On Fri, Feb 24, 2017 at 10:09:50PM +0200, Ants Aasma wrote: >> On Fri, Feb 24, 2017 at 9:37 PM, Bruce Momjian wrote: >> > Oh, that's why we will hopefully eventually change the page checksum >> > al

Re: [HACKERS] better atomics - v0.5

2014-07-01 Thread Ants Aasma
an implementation for Apache Derby. You may find some interesting ideas in there. [1] http://code.google.com/p/derby-nb/source/browse/trunk/derby-nb/ICDE10_conf_full_409.pdf Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.pos

Re: [HACKERS] Reporting the commit LSN at commit time

2014-08-07 Thread Ants Aasma
r for a while after commit. Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-08 Thread Ants Aasma
arisons, > especially of strings, ISTM, and that could still be available - this would > just be a bit of extra arithmetic. I don't think binary search is the main problem here. Objects are usually reasonably sized, while arrays are more likely to be huge. To make matters worse, js

Re: [HACKERS] WIP: Faster Expression Processing v4

2017-03-25 Thread Ants Aasma
terpreter is about 15% (5% speedup on a workload that spends 1/3 in ExecInterpExpr). My idea of prefetching op->resnull/resvalue to local vars before the indirect jump is somewhere between a tiny benefit and no effect, certainly not worth introducing extra complexity. Clang 3.8 does the correct thi

Re: [HACKERS] Reducing pg_ctl's reaction time

2017-06-28 Thread Ants Aasma
We use a regular fast shutdown, but that can take a long time due to the shutdown checkpoint. The leader lease may run out during this time so we would have to escalate to immediate shutdown or have a watchdog fence the node. If we knew that no user backends are left we can let the shutdown check

Re: [HACKERS] emergency outage requiring database restart

2017-08-10 Thread Ants Aasma
On Wed, Jan 18, 2017 at 4:33 PM, Merlin Moncure wrote: > On Wed, Jan 18, 2017 at 4:11 AM, Ants Aasma wrote: >> On Wed, Jan 4, 2017 at 5:36 PM, Merlin Moncure wrote: >>> Still getting checksum failures. Over the last 30 days, I see the >>> following. Since enabling

Re: [HACKERS] WIP: Data at rest encryption

2017-06-12 Thread Ants Aasma
On Mon, Jun 12, 2017 at 10:38 PM, Robert Haas wrote: > On Mon, Jun 13, 2016 at 11:07 AM, Peter Eisentraut > wrote: >> On 6/7/16 9:56 AM, Ants Aasma wrote: >>> >>> Similar things can be achieved with filesystem level encryption. >>> However this is not alw

Re: [HACKERS] WIP: Data at rest encryption

2017-06-14 Thread Ants Aasma
ould. I don't know how many places we've got assumptions > like this baked into the system, but I'm guessing there are a bunch. I think we need to require wal_log_hints=on when encryption is enabled. Currently I have not considered tearing on CLOG bits. Other SLRUs probably have sim

Re: [HACKERS] WIP: Data at rest encryption

2017-06-14 Thread Ants Aasma
. Me neither, but it currently is, and it looks like that's broken in a "silently corrupts your data" way in face of torn writes. Using OFB mode (xor plaintext with pseudorandom stream for cipher) looks like it might help here, if other approaches fail. Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Hooks to track changed pages for backup purposes

2017-09-13 Thread Ants Aasma
0 minutes - making the ratio of work from generating WAL to parsing it be about 750:1. Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] make async slave to wait for lsn to be replayed

2017-09-26 Thread Ants Aasma
ing it so the token is an opaque commit visibility token that just happens to be a LSN would still allow for progress in transaction management. For example, making PostgreSQL distributed will likely want timestamp and/or vector clock based visibility rules. Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Memory ordering issue in LWLockRelease, WakeupWaiters, WALInsertSlotRelease

2014-02-12 Thread Ants Aasma
er) provides that guarantee in this specific case. Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your su

[HACKERS] Master-slave visibility order

2013-08-28 Thread Ants Aasma
required. 4. Make the choice between 1 and 2 user configurable (it seems to me that it could even be changed without a restart). Thoughts? Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via

Re: [HACKERS] Master-slave visibility order

2013-08-29 Thread Ants Aasma
ommunication channel, neither can the slave. Timeouts won't help either as that would need clock synchronization between servers, similarly to Google's F1 system. Speaking of F1, they solve the same problem by having clients be aware of how fresh they want their snapshot to be. If we add this ca

Re: [HACKERS] Master-slave visibility order

2013-08-29 Thread Ants Aasma
t what is behind the CPU overhead? Maybe the solution is to make WAL insertion cheap enough to not matter. That won't be easy, but neither are the alternatives. Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subs

Re: [HACKERS] Master-slave visibility order

2013-08-29 Thread Ants Aasma
on to do something about it later. [1] http://www.postgresql.org/message-id/CA+CSw_tEpJ=md1zgxPkjH6CWDnTDft4gBi=+p9snoc+wy3p...@mail.gmail.com Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WAL CPU overhead/optimization (was Master-slave visibility order)

2013-08-29 Thread Ants Aasma
On Fri, Aug 30, 2013 at 1:30 AM, Andres Freund wrote: > On 2013-08-30 01:10:40 +0300, Ants Aasma wrote: >> On Fri, Aug 30, 2013 at 12:33 AM, Andres Freund >> wrote: >> > FWIW, WAL is still the major bottleneck for INSERT heavy workloads. The >> > per CPU over

Re: [HACKERS] WAL CPU overhead/optimization (was Master-slave visibility order)

2013-08-29 Thread Ants Aasma
nderstand the appeal of staying with what we have, but this would cap the speedup at 4x and has large caveats with the extra lookup tables. A 28x speedup might be worth the extra effort. Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Questions about checksum feature in 9.3

2013-09-16 Thread Ants Aasma
o imagine it having any measurable effect. A single core can checksum several gigabytes per second of I/O without vectorization, and about 30GB/s with vectorization. Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-su

Re: [HACKERS] Freezing without write I/O

2013-09-19 Thread Ants Aasma
art, at least it would make the intention more clear than the current approach of sprinkling around volatile pointers. Regards, Ants Aasma [1] http://en.cppreference.com/w/c/atomic [2] (long video about atomics) http://channel9.msdn.com/Shows/Going+Deep/Cpp-and-Beyond-2012-Herb-Sutter-atomic-W

Re: [HACKERS] [PERFORM] Cpu usage 100% on slave. s_lock problem.

2013-09-30 Thread Ants Aasma
s that would be insane. [1] http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-vol-3a-part-1-manual.pdf Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-suppo

Re: [HACKERS] Freezing without write I/O

2013-09-30 Thread Ants Aasma
ompiler fences. That would > eliminate the need for scads of volatile references all over the > place. +1. The commits that you showed fixing issues in this area show quite well why this is a good idea. Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wie

Re: [HACKERS] Freezing without write I/O

2013-10-01 Thread Ants Aasma
s good enough for me. I would consider using a naming scheme that accounts for possible future uint64 atomics. Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hacker

Re: [HACKERS] [PERFORM] Cpu usage 100% on slave. s_lock problem.

2013-10-01 Thread Ants Aasma
ds and even stores out from the conditional block losing the control dependency. :( It's quite unlikely to do so as it would be a very large code motion and it probably has no reason to do it, but I don't see anything that would disallow it. I wonder if we should just emit a full fence

Re: [HACKERS] [PERFORM] Cpu usage 100% on slave. s_lock problem.

2013-10-02 Thread Ants Aasma
On Wed, Oct 2, 2013 at 4:39 PM, Merlin Moncure wrote: > On Mon, Sep 30, 2013 at 7:51 PM, Ants Aasma wrote: >> So we need a read barrier somewhere *after* reading the flag in >> RecoveryInProgress() and reading the shared memory structures, and in >> theory a full barrier

Re: [HACKERS] [PERFORM] Cpu usage 100% on slave. s_lock problem.

2013-10-02 Thread Ants Aasma
On Wed, Oct 2, 2013 at 10:37 PM, Merlin Moncure wrote: > On Wed, Oct 2, 2013 at 9:45 AM, Ants Aasma wrote: >> I haven't reviewed the code in as much detail to say if there is an >> actual race here, I tend to think there's probably not, but the >> specific pattern

Re: [HACKERS] removing old ports and architectures

2013-10-18 Thread Ants Aasma
reate a new architecture with a similarly loose memory model. The experience with Alpha and other microprocessors shows that the extra hardware needed for fast and strong memory ordering guarantees more than pays for itself in performance. Regards, Ants Aasma -- Sent via pgsql-hackers mailing l

Re: [HACKERS] removing old ports and architectures

2013-10-18 Thread Ants Aasma
On Fri, Oct 18, 2013 at 8:04 PM, Peter Geoghegan wrote: > On Fri, Oct 18, 2013 at 9:55 AM, Ants Aasma wrote: >> FWIW, I think that if we approach coding lock free algorithms >> correctly - i.e. "which memory barriers can we avoid while being >> safe", instead of &q

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2013-10-21 Thread Ants Aasma
the fraction of queries running at each order of magnitude from less than 1ms to more than 1000s. Or with 31 bins you can cover factor of 2 increments from 100us to over 27h. And the code is almost trivial, just take a log of the duration and calculate the bin number from that and increment the val

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2013-10-21 Thread Ants Aasma
in RAM and/or stuff is on SSD's. Selecting a single row takes about 20us on my computer, I picked 100us as a reasonable limit below where the exact speed doesn't matter anymore. Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http:

Re: [HACKERS] better atomics

2013-11-06 Thread Ants Aasma
nchronized by a lock. I guess the best approach for deciding would be to try to convert a couple of the existing unlocked accesses to the API and see what the patch looks like. Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.post

Re: [HACKERS] better atomics

2013-11-06 Thread Ants Aasma
ssimization. Currently it's more of a catch all "here be dragons" declaration for data structures. Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal: "Causal reads" mode for load balancing reads without stale data

2015-11-11 Thread Ants Aasma
n the token, and then there are "dumb" clients that want to use write side waits. Also, it should be possible to configure which standbys are considered for waiting on. Otherwise a reporting slave will occasionally catch up enough to be considered "available" and then cause a latenc

Re: [HACKERS] Re: Add minor version to v3 protocol to allow changes without breaking backwards compatibility

2012-01-21 Thread Ants Aasma
e had minimal complaints because anyone who gets bitten > can easily set bytea_output='escape' and the problem goes away. I had a run in with this. JDBC driver versions < 9.0 with the default configuration resulted in silent data corruption. The fix was easy, but not having an us

Re: [HACKERS] CLOG contention, part 2

2012-02-10 Thread Ants Aasma
he disease. I think that those cases would be better handled by increasing the maximum CLOG SLRU size. The increase in memory usage should be a drop in the bucket for systems that have enough transaction processing velocity for that to be a problem. -- Ants Aasma

Re: [HACKERS] Patch: add timing of buffer I/O requests

2012-02-22 Thread Ants Aasma
the general design could be hashed out? Anyway, the user visible information from this patch should be trivial to extract from a general wait timing framework. Pushing my own agenda a bit - having this patch in the current release would help to get some field experience on any issues surroundin

Re: [HACKERS] Initial 9.2 pgbench write results

2012-02-27 Thread Ants Aasma
ould help end user latency. Is there a standard benchmark to measure that? -- Ants Aasma

Re: [HACKERS] performance results on IBM POWER7

2012-03-01 Thread Ants Aasma
lable_clocksource To switch the clocksource, just write the desired clocksource like this: # echo hpet > /sys/devices/system/clocksource/clocksource0/current_clocksource Thanks, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to you

Re: [HACKERS] pg_upgrade and statistics

2012-03-16 Thread Ants Aasma
re, optimal histogram construction is NP-hard in the multi-dimensional case and so people will want to try different algorithms, or make different tradeoffs on effort spent on constructing the histogram. Or even build one by hand. Cheers, Ants Aasma -- Sent via pgsql-hackers mailing lis

[HACKERS] pg_upgrade incorrectly equates pg_default and database tablespace

2012-03-25 Thread Ants Aasma
ates empty spclocation with database tablespace: tblspace = PQgetvalue(res, relnum, i_spclocation); /* if no table tablespace, use the database tablespace */ if (strlen(tblspace) == 0) tblspace = dbinfo->db_tblspace; Patch to fix this is attached. Regards, Ants Aasma -- Cybertec Schönig &

Re: [HACKERS] Patch: add timing of buffer I/O requests

2012-03-27 Thread Ants Aasma
med by others should be the minority anyway. I won't really miss the per table stats. But if the pg_stat_statements normalisation patch gets commited, it would be really neat to also have IO waits there. It would be much easier to quickly diagnose "what is causing all this IO" issues

[HACKERS] [PATCH] Lazy hashaggregate when no aggregation is needed

2012-03-27 Thread Ants Aasma
orted aggregation. Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de diff --git a/src/backend/executor/nodeAgg.c b/src/backend/executor/nodeAgg.c index c9aa921..53cdd09 100644 --- a/src/backend/executor/nodeAgg.c +++ b/src/backend

Re: [HACKERS] pg_rewind, a tool for resynchronizing an old master after failover

2013-05-25 Thread Ants Aasma
erformed certain kind of > operations on old cluster after new cluster forked off? Truncate and all kinds of DROP come to mind, also table rewrites from ALTER. The tool should probably just flag those relation files to be copied wholesale. Regards, Ants Aasma -- Cybertec Schönig & Schöni

Re: [HACKERS] XLogInsert scaling, revisited

2013-05-29 Thread Ants Aasma
serves a comment why it's not needed so future refactorings don't create a data race. Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postg

Re: [HACKERS] Vacuum, Freeze and Analyze: the big picture

2013-06-03 Thread Ants Aasma
or or against, but it's possible that OS write-out routines defeat the careful cost based throttling that PostgreSQL does by periodically dumping a large portion of dirty pages into the write queue at once. That does nasty things to query latencies as evidenced by the work on checkpoint spre

[HACKERS] Proposal for CSN based snapshots

2013-06-06 Thread Ants Aasma
not be as well off, but I doubt there will be a regression. At this point I don't see any major issues with this approach. If the ensuing discussion doesn't find any major showstoppers then I will start to work on a patch bit-by-bit. It might take a while though as my free hacking time ha

Re: [HACKERS] Proposal for CSN based snapshots

2013-06-07 Thread Ants Aasma
ee so they don't cause any contention. The number of times each cache line can be invalidated is bounded by 8. Overall I think actual performance tests are needed to see if there is a problem, or perhaps if having the data shared actually helps with cache hit rates. >> At this point I

Re: [HACKERS] Proposal for CSN based snapshots

2013-06-07 Thread Ants Aasma
On Fri, Jun 7, 2013 at 3:47 PM, Greg Stark wrote: > On Thu, Jun 6, 2013 at 11:42 PM, Ants Aasma wrote: >> To refresh your memory the basic idea is to change visibility >> determination to be based on a commit sequence number (CSN for short) >> - a 8 byte number increm

Re: [HACKERS] Parallell Optimizer

2013-06-12 Thread Ants Aasma
l to solve with Commit Sequence Number based snapshot scheme that I proposed. Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Parallell Optimizer

2013-06-12 Thread Ants Aasma
ssue prohibiting it is the fact that visibility order of commits on the master is determined by the order that commiters acquire ProcArrayLock, and that can be different from the order of WALInsertLock that determines the ordering of LSNs, whereas visibility on the slave instance is determined purely

Re: [HACKERS] Parallell Optimizer

2013-06-12 Thread Ants Aasma
On Jun 13, 2013 4:18 AM, "Stephen Frost" wrote: > * Ants Aasma (a...@cybertec.at) wrote: > > In a cluster setting you take the CSN value on the master, then before > > starting execution on a standby you wait until that the standby has > > replayed enough WAL to re

Re: [HACKERS] Parallell Optimizer

2013-06-13 Thread Ants Aasma
th with the same solution. [1] http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en//archive/spanner-osdi2012.pdf Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Parallell Optimizer

2013-06-13 Thread Ants Aasma
er is easier than synchronizing lists of running transactions. Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subs

Re: [HACKERS] Implementing incremental backup

2013-06-19 Thread Ants Aasma
d be to apply the incremental backup to the previous backup while copying out old blocks so you could have the latest full backup available and incremental changes to rewind it to the previous version. Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt W

Re: [HACKERS] Bloom Filter lookup for hash joins

2013-06-26 Thread Ants Aasma
ns when the planner has a bad estimate. Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Hash partitioning.

2013-06-27 Thread Ants Aasma
you can create a predicate index on (key, tstamp) where tstamp > [some date in recent past], and replace the index with a newer one every so often to keep the size small. This way you can have a non-partitioned index for batch queries and a small one for the OLTP workload. If we added the option to build indexes using an index only scan, building the replacement index would be quite cheap. Regards, Ants Aasma

Re: [HACKERS] XLogInsert scaling, revisited

2013-07-08 Thread Ants Aasma
blocks. A read barrier is enough here unless there is some other undocumented interaction. I don't think it matters for performance, but it seems like good practice to have the barriers exactly matching the documentation. Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 2

Re: [HACKERS] XLogInsert scaling, revisited

2013-07-08 Thread Ants Aasma
disjoint memory locations. This way all memory locations in xlog buffer page are stored exactly once and there is no data race between writes making it possible to omit the barrier from GetXLogBuffer. WaitXLogInsertionsToFinish() takes care of the memory barrier for XlogWrite(). Regards, Ants A

Re: [HACKERS] Improvement of checkpoint IO scheduler for stable transaction responses

2013-07-16 Thread Ants Aasma
;m not yet sure if it's repeatable or just noise. Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Improvement of checkpoint IO scheduler for stable transaction responses

2013-07-16 Thread Ants Aasma
On Tue, Jul 16, 2013 at 9:17 PM, Greg Smith wrote: > On 7/16/13 12:46 PM, Ants Aasma wrote: > >> Spread checkpoints sprinkles the writes out over a long >> period and the general tuning advice is to heavily bound the amount of >> memory the OS willing to keep dirty. >

Re: [HACKERS] Improvement of checkpoint IO scheduler for stable transaction responses

2013-07-17 Thread Ants Aasma
On Wed, Jul 17, 2013 at 1:54 PM, Greg Smith wrote: > On 7/16/13 11:36 PM, Ants Aasma wrote: >> >> As you know running a full suite of write benchmarks takes a very long >> time, with results often being inconclusive (noise is greater than >> effect we are trying to m

Re: [HACKERS] Improvement of checkpoint IO scheduler for stable transaction responses

2013-07-17 Thread Ants Aasma
the patent also looks like it would be easy to bypass by doing the equivalent thing in a slightly different way. Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mail

Re: [HACKERS] InvokeObjectPostAlterHook() vs. CommandCounterIncrement()

2013-07-21 Thread Ants Aasma
about taking a snapshot before altering and passing this to the hook. Would there be other issues besides performance? If the snapshot is taken only when there is a hook present then the performance can be fixed later. Regards, Ants Aasma

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-09-04 Thread Ants Aasma
enough to just interleave writes of each tablespace, weighed by the amount of writes per tablespace. Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-09-04 Thread Ants Aasma
sort with the full sized array and then compress it to a list of buffer IDs that need to be written out. This way most of the time you only need a small array and the large array is only needed for a fraction of a second. Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmü

Re: [HACKERS] Scaling shared buffer eviction

2014-09-12 Thread Ants Aasma
usive state introducing coherency traffic. Not locking the buffer only saves transfering the cacheline back to the pinning backend, not a huge amount of savings. Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-suppor

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-09-12 Thread Ants Aasma
CRC calculations for testing this patch to see if the performance improvement is due to less data being checksummed. Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-09-12 Thread Ants Aasma
shing lookup tables. If we choose to stay with CRC we must accept that we can only solve the performance issues for Intel CPUs and provide slight alleviation for others. Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgres

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-09-24 Thread Ants Aasma
On Tue, Sep 23, 2014 at 8:15 PM, Florian Weimer wrote: > * Ants Aasma: > >> CRC has exactly one hardware implementation in general purpose CPU's > > I'm pretty sure that's not true. Many general purpose CPUs have CRC > circuity, and there must be some which al

Re: [HACKERS] Scaling shared buffer eviction

2014-09-26 Thread Ants Aasma
blocknum first and possibly have better branch prediction. Do you have a workload where I could test if this helps alleviate the comparison overhead? Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de diff --g

Re: [HACKERS] "Value locking" Wiki page

2014-10-01 Thread Ants Aasma
predicate (e.g. soft delete). AFAICS this is currently not possible to implement correctly without a retry loop. The hypothetical ON CONFLICT REPLACE and ON CONFLICT UPDATE-AND-THEN-INSERT modes would also make sense in the unique index case. Not saying that I view this as necessary for the first c

Re: [HACKERS] WIP: dynahash replacement for buffer table

2014-10-15 Thread Ants Aasma
sing asymmetrically sized tables. [1] https://www.cs.princeton.edu/~mfreed/docs/cuckoo-eurosys14.pdf Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP: dynahash replacement for buffer table

2014-10-16 Thread Ants Aasma
On Oct 15, 2014 7:32 PM, "Ants Aasma" wrote: > I'm imagining a bucketized cuckoo hash with 5 item buckets (5-way > associativity). This allows us to fit the bucket onto 2 regular sized > cache lines and have 8 bytes left over. Buckets would be protected by > seqlocks st

Re: [HACKERS] Failback to old master

2014-11-12 Thread Ants Aasma
re sequential integers, not GUID's, but at least it's significantly harder. Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql

Re: [HACKERS] Failback to old master

2014-11-13 Thread Ants Aasma
nd of xlog on old master at x2, x1 < x2, master will request streaming at tli1.x2, wal sender does tliSwitchPoint(tli1) to lookup x1, finds that x1 < x2 and gives the error "requested starting point %X/%X on timeline %u is not in this server's history". The alignmen

Re: [HACKERS] What exactly is our CRC algorithm?

2014-11-21 Thread Ants Aasma
On Fri, Nov 21, 2014 at 12:11 PM, Abhijit Menon-Sen wrote: > If anyone has other suggestions, I'm all ears. Do you have a WIP patch I could take a look at and tweak? Maybe there's something about the compilers code generation that could be improved. Regards, Ants Aasma -- Cyb

[HACKERS] Using RTLD_DEEPBIND to handle symbol conflicts in loaded libraries

2014-11-26 Thread Ants Aasma
ldn't be a good idea? Are there any similar options for other platforms? Alternatively, does anyone know of linker flags that would give a similar effect? Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.

Re: [HACKERS] parallelism and sorting

2015-11-24 Thread Ants Aasma
ith a newer timestamp constant, and the rebuild would be a lot faster if it could use the existing index to perform an index only scan of 10% of data instead of scanning and sorting the full table. Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: htt

Re: [HACKERS] Re: Revisiting pg_stat_statements and IN() (Was: Re: pg_stat_statements fingerprinting logic and ArrayExpr)

2015-11-24 Thread Ants Aasma
eSQL that would be capable of distinguishing interesting variations from irrelevant doesn't seem like a feasible plan. In my view the best we could do is to aim to have entries roughly correspond to application query invocation points and leave the more complex statistical analysis use cases t

Re: [HACKERS] W-TinyLfu for cache eviction

2015-12-09 Thread Ants Aasma
k a global lock would be good enough for a proof of concept that only evaluates cache hit ratios. Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Using quicksort for every external sort run

2015-12-12 Thread Ants Aasma
and use that to drive a simple loop. The code size would be pretty similar to insertion sort and the loop overhead should mostly be hidden by the CPU OoO machinery. Probably won't help much, but would be interesting and simple enough to try out. Can you share you code for the benchmark so I ca

Re: [HACKERS] 9.2.3 crashes during archive recovery

2013-02-14 Thread Ants Aasma
ommunicate the necessary WAL location. -- Ants Aasma

Re: [HACKERS] Alias hstore's ? to ~ so that it works with JDBC

2013-02-14 Thread Ants Aasma
ng applications to bump their minimum db version. With operator("?") older JDBC clients can work too in case the library version is fixed due to policies (I'm assuming here that question marks already work within quoted identifiers/literals). -- Ants Aasma

Re: [HACKERS] 9.2.3 crashes during archive recovery

2013-02-15 Thread Ants Aasma
hat section has confused at least one of my clients. That chapter is rather scarily complicated compared to what's usually necessary. Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers ma

Re: [HACKERS] 9.2.3 crashes during archive recovery

2013-02-16 Thread Ants Aasma
ear out pg_xlog seems like > an unnecessary complication. It's safe to do so, if you restore with an > archive, but unnecessary. > > The "File System Level Backup" chapter > (http://www.postgresql.org/docs/devel/static/backup-file.html) probably > should mention &qu

Re: [HACKERS] 9.2.3 crashes during archive recovery

2013-02-19 Thread Ants Aasma
> end-of-backup record. Thank you for explaining this, I can see how it works now. I'll see if I can document this better so others won't have to go through as much effort. Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgr

Re: [HACKERS] Materialized views WIP patch

2013-03-01 Thread Ants Aasma
inaccessible until the next refresh. Sorry for weighing in late, but it seemed to me that this point didn't get enough consideration. Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql

Re: [HACKERS] Materialized views WIP patch

2013-03-01 Thread Ants Aasma
TRUNCATE semantics for slight implementation and notation convenience. To give another example of potential future update semantics, if we were to allow users manually maintaining materialized view contents using DML commands, one would expect TRUNCATE to mean "make this matview empty", not

Re: [HACKERS] Using indexes for partial index builds

2013-03-12 Thread Ants Aasma
his is an increasingly widespread pattern with a proliferation of mobile devices that need syncing. Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgres

  1   2   3   >