[HACKERS] B-tree index row size limit

2016-10-09 Thread Florian Weimer
The index row size limit reared its ugly head again. My current use of PostgreSQL is to load structured data into it but from sources I don't have control over, to support a wide range of queries whose precise nature is not yet known to me. (Is this called a data warehouse?) Anyway, what happens

Re: [HACKERS] atexit_callback can be a net negative

2014-03-07 Thread Florian Weimer
ow we should not joggle. Can anyone see a flaw in that? There's the PID reuse problem. Forking twice (with a delay) could end up with the same PID as MyProcPid. Comparing the process start time would protect against that. Checking getppid() would have the same theoretical problem.

Re: [HACKERS] atexit_callback can be a net negative

2014-03-07 Thread Florian Weimer
down further. On Linux, linking against pthread_atfork currently requires linking against pthread (although this is about to change), and it might incur the pthread-induced overhead on some configurations. -- Florian Weimer / Red Hat Product Security Team -- Sent via pgsql-hackers mailing lis

Re: [HACKERS] atexit_callback can be a net negative

2014-03-07 Thread Florian Weimer
On 03/07/2014 04:10 PM, Tom Lane wrote: Florian Weimer writes: On 03/07/2014 03:57 PM, Tom Lane wrote: It's not a reason not to do something about the much larger chance of this happening in a direct child process, which certainly won't have a matching PID. Indeed. Checking ge

Re: [HACKERS] get rid of SQL_ASCII?

2013-09-06 Thread Florian Weimer
ly storing, retrieving, and working with and how they're doing it. Practically speaking, the escaping gets in the way, and there isn't full feature parity with TEXT. Regular expression matching seems to be missing, for instance. But apart from that, yes, BYTEA would be the more appropriate

Re: [HACKERS] Why the asprintf patch is still breaking the buildfarm

2013-10-22 Thread Florian Weimer
e (because it is claimed that C99 and POSIX conflict, and glibc implements neither behavior). -- Florian Weimer / Red Hat Product Security Team -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Reasons not to like asprintf

2013-10-23 Thread Florian Weimer
rrno == EOVERFLOW, as required by POSIX. It's probably better to move it after the error logging. -- Florian Weimer / Red Hat Product Security Team -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] RFC: Async query processing

2013-11-03 Thread Florian Weimer
t the end of the transaction. With a result-buffering implementation of PQASYNC_RESULT, we could do that as well, but this might be too much magic. I thought I'd ask for comments before starting coding because this looks a bit more complicated than I expected. :) -- Florian Weimer / Red H

Re: [HACKERS] Re: Add minor version to v3 protocol to allow changes without breaking backwards compatibility

2012-01-23 Thread Florian Weimer
t the change was coming and could push updated Java and Perl client libraries well before the server-side change hit our internal repository, but I really don't want to have to pay attention to such details. -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/

Re: [HACKERS] Re: Add minor version to v3 protocol to allow changes without breaking backwards compatibility

2012-01-23 Thread Florian Weimer
EA values received in text format, which led to the compatibility issue.) If you've version negotiation and you don't expose the binary format, then all clients can use the most efficient format automatically. If I understand things correctly, this is where the JDBC driver is heading

Re: [HACKERS] Page Checksums

2012-01-23 Thread Florian Weimer
ir of course, but it's > interesting how small the camp of those using checksummed filesystems > is. Don't checksumming file systems currently come bundled with other features you might not want (such as certain vendors)? -- Florian Weimer BFK edv-consulting GmbH

Re: [HACKERS] Page Checksums

2012-01-24 Thread Florian Weimer
eally quite limited. > And yes, I would for sure turn such functionality on if it were present. Same here. I already use page-level checksum with Berkeley DB. -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-7

Re: [HACKERS] initdb and fsync

2012-02-04 Thread Florian Weimer
* Tom Lane: > I wonder whether it wouldn't be sufficient to call sync(2) at the end, > anyway, rather than cluttering the entire initdb codebase with fsync > calls. We tried to do this in the Debian package mananger. It works as expected on Linux systems, but it can cause a lot of data to hit th

Re: [HACKERS] WIP: URI connection string support for libpq

2012-02-24 Thread Florian Weimer
which sort of defeats the purpose of URIs). -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org

Re: [HACKERS] WIP: URI connection string support for libpq

2012-02-24 Thread Florian Weimer
-d? It's not nice URI syntax, but it's better than an out-of-band mechanism. -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 -- Sent via pgsq

Re: [HACKERS] [PATCH] Documentation: remove confusing paragraph about backslash escaping

2012-02-27 Thread Florian Weimer
rameters, but this seems a pretty fine distinction. -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgres

Re: [HACKERS] Remove fsync ON/OFF as a visible option?

2015-03-22 Thread Florian Weimer
* David G. Johnston: > "​enables or disables data durability ​promise of ACID." ? “fsync = on” only works if the storage stack doesn't do funny things. Depending on the system, it might not be sufficient. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to

Re: [HACKERS] Remove fsync ON/OFF as a visible option?

2015-03-22 Thread Florian Weimer
* David G. Johnston: > On Sunday, March 22, 2015, Florian Weimer wrote: > >> * David G. Johnston: >> >> > "​enables or disables data durability ​promise of ACID." ? >> >> “fsync = on” only works if the storage stack doesn't do funny things.

Re: [HACKERS] SSL renegotiation

2015-03-23 Thread Florian Weimer
as a substitute for renegotiation? Theoretical considerations, mostly. If rekeying is strictly required after processing just a few petabytes, the cipher is severely broken and should no longer be used. -- Florian Weimer / Red Hat Product Security -- Sent via pgsql-hackers mailing list (pgsql

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-09-23 Thread Florian Weimer
* Ants Aasma: > CRC has exactly one hardware implementation in general purpose CPU's I'm pretty sure that's not true. Many general purpose CPUs have CRC circuity, and there must be some which also expose them as instructions. > and Intel has a patent on the techniques they used to implement > i

Re: [HACKERS] [PATCH] Custom code int(32|64) => text conversions out of performance reasons

2010-12-03 Thread Florian Weimer
division, too. It might also be helpful to break down the dependency chain for large input values. The int8 version should probably work in 1e9 chunks and use a zero-padding variant of the 32-bit code. -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstr

Re: [HACKERS] crash-safe visibility map, take three

2010-12-03 Thread Florian Weimer
t least in principle, I'm not sure how much that would interfere with the executor architecture.) -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 -- S

Re: [HACKERS] Idle git question: how come so many "objects"?

2010-12-03 Thread Florian Weimer
s come from? 89 is the number of objects which need to be transmitted. Of those, 80 were compressed by diffing them to some other object (which might, in turn, be a diff). -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-72

Re: [HACKERS] WIP: Range Types

2011-01-04 Thread Florian Weimer
they behave sanely. So > I was thinking that the user might need to specify a function that > converts the subtype into a float that approximates a value's position > in the total order. Doesn't the eqsel hint already provide this information? -- Florian Weimer BFK

Re: [HACKERS] WIP: Range Types

2011-01-07 Thread Florian Weimer
* Jeff Davis: > On Tue, 2011-01-04 at 14:18 +0000, Florian Weimer wrote: >> * Jeff Davis: >> >> > 4. For the GiST penalty function, and perhaps some picksplit algorithms, >> > it might be nice to know the length of a range, or do some other kinds >> >

Re: [HACKERS] posix_fadvise missing in the walsender

2013-03-01 Thread Florian Weimer
* Jeff Janes: > Does the kernel really read a data block from disk into memory in > order to immediately overwrite it? I would have thought it would > optimize that away, at least if the writes are sized and aligned to > 512 or 1024 bytes blocks (which WAL should be). With Linux, you'd have to u

Re: [HACKERS] [GENERAL] Floating point error

2013-03-06 Thread Florian Weimer
EE floats, you may or may not get equivalent results --- but it's pretty hard to make any guarantees at all in such a case. There's also gdtoa, which returns the shortest decimal representation which rounds to the same decimal number. It would print 0.1 as 0.1, but 0.1 + 0.2 as 0.300

Re: [HACKERS] Enabling Checksums

2013-04-18 Thread Florian Weimer
* Greg Smith: > The TCP/IP checksum spec is at https://tools.ietf.org/html/rfc793 ; > its error detection limitations are described at > http://www.noahdavids.org/self_published/CRC_and_checksum.html ; and a > good article about optimizing its code is at > http://www.locklessinc.com/articles/tcp_c

Re: [HACKERS] Covering Indexes

2012-08-25 Thread Florian Weimer
* Jeff Janes: > I don't see the virtue of this in this case. Since the index is not > unique, why not just put the index on (a,b,c,d) and be done with it? AFAICT, SQLite 4 encodes keys in a way that is not easily reversed (although the encoding is injective, so it's reversible in principle). The

Re: [HACKERS] Synchronous commit not... synchronous?

2012-11-03 Thread Florian Weimer
* Daniel Farina: > The idea of canceling a COMMIT statement causing a COMMIT seems pretty > strange to me. Canceling commits is inherently racy, so I'm not sure if this behavior so strange after all. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] Confusion over Python drivers

2010-02-08 Thread Florian Weimer
* Jeff Davis: > I have written up a set of guidelines for driver development based on > what I learned working on ruby-pg: > > http://wiki.postgresql.org/wiki/Driver_development Interesting, thanks. I'm contemplating to create a new language binding for libpq (or, to be more precise, turn an exi

Re: [HACKERS] Confusion over Python drivers

2010-02-08 Thread Florian Weimer
* Andrew McNamara: >>Any other suggestions before I turn the above into a roadmap page on the >>wiki? > > I got sick of the constant stream of escaping bugs impacting on psycopg > and pyPgSQL, and wrote my own DB-API driver, using the more modern > libpq/binary/protocol 3 APIs where ever possible

Re: [HACKERS] Confusion over Python drivers

2010-02-14 Thread Florian Weimer
* Jeff Davis: > Agreed. Ultimately, the conversion has to be done somewhere, but I don't > believe the driver is the place for it. Type conversions are always > going to be imperfect, and this has some important consequences: > * The type conversion system will be endlessly tweaked to improve it

Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync

2010-02-14 Thread Florian Weimer
* Tom Lane: >> Which options would that be? I am not aware that there any for any of the >> recent linux filesystems. > > Shouldn't journaling of metadata be sufficient? You also need to enforce ordering between the directory update and the file update. The file metadata is flushed with fsync()

Re: [HACKERS] .gitignore files, take two

2010-10-11 Thread Florian Weimer
* James Cloos: > I'm reading this a bit late, but... > > We (Xorg) found that ignoring: > >*~ >*.bak >*.patch > > in addition to the files generated by building is very helpful. I tend to put those into .git/info/exclude. They are somewhat developer-s

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-10-20 Thread Florian Weimer
* Terry Laurenzo: > Agreed. BSON was born out of implementations that either lacked > arbitrary precision numbers or had a strong affinity to an > int/floating point way of thinking about numbers. I believe that if > BSON had an arbitrary precision number type, it would be a proper > superset of

Re: Latches with weak memory ordering (Re: [HACKERS] max_wal_senders must die)

2010-11-19 Thread Florian Weimer
ing is that once you've seen the result of an atomic operation on i386 and amd64, you are guaranteed to observe all prior writes performed by the thread which did the atomic operation, too. Explicit fencing is only necessary if you need synchronization without atomic operations. -- Florian We

Re: [HACKERS] POSIX shared memory redux

2011-04-14 Thread Florian Weimer
d you have to start postmasters on separate nodes (which you normally wouldn't do), doesn't this make it increasingly unlikely that it's going to be triggered in the wild? -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100

Re: [HACKERS] POSIX question

2011-06-20 Thread Florian Weimer
for that... mmap with PROT_NONE and subsequent update with mprotect does this on Linux. (It's not clear to me what this is trying to solve, though.) -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-7613

Re: [HACKERS] Range Types - efficiency

2011-02-10 Thread Florian Weimer
) > Total runtime: 111.780 ms > (3 rows) > > This, alas, reverts to a seq scan on the table, rather than restricting > itself to the tuples of interest. This is quite similar to LIKE and regexp matches. The backend has a kludge to use the index in such cases. It did not seem

Re: [HACKERS] Debian readline/libedit breakage

2011-02-14 Thread Florian Weimer
ed copies. We wouldn't face this issue with LGPL code. -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 -- Sent via pgsql-hackers mailing list (pgsql-hack

Re: [HACKERS] Logging WAL when updating hintbit

2013-11-14 Thread Florian Weimer
On 11/14/2013 07:02 AM, Sawada Masahiko wrote: I attached patch adds new wal_level 'all'. Shouldn't this be a separate setting? It's useful for storage which requires rewriting a partially written sector before it can be read again. -- Florian Weimer / Red Hat P

Re: [HACKERS] Can we trust fsync?

2013-11-22 Thread Florian Weimer
ty work. With ext4 and XFS on plain/LVM/md block devices, this issue should really be a thing of the past. I think the kernel folks would treat this as bugs nowadays, too. -- Florian Weimer / Red Hat Product Security Team -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)

Re: [HACKERS] RFC: Async query processing

2013-12-18 Thread Florian Weimer
On 11/04/2013 02:51 AM, Claudio Freire wrote: On Sun, Nov 3, 2013 at 3:58 PM, Florian Weimer wrote: I would like to add truly asynchronous query processing to libpq, enabling command pipelining. The idea is to to allow applications to auto-tune to the bandwidth-delay product and reduce the

Re: [HACKERS] RFC: Async query processing

2014-01-03 Thread Florian Weimer
ady streams of queries, it will work. But not for short bursts, which will be the most heavily used case I believe (most apps create short bursts of inserts and not continuous streams at full bandwidth). Loading data into the database isn't such an uncommon task. Not everything is OLTP.

Re: [HACKERS] RFC: Async query processing

2014-01-03 Thread Florian Weimer
fully reusing them when possible might avoid that. Again, due to the variance in lengths of runs, the staging tables are not always beneficial. I understand that pipelining introduces complexity. But solving the issues described above is no picnic, either. -- Florian Weimer / Red Hat Pr

Re: [HACKERS] RFC: Async query processing

2014-01-05 Thread Florian Weimer
uests. Purely from a perspective of latency avoidance, sub-queries, WITH or stored procedures can achieve the same thing, and work even if the intermediate result has to undergo some transformation. :-) -- Florian Weimer / Red Hat Product Security Team -- Sent via pgsql-hackers mailing list (

Re: [HACKERS] RFC: Async query processing

2014-01-05 Thread Florian Weimer
avoid any server-side changes, so that applications do not need fallback code for talking to old servers. -- Florian Weimer / Red Hat Product Security Team -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref

Re: [HACKERS] gettimeofday is at the end of its usefulness?

2017-01-13 Thread Florian Weimer
* Tom Lane: > Florian Weimer writes: >> * Tom Lane: >>> On Linux (RHEL6, 2.4GHz x86_64), I find that gettimeofday(), >>> clock_gettime(CLOCK_MONOTONIC), and clock_gettime(CLOCK_REALTIME) >>> all take about 40ns. Of course gettimeofday() only has 1us re

Re: [HACKERS] gettimeofday is at the end of its usefulness?

2016-12-29 Thread Florian Weimer
* Tom Lane: > On Linux (RHEL6, 2.4GHz x86_64), I find that gettimeofday(), > clock_gettime(CLOCK_MONOTONIC), and clock_gettime(CLOCK_REALTIME) > all take about 40ns. Of course gettimeofday() only has 1us resolution, > but the other two have perhaps 10ns resolution (I get no duplicate > readings i

Re: [HACKERS] SSL renegotiation

2015-02-23 Thread Florian Weimer
e rekeying, that mechanism will be provided at the TLS layer in the future. I think you could remove renegotiation from PostgreSQL as long as you offer something better than RC4 in the TLS handshake. -- Florian Weimer / Red Hat Product Security -- Sent via pgsql-hackers mailing list (pgsql-hac

Re: [HACKERS] [BUGS] Invalid YAML output from EXPLAIN

2010-06-07 Thread Florian Weimer
been claimed before that YAML is a superset of JSON, so why can't the YAML folks use the existing JSON output instead? -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax

Re: [HACKERS] [BUGS] Invalid YAML output from EXPLAIN

2010-06-08 Thread Florian Weimer
* Greg Smith: > Florian Weimer wrote: >> It has been claimed before that YAML is a superset of JSON, so why >> can't the YAML folks use the existing JSON output instead? >> > > Because JSON just crosses the line where it feels like there's so much

Re: [HACKERS] antisocial things you can do in git (but not CVS)

2010-07-23 Thread Florian Weimer
ry of all of our > branches is linear. But it seems to me that someone could > accidentally push a merge commit, either because they forgot to squash > locally, or because of a conflict between their local git repo's > master branch and origin/master. Can we forbid thi

Re: [HACKERS] UTF16 surrogate pairs in UTF8 encoding

2010-08-22 Thread Florian Weimer
nverted to regular UTF-8, I really question the sense of this. Usually, people want CESU-8 to preserve ordering between languages such as C# and Java and their database, and conversion destroys this property. -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de

Re: [HACKERS] reducing the overhead of frequent table locks, v4

2011-07-11 Thread Florian Weimer
mplemented this as "brlock" in Linux 2.4. The earliest citation seems to be W.C. Hsiesh, W. E. Weihl, "Scalable reader-writer locks for parallel systems.", MIT-LCS-TR-521, published in 1991. -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kr

Re: [HACKERS] libedit memory stomp is apparently fixed in OS X Lion

2011-08-09 Thread Florian Weimer
, might well fall under the system library exception on MacOS X. Curiously, the GPL favors proprietary operating systems in this area. -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe

Re: [HACKERS] synchronized snapshots

2011-08-15 Thread Florian Weimer
* Simon Riggs: > I don't see the need to change the BEGIN command, which is SQL > Standard. We don't normally do that. Some language bindings treat BEGIN specially, so it might be difficult to use this feature. -- Florian Weimer BFK edv-consulting GmbH h

Re: [HACKERS] How to define global variable in postgresql

2011-08-19 Thread Florian Weimer
> Hello. How can we define a global variable in postgresql? Do you mean session-private, but persistent across transactions? Configuration parameters can be abused for this purpose. -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße

Re: [HACKERS] strict aliasing

2011-11-15 Thread Florian Weimer
se). Correct, if code is not strict-aliasing-safe and you compile with -f-strict-aliasing, GCC may silently produce wrong code. (Same with -fwrapv, by the way.) -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-962

Re: [HACKERS] Notes on implementing URI syntax for libpq

2011-11-23 Thread Florian Weimer
X domain socket? Would it be possible to add something like psql -d postgresql+ssh://fweimer@db5/var/run/postgresql/.s.PGSQL.5432 similar to what Subversion supports? (This might have security implications when used from untrusted PHP scripts.) -- Florian Weimer BFK edv-

[HACKERS] Wire protocol: type-specific opt-in to binary format

2011-11-24 Thread Florian Weimer
wire protocol, it could be a regular run-time parameter. Do you think this would make sense? -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 -- Sent via pgsql-ha

[HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Florian Weimer
parameters). I haven't even looked how difficult it would be to implement this. Do you think it's worth the trouble? -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fa

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Florian Weimer
* Pavel Stehule: > Hello > > 2011/11/24 Florian Weimer : >> Occasionally, we get bitten by embedded NUL bytes in TEXT values.  We >> take care of generating proper UTF-8, but this additional restriction >> sometimes slips by.  It would be really helpful if PostgreS

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Florian Weimer
e NUL-transparent. By the way, I refuse the notion that UTF-8 strings with embedded NULs are "broken". I can't recall any other system which enforces UTF-8 well-formedness, but does not permit embedded NULs. -- Florian Weimer BFK edv-consulting GmbH http://www.bf

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Florian Weimer
n for embedded NULs inside > strings? The source system does not enforce that constraint, so from time to time, such data slips through. I don't know why it's there in the first place, and I have no control over the original data source. Usually, it's okay to silently strip N

Re: [HACKERS] Wire protocol: type-specific opt-in to binary format

2011-11-24 Thread Florian Weimer
* Florian Pflug: > On Nov24, 2011, at 10:03 , Florian Weimer wrote: >> I would like to add functionality which allows a client to tell the >> server which types can be sent in binary format. The immediate goal is >> to suppress hex quoting for BYTEA values, but it seems to

Re: [HACKERS] Notes on implementing URI syntax for libpq

2011-11-24 Thread Florian Weimer
ternally for management functions). -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)

Re: [HACKERS] Wire protocol: type-specific opt-in to binary format

2011-11-24 Thread Florian Weimer
_output GUC? libpq doesn't hide that at all, but the JDBC driver does---similar to the text/binary distinction. -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-

Re: [HACKERS] Notes on implementing URI syntax for libpq

2011-11-24 Thread Florian Weimer
ntax that > libpq ends up adopting. I just meant "something that will not work in > JDBC *right now*" (i.e. with no local socket support). Ah, okay, your proposal looked like something which couldn't work with JDBC *at all* because of invalid URI syntax (but admitted

[HACKERS] Large number of open(2) calls with bulk INSERT into empty table

2011-11-27 Thread Florian Weimer
such file or directory) I'm not sure if this result in a significant performance hit on Linux (because the dentry cache covers negative lookups, too), but I suspect that it could be an issue with other systems. This happens with PostgreSQL 9.1.0. -- Florian Weimer BFK edv-consu

Re: [HACKERS] Large number of open(2) calls with bulk INSERT into empty table

2011-12-06 Thread Florian Weimer
xercise file system lookup performance, you need a larger directory, which presumably means a large number of tables. -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-

Re: [HACKERS] Page Checksums + Double Writes

2011-12-22 Thread Florian Weimer
recovery time significantly because it avoids read-modify-write cycles with a cold cache? -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 -- Sent via pgsql-h

Re: [HACKERS] Making TEXT NUL-transparent

2011-12-23 Thread Florian Weimer
* Florian Pflug: > On Nov24, 2011, at 10:54 , Florian Weimer wrote: >>> Or is it not only about being able to *store* NULs in a text field? >> >> No, the entire core should be NUL-transparent. > > That's unlikely to happen. Yes, with the type input/outp

Re: [HACKERS] optimizing repeated MVCC snapshots

2012-01-05 Thread Florian Weimer
* Robert Haas: > and (c) architectures (like 32-bit x86) where ordinary 64-bit > operations aren't atomic but special instructions (cmpxchg8b) can be > used to get that behavior. FILD and FIST are atomic, too, and are supported by more micro-architectures. -- Florian Weimer

Re: [HACKERS] checkpoint writeback via sync_file_range

2012-01-11 Thread Florian Weimer
ty_bytes? We generally set it to pretty low values, and seems to help to smoothen the checkpoints. -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 --

Re: [HACKERS] Copyrights on files

2007-03-26 Thread Florian Weimer
/email.phtml>. -- Florian Weimer<[EMAIL PROTECTED]> BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 ---(end of broadcast)--- TIP

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-29 Thread Florian Weimer
, where it would be pretty straightforward). Something like an Adler32 checksum (not a full CRC) on each page might be helpful. However, what I'd really like to see is something that catches missed writes, but this is very difficult to implement AFAICT. -- Florian Weimer<

Re: [HACKERS] Something's been bugging me

2007-10-04 Thread Florian Weimer
it comes to aliasing), so it's better not to try to outsmart the C standard here. -- Florian Weimer<[EMAIL PROTECTED]> BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99

Re: [HACKERS] [GENERAL] 8.2.3: Server crashes on Windows using Eclipse/Junit

2007-10-22 Thread Florian Weimer
* Magnus Hagander: > Oh, that's interesting. That's actually a sideeffect of us increasing > the stack size for the postgres.exe executable in order to work on other > things. By default, it burns 1MB/thread, but ours will do 4MB. Never > really thought of the problem that it'll run out of address

Re: [HACKERS] Opportunity for a Radical Changes in Database Software

2007-10-27 Thread Florian Weimer
* J. Andrew Rogers: > Everything you are looking for is here: > > http://web.mit.edu/dna/www/vldb07hstore.pdf > > It is the latest Stonebraker et al on massively distributed in-memory > OLTP architectures. "Ruby-on-Rails compiles into standard JDBC, but hides all the complexity of that interface.

Re: [HACKERS] BK-Tree Implementation on top of GiST

2007-10-28 Thread Florian Weimer
* Volkan YAZICI: > [1] Some approaches to best-match file searching > http://portal.acm.org/citation.cfm?id=362003.362025 http://citeseer.ist.psu.edu/1593.html suggests that this uninteresting (too much of the database is examined) once you go past an edit distance of 1. I don't know if this

Re: [HACKERS] Hmmm ... isn't count_nondeletable_pages all wet?

2007-10-29 Thread Florian Weimer
* Tom Lane: > I am fairly sure that this bug explains problems previously reported > by Merlin Moncure: > http://archives.postgresql.org/pgsql-general/2006-10/msg01312.php > and Florian Weimer: > http://archives.postgresql.org/pgsql-general/2006-11/msg00305.php > In both tho

Re: [HACKERS] Obfuscated definitions of database objects

2007-10-29 Thread Florian Weimer
* Joshua D. Drake: > If you need obfuscation (and you don't, you just think you do, no > offense) use C. Or put the relevant code into some package/module/whatever, stored on the file system, and include that. -- Florian Weimer<[EMAIL PROTECTED]> BFK ed

[HACKERS] Asynchronous commit documentation gap

2007-11-02 Thread Florian Weimer
The documentation doesn't really tell how to disable synchronous commits for a single commit. I believe the correct command is SET LOCAL synchronous_commit TO OFF; just before the COMMIT statement. -- Florian Weimer<[EMAIL PROTECTED]> BFK edv-consulting GmbH

Re: [HACKERS] Ordered Append Node

2007-11-23 Thread Florian Weimer
* Markus Schiltknecht: >> uses a heap to efficiently find the next value from the source tapes. > > Well, maybe my point here is: why do you need the heap to sort? I think you need it because there are potentially many input types. -- Florian Weimer<[EMAIL PROT

Re: [HACKERS] Ordered Append Node

2007-11-23 Thread Florian Weimer
t; == "priority queue" here, I guess. Looking at your zipper again, it's actually an implementation of a heap. -- Florian Weimer<[EMAIL PROTECTED]> BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96

Re: [HACKERS] Ordered Append Node

2007-11-23 Thread Florian Weimer
* Markus Schiltknecht: > Florian Weimer wrote: >> I think you need it because there are potentially many input types. Eh, "tapes". > Given the partitioning case, I'd expect all rows to have an equal > tuple descriptor. Maybe this is a matter of what to optimize, th

Re: [HACKERS] [GENERAL] SHA1 on postgres 8.3

2008-01-20 Thread Florian Weimer
7;s not clear if they add significant additional security. (Sorry if this is what you've said.) -- Florian Weimer<[EMAIL PROTECTED]> BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsru

Re: [HACKERS] [GENERAL] SHA1 on postgres 8.3

2008-01-21 Thread Florian Weimer
h better, really, and the difference doesn't actually matter for many application). It's a bit like justifying that you don't need a virus scanner on your non-Windows server or database server. 8-P BTW, I'd like to see MD5/SHA-1 for BYTEA, not just TEXT, and with a BYT

[HACKERS] plperl: Documentation on BYTEA decoding is wrong

2008-01-25 Thread Florian Weimer
t;\\"!ge; You need to do this in one go because pre-escaped backslashes like like '\\101' cause problems otherwise. (All ''-delimited strings in this posting use strict SQL syntax, i.e. no escaped backslashes.) -- Florian Weimer<[EMAIL PROTECTED

Re: [HACKERS] plperl: Documentation on BYTEA decoding is wrong

2008-01-28 Thread Florian Weimer
* Robert Treat: > Note we've been using Theo's plperl bytea patch on one of our > production servers for some time; if anyone wants access to that > lmk. I'm interested. Could you post a pointer to this code, please? -- Florian Weimer<[EMAIL PROTECTE

Re: [HACKERS] [GENERAL] SHA1 on postgres 8.3

2008-01-29 Thread Florian Weimer
eral use and some function in > pgcrypto should be used instead? Yes, that would probably help those folks doing checklist-based security audits. -- Florian Weimer<[EMAIL PROTECTED]> BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721

Re: [HACKERS] configurability of OOM killer

2008-02-02 Thread Florian Weimer
verall, even if the > sysadmin hasn't configured it. How much does that help? Postmaster &c still need to be shut down when a regular backend dies due to SIGKILL. -- Florian Weimer<[EMAIL PROTECTED]> BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße

Re: [HACKERS] configurability of OOM killer

2008-02-02 Thread Florian Weimer
of the parent, the parent might just reattempt the fork() (which succeeds thanks to COW), and the child runs into the same OOM condition. -- Florian Weimer<[EMAIL PROTECTED]> BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-72

Re: [HACKERS] configurability of OOM killer

2008-02-02 Thread Florian Weimer
ore (for me, at least). Some applications are still not fully compatible with this mode (SBCL, for instance, and the Sun JVM doesn't perform as well as it could, either), but there are astonishingly few problems with vm.overcommit_memory=2. -- Florian Weimer<[EM

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Florian Weimer
as > LP_DEAD, since that will happen very quickly anyway. What about putting the whole visibility information out-of-line, into its own B-tree, indexed by page number? -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +4

Re: [HACKERS] YAML Was: CommitFest status/management

2009-12-07 Thread Florian Weimer
class instantiations (and it's not possible to switch this off, or it's rather involved to do so). Plain JSON doesn't have this issue. -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe

Re: [HACKERS] YAML Was: CommitFest status/management

2009-12-07 Thread Florian Weimer
* Alvaro Herrera: > Florian Weimer escribió: >> * Dimitri Fontaine: >> >> > Well we have JSON and agreed it was a good idea to have it. Now JSON is >> > a subset of YAML and some would prefer another YAML style (me included). >> >> YAML is rather

Re: [HACKERS] Update on true serializable techniques in MVCC

2009-12-16 Thread Florian Weimer
ble to tell a unique constraint violation caused by a phantom from an application bug. (We currently faking this by retrying a fixed number of times and bailing out if the error returned by PostgreSQL looks like a unique constraint violation.) -- Florian Weimer BFK edv-consulting Gm

  1   2   >