Re: pread() and pwrite()
On Sat, Jul 21, 2018 at 3:34 AM Tom Lane wrote: > Heikki Linnakangas writes: > > No objections, if you want to make the effort. But IMHO the lseek+read > > fallback is good enough on Windows. Unless you were thinking that we > > could then remove the !HAVE_PREAD fallback altogether. Are there any > > other platforms out there that don't have pread/pwrite that we care about? > > AFAICT, macOS has them as far back as we care about (prairiedog does). > HPUX 10.20 (gaur/pademelon) does not, so personally I'd like to keep > the lseek+read workaround. Don't know about the oldest Solaris critters > we have in the buildfarm. FreeBSD has had 'em at least since 4.0 (1994); > didn't check the other BSDen. > > SUS v2 (POSIX 1997) does specify both functions, so we could insist on > their presence without breaking any of our own portability guidelines. > However, if we have to have some workaround anyway for Windows, it > seems like including an lseek+read code path is reasonable so that we > needn't retire those oldest buildfarm critters. Yeah it seems useful and cheap to carry the lseek() fallback. But actually there is a good reason to implement proper pread/pwrite (equivalent) on Windows: this patch removes the position tracking, so that the fallback code generates *more* lseek() calls than current master. For example with sequential reads today we are smart enough to skip redundant lseek() calls, but this patch removes those smarts. I doubt anyone cares about that on HPUX 10.20 but I don't think we should do that on Windows. -- Thomas Munro http://www.enterprisedb.com
Re: Loaded footgun open_datasync on Windows
On Wed, Jun 27, 2018 at 12:09:24PM +0200, Laurenz Albe wrote: > Michael Paquier wrote: > > > I have added it to the July commitfest. > > > > Have you looked at the possibility of removing the log file constraints > > in pg_upgrade with the change you are doing here so as things would be > > more consistent with non-Windows platforms, simplifying some code on the > > way? > > Can you explain what the "log file constraints" are? > If it is in any way related, and I can handle it, sure. See this comment in pg_upgrade.h: /* * WIN32 files do not accept writes from multiple processes * * On Win32, we can't send both pg_upgrade output and command output to the * same file because we get the error: "The process cannot access the file * because it is being used by another process." so send the pg_ctl * command-line output to a new file, rather than into the server log file. * Ideally we could use UTILITY_LOG_FILE for this, but some Windows platforms * keep the pg_ctl output file open by the running postmaster, even after * pg_ctl exits. * * We could use the Windows pgwin32_open() flags to allow shared file * writes but is unclear how all other tools would use those flags, so * we just avoid it and log a little differently on Windows; we adjust * the error message appropriately. */ If you grep src/bin/pg_upgrade for WIN32, roughly a third of the hits are workarounds for this problem. I agree with Michael that removing those workarounds would be a good test of frontend pgwin32_open() and worth including in the initial commit.
Re: pg_verify_checksums and -fno-strict-aliasing
On Sat, Sep 01, 2018 at 03:32:10PM -0400, Tom Lane wrote: > Fair enough. I renamed the types as suggested, changed a few more > places for consistency's sake, and pushed. Thanks! > There still remain some places where palloc(BLCKSZ) or equivalent is used, > but there's no matching pfree. In a lot of them the buffer is returned > to the caller so there's no choice. It's likely that some are just > leaking the storage transiently and we could convert them to using a > PGAlignedBlock local variable, but I didn't bother trying to do the > analysis. At quick glance, I am not seeing anything critical. So the result looks good to me. -- Michael signature.asc Description: PGP signature
Re: Configuring messages language on Windows
On Fri, Jan 26, 2018 at 04:54:08PM +0300, a.parfe...@postgrespro.ru wrote:
> As it mentioned in pg_locale.c, the variable LC_MESSAGES is ignored in
> Windows(pg_locale.c:162).
That comment says "On Windows, setlocale(LC_MESSAGES) does not work". It says
nothing about the LC_MESSAGES environment variable.
> Additionally, this affects regress tests, since language
> for messages generated by psql is also configured via LC_MESSAGES and
> ignored on Windows installations and cause failure of tests on
> non-English Windows installations.
I vaguely recall having seen such a problem with an old version of GNU
gettext. On Windows Server 2016, with binaries from
https://get.enterprisedb.com/postgresql/postgresql-11.0-beta3-windows-x64-binaries.zip,
in an account having its Windows display language set to "Español (España)", I
get this behavior:
$ for lang in '' it C; do LC_MESSAGES=$lang ./psql.exe --help; done | grep -e
--output
-o, --output=ARCHIVO enviar resultados de consultas a archivo (u |orden)
-o, --output=NOME_FILE reindirizza i risultati al file specificato
-o, --output=FILENAMEsend query results to file (or |pipe)
If you run those commands without your patch, what do you see? What do you
see with your patch? What does your environment have for each of the
following characteristics?
- Windows version
- gettext version
- user's Windows display language
- user's locale ("Format" in "Region" control panel)
- Windows ANSI code page
("Language for non-Unicode programs" in "Region" control panel)
Re: Seeking exciting PostgreSQL development opportunities
On Sat, Sep 1, 2018 at 2:13 PM, Charles Cui wrote: >Thanks again for supporting me to pass the GSoC summer project and I > found that I have a learnt a lot during this project! Although the project > is completed, I am thinking of contributing to PostgreSQL community in the > long term. So, the purpose of this email is to seek the exciting project you > are currently working on or you plan to work on and see whether I am a fit > for these projects. I should be able to contribute in my part time and I can > work remotely in China. Please feel free to share your project and let's > start coding :) I always have difficulty answering this question. It is a question that can be personal, at least in my opinion. It may seem like it should be easy for experienced contributors to offer specific guidance, but I doubt that it is. I can offer you my personal opinion on how to go about working on Postgres as a novice hacker: What are you interested in? The project that has the greatest chance of success may be the one that you have the greatest intrinsic motivation to work on. You are the one that will be doing the work -- nobody else will. I don't mean the project that is the most fun, though it may be a fun project. There are large differences in how individual contributors assess how important a project is to users. You may be able to identify an area where Postgres is weak from the point of view of a large number of users -- people expect many things from a database system. It might even be relatively easy to add an enhancement that satisfies these users. Don't assume that somebody else would have seen the opportunity already, so it must not be there. This mentality can be very counter-productive. A long term outlook is absolutely essential. I also think it's important to know *why* you're doing something. Ideally, you'll have multiple reasons for working on something, and not just one. Ideally, you'll be able to pick a project that could easily lead to another project, and then another. A "virtuous circle" can be created. -- Peter Geoghegan
Seeking exciting PostgreSQL development opportunities
Hi mentors and hackers, Thanks again for supporting me to pass the GSoC summer project and I found that I have a learnt a lot during this project! Although the project is completed, I am thinking of contributing to PostgreSQL community in the long term. So, the purpose of this email is to seek the exciting project you are currently working on or you plan to work on and see whether I am a fit for these projects. I should be able to contribute in my part time and I can work remotely in China. Please feel free to share your project and let's start coding :) Thanks Charles.
Re: Some pgq table rewrite incompatibility with logical decoding?
On 01/09/18 18:25, Jeremy Finzel wrote: > > Interesting. > > So you were running 9.6.9 before, it triggered the issue (and was not > able to recover). You took a filesystem snapshot, started a 9.6.10 on > the snapshot, and it recovered without hitting the issue? > > > I am resposting this to the list and not only to Tomas. Tomas, I can’t > promise just yet to delve into this because given the patch fixes the > issue it’s obviously much lower priority for our team. Are you hoping > for me to confirm the exact scenario in which the 9.6.10 patch fixes the > bug? > > Actually, there were more things changed than that so I'm not positive > it was the last patch: > > BEFORE: > Provider - 9.6.8-1.pgdg16.04+1, pglogical 2.1.1-1.xenial+1 > Subscriber - 9.6.9-2.pgdg16.04+1, 2.1.1-1.xenial+1 > > AFTER: > Provider - 9.6.10-1.pgdg16.04+1, pglogical 2.2.0-3.xenial+1 > Subscriber - 9.6.10-1.pgdg16.04+1, pglogical 2.2.0-3.xenial+1 > I finally managed to reproduce this and it's indeed fixed in the 9.6.10 by the commit da10d6a. It was caused by the issue with subtransaction and SnapBuilder snapshots described there. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: pg_verify_checksums and -fno-strict-aliasing
Michael Paquier writes: > On Fri, Aug 31, 2018 at 07:59:58PM -0400, Tom Lane wrote: >> The others you mention could be changed, probably, but I didn't >> bother as they didn't seem performance-critical. > It is not really critical indeed. There is an argument to change them > so as other folks get used to it though. Fair enough. I renamed the types as suggested, changed a few more places for consistency's sake, and pushed. There still remain some places where palloc(BLCKSZ) or equivalent is used, but there's no matching pfree. In a lot of them the buffer is returned to the caller so there's no choice. It's likely that some are just leaking the storage transiently and we could convert them to using a PGAlignedBlock local variable, but I didn't bother trying to do the analysis. regards, tom lane
Re: [HACKERS] PATCH: Keep one postmaster monitoring pipe per process
On Thu, Jul 19, 2018 at 11:51 PM Thomas Munro wrote: > On Thu, Jul 19, 2018 at 10:30 PM, Kyotaro HORIGUCHI > > Yeah. That seems good. Couldn't we reuse prepared WaitEventSet in > > other places? For example PgstatCollectorMain has the same > > characteristics, where WaitLatchOrSocket is used with fixed > > parameters and waiting on a socket which gets frequent receipts. > > +1, but I'm considering that to be a separate project, or I'll never > get this patch committed. It may be possible to have a small number > of them reused in many places, and it may be possible for > WaitLatchXXX() to reuse them automatically (so we don't have to change > every call site). > > > # Is it intentional that the patch doesn't touch pgstat.c? > > Yes. pgstat.c still uses WL_POSTMASTER_DEATH because it does > something special: it calls pgstat_write_statsfiles() before it exits. Rebased. -- Thomas Munro http://www.enterprisedb.com 0001-Add-WL_EXIT_ON_PM_DEATH-pseudo-event-v3.patch Description: Binary data
Re: [PATCH] Fix docs to JOHAB encoding on server side
Lars Kanis writes: > The current documentation is inconsistent about the JOHAB character encoding > on server side. While the first table says that it is not possible to use > JOHAB on server side, it is still listed in table 23.2 as a server character > set: > https://www.postgresql.org/docs/devel/static/multibyte.html#MULTIBYTE-CHARSET-SUPPORTED Hm, yeah. Probably this is because JOHAB was first installed as being a server-allowed encoding, and it was only later we realized that wasn't OK. This table must've been missed while changing it. Cross-checking the other entries, I notice SHIFT_JIS_2004 is also not listed in that table ... we've been a little sloppy there. Thanks for noticing! regards, tom lane
Re: Undo logs
On Fri, Aug 31, 2018 at 10:24 PM Dilip Kumar wrote: > On Fri, Aug 31, 2018 at 3:08 PM, Dilip Kumar wrote: > > As Thomas has already mentioned upthread that we are working on an > > undo-log based storage and he has posted the patch sets for the lowest > > layer called undo-log-storage. > > > > This is the next layer which sits on top of the undo log storage, > > which will provide an interface for prepare, insert, or fetch the undo > > records. This layer will use undo-log-storage to reserve the space for > > the undo records and buffer management routine to write and read the > > undo records. I have also pushed a new WIP version of the lower level undo log storage layer patch set to a public branch[1]. I'll leave the earlier branch[2] there because the record-level patch posted by Dilip depends on it for now. The changes are mostly internal: it doesn't use DSM segments any more. Originally I wanted to use DSM because I didn't want arbitrary limits, but in fact DSM slots can run out in unpredictable ways, and unlike parallel query the undo log subsystem doesn't have a plan B for when it can't get the space it needs due to concurrent queries. Instead, this version uses a pool of size 4 * max_connections, fixed at startup in regular shared memory. This creates an arbitrary limit on transaction size, but it's a large at 1TB per slot, can be increased, doesn't disappear unpredictably, is easy to monitor (pg_stat_undo_logs), and is probably a useful brake on a system in trouble. More soon. [1] https://github.com/EnterpriseDB/zheap/tree/undo-log-storage-v2 [2] https://github.com/EnterpriseDB/zheap/tree/undo-log-storage -- Thomas Munro http://www.enterprisedb.com
Re: A strange GiST error message or fillfactor of GiST build
Alexander Korotkov writes: > Thus, I would vote for removing GiST fillfactor altogether. Assuming > we can't do this for compatibility reasons, I would vote for setting > default GiST fillfactor to 100, and don't introduce new places where > we take it into account. We probably can't remove the fillfactor storage parameter, both for backwards compatibility and because I think it's implemented independently of index type. But there's no backwards-compatibility argument against simply ignoring it, if we conclude it's a bad idea. regards, tom lane
Re: A strange GiST error message or fillfactor of GiST build
On Sat, Sep 1, 2018 at 6:03 PM Robert Haas wrote: > On Wed, Aug 29, 2018 at 4:32 AM, Kyotaro HORIGUCHI > wrote: > > After the attached patch applied, the above messages becomes as > > follows. (And index can be built being a bit sparse by fill > > factor.) > > > >> ERROR: index row size 8016 exceeds maximum 7333 for index "y_cube_idx" > > > > I'm not sure why 277807bd9e didn't do that completely so I may be > > missing something. Is there any thoughts? > > It seems strange to me that we consider respecting the fillfactor to > be more important than letting the operation succeed. I would have > thought that the fillfactor would not apply when placing a tuple into > a completely empty page. The point of the setting is, of course, to > leave some free space available on the page for future tuples, but if > the tuples are big enough that only one fits in a page anyway, that's > pointless. IIRC, I've already wrote that I think we don't need GiST fillfactor parameter at all. As you pointed, the purpose of fillfactor parameter is to leave some free space in the pages. That, in turn, allow us to evade the flood of page splits, which may happen when you start insertions into freshly build and perfectly packed index. But thats makes sense only for index building algorithms, which can pack index pages as tight as possible. Our B-tree build algorithm is one of such alogirhtms: at first it sorts tuples and then packs them into pages as tight as required. But GiST is another story: GiST index build in the pretty same as insertion tuples one by one. Yes, we have some bulk insert optimization for GiST, but it optimizes only IO and internally still uses picksplit. So, GiST indexes are never perfectly packed even with fillfactor = 100. Why should we bother setting lower fillfactor? Thus, I would vote for removing GiST fillfactor altogether. Assuming we can't do this for compatibility reasons, I would vote for setting default GiST fillfactor to 100, and don't introduce new places where we take it into account. -- Alexander Korotkov Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
Re: Reopen logfile on SIGHUP
On Thu, Aug 30, 2018 at 2:44 PM Kyotaro HORIGUCHI wrote: > At Thu, 30 Aug 2018 13:42:42 +0300, Alexander Korotkov > wrote in > > > It seems that http://commitfest.cputube.org/ runs only "make check" on > > Windows. But my Postgres Pro colleagues checked that tests passed on > > 32-bit and 64-bit versions of Windows Server 2008. Also I made some > > minor beautifications on code and documentation. > > > > This patch seems to have good shape and generally being quite > > harmless. Do we have any objections to committing this? > > I checked that on my Win7 box and worked. Of course I have no > objection. So, pushed. Thank you. -- Alexander Korotkov Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
Re: Some pgq table rewrite incompatibility with logical decoding?
> > Interesting. > > So you were running 9.6.9 before, it triggered the issue (and was not > able to recover). You took a filesystem snapshot, started a 9.6.10 on > the snapshot, and it recovered without hitting the issue? > I am resposting this to the list and not only to Tomas. Tomas, I can’t promise just yet to delve into this because given the patch fixes the issue it’s obviously much lower priority for our team. Are you hoping for me to confirm the exact scenario in which the 9.6.10 patch fixes the bug? Actually, there were more things changed than that so I'm not positive it was the last patch: BEFORE: Provider - 9.6.8-1.pgdg16.04+1, pglogical 2.1.1-1.xenial+1 Subscriber - 9.6.9-2.pgdg16.04+1, 2.1.1-1.xenial+1 AFTER: Provider - 9.6.10-1.pgdg16.04+1, pglogical 2.2.0-3.xenial+1 Subscriber - 9.6.10-1.pgdg16.04+1, pglogical 2.2.0-3.xenial+1 > I quickly went through the commits in 9.6 branch between 9.6.9 and > 9.6.10, looking for stuff that might be related, and these three commits > seem possibly related (usually because of invalidations, vacuum, ...): > > 6a46aba1cd6dd7c5af5d52111a8157808cbc5e10 > Fix bugs in vacuum of shared rels, by keeping their relcache entries > current. > > da10d6a8a94eec016fa072d007bced9159a28d39 > Fix "base" snapshot handling in logical decoding > > 0a60a291c9a5b8ecdf44cbbfecc4504e3c21ef49 > Add table relcache invalidation to index builds. > > But it's hard to say if/which of those commits did the trick, without > more information. > Let me know if that info gives you any more insight - actually 2 point version jumps for provider, 1 for subscriber. Thanks, Jeremy
[PATCH] Fix docs to JOHAB encoding on server side
The current documentation is inconsistent about the JOHAB character encoding on server side. While the first table says that it is not possible to use JOHAB on server side, it is still listed in table 23.2 as a server character set: https://www.postgresql.org/docs/devel/static/multibyte.html#MULTIBYTE-CHARSET-SUPPORTED -- Kind Regards, Lars From 65d60fc66cf62af9ea2d570590a541fbdf397767 Mon Sep 17 00:00:00 2001 From: Lars Kanis Date: Sat, 1 Sep 2018 18:03:14 +0200 Subject: [PATCH] Fix docs to JOHAB encoding on server side This is probably left over from commit 6041b92238897b06fe7bbe229a6e99f80121fa4a . --- doc/src/sgml/charset.sgml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/doc/src/sgml/charset.sgml b/doc/src/sgml/charset.sgml index dc3fd34a62..452e625e01 100644 --- a/doc/src/sgml/charset.sgml +++ b/doc/src/sgml/charset.sgml @@ -1538,8 +1538,7 @@ $ psql -l JOHAB - JOHAB, - UTF8 + not supported as a server encoding -- 2.17.1
Re: A strange GiST error message or fillfactor of GiST build
On Wed, Aug 29, 2018 at 4:32 AM, Kyotaro HORIGUCHI wrote: > After the attached patch applied, the above messages becomes as > follows. (And index can be built being a bit sparse by fill > factor.) > >> ERROR: index row size 8016 exceeds maximum 7333 for index "y_cube_idx" > > I'm not sure why 277807bd9e didn't do that completely so I may be > missing something. Is there any thoughts? It seems strange to me that we consider respecting the fillfactor to be more important than letting the operation succeed. I would have thought that the fillfactor would not apply when placing a tuple into a completely empty page. The point of the setting is, of course, to leave some free space available on the page for future tuples, but if the tuples are big enough that only one fits in a page anyway, that's pointless. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Re: buildfarm: could not read block 3 in file "base/16384/2662": read only 0 of 8192 bytes
Andres Freund writes: > One concern I have with your approach is that it isn't particularly > bullet-proof for cases where the rebuild is triggered by something that > doesn't hold a conflicting lock. Wouldn't that be a bug in the something-else? The entire relation cache system is based on the assumptions that (a) if you hold lock, you can read a consistent and valid set of information about the rel from the catalogs, and (b) anyone changing that info must hold a conflicting lock and send an SINVAL message *before* releasing said lock. I'm not prepared to consider a redesign of those assumptions, especially not for back-patching. regards, tom lane
Re: Online verification of checksums
Hallo Michael, I've now forward-ported this change to pg_verify_checksums, in order to make this application useful for online clusters, see attached patch. Patch does not seem to apply anymore, could you rebase it? -- Fabien.
