Re: Corruption during WAL replay

2022-03-16 Thread Robert Haas
please inform me. (It > may more than likely be too late..) If you want to take a crack at that, I'd be delighted. -- Robert Haas EDB: http://www.enterprisedb.com

Re: Unhyphenation of crash-recovery

2022-03-16 Thread Robert Haas
ency)-recovery". We don't have such wordings for "archive > >recovery" This patch fixes non-user-facing texts as well as > >user-facing ones. > > I don't see the point of this kind of change. It seems like better grammar to me, although whether it is worth th

Re: refactoring basebackup.c (zstd workers)

2022-03-17 Thread Robert Haas
On Mon, Mar 14, 2022 at 1:21 PM Robert Haas wrote: > There's some appeal to that, but one downside is that it means that > the client can't be used to fetch data that is compressed in a way > that the server knows about and the client doesn't. I don't think > that&

Re: Granting SET and ALTER SYSTE privileges for GUCs

2022-03-17 Thread Robert Haas
obvious. But maintaining a million lines of code is a lot of work, and I can't think of any reason why the burden of maintaining relatively little-used features should fall entirely on people who don't care about them. -- Robert Haas EDB: http://www.enterprisedb.com

Re: Granting SET and ALTER SYSTE privileges for GUCs

2022-03-17 Thread Robert Haas
ote the use of the word "setting" and use it in my own writing, especially to end-users. That is definitely more intelligible to random users, but it's admittedly also awkward. "Set a setting" just sounds redundant. But "set a configuration variable" sounds wordy, so I don't know. -- Robert Haas EDB: http://www.enterprisedb.com

Re: Granting SET and ALTER SYSTE privileges for GUCs

2022-03-17 Thread Robert Haas
every committer to understand exactly what's required to make it all work. I try my best to commit things that don't break it, but sometimes I do, and then Andrew helps sort it out, because he understands it and I don't. I think DAC should fall into that kind of category as well. -- Robert Haas EDB: http://www.enterprisedb.com

Re: Granting SET and ALTER SYSTE privileges for GUCs

2022-03-17 Thread Robert Haas
eciate you providing it. -- Robert Haas EDB: http://www.enterprisedb.com

Re: Unhyphenation of crash-recovery

2022-03-17 Thread Robert Haas
henate it, but I also wouldn't spend a lot of time arguing if someone else did. Except maybe that's exactly what I am doing. Perhaps I should find something else to do. -- Robert Haas EDB: http://www.enterprisedb.com

Re: Optimize external TOAST storage

2022-03-17 Thread Robert Haas
g, even if it gets longer by only 1 byte. But for larger values it's hard for that to be significant. -- Robert Haas EDB: http://www.enterprisedb.com

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-17 Thread Robert Haas
is actually totally fine, but I don't know what it is. On the other hand, it may be that things went horribly wrong and the tests are just smart enough to catch it, or maybe there's a problematic scenario which those tests just don't hit. I don't know. Thoughts? -- Robert Haas EDB: http://www.enterprisedb.com

Re: refactoring basebackup.c (zstd workers)

2022-03-17 Thread Robert Haas
e last one should just win (as it apparently does, here). We do that in some places and throw an error in others and I'm not sure if we have a 100% consistent rule for it, but flipping one location between one behavior and the other isn't going to make things more consistent overall. -- Robert Haas EDB: http://www.enterprisedb.com

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-19 Thread Robert Haas
. That acts as a universal reset of everything. So you can't reason on the basis of what somebody might be sending. -- Robert Haas EDB: http://www.enterprisedb.com

Re: refactoring basebackup.c (zstd workers)

2022-03-20 Thread Robert Haas
What's your specific thinking here? > + unsignedoptions; /* OR of > BACKUP_COMPRESSION_OPTION constants */ > > Should be "unsigned int" or "bits32" ? I do not see why either of those would be better. > The server crashes if I send an unknown option - you should hit that in the > regression tests. Turns out I was testing this on the client side but not the server side. Fixed and added more tests. v2 attached. -- Robert Haas EDB: http://www.enterprisedb.com v2-0001-Replace-BASE_BACKUP-COMPRESSION_LEVEL-option-with.patch Description: Binary data

Re: refactoring basebackup.c (zstd workers)

2022-03-20 Thread Robert Haas
matter. I think I'm guilty of verbal inexactitude here but not bad coding. Checking for *endptr != '\0', as I did, is not sufficient to detect "whether an error occurred," as I alleged. But, in the part of my response you didn't quote, I believe I made it clear that

Re: refactoring basebackup.c (zstd workers)

2022-03-20 Thread Robert Haas
ill fix. > I wonder whether the syntax should really use both ":" and ",". > Maybe ":" isn't needed at all. I don't think we should treat the compression method name in the same way as a compression algorithm option. > This patch also needs to update the other user-facing docs. Which ones exactly? > typo: contain a an OK, will fix. -- Robert Haas EDB: http://www.enterprisedb.com

Re: refactoring basebackup.c (zstd workers)

2022-03-20 Thread Robert Haas
On Sun, Mar 20, 2022 at 9:32 PM Tom Lane wrote: > Robert Haas writes: > > I think I'm guilty of verbal inexactitude here but not bad coding. > > Checking for *endptr != '\0', as I did, is not sufficient to detect > > "whether an error occurred," as

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-21 Thread Robert Haas
hat > is not possible in this case. Well, I don't see how that chain of logic addresses my concern about sinval reset. Mind you, I'm not sure there's an actual problem here, because I tried testing the patch with debug_discard_caches=1 and nothing failed. But I still don't under

Re: refactoring basebackup.c (zstd workers)

2022-03-21 Thread Robert Haas
On Mon, Mar 21, 2022 at 9:18 AM Justin Pryzby wrote: > On Sun, Mar 20, 2022 at 09:38:44PM -0400, Robert Haas wrote: > > > This patch also needs to update the other user-facing docs. > > > > Which ones exactly? > > I mean pg_basebackup -Z > > -Z level

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-21 Thread Robert Haas
would be a problem. I'm just thinking perhaps taking locks is fairly harmless and might avoid some hypothetical problem later. Thoughts? -- Robert Haas EDB: http://www.enterprisedb.com

Re: refactoring basebackup.c (zstd workers)

2022-03-21 Thread Robert Haas
enough time on it now :) Oh dear. But yes. -- Robert Haas EDB: http://www.enterprisedb.com v4-0001-Replace-BASE_BACKUP-COMPRESSION_LEVEL-option-with.patch Description: Binary data

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-22 Thread Robert Haas
e's my worked-over version of your previous patch. I haven't tried to incorporate your incremental patch that you just posted. -- Robert Haas EDB: http://www.enterprisedb.com v1-0001-Add-new-block-by-block-strategy-for-CREATE-DATABA.patch Description: Binary data

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-22 Thread Robert Haas
On Tue, Mar 22, 2022 at 11:23 AM Robert Haas wrote: > Here's my worked-over version of your previous patch. I haven't tried > to incorporate your incremental patch that you just posted. Also, please have a look at the XXX comments that I added in a few places where I think

Re: refactoring basebackup.c (zstd workers)

2022-03-22 Thread Robert Haas
On Mon, Mar 21, 2022 at 2:41 PM Dagfinn Ilmari Mannsåker wrote: > This is no longer the accurate. How about something like like "Specifies > details of the chosen compression method"? Good catch. v5 attached. -- Robert Haas EDB: http://www.enterprisedb.com v5-0001-Rep

Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce)

2022-03-22 Thread Robert Haas
sed for refilenodes after pg_upgrade relfilenode changes Reply-To:" thread, and I think the ball is in your court there. -- Robert Haas EDB: http://www.enterprisedb.com

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-22 Thread Robert Haas
; that could break it (e.g. snapshot computation, horizon stuff). Please see the function header comment for ScanSourceDatabasePgClass. I don't quite see how changes in those places would break this, but if you want to be more specific perhaps I will see the light? -- Robe

Re: pg14 psql broke \d datname.nspname.relname

2022-03-22 Thread Robert Haas
"}" against the database name and see whether it happens to match. -- Robert Haas EDB: http://www.enterprisedb.com 0002-Remove-dbname_is_literal-stuff.patch Description: Binary data 0001-Reject-patterns-with-too-many-parts-or-wrong-db.patch Description: Binary data

Re: LockAcquireExtended() dontWait vs weaker lock levels than already held

2022-03-22 Thread Robert Haas
d "weaker" aren't even necessarily well-defined ideas: A could conflict with B, B with C, and C with A, or something crazy like that. It seems rather unlikely to me that we'd ever do such a thing at this point. In fact, there are a lot of things in lock.c that we'd probably do differently if we were doing that work over. -- Robert Haas EDB: http://www.enterprisedb.com

Re: LockAcquireExtended() dontWait vs weaker lock levels than already held

2022-03-22 Thread Robert Haas
something like I don't mind you fixing the behavior. I just couldn't pass up an opportunity to complain about the structure of lock.c. -- Robert Haas EDB: http://www.enterprisedb.com

Re: Optimize external TOAST storage

2022-03-22 Thread Robert Haas
On Thu, Mar 17, 2022 at 4:05 PM Nathan Bossart wrote: > On Thu, Mar 17, 2022 at 01:04:17PM -0400, Robert Haas wrote: > > Right, so perhaps the ultimate thing here would be a more fine-grained > > knob than SET STORAGE EXTERNAL -- something that allows you to specify > > that

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-22 Thread Robert Haas
On Mon, Mar 21, 2022 at 2:23 PM Robert Haas wrote: > On Mon, Mar 21, 2022 at 11:21 AM Dilip Kumar wrote: > > I tried to debug the case but I realized that somehow > > CHECK_FOR_INTERRUPTS() is not calling the > > AcceptInvalidationMessages() and I could not find the same w

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-23 Thread Robert Haas
and directly calling smgrcreate(), or by using RelationPreserveStorage() afterwards to yank the file back out of the list. -- Robert Haas EDB: http://www.enterprisedb.com

Re: make MaxBackends available in _PG_init

2022-03-23 Thread Robert Haas
that memory based on MaxBackends is totally reasonable. Do we need to add another function, alongside _PG_init(), that gets called after MaxBackends is determined and before it's too late to RequestAddinShmemSpace()? -- Robert Haas EDB: http://www.enterprisedb.com

Re: Support isEmptyStringInfo

2022-03-23 Thread Robert Haas
tringInfo. I think this will be useful to extensions > outside core which also have these checks. They won't need to worry > about that logic/code being changed in future; rare but not impossible > case. I think that the code is perfectly readable as it is and that this change makes it

Re: refactoring basebackup.c (zstd workers)

2022-03-23 Thread Robert Haas
On Tue, Mar 22, 2022 at 11:37 AM Robert Haas wrote: > On Mon, Mar 21, 2022 at 2:41 PM Dagfinn Ilmari Mannsåker > wrote: > > This is no longer the accurate. How about something like like "Specifies > > details of the chosen compression method"? > > Good

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-23 Thread Robert Haas
think that this version handles pending syncs correctly? I think perhaps that is overlooked. -- Robert Haas EDB: http://www.enterprisedb.com

Re: pgsql: Unbreak the build.

2022-03-23 Thread Robert Haas
On Wed, Mar 23, 2022 at 12:03 PM Robert Haas wrote: > I'm looking into this now, but that's not the same Assert(false). The > one Andres is talking about is at the end of get_bc_algorithm_name(). > This one is in tar_open_for_write(). AFAIK, the first of these is > actu

multithreaded zstd backup compression for client and server

2022-03-23 Thread Robert Haas
, though. This patch contains a trivial adjustment to PostgreSQL::Test::Cluster::run_log to make it return a useful value instead of not. I think that should be pulled out and committed independently regardless of what happens to this patch overall, and possibly back-patched. Thanks, -- Robert Haas EDB: http://www.enterprisedb.com 0001-Allow-parallel-zstd-compression-when-taking-a-base-b.patch Description: Binary data

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-03-23 Thread Robert Haas
lem. The option isn't named CARE_ABOUT_VISIBILITY_MAP. It's named DISABLE_PAGE_SKIPPING. -- Robert Haas EDB: http://www.enterprisedb.com

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-03-23 Thread Robert Haas
On Wed, Mar 23, 2022 at 4:49 PM Peter Geoghegan wrote: > On Wed, Mar 23, 2022 at 1:41 PM Robert Haas wrote: > > It seems to me that if DISABLE_PAGE_SKIPPING doesn't completely > > disable skipping pages, we have a problem. > > It depends on how you define skipping.

Re: multithreaded zstd backup compression for client and server

2022-03-23 Thread Robert Haas
el zstd, 4 workers, server-side: 1.3GB, 7.2 seconds > > What causes this fairly massive client-side/server-side size difference? You seem not to have read what I wrote about this exact point in the text which you quoted. > Will this cause test failures on systems with older zstd? I put a bunch of logic in the test case to try to avoid that, so hopefully not, but if it does, we can adjust the logic. -- Robert Haas EDB: http://www.enterprisedb.com

Re: multithreaded zstd backup compression for client and server

2022-03-23 Thread Robert Haas
MAXZSTD_BLOCKSIZE_MAX > lib/zstd.h:#define ZSTD_BLOCKSIZE_MAX (1< lib/zstd.h:#define ZSTD_BLOCKSIZELOG_MAX 17 > ; -1<<17 > -131072 So does that, like, compress the value by making it way bigger? :-) -- Robert Haas EDB: http://www.enterprisedb.com

Re: libpq compression

2020-12-02 Thread Robert Haas
client side, so that the server GUC can never influence the selection of which algorithm is actually chosen, but only rule things out. But that is just a tentative opinion; maybe it's not the right idea. -- Robert Haas EDB: http://www.enterprisedb.com

Re: libpq compression

2020-12-09 Thread Robert Haas
eed is basically independent of compression method for all three of these compressors; to the extent that there is a difference, higher compression levels are generally slightly faster to decompress. I don't really see the argument for letting either side be proscriptive here. Deci

Re: [Patch] ALTER SYSTEM READ ONLY

2020-12-09 Thread Robert Haas
n at that point the previously-started checkpoint will finish normally. The latter seems better if we can make it work, but the former is probably also acceptable. What you've got right now is not. -- Robert Haas EDB: http://www.enterprisedb.com

pg_basebackup test coverage

2020-12-10 Thread Robert Haas
27;t turn up similar precedents in the existing tests. Any ideas? -- Robert Haas EDB: http://www.enterprisedb.com

Re: Autovacuum worker doesn't immediately exit on postmaster death

2020-12-10 Thread Robert Haas
like if a single system call could somehow answer both questions. -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_basebackup test coverage

2020-12-11 Thread Robert Haas
perl package approach might be better than shell commands because perhaps it is more likely to work on Windows, but I'm not positive. -- Robert Haas EDB: http://www.enterprisedb.com

Re: Incorrect allocation handling for cryptohash functions with OpenSSL

2020-12-21 Thread Robert Haas
to check that ctx is non-NULL, we should do that with an Assert(). Complicating the code with error checks that have to be added in multiple places that are far removed from where the actual problem was detected stinks. -- Robert Haas EDB: http://www.enterprisedb.com

Re: libpq compression

2020-12-22 Thread Robert Haas
On Mon, Dec 14, 2020 at 12:53 PM Daniil Zakhlystov wrote: > > On Dec 10, 2020, at 1:39 AM, Robert Haas wrote: > > Good points. I guess you need to arrange to "flush" at the compression > > layer as well as the libpq layer so that you don't end up with data >

Re: libpq compression

2020-12-22 Thread Robert Haas
00 and a user who wants to make trouble, you have a really big problem. Perhaps I'm being too pessimistic here, but man that's a lot of memory. -- Robert Haas EDB: http://www.enterprisedb.com

Re: libpq compression

2020-12-22 Thread Robert Haas
tween the sender and the receiver. What's alleged here is that the sender can do a thing which causes the receiver to burn through tons of memory. It doesn't help anything to say, well, the sender ought to use a window size of N or less. What if they don't? -- Robert Haas EDB: http://www.enterprisedb.com

Re: Proposal: Global Index

2021-01-07 Thread Robert Haas
feature to mature as soon as possible. > I summarized the scheme mentioned in the email and completed the POC > patch(base on PG_13). You need to summarize the basic design choices you've made here. Like, what did you do about the fact that TIDs have to be 6 bytes? -- Robert Haas EDB: http://www.enterprisedb.com

Re: Disable WAL logging to speed up data loading

2021-01-07 Thread Robert Haas
avoid accidental mistakes. -- Robert Haas EDB: http://www.enterprisedb.com

Re: [Patch] ALTER SYSTEM READ ONLY

2021-02-16 Thread Robert Haas
eptWrites() at the end of StartupXLOG() and instead have the checkpointer do it later when the system is going read-write for the first time. -- Robert Haas EDB: http://www.enterprisedb.com

Re: new heapcheck contrib module

2021-02-17 Thread Robert Haas
; such failures, and I'm not sure that pg_amcheck should just abort. I agree that exit(1) should happen after any error intended to stop the program. But I think it should also happen at the end of the run if we hit any problems for which we did not stop, so that exit(0) means your database is healthy. -- Robert Haas EDB: http://www.enterprisedb.com

Re: new heapcheck contrib module

2021-02-17 Thread Robert Haas
or example pg_dump and psql don't do this, instead putting everything into one ordered list, despite also having a lot of options. This is arguably worse if you want to understand which options are related to each other, but it's better if you are just looking for something based on alphabetical order. -- Robert Haas EDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2021-02-19 Thread Robert Haas
don't know what that reason might be. We seem to have no test cases either in the main test suite or in the plpgsql test suite where ER_flatten_info gets called with ER_FLAG_HAVE_EXTERNAL is set, which seems a little unfortunate. If there is such a bug here it's independent of this patch, I suppose, but it would still be nice to understand what's going on here better than I do. -- Robert Haas EDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2021-02-19 Thread Robert Haas
On Fri, Feb 19, 2021 at 4:21 PM Robert Haas wrote: > What makes me a bit uncomfortable about that approach is that it > presupposes that everything that uses expanded records has some other > defense against those tuples getting written to disk without first > expanding any external

Re: libpq compression

2021-02-22 Thread Robert Haas
rk consumption. So at the end of the day I'm not really quite sure what is best here. I agree with all of Craig's points about the advantages of packet-level compression, so I'd really prefer to make that approach work if we can. However, it also seems to me that there's room to be fairly concerned about what these test results are showing. -- Robert Haas EDB: http://www.enterprisedb.com

Re: new heapcheck contrib module

2021-02-24 Thread Robert Haas
exit(0) means your > database was successfully checked and is healthy. wfm. -- Robert Haas EDB: http://www.enterprisedb.com

Re: new heapcheck contrib module

2021-03-01 Thread Robert Haas
infoParse() in the right place, so that what ends up in the cparams is just the db name, not a whole connection string. This is just based on a relatively short amount of time spent studying the patch, so I might well be off-base here. What do you think? -- Robert Haas EDB: http://www.enterprisedb.com

Re: new heapcheck contrib module

2021-03-02 Thread Robert Haas
ks about a cparams argument that it no longer has. The usual idiom for sizing a memory allocation involving FLEXIBLE_ARRAY_MEMBER is something like offsetof(ParallelSlotArray, slots) + numslots * sizeof(ParallelSlot). Your version uses sizeof(); don't. Other than that 000

Re: new heapcheck contrib module

2021-03-02 Thread Robert Haas
On Tue, Mar 2, 2021 at 1:24 PM Robert Haas wrote: > Other than that 0001 looks to me to be in pretty good shape now. Incidentally, we might want to move this to a new thread with a better subject line, since the current subject line really doesn't describe the uncommitted portion of

Re: [Patch] ALTER SYSTEM READ ONLY

2021-03-03 Thread Robert Haas
r "pg_prohibit_wal" I think we also ought to rename the GUC, system_is_read_only -> wal_prohibited. -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_amcheck contrib application

2021-03-03 Thread Robert Haas
er about making -v print a message per relation so that we can see, in human-readable format, which relations are getting checked. Right now, if you got an error checking just one relation, how would you know which relation you got it from? Unless the server happens to report that information in the message, you're just in the dark, because pg_amcheck won't tell you. The line "Read the description of the amcheck contrib module for details" seems like it could be omitted. Perhaps the first line of the help message could be changed to read "pg_amcheck uses amcheck to find corruption in a PostgreSQL database." or something like that, instead. -- Robert Haas EDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2021-03-03 Thread Robert Haas
the code base and widely-used APIs for everyone. Another advantage is that getting those first two patches committed or even just applied locally on a branch would, at least IMHO, make it a lot simpler to see what potential problem spots remain - and by "problem" I mean mostly from a pe

Re: pg_amcheck contrib application

2021-03-04 Thread Robert Haas
, you have got a bad UI. If it dumps out 300 lines of output instead of 20 or 40, you have a UI that is so verbose that usability is going to be somewhat impaired, which is why I suggested only showing the query in verbose mode. BTW, another thing that might be interesting is to call PQsetErrorVerbosity(conn, PQERRORS_VERBOSE) in verbose mode. It's probably possible to contrive a case where the server error message is something generic like "cache lookup failed for relation %u" which occurs in a whole bunch of places in the source code, and being able get the file and line number information can be really useful when trying to track such things down. -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_amcheck contrib application

2021-03-04 Thread Robert Haas
On Thu, Mar 4, 2021 at 10:29 AM Robert Haas wrote: > Most of these changes sound good. I'll go through the whole patch > again today, or as much of it as I can. But before I do that, I want > to comment on this point specifically. Just a thought - I don't feel strongly about

Re: pg_amcheck contrib application

2021-03-04 Thread Robert Haas
On Thu, Mar 4, 2021 at 12:27 PM Robert Haas wrote: > More in a bit, need to grab some lunch. Moving on to the tests, in 003_check.pl, I think it would be slightly better if relation_toast were to select ct.oid::regclass and then just have the caller use that value directly. We'd certai

Re: pg_amcheck contrib application

2021-03-08 Thread Robert Haas
you put it in amcheck itself and just expose it through pg_amcheck, then it's accessible either way. It's probably cleaner and more performant to do it that way, too. So if you did add a sampling option in the future, that's the way I would recommend doing it, but I think it is probably best not to go there right now. -- Robert Haas EDB: http://www.enterprisedb.com

Re: New IndexAM API controlling index vacuum strategies

2021-03-08 Thread Robert Haas
MaxHeapTuplesPerPage is used to size some stack-allocated arrays, especially the stack-allocated PruneState. I thought for a while about this and I can't really see why it would be a big problem, even with a large increase in MaxHeapTuplesPerPage, so I'm just mentioning this in case it makes somebody else think of something I've missed. -- Robert Haas EDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2021-03-08 Thread Robert Haas
ntroduces is a pain, too. I think accepting a feature that gives us LZ4 compression is better than rejecting it because we can't solve those problems. -- Robert Haas EDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2021-03-08 Thread Robert Haas
DefaultCompressionMethod. Surely lotsa people are going to want to set > > toast_compression = lz4 in postgresql.conf and forget about it. > > I don't understand - isn't that what 0002 does ? Oh, man, you want me to look at all the patches and not just the first one? :-) -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_amcheck contrib application

2021-03-10 Thread Robert Haas
It doesn't seem to make sense to connect to a database, set the search path, exit, and then immediately reconnect to the same database. This is slightly inconsistent: pg_amcheck: checking heap table "rhaas"."public"."foo" heap relation "rhaas"."public"."foo": ERROR: XX000: catalog is missing 144 attribute(s) for relid 16392 LOCATION: RelationBuildTupleDesc, relcache.c:652 query was: SELECT blkno, offnum, attnum, msg FROM "public".verify_heapam( relation := 16392, on_error_stop := false, check_toast := true, skip := 'none') In line 1 it's a heap table, but in line 2 it's a heap relation. That's all I've got. -- Robert Haas EDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2021-03-10 Thread Robert Haas
still need to make sure that we don't just blindly let compressed values wander around everywhere. When we insert into a table column with a compression method, we should recompress any data that is compressed using some other method. -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_amcheck contrib application

2021-03-11 Thread Robert Haas
quite a large number of binaries that we ship, so maybe it's best not to add to it, especially for a tool which I at least suspect is going to get a lot more use than either of those. Anyone else want to vote for or against moving this to src/bin? -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_amcheck contrib application

2021-03-11 Thread Robert Haas
that there should only ever be EITHER a bare command-line argument OR options that require querying for a list of databases OR neither BUT NOT both. Then it's simple: 0. If you have both options which require querying for a list of databases and also a bare database name, error and die. 1. As above. 2. As above except the only possibility is now increasing the list of target databases from length 0 to length 1. 3. As above. -- Robert Haas EDB: http://www.enterprisedb.com

Re: New IndexAM API controlling index vacuum strategies

2021-03-11 Thread Robert Haas
t fragmented. But I'm not sure it's OK to relocate even a HOT tuple to a different TID. Can someone, perhaps even just the user, still have a reference to the old one and care about us invalidating it? Maybe. But even if not, I'm not sure this helps much with the situation you're concern

Re: pg_amcheck contrib application

2021-03-11 Thread Robert Haas
On Wed, Mar 10, 2021 at 11:02 PM Mark Dilger wrote: > [ new patches ] Seems like this is mostly ready to commit now, modulo exactly what to do about the maintenance DB stuff, and whether to move it to src/bin. Since neither of those affects 0001, I went ahead and committed that part. -- Rob

Re: Is Recovery actually paused?

2021-03-11 Thread Robert Haas
On Mon, Mar 1, 2021 at 12:08 AM Dilip Kumar wrote: > > > > I'll wait for a day before marking this RfC in case anyone have > > > > further comments. > > > > > > Okay. > > > > Hearing nothing, done that. > > Thanks.

Re: pg_amcheck contrib application

2021-03-11 Thread Robert Haas
On Thu, Mar 11, 2021 at 11:09 AM Robert Haas wrote: > An alternate possibility would be to say that there should only ever > be EITHER a bare command-line argument OR options that require > querying for a list of databases OR neither BUT NOT both. Then it's > simple: > &

Re: pg_amcheck contrib application

2021-03-12 Thread Robert Haas
ince nobody is saying we *shouldn't* move this to src/bin, I think you may as well go put it there per Peter's suggestion. Then I think it's time to get this committed and move on to the next thing. -- Robert Haas EDB: http://www.enterprisedb.com more-doc-hacking.patch Description: Binary data

Re: pg_amcheck contrib application

2021-03-12 Thread Robert Haas
farm thinks. -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_amcheck contrib application

2021-03-12 Thread Robert Haas
omething that already exists, probably because he knows that I'm finnicky about consistency -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_amcheck contrib application

2021-03-12 Thread Robert Haas
_btree’ set but not used > [-Wunused-but-set-variable] > 2060 | bool is_btree = false; > | ^~~~ > > Looks like this 'is_btree' variable should be PG_USED_FOR_ASSERTS_ONLY. I'll commit something shortly to address these. -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_amcheck contrib application

2021-03-12 Thread Robert Haas
On Fri, Mar 12, 2021 at 2:31 PM Robert Haas wrote: > I'll commit something shortly to address these. There are some interesting failures in the test cases on the buildfarm. One of the tests ($offnum == 13) corrupts the TOAST pointer with a garbage value, expecting to get the message &quo

Re: pg_amcheck contrib application

2021-03-12 Thread Robert Haas
On Fri, Mar 12, 2021 at 5:24 PM Mark Dilger wrote: > This does nothing to change the verbiage from contrib/amcheck, but it should > address the problems discussed here in pg_amcheck's regression tests. Committed. -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_amcheck contrib application

2021-03-12 Thread Robert Haas
> spotted by Robert. Here is a patch: OK, I committed this too, along with the one I hadn't committed yet from your previous email. Gah, tests are so annoying. :-) -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_amcheck contrib application

2021-03-13 Thread Robert Haas
ong, a zero value is more likely to look like something normal than whatever else. I suggest picking a value where all 8 bytes are the same, but not zero, and ideally chosen so that they don't look much like any of the surrounding bytes. -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_amcheck contrib application

2021-03-13 Thread Robert Haas
ttle > easier to remember than other permutations of those nibbles. OK, committed. The nibbles seem less relevant than the bytes as a whole, but that's fine. -- Robert Haas EDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2021-03-15 Thread Robert Haas
lz4'd and de-lz4 it before returning it, since a compressed lz4 datum is impossible. I'm open to being convinced that we don't need to do either of these things, and that the cost of iterating over all varlenas in the tuple is not so bad as to preclude doing things as you have them here. But, I'm afraid it's going to be too expensive. -- Robert Haas EDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2021-03-16 Thread Robert Haas
hat's optional because we don't have a good way of making it fast seems like a major cop-out, at least to me. I think from a user perspective you don't expect INSERT INTO .. SELECT FROM to create a different final state than a dump and reload, and that if we deviate from that people are gonna be unhappy. I could be wrong; maybe it's only me who would be unhappy. -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_amcheck contrib application

2021-03-16 Thread Robert Haas
we then get garbage when we deform the tuple? I mean, heap_deform_tuple() is going to get the alignment from the tuple descriptor, which is a table property. It doesn't (and can't) know what type of value is stored inside any of these fields for real, right? In other words, i

Re: pg_amcheck contrib application

2021-03-16 Thread Robert Haas
;t decide to use a larger OR smaller alignment than expected without breaking stuff. In what context is it OK to just add extra alignment padding? -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_amcheck contrib application

2021-03-16 Thread Robert Haas
f corruption. Maybe we should be warning about tuples that are have xmin or xmax flagged as committed or invalid when in fact clog disagrees. That's not a particularly uncommon case, and it's hard to check. -- Robert Haas EDB: http://www.enterprisedb.com

Re: shared-memory based stats collector

2021-03-16 Thread Robert Haas
tremely widely-included already. Maybe it should be broken up into pieces so that we're not including so MUCH stuff in a zillion places, but the header that contains the definition of CHECK_FOR_INTERRUPTS() is always going to be needed in a ton of spots. Honestly, I wonder why w

Re: pg_amcheck contrib application

2021-03-16 Thread Robert Haas
avaluesX values on d-aligned boundaries. > > It could be that a particular entry is of an array type that > only requires i-alignment. But that doesn't break anything, > it just means we inserted more padding than an omniscient > implementation would do. OK, yeah, I just misunder

Re: [HACKERS] Custom compression methods

2021-03-16 Thread Robert Haas
e compressed data as it is to the target table. Uggh. I don't like that behavior either, but I guess if it's the long-established way things work then perhaps this is no worse. -- Robert Haas EDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2021-03-16 Thread Robert Haas
t the uncompressed size from the TOAST pointer itself. But the code would have to know to do that, as you say. -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_amcheck contrib application

2021-03-16 Thread Robert Haas
dea, and could apply to any new messages as well as some existing ones. Maybe there are also more fields in the TOAST pointer for which we could add checks. -- Robert Haas EDB: http://www.enterprisedb.com

<    1   2   3   4   5   6   7   8   9   10   >