On Mon, Aug 24, 2026 at 9:01 PM Dilip Kumar <[email protected]> wrote: > > On Mon, Aug 24, 2026 at 3:02 PM Amit Kapila <[email protected]> wrote: > > > > On Fri, Aug 21, 2026 at 12:13 PM vignesh C <[email protected]> wrote: > > > > > > Thanks for the patch Dilip. > > > Logical replication breaks when conflict_log_destination = 'table' and > > > an update_origin_differs conflict involves a large row. > > > When the conflict occurs on a row containing a large value (~190 MB), > > > the conflict log table path converts the old row data into a string > > > representation. During this conversion, the resulting string buffer > > > grows beyond 1 GB and exceeds PostgreSQL's maximum StringInfo buffer > > > size of 1073741823 bytes. > > > > > > This results in: > > > "ERROR: string buffer exceeds maximum allowed length (1073741823 bytes)" > > > "DETAIL: Cannot enlarge string buffer containing 1073741808 bytes by > > > 32 more bytes." > > > > > > As a result, the logical replication apply worker exits, and > > > replication cannot make further progress. > > > Interestingly, the same scenario works successfully with > > > conflict_log_destination = 'log'. In that case, the conflict is logged > > > successfully and logical replication continues without any error. > > > > > > Reproducer > > > The issue can be reproduced with the following steps: > > > 1) Set up logical replication with conflict_log_destination = 'table' > > > and create the following table: > > > "CREATE TABLE tab (a int PRIMARY KEY, b text);" > > > 2) On the subscriber, enable track_commit_timestamp = on to detect > > > update_origin_differs conflicts. > > > 3) Insert a row on the publisher: > > > "INSERT INTO tab VALUES (1, 'small');" > > > 4) On the subscriber, update the replicated row with a large value: > > > "UPDATE tab SET b = repeat(chr(1), 190_000_000) WHERE a = 1;" > > > > > > > Here, the chr(1)'s special value makes the string longer. So, the > > failure is not really about row size, it is about JSON escaping > > expansion. escape_json* turns every byte below 0x20 into a six-byte > > \uXXXX sequence. The reproducer's value is 190MB of chr(1), which > > expands to ~1.14GB against the 1073741823-byte StringInfo limit, > > matching the reported error exactly. As per my understanding, this > > cannot be fixed by enlarging a buffer. The result has to become a json > > Datum, and varlena is capped at the same 1GB, so a tuple whose JSON > > form exceeds 1GB cannot be stored in a json column at all. It is a > > representational ceiling. So, the possible options to fix this are: > > (a) Don't LOG any column_value_size greater than 2K. (b) Truncate > > bigger column values(>2K) to 2K or even shorter number 64 bytes. (c) > > LOG only index/RI columns but what should we do for large columns on > > the subscriber side or if RI is set to FULL? > > > > If we choose option (b), then a truncated value in a queryable table > > silently produces wrong results: WHERE tuple->>'col' = ..., joins and > > old-vs-new diffs all answer incorrectly with no way for the consumer > > to tell. In a server log a human can see the value is clipped; in SQL > > nothing can. I think option (c) is not workable because of reasons > > mentioned along with option (c). I would prefer to go with option (a) > > aka store a value verbatim when it is below a cap; above the cap, > > record no value plus an explicit marker. Every value present in the > > CLT is then byte-exact, and every value that could not be stored says > > so. > > > > If we would like to go with option (a) then the next thing to decide > > is how to display/show/represent omitted columns. > > > > (1) The first option is to mark the omission on the value itself, e.g. > > "b": {"omitted": true, "length": 190000000}. This is self-locating > > (the same column can be oversized in one payload and fine in another), > > it cannot collide with a user column name (any scheme that adds a > > metadata key alongside column-name keys is unsafe, since a real column > > can be named anything), it travels with the row, and it fails loudly > > rather than looking like a genuine NULL. (2) The other option is to > > use a separate column like omitted_columns. There are three > > independent JSON payloads per row — replica_identity, remote_tuple, > > and local_conflicts (an array whose elements each contain a tuple). > > That's what creates the problems because this option can't say which > > payload. (3) Yet another option is to simply LOG the omitted column > > names but may be difficult for users to find out. Or maybe some > > combination of these options. > > > > The one benefit of option (2) is cheap filtering. WHERE > > omitted_columns IS NOT NULL to find incomplete rows is trivial, > > whereas with in-value markers you'd need a JSON search. So, I suggest > > to go with a mix of option 1 and 2, put the detail on the value — "b": > > {"omitted": true, "length": 190000000} and add a plain bool (say > > has_omitted_values) purely as a filter. > > Yes this makes sense. Here is the new version of patches which usage > v71_1-0001 as base patch and v71-0002 is rebased on top of that and > the the ommitting the large column patch is implemented as a top up > patch, which is slightly modified version(comments changes) of what > Amit shared offlist. In this, we use a 64kB max size for each column > and if a column's value exceeds this size, it will be omitted, as > suggested in option a. With a max value size of 64kB, even a table > with the maximum > supported number of columns (MaxHeapAttributeNumber = 1600) undergoing > the worst-case 6x JSON escape expansion will produce a serialized JSON > datum of at most ~600MB, safely below PostgreSQL's 1GB varlena limit. > Furthermore, across all three JSON columns in ConflictLogSchema > (replica_identity, remote_tuple, local_conflicts), the total in-memory > formed row size is bounded by ~1.8GB, ensuring it stays within > PostgreSQL's 2GB row size limit (assuming local_conflicts contains at > most one local tuple, which holds for all resolvable conflicts today). >
A few comments: 1) + * bounded by ~1.8GB, ensuring it stays within PostgreSQL's 2GB row size The comments mention a “2GB row limit,” but it is not clear to me where this limit is derived from. The doc does not mention such a limit: https://www.postgresql.org/docs/19/limits.html 2) Also going by worst case scenario as you mentioned above, the total in-memory row size could go upto 1.8GB; won't that fail in heap_form_tuple() in insert_conflict_log_tuple(). heap_form_tuple() does palloc0 for total length which will MaxAllocSize limit of 1GB in such a case. 3) +#include "utils/jsonfuncs.h" It compiles without this. 4) + <type>json</type> + <type>JSON</type> We can make the case same for all. There are a few such references. I think it is 'json' if we want to say it as pg-datatype and 'JSON' as a generic term. 5) + <row> + <entry><literal>has_omitted_values</literal></entry> + <entry><type>boolean</type></entry> + <entry>Indicates whether any value in this row was too large to be + recorded, and was replaced by a marker. See below.</entry> + </row> Suggestion: Indicates whether any value in this row was too large to be recorded and was replaced by an <literal>omitted</literal> marker containing its length in bytes. thanks Shveta
