Re: COPY FROM WITH HEADER skips a tuple every 4 billion tuples

[Tom Lane]

Andres Freund  writes:
> On 2018-05-23 09:04:35 +1200, David Rowley wrote:
>> I thought the output I pasted was clearly showing it not to be the
>> same. 42 vs 43.

> Well, the row-returned counter is obviously wide enough, otherwise
> 42 couldn't be returned. Tom's point, as I understood it, is
> that we obviously have one wide enough counter - why can't we reuse that
> for the one you made wider.  And it doesn't seem entirely trivial to do
> so, so your patch is easier.

Right.  Obviously there was a 64-bit counter someplace, but it wasn't
being used for this purpose.

I think after looking at the code that the cur_lineno counter is
counting input *lines* whereas the other thing counts finished *rows*,
so unifying them would be a bad idea anyway.

regards, tom lane

Re: COPY FROM WITH HEADER skips a tuple every 4 billion tuples

[David Rowley]

On 23 May 2018 at 09:31, Andres Freund  wrote:
>> On 23 May 2018 at 03:55, Tom Lane  wrote:
>> > Hm, so why is the correct rowcount returned --- are we running
>> > a separate counter for that purpose, and if so why?
>>
>> I thought the output I pasted was clearly showing it not to be the
>> same. 42 vs 43.
>>
>> Did I misunderstand you?
>
> Well, the row-returned counter is obviously wide enough, otherwise
> 42 couldn't be returned. Tom's point, as I understood it, is
> that we obviously have one wide enough counter - why can't we reuse that
> for the one you made wider.  And it doesn't seem entirely trivial to do
> so, so your patch is easier.

*moment of realisation*

Oh, this makes sense now.

They can't be the same. One tracks the line number in the COPY FROM
input, the other tracks the number of rows inserted. You'd only have
to add a BEFORE INSERT ROW trigger which blocks some rows to
understand why they need to be separate.

-- 
 David Rowley   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: COPY FROM WITH HEADER skips a tuple every 4 billion tuples

[Andres Freund]

On 2018-05-23 09:04:35 +1200, David Rowley wrote:
> Thanks for pushing.
> 
> On 23 May 2018 at 03:55, Tom Lane  wrote:
> > Hm, so why is the correct rowcount returned --- are we running
> > a separate counter for that purpose, and if so why?
> 
> I thought the output I pasted was clearly showing it not to be the
> same. 42 vs 43.
> 
> Did I misunderstand you?

Well, the row-returned counter is obviously wide enough, otherwise
42 couldn't be returned. Tom's point, as I understood it, is
that we obviously have one wide enough counter - why can't we reuse that
for the one you made wider.  And it doesn't seem entirely trivial to do
so, so your patch is easier.

Greetings,

Andres Freund

Re: COPY FROM WITH HEADER skips a tuple every 4 billion tuples

[David Rowley]

On 23 May 2018 at 09:16, Vik Fearing  wrote:
> I think Tom was wondering why it isn't showing 5032703.

You'll need to explain that one. The number just looks like nonsense to me.

-- 
 David Rowley   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: COPY FROM WITH HEADER skips a tuple every 4 billion tuples

[Vik Fearing]

On 22/05/18 23:04, David Rowley wrote:
> Thanks for pushing.
> 
> On 23 May 2018 at 03:55, Tom Lane  wrote:
>> Hm, so why is the correct rowcount returned --- are we running
>> a separate counter for that purpose, and if so why?
> 
> I thought the output I pasted was clearly showing it not to be the
> same. 42 vs 43.
> 
> Did I misunderstand you?

I think Tom was wondering why it isn't showing 5032703.
-- 
Vik Fearing  +33 6 46 75 15 36
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

Re: COPY FROM WITH HEADER skips a tuple every 4 billion tuples

[David Rowley]

Thanks for pushing.

On 23 May 2018 at 03:55, Tom Lane  wrote:
> Hm, so why is the correct rowcount returned --- are we running
> a separate counter for that purpose, and if so why?

I thought the output I pasted was clearly showing it not to be the
same. 42 vs 43.

Did I misunderstand you?

-- 
 David Rowley   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: COPY FROM WITH HEADER skips a tuple every 4 billion tuples

[Tom Lane]

Andres Freund  writes:
> On 2018-05-22 11:55:26 -0400, Tom Lane wrote:
>> Hm, so why is the correct rowcount returned --- are we running
>> a separate counter for that purpose, and if so why?

> Yes, it's a local counter in CopyFrom/CopyTo.  It's probably not
> entirely trivial to unify the two. The batching etc makes us modify
> cur_lineno in a bit weird ways at times.

OK, we'll just do it like David suggests then.  I haven't checked the
patch in detail yet, but it seemed generally sane if we're just going
to widen the duplicate counter.

regards, tom lane

Re: COPY FROM WITH HEADER skips a tuple every 4 billion tuples

[Andres Freund]

On 2018-05-22 11:55:26 -0400, Tom Lane wrote:
> David Rowley  writes:
> > while it might not look too scary by itself, it gets a bit more so
> > when you learn that the cur_lineno is only 32 bits wide. This will
> > result in skipping a tuple every time the 32-bit variable wraps back
> > around to 0 again.
> 
> Hm, so why is the correct rowcount returned --- are we running
> a separate counter for that purpose, and if so why?

Yes, it's a local counter in CopyFrom/CopyTo.  It's probably not
entirely trivial to unify the two. The batching etc makes us modify
cur_lineno in a bit weird ways at times.  It's noteworthy that the
comment for cur_lineno says: /* line number for error messages */

Greetings,

Andres Freund

Re: COPY FROM WITH HEADER skips a tuple every 4 billion tuples

[Tom Lane]

David Rowley  writes:
> while it might not look too scary by itself, it gets a bit more so
> when you learn that the cur_lineno is only 32 bits wide. This will
> result in skipping a tuple every time the 32-bit variable wraps back
> around to 0 again.

Hm, so why is the correct rowcount returned --- are we running
a separate counter for that purpose, and if so why?

regards, tom lane