Re: COPY threads

2018-10-11 Thread Ravi Krishna
> > Well the result is not what I expected. > > this is the sql I used > > copy TEST.TABLE1 from '/var/lib/postgresql/data/rkrishna/copytesting/xaa' > with delimiter '|' NULL as '' CSV HEADER; > > From another session > > copy TEST.TABLE1 from

Re: COPY threads

2018-10-11 Thread Ravi Krishna
>>> >> Thank you. Let me test it and see the benefit. We have a use case for this. > Well the result is not what I expected. this is the sql I used copy TEST.TABLE1 from '/var/lib/postgresql/data/rkrishna/copytesting/xaa' with delimiter '|' NULL as '' CSV HEADER; From another session copy

Re: COPY threads

2018-10-11 Thread Thomas Kellerer
Rob Sargent schrieb am 10.10.2018 um 00:45:> Can anyone here tell me whether or not the CopyManager facility in JDBC via org.postgresql:postgresql:42.1.4 is internally multithreaded? Running on CentOS 7 (all participants), java8, postgres 10.5 An alternative to creating your own

Re: COPY threads

2018-10-11 Thread Peter J. Holzer
On 2018-10-10 17:19:50 -0400, Ravi Krishna wrote: > > On Oct 10, 2018, at 17:18 , Andres Freund wrote: > > On October 10, 2018 2:15:19 PM PDT, Ravi Krishna wrote: > >> If I have a large file with say 400 million rows, can I first split it > >> into 10 files of 40 million rows each and then fire

Re: COPY threads

2018-10-10 Thread Ravi Krishna
Thank you. Let me test it and see the benefit. We have a use case for this. > On Oct 10, 2018, at 17:18 , Andres Freund wrote: > > > > On October 10, 2018 2:15:19 PM PDT, Ravi Krishna wrote: >>> >>> pg_restore doesn't take locks on the table for the COPY, it does so >>> because creating

Re: COPY threads

2018-10-10 Thread Andres Freund
On October 10, 2018 2:15:19 PM PDT, Ravi Krishna wrote: >> >> pg_restore doesn't take locks on the table for the COPY, it does so >> because creating the table takes an exclusive lock. > > >Interesting. I seem to recollect reading here that I can't have >concurrent COPY on the same table

Re: COPY threads

2018-10-10 Thread Ravi Krishna
> > pg_restore doesn't take locks on the table for the COPY, it does so > because creating the table takes an exclusive lock. Interesting. I seem to recollect reading here that I can't have concurrent COPY on the same table because of the lock. To give an example: If I have a large file with

Re: COPY threads

2018-10-10 Thread Andres Freund
Hi, On 2018-10-10 17:02:59 -0400, Ravi Krishna wrote: > I am talking about pg_restore which is COPY FROM, which takes exclusive lock > on the table > while the data is loaded into the table. pg_restore doesn't take locks on the table for the COPY, it does so because creating the table takes an

Re: COPY threads

2018-10-10 Thread Ravi Krishna
> > No, why would that seem to be the case? If it did so, then you could > not run pg_dump to dump data while regular activity was going on. Not sure. In fact I am now confused. I am talking about pg_restore which is COPY FROM, which takes exclusive lock on the table while the data is loaded

Re: COPY threads

2018-10-10 Thread Christopher Browne
On Wed, 10 Oct 2018 at 16:22, Ravi Krishna wrote: > You obviously are referring to multiple connections running COPY on different > tables, right? Like what pg_restore does with -j option. > Doesn't copy take an exclusive lock on the table which makes it incompatible > with parallelization.

Re: COPY threads

2018-10-10 Thread Ravi Krishna
> > You obviously can just copy the data into postgres over multiple > connections if you need to speed COPY up. But that requires splitting up > the data on the clientside. > You obviously are referring to multiple connections running COPY on different tables, right? Like what pg_restore

Re: COPY threads

2018-10-10 Thread Rob Sargent
> On Oct 10, 2018, at 1:24 PM, Andres Freund wrote: > > On 2018-10-10 18:50:02 +0200, Laurenz Albe wrote: >> Rob Sargent wrote: >>> Can anyone here tell me whether or not the CopyManager facility in JDBC >>> via org.postgresql:postgresql:42.1.4 is internally multithreaded? >>> Running on

Re: COPY threads

2018-10-10 Thread Andres Freund
On 2018-10-10 18:50:02 +0200, Laurenz Albe wrote: > Rob Sargent wrote: > > Can anyone here tell me whether or not the CopyManager facility in JDBC > > via org.postgresql:postgresql:42.1.4 is internally multithreaded? > > Running on CentOS 7 (all participants), java8, postgres 10.5 > > It isn't,

Re: COPY threads

2018-10-10 Thread Rob Sargent
> On Oct 10, 2018, at 10:50 AM, Laurenz Albe wrote: > > Rob Sargent wrote: >> Can anyone here tell me whether or not the CopyManager facility in JDBC >> via org.postgresql:postgresql:42.1.4 is internally multithreaded? >> Running on CentOS 7 (all participants), java8, postgres 10.5 > > It

Re: COPY threads

2018-10-10 Thread Laurenz Albe
Rob Sargent wrote: > Can anyone here tell me whether or not the CopyManager facility in JDBC > via org.postgresql:postgresql:42.1.4 is internally multithreaded? > Running on CentOS 7 (all participants), java8, postgres 10.5 It isn't, and there would not be much reason for it to be, as COPY in

COPY threads

2018-10-09 Thread Rob Sargent
Can anyone here tell me whether or not the CopyManager facility in JDBC via org.postgresql:postgresql:42.1.4 is internally multithreaded? Running on CentOS 7 (all participants), java8, postgres 10.5