Re: [HACKERS] Parallell Optimizer

2013-06-13 Thread Ants Aasma
On Thu, Jun 13, 2013 at 11:24 AM, Simon Riggs wrote: > That idea is not dependent upon CSNs. > > It is an option for us to implement snapshot synchronisation now, we > just haven't done it yet. > > I'm currently working on exporting/importing snapshots on standbys, > which is a precursor to that i

Re: [HACKERS] Parallell Optimizer

2013-06-13 Thread Ants Aasma
On Thu, Jun 13, 2013 at 11:39 AM, Hannu Krosing wrote: >>> Coincidentally getting cluster wide consistent snapshots and delaying >>> until some specific point in commit ordering is almost trivial to >>> solve with Commit Sequence Number based snapshot scheme that I >>> proposed. >> Can you elabora

Re: [HACKERS] Parallell Optimizer

2013-06-13 Thread Hannu Krosing
On 06/13/2013 02:22 AM, Tatsuo Ishii wrote: >> On Jun 12, 2013 2:02 AM, "Tatsuo Ishii" wrote: >>> No, I'm not talking about conflict resolution. >>> >>> From >>> http://www.cs.cmu.edu/~natassa/courses/15-823/F02/papers/replication.pdf: >>> -- >>> Eager

Re: [HACKERS] Parallell Optimizer

2013-06-13 Thread Simon Riggs
On 13 June 2013 02:18, Stephen Frost wrote: > * Ants Aasma (a...@cybertec.at) wrote: >> In a cluster setting you take the CSN value on the master, then before >> starting execution on a standby you wait until that the standby has >> replayed enough WAL to reach the CSN point read from the master a

Re: [HACKERS] Parallell Optimizer

2013-06-13 Thread Simon Riggs
On 11 June 2013 15:59, Tatsuo Ishii wrote: > I wonder why "true" synchronous replication nor "eager replication" > are not in the developer TODO list. If we want them in the future, > they should be on it. I think you still need to explain what "true" synchronous replication is. IMHO eager repl

Re: [HACKERS] Parallell Optimizer

2013-06-12 Thread Stephen Frost
Ants, * Ants Aasma (ants.aa...@eesti.ee) wrote: > On Jun 13, 2013 4:18 AM, "Stephen Frost" wrote: > > * Ants Aasma (a...@cybertec.at) wrote: > > > In a cluster setting you take the CSN value on the master, then before > > > starting execution on a standby you wait until that the standby has > > >

Re: [HACKERS] Parallell Optimizer

2013-06-12 Thread Ants Aasma
On Jun 13, 2013 4:18 AM, "Stephen Frost" wrote: > * Ants Aasma (a...@cybertec.at) wrote: > > In a cluster setting you take the CSN value on the master, then before > > starting execution on a standby you wait until that the standby has > > replayed enough WAL to reach the CSN point read from the m

Re: [HACKERS] Parallell Optimizer

2013-06-12 Thread Stephen Frost
* Ants Aasma (a...@cybertec.at) wrote: > In a cluster setting you take the CSN value on the master, then before > starting execution on a standby you wait until that the standby has > replayed enough WAL to reach the CSN point read from the master and > you know that after that everything that the

Re: [HACKERS] Parallell Optimizer

2013-06-12 Thread Tatsuo Ishii
> On Thu, Jun 13, 2013 at 3:22 AM, Tatsuo Ishii wrote: >>> Parallel query execution doesn't require commits to synchronize all >>> nodes. Parallel execution needs consistent snapshots across all nodes. >>> In effect this means that nodes need to agree on commit ordering, >>> either total order or

Re: [HACKERS] Parallell Optimizer

2013-06-12 Thread Ants Aasma
On Thu, Jun 13, 2013 at 3:22 AM, Tatsuo Ishii wrote: >> Parallel query execution doesn't require commits to synchronize all >> nodes. Parallel execution needs consistent snapshots across all nodes. >> In effect this means that nodes need to agree on commit ordering, >> either total order or a part

Re: [HACKERS] Parallell Optimizer

2013-06-12 Thread Tatsuo Ishii
> On Jun 12, 2013 2:02 AM, "Tatsuo Ishii" wrote: >> No, I'm not talking about conflict resolution. >> >> From >> http://www.cs.cmu.edu/~natassa/courses/15-823/F02/papers/replication.pdf: >> -- >> Eager or Lazy Replication? >> Eager replication: >> kee

Re: [HACKERS] Parallell Optimizer

2013-06-12 Thread Ants Aasma
On Jun 12, 2013 2:02 AM, "Tatsuo Ishii" wrote: > No, I'm not talking about conflict resolution. > > From http://www.cs.cmu.edu/~natassa/courses/15-823/F02/papers/replication.pdf: > -- > Eager or Lazy Replication? > Eager replication: > keep all replica

Re: [HACKERS] Parallell Optimizer

2013-06-12 Thread Yeb Havinga
On 2013-06-07 19:09, Fred&Dani&Pandora&Aquiles wrote: I asked a while ago in this group about the possibility to implement a parallel planner in a multithread way, and the replies were that the proposed approach couldn't be implemented, because the postgres is not thread-safe. With the new fea

Re: [HACKERS] Parallell Optimizer

2013-06-11 Thread Craig Ringer
On 06/12/2013 07:01 AM, Tatsuo Ishii wrote: Please explain what you mean by the word "true" used here. >>> In another word, "eager replication". >> Do you mean something along these lines : >> >> "Most synchronous or eager replication solutions do conflict prevention, >> while asynchronous sol

Re: [HACKERS] Parallell Optimizer

2013-06-11 Thread Tatsuo Ishii
>> No, I'm not talking about conflict resolution. >> >> From >> http://www.cs.cmu.edu/~natassa/courses/15-823/F02/papers/replication.pdf: >> -- >> Eager or Lazy Replication? >> Eager replication: >> keep all replicas synchronized by updating all >> re

Re: [HACKERS] Parallell Optimizer

2013-06-11 Thread Hannu Krosing
On 06/12/2013 01:01 AM, Tatsuo Ishii wrote: Please explain what you mean by the word "true" used here. >>> In another word, "eager replication". >> Do you mean something along these lines : >> >> "Most synchronous or eager replication solutions do conflict prevention, >> while asynchronous sol

Re: [HACKERS] Parallell Optimizer

2013-06-11 Thread Tatsuo Ishii
>>> Please explain what you mean by the word "true" used here. >> In another word, "eager replication". > Do you mean something along these lines : > > "Most synchronous or eager replication solutions do conflict prevention, > while asynchronous solutions have to do conflict resolution. For instan

Re: [HACKERS] Parallell Optimizer

2013-06-11 Thread Hannu Krosing
On 06/11/2013 04:53 PM, Tatsuo Ishii wrote: >> On 11 June 2013 01:45, Tatsuo Ishii wrote: On Sat, Jun 8, 2013 at 5:04 AM, Simon Riggs wrote: > On 7 June 2013 20:23, Tom Lane wrote: > >> As for other databases, I suspect that ones that have parallel execution >> are prob

Re: [HACKERS] Parallell Optimizer

2013-06-11 Thread Gavin Flower
On 11/06/13 19:24, Hannu Krosing wrote: On 06/10/2013 10:37 PM, Fred&Dani&Pandora&Aquiles wrote: Hi, >> I asked a while ago in this group about the possibility to implement a >> parallel planner in a multithread way, and the replies were that the >> proposed approach couldn

Re: [HACKERS] Parallell Optimizer

2013-06-11 Thread Jim Nasby
On 6/7/13 2:23 PM, Tom Lane wrote: As for other databases, I suspect that ones that have parallel execution are probably doing it with a thread model not a process model. Oracle 9i was multi-process, not multi-threaded. IIRC it actually had dedicated IO processes too; backends didn't do their

Re: [HACKERS] Parallell Optimizer

2013-06-11 Thread Tatsuo Ishii
> On Tue, Jun 11, 2013 at 9:45 AM, Tatsuo Ishii wrote: > >> > On Sat, Jun 8, 2013 at 5:04 AM, Simon Riggs >> wrote: >> > >> >> On 7 June 2013 20:23, Tom Lane wrote: >> >> >> >> > As for other databases, I suspect that ones that have parallel >> execution >> >> > are probably doing it with a thr

Re: [HACKERS] Parallell Optimizer

2013-06-11 Thread Tatsuo Ishii
> On 11 June 2013 01:45, Tatsuo Ishii wrote: >>> On Sat, Jun 8, 2013 at 5:04 AM, Simon Riggs wrote: >>> On 7 June 2013 20:23, Tom Lane wrote: > As for other databases, I suspect that ones that have parallel execution > are probably doing it with a thread model not a process m

Re: [HACKERS] Parallell Optimizer

2013-06-11 Thread Hannu Krosing
On 06/10/2013 10:37 PM, Fred&Dani&Pandora&Aquiles wrote: > Hi, > > > >> I asked a while ago in this group about the possibility to > implement a > >> parallel planner in a multithread way, and the replies were > that the > >> proposed approach couldn't be implemented, because

Re: [HACKERS] Parallell Optimizer

2013-06-10 Thread Simon Riggs
On 11 June 2013 01:45, Tatsuo Ishii wrote: >> On Sat, Jun 8, 2013 at 5:04 AM, Simon Riggs wrote: >> >>> On 7 June 2013 20:23, Tom Lane wrote: >>> >>> > As for other databases, I suspect that ones that have parallel execution >>> > are probably doing it with a thread model not a process model. >>

Re: [HACKERS] Parallell Optimizer

2013-06-10 Thread Michael Paquier
On Tue, Jun 11, 2013 at 9:45 AM, Tatsuo Ishii wrote: > > On Sat, Jun 8, 2013 at 5:04 AM, Simon Riggs > wrote: > > > >> On 7 June 2013 20:23, Tom Lane wrote: > >> > >> > As for other databases, I suspect that ones that have parallel > execution > >> > are probably doing it with a thread model no

Re: [HACKERS] Parallell Optimizer

2013-06-10 Thread Tatsuo Ishii
> On Sat, Jun 8, 2013 at 5:04 AM, Simon Riggs wrote: > >> On 7 June 2013 20:23, Tom Lane wrote: >> >> > As for other databases, I suspect that ones that have parallel execution >> > are probably doing it with a thread model not a process model. >> >> Separate processes are more common because it

Re: [HACKERS] Parallell Optimizer

2013-06-10 Thread Michael Paquier
On Sat, Jun 8, 2013 at 5:04 AM, Simon Riggs wrote: > On 7 June 2013 20:23, Tom Lane wrote: > > > As for other databases, I suspect that ones that have parallel execution > > are probably doing it with a thread model not a process model. > > Separate processes are more common because it covers th

Re: [HACKERS] Parallell Optimizer

2013-06-10 Thread
Hi, > >> I asked a while ago in this group about the possibility to implement a > >> parallel planner in a multithread way, and the replies were that the > >> proposed approach couldn't be implemented, because the postgres is not > >> thread-safe. With the new feature Background Worker Processes

Re: [HACKERS] Parallell Optimizer

2013-06-07 Thread Simon Riggs
On 7 June 2013 20:23, Tom Lane wrote: > As for other databases, I suspect that ones that have parallel execution > are probably doing it with a thread model not a process model. Separate processes are more common because it covers the general case where query execution is spread across multiple

Re: [HACKERS] Parallell Optimizer

2013-06-07 Thread Robert Haas
On Fri, Jun 7, 2013 at 3:23 PM, Tom Lane wrote: > Robert Haas writes: >> On Fri, Jun 7, 2013 at 2:27 PM, Tom Lane wrote: >>> I don't think that bgworkers as currently implemented make this any more >>> practical than it was before. The communication overhead with a >>> separate process would sw

Re: [HACKERS] Parallell Optimizer

2013-06-07 Thread Tom Lane
Robert Haas writes: > On Fri, Jun 7, 2013 at 2:27 PM, Tom Lane wrote: >> I don't think that bgworkers as currently implemented make this any more >> practical than it was before. The communication overhead with a >> separate process would swamp any benefit in most cases. > I'm baffled by your s

Re: [HACKERS] Parallell Optimizer

2013-06-07 Thread Robert Haas
On Fri, Jun 7, 2013 at 2:27 PM, Tom Lane wrote: > "Fred&Dani&Pandora&Aquiles" writes: >> I asked a while ago in this group about the possibility to implement a >> parallel planner in a multithread way, and the replies were that the >> proposed approach couldn't be implemented, because the postgr

Re: [HACKERS] Parallell Optimizer

2013-06-07 Thread Tom Lane
"Fred&Dani&Pandora&Aquiles" writes: > I asked a while ago in this group about the possibility to implement a > parallel planner in a multithread way, and the replies were that the > proposed approach couldn't be implemented, because the postgres is not > thread-safe. With the new feature Backgrou

[HACKERS] Parallell Optimizer

2013-06-07 Thread
I asked a while ago in this group about the possibility to implement a parallel planner in a multithread way, and the replies were that the proposed approach couldn't be implemented, because the postgres is not thread-safe. With the new feature Background Worker Processes, such implementation woul