Re: [HACKERS] Logical tape pause/resume

2016-12-28 Thread Peter Geoghegan
On Wed, Dec 21, 2016 at 4:25 AM, Heikki Linnakangas wrote: > In the meanwhile, Robert committed the cap on the number of tapes. Since > that's in, I'm not sure if the pause/resume part of this is worthwhile. > Maybe it is. I rebased my parallel tuplesort patch on top of what you

Re: [HACKERS] Logical tape pause/resume

2016-12-21 Thread Heikki Linnakangas
On 12/21/2016 03:22 PM, Robert Haas wrote: On Wed, Dec 21, 2016 at 7:25 AM, Heikki Linnakangas wrote: For now, barring objections, I'm going to commit the first patch. It seems like a worthwhile simplification in any case, especially for Peter's Parallel tuplesort patch set.

Re: [HACKERS] Logical tape pause/resume

2016-12-21 Thread Robert Haas
On Wed, Dec 21, 2016 at 7:25 AM, Heikki Linnakangas wrote: > For now, barring objections, I'm going to commit the first patch. It seems > like a worthwhile simplification in any case, especially for Peter's > Parallel tuplesort patch set. Well, it removes more code than it adds.

Re: [HACKERS] Logical tape pause/resume

2016-12-21 Thread Heikki Linnakangas
On 10/12/2016 05:30 PM, Heikki Linnakangas wrote: On 10/10/2016 10:31 PM, Peter Geoghegan wrote: On Sun, Oct 9, 2016 at 11:52 PM, Heikki Linnakangas wrote: Ok, good. I think we're in agreement on doing this, then, even if we don't agree on the justification :-). Agreed. :-)

Re: [HACKERS] Logical tape pause/resume

2016-10-12 Thread Heikki Linnakangas
On 10/10/2016 10:31 PM, Peter Geoghegan wrote: On Sun, Oct 9, 2016 at 11:52 PM, Heikki Linnakangas wrote: Ok, good. I think we're in agreement on doing this, then, even if we don't agree on the justification :-). Agreed. :-) Attached are new patch versions. Rebased them

Re: [HACKERS] Logical tape pause/resume

2016-10-10 Thread Peter Geoghegan
On Sun, Oct 9, 2016 at 11:52 PM, Heikki Linnakangas wrote: > Regardless of the number of tapes, the memory used for the tape buffers, > while building the initial runs, is wasted. It's not entirely wasted when > you have multiple merge passes, because without the buffer you need

Re: [HACKERS] Logical tape pause/resume

2016-10-10 Thread Heikki Linnakangas
Here are freshly rebased versions of these patches. No big changes, but edited some comments slightly. - Heikki >From 324706fb2e15facbc5686e98ee26bbdc5017366a Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas Date: Mon, 10 Oct 2016 10:59:38 +0300 Subject: [PATCH

Re: [HACKERS] Logical tape pause/resume

2016-10-10 Thread Heikki Linnakangas
On 10/09/2016 03:27 AM, Peter Geoghegan wrote: You shouldn't really waste 8% of the budget with low work_mem settings with my cap patch applied, because the new cap never limits the number of tapes. IIRC, the cap of 500 tapes doesn't start to matter until you have about 1GB of work_mem. So, if

Re: [HACKERS] Logical tape pause/resume

2016-10-08 Thread Peter Geoghegan
Apologies for the delayed response to this. On Tue, Oct 4, 2016 at 3:47 AM, Heikki Linnakangas wrote: > One of the patches in Peter Geoghegan's Parallel tuplesort patch set [1] is > to put a cap on the number of tapes to use for the sort. > That amounts to about 8% of the

Re: [HACKERS] Logical tape pause/resume

2016-10-04 Thread Heikki Linnakangas
On 10/04/2016 07:06 PM, Claudio Freire wrote: On Tue, Oct 4, 2016 at 7:47 AM, Heikki Linnakangas wrote: 1. The first patch changes the way we store the logical tapes on disk. Instead of using indirect blocks, HFS style, the blocks form a linked list. Each block has a trailer,

Re: [HACKERS] Logical tape pause/resume

2016-10-04 Thread Claudio Freire
On Tue, Oct 4, 2016 at 7:47 AM, Heikki Linnakangas wrote: > 1. The first patch changes the way we store the logical tapes on disk. > Instead of using indirect blocks, HFS style, the blocks form a linked list. > Each block has a trailer, with the block numbers of the previous and

Re: [HACKERS] Logical tape pause/resume

2016-10-04 Thread Heikki Linnakangas
On 10/04/2016 05:58 PM, Simon Riggs wrote: On 4 October 2016 at 12:47, Heikki Linnakangas wrote: Why not just make each new run start at a block boundary? That way we waste on average BLCKSZ/2 disk space per run, which is negligible but we avoid any need to have code to read

Re: [HACKERS] Logical tape pause/resume

2016-10-04 Thread Simon Riggs
On 4 October 2016 at 12:47, Heikki Linnakangas wrote: >> Why not just make each new run start at a block boundary? >> That way we waste on average BLCKSZ/2 disk space per run, which is >> negligible but we avoid any need to have code to read back in the last >> block. > > > Hmm.

Re: [HACKERS] Logical tape pause/resume

2016-10-04 Thread Heikki Linnakangas
On 10/04/2016 02:09 PM, Simon Riggs wrote: On 4 October 2016 at 11:47, Heikki Linnakangas wrote: When we finish writing an initial run to a tape, we keep the tape buffers around. That's the reason we need to reserve that memory. But there's no fundamental reason we have to do

Re: [HACKERS] Logical tape pause/resume

2016-10-04 Thread Simon Riggs
On 4 October 2016 at 11:47, Heikki Linnakangas wrote: > When we finish writing an initial run to a tape, we keep the tape buffers > around. That's the reason we need to reserve that memory. But there's no > fundamental reason we have to do that. As I suggested in [2], we could

[HACKERS] Logical tape pause/resume

2016-10-04 Thread Heikki Linnakangas
One of the patches in Peter Geoghegan's Parallel tuplesort patch set [1] is to put a cap on the number of tapes to use for the sort. The reason is that each tape consumes 3 * BLCKSZ worth of memory, for the buffers. We decide the max number of tapes upfront, so if we decide that e.g. the max