date:20100222

Re: [HACKERS] [COMMITTERS] pgsql: Oops, don't forget to rewind the directory before scanning it to

2010-02-22 Thread Andres Freund

On Monday 22 February 2010 04:58:29 Tom Lane wrote:
 Fujii Masao masao.fu...@gmail.com writes:
  +   Free(xldir);
  
  s/Free/FreeDir ?
 
 Yeah, that too.  I think it's all good now, but please test.
At least I havent seen any of the problems pointed out in fsync fun.

 One thing I was wondering was whether the stat-wrong-file problem
 could explain the buildfarm failures that we thought were evidence
 of a portability issue.  I was tempted to re-enable the #ifdef NOTYET
 code, but didn't want to pull that trigger while there were other
 problems outstanding.
I unfortunately dont think so. The mkdir above should not have been bothered 
by the stat issue - especially as it was only introduced by the commit to 
disable the fsyncing.

 I also think it should scan the todir not the fromdir, just on
 general principles to avoid any possibility of race conditions.
That one actually was my idea/code and intentional with the idea to error out 
at one more place if anything goes wrong in the copy loop - I could not think 
of any race issues created by that which were not there before.
On the other hand its unlikely to catch anything so I dont mind.

Andres

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] A thought on Index Organized Tables

2010-02-22 Thread Gokulakannan Somasundaram

Heikki,
 I had a quick look at the discussion on visibility map design. The
main differences as i see it are
a) IOT has both table and index in one structure. So no duplication of data
b) With visibility maps, we have three structures a) Table b) Index c)
Visibility map. So the disk footprint of the same data will be higher in
postgres ( 2x + size of the visibility map).
c) More than that, inserts and updates will incur two extra random i/os
every time. - one for updating the table and one for updating the visibility
map.

Are we going to ignore these differences?

Thanks,
Gokul.

On Mon, Feb 22, 2010 at 12:21 PM, Heikki Linnakangas 
heikki.linnakan...@enterprisedb.com wrote:

 Gokulakannan Somasundaram wrote:
  Hi,
  As you all know, Index Organized tables are a way by which we can
  automatically cluster the data based on the primary key. While i was
  thinking about an implementation for postgres, it looks like an
 impossible
  with the current ideologies. In an IOT, if a record gets updated, we need
 to
  mark the old row as deleted immediately, as we do with any other table.
 But
  since Postgres supports user defined data types and if they happen to be
 a
  broken data type, then we have an unstable IOT.(as there is no guarantee,
 we
  might hit the same record)
  This was the reason for which, the proposal on creating  indexes with
  snapshot was rejected.
  May i get a little clarification on this issue? Will we be supporting
  the IOT feature in postgres in future?

 What seems like the best path to achieve the kind of performance
 benefits that IOTs offer is allowing index-only-scans using the
 visibility map. I worked on that last summer, but unfortunately didn't
 have the time to finish anything.

 --
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

Re: [HACKERS] scheduler in core

2010-02-22 Thread Simon Riggs

On Sun, 2010-02-21 at 20:46 +0100, Dimitri Fontaine wrote:
 Simon Riggs si...@2ndquadrant.com writes:
  There is currently no way to run a separate daemon process that runs
  user code as part of Postgres, so that the startup code gets run
  immediately we startup, re-run if we crash and shut down cleanly when
  the server does. If there were some way to run arbitrary code in a
  daemon using an extensibility API then we wouldn't ever get any requests
  for the scheduler, cos you could write it yourself without troubling
  anybody here.
 
 Please do include the Skytools / PGQ ticker as one use case in the
 design discussion, and pgbouncer too. Having user daemons as part as the
 PostgreSQL extensibility would be awesome indeed!
 
 Bonus point if you build them with PGXS and install them from SQL, so
 that the current extension packaging design applies.
 
 I guess we can say that the archive and restore command are precursors
 of managed user daemons, or say, integrated processes. So adding them
 to the use cases to cover would make sense.

Yes, I think so. Rough design...

integrated_user_processes = 'x, y, z'

would run x(), y() and z() in their own processes. These would execute
after startup, or at consistent point in recovery. The code for these
would come from preload_libraries etc.

They would not block smart shutdown, though their shudown sequence might
delay it. User code would be executed last at startup and first thing at
shutdown.

API would be user_process_startup(), user_process_shutdown().

-- 
 Simon Riggs   www.2ndQuadrant.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] A thought on Index Organized Tables

2010-02-22 Thread Greg Stark

On Mon, Feb 22, 2010 at 8:18 AM, Gokulakannan Somasundaram
gokul...@gmail.com wrote:
 a) IOT has both table and index in one structure. So no duplication of data
 b) With visibility maps, we have three structures a) Table b) Index c)
 Visibility map. So the disk footprint of the same data will be higher in
 postgres ( 2x + size of the visibility map).

These sound like the same point to me. I don't think we're concerned
with footprint -- only with how much of that footprint actually needs
to be scanned. So if we have a solution allowing the scan to only need
to look at the index then the extra footprint of the table doesn't
cost anything at run-time. And the visibility map is very small.


I think you would be better off looking for incremental improvements
rather than major architectural changes like having no heap for a
table. There are so many design decisions hinged on having a heap that
it would be impractical to rethink them all.

-- 
greg

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] A thought on Index Organized Tables

2010-02-22 Thread Gokulakannan Somasundaram

Forgot to include the group...

On Mon, Feb 22, 2010 at 4:31 PM, Gokulakannan Somasundaram 
gokul...@gmail.com wrote:



 These sound like the same point to me. I don't think we're concerned
 with footprint -- only with how much of that footprint actually needs
 to be scanned. So if we have a solution allowing the scan to only need
 to look at the index then the extra footprint of the table doesn't
 cost anything at run-time. And the visibility map is very small.


 Yep.. They are one and the same...
 Just wanted a clarification on the design goals going forward.

 Thanks,
 Gokul.

Re: [HACKERS] Streaming replication and pg_xlogfile_name()

2010-02-22 Thread Heikki Linnakangas

Fujii Masao wrote:
 On Thu, Jan 28, 2010 at 5:28 PM, Heikki Linnakangas
 heikki.linnakan...@enterprisedb.com wrote:
 How about extending the format of the string returned by
 pg_last_xlog_receive/replay_location() to include the timeline ID? When
 it currently returns e.g '6/200016C', it could return '1/6/200016C',
 where 1 is the timeline ID. Then just teach pg_xlogfile_name[_offset]()
 to accept that format as well.
 
 Sounds good. The attached patch does so. Also the code is available
 in the 'replication' branch in my git repository.

 --- 5866,5882 
   /* use volatile pointer to prevent code rearrangement */
   volatile XLogCtlData *xlogctl = XLogCtl;
   
 ! /*
 !  * initialize shared replayEndRecPtr, 
 recoveryLastRecPtr and
 !  * recoveryLastTLI. Actually, the latter two variables 
 don't need to
 !  * be initialized here since they are expected to be 
 updated at least
 !  * once until read only connections will have read 
 them. But just in
 !  * case.
 !  */
   SpinLockAcquire(xlogctl-info_lck);
   xlogctl-replayEndRecPtr = ReadRecPtr;
   xlogctl-recoveryLastRecPtr = ReadRecPtr;
 + xlogctl-recoveryLastTLI = curFileTLI;
   SpinLockRelease(xlogctl-info_lck);
   
   InRedo = true;

Thinking about this again, I'm not sure this is a good idea. Using
curFileTLI makes sense if you're going to call pg_xlogfile_name() and
would expect it to return the filename of the file containing the WAL
record being replayed. But in other contexts, it seems strange for
pg_last_replay_timeline() to return the TLI of the first record in the
file, rather than the actual record replayed.

I don't have any better ideas, though.

-- 
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Time travel on the buildfarm

2010-02-22 Thread Andrew Dunstan




Tom Lane wrote:

It is currently 22:21:59 EST here.  At 21:50 I committed a fix to
copydir.c that cleaned up a couple of thinkos by Greg, including
a misspelling that had been making all the builds fail for several
hours.  I went to see if any of the buildfarm had gone green yet,
and indeed half a dozen members had --- but they are all claiming
to be using snapshots between 00:44:53 and 00:51:20 old.  Unless
the cvs mirror can see into the future, that's a lie.  I'm not
sure how the ages on the buildfarm show_status.pl page are generated,
but I'm betting somebody's local clock is off.


  


Thanks for the report.

The buildfarm server is currently reporting a time 25 minutes too fast. 
I have previously discussed with the server administrators the necessity 
of running an NTP daemon on the machine, but I will do so again.


This is one area where git (or almost anything but CVS) would server us 
better. With git or even Subversion, we would report the tree version as 
well as the timestamp.


cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] A thought on Index Organized Tables

2010-02-22 Thread Alvaro Herrera

Greg Stark escribió:
 On Mon, Feb 22, 2010 at 8:18 AM, Gokulakannan Somasundaram
 gokul...@gmail.com wrote:
  a) IOT has both table and index in one structure. So no duplication of data
  b) With visibility maps, we have three structures a) Table b) Index c)
  Visibility map. So the disk footprint of the same data will be higher in
  postgres ( 2x + size of the visibility map).
 
 These sound like the same point to me. I don't think we're concerned
 with footprint -- only with how much of that footprint actually needs
 to be scanned. So if we have a solution allowing the scan to only need
 to look at the index then the extra footprint of the table doesn't
 cost anything at run-time. And the visibility map is very small.

Moreover, the visibility map is already there.

-- 
Alvaro Herrerahttp://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Plans for 9.1, Grouping Sets, disabling multiqueries, contrib module for string, plpgpsm, preload dictionaries

2010-02-22 Thread Alvaro Herrera

Pavel Stehule escribió:
 Hello,
 
 * Now I am working on migration of plpgpsm to plpgsql 9.0 base. I hope
 so I understand SQL/PSM well so I am able to write production quality
 implementation. If you like, I can integrate it to core. It can share
 about 40-50% code with plpgpsm. The behave of plpgpsm is same as
 plpgsql - without some plpgsql's historical issues (about FOUND, about
 NULL and record type). SQL/PSM is litlle bit richer language. Now we
 have not any wide used runtime so I don't thinking about rewriting.
 Maybe we can rewrite these PL language for parrot or lua runtime in
 future. But this step isn't necessary - people hasn't performance
 problems with PL based on PL runtime.

How do you plan to go about code sharing?  I'm wondering if we're going
to have src/pl/common or something like that.  Since there's a huge
amount of common code it doesn't make any sense to keep it duplicate.

Also, AFAIR that was the main rejection point for the plpgpsm patch last
time around, so it would be good to discuss this thoroughly.

-- 
Alvaro Herrerahttp://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] ALTER TABLE documentation

2010-02-22 Thread Robert Haas

I just noticed that the current ALTER TABLE documentation doesn't do a
very good job differentiating between commands that affect the whole
table and commands that only affect specific columns.

http://developer.postgresql.org/pgdocs/postgres/sql-altertable.html

That didn't matter as much before, but now that we have both
table-level options and attribute-level options, there are two
sections in there that look all-too-similar...  suggestions?

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [COMMITTERS] Re: pgsql: Speed up CREATE DATABASE by deferring the fsyncs until after

2010-02-22 Thread Tom Lane

Greg Stark gsst...@mit.edu writes:
 On Mon, Feb 22, 2010 at 2:54 AM, Tom Lane t...@sss.pgh.pa.us wrote:
 I also think it should scan the todir not the fromdir, just on
 general principles to avoid any possibility of race conditions.

 I had concluded that scanning the original directory was odd but
 better because it served to double-check that all the original files
 actually made it and also because if there were any unrelated files
 present there was no need to fsync them.

Well, just for the record: if that was actually intentional then both of
you erred seriously by not including a comment that explained that the
coding was intentional (and giving the reasoning).  Any reader of the
code would have assumed that it was a copy-and-paste error, as I did.

 But I agree it's odd and not
 very general for copydir if we decide to use it elsewhere other than
 create database.

Yeah, to me it seems more likely to cause problems down the road than
to catch anything.  If the system is missing directory entries during
ReadDir then we have problems far beyond what copydir can deal with.
The point of the fsync loop is just to force the copy results down to
the platter, not to cross-check that the source directory isn't
changing.  (And, what's more, I don't believe that the source directory
can't change during CREATE DATABASE.  Consider delayed cleanup of
deleted relations during checkpoints.)

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Plans for 9.1, Grouping Sets, disabling multiqueries, contrib module for string, plpgpsm, preload dictionaries

2010-02-22 Thread Pavel Stehule

2010/2/22 Alvaro Herrera alvhe...@commandprompt.com:
Pavel Stehule escribió:
Hello,

* Now I am working on migration of plpgpsm to plpgsql 9.0 base. I hope
so I understand SQL/PSM well so I am able to write production quality
implementation. If you like, I can integrate it to core. It can share
about 40-50% code with plpgpsm. The behave of plpgpsm is same as
plpgsql - without some plpgsql's historical issues (about FOUND, about
NULL and record type). SQL/PSM is litlle bit richer language. Now we
have not any wide used runtime so I don't thinking about rewriting.
Maybe we can rewrite these PL language for parrot or lua runtime in
future. But this step isn't necessary - people hasn't performance
problems with PL based on PL runtime.

How do you plan to go about code sharing? I'm wondering if we're going
to have src/pl/common or something like that. Since there's a huge
amount of common code it doesn't make any sense to keep it duplicate.

sure - a there a a few parts - simple query diagnostic, namespace
support. Now I would to migrate to 9.0 and have initial version. This
version and regress tests can be used as etalon. Next stage can be
code cleaning and migration to shared code.

Pavel

Also, AFAIR that was the main rejection point for the plpgpsm patch last
time around, so it would be good to discuss this thoroughly.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

90 matches

Mail list logo