Re: [GENERAL] effective_cache_size and work_ram

2012-12-11 Thread Pavel Stehule
hello 2012/12/12 a...@hsk.hk : > Hi, > > I have a new server for PostgreSQL 9.2.2 with 8GB physical RAM, I want to > turn the server with the following changes from default: > > max_connections = 100 # default > shared_buffers = 2048MB # change from 24MB to 2048MB as I think 24MB is not

[GENERAL] effective_cache_size and work_ram

2012-12-11 Thread a...@hsk.hk
Hi, I have a new server for PostgreSQL 9.2.2 with 8GB physical RAM, I want to turn the server with the following changes from default: max_connections = 100 # default shared_buffers = 2048MB # change from 24MB to 2048MB as I think 24MB is not enough maintenance_work_mem = 400

Re: [GENERAL] large database

2012-12-11 Thread Adrian Klaver
On 12/11/2012 01:58 PM, Mihai Popa wrote: On Tue, 2012-12-11 at 10:00 -0800, Jeff Janes wrote: On Mon, Dec 10, 2012 at 12:26 PM, Mihai Popa wrote: Hi, I've recently inherited a project that involves importing a large set of Access mdb files into a Postgres or MySQL database. The process is to

Re: [GENERAL] large database

2012-12-11 Thread John R Pierce
On 12/11/2012 1:58 PM, Mihai Popa wrote: 1TB of storage sounds desperately small for loading 300GB of csv files. really? that's good to know; I wouldn't have guessed on many of our databases, the indexes are as large as the tables. -- Sent via pgsql-general mailing list (pgsql-general@pos

Re: [GENERAL] Deleting WAL archives and pg_xlog when there is not a shared drive

2012-12-11 Thread Sergey Konoplev
On Tue, Dec 11, 2012 at 9:37 AM, Eng. AlSamman wrote: > I am trying to implement a high-availability cluster using only two nodes, > without any shared disk storage. > > In my implementation, the primary database has continuous archiving set up > to a directory residing on the second node, where t

Re: [GENERAL] large database

2012-12-11 Thread Mihai Popa
On Tue, 2012-12-11 at 14:28 -0700, David Boreham wrote: > Try SoftLayer instead for physical machines delivered on-demand : > http://www.softlayer.com/dedicated-servers/ > > If you're looking for low cost virtual hosting alternative to Amazon, > try Rackspace. Thank you, I will regards, --

Re: [GENERAL] large database

2012-12-11 Thread Mihai Popa
On Tue, 2012-12-11 at 10:00 -0800, Jeff Janes wrote: > On Mon, Dec 10, 2012 at 12:26 PM, Mihai Popa wrote: > > Hi, > > > > I've recently inherited a project that involves importing a large set of > > Access mdb files into a Postgres or MySQL database. > > The process is to export the mdb's to comm

Re: [GENERAL] NOTIFY/LISTEN on server, asynchronous processing

2012-12-11 Thread rektide
Hiya! PgQ looks like cool technology, but it still violates the #1 thing I'm looking for, which is a postgres-only solution. I don't see any reason why I'd want an external daemon ticking the system over, and I'm hoping perhaps there's some way to coax Postgres itself into doing this asychronou

Re: [GENERAL] large database

2012-12-11 Thread David Boreham
On 12/11/2012 2:03 PM, Mihai Popa wrote: I actually looked at Linode, but Amazon looked more competitive... Checking Linode's web site just now it looks like they have removed physical machines as an option. Try SoftLayer instead for physical machines delivered on-demand : http://www.softlayer.

Re: [GENERAL] Bug in GET STACKED DIAGNOSTICS ?

2012-12-11 Thread Dan S
I got another unexpected behaviour of the call stack by this invocation : select testfunc2(true) from generate_series(1,10); The first call stack is different from the nine folllowing. Shouldn't it be identical to the others ? Best Regards Dan S 2012/12/11 Pavel Stehule > Hello > > 2012/12/1

Re: [GENERAL] large database

2012-12-11 Thread Mihai Popa
On Tue, 2012-12-11 at 09:47 -0700, David Boreham wrote: > Finally, note that there is a middle-ground available between cloud > hosting and outright machine purchase -- providers such as Linode and > SoftLayer will sell physical machines in a way that gives much of the > convenience of cloud ho

Re: [GENERAL] NOTIFY/LISTEN on server, asynchronous processing

2012-12-11 Thread Sergey Konoplev
Hi, On Tue, Dec 11, 2012 at 11:29 AM, rektide wrote: > Does anyone have suggestions for decoupling work done on a server, for > breaking up a task > into multiple asychronous pieces? I believe I've described 1. a viable if > ugly means of > doing so, and 2. limitations in the primary asynchrono

Re: [GENERAL] large database

2012-12-11 Thread Ben Chobot
On Dec 11, 2012, at 2:25 AM, Chris Angelico wrote: > On Tue, Dec 11, 2012 at 7:26 AM, Mihai Popa wrote: >> Second, where should I deploy it? The cloud or a dedicated box? > > Forget cloud. For similar money, you can get dedicated hosting with > much more reliable performance. We've been looking

[GENERAL] PgAdmin question (maybe off topic)

2012-12-11 Thread Edson Richter
Dear list, When using PgAdmin on Windows 7 (64bit), sometimes I can manage indexes, sometimes cannot. Example (can manage indexes normally): Cannot manage indexes at all: Must be something related to the table - but it is not pe

Re: [GENERAL] NOTIFY/LISTEN on server, asynchronous processing

2012-12-11 Thread Kiriakos Georgiou
If I'm understanding you correctly, you want a job queue. This involves polling and retrieving jobs to work on them. The polling can be assisted by listen/notify so workers don't poll unnecessarily. The retrieving has to be done intelligently to avoid concurrency issues. Kiriakos Georgiou htt

[GENERAL] NOTIFY/LISTEN on server, asynchronous processing

2012-12-11 Thread rektide
Hi all, I'm writing seeking help for making asynchronous & decoupled processes run on a Postgres server. Here's my current hairbraned workingis: 1. Create an table "async_process" and attach a trigger after. 2. Establish a dblink to localhost. 3. dblink_send_query("update async_process set counte

Re: [GENERAL] execution plan is wrong, or the query ?

2012-12-11 Thread Alex Burkoff
Understood. Thank you! From: Tom Lane [t...@sss.pgh.pa.us] Sent: Tuesday, December 11, 2012 9:10 AM To: Alex Burkoff Cc: pgsql-general@postgresql.org Subject: Re: [GENERAL] execution plan is wrong, or the query ? Alex Burkoff writes: > I think I am just l

Re: [GENERAL] Postgresql PL parallel processing inside Postgresql function....

2012-12-11 Thread David Johnston
From: pgsql-general-ow...@postgresql.org [mailto:pgsql-general-ow...@postgresql.org] On Behalf Of Misa Simic Sent: Tuesday, December 11, 2012 10:52 AM To: pgsql Subject: [GENERAL] Postgresql PL parallel processing inside Postgresql function Hi, I have a table of bunch of records:

Re: [GENERAL] large database

2012-12-11 Thread Jeff Janes
On Mon, Dec 10, 2012 at 12:26 PM, Mihai Popa wrote: > Hi, > > I've recently inherited a project that involves importing a large set of > Access mdb files into a Postgres or MySQL database. > The process is to export the mdb's to comma separated files than import > those into the final database. >

Re: [GENERAL] SSDs - SandForce or not?

2012-12-11 Thread Merlin Moncure
On Tue, Dec 11, 2012 at 10:48 AM, Bruce Momjian wrote: > On Mon, Dec 10, 2012 at 02:05:09AM -0500, Greg Smith wrote: >> On 11/14/12 2:11 AM, Toby Corkindale wrote: >> >So on the face of it, I think the Sandforce-based drives are probably a >> >winner here, so I should look at the Intel 520s for ev

Re: [GENERAL] Bug in GET STACKED DIAGNOSTICS ?

2012-12-11 Thread Dan S
It would be nice with a consistent behaviour, with the callstack always looking the same despite different causes of the exception. I think it is violating the 'Principle of least astonishment' . Best Regards Dan S 2012/12/11 Pavel Stehule > Hello > > 2012/12/10 Dan S : > > Hi ! > > > > I'm run

[GENERAL] Deleting WAL archives and pg_xlog when there is not a shared drive

2012-12-11 Thread Eng. AlSamman
Hello everyone, I am trying to implement a high-availability cluster using only two nodes, without any shared disk storage. In my implementation, the primary database has continuous archiving set up to a directory residing on the second node, where the standby database is. Streaming replicati

Re: [GENERAL] execution plan is wrong, or the query ?

2012-12-11 Thread Tom Lane
Alex Burkoff writes: > I think I am just looking for an opinion on fundamentals. Can WHERE clause > impact an OUTER JOIN ? Sure. I noticed for instance that your query had some outer joins that were simplified to plain joins as a result of strict WHERE clauses above them. That's not incorrect.

Re: [GENERAL] SSDs - SandForce or not?

2012-12-11 Thread Bruce Momjian
On Mon, Dec 10, 2012 at 02:05:09AM -0500, Greg Smith wrote: > On 11/14/12 2:11 AM, Toby Corkindale wrote: > >So on the face of it, I think the Sandforce-based drives are probably a > >winner here, so I should look at the Intel 520s for evaluation, and > >whatever the enterprise equivalent are for p

Re: [GENERAL] large database

2012-12-11 Thread David Boreham
On 12/11/2012 8:28 AM, Mihai Popa wrote: I guess Chris was right, I have to better understand the usage pattern and do some testing of my own. I was just hoping my hunch about Amazon being the better alternative would be confirmed, but this does not seem to be the case; most of you recommend pu

Re: [GENERAL] execution plan is wrong, or the query ?

2012-12-11 Thread Alex Burkoff
I think I am just looking for an opinion on fundamentals. Can WHERE clause impact an OUTER JOIN ? What I am seeing is that on 9.2 rows from the joined table are restricted prior to joining them with the rest of the tables, and that leads to incorrect results. I have noticed such behaviour only w

[GENERAL] Postgresql PL parallel processing inside Postgresql function....

2012-12-11 Thread Misa Simic
Hi, I have a table of bunch of records: MainTable: -MainID -Other attributes Several Tables what Track Activities about MainIDs... And one routine written as plpgsql function what Loops trough MainTable and for each record, calls ProccessTheThing(MainID), What again sequentially calls, s

Re: [GENERAL] large database

2012-12-11 Thread Mihai Popa
On 12/11/2012 07:27 AM, Bill Moran wrote: On Mon, 10 Dec 2012 15:26:02 -0500 (EST) "Mihai Popa" wrote: Hi, I've recently inherited a project that involves importing a large set of Access mdb files into a Postgres or MySQL database. The process is to export the mdb's to comma separated files t

Re: [GENERAL] Problem with aborting entire transactions on error

2012-12-11 Thread Merlin Moncure
On Sun, Dec 9, 2012 at 9:20 PM, Zbigniew wrote: > Hello, > > As I read while googling the web, many people complained about this > before. Couldn't it be made optional (can be even with "default ON")? > I understand, that there are situations, when it is a must - for > example, when the rest of qu

Re: [GENERAL] Problem with aborting entire transactions on error

2012-12-11 Thread David Johnston
On Dec 11, 2012, at 6:48, Zbigniew wrote: > 2012/12/11, Abel Abraham Camarillo Ojeda : > >> Obviously, it's not fast... > > Well, the problem was exactly speed-related. If not this - I could > simply dispose of the transaction... > -- > regards, > Zbigniew > > It seems like if you switched

Re: [GENERAL] Looking for cooperators

2012-12-11 Thread Merlin Moncure
On Mon, Dec 10, 2012 at 7:02 PM, Fan, Yi wrote: > To whom it my concern, > > We are a Chinese company (www.highgo.com/en/index.html), working on open > source DB products for a while. Our product calls “HighGo Database System” > which is developed base on PostgreSQL. > > HighGo DB already gained a

Re: [GENERAL] large database

2012-12-11 Thread David Boreham
On 12/10/2012 1:26 PM, Mihai Popa wrote: Second, where should I deploy it? The cloud or a dedicated box? Amazon seems like the sensible choice; you can scale it up and down as needed and backup is handled automatically. I was thinking of an x-large RDS instance with 1 IOPS and 1 TB of stora

Re: [GENERAL] Problem with aborting entire transactions on error

2012-12-11 Thread David Johnston
On Dec 11, 2012, at 6:26, Zbigniew wrote: > 2012/12/10, Kevin Grittner : > >> Could an option like this be added to the server? Probably. Has >> anyone worked on such a feature? Not that I know of. Would such a >> feature be accepted if written? I think a better case would need to >> be made for

Re: [GENERAL] Problem with aborting entire transactions on error

2012-12-11 Thread David Johnston
On Dec 11, 2012, at 6:28, Zbigniew wrote: > 2012/12/11, Tom Lane : > >> The complexity and performance costs that people have mentioned are other >> good reasons not to change it; but even if the change were free on the >> server side, history says it's not something we ought to mess with. > >

Re: [GENERAL] large database

2012-12-11 Thread Tony CL Chan
Hi, If you have big table you could also think about Hadoop/HBase or Cassandra but do not put large data set in MySQL. I agree with Bill that "Despite the fact that lots of people have been able to make it (MySQL) work" (me too, another example), there are issues with it. I have been using My

Re: [GENERAL] Problem with aborting entire transactions on error

2012-12-11 Thread Craig Ringer
On 12/11/2012 08:05 PM, Zbigniew wrote: > 2012/12/11, Chris Angelico : > >> I have considered. It will be beneficial to a small group of users, >> including yourself > No, not that small whatsoever; it's easy to find out using Google, how > many people were looking for some help because of this. Yo

Re: [GENERAL] large database

2012-12-11 Thread Bill Moran
On Mon, 10 Dec 2012 15:26:02 -0500 (EST) "Mihai Popa" wrote: > Hi, > > I've recently inherited a project that involves importing a large set of > Access mdb files into a Postgres or MySQL database. > The process is to export the mdb's to comma separated files than import > those into the final d

Re: [GENERAL] Problem with aborting entire transactions on error

2012-12-11 Thread Abel Abraham Camarillo Ojeda
It's faster than restarting the big transaction, maybe... On Tue, Dec 11, 2012 at 5:48 AM, Zbigniew wrote: > 2012/12/11, Abel Abraham Camarillo Ojeda : > > > Obviously, it's not fast... > > Well, the problem was exactly speed-related. If not this - I could > simply dispose of the transaction...

Re: [GENERAL] Problem with aborting entire transactions on error

2012-12-11 Thread Zbigniew
2012/12/11, Abel Abraham Camarillo Ojeda : > Obviously, it's not fast... Well, the problem was exactly speed-related. If not this - I could simply dispose of the transaction... -- regards, Zbigniew -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your s

Re: [GENERAL] Problem with aborting entire transactions on error

2012-12-11 Thread Chris Angelico
On Tue, Dec 11, 2012 at 10:36 PM, Zbigniew wrote: > There are always TWO sides (at least two): creators/designers - and > the users. Considering how much complexity some kind of modification > adds to your - programmer's - code, and how it'll make your life more > difficult, at the same time try t

Re: [GENERAL] Problem with aborting entire transactions on error

2012-12-11 Thread Abel Abraham Camarillo Ojeda
Obviously, it's not fast... On Tue, Dec 11, 2012 at 5:42 AM, Abel Abraham Camarillo Ojeda < acam...@the00z.org> wrote: > I generally do: > > DO $$ > BEGIN > INSERT INTO ...; > EXCEPTION > WHEN UNIQUE_VIOLATION or EXCLUSION_VIOLATION THEN > NULL; /* ignore this rec

Re: [GENERAL] Problem with aborting entire transactions on error

2012-12-11 Thread Abel Abraham Camarillo Ojeda
I generally do: DO $$ BEGIN INSERT INTO ...; EXCEPTION WHEN UNIQUE_VIOLATION or EXCLUSION_VIOLATION THEN NULL; /* ignore this record */ END; $$; On Sun, Dec 9, 2012 at 9:20 PM, Zbigniew wrote: > Hello, > > As I read while googling the web, many people complaine

Re: [GENERAL] large database

2012-12-11 Thread Chris Angelico
On Tue, Dec 11, 2012 at 9:33 PM, Gavin Flower wrote: > > On Tue, Dec 11, 2012 at 7:26 AM, Mihai Popa wrote: > > Second, where should I deploy it? The cloud or a dedicated box? > > Would you say the issue is cloudy? > (I'm not being entirely facetious!) *Groan* :) It's certainly not clear-cut in

Re: [GENERAL] Problem with aborting entire transactions on error

2012-12-11 Thread Zbigniew
2012/12/10, Kevin Grittner : > Could an option like this be added to the server? Probably. Has > anyone worked on such a feature? Not that I know of. Would such a > feature be accepted if written? I think a better case would need to > be made for its value compared to the added code complexity and

Re: [GENERAL] Bug in GET STACKED DIAGNOSTICS ?

2012-12-11 Thread Pavel Stehule
Hello 2012/12/10 Dan S : > Hi ! > > I'm running "PostgreSQL 9.2.2 on i686-pc-linux-gnu, compiled by gcc > (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3, 32-bit" on ubuntu 12.04 . > > I'm a bit perplexed by the difference in PG_EXCEPTION_CONTEXT of the > following two invocations of a function where I try

Re: [GENERAL] large database

2012-12-11 Thread Johannes Lochmann
Hi all, On 12/11/2012 11:02 AM, Jan Kesten wrote: >> I would very much appreciate a copy or a link to these slides! > here they are: > > http://www.scribd.com/mobile/doc/61186429 > thank you very much! Johannes -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make chang

Re: [GENERAL] large database

2012-12-11 Thread Chris Travers
On Mon, Dec 10, 2012 at 12:26 PM, Mihai Popa wrote: > Hi, > > I've recently inherited a project that involves importing a large set of > Access mdb files into a Postgres or MySQL database. > The process is to export the mdb's to comma separated files than import > those into the final database. >

Re: [GENERAL] large database

2012-12-11 Thread Gavin Flower
On 11/12/12 23:25, Chris Angelico wrote: On Tue, Dec 11, 2012 at 7:26 AM, Mihai Popa wrote: Second, where should I deploy it? The cloud or a dedicated box? Forget cloud. For similar money, you can get dedicated hosting with much more reliable performance. We've been looking at places to deploy

Re: [GENERAL] large database

2012-12-11 Thread Chris Angelico
On Tue, Dec 11, 2012 at 7:26 AM, Mihai Popa wrote: > Second, where should I deploy it? The cloud or a dedicated box? Forget cloud. For similar money, you can get dedicated hosting with much more reliable performance. We've been looking at places to deploy a new service, and to that end, we booked

Re: [GENERAL] large database

2012-12-11 Thread Jan Kesten
Hi all, > I would very much appreciate a copy or a link to these slides! here they are: http://www.scribd.com/mobile/doc/61186429 Have fun! -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-ge

Re: [GENERAL] large database

2012-12-11 Thread Johannes Lochmann
Hello Jan, hello List On 12/11/2012 09:10 AM, Jan Kesten wrote: > There are some sildes from Sun/Oracle about ZFS, ZIL, SSD and > PostgreSQL performance (I can look if I find them if needed). I would very much appreciate a copy or a link to these slides! Johannes -- Sent via pgsql-general mai

Re: [GENERAL] large database

2012-12-11 Thread Jan Kesten
Hi Mihai. > We are now at the point where the csv files are all created and amount > to some 300 GB of data. > I would like to get some advice on the best deployment option. First - and maybe best - advice: Do some testing on your own and plan some time for this. > First, the project has been s