Re: [GENERAL] Build in function to verify email addresses

2017-11-16 Thread Chris Withers
I'm pretty sure this isn't as simple as you think it is, I'd suggest having a good read of: https://stackoverflow.com/a/201378/216229 Chris On 16/11/2017 07:56, Nick Dro wrote: I beleieve that every information system has the needs to send emails. Currently PostgreSQL doesn't have a

Re: [GENERAL] options for logical replication plugins?

2017-03-16 Thread Chris Withers
On 15/03/2017 19:18, Andres Freund wrote: Hi, On 2017-03-15 18:29:06 +, Chris Withers wrote: Shame the decoding has to be done on the server-side rather than the client side. Why? Requiring compiled extensions to be installed on the server is always going to be a pain, especially

[GENERAL] options for logical replication plugins?

2017-03-15 Thread Chris Withers
Hi All, What are the current "best" options for logical replication plugins? I saw wal2json and one that serialized to protobuf instead, and then of course there's bottledwater, but none of these smell particularly production ready. Shame the decoding has to be done on the server-side rather

[GENERAL] json aggregation question

2017-02-28 Thread Chris Withers
Hi All, Given the following table: |#createtablething (id serial,tags jsonb);#\d thing Table"public.thing"Column|Type |Modifiers +-+id |integer |notnulldefaultnextval('thing_id_seq'::regclass)tags |jsonb || ...and the

Re: [GENERAL] json aggregation question

2017-02-28 Thread Chris Withers
g2'?'t2val1' count | tag1 2| {val1,val2,val3} 28 Şub 2017 Sal, 19:22 tarihinde, Chris Withers <ch...@simplistix.co.uk <mailto:ch...@simplistix.co.uk>> şunu yazdı: Hi All, Given the following table: |#createtablething (id serial,tags jsonb);#\d thing Ta

[GENERAL] json aggregation question

2017-02-28 Thread Chris Withers
Hi All, Given the following table: |#createtablething (id serial,tags jsonb);#\d thing Table"public.thing"Column|Type |Modifiers +-+id |integer |notnulldefaultnextval('thing_id_seq'::regclass)tags |jsonb || ...and the

Re: [GENERAL] default representation of null in psql

2017-01-01 Thread Chris Withers
On 12/12/2016 14:33, Adrian Klaver wrote: On 12/11/2016 11:34 PM, Chris Withers wrote: On 01/12/2016 12:12, Francisco Olarte wrote: On Thu, Dec 1, 2016 at 12:56 PM, Chris Withers <ch...@simplistix.co.uk> wrote: So, first observation: if I make room nullable, the exclude constrain

Re: [GENERAL] schema advice for event stream with tagging and filtering

2016-12-12 Thread Chris Withers
On 16/08/2016 15:10, Ilya Kazakevich wrote: An event is a row with a primary key along the lines of (colo, host, category) and an associated set of tags, where each tag has a type and a value (eg: {"color": "red", "owner": "fred", "status": "open"...}). What about simple table with several

Re: [GENERAL] btree gist indices, null and open-ended tsranges

2016-12-12 Thread Chris Withers
On 01/12/2016 12:12, Francisco Olarte wrote: On Thu, Dec 1, 2016 at 12:56 PM, Chris Withers <ch...@simplistix.co.uk> wrote: So, first observation: if I make room nullable, the exclude constraint does not apply for rows that have a room of null. I guess that's to be expected, right? I

[GENERAL] btree gist indices, null and open-ended tsranges

2016-12-01 Thread Chris Withers
Hi All, Working with the exclude constraint example from https://www.postgresql.org/docs/current/static/rangetypes.html: CREATE EXTENSION btree_gist; CREATE TABLE room_reservation ( room text, during tsrange, EXCLUDE USING GIST (room WITH =, during WITH &&) ); So, first

Re: [GENERAL] performance problems with bulk inserts/updates on tsrange with gist-based exclude constrains

2016-11-25 Thread Chris Withers
Hey Tom, I appreciate you're busy, but did you ever get a chance to look at this? On 19/09/2016 08:40, Chris Withers wrote: On 16/09/2016 15:29, Tom Lane wrote: Chris Withers <ch...@simplistix.co.uk> writes: On 16/09/2016 14:54, Igor Neyman wrote: So, what is the value for "en

Re: [GENERAL] performance problems with bulk inserts/updates on tsrange with gist-based exclude constrains

2016-11-24 Thread Chris Withers
Hey Tom, I appreciate you're busy, but did you ever get a chance to look at this? On 19/09/2016 08:40, Chris Withers wrote: On 16/09/2016 15:29, Tom Lane wrote: Chris Withers <ch...@simplistix.co.uk> writes: On 16/09/2016 14:54, Igor Neyman wrote: So, what is the value for "en

Re: [GENERAL] dumping table contents in a sensible order

2016-11-15 Thread Chris Withers
On 16/11/2016 01:05, Adrian Klaver wrote: INSERT 0 1 ERROR: insert or update on table "table_one" violates foreign key constraint "table_one_parent_id_fkey" DETAIL: Key (parent_id)=(xxx) is not present in table "table_one". So, the problem appears to be that table_one is self-referential by

[GENERAL] dumping table contents in a sensible order

2016-11-15 Thread Chris Withers
Hi All, I have a database that I want to dump three tables from, for use in development. They form a subset of the data, so I was dumping like this: pg_dump thedatabase --inserts -t table_one -t depends_on_table_one -t depends_on_previous_two > dump.sql However, when I try to load this

Re: [GENERAL] performance problems with bulk inserts/updates on tsrange with gist-based exclude constrains

2016-09-19 Thread Chris Withers
On 16/09/2016 15:29, Tom Lane wrote: Chris Withers <ch...@simplistix.co.uk> writes: On 16/09/2016 14:54, Igor Neyman wrote: So, what is the value for "end ts", when the record is inserted (the range just started)? It's open ended, so the period is [start_ts, ) I've not lo

Re: [GENERAL] performance problems with bulk inserts/updates on tsrange with gist-based exclude constrains

2016-09-16 Thread Chris Withers
On 16/09/2016 10:26, John R Pierce wrote: On 9/16/2016 2:23 AM, John R Pierce wrote: wait, what is a tsrange? the standard textsearch data types in postgres are tsvector and tsquery, never mind, I should have known, its a timestamp range. ... when you do updates, are you changing any

Re: [GENERAL] performance problems with bulk inserts/updates on tsrange with gist-based exclude constrains

2016-09-16 Thread Chris Withers
On 16/09/2016 12:00, John R Pierce wrote: On 9/16/2016 3:46 AM, Chris Withers wrote: when you do updates, are you changing any of the indexed fields, or just "value" ? Yeah, it's a temporal table, so "updates" involve modifying the period column for a row to set its end ts

[GENERAL] performance problems with bulk inserts/updates on tsrange with gist-based exclude constrains

2016-09-16 Thread Chris Withers
Hi All, I have quite a few tables that follow a pattern like this: Table "public.my_model" Column | Type| Modifiers +---+--- period | tsrange | not null key| character varying | not null value | integer |

Re: [GENERAL] performance problems with bulk inserts/updates on tsrange with gist-based exclude constrains

2016-09-16 Thread Chris Withers
On 16/09/2016 14:54, Igor Neyman wrote: -Original Message- From: pgsql-general-ow...@postgresql.org [mailto:pgsql-general-ow...@postgresql.org] On Behalf Of Chris Withers Sent: Friday, September 16, 2016 6:47 AM To: John R Pierce <pie...@hogranch.com>; pgsql-general@postgres

Re: [GENERAL] performance problems with bulk inserts/updates on tsrange with gist-based exclude constrains

2016-09-16 Thread Chris Withers
On 16/09/2016 12:00, John R Pierce wrote: On 9/16/2016 3:46 AM, Chris Withers wrote: when you do updates, are you changing any of the indexed fields, or just "value" ? Yeah, it's a temporal table, so "updates" involve modifying the period column for a row to set its end ts

Re: [GENERAL] performance problems with bulk inserts/updates on tsrange with gist-based exclude constrains

2016-09-16 Thread Chris Withers
On 16/09/2016 10:26, John R Pierce wrote: On 9/16/2016 2:23 AM, John R Pierce wrote: wait, what is a tsrange? the standard textsearch data types in postgres are tsvector and tsquery, never mind, I should have known, its a timestamp range. ... when you do updates, are you changing any

[GENERAL] performance problems with bulk inserts/updates on tsrange with gist-based exclude constrains

2016-09-16 Thread Chris Withers
Hi All, I have quite a few tables that follow a pattern like this: Table "public.my_model" Column | Type| Modifiers +---+--- period | tsrange | not null key| character varying | not null value | integer |

[GENERAL] schema advice for event stream with tagging and filtering

2016-08-16 Thread Chris Withers
Hi All, What would be the best schema to use when looking to implement an event stream with tagging and filtering? An event is a row with a primary key along the lines of (colo, host, category) and an associated set of tags, where each tag has a type and a value (eg: {"color": "red",

Re: [GENERAL] schema advice for event stream with tagging and filtering

2016-08-16 Thread Chris Withers
On 16/08/2016 14:29, Ilya Kazakevich wrote: Hi, An event is a row with a primary key along the lines of (colo, host, category) and an associated set of tags, where each tag has a type and a value (eg: {"color": "red", "owner": "fred", "status": "open"...}). What about simple table with

[GENERAL] schema advice for event stream with tagging and filtering

2016-08-16 Thread Chris Withers
Hi All, What would be the best schema to use when looking to implement an event stream with tagging and filtering? An event is a row with a primary key along the lines of (colo, host, category) and an associated set of tags, where each tag has a type and a value (eg: {"color": "red",

Re: [GENERAL] connection file descriptors created with identical number after process fork on mac

2016-08-03 Thread Chris Withers
On 04/08/2016 00:20, Tom Lane wrote: Chris Withers <ch...@simplistix.co.uk> writes: I'm writing some multi-process code in Python and trying to make sure I open a new connection for each process. Here's the really cut down code: ... What's really surpising to me is the output on

[GENERAL] connection file descriptors created with identical number after process fork on mac

2016-08-03 Thread Chris Withers
Hi All, I'm writing some multi-process code in Python and trying to make sure I open a new connection for each process. Here's the really cut down code: """ import os, time import psycopg2 from multiprocessing import Pool def init(): conn = psycopg2.connect("dbname=...host=localhost")

Re: [GENERAL] create create of database of a particular name

2016-03-09 Thread Chris Withers
On 09/03/2016 14:53, Adrian Klaver wrote: On 03/09/2016 05:49 AM, Chris Withers wrote: Hi All, Django, a popular python web framework, runs its unit tests by creating a 'test_'-prefixed database and then dropping it at the end of the test run. FYI in Django 1.8+ you can prevent that: https

[GENERAL] create create of database of a particular name

2016-03-09 Thread Chris Withers
Hi All, Django, a popular python web framework, runs its unit tests by creating a 'test_'-prefixed database and then dropping it at the end of the test run. Is there any way I can grant a user in postgres the ability only to create a database of a particular name? cheers, Chris -- Sent

[GENERAL] foreign key to "some rows" of a second table

2016-02-22 Thread Chris Withers
Hi All, So, I have a table that looks like this: CREATE TABLE config (     region    varchar(10),     name    varchar(10),     value    varchar(40) ); Another looks like this: CREATE TABLE tag (     host    varchar(10),    

[GENERAL] aggregation question

2015-12-08 Thread Chris Withers
Hi All, I hope this is quite a simple one... I have a fixtures table containing home_club and away_club, and I'd like to select the distinct list of clubs contained in all rows but in either the home_club or away_club columns. How would I do that? thanks for the help! Chris -- Sent

Re: [GENERAL] union/difference/intersection question

2015-12-08 Thread Chris Withers
On 08/12/2015 11:51, David Rowley wrote: On 9 December 2015 at 00:44, Chris Withers <ch...@simplistix.co.uk <mailto:ch...@simplistix.co.uk>> wrote: Hi All, I hope this is quite a simple one... I have a fixtures table containing home_club and away_club, and I'd li

[GENERAL] "trust" authentication in pg_hba.conf

2015-11-24 Thread Chris Withers
Hi All, What's the default contents of pg_hba.conf that postgres ships with? I've been to it contains 'trust' for all local connections. Is this wise? Anyone who can get a shell on your database server can connect to any database as any user? cheers, Chris -- Sent via pgsql-general

Re: [GENERAL] current_query='IDLE" in pg_stat_activity

2015-11-23 Thread Chris Withers
On 23/11/2015 10:33, paramjib baruah wrote: I am trying to check current running sql queries in postgres through pg_stat_activity .However, in the current_query column I could see only "IDLE" . I am not able to see the query . That would suggest no query is running, what suggests to you that

[GENERAL] using a postgres table as a multi-writer multi-updater queue

2015-11-23 Thread Chris Withers
Hi All, I wondered if any of you could recommend best practices for using a postgres table as a queue. Roughly speaking, 100-200 workers will vomit rows and rates of a few hundres per second into the table leaving the status as new and then as many workers as needed to keep up with the load

Re: [GENERAL] scaling postgres - can child tables be in a different tablespace?

2015-08-04 Thread Chris Withers
On 03/08/2015 08:40, Jony Cohen wrote: Servers now days reach very impressive write speeds and at rather low prices - it's simpler to split the write to 2 tables on different tablespaces/devices than 2 servers. This raises an interesting question: can a child table be in a different tablespace

Re: [GENERAL] scaling postgres

2015-08-04 Thread Chris Withers
On 04/08/2015 09:11, Seref Arikan wrote: I work in healthcare and patient centric records let me consider multiple servers for lots and lots of patients. The engineering team from instagram has been sharing their experience with postgres, which is

Re: [GENERAL] scaling postgres

2015-08-04 Thread Chris Withers
On 03/08/2015 08:34, Seref Arikan wrote: At what point does postgres stop scaling? What happens when the computational load no longer fits on one machine? What are the options then? I think it is hard to come up with blanket responses to generic questions such as What happens

[GENERAL] scaling postgres

2015-08-03 Thread Chris Withers
Hi All, I'm curious if there are recommendations for scaling postgres to what, for me, seems like a lot of data... The app in question currently writes around 1.5 billion rows into a table before rolling them up into tables that have a few million roll up rows each. That 1.5 billion row

Re: [GENERAL] scaling postgres

2015-08-03 Thread Chris Withers
On 03/08/2015 08:15, Jony Cohen wrote: SSD disks are cheep these days but they don't like repeated writes/deletes so it might cause problems down the line (hence my first RAM recommendation) as for keeping the raw data - you could easily do it if you use partitions, if you have daily

Re: [GENERAL] pgbench tps drop from 5000 to 37 going from localhost to a server 13ms away

2015-07-27 Thread Chris Withers
On 24/07/2015 22:51, Jeff Janes wrote: starting vacuum...end. transaction type: TPC-B (sort of) scaling factor: 1 This is your problem. There is only one row in the pgbench_branch table, and every transaction has to update that one row. This is inherently a seriaized event.

Re: [GENERAL] pgbench tps drop from 5000 to 37 going from localhost to a server 13ms away

2015-07-24 Thread Chris Withers
On 24/07/2015 19:21, Jan Lentfer wrote: I've been doing some lightweight load testing with “pgbench -c8 -j8 -T10” When run locally on the postgres server I've testing, this gives around 5000tps When I do it from a server that has a 13ms ping latency, it drops to 37tps. This is using the

[GENERAL] pgbench tps drop from 5000 to 37 going from localhost to a server 13ms away

2015-07-24 Thread Chris Withers
Hi all, I've been doing some lightweight load testing with “pgbench -c8 -j8 -T10” When run locally on the postgres server I've testing, this gives around 5000tps When I do it from a server that has a 13ms ping latency, it drops to 37tps. This is using the default pgbench script, is it to be

[GENERAL] how do I disable automatic start on mac os x?

2010-01-04 Thread Chris Withers
Hi All, I installed postgres from the enterprisedb-maintained one-click installer at http://www.postgresql.org/download/macosx. However, it starts automatically when the machine boots up, and I'd like it only to start when I manually start it. I've found an entry in launchd: $ sudo

Re: [GENERAL] how do I disable automatic start on mac os x?

2010-01-04 Thread Chris Withers
Naoko Reeves wrote: Try check this directory: Library: StartupItems: postgres-plus-8.xx there should be script to start automatically. /Library/StartupItems is empty. I think putting stuff in there is an old-fashioned way of doing things that went out with Mac OS X 10.4, and I'm glad to say

Re: [GENERAL] how do I disable automatic start on mac os x?

2010-01-04 Thread Chris Withers
Dave Page wrote: On Mon, Jan 4, 2010 at 1:38 PM, Chris Withers ch...@simplistix.co.uk wrote: Naoko Reeves wrote: Try check this directory: Library: StartupItems: postgres-plus-8.xx there should be script to start automatically. /Library/StartupItems is empty. I think putting stuff

Re: [GENERAL] how do I disable automatic start on mac os x?

2010-01-04 Thread Chris Withers
Dave Page wrote: Iirc, you have to change: keyRunAtLoad/key true/ to keyRunAtLoad/key false/ I feel slightly stupid asking this, but this is a Mac, surely there's some pretty shiny GUI for editing things like this? (even Windows has the Services stuff in

Re: [GENERAL] how do I disable automatic start on mac os x?

2010-01-04 Thread Chris Withers
Sachin Srivastava wrote: Iirc, you have to change: keyRunAtLoad/key true/ to keyRunAtLoad/key false/ Sachin; can you confirm please? Yes. Though the daemon is loaded during startup. Will setting RunAtLoad to false not stop that? Chris -- Simplistix -

Re: [GENERAL] how do I disable automatic start on mac os x?

2010-01-04 Thread Chris Withers
Sachin Srivastava wrote: Will setting RunAtLoad to false not stop that? RunAtLoad = false means, the daemon is loaded not executed/launched. If you dont want it to load at system startup. Remove the plist from that directory. (i.e. /Library/LaunchDaemons) I don't want postgres to start on