Re: [GENERAL] Recover from corrupted database due to failing disk

2016-11-02 Thread Jim Nasby
On 11/2/16 6:21 PM, Jim Nasby wrote: I wouldn't trust the existing cluster that far. Since it sounds like you have no better options, you could use zero_damaged_pages to allow a pg_dumpall to complete, but you're going to end up with missing data. So what I'd suggest would be: stop Postgres

Re: [GENERAL] Recover from corrupted database due to failing disk

2016-11-02 Thread Jim Nasby
ou could use zero_damaged_pages to allow a pg_dumpall to complete, but you're going to end up with missing data. So what I'd suggest would be: stop Postgres make a copy of the cluster start with zero_damaged_pages pg_dumpall stop and remove the cluster (make sure you've got that backup) create a ne

Re: [GENERAL] Questions on Post Setup MASTER and STANDBY replication - Postgres9.1

2016-11-02 Thread Jim Nasby
replication? Also, it seems the startup process stucks on “recovering 00010004”, how to resolve it? As far as I know that's normal while in streaming mode. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data

Re: [GENERAL] initdb createuser commands

2016-11-02 Thread Jim Nasby
and push new users away. And we wonder why we're having trouble attracting new developers... This has actually been discussed recently on -hackers as well[1], and there is some general consensus that simplification in this area would be a good idea. 1: https://www.postgresql.org/message-id/20

Re: [GENERAL] Checking Postgres Streaming replication delay

2016-11-02 Thread Jim Nasby
On 10/31/16 3:39 PM, Patrick B wrote: |( ||extract(epoch FROMnow())- ||extract(epoch FROMpg_last_xact_replay_timestamp()) ||)::int lag| You could certainly simplify it though... extract(epoch FROM now()-pg_last_xact_replay_timestamp()) -- Jim Nasby, Data Architect, Blue Treble Consulting

Re: [GENERAL] 2.5TB Migration from SATA to SSD disks - PostgreSQL 9.2

2016-09-24 Thread Jim Nasby
integrity of the data. AFAIK pg_basebackup blindly copies all data files, while rsync will transfer only the parts of the files that have actually changed (see --block-size). If the source and destination are on different servers, that can mean less data transferred over the network. -- Jim N

Re: [GENERAL] overwrite column data select - Postgres 9.2

2016-09-21 Thread Jim Nasby
aving a default set largely defeats the purpose of NOT NULL (IMHO). -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855-873-2532) mobile: 512-569-9461 --

Re: [GENERAL] Duplicate data despite unique constraint

2016-09-10 Thread Jim Nasby
On 9/2/16 8:02 AM, Adrian Klaver wrote: Best guess is the INDEX on the column is corrupted and needs to be reindexed: You should contact AWS support about this; they'd want to know. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture

Re: [GENERAL] What limits Postgres performance when the whole database lives in cache?

2016-09-10 Thread Jim Nasby
. MySQL used to tout how fast it was compared to Postgres, using a benchmark it created specifically for that purpose that had very little to do with the real world. People eventually discovered that as soon as you had a concurrent workload Postgres was actually faster. -- Jim Nasby, Data Architect

Re: [GENERAL] Trigger is not working for Inserts from the application

2016-09-10 Thread Jim Nasby
es the app is actually running. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855-873-2532) mobile: 512-569-9461 -- Sent via pgsql-general mailing l

Re: [GENERAL] 2.5TB Migration from SATA to SSD disks - PostgreSQL 9.2

2016-09-10 Thread Jim Nasby
to 1 second). -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855-873-2532) mobile: 512-569-9461 -- Sent via pgsql-general mailing list (pgsql-general

Re: [GENERAL] 2.5TB Migration from SATA to SSD disks - PostgreSQL 9.2

2016-09-07 Thread Jim Nasby
, rsync -va --modify-window=1 would remove your concern about a same second race condition without forcing the sync to read through all the files. Very interesting and useful! -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data

Re: [GENERAL] What limits Postgres performance when the whole database lives in cache?

2016-09-07 Thread Jim Nasby
." - Me, circa 1999. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855-873-2532) mobile: 512-569-9461 -- Sent via pgsql-general mailing

Re: [GENERAL] IDE for function/stored proc development.

2016-09-07 Thread Jim Nasby
'll want to create a separate sqitch migration for each object. [1] http://sqitch.org/ -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855-873-2532) mobi

Re: [GENERAL] How to assemble all fields of (any) view into a string?

2016-09-07 Thread Jim Nasby
to JSON would make it relatively easy to accomplish what you want in a plpgsql (or maybe even plsql) function. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855

Re: [GENERAL] Clustered index to preserve data locality in a multitenant application?

2016-09-07 Thread Jim Nasby
the replication. Depending on your needs, a major benefit to this method is it makes major version upgrades very simple: you just stand up a new replica on the new version and then failover to it. If anything goes wrong, you can fail back to the old version without losing any data. -- Jim Nasby, Data

Re: [GENERAL] 2.5TB Migration from SATA to SSD disks - PostgreSQL 9.2

2016-09-07 Thread Jim Nasby
after rsync reads). You need to add the --checksum flag to rsync (which means it will still have to read everything that's in /var/lib/pgsql). -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble

Re: [GENERAL] Privileges on public schema can't be revoked?

2016-09-07 Thread Jim Nasby
relatively easy to do today; see the has_*_privilege() functions. You might also find http://pgxn.org/dist/pg_acl useful. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855

Re: [GENERAL] Materialized view auto refresh

2016-09-07 Thread Jim Nasby
(and it would be far easier for them to submit improvements). -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855-873-2532) mobile: 512-569-9461 -- Sent via

Re: [GENERAL] Postgres UPGRADE from 9.2 to 9.4

2016-09-07 Thread Jim Nasby
of replication, so you might be better off just sticking with Postgres tools. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855-873-2532) mobile: 512-569-9461

Re: [GENERAL] How to assemble all fields of (any) view into a string?

2016-09-07 Thread Jim Nasby
this function? Or some other way? Thanks. A PL that can accept composite types (such as plpythonu) should be able to do this. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http

Re: [GENERAL] PostgreSQL Database performance

2016-09-07 Thread Jim Nasby
occurrence if it's enough to make a difference then you're already close to the limits of your IO, and if that's true then you definitely want to spread the checkpoint out over a longer interval. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture

Re: [GENERAL] a column definition list is required for functions returning "record"

2016-09-07 Thread Jim Nasby
mething (like a function) that has explicitly > defined what the contents of the record are. We have that already, it's named 'json_each_text' Apparently you haven't looked at json parse/deparse costs ;P -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data

Re: [GENERAL] a column definition list is required for functions returning "record"

2016-09-02 Thread Jim Nasby
On 8/29/16 6:28 AM, Tom Lane wrote: Pavel Stehule <pavel.steh...@gmail.com> writes: > 2016-08-29 1:59 GMT+02:00 Jim Nasby <jim.na...@bluetreble.com>: >> It would be nice if there was a way to pass dynamically formed records >> around, similar to how you can pass

Re: [GENERAL] LOG: could not fork new process for connection: Cannot allocate memory

2016-08-28 Thread Jim Nasby
a breakpoint at ereport and then trying to connect. You could then get a backtrace; just don't leave the system in that state for long. (There might be a more elegant way to do that...) -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and Pos

Re: [GENERAL] a column definition list is required for functions returning "record"

2016-08-28 Thread Jim Nasby
is true for SQL functions. But you'd probably need a more complex query for that to be a win over the lighter weight nature of SQL functions. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble

Re: [GENERAL] Any reasons for 'DO' statement not returning result?

2016-08-16 Thread Jim Nasby
On 8/16/16 1:05 PM, Adrian Klaver wrote: On 08/16/2016 07:54 AM, Jim Nasby wrote: On 8/14/16 5:13 AM, Xtra Coder wrote: - ability to switch session language from 'sql' to 'pl/pgsql' Actually, something I wish I had was the ability to temporarily switch to an entirely different interpreter

Re: [GENERAL] C++ port of Postgres

2016-08-16 Thread Jim Nasby
...@bluetreble.com) -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855-873-2532) mobile: 512-569-9461 -- Sent via pgsql-general mailing list (pgsql-general

Re: [GENERAL] Any reasons for 'DO' statement not returning result?

2016-08-16 Thread Jim Nasby
almost *any* object in pg_temp!). What would be a lot more interesting is if creating a temp function didn't involve writing an entry to the catalog (something being discussed for temp tables right now). -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data

Re: [GENERAL] Jsonb extraction very slow

2016-08-16 Thread Jim Nasby
ct might be able to do it's expansion in the original object so that subsequent references to that key wouldn't need to re-expand it. I don't think the current EO framework supports that, but it doesn't seem impossible to add... -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in A

Re: [GENERAL] Jsonb extraction very slow

2016-08-16 Thread Jim Nasby
On 8/11/16 8:45 AM, Tom Lane wrote: Jim Nasby <jim.na...@bluetreble.com> writes: I never dug into why. As Tom posited, decompression might explain the time to get a single key out. Getting 10 keys instead of just 1 wasn't 10x more expensive, but it was significantly more expensive tha

Re: [GENERAL] Any reasons for 'DO' statement not returning result?

2016-08-16 Thread Jim Nasby
. That would be especially useful for debugging plpython functions. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855-873-2532) mobile: 512-569-9461

Re: [GENERAL] Jsonb extraction very slow

2016-08-11 Thread Jim Nasby
more expensive, but it was significantly more expensive than just getting a single key. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855-873-2532

Re: [GENERAL] Jsonb extraction very slow

2016-08-10 Thread Jim Nasby
ets (to improve compression), but I never got around to actually tracing it. I suspect there's a win to be had by having both json types use the ExpandedObject stuff. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble?

Re: [GENERAL] Any reasons for 'DO' statement not returning result?

2016-08-10 Thread Jim Nasby
least one correctly typed input (even if it's NULL) so the pseudotype will work. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855-873-2532) mobile: 512-569-9461

[GENERAL] plpython.h not installed in 9.4

2016-08-10 Thread Jim Nasby
9.5+ installs plpython.h under include/server. 9.4 apparently doesn't. I'm guessing that changed in 9.5? Or am I doing something wrong? -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http

Re: [GENERAL] Array value from table as parameter

2016-07-22 Thread Jim Nasby
u're seeing. I suspect that in the process of doing that you're going to uncover a bug in your code, but if not then we'll have something concrete we can look at. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get

Re: [GENERAL] unique constraint with several null values

2016-07-22 Thread Jim Nasby
you could then do a unique index on that WHERE array != array[]. Maybe a less obtuse option would be to use a boolean array. Storage would be ~8x larger, but since there should be very few rows I doubt that matters. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Expe

Re: [GENERAL] Constraint using a SQL function executed during SELECT

2016-07-19 Thread Jim Nasby
not exist ... CONTEXT: SQL function "f" during inlining In this example, you should be able to avoid that by setting constraint_exclusion=off. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Ge

Re: [GENERAL] Stored procedure version control

2016-07-01 Thread Jim Nasby
(as in, database unit tests). In 9 years in an environment where downtime was 6 figures per hour I only had 1 or 2 deployments that had problems, and never bad enough to consider reverting. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture

Re: [GENERAL] How safe is pg_basebackup + continuous archiving?

2016-07-01 Thread Jim Nasby
ely. If someone did want that though, it could probably be done as an extension. I believe you just have to pull all of each relation into shared buffers for the checksums to be verified. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architect

Re: [GENERAL] [HACKERS] sign function with INTERVAL?

2016-04-13 Thread Jim Nasby
code for interval_lt, all that needs to happen to add this support is to expose interval_cmp_internal() as a strict function. It already does exactly what you want. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Troubl

[GENERAL] Odd quoting behavior from \set

2016-01-19 Thread Jim Nasby
Is this odd quoting behavior expected? (Note difference between "'a':1" and "'b':'a'") ~@decina.local/53896# \set df pd.DataFrame.from_dict([{'a':1,'b':'a'},{'a':2,'b':'b'}]) ~@decina.local/53896# \echo :df pd.DataFrame.from_dict([{a:1,b:'a'},{a:2,b:'b'}]) ~@decina.local/5

Re: [GENERAL] Execute commands in single-user mode

2016-01-18 Thread Jim Nasby
ed in because they became extremely popular. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make ch

Re: [GENERAL] Unable to build python extension with PGXS

2016-01-15 Thread Jim Nasby
On 1/13/16 3:11 PM, Jim Nasby wrote: On 1/12/16 10:04 PM, Jim Nasby wrote: Attempting to build a python extension, I'm getting: Undefined symbols for architecture x86_64: "_PyErr_Clear", referenced from: _PLyNdarray_FromDatum in pg_ndarray.o _PLyObject_

Re: [GENERAL] Moving a large DB (> 500GB) to another DB with different locale

2016-01-13 Thread Jim Nasby
. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http

Re: [GENERAL] Changing varchar length by manipulating pg_attribute

2016-01-13 Thread Jim Nasby
hat. Obviously you should test all of this thoroughly before doing it in production. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-general mailing

Re: [GENERAL] Unable to build python extension with PGXS

2016-01-13 Thread Jim Nasby
On 1/12/16 10:04 PM, Jim Nasby wrote: Attempting to build a python extension, I'm getting: Undefined symbols for architecture x86_64: "_PyErr_Clear", referenced from: _PLyNdarray_FromDatum in pg_ndarray.o _PLyObject_To_ndarray in pg_ndarray.o "_PyImpo

Re: [GENERAL] Moving a large DB (> 500GB) to another DB with different locale

2016-01-13 Thread Jim Nasby
On 1/13/16 2:39 PM, Andreas Joseph Krogh wrote: Where can I find more info about how to use and configure pg_logical to replicate a 9.4 DB to 9.5? http://2ndquadrant.com/en/resources/pglogical/ -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data

Re: [GENERAL] Call postgres PL/Python stored function from another PL/Python block.

2016-01-13 Thread Jim Nasby
ore information. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your sub

[GENERAL] Unable to build python extension with PGXS

2016-01-12 Thread Jim Nasby
erenced from: _PLyObject_To_ndarray in pg_ndarray.o ld: symbol(s) not found for architecture x86_64 I've included $(python_includespec) in my Makefile: override CPPFLAGS := $(python_includespec) $(CPPFLAGS) Is there some other magic I need? Do I need to switch to using MODULE_big or something? -

[GENERAL] New hacker item posted

2016-01-11 Thread Jim Nasby
Anyone looking to get their feet wet in the backend code, please take a look at http://www.postgresql.org/message-id/568f03ef.4070...@bluetreble.com. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get

Re: [GENERAL] Using xmax to detect deleted rows

2016-01-10 Thread Jim Nasby
be a lot simpler to just do whatever you need to do when the row is actually deleted. Just be sure you deal with rollbacks correctly if you're doing something external. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Tr

Re: [GENERAL] Trigger function interface

2016-01-10 Thread Jim Nasby
e info. But is this the parse tree for the top level query which involves the trigger? Hrm, apparently not. fcinfo->context would maybe be helpful, but I'm not sure. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it

Re: [GENERAL] Execute commands in single-user mode

2016-01-10 Thread Jim Nasby
issues with large objects, notably their use of OIDs. Lots of LOs can lead to OID depletion. There was a thread about this recently. It might be about time to come up with an extension that's a replacement for large objects. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX

Re: [GENERAL] COPY FROM STDIN

2016-01-10 Thread Jim Nasby
would be much appreciated. I don't know off-hand. I suggest you look at what psql does to implement \copy (note the \). -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com I

Re: [GENERAL] Code of Conduct: Is it time?

2016-01-10 Thread Jim Nasby
reporting. Also, not allowing your CoC to become a weapon that someone can use offensively. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-general

Re: [GENERAL] Code of Conduct: Is it time?

2016-01-06 Thread Jim Nasby
, ask your questions! The Postgres community really is one of the most patient and helpful OSS communities out there, and there's plenty of people that would be happy to explain things. Questions are also a good way to show where things could possibly be better commented/documented. -- Jim Nasby

Re: [GENERAL] Code of Conduct: Is it time?

2016-01-06 Thread Jim Nasby
or dealing with an actual situation. (BTW, if your concern on enforcement is about control, not only can people be removed from mailing lists and the like, but there actually is a Postgres legal entity that could start legal proceedings if it ever came to it.) [1] http://couchdb.apache.org/conduc

Re: [GENERAL] Code of Conduct: Is it time?

2016-01-06 Thread Jim Nasby
age (which with the planner hooks might actually be possible). -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-general mailing list (pgsql-general@postgresql.or

Re: [GENERAL] Definitive answer: can functions use indexes?

2016-01-06 Thread Jim Nasby
RE LEFT(field,5) = ... -- CAN use index ... WHERE LEFT(field,6) = ... -- can NOT use index -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-general mailin

Re: [GENERAL] Definitive answer: can functions use indexes?

2016-01-06 Thread Jim Nasby
principle there could be, but I've not heard enough requests to make me think we'd ever pursue it.) BTW, the case where this would be highly valuable is timestamps. Being able to do something like date_part('month',timestamptz)='Jan' would be a big, big deal for warehousing. -- Jim Nasby, Data Archi

Re: [GENERAL] COPY FROM STDIN

2016-01-06 Thread Jim Nasby
-hand. I suggest you look at what psql does to implement \copy (note the \). -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-general mailing list

Re: [GENERAL] Trigger function interface

2016-01-06 Thread Jim Nasby
On 1/6/16 7:03 PM, Tatsuo Ishii wrote: Is it possible to get the parse tree in a C trigger function which is invoked when DML (INSERT/UPDATE/DELETE against a view) is executed? Yes, it's in fcinfo->flinfo->fn_expr. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX E

Re: [GENERAL] Efficiently selecting single row from a select with window functions row_number, lag and lead

2016-01-05 Thread Jim Nasby
re the filter can actually be pushed past the window functions to get the result you want. That Index Only Scan could still be pulling every row in the table. BTW, if you switch the order by to id, shortname then it might be able to use the index, but of course the results would be different.

Re: [GENERAL] Code of Conduct: Is it time?

2016-01-05 Thread Jim Nasby
On 1/5/16 10:03 PM, John R Pierce wrote: On 1/5/2016 5:31 PM, Jim Nasby wrote: IMHO, the real problem here is not simply a CoC, it is that the Postgres community doesn't focus on developing the community itself. The closest we come to "focus" is occasional talk on -hackers about h

Re: [GENERAL] Code of Conduct: Is it time?

2016-01-05 Thread Jim Nasby
thout active support and some encouragement things aren't going to change. [1] https://www.facebook.com/jon.erdman.jr/posts/10153828693183899 [2] http://blog.randi.io/2015/12/31/the-developer-formerly-known-as-freebsdgirl/ [3] https://opensource.com/life/15/8/couchdb-community-apache-way -- Jim Na

Re: [GENERAL] Cannot upgrade from 9.3 to 9.4 using pg_upgrade

2016-01-05 Thread Jim Nasby
On 1/4/16 7:40 PM, Adrian Klaver wrote: or even better yet could you post the section of the log above the error? The server log itself might be useful, especially if full query logging was turned on. Dunno how easy/possible that is with pg_upgrade. -- Jim Nasby, Data Architect, Blue Treble

Re: [GENERAL] Getting the function definition from oid in before firing the function

2016-01-05 Thread Jim Nasby
hook. Have you looked at https://github.com/2ndQuadrant/pgaudit? -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-general mailing list (pgsql-general

Re: [GENERAL] to_timestamp alternatives

2016-01-05 Thread Jim Nasby
gmt_date||' '||gmt_time::timestamp I suspect you need to wrap that in (). Even if the parser does the right thing there, it'd certainly make the intent a lot clearer. SELECT (gmt_date || ' ' || gmt_time)::timestamp -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics

Re: [GENERAL] COPY FROM STDIN

2016-01-05 Thread Jim Nasby
connection via SPI_connect() and then use the libpq library to issue the copy commands via PQputCopyData, PQputCopyEnd. C functions can use SPI, so I'm not sure what the issue is? http://www.postgresql.org/docs/9.5/static/spi.html (BTW, you'll want to scroll to the bottom of that page...) -- Jim

Re: [GENERAL] to_timestamp alternatives

2016-01-04 Thread Jim Nasby
nce my suggestion. Normally you'd want to figure out the correct TZ for the lat/long. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-general mailing list

Re: [GENERAL] cannot get stable function to use index

2015-12-31 Thread Jim Nasby
also set a default for than on a specific database, or a specific user, using ALTER DATABASE SET or ALTER USER SET.) -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com

Re: [GENERAL] Efficiently Triggering Autovacuum Analyze?

2015-12-31 Thread Jim Nasby
hot in cache. If you don't do that, the next query that reads the tuple will have to set the hints, which also dirties the page. VACUUM does that too, but there's really no point in having vacuum run through the entire table just to set hints on less than 1% of it. -- Jim Nasby, Data Architect

Re: [GENERAL] to_timestamp alternatives

2015-12-31 Thread Jim Nasby
this dynamically either, either by just blindly dividing longitude by 15 or using actual timezone shape polygons and @> or <@. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreb

Re: [GENERAL] cannot get stable function to use index

2015-12-29 Thread Jim Nasby
the same data to to_tsquery. Your hard-coded example is where search_vec @@ to_tsquery('213 & E & 13 & ST & N'); but your second query becomes '213 & E & 13 & ST & N:*'. Have you tried that as a hard-coded value? -- Jim Nasby, Data Architect, Blue Treble Co

Re: [GENERAL] efficient math vector operations on arrays

2015-12-29 Thread Jim Nasby
there is an internal C function array_map that can do it. There's no SQL interface to it, but it shouldn't be hard to add one. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http

Re: [GENERAL] efficient math vector operations on arrays

2015-12-29 Thread Jim Nasby
On 12/29/15 6:50 PM, Tom Lane wrote: Jim Nasby<jim.na...@bluetreble.com> writes: >BTW, if you want to simply apply a function to all elements in an array >there is an internal C function array_map that can do it. There's no SQL >interface to it, but it shouldn't be

Re: [GENERAL] Shared system resources

2015-12-25 Thread Jim Nasby
are better off concentrating on proper O/S security and user/table permissions. That is how to implement database security! True, but in my experience security audits have nothing to do with security and everything to do with marking off checkboxes and complicating lawsuits. ;) -- Jim Nasby

Re: [GENERAL] efficient math vector operations on arrays

2015-12-25 Thread Jim Nasby
On 12/24/15 1:56 AM, Pavel Stehule wrote: I don't know any extension that calculate euclid distance, but it should be trivial in C - if you don't need to use generic types and generic operations. Before messing around with that, I'd recommend trying either pl/r or pl/pythonu. -- Jim Nasby

Re: [GENERAL] Shared system resources

2015-12-23 Thread Jim Nasby
the obvious side effect of obliterating any data that was in the page. Uh, only thing is, I don't know if this is done if we're going to be returning the memory to the OS. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL

Re: [GENERAL] Table with seemingly duplicated primary key values

2015-12-22 Thread Jim Nasby
errors and your Postgres and filesystem settings for anything dangerous. Index corruption is not normal and indicates the underlying hardware or OS is faulty (or maybe a bug in Postgres, but that's very unlikely). You should also consider turning on page checksums if you haven't already. -- Jim

Re: [GENERAL] Shared system resources

2015-12-22 Thread Jim Nasby
, database processes stored in memory (memory would be a common resource here). Of far larger concern at that point is unauthorized access to the database files. Basically, if someone gains access to the OS user that Postgres is running as, or to root, it's game-over. -- Jim Nasby, Data Architect

Re: [GENERAL] Test disk reliability (or HGST HTS721010A9E630 surprisingly reliable)

2015-12-22 Thread Jim Nasby
you control it too; I know FreeBSD has support for this. (Whether the drive obeys or not is a different matter...) -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- S

Re: [GENERAL] Re: Moving Postgresql 9.1 instance from FreeBSD 9.0 machine to FreeBSD 10.2

2015-12-20 Thread Jim Nasby
like big vs little endian would be an obvious example. Postgres would detect all the obvious examples of this and refuse to start. One thing I'm not sure about is if there were different locals installed on the two machines. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts

Re: [GENERAL] Test disk reliability (or HGST HTS721010A9E630 surprisingly reliable)

2015-12-20 Thread Jim Nasby
on " Since it is not, a high end disk, I expect some errors. Why? Just because a disk isn't enterprise-grade doesn't mean it has to lie about fsync, which is the only thing diskchecker.pl tests for. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, D

Re: [GENERAL] Unique index problem

2015-12-20 Thread Jim Nasby
... -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http

Re: [GENERAL] wal receiver process always start after startup process recovering all WALs and need new WAL?

2015-12-15 Thread Jim Nasby
. Or configure WAL archiving and let the replica replay from the archive. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-general mailing list (pgsql

Re: [GENERAL] Trigger function, C, lookup of attribute type

2015-12-15 Thread Jim Nasby
/. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http

Re: [GENERAL] How to get the size of JSONB in bytes?

2015-12-15 Thread Jim Nasby
. If you're accepting data from a web form or something you certainly want it to also check things, so the user gets immediate feedback. But for anything you need to guarantee, you need to use the database. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data

Re: [GENERAL] HELP!!! The WAL Archive is taking up all space

2015-12-14 Thread Jim Nasby
On 12/9/15 7:05 PM, Andreas Kretschmer wrote: I'm really newbie to PostgreSQL but the boss pushed me to handle it >and implement it in production f*&%*$%%$#%$#

Re: [GENERAL] Overhead changing varchar(2000) to text

2015-12-14 Thread Jim Nasby
usually put some limit on fields that a webapp can write to in the database. That way a bug in the app (or malicious action) can't just start allocating gigabytes of stuff in your database. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture

Re: [GENERAL] Feature Request: Faceting for full text search

2015-12-14 Thread Jim Nasby
4) Pay one of the support companies to develop the feature. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To ma

Re: [GENERAL] Deletion Challenge

2015-12-14 Thread Jim Nasby
rtitions is enough. This project is a game, btw, described at You might be interested in https://schemaverse.com/ -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com --

Re: [GENERAL] Table with invalid page blocks

2015-12-06 Thread Jim Nasby
HERS RAISE WARNING E'Error %: %\nLast good CTID %', SQLSTATE, SQLERRM, last_good; bad := true; END; END LOOP; END; -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent

Re: [GENERAL] FATAL: unable to read data from DB node 0

2015-12-06 Thread Jim Nasby
the test because all connections it lots. I've never run pgPool myself, but maybe this means the connection was broken for some reason. Are there related errors in the Postgres server log for that node? -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data

Re: [GENERAL] plperlu stored procedure seems to freeze for a minute

2015-12-06 Thread Jim Nasby
able all signal handling during UDF execution. Just for starters, that means that if a UDF went into an infinite loop your only way to recover would be to PANIC the entire database. It would probably create a bunch of other problems as well. In other words, UDFs *must* be capable of handling an interrupt.

Re: [GENERAL] Convert from hex to string

2015-12-06 Thread Jim Nasby
there's any built-in conversion functions with the correct parameters. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-general mailing list (pgsql-general

Re: [GENERAL] JSONB performance enhancement for 9.6

2015-12-03 Thread Jim Nasby
there, so why not one specialized to Postgres? (And of course we'd want to be able to cast from that to JSON and back...) -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com

Re: [GENERAL] loading data into cluster - can I daisy-chain streaming replication?

2015-12-03 Thread Jim Nasby
A to B. But (assuming A and B are binary compatible) you'd be better off just breaking the B to C replication, setting B up as a replica of A, and then setting C up as a replica of B. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture

  1   2   3   4   5   >