Re: [GENERAL] Recover from corrupted database due to failing disk

2016-11-02 Thread Jim Nasby
On 11/2/16 6:21 PM, Jim Nasby wrote: I wouldn't trust the existing cluster that far. Since it sounds like you have no better options, you could use zero_damaged_pages to allow a pg_dumpall to complete, but you're going to end up with missing data. So what I'd suggest would be:

Re: [GENERAL] Recover from corrupted database due to failing disk

2016-11-02 Thread Jim Nasby
tions, you could use zero_damaged_pages to allow a pg_dumpall to complete, but you're going to end up with missing data. So what I'd suggest would be: stop Postgres make a copy of the cluster start with zero_damaged_pages pg_dumpall stop and remove the cluster (make sure you've got

Re: [GENERAL] Questions on Post Setup MASTER and STANDBY replication - Postgres9.1

2016-11-02 Thread Jim Nasby
setup replication? Also, it seems the startup process stucks on “recovering 00010004”, how to resolve it? As far as I know that's normal while in streaming mode. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and Postg

Re: [GENERAL] initdb createuser commands

2016-11-02 Thread Jim Nasby
arn it" are unproductive and push new users away. And we wonder why we're having trouble attracting new developers... This has actually been discussed recently on -hackers as well[1], and there is some general consensus that simplification in this area would be a good idea. 1: https

Re: [GENERAL] Checking Postgres Streaming replication delay

2016-11-02 Thread Jim Nasby
On 10/31/16 3:39 PM, Patrick B wrote: |( ||extract(epoch FROMnow())- ||extract(epoch FROMpg_last_xact_replay_timestamp()) ||)::int lag| You could certainly simplify it though... extract(epoch FROM now()-pg_last_xact_replay_timestamp()) -- Jim Nasby, Data Architect, Blue Treble Consulting

Re: [GENERAL] 2.5TB Migration from SATA to SSD disks - PostgreSQL 9.2

2016-09-24 Thread Jim Nasby
rve the integrity of the data. AFAIK pg_basebackup blindly copies all data files, while rsync will transfer only the parts of the files that have actually changed (see --block-size). If the source and destination are on different servers, that can mean less data transferred over the network.

Re: [GENERAL] overwrite column data select - Postgres 9.2

2016-09-21 Thread Jim Nasby
is then it should be NULL. Or to put it another way, having a default set largely defeats the purpose of NOT NULL (IMHO). -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com

Re: [GENERAL] Duplicate data despite unique constraint

2016-09-10 Thread Jim Nasby
On 9/2/16 8:02 AM, Adrian Klaver wrote: Best guess is the INDEX on the column is corrupted and needs to be reindexed: You should contact AWS support about this; they'd want to know. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architectur

Re: [GENERAL] What limits Postgres performance when the whole database lives in cache?

2016-09-10 Thread Jim Nasby
commercial companies. MySQL used to tout how fast it was compared to Postgres, using a benchmark it created specifically for that purpose that had very little to do with the real world. People eventually discovered that as soon as you had a concurrent workload Postgres was actually faster. -- Jim N

Re: [GENERAL] Trigger is not working for Inserts from the application

2016-09-10 Thread Jim Nasby
hat queries the app is actually running. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855-873-2532) mobile: 512-569-9461 -- Sent via pgsql-general

Re: [GENERAL] 2.5TB Migration from SATA to SSD disks - PostgreSQL 9.2

2016-09-10 Thread Jim Nasby
by up to 1 second). -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855-873-2532) mobile: 512-569-9461 -- Sent via pgsql-general mailing list (pgsql-ge

Re: [GENERAL] 2.5TB Migration from SATA to SSD disks - PostgreSQL 9.2

2016-09-07 Thread Jim Nasby
ns. Hence, rsync -va --modify-window=1 would remove your concern about a same second race condition without forcing the sync to read through all the files. Very interesting and useful! -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreS

Re: [GENERAL] What limits Postgres performance when the whole database lives in cache?

2016-09-07 Thread Jim Nasby
suck, each one just sucks in a different way." - Me, circa 1999. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855-873-2532) mobile: 512

Re: [GENERAL] IDE for function/stored proc development.

2016-09-07 Thread Jim Nasby
. Note that you'll want to create a separate sqitch migration for each object. [1] http://sqitch.org/ -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (

Re: [GENERAL] How to assemble all fields of (any) view into a string?

2016-09-07 Thread Jim Nasby
hat and converting the row to JSON would make it relatively easy to accomplish what you want in a plpgsql (or maybe even plsql) function. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://B

Re: [GENERAL] Clustered index to preserve data locality in a multitenant application?

2016-09-07 Thread Jim Nasby
ing to maintain the replication. Depending on your needs, a major benefit to this method is it makes major version upgrades very simple: you just stand up a new replica on the new version and then failover to it. If anything goes wrong, you can fail back to the old version without losing any data.

Re: [GENERAL] 2.5TB Migration from SATA to SSD disks - PostgreSQL 9.2

2016-09-07 Thread Jim Nasby
after rsync reads). You need to add the --checksum flag to rsync (which means it will still have to read everything that's in /var/lib/pgsql). -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in T

Re: [GENERAL] Privileges on public schema can't be revoked?

2016-09-07 Thread Jim Nasby
s actually relatively easy to do today; see the has_*_privilege() functions. You might also find http://pgxn.org/dist/pg_acl useful. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreb

Re: [GENERAL] Materialized view auto refresh

2016-09-07 Thread Jim Nasby
itHub (and it would be far easier for them to submit improvements). -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855-873-2532) mobile: 512-569-9461 -- Sen

Re: [GENERAL] Postgres UPGRADE from 9.2 to 9.4

2016-09-07 Thread Jim Nasby
rm of replication, so you might be better off just sticking with Postgres tools. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855-873-2532) mobile: 512-569

Re: [GENERAL] How to assemble all fields of (any) view into a string?

2016-09-07 Thread Jim Nasby
accomplish this function? Or some other way? Thanks. A PL that can accept composite types (such as plpythonu) should be able to do this. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it

Re: [GENERAL] PostgreSQL Database performance

2016-09-07 Thread Jim Nasby
ommon occurrence if it's enough to make a difference then you're already close to the limits of your IO, and if that's true then you definitely want to spread the checkpoint out over a longer interval. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, D

Re: [GENERAL] a column definition list is required for functions returning "record"

2016-09-07 Thread Jim Nasby
is something (like a function) that has explicitly > defined what the contents of the record are. We have that already, it's named 'json_each_text' Apparently you haven't looked at json parse/deparse costs ;P -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Ex

Re: [GENERAL] a column definition list is required for functions returning "record"

2016-09-02 Thread Jim Nasby
On 8/29/16 6:28 AM, Tom Lane wrote: Pavel Stehule writes: > 2016-08-29 1:59 GMT+02:00 Jim Nasby : >> It would be nice if there was a way to pass dynamically formed records >> around, similar to how you can pass the results of row() around. Someone >> else has actually be

Re: [GENERAL] LOG: could not fork new process for connection: Cannot allocate memory

2016-08-28 Thread Jim Nasby
postmaster and setting a breakpoint at ereport and then trying to connect. You could then get a backtrace; just don't leave the system in that state for long. (There might be a more elegant way to do that...) -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analyt

Re: [GENERAL] a column definition list is required for functions returning "record"

2016-08-28 Thread Jim Nasby
ure if the same is true for SQL functions. But you'd probably need a more complex query for that to be a win over the lighter weight nature of SQL functions. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Troub

Re: [GENERAL] Any reasons for 'DO' statement not returning result?

2016-08-16 Thread Jim Nasby
On 8/16/16 1:05 PM, Adrian Klaver wrote: On 08/16/2016 07:54 AM, Jim Nasby wrote: On 8/14/16 5:13 AM, Xtra Coder wrote: - ability to switch session language from 'sql' to 'pl/pgsql' Actually, something I wish I had was the ability to temporarily switch to an entirely d

Re: [GENERAL] C++ port of Postgres

2016-08-16 Thread Jim Nasby
...@bluetreble.com) -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855-873-2532) mobile: 512-569-9461 -- Sent via pgsql-general mailing list (pgsql-general

Re: [GENERAL] Any reasons for 'DO' statement not returning result?

2016-08-16 Thread Jim Nasby
reate almost *any* object in pg_temp!). What would be a lot more interesting is if creating a temp function didn't involve writing an entry to the catalog (something being discussed for temp tables right now). -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analy

Re: [GENERAL] Jsonb extraction very slow

2016-08-16 Thread Jim Nasby
ed to duplicate everything... and in fact might be able to do it's expansion in the original object so that subsequent references to that key wouldn't need to re-expand it. I don't think the current EO framework supports that, but it doesn't seem impossible to add... -- Jim Nasby, Da

Re: [GENERAL] Jsonb extraction very slow

2016-08-16 Thread Jim Nasby
On 8/11/16 8:45 AM, Tom Lane wrote: Jim Nasby writes: I never dug into why. As Tom posited, decompression might explain the time to get a single key out. Getting 10 keys instead of just 1 wasn't 10x more expensive, but it was significantly more expensive than just getting a single key.

Re: [GENERAL] Any reasons for 'DO' statement not returning result?

2016-08-16 Thread Jim Nasby
nnection and context. That would be especially useful for debugging plpython functions. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855-873-2532)

Re: [GENERAL] Jsonb extraction very slow

2016-08-11 Thread Jim Nasby
t 1 wasn't 10x more expensive, but it was significantly more expensive than just getting a single key. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-T

Re: [GENERAL] Jsonb extraction very slow

2016-08-10 Thread Jim Nasby
ts (to improve compression), but I never got around to actually tracing it. I suspect there's a win to be had by having both json types use the ExpandedObject stuff. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Tr

Re: [GENERAL] Any reasons for 'DO' statement not returning result?

2016-08-10 Thread Jim Nasby
;s not quite as convenient as DO, and you also must ALWAYS provide at least one correctly typed input (even if it's NULL) so the pseudotype will work. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treb

[GENERAL] plpython.h not installed in 9.4

2016-08-10 Thread Jim Nasby
9.5+ installs plpython.h under include/server. 9.4 apparently doesn't. I'm guessing that changed in 9.5? Or am I doing something wrong? -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Tr

Re: [GENERAL] Array value from table as parameter

2016-07-22 Thread Jim Nasby
Please create a stand-alone scenario that demonstrates the problem you're seeing. I suspect that in the process of doing that you're going to uncover a bug in your code, but if not then we'll have something concrete we can look at. -- Jim Nasby, Data Architect, Blue Treble Co

Re: [GENERAL] unique constraint with several null values

2016-07-22 Thread Jim Nasby
you could then do a unique index on that WHERE array != array[]. Maybe a less obtuse option would be to use a boolean array. Storage would be ~8x larger, but since there should be very few rows I doubt that matters. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts

Re: [GENERAL] Constraint using a SQL function executed during SELECT

2016-07-19 Thread Jim Nasby
not exist ... CONTEXT: SQL function "f" during inlining In this example, you should be able to avoid that by setting constraint_exclusion=off. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Ge

Re: [GENERAL] Stored procedure version control

2016-07-01 Thread Jim Nasby
have good test practices (as in, database unit tests). In 9 years in an environment where downtime was 6 figures per hour I only had 1 or 2 deployments that had problems, and never bad enough to consider reverting. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analyti

Re: [GENERAL] How safe is pg_basebackup + continuous archiving?

2016-07-01 Thread Jim Nasby
tunately. If someone did want that though, it could probably be done as an extension. I believe you just have to pull all of each relation into shared buffers for the checksums to be verified. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architec

Re: [GENERAL] [HACKERS] sign function with INTERVAL?

2016-04-13 Thread Jim Nasby
nically difficult to make it "just work"? Actually, after looking at the code for interval_lt, all that needs to happen to add this support is to expose interval_cmp_internal() as a strict function. It already does exactly what you want. -- Jim Nasby, Data Architect, Blue Treble C

[GENERAL] Odd quoting behavior from \set

2016-01-19 Thread Jim Nasby
Is this odd quoting behavior expected? (Note difference between "'a':1" and "'b':'a'") ~@decina.local/53896# \set df pd.DataFrame.from_dict([{'a':1,'b':'a'},{'a':2,'b':'b'}]) ~@decina

Re: [GENERAL] Execute commands in single-user mode

2016-01-18 Thread Jim Nasby
ife as add-ons and were eventually pulled in because they became extremely popular. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-general mailing list (

Re: [GENERAL] Unable to build python extension with PGXS

2016-01-15 Thread Jim Nasby
On 1/13/16 3:11 PM, Jim Nasby wrote: On 1/12/16 10:04 PM, Jim Nasby wrote: Attempting to build a python extension, I'm getting: Undefined symbols for architecture x86_64: "_PyErr_Clear", referenced from: _PLyNdarray_FromDatum in pg_ndarray.o _PLyObjec

Re: [GENERAL] Moving a large DB (> 500GB) to another DB with different locale

2016-01-13 Thread Jim Nasby
On 1/13/16 2:39 PM, Andreas Joseph Krogh wrote: Where can I find more info about how to use and configure pg_logical to replicate a 9.4 DB to 9.5? http://2ndquadrant.com/en/resources/pglogical/ -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data

Re: [GENERAL] Unable to build python extension with PGXS

2016-01-13 Thread Jim Nasby
On 1/12/16 10:04 PM, Jim Nasby wrote: Attempting to build a python extension, I'm getting: Undefined symbols for architecture x86_64: "_PyErr_Clear", referenced from: _PLyNdarray_FromDatum in pg_ndarray.o _PLyObject_To_ndarray in pg_ndarray.o "_P

Re: [GENERAL] Changing varchar length by manipulating pg_attribute

2016-01-13 Thread Jim Nasby
ypmod to -1 at the same time if you do that. Obviously you should test all of this thoroughly before doing it in production. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble

Re: [GENERAL] Moving a large DB (> 500GB) to another DB with different locale

2016-01-13 Thread Jim Nasby
x27;d use pg_logical. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to you

Re: [GENERAL] Call postgres PL/Python stored function from another PL/Python block.

2016-01-13 Thread Jim Nasby
html#AEN65599 for more information. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make

[GENERAL] Unable to build python extension with PGXS

2016-01-12 Thread Jim Nasby
erenced from: _PLyObject_To_ndarray in pg_ndarray.o ld: symbol(s) not found for architecture x86_64 I've included $(python_includespec) in my Makefile: override CPPFLAGS := $(python_includespec) $(CPPFLAGS) Is there some other magic I need? Do I need to switch to using MODULE_big or so

[GENERAL] New hacker item posted

2016-01-11 Thread Jim Nasby
Anyone looking to get their feet wet in the backend code, please take a look at http://www.postgresql.org/message-id/568f03ef.4070...@bluetreble.com. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in

Re: [GENERAL] Trigger function interface

2016-01-10 Thread Jim Nasby
is this the parse tree for the top level query which involves the trigger? Hrm, apparently not. fcinfo->context would maybe be helpful, but I'm not sure. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Troubl

Re: [GENERAL] Using xmax to detect deleted rows

2016-01-10 Thread Jim Nasby
27;ll be a lot simpler to just do whatever you need to do when the row is actually deleted. Just be sure you deal with rollbacks correctly if you're doing something external. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Dat

Re: [GENERAL] Code of Conduct: Is it time?

2016-01-10 Thread Jim Nasby
away from reporting. Also, not allowing your CoC to become a weapon that someone can use offensively. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgs

Re: [GENERAL] Execute commands in single-user mode

2016-01-10 Thread Jim Nasby
some other issues with large objects, notably their use of OIDs. Lots of LOs can lead to OID depletion. There was a thread about this recently. It might be about time to come up with an extension that's a replacement for large objects. -- Jim Nasby, Data Architect, Blue Treble Consul

Re: [GENERAL] COPY FROM STDIN

2016-01-10 Thread Jim Nasby
atter would be much appreciated. I don't know off-hand. I suggest you look at what psql does to implement \copy (note the \). -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTre

Re: [GENERAL] COPY FROM STDIN

2016-01-06 Thread Jim Nasby
t know off-hand. I suggest you look at what psql does to implement \copy (note the \). -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-general ma

Re: [GENERAL] Trigger function interface

2016-01-06 Thread Jim Nasby
On 1/6/16 7:03 PM, Tatsuo Ishii wrote: Is it possible to get the parse tree in a C trigger function which is invoked when DML (INSERT/UPDATE/DELETE against a view) is executed? Yes, it's in fcinfo->flinfo->fn_expr. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX

Re: [GENERAL] Definitive answer: can functions use indexes?

2016-01-06 Thread Jim Nasby
res. (In principle there could be, but I've not heard enough requests to make me think we'd ever pursue it.) BTW, the case where this would be highly valuable is timestamps. Being able to do something like date_part('month',timestamptz)='Jan' would be a big, big de

Re: [GENERAL] Definitive answer: can functions use indexes?

2016-01-06 Thread Jim Nasby
e index ... WHERE LEFT(field,5) = ... -- CAN use index ... WHERE LEFT(field,6) = ... -- can NOT use index -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql

Re: [GENERAL] Code of Conduct: Is it time?

2016-01-06 Thread Jim Nasby
useless for dealing with an actual situation. (BTW, if your concern on enforcement is about control, not only can people be removed from mailing lists and the like, but there actually is a Postgres legal entity that could start legal proceedings if it ever came to it.) [1] http://couchdb.apache.

Re: [GENERAL] Code of Conduct: Is it time?

2016-01-06 Thread Jim Nasby
owed things like a planner written in another language (which with the planner hooks might actually be possible). -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent

Re: [GENERAL] Code of Conduct: Is it time?

2016-01-06 Thread Jim Nasby
s one. Please, ask your questions! The Postgres community really is one of the most patient and helpful OSS communities out there, and there's plenty of people that would be happy to explain things. Questions are also a good way to show where things could possibly be better commented/docum

Re: [GENERAL] Efficiently selecting single row from a select with window functions row_number, lag and lead

2016-01-05 Thread Jim Nasby
I'm not sure the filter can actually be pushed past the window functions to get the result you want. That Index Only Scan could still be pulling every row in the table. BTW, if you switch the order by to id, shortname then it might be able to use the index, but of course the results would

Re: [GENERAL] Code of Conduct: Is it time?

2016-01-05 Thread Jim Nasby
On 1/5/16 10:03 PM, John R Pierce wrote: On 1/5/2016 5:31 PM, Jim Nasby wrote: IMHO, the real problem here is not simply a CoC, it is that the Postgres community doesn't focus on developing the community itself. The closest we come to "focus" is occasional talk on -hackers ab

Re: [GENERAL] COPY FROM STDIN

2016-01-05 Thread Jim Nasby
ng db connection via SPI_connect() and then use the libpq library to issue the copy commands via PQputCopyData, PQputCopyEnd. C functions can use SPI, so I'm not sure what the issue is? http://www.postgresql.org/docs/9.5/static/spi.html (BTW, you'll want to scroll to the bottom of that

Re: [GENERAL] to_timestamp alternatives

2016-01-05 Thread Jim Nasby
ad() either: SELECT gmt_date||' '||gmt_time::timestamp I suspect you need to wrap that in (). Even if the parser does the right thing there, it'd certainly make the intent a lot clearer. SELECT (gmt_date || ' ' || gmt_time)::timestamp -- Jim Nasby, Data Architect, Blue

Re: [GENERAL] Getting the function definition from oid in before firing the function

2016-01-05 Thread Jim Nasby
ith an executor hook. Have you looked at https://github.com/2ndQuadrant/pgaudit? -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-general mailing list

Re: [GENERAL] Cannot upgrade from 9.3 to 9.4 using pg_upgrade

2016-01-05 Thread Jim Nasby
On 1/4/16 7:40 PM, Adrian Klaver wrote: or even better yet could you post the section of the log above the error? The server log itself might be useful, especially if full query logging was turned on. Dunno how easy/possible that is with pg_upgrade. -- Jim Nasby, Data Architect, Blue Treble

Re: [GENERAL] Code of Conduct: Is it time?

2016-01-05 Thread Jim Nasby
in the community that would be interested in doing that, but without active support and some encouragement things aren't going to change. [1] https://www.facebook.com/jon.erdman.jr/posts/10153828693183899 [2] http://blog.randi.io/2015/12/31/the-developer-formerly-known-as-freebsdgirl/ [3] h

Re: [GENERAL] to_timestamp alternatives

2016-01-04 Thread Jim Nasby
re DST, hence my suggestion. Normally you'd want to figure out the correct TZ for the lat/long. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-genera

Re: [GENERAL] to_timestamp alternatives

2015-12-31 Thread Jim Nasby
g, it shouldn't be hard to do this dynamically either, either by just blindly dividing longitude by 15 or using actual timezone shape polygons and @> or <@. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Tr

Re: [GENERAL] cannot get stable function to use index

2015-12-31 Thread Jim Nasby
u can also set a default for than on a specific database, or a specific user, using ALTER DATABASE SET or ALTER USER SET.) -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTrebl

Re: [GENERAL] Efficiently Triggering Autovacuum Analyze?

2015-12-31 Thread Jim Nasby
hile the pages are still hot in cache. If you don't do that, the next query that reads the tuple will have to set the hints, which also dirties the page. VACUUM does that too, but there's really no point in having vacuum run through the entire table just to set hints on less than 1% o

Re: [GENERAL] efficient math vector operations on arrays

2015-12-29 Thread Jim Nasby
On 12/29/15 6:50 PM, Tom Lane wrote: Jim Nasby writes: >BTW, if you want to simply apply a function to all elements in an array >there is an internal C function array_map that can do it. There's no SQL >interface to it, but it shouldn't be hard to add one. That wouldn&

Re: [GENERAL] efficient math vector operations on arrays

2015-12-29 Thread Jim Nasby
nts in an array there is an internal C function array_map that can do it. There's no SQL interface to it, but it shouldn't be hard to add one. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Troubl

Re: [GENERAL] cannot get stable function to use index

2015-12-29 Thread Jim Nasby
text))) Part of this could well be that you're not feeding the same data to to_tsquery. Your hard-coded example is where search_vec @@ to_tsquery('213 & E & 13 & ST & N'); but your second query becomes '213 & E & 13 & ST & N:*'. Have

Re: [GENERAL] Shared system resources

2015-12-25 Thread Jim Nasby
. You are better off concentrating on proper O/S security and user/table permissions. That is how to implement database security! True, but in my experience security audits have nothing to do with security and everything to do with marking off checkboxes and complicating lawsuits. ;) -- Jim

Re: [GENERAL] efficient math vector operations on arrays

2015-12-25 Thread Jim Nasby
On 12/24/15 1:56 AM, Pavel Stehule wrote: I don't know any extension that calculate euclid distance, but it should be trivial in C - if you don't need to use generic types and generic operations. Before messing around with that, I'd recommend trying either pl/r or pl/pythonu

Re: [GENERAL] Shared system resources

2015-12-23 Thread Jim Nasby
e to help hunt down memory access bugs, but would have the obvious side effect of obliterating any data that was in the page. Uh, only thing is, I don't know if this is done if we're going to be returning the memory to the OS. -- Jim Nasby, Data Architect, Blue Treble Consulting,

Re: [GENERAL] Shared system resources

2015-12-22 Thread Jim Nasby
data, database processes stored in memory (memory would be a common resource here). Of far larger concern at that point is unauthorized access to the database files. Basically, if someone gains access to the OS user that Postgres is running as, or to root, it's game-over. -- Jim Nasby, Data

Re: [GENERAL] Table with seemingly duplicated primary key values

2015-12-22 Thread Jim Nasby
errors and your Postgres and filesystem settings for anything dangerous. Index corruption is not normal and indicates the underlying hardware or OS is faulty (or maybe a bug in Postgres, but that's very unlikely). You should also consider turning on page checksums if you haven't alrea

Re: [GENERAL] Test disk reliability (or HGST HTS721010A9E630 surprisingly reliable)

2015-12-22 Thread Jim Nasby
r OS might let you control it too; I know FreeBSD has support for this. (Whether the drive obeys or not is a different matter...) -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTr

Re: [GENERAL] Test disk reliability (or HGST HTS721010A9E630 surprisingly reliable)

2015-12-20 Thread Jim Nasby
n " Since it is not, a high end disk, I expect some errors. Why? Just because a disk isn't enterprise-grade doesn't mean it has to lie about fsync, which is the only thing diskchecker.pl tests for. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in A

Re: [GENERAL] Re: Moving Postgresql 9.1 instance from FreeBSD 9.0 machine to FreeBSD 10.2

2015-12-20 Thread Jim Nasby
like big vs little endian would be an obvious example. Postgres would detect all the obvious examples of this and refuse to start. One thing I'm not sure about is if there were different locals installed on the two machines. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Ex

Re: [GENERAL] Unique index problem

2015-12-20 Thread Jim Nasby
gle values... -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscrip

Re: [GENERAL] wal receiver process always start after startup process recovering all WALs and need new WAL?

2015-12-15 Thread Jim Nasby
segments that are needed. Or configure WAL archiving and let the replica replay from the archive. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-ge

Re: [GENERAL] How to get the size of JSONB in bytes?

2015-12-15 Thread Jim Nasby
laces. If you're accepting data from a web form or something you certainly want it to also check things, so the user gets immediate feedback. But for anything you need to guarantee, you need to use the database. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analy

Re: [GENERAL] Trigger function, C, lookup of attribute type

2015-12-15 Thread Jim Nasby
t.org/. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription:

Re: [GENERAL] Feature Request: Faceting for full text search

2015-12-14 Thread Jim Nasby
Pay one of the support companies to develop the feature. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org)

Re: [GENERAL] Overhead changing varchar(2000) to text

2015-12-14 Thread Jim Nasby
s worth, I usually put some limit on fields that a webapp can write to in the database. That way a bug in the app (or malicious action) can't just start allocating gigabytes of stuff in your database. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data A

Re: [GENERAL] HELP!!! The WAL Archive is taking up all space

2015-12-14 Thread Jim Nasby
just isn't that important (as hard as database people find that to believe!). If that's your case then you can probably just turn off archiving and not worry about it. Finally, as someone else said, *a replica is NOT a backup!* -- Jim Nasby, Data Architect, Blue Treble Consulting, Aus

Re: [GENERAL] Deletion Challenge

2015-12-14 Thread Jim Nasby
;active" and "history" partitions is enough. This project is a game, btw, described at You might be interested in https://schemaverse.com/ -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get i

Re: [GENERAL] FATAL: unable to read data from DB node 0

2015-12-06 Thread Jim Nasby
b stop the test because all connections it lots. I've never run pgPool myself, but maybe this means the connection was broken for some reason. Are there related errors in the Postgres server log for that node? -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in

Re: [GENERAL] Convert from hex to string

2015-12-06 Thread Jim Nasby
ng and I don't think there's any built-in conversion functions with the correct parameters. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-gener

Re: [GENERAL] plperlu stored procedure seems to freeze for a minute

2015-12-06 Thread Jim Nasby
during UDF execution. Just for starters, that means that if a UDF went into an infinite loop your only way to recover would be to PANIC the entire database. It would probably create a bunch of other problems as well. In other words, UDFs *must* be capable of handling an interrupt. -- Jim

Re: [GENERAL] Table with invalid page blocks

2015-12-06 Thread Jim Nasby
d; EXCEPTION WHEN OTHERS RAISE WARNING E'Error %: %\nLast good CTID %', SQLSTATE, SQLERRM, last_good; bad := true; END; END LOOP; END; -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! h

Re: [GENERAL] fast refresh materialized view

2015-12-03 Thread Jim Nasby
internally, and maybe your work will help that. What license is it released under? -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-general mailing list

Re: [GENERAL] Can row level security policies also be implemented for views?

2015-12-03 Thread Jim Nasby
ard select statement (possibly a dynamic one via EXECUTE) from the SRF, in the hope that the system would just consider that to be a plain-old SELECT against the table, allowing RLS to function. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and

  1   2   3   4   5   >