Re: [GENERAL] Multimaster

2016-04-10 Thread Dorian Hoxha
@Konstantin
1. It's ok in my cases.
2. Not required in my cases.
3. Just require users to use different servers for now I think.
Sometimes(always?) users can be greedy with feature requests.
4. I want magically consistency + failover (I can instruct the client to
retry all masters).

Good-cluster is the only thing that is stopping postgresql from killing
many nosqls in my opinion. Citusdb is single-master(can't even update many
rows on the same shard), Postgres-XL has no highavailibility (and having 3
servers per node + global dtm), GreenPlum is olap(and old version of
postgres), app-side sucks.

With sharding, the holy-grail is to have
automatic-global-range-partitioning (hypertable,hbase,cockroachdb,bigtable
etc) on the primary-key or a special column. The con of this, is that
indexes of a row reside in another region, which may mean another server,
which you need cross-server-transaction to update a row.
In this case, I think you can save the indexes on the same server as the
row and be fine.

To have good speed you can implement: 1-shard-only-queries (no cross-nodes
transaction), async-disk changes (so the changes are done only in the
memory of all replicated servers and not in commit-log (used by couchbase)).

Multi-shard transactions aren't needed as much in my opinion.

5. The fewer places/files there are to configure any system the better it
is IMHO.

Cheers and good luck!

On Fri, Apr 1, 2016 at 8:15 PM, Moreno Andreo 
wrote:

> Il 01/04/2016 17:22, Joshua D. Drake ha scritto:
>
>> On 04/01/2016 04:50 AM, Konstantin Knizhnik wrote:
>>
>> There are also some minor technical issues which lead us to making few
>>> changes in pglogical code but we tried to do our best to keep original
>>> versions unchanged, so we can switch to public version in future.
>>>
>>
>> It seems the better solution for all of this would be for BDR and this
>> project to come together as a community project to get multimaster for
>> PostgreSQL. Otherwise we are going to end up with the same situation we had
>> before Pg had master-slave replication.
>>
>> JD
>>
>> I'd like to post my use case, hoping it could be of any interest.
> We are giving a cloud service, based on our professional application,
> hosting databases on our server and replicating them to every site the
> users register with.
> Every user have (for now) a unique database to keep in sync.
> Users write on their local database and data is replicated with the server
> via a background process running while user is connected.
> Actually we have to improve what our replicator is doing: it's only
> replicating the single user's database. The improvement should that we can
> put it on the "server" (in some cases there are groups of users sharing a
> dedicated server) and, given a configuration of what and how to replicate,
> it should replicate more than one DB a time.
> Actually, it's a stand-alone program, but what we'd like would be
> something more integrated in PostgreSQL, so where PostgreSQL can run
> (everywhere!), so can it.
> We were beginning to "translate" (and then improve) this program in c#,
> when I bumped into articles pointing to BDR, and I started taking a look.
> But it seems that is good to replicahe whole servers, and still hasn't the
> granularity we need.
>
> My 2 cent...
>
> Cheers,
> Moreno.
>
>
>
>
> --
> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
>


Re: [GENERAL] Bypassing NULL elements in row_to_json function

2016-04-10 Thread Adrian Klaver

On 04/10/2016 09:24 AM, David G. Johnston wrote:

On Sun, Apr 10, 2016 at 8:39 AM, Michael Nolan >wrote:

Here's what I did:

  \d gold1604_test
Table "uscf.gold1604_test"
  Column | Type | Modifiers
+--+---
  data   | json |

Some sample data:
  {"id":"1001","name":"MISNER, J
NATHAN","st":"NY","exp":"2012-05-31","sts":
"A"} +

   {"id":"1002","name":"MISNER,
JUDY","st":"TN","exp":"2007-07-31","sts":"I"}
  +

   {"id":"1003","name":"MISNER, J
AMSCHEL","st":"NY","exp":"2007-05-31","sts"
:"A"}+


​(I think) PostgreSQL assumes that there is only a single top-level json
element, whether it be an array or an object.  The first thing you'd
have to do is split on the newline and create a PostgreSQL text array.


But,
uscf=> insert into goldmast_test select * from
json_populate_record(NULL::"goldmast_test", (select * from
gold1604_test limit 2) )
uscf-> \g
ERROR:  more than one row returned by a subquery used as an expression

Is there a way to get around the one row per subquery issue?


​Yes, use LATERAL.

Something like the following should work (not tested):

INSERT INTO goldmast_test
SELECT jpr.*
FROM gold1604_test
LATERAL json_populate_record(null::goldmast_test", data) AS jpr


I can confirm this works after a little clean up:

test=> INSERT INTO goldmast_test
SELECT jpr.*
FROM gold1604_test,
LATERAL json_populate_record(null::"goldmast_test", data) AS jpr;
INSERT 0 3

test=> select * from goldmast_test ;
id|   name| st |exp | sts | supp | rrtg | 
qrtg | brtg | oqrtg | obrtg | fid

--+---+++-+--+--+--+--+---+---+-
 1001 | MISNER, J NATHAN  | NY | 2012-05-31 | A   |  |  | 
|  |   |   |
 1002 | MISNER, JUDY  | TN | 2007-07-31 | I   |  |  | 
|  |   |   |
 1003 | MISNER, J AMSCHEL | NY | 2007-05-31 | A   |  |  | 
|  |   |   |

(3 rows)

While trying to figure out how it works I discovered the LATERAL is not 
necessary:


test=> INSERT INTO goldmast_test
SELECT jpr.*
FROM gold1604_test,
json_populate_record(null::"goldmast_test", data) AS jpr;
INSERT 0 3

test=> select * from goldmast_test ;
id|   name| st |exp | sts | supp | rrtg | 
qrtg | brtg | oqrtg | obrtg | fid

--+---+++-+--+--+--+--+---+---+-
 1001 | MISNER, J NATHAN  | NY | 2012-05-31 | A   |  |  | 
|  |   |   |
 1002 | MISNER, JUDY  | TN | 2007-07-31 | I   |  |  | 
|  |   |   |
 1003 | MISNER, J AMSCHEL | NY | 2007-05-31 | A   |  |  | 
|  |   |   |
 1001 | MISNER, J NATHAN  | NY | 2012-05-31 | A   |  |  | 
|  |   |   |
 1002 | MISNER, JUDY  | TN | 2007-07-31 | I   |  |  | 
|  |   |   |
 1003 | MISNER, J AMSCHEL | NY | 2007-05-31 | A   |  |  | 
|  |   |   |

(6 rows)




ideally you could just do (not tested):

INSERT INTO goldmast_test
SELECT jpr.*
FROM (SELECT t::json FROM unnest(string_to_array(?::text, E'\n') un
(t))) src j
LATERAL json_populate_record(null::goldmast_test", j) AS jpr

Where the "?::text" is placeholder for the textual JSON being handed to
the query thus avoiding the temporary "gold1604_test" table.

David J.




--
Adrian Klaver
adrian.kla...@aklaver.com


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Bypassing NULL elements in row_to_json function

2016-04-10 Thread Michael Nolan
In case it wasn't clear, the sample data was 3 rows of data.  (There are
actually around 890K rows in the table pgfutter built from the JSON file.)
-
Mike Nolan


Re: [GENERAL] Bypassing NULL elements in row_to_json function

2016-04-10 Thread David G. Johnston
On Sun, Apr 10, 2016 at 8:39 AM, Michael Nolan  wrote:

> Here's what I did:
>
>  \d gold1604_test
> Table "uscf.gold1604_test"
>  Column | Type | Modifiers
> +--+---
>  data   | json |
>
> Some sample data:
>  {"id":"1001","name":"MISNER, J
> NATHAN","st":"NY","exp":"2012-05-31","sts":
> "A"} +
>
>   {"id":"1002","name":"MISNER,
> JUDY","st":"TN","exp":"2007-07-31","sts":"I"}
>  +
>
>   {"id":"1003","name":"MISNER, J
> AMSCHEL","st":"NY","exp":"2007-05-31","sts"
> :"A"}+
>
>
​(I think) PostgreSQL assumes that there is only a single top-level json
element, whether it be an array or an object.  The first thing you'd have
to do is split on the newline and create a PostgreSQL text array.

>
> But,
> uscf=> insert into goldmast_test select * from
> json_populate_record(NULL::"goldmast_test", (select * from gold1604_test
> limit 2) )
> uscf-> \g
> ERROR:  more than one row returned by a subquery used as an expression
>
> Is there a way to get around the one row per subquery issue?
>

​Yes, use LATERAL.

Something like the following should work (not tested):

INSERT INTO goldmast_test
SELECT jpr.*
FROM gold1604_test
LATERAL json_populate_record(null::goldmast_test", data) AS jpr

ideally you could just do (not tested):

INSERT INTO goldmast_test
SELECT jpr.*
FROM (SELECT t::json FROM unnest(string_to_array(?::text, E'\n') un (t)))
src j
LATERAL json_populate_record(null::goldmast_test", j) AS jpr

Where the "?::text" is placeholder for the textual JSON being handed to the
query thus avoiding the temporary "gold1604_test" table.

David J.


Re: [GENERAL] Really unique session ID - PID + connection timestamp?

2016-04-10 Thread Adrian Klaver

On 04/10/2016 06:29 AM, durumd...@gmail.com wrote:


Dear Alban!


2016.04.10. 13:05 keltezéssel, Alban Hertroys írta:

On 10 Apr 2016, at 9:07, Durumdara  wrote:

Dear Adrian!

Again. As I see the beginning blocks are removed by mailing system in
the code.

We have an "ourlocks" table which hold records (TableName, RecordID,
SessionInnerID, TimeStamp, etc, with TableName/RecordID prikey).

If anybody wants to lock record "for long time", "over the
transactions" it try to insert a new record here.

Why are those records being locked? Reading on, it seems like you're
trying to solve a fairly standard concurrency problem. Any RDBMS worth
their salt can handle that for you, you don't need to manually do any
of that.


This is not real locks. They are logical locks.
Products, offers are edited for long time.


Define long time, a session, a day, days, etc?


But we must save subdata. This is not a "word like document" which can
saved at once, in a transaction.
When a product edited, we must protect it from other user's edit.
But it's subdata must be posted/commited to the DB, for example
shipping, article quantity changes, vouchers, etc.


So folks can make changes to the attributes of a Product, Offer, etc 
while it is being changed in ways they can not see?


Or do they get a read only view that changes as the 'locking' user makes 
edits?








This sounds much more like a use-case for sub-transactions and select
for update (which puts a temporary RDBMS-controlled lock on the
relevant records) than for manual locking.


Yes, this is like sub-transaction.
But for only sub-data. The main data must be edited by only the first
user who started the edit.
This is a long time "lock" like thing. This what we simulate here.

Thanks for your suggestions. I will check this in our client library.


To be clear you are trying to come up with a solution that allows your 
application to run against different databases(Firebird, SQL Server, 
Postgres, etc?), using a single code base, correct?




dd





--
Adrian Klaver
adrian.kla...@aklaver.com


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Bypassing NULL elements in row_to_json function

2016-04-10 Thread Adrian Klaver

On 04/10/2016 08:39 AM, Michael Nolan wrote:

Here's what I did:

  \d gold1604_test
Table "uscf.gold1604_test"
  Column | Type | Modifiers
+--+---
  data   | json |

Some sample data:
  {"id":"1001","name":"MISNER, J
NATHAN","st":"NY","exp":"2012-05-31","sts":
"A"} +

   {"id":"1002","name":"MISNER,
JUDY","st":"TN","exp":"2007-07-31","sts":"I"}
  +

   {"id":"1003","name":"MISNER, J
AMSCHEL","st":"NY","exp":"2007-05-31","sts"
:"A"}+


uscf-> \d goldmast_test
  Table "uscf.goldmast_test"
  Column | Type  | Modifiers
+---+---
  id | character varying(8)  |
  name   | character varying(40) |
  st | character varying(2)  |
  exp| date  |
  sts| character(1)  |
  supp   | date  |
  rrtg   | character varying(8)  |
  qrtg   | character varying(8)  |
  brtg   | character varying(8)  |
  oqrtg  | character varying(8)  |
  obrtg  | character varying(8)  |
  fid| character varying(12) |




insert into goldmast_test select * from
json_populate_record(NULL::"goldmast_test", (select * from gold1604_test
limit 1) )
  produces:
uscf=> select * from goldmast_test;
 id|   name   | st |exp | sts | supp | rrtg |
qrtg | brtg
  | oqrtg | obrtg | fid
--+--+++-+--+--+--+-
-+---+---+-
  1001 | MISNER, J NATHAN | NY | 2012-05-31 | A   |  |  |  |
  |   |   |
(1 row)

The fact that the null values were stripped out is not an issue here.

But,
uscf=> insert into goldmast_test select * from
json_populate_record(NULL::"goldmast_test", (select * from gold1604_test
limit 2) )
uscf-> \g
ERROR:  more than one row returned by a subquery used as an expression

Is there a way to get around the one row per subquery issue?


Per Davids post:

http://www.postgresql.org/docs/9.5/interactive/functions-json.html

json_populate_recordset(base anyelement, from_json json)

Expands the outermost array of objects in from_json to a set of rows 
whose columns match the record type defined by base (see note below). 
select * from json_populate_recordset(null::myrowtype, 
'[{"a":1,"b":2},{"a":3,"b":4}]')



--
Mike Nolan



--
Adrian Klaver
adrian.kla...@aklaver.com


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Bypassing NULL elements in row_to_json function

2016-04-10 Thread Michael Nolan
Here's what I did:

 \d gold1604_test
Table "uscf.gold1604_test"
 Column | Type | Modifiers
+--+---
 data   | json |

Some sample data:
 {"id":"1001","name":"MISNER, J
NATHAN","st":"NY","exp":"2012-05-31","sts":
"A"} +

  {"id":"1002","name":"MISNER,
JUDY","st":"TN","exp":"2007-07-31","sts":"I"}
 +

  {"id":"1003","name":"MISNER, J
AMSCHEL","st":"NY","exp":"2007-05-31","sts"
:"A"}+


uscf-> \d goldmast_test
 Table "uscf.goldmast_test"
 Column | Type  | Modifiers
+---+---
 id | character varying(8)  |
 name   | character varying(40) |
 st | character varying(2)  |
 exp| date  |
 sts| character(1)  |
 supp   | date  |
 rrtg   | character varying(8)  |
 qrtg   | character varying(8)  |
 brtg   | character varying(8)  |
 oqrtg  | character varying(8)  |
 obrtg  | character varying(8)  |
 fid| character varying(12) |




insert into goldmast_test select * from
json_populate_record(NULL::"goldmast_test", (select * from gold1604_test
limit 1) )
 produces:
uscf=> select * from goldmast_test;
id|   name   | st |exp | sts | supp | rrtg | qrtg |
brtg
 | oqrtg | obrtg | fid
--+--+++-+--+--+--+-
-+---+---+-
 1001 | MISNER, J NATHAN | NY | 2012-05-31 | A   |  |  |  |
 |   |   |
(1 row)

The fact that the null values were stripped out is not an issue here.

But,
uscf=> insert into goldmast_test select * from
json_populate_record(NULL::"goldmast_test", (select * from gold1604_test
limit 2) )
uscf-> \g
ERROR:  more than one row returned by a subquery used as an expression

Is there a way to get around the one row per subquery issue?
--
Mike Nolan


Re: [GENERAL] Really unique session ID - PID + connection timestamp?

2016-04-10 Thread David G. Johnston
On Sun, Apr 10, 2016 at 6:29 AM, durumd...@gmail.com 
wrote:

>
> Dear Alban!
>
>
> 2016.04.10. 13:05 keltezéssel, Alban Hertroys írta:
>
>> On 10 Apr 2016, at 9:07, Durumdara  wrote:
>>>
>>> Dear Adrian!
>>>
>>> Again. As I see the beginning blocks are removed by mailing system in
>>> the code.
>>>
>>> We have an "ourlocks" table which hold records (TableName, RecordID,
>>> SessionInnerID, TimeStamp, etc, with TableName/RecordID prikey).
>>>
>>> If anybody wants to lock record "for long time", "over the transactions"
>>> it try to insert a new record here.
>>>
>> Why are those records being locked? Reading on, it seems like you're
>> trying to solve a fairly standard concurrency problem. Any RDBMS worth
>> their salt can handle that for you, you don't need to manually do any of
>> that.
>>
>
> This is not real locks. They are logical locks.
> Products, offers are edited for long time.
> But we must save subdata. This is not a "word like document" which can
> saved at once, in a transaction.
> When a product edited, we must protect it from other user's edit.
> But it's subdata must be posted/commited to the DB, for example shipping,
> article quantity changes, vouchers, etc.
>
>
>
>
>> This sounds much more like a use-case for sub-transactions and select for
>> update (which puts a temporary RDBMS-controlled lock on the relevant
>> records) than for manual locking.
>>
>
> Yes, this is like sub-transaction.
> But for only sub-data. The main data must be edited by only the first user
> who started the edit.
> This is a long time "lock" like thing. This what we simulate here.
>
> Thanks for your suggestions. I will check this in our client library.
>

​I get what you are trying to do, and was trying to figure out a way to
make "FOR UPDATE" and "SKIP LOCKED" or "NOWAIT" work in concert to solve
the problem.  I'm not familiar enough with the problem to have come up with
a viable solution.  But I kept coming back to the theory that maybe making
the database solve the "long running transaction" problem isn't the best
idea.  I'd tend to gravitate toward having a dedicated "librarian"
application who is responsible accepting checkout (select) requests and
processing returns (updates) over the relevant data.

PostgreSQL doesn't make doing pessimistic concurrency via long-running
transactions that fun to implement - namely because it is not very scalable
and can wreck havoc on the system if done improperly as many of the
maintenance routines can be prevented from doing their work.  That said its
likely there is some way to accomplish your goal.

In fact, I just remembered that we implemented "advisory locks" for just
that reason.

http://www.postgresql.org/docs/current/static/explicit-locking.html

David J.


Re: [GENERAL] max_stack_depth problem though query is substantially smaller

2016-04-10 Thread Bannert Matthias
Heureka! thanks so much for your help, patience and the right hunch. Actually I 
am glad now I ran into that stack issue (and you) cause the entire thing is 
also much faster now. 
I changed my app to emit strings like you suggested and it works, also with 
smaller max_stack_depth. 

Fwiw, I was not stubbornly insisting on nesting operators. Actually I switched 
from "=>" to the hstore function cause
a note in the manual said it was deprecated 
(http://www.postgresql.org/docs/9.0/static/hstore.html). Somehow I must have 
understand that note the wrong way. 
What's your take on that operator being deprecated? 

regards, matt bannert




From: Tom Lane [t...@sss.pgh.pa.us]
Sent: Saturday, April 09, 2016 5:25 PM
To: Bannert  Matthias
Cc: Charles Clavadetscher; pgsql-general@postgresql.org
Subject: Re: [GENERAL] max_stack_depth problem though query is substantially 
smaller

"Bannert  Matthias"  writes:
> [ very deep stack of parser transformExprRecurse calls ]

> #20137 0x7fe7fb80ab8c in pg_analyze_and_rewrite 
> (parsetree=parsetree@entry=0x7fe7fffdb2a0, 
> query_string=query_string@entry=0x7fe7fdf606b0 "INSERT INTO 
> ts_updates(ts_key, ts_data, ts_frequency) VALUES 
> ('some_id.sector_all.news_all_d',hstore('1900-01-01','-0.395131869823009')||hstore('1900-01-02','-0.395131869823009')||hstore('1"...,
>  paramTypes=paramTypes@entry=0x0, numParams=numParams@entry=0) at 
> /build/postgresql-9.3-G1RSAD/postgresql-9.3-9.3.11/build/../src/backend/tcop/postgres.c:640

The SQL fragment we can see here suggests that your "40K entry hstore" is
getting built up by stringing together 40K hstore concatenation operators.
Don't do that.  Even without the parser stack depth issue, it's uselessly
inefficient.  I presume you're generating this statement mechanically,
not by hand, so you could equally well have the app emit

'1900-01-01 => -0.395131869823009, 1900-01-02 => -0.395131869823009, 
...'::hstore

which would look like a single hstore literal to the parser, and be
processed much more quickly.

If you insist on emitting SQL statements that have operators nested
to such depths, then yes you'll need to increase max_stack_depth to
whatever it takes to allow it.

regards, tom lane


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Bypassing NULL elements in row_to_json function

2016-04-10 Thread Adrian Klaver

On 04/10/2016 07:49 AM, Michael Nolan wrote:



On Sun, Apr 10, 2016 at 2:30 AM, David G. Johnston
> wrote:

On Sat, Apr 9, 2016 at 9:48 PM, Michael Nolan >wrote:


2nd Followup:  It turns out that loading a table from a JSON
string is more complicated than going from a table to JSON,
perhaps for good reason.  There does not appear to be a direct
inverse to the row_to_json() function, but it wasn't difficult
for me to write a PHP program that takes the JSON file I created
the other day and converts it back to a series of inserts,
recreating the original table.

Of course this simple program does NO validation (not that this
file needed any), so if the JSON string is not well-formed for
any of a number of reasons, or if it is not properly mapped to
the table into which the inserts are made, an insert could fail
or result in incorrect data.
--
Mike Nolan


​See: http://www.postgresql.org/docs/9.5/interactive/functions-json.html

​json_populate_record(base anyelement, from_json json)
json_populate_recordset(base anyelement, from_json json)

Exists in 9.3 too...though if you are going heavy json I'd suggest
doing whatever you can to keep up with the recent releases.

David J.


If there's a way to use the json_populate_record() or
json_populate_recordset() functions to load a table from a JSON file
(eg, using copy), it would be nice if it was better documented.  I did
find a tool that loads a JSON file into a table (pgfutter), and even
loaded one row from that table into another table using
json_populate_record(), but the 'subquery returned multiple rows' issue
wouldn't let me do the entire table.


Does the receiving table have the same structure as the sending table?

Is the receiving table already populated with data?



But that still doesn't deal with validating individual fields or
checking that the JSON is complete and consistent with the table to be
loaded.


Well you know the JSON is not complete as you dropped all the fields in 
each row that had NULL values.
Validation is a more complex subject and honestly something I do not 
think could be accomplished in straight SQL. In other words it would 
need to be run through some ETL tool. I use Python so as an example:


https://petl.readthedocs.org/en/latest/

In particular:

https://petl.readthedocs.org/en/latest/io.html#json-files

https://petl.readthedocs.org/en/latest/transform.html#validation


--
Mike Nolan



--
Adrian Klaver
adrian.kla...@aklaver.com


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] max_stack_depth problem though query is substantially smaller

2016-04-10 Thread Bannert Matthias
I am not sure if I did it the right way, cause it's my first time stack 
tracing, but I did get some more information. 
Here's what I did. 

1. Switched to ubuntu server 14.04 /w postgres 9.3 server (in a Virtual Box VM)
2. setup postgres and made sure I was able to reproduce the same error. 
3. followed the link to get gdb etc going. 
4. started a psql session and determined the pid using SELECT pg_backend_pid();
5. attached the backend to gdb: sudo gdb -p 1521
6. in gdb: 
(gdb) set pagination off
(gdb) set logging file debuglog.txt
(gdb) set logging on
break errfinish
cont
7. in psql: \i query.sql 
query.sql contains a create TABLE statement and the 1.7 MB INSERT with the 40K 
pairs hstore. 

8. in gdb: ctrl + c 
bt
detach
quit

Finally that logfile hast almost 4 MB, so I am not posting it here. 
Yet, here are some lines that might be helpful In case there is anything in 
particular I could look for, please let me know... 

Breakpoint 1, errfinish (dummy=0) at 
/build/postgresql-9.3-G1RSAD/postgresql-9.3-9.3.11/build/../src/backend/utils/error/elog.c:406
406 
/build/postgresql-9.3-G1RSAD/postgresql-9.3-9.3.11/build/../src/backend/utils/error/elog.c:
 No such file or directory.
#0  errfinish (dummy=0) at 
/build/postgresql-9.3-G1RSAD/postgresql-9.3-9.3.11/build/../src/backend/utils/error/elog.c:406
#1  0x7fe7fb809c67 in check_stack_depth () at 
/build/postgresql-9.3-G1RSAD/postgresql-9.3-9.3.11/build/../src/backend/tcop/postgres.c:3115
#2  0x7fe7fb6b9177 in transformExprRecurse 
(pstate=pstate@entry=0x7fe7fffdb340, expr=expr@entry=0x7fe7ff8fc8d0) at 
/build/postgresql-9.3-G1RSAD/postgresql-9.3-9.3.11/build/../src/backend/parser/parse_expr.c:132
#3  0x7fe7fb6ba314 in transformAExprOp (a=0x7fe7ff8fcb88, 
pstate=0x7fe7fffdb340) at 
/build/postgresql-9.3-G1RSAD/postgresql-9.3-9.3.11/build/../src/backend/parser/parse_expr.c:907
#4  transformExprRecurse (pstate=pstate@entry=0x7fe7fffdb340, 
expr=expr@entry=0x7fe7ff8fcb88) at 
/build/postgresql-9.3-G1RSAD/postgresql-9.3-9.3.11/build/../src/backend/parser/parse_expr.c:223
#5  0x7fe7fb6ba314 in transformAExprOp (a=0x7fe7ff8fce58, 
pstate=0x7fe7fffdb340) at 
/build/postgresql-9.3-G1RSAD/postgresql-9.3-9.3.11/build/../src/backend/parser/parse_expr.c:907
 

what follows are tons of similar lines, just at the end it seems to get more 
specific, i.e. mentioning the hstore:

#20133 0x7fe7fb698d74 in transformInsertStmt (stmt=0x7fe7fffdb2a0, 
pstate=0x7fe7fffdb340) at 
/build/postgresql-9.3-G1RSAD/postgresql-9.3-9.3.11/build/../src/backend/parser/analyze.c:701
#20134 transformStmt (pstate=pstate@entry=0x7fe7fffdb340, 
parseTree=0x7fe7fffdb2a0) at 
/build/postgresql-9.3-G1RSAD/postgresql-9.3-9.3.11/build/../src/backend/parser/analyze.c:225
#20135 0x7fe7fb699e43 in transformTopLevelStmt 
(pstate=pstate@entry=0x7fe7fffdb340, parseTree=, 
parseTree@entry=0x7fe7fffdb2a0) at 
/build/postgresql-9.3-G1RSAD/postgresql-9.3-9.3.11/build/../src/backend/parser/analyze.c:207
#20136 0x7fe7fb699ee9 in parse_analyze (parseTree=0x7fe7fffdb2a0, 
sourceText=0x7fe7fdf606b0 "INSERT INTO ts_updates(ts_key, ts_data, 
ts_frequency) VALUES 
('some_id.sector_all.news_all_d',hstore('1900-01-01','-0.395131869823009')||hstore('1900-01-02','-0.395131869823009')||hstore('1"...,
 paramTypes=0x0, numParams=0) at 
/build/postgresql-9.3-G1RSAD/postgresql-9.3-9.3.11/build/../src/backend/parser/analyze.c:100
#20137 0x7fe7fb80ab8c in pg_analyze_and_rewrite 
(parsetree=parsetree@entry=0x7fe7fffdb2a0, 
query_string=query_string@entry=0x7fe7fdf606b0 "INSERT INTO ts_updates(ts_key, 
ts_data, ts_frequency) VALUES 
('some_id.sector_all.news_all_d',hstore('1900-01-01','-0.395131869823009')||hstore('1900-01-02','-0.395131869823009')||hstore('1"...,
 paramTypes=paramTypes@entry=0x0, numParams=numParams@entry=0) at 
/build/postgresql-9.3-G1RSAD/postgresql-9.3-9.3.11/build/../src/backend/tcop/postgres.c:640
#20138 0x7fe7fb80cff3 in exec_simple_query (query_string=0x7fe7fdf606b0 
"INSERT INTO ts_updates(ts_key, ts_data, ts_frequency) VALUES 
('some_id.sector_all.news_all_d',hstore('1900-01-01','-0.395131869823009')||hstore('1900-01-02','-0.395131869823009')||hstore('1"...)
 at 
/build/postgresql-9.3-G1RSAD/postgresql-9.3-9.3.11/build/../src/backend/tcop/postgres.c:1001
#20139 PostgresMain (argc=, argv=argv@entry=0x7fe7fdd26090, 
dbname=0x7fe7fdd25f40 "postgres", username=) at 
/build/postgresql-9.3-G1RSAD/postgresql-9.3-9.3.11/build/../src/backend/tcop/postgres.c:4072
#20140 0x7fe7fb5fe424 in BackendRun (port=0x7fe7fdd69900) at 
/build/postgresql-9.3-G1RSAD/postgresql-9.3-9.3.11/build/../src/backend/postmaster/postmaster.c:4177
#20141 BackendStartup (port=0x7fe7fdd69900) at 
/build/postgresql-9.3-G1RSAD/postgresql-9.3-9.3.11/build/../src/backend/postmaster/postmaster.c:3840
#20142 ServerLoop () at 
/build/postgresql-9.3-G1RSAD/postgresql-9.3-9.3.11/build/../src/backend/postmaster/postmaster.c:1691
#20143 0x7fe7fb7c6361 in PostmasterMain (argc=5, argv=) at 

Re: [GENERAL] max_stack_depth problem though query is substantially smaller

2016-04-10 Thread Bannert Matthias
I guess you are right. I have narrowed the query down 
to a simple create table, followed by an insert, one text field, one hstore 
field and an integer field.
No temporary table, no BEGIN etc. One record, yet the hstore has 40K kvp. No R 
involved. 
and I still end up with the same error. 

Thanks for the pointer to the stack trace backend. I'll try to set that up and 
report what I find. 





From: Tom Lane [t...@sss.pgh.pa.us]
Sent: Friday, April 08, 2016 9:39 PM
To: Bannert  Matthias
Cc: Charles Clavadetscher; pgsql-general@postgresql.org
Subject: Re: [GENERAL] max_stack_depth problem though query is substantially 
smaller

"Bannert  Matthias"  writes:
> Thanks for your reply. I do think it is rather a postgres than an R issue, 
> here's why:
> a) R simply puts an SQL string together. What Charles had posted was an 
> excerpt of that string.
> Basically we have 1.7 MB of that string. Everything else is equal just the 
> hstore contains 40K key value pairs.

Well, as a test I ran a query that included an hstore literal with 4
million key/value pairs (a bit shy of 70MB of query text).  I didn't see
any misbehavior on a machine with 2MB max_stack_depth.  So there's
something else going on in your situation.

I concur with the suggestion to try to get a stack backtrace from the
point of the error.  Setting a breakpoint at errfinish() is usually
an effective strategy when you know that the query will provoke a SQL
error report.

https://wiki.postgresql.org/wiki/Generating_a_stack_trace_of_a_PostgreSQL_backend

regards, tom lane


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Bypassing NULL elements in row_to_json function

2016-04-10 Thread David G. Johnston
On Sun, Apr 10, 2016 at 7:49 AM, Michael Nolan  wrote:

>
>
> On Sun, Apr 10, 2016 at 2:30 AM, David G. Johnston <
> david.g.johns...@gmail.com> wrote:
>
>> On Sat, Apr 9, 2016 at 9:48 PM, Michael Nolan  wrote:
>>
>>>
>>> 2nd Followup:  It turns out that loading a table from a JSON string is
>>> more complicated than going from a table to JSON, perhaps for good reason.
>>> There does not appear to be a direct inverse to the row_to_json() function,
>>> but it wasn't difficult for me to write a PHP program that takes the JSON
>>> file I created the other day and converts it back to a series of inserts,
>>> recreating the original table.
>>>
>>> Of course this simple program does NO validation (not that this file
>>> needed any), so if the JSON string is not well-formed for any of a number
>>> of reasons, or if it is not properly mapped to the table into which the
>>> inserts are made, an insert could fail or result in incorrect data.
>>> --
>>> Mike Nolan
>>>
>>
>> ​See: http://www.postgresql.org/docs/9.5/interactive/functions-json.html
>>
>> ​json_populate_record(base anyelement, from_json json)
>> json_populate_recordset(base anyelement, from_json json)
>>
>> Exists in 9.3 too...though if you are going heavy json I'd suggest doing
>> whatever you can to keep up with the recent releases.
>>
>> David J.
>>
>>
> If there's a way to use the json_populate_record() or
> json_populate_recordset() functions to load a table from a JSON file (eg,
> using copy), it would be nice if it was better documented.  I did find a
> tool that loads a JSON file into a table (pgfutter), and even loaded one
> row from that table into another table using json_populate_record(), but
> the 'subquery returned multiple rows' issue wouldn't let me do the entire
> table.
>
> But that still doesn't deal with validating individual fields or checking
> that the JSON is complete and consistent with the table to be loaded.
>
>
It isn't that involved once you've learned generally how to call normal
record functions and also set-returning functions (that later must be in
the FROM clause of the query).  If you provide what you attempted its
becomes easier to explain away your mis-understanding.

It doesn't work with COPY.  You have to write an explicit INSERT+SELECT
query where the text of the JSON is a parameter.  Your client library
should let you do this.  If you are using "psql", which doesn't support
parameters, you up having to store the json in a psql variable and
reference that in the function.
​​
INSERT INTO %I SELECT * FROM json_populate_recordset(null::%I, $1)

​The function ensures that column order is consistent so "INSERT INTO %I"
is all you need to write.

​
​
​Data validation is why we invented CHECK constraints - if you need more
functionality than the simple mechanical conversion from a json object to a
table row you will need to write code somewhere to do the additional work.
All json_populate_record(set) promises is that the above command will work.


I suppose the way you'd write your attempt that failed would be similar to:

INSERT INTO %I
SELECT rec.* FROM src_table_with_json LATERAL
json_populate_record(null::%I, src_table_with_json.json_column);

Again, seeing what you actually did would be helpful - I'm having trouble
imaging what you did to provoke that particular error.

David J.


Re: [GENERAL] Bypassing NULL elements in row_to_json function

2016-04-10 Thread Michael Nolan
On Sun, Apr 10, 2016 at 2:30 AM, David G. Johnston <
david.g.johns...@gmail.com> wrote:

> On Sat, Apr 9, 2016 at 9:48 PM, Michael Nolan  wrote:
>
>>
>> 2nd Followup:  It turns out that loading a table from a JSON string is
>> more complicated than going from a table to JSON, perhaps for good reason.
>> There does not appear to be a direct inverse to the row_to_json() function,
>> but it wasn't difficult for me to write a PHP program that takes the JSON
>> file I created the other day and converts it back to a series of inserts,
>> recreating the original table.
>>
>> Of course this simple program does NO validation (not that this file
>> needed any), so if the JSON string is not well-formed for any of a number
>> of reasons, or if it is not properly mapped to the table into which the
>> inserts are made, an insert could fail or result in incorrect data.
>> --
>> Mike Nolan
>>
>
> ​See: http://www.postgresql.org/docs/9.5/interactive/functions-json.html
>
> ​json_populate_record(base anyelement, from_json json)
> json_populate_recordset(base anyelement, from_json json)
>
> Exists in 9.3 too...though if you are going heavy json I'd suggest doing
> whatever you can to keep up with the recent releases.
>
> David J.
>
>
If there's a way to use the json_populate_record() or
json_populate_recordset() functions to load a table from a JSON file (eg,
using copy), it would be nice if it was better documented.  I did find a
tool that loads a JSON file into a table (pgfutter), and even loaded one
row from that table into another table using json_populate_record(), but
the 'subquery returned multiple rows' issue wouldn't let me do the entire
table.

But that still doesn't deal with validating individual fields or checking
that the JSON is complete and consistent with the table to be loaded.
--
Mike Nolan


Re: [GENERAL] max_stack_depth problem though query is substantially smaller

2016-04-10 Thread Tom Lane
"Bannert  Matthias"  writes:
> Fwiw, I was not stubbornly insisting on nesting operators. Actually I 
> switched from "=>" to the hstore function cause
> a note in the manual said it was deprecated 
> (http://www.postgresql.org/docs/9.0/static/hstore.html). Somehow I must have 
> understand that note the wrong way. 
> What's your take on that operator being deprecated? 

That's the old SQL operator (which is not even there anymore) that's
equivalent to the hstore(text,text) constructor function, ie
"text => text returning hstore".  It's quite a different concept
from the => notation inside an hstore literal.  That is:
'foo'::text => 'bar'::text
is not like
'"foo" => "bar"'::hstore
even though they have the same end result.

regards, tom lane


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Really unique session ID - PID + connection timestamp?

2016-04-10 Thread durumd...@gmail.com


Dear Alban!


2016.04.10. 13:05 keltezéssel, Alban Hertroys írta:

On 10 Apr 2016, at 9:07, Durumdara  wrote:

Dear Adrian!

Again. As I see the beginning blocks are removed by mailing system in the code.

We have an "ourlocks" table which hold records (TableName, RecordID, 
SessionInnerID, TimeStamp, etc, with TableName/RecordID prikey).

If anybody wants to lock record "for long time", "over the transactions" it try 
to insert a new record here.

Why are those records being locked? Reading on, it seems like you're trying to 
solve a fairly standard concurrency problem. Any RDBMS worth their salt can 
handle that for you, you don't need to manually do any of that.


This is not real locks. They are logical locks.
Products, offers are edited for long time.
But we must save subdata. This is not a "word like document" which can 
saved at once, in a transaction.

When a product edited, we must protect it from other user's edit.
But it's subdata must be posted/commited to the DB, for example 
shipping, article quantity changes, vouchers, etc.






This sounds much more like a use-case for sub-transactions and select for 
update (which puts a temporary RDBMS-controlled lock on the relevant records) 
than for manual locking.


Yes, this is like sub-transaction.
But for only sub-data. The main data must be edited by only the first 
user who started the edit.

This is a long time "lock" like thing. This what we simulate here.

Thanks for your suggestions. I will check this in our client library.

dd


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Really unique session ID - PID + connection timestamp?

2016-04-10 Thread Alban Hertroys

> On 10 Apr 2016, at 9:07, Durumdara  wrote:
> 
> Dear Adrian!
> 
> Again. As I see the beginning blocks are removed by mailing system in the 
> code.
> 
> We have an "ourlocks" table which hold records (TableName, RecordID, 
> SessionInnerID, TimeStamp, etc, with TableName/RecordID prikey).
> 
> If anybody wants to lock record "for long time", "over the transactions" it 
> try to insert a new record here.

Why are those records being locked? Reading on, it seems like you're trying to 
solve a fairly standard concurrency problem. Any RDBMS worth their salt can 
handle that for you, you don't need to manually do any of that.

> If other process want to lock same record, it can see this record (or got 
> violation error), so it stopped.
> 
> This is not for protect all tables, only for protect main entities have many 
> subtables like "Products", "Offers", etc.
> We can't use transactions, because in the editor they must post/commit 
> subdata.
> And because PG is different from other DBs, so if ANY of statements failed, 
> it rollback whole thing automatically (see my prior mail).
> In FireBird we can do simple record lock protection with main connection's 
> second transaction, but in MS and PG not.

This sounds much more like a use-case for sub-transactions and select for 
update (which puts a temporary RDBMS-controlled lock on the relevant records) 
than for manual locking.
See: http://www.postgresql.org/docs/9.5/static/sql-begin.html and 
http://www.postgresql.org/docs/9.5/static/sql-select.html

You might also want to look into transaction isolation levels: 
http://www.postgresql.org/docs/9.5/interactive/mvcc.html

As an example of how a concurrent workflow with the above goes:

Session 1:
begin;
savepoint offer_update;
select product_id from offers where offer_id = 1234567 for update;

Session 2:
begin;
savepoint offer_update;
select product_id from offers where offer_id = 1234567 for update;
update offers set discount = 0.10 where product_id = 1234567;
#   ERROR (the record is locked by session 1)
rollback to offer_update;

Session 1:
update offers set discount = 0.15 where product_id = 1234567;
#   success
commit;

Session 2: (retrying earlier update)
select product_id from offers where offer_id = 1234567 for update;
update offers set discount = 0.10 where product_id = 1234567;
#   success
commit; 

You'll need to add some logic to your application (that editor you were talking 
about) so that it inserts savepoints and handles failures of sub-transactions 
appropriately.

> So we used real records in a real table. But how to clean if client 
> disconnected without delete own records?
> For this we created own sessioninfo table with inner id, user id, timestamp, 
> and [connectionid, connectiontime].
> The locking mechanism checks for same lock (Offer, 117), if anybody locks the 
> record, it checks for he's on or not.
> If active connection (ID + TS) then lock is valid, and we can show 
> information that "who is editing, please ask for him to release, etc.".
> If not, we can eliminate the record and insert our.

It sounds to me like you're complicating your code where you could be 
simplifying it. Possibly, because you're used to a database that provides 
certain features to make up for the lack of others that are harder to 
implement. Both MS Access and Firebird are very much file-based desktop 
databases that are not really meant for concurrent access. The big RDBMSes (PG, 
MS SQL server, Oracle, DB2) are _designed_ for such workloads.

> The main goal is to protect the main entities. It is working in MS.
> My question was about how to get my client's connection timestamp as 
> get_backend_pid.
> But as you wrote I can get it from activity log. Because PID can't be same as 
> mine, I can select my from the table.
> 
> You said it have danger (like guid repetition). Yes, it have. And not the BAD 
> SYSADMIN, but the summer/winter time changes are dangerous (the backward 
> version). In PG we can extend our "guid" with IP and Port too, and this could 
> be enough safe for us.

In that case you should at least use UTC timestamps. Still, with such an 
implementation it will be hard to create a reliable system.

> Thanks
> 
> 
> 
> 
> 2016-04-09 16:05 GMT+02:00 Adrian Klaver :
> On 04/09/2016 01:30 AM, Durumdara wrote:
> Dear Everybody!
> 
> 
> In MS we had a "persistent lock" structure and method.
> This over  transactions because based on real records in a real table
> with pri key (tablename + id).
> 
> For garbaging we had a special session info.
> In MS the session id is smallint, so it can repeats after server
> restarts, but my coll. found a "session creation timestamp".
> This is a key which unique.
> With this we can check for died sessions and we can clean their records.
> 
> It might help to explain more what it is you are 

Re: [GENERAL] Shipping big WAL archives to hot standby

2016-04-10 Thread Venkata Balaji N
> What would be the effect of suddenly introducing a 1-2 GB of WAL archives
> to the WAL restore folder on the slave? Would there be a big performance
> effect on the incoming queries to the slave? Would the slave be available
> for queries while the WAL logs are restored into the DB?
>

If the Queries are running on slave whilst WAL archives are being applied,
there is a bright chance that, queries would encounter a conflict with the
recovery and would eventually get cancelled. You can see messages in the
PostgreSQL logfiles. The recommended approach would be to apply WALs when
queries are being executed on slave.

Regards,
Venkata B N

Fujitsu Australia


Re: [GENERAL] Bypassing NULL elements in row_to_json function

2016-04-10 Thread David G. Johnston
On Sat, Apr 9, 2016 at 9:48 PM, Michael Nolan  wrote:

>
> 2nd Followup:  It turns out that loading a table from a JSON string is
> more complicated than going from a table to JSON, perhaps for good reason.
> There does not appear to be a direct inverse to the row_to_json() function,
> but it wasn't difficult for me to write a PHP program that takes the JSON
> file I created the other day and converts it back to a series of inserts,
> recreating the original table.
>
> Of course this simple program does NO validation (not that this file
> needed any), so if the JSON string is not well-formed for any of a number
> of reasons, or if it is not properly mapped to the table into which the
> inserts are made, an insert could fail or result in incorrect data.
> --
> Mike Nolan
>

​See: http://www.postgresql.org/docs/9.5/interactive/functions-json.html

​json_populate_record(base anyelement, from_json json)
json_populate_recordset(base anyelement, from_json json)

Exists in 9.3 too...though if you are going heavy json I'd suggest doing
whatever you can to keep up with the recent releases.

David J.


Re: [GENERAL] Really unique session ID - PID + connection timestamp?

2016-04-10 Thread Durumdara
Dear Adrian!

Again. As I see the beginning blocks are removed by mailing system in the
code.

We have an "ourlocks" table which hold records (TableName, RecordID,
SessionInnerID, TimeStamp, etc, with TableName/RecordID prikey).

If anybody wants to lock record "for long time", "over the transactions" it
try to insert a new record here.
If other process want to lock same record, it can see this record (or got
violation error), so it stopped.

This is not for protect all tables, only for protect main entities have
many subtables like "Products", "Offers", etc.
We can't use transactions, because in the editor they must post/commit
subdata.
And because PG is different from other DBs, so if ANY of statements failed,
it rollback whole thing automatically (see my prior mail).
In FireBird we can do simple record lock protection with main connection's
second transaction, but in MS and PG not.

So we used real records in a real table. But how to clean if client
disconnected without delete own records?
For this we created own sessioninfo table with inner id, user id,
timestamp, and [connectionid, connectiontime].
The locking mechanism checks for same lock (Offer, 117), if anybody locks
the record, it checks for he's on or not.
If active connection (ID + TS) then lock is valid, and we can show
information that "who is editing, please ask for him to release, etc.".
If not, we can eliminate the record and insert our.

The main goal is to protect the main entities. It is working in MS.
My question was about how to get my client's connection timestamp as
get_backend_pid.
But as you wrote I can get it from activity log. Because PID can't be same
as mine, I can select my from the table.

You said it have danger (like guid repetition). Yes, it have. And not the
BAD SYSADMIN, but the summer/winter time changes are dangerous (the
backward version). In PG we can extend our "guid" with IP and Port too, and
this could be enough safe for us.

Thanks




2016-04-09 16:05 GMT+02:00 Adrian Klaver :

> On 04/09/2016 01:30 AM, Durumdara wrote:
>
>> Dear Everybody!
>>
>>
>> In MS we had a "persistent lock" structure and method.
>> This over  transactions because based on real records in a real table
>> with pri key (tablename + id).
>>
>> For garbaging we had a special session info.
>> In MS the session id is smallint, so it can repeats after server
>> restarts, but my coll. found a "session creation timestamp".
>> This is a key which unique.
>> With this we can check for died sessions and we can clean their records.
>>
>
> It might help to explain more what it is you are trying to achieve.
>
> First I am not sure what you mean by 'persistent lock', especially as it
> applies to Postgres?
>
> Second, I assume by garbaging you mean garbage collection of something?
> If that is the case what exactly are you garbage collecting?
> I see 'clean records', what records would these be?
> In particular, on Postgres, where are you going to do this?
>
>
>> We want create same mechanism.
>>
>
> If the above questions did not already cover this, what mechanism?
>
>
> I know there are adv. locks in PG, but I want to use session id.
>>
>> This could be:
>> |pg_backend_pid|()
>>
>> May pid repeats.
>> Where I can get timestamp or some other unique data with I can create a
>> combined primary key?
>>
>> Thanks for your help!
>>
>> dd
>>
>>
>
> --
> Adrian Klaver
> adrian.kla...@aklaver.com
>