Re: [GENERAL] plugin dev, oid to pointer map

[Julien Rouhaud]

On 29/05/2016 22:10, Attila Soki wrote:
> Hi all,
> 

Hello,

> i am about to begin with postgresql plugin development.
>H Currently i'm trying to become somewhat familiar with the postgresql sources.

> 
> Without going too deep into details about the plugin, i want to use
> many Oid to pointer relations.
> The pointer is a pointer to my own struct (allocated with palloc).
> There will be approx. 1000 unique oid/pointer pairs.
> 
> Basically, what i want is, to be able to get the pointer to my struct by Oid.
> 
> Is there is a suitable hashmap or key-value storage solution in the pg code?
> if so, please point me to the right part of the source.
> 

Yes, there's an hashtable implementation, see dynahash.c

If you want to use that in shared memory in your extension, you can look
at the pg_stat_statements extension (look for pgss_hash) for an example.

Regards.


-- 
Julien Rouhaud
http://dalibo.com - http://dalibo.org


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

[GENERAL] plugin dev, oid to pointer map

[Attila Soki]

Hi all,

i am about to begin with postgresql plugin development.
Currently i'm trying to become somewhat familiar with the postgresql sources.

Without going too deep into details about the plugin, i want to use
many Oid to pointer relations.
The pointer is a pointer to my own struct (allocated with palloc).
There will be approx. 1000 unique oid/pointer pairs.

Basically, what i want is, to be able to get the pointer to my struct by Oid.

Is there is a suitable hashmap or key-value storage solution in the pg code?
if so, please point me to the right part of the source.

thanks,
Attila Soki


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Slides for PGCon2016; "FTS is dead ? Long live FTS !"

[Karsten Hilbert]

>> I submitted slides to pgcon site, but it usually takes awhile, so you can
>> download our presentation directly
>> http://www.sai.msu.su/~megera/postgres/talks/pgcon-2016-fts.pdf

Looking at slide 39 (attached) I get the impression that I
should be able to do the following:


- turn a coding system (say, ICD-10) into a dictionary
  by splitting the terms into single words

say, "diabetes mellitus -> "diabetes", "mellitus"

- define stop words like "left", "right", ...

say, "fracture left ulna" -> the "left" doesn't
matter as far as coding is concerned

- also turn that coding system into queries by splitting
  the terms into single words, concatenating them
  with "&", and setting the ICD 10 code as tag on them

say, "diabetes mellitus" -> "diabetes & mellitus [E11]"

- run an inverse FTS (FQS) against a user supplied string
  thereby finding queries (= tags = ICD10 codes) likely
  relevant to the input

say, to_tsvector("patient was suspected to suffer from diabetes 
mellitus")
-> tag = E11


Possible, not possible, insane, unintended use ?

Thanks,
Karsten
-- 
GPG key ID E4071346 @ eu.pool.sks-keyservers.net
E167 67FD A291 2BEA 73BD  4537 78B9 A9F9 E407 1346


pgcon-2016-fts-Seite-39.pdf
Description: Adobe PDF document

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] BDR to ignore table exists error

[Martín Marqués]

Hi,

El 29/05/16 a las 06:01, Nikhil escribió:
> 
> *​Nik>> skip_ddl_locking is set to True in my configuration. As this
> was preventing single*
> 
> *​node from doing DDL operation (if one is down majority is not there
> for doing DDL on available node)*​

Well, you have to be prepared to deal with burn wounds if you play with
fire. ;)

If you decide to have skip_ddl_locking on you have to be sure all DDLs
happen on one node, else you end up with conflicts like this.

I suggest you find out why the table was already created on the
downstream node (as a forensics task so you can avoid bumping into the
same issue).

> ​Nik>> DDL used is
> 
> ​
> ERROR:  relation "af_npx_l3_16_146_10" already exists
> <596802016-05-29 08:53:07 GMT%CONTEXT:  during DDL replay of ddl
> statement: CREATE  TABLE  public.af_npx_license_l3_16_146_
> 10 (CONSTRAINT af_npx_license_l3_16_146_10_rpt_sample_time_check CHECK
> (((rpt_sample_time OPERATOR(pg_catalog.>=) 146417040
> 0) AND (rpt_sample_time OPERATOR(pg_catalog.<=) 1464173999))) ) INHERITS
> (public.af_npx_l3) WITH (oids=OFF)
> <554132016-05-29 08:53:07 GMT%LOG:  worker process: bdr
> (6288512113617339435,2,16384,)->bdr (6288505144157102317,1, (PID 59
> 680) exited with exit code 1

On the node where the CREATE TABLE is trying to get applied run this:

BEGIN;
SET LOCAL bdr.skip_ddl_replication TO 'on';
SET LOCAL bdr.skip_ddl_locking TO 'on';
DROP TABLE af_npx_l3_16_146_10;
END;

After that, the DDL that's stuck will get applied and the stream of
changes will continue.

By the looks of what you're dealing with, I wouldn't be surprised if the
replication gets stuck again on another DDL conflict.

I suggest rethinking the locking strategy, because this shows that
there's something fishy there.

Regards,

-- 
Martín Marquéshttp://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Slides for PGCon2016; "FTS is dead ? Long live FTS !"

[Oleg Bartunov]

On Sun, May 29, 2016 at 2:43 PM, Stefan Keller  wrote:

> Hi,
>
> Nice work from you postgrespro.ru guys! Especially the RUM index which
> demonstrates the power of 9.6 to let third party SW create access methods
> as extension: https://github.com/postgrespro/rum
>
> 1. I don't understand the benchmarks on slide 25 "20 mln descriptions"
> (and the one before "6.7 mln classifieds"): What does "Queries in 8 h 9.2
> +patch (9.6 rum)" mean?
>

We run queries for 8 hours and recorded the number of executed queries.
Four years ago, when I and Alexander developed an initial version of patch
we got results marked by "9.2+patch", and now we run the same queries on
the same database and put rum results into (). I'd not consider to this
numbers, since we used queries from 6 mln database. We'd be happy if
somebody run independent benchmarks.


>
> 2. What does R-U-M mean? (can't mean "Range Usage Metadata" which was
> finally coined range index BRIN)?
>


We chose RUM just because there are GIN and VODKA :) But some people
already suggested several meanings like Really Useful iMdex :)  We are open
for suggestion.


>
> :Stefan, co-organizer of Swiss PGDay
>
>
> 2016-05-29 11:29 GMT+02:00 Andreas Joseph Krogh :
>
>> På lørdag 28. mai 2016 kl. 23:59:55, skrev Oleg Bartunov <
>> obartu...@gmail.com>:
>>
>>
>>
>> On Thu, May 26, 2016 at 11:26 PM, Andreas Joseph Krogh <
>> andr...@visena.com> wrote:
>>>
>>> Hi.
>>>
>>> Any news about when slides for $subject will be available?
>>>
>>
>> I submitted slides to pgcon site, but it usually takes awhile, so you can
>> download our presentation directly
>> http://www.sai.msu.su/~megera/postgres/talks/pgcon-2016-fts.pdf
>>
>> There are some missing features in rum index, but I hope we'll update
>> github repository really soon.
>>
>>
>> This is simply amazing!
>>
>> I want to run 9.6 beta in production right now because of this:-)
>>
>> Hats off guys, congrats to PostgresPro, and huge thanks!!
>>
>> --
>> *Andreas Joseph Krogh*
>> CTO / Partner - Visena AS
>> Mobile: +47 909 56 963
>> andr...@visena.com
>> www.visena.com
>> 
>>
>>
>
>

Re: [GENERAL] Slides for PGCon2016; "FTS is dead ? Long live FTS !"

[Andreas Joseph Krogh]

På søndag 29. mai 2016 kl. 19:49:06, skrev Oleg Bartunov >:
[snip]  
I want to run 9.6 beta in production right now because of this:-)
 
wait-wait :)  We'd be happy to have feedback from production, of course, but 
please, wait a bit. We are adding support of sorting posting list/tree not by 
item pointer as in gin, but make use of additional information, for example, 
timestamp, which will provide additional speedup to the existing one.



 
Awesome!
 
 
Also, we are sure there are some bugs :)



 
He he, I reported 1st issue: https://github.com/postgrespro/rum/issues/1
 
Would be cool to see this fixed so I actually could have a sip of the rum:-)

 
-- Andreas Joseph Krogh
CTO / Partner - Visena AS
Mobile: +47 909 56 963
andr...@visena.com 
www.visena.com 
 


 

Re: [GENERAL] Slides for PGCon2016; "FTS is dead ? Long live FTS !"

[Oleg Bartunov]

On Sun, May 29, 2016 at 12:29 PM, Andreas Joseph Krogh 
wrote:

> På lørdag 28. mai 2016 kl. 23:59:55, skrev Oleg Bartunov <
> obartu...@gmail.com>:
>
>
>
> On Thu, May 26, 2016 at 11:26 PM, Andreas Joseph Krogh  > wrote:
>>
>> Hi.
>>
>> Any news about when slides for $subject will be available?
>>
>
> I submitted slides to pgcon site, but it usually takes awhile, so you can
> download our presentation directly
> http://www.sai.msu.su/~megera/postgres/talks/pgcon-2016-fts.pdf
>
> There are some missing features in rum index, but I hope we'll update
> github repository really soon.
>
>
> This is simply amazing!
>
> I want to run 9.6 beta in production right now because of this:-)
>

wait-wait :)  We'd be happy to have feedback from production, of course,
but please, wait a bit. We are adding support of sorting posting list/tree
not by item pointer as in gin, but make use of additional information, for
example, timestamp, which will provide additional speedup to the existing
one. Also, we are sure there are some bugs :)


>
> Hats off guys, congrats to PostgresPro, and huge thanks!!
>
> --
> *Andreas Joseph Krogh*
> CTO / Partner - Visena AS
> Mobile: +47 909 56 963
> andr...@visena.com
> www.visena.com
> 
>
>

Re: [GENERAL] Slides for PGCon2016; "FTS is dead ? Long live FTS !"

[Stefan Keller]

Hi,

Nice work from you postgrespro.ru guys! Especially the RUM index which
demonstrates the power of 9.6 to let third party SW create access methods
as extension: https://github.com/postgrespro/rum

1. I don't understand the benchmarks on slide 25 "20 mln descriptions" (and
the one before "6.7 mln classifieds"): What does "Queries in 8 h 9.2 +patch
(9.6 rum)" mean?

2. What does R-U-M mean? (can't mean "Range Usage Metadata" which was
finally coined range index BRIN)?

:Stefan, co-organizer of Swiss PGDay


2016-05-29 11:29 GMT+02:00 Andreas Joseph Krogh :

> På lørdag 28. mai 2016 kl. 23:59:55, skrev Oleg Bartunov <
> obartu...@gmail.com>:
>
>
>
> On Thu, May 26, 2016 at 11:26 PM, Andreas Joseph Krogh  > wrote:
>>
>> Hi.
>>
>> Any news about when slides for $subject will be available?
>>
>
> I submitted slides to pgcon site, but it usually takes awhile, so you can
> download our presentation directly
> http://www.sai.msu.su/~megera/postgres/talks/pgcon-2016-fts.pdf
>
> There are some missing features in rum index, but I hope we'll update
> github repository really soon.
>
>
> This is simply amazing!
>
> I want to run 9.6 beta in production right now because of this:-)
>
> Hats off guys, congrats to PostgresPro, and huge thanks!!
>
> --
> *Andreas Joseph Krogh*
> CTO / Partner - Visena AS
> Mobile: +47 909 56 963
> andr...@visena.com
> www.visena.com
> 
>
>

Re: [GENERAL] Slides for PGCon2016; "FTS is dead ? Long live FTS !"

[Andreas Joseph Krogh]

På lørdag 28. mai 2016 kl. 23:59:55, skrev Oleg Bartunov >:
    On Thu, May 26, 2016 at 11:26 PM, Andreas Joseph Krogh > wrote: Hi.
 
Any news about when slides for $subject will be available?
 
I submitted slides to pgcon site, but it usually takes awhile, so you can 
download our presentation directly
http://www.sai.msu.su/~megera/postgres/talks/pgcon-2016-fts.pdf 

  
There are some missing features in rum index, but I hope we'll update github 
repository really soon.



 
This is simply amazing!
 
I want to run 9.6 beta in production right now because of this:-)
 
Hats off guys, congrats to PostgresPro, and huge thanks!!
 
-- Andreas Joseph Krogh
CTO / Partner - Visena AS
Mobile: +47 909 56 963
andr...@visena.com 
www.visena.com 
 


 

Re: [GENERAL] Migrate 2 DB's - v8.3

[Francisco Olarte]

Martin:

Could you please avoid unedited top posts?

On Sat, May 28, 2016 at 7:53 PM, Martín Marqués  wrote:
> I still don't understand why the OP is getting into so much trouble and
> doesn't upgrade to a newer version like 9.3 or 9.4 (or even 9.5).

Neither do I, but the thing is trying to find the fastest way to move
a single database between two clusters on different machines, or
trying to find a way to do it without downtime, is useful on its own,
whicever the versions are.

> All this hassle to stay on an unsupported postgres is just useless, IMNSHO.

You can ask the OP for the reason to stay in 8.3 directly. Maybe is
something as simple as "I'm the one who pays, you do what I pay you
for.". I've had several of these.


Francisco Olarte.


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] BDR to ignore table exists error

[Nikhil]

Please see my replies inline.

On Sat, May 28, 2016 at 8:08 PM, Martín Marqués 
wrote:

> El 28/05/16 a las 08:57, Nikhil escribió:
> > Once the node which was down is brought back the replication slot is not
> > turned active. The reason being replication slot is trying to create a
> > partition table which already exists. Because of this error replication
> > slot is stuck in inactive mode. Is there any way to ignore this error?
>
> BTW, how did you end up in such a state? Did you create the partition
> table skipping ddl locking?
> *​Nik>> skip_ddl_locking is set to True in my configuration. As this was
> preventing single*
>
*​node from doing DDL operation (if one is down majority is not there for
doing DDL on available node)*​


>
> At this point the easiest way out is to drop the table on the node where
> it's trying to get applied with bdr_replication off or
> skip_ddl_replication on, so the table is dropped locally but not
> replicated, and the create table from the slot can be consumed.
>
> The other option is to consume the create table statement from the slot
> directly.
> ​Nik>> DDL used is
>
​
ERROR:  relation "af_npx_l3_16_146_10" already exists
<596802016-05-29 08:53:07 GMT%CONTEXT:  during DDL replay of ddl statement:
CREATE  TABLE  public.af_npx_license_l3_16_146_
10 (CONSTRAINT af_npx_license_l3_16_146_10_rpt_sample_time_check CHECK
(((rpt_sample_time OPERATOR(pg_catalog.>=) 146417040
0) AND (rpt_sample_time OPERATOR(pg_catalog.<=) 1464173999))) ) INHERITS
(public.af_npx_l3) WITH (oids=OFF)
<554132016-05-29 08:53:07 GMT%LOG:  worker process: bdr
(6288512113617339435,2,16384,)->bdr (6288505144157102317,1, (PID 59
680) exited with exit code 1
​


> ​
>
> Be aware of the dangers of changing the default values for such
> parameters, (bdr_replication, skip_ddl_replication, skip_ddl_locking)
> and when needed they should be used with special care.
> *​Nik>>. The DDL replay is started once the node join back to bdr group. I
> think its started from an old check point causing partition already exists
> error. Is there any way to ignore replay error ? or ignore DDL errors while
> replay ?​*
>
> Regards,
>
> --
> Martín Marquéshttp://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Training & Services
>