Re: [HACKERS] Tackling JsonPath support

[Christian Convey]

On Fri, Dec 2, 2016 at 1:32 PM, Nico Williams <n...@cryptonector.com> wrote:
...
On Fri, Dec 02, 2016 at 08:53:33AM -0500, Robert Haas wrote:

> > On Tue, Nov 29, 2016 at 11:50 AM, Christian Convey
> > <christian.con...@gmail.com> wrote:
> > > I think I can satisfy (3) with a PG extension which provides a
> function that
> > > approximately implements JSONPath.  My short-term plans are to submit
> such a
> > > patch.
> >
> > FWIW, I think that's a fine plan.  I don't really know whether
> > JSONPath is the right standard to pick for the task of extracting bits
>
> It's not even a standard.  Are there particular proposals that the ANSI
> SQL working group is considering?
>

​Hi Nico, it seems to be something in the works with the standards
committee.  We were discussing it earlier in the thread: [1]

Kind regards,
Christian

[1] http://www.mail-archive.com/pgsql-hackers@postgresql.org/msg297732.html​

Re: [HACKERS] Tackling JsonPath support

[Christian Convey]

On Tue, Nov 29, 2016 at 8:18 AM, Petr Jelinek  wrote:
...

> Just to add to this, the SQL/JSON proposals I've seen so far, and what
> Oracle, MSSQL and Teradata chose to implement already is basically
> subset of jsonpath (some proposals/implementations also include
> lax/strict prefix keyword on top of that). I think that should give us
> some hint on what the base functionality should look like.


I agree.  My guess is that PG users would benefit most from:

(1) Conformance to whatever ISO standard regarding JSON operators
eventually makes it out of the working group.
(2) Compatibility with other widely-used DBMS's.
(3) Compatibility with the JSONPath functionality ​used by web developers.
 (Although I don't currently have a grasp on which frameworks / libraries
this entails.)

I *think* that (1), (2), and (3) are in approximate agreement about the
syntax and semantics of the path-expression language: the language proposed
by Stefan Groessner, plus the strict vs. lax distinction.

I think I can satisfy (3) with a PG extension which provides a function
that approximately implements JSONPath.  My short-term plans are to submit
such a patch.

Hopefully that patch's function will be a helpful starting point for
satisfying (1) and (2) as well.  But that can be decided later.

Nico Williams has argued for using "jq".  I don't think jq satisfies any of
(1), (2), or (3), so I don't see a good case for incorporating it in my
short-term plans.  There *may* be a case for using jq internally to my
implementation; I'll try to look into that.

Re: [HACKERS] Tackling JsonPath support

[Christian Convey]

On Mon, Nov 28, 2016 at 10:37 PM, Pavel Stehule <pavel.steh...@gmail.com>
wrote:

>
>
> 2016-11-29 7:34 GMT+01:00 Christian Convey <christian.con...@gmail.com>:
>
>> On Mon, Nov 28, 2016 at 9:28 PM, Pavel Stehule <pavel.steh...@gmail.com>
>> wrote:
>>
>>> ​​
>>> We now support XPath function - JSONPath is similar to XPath -
>>> ​​
>>> it is better for user, because have to learn only one language.
>>>
>>
>> I'm not sure I understand.
>>
>> Are you suggesting that we use XPath, not JSONPath, as our language for
>> json-path expressions?
>>
>
> surely not.
>
> follow ANSI/SQL :)
>

I see.  Then I'm afraid I still don't understand what you're main point was
when you wrote:

​
> We now support XPath function - JSONPath is similar to XPath -
> ​​
> it is better for user, because have to learn only one language.
>

- C

Re: [HACKERS] Tackling JsonPath support

[Christian Convey]

On Mon, Nov 28, 2016 at 9:28 PM, Pavel Stehule 
wrote:

> We now support XPath function - JSONPath is similar to XPath - it is
> better for user, because have to learn only one language.
>

I'm not sure I understand.

Are you suggesting that we use XPath, not JSONPath, as our language for
json-path expressions?

​- C

Re: [HACKERS] Tackling JsonPath support

[Christian Convey]

On Mon, Nov 28, 2016 at 6:26 PM, Nico Williams <n...@cryptonector.com>
wrote:

> On Mon, Nov 28, 2016 at 05:50:40PM -0800, Christian Convey wrote:
> > On Mon, Nov 28, 2016 at 11:23 AM, Nico Williams <n...@cryptonector.com>
> > wrote:
> > ...
> > > JSON Path is not expressive enough (last I looked) and can be mapped
> > > onto jq if need be anyways.
> >
> > Hi Nico,
> >
> > Could you please clarify what you mean by "not expressive enough"?
>
> jq is a functional language that has these and other features:
>
>  - recursion
>  - generators
>  - lazy evaluation (of sorts)
>  - path expressions
>  - math functionality (libm, basically)
>  - reduction
>  - functions
>  - and other things
>
> (jq does not have higher-order functions in that functions cannot return
> functions and functions are not values, though it does have closures.)
>
> jq is and feels a lot like a SQL, but for JSON.
>
> > I ask because I've been struggling to identify clear requirements for the
> > json-path functionality I'm trying to provide.  It sounds like perhaps
> you
> > have something concrete in mind.
>
> SQL imposes structure on data.  Recursion makes SQL structure looser in
> the sense that it may not be easy or possible to express certain
> desirable schema constraints in SQL terms without resorting to triggers,
> say.  Storing documents in XML, JSON, or other such recursion-friendly
> formats (perhaps in semantically equivalent but query-optimized forms)
> is also a way to avoid strict structure (thus one needs schema
> validators for XML, for example).
>
> Less rigid schema constraints do not and should not preclude powerful
> query languages.
>
> One could convert such documents to a SQL EAV schema, if one has an
> RDBMS with an ANY type (e.g., something like SQLite3's duck typing), and
> then use SQL to query them.  But that may be more difficult to use than
> a SQL with support for XML/JSON/... and query sub-languages for those.
>
> SQL is very powerful.  One might like to have similarly powerful,
> format-specific query languages for documents stored in XML, JSON,
> etcetera, in a SQL RDBMS.  jq is such a language, for JSON documents.
> Ditto XPath/XSLT, for XML.  While XPath is expressive and compact, XSLT
> is rather verbose; jq is as expressive as XSLT, but with the compact
> verbosity of XPath.
>
> > Since I myself have no need currently for this functionality, I'm left
> > guessing about hypothetical users of it.  My current mental model is:
>
> That's a bit like asking what is the use for SQL :^)  The point is that
> SQL is a powerful query language, and so is jq.  Each is appropriate to
> its own domain; both could be used together.
>

Thanks for the explanation.  It sounds like your original point was NOT
that json-path isn't sufficient for "${specific use X}".

Instead, your point was that jq seems to have many advantages over
json-path in general, and therefore PG should offer jq instead or, or in
addition to, json-path.

Is that what you're saying?

​...


> > (c) Pavel.  (I'm still somewhat unclear on what has him interested in
> this,
> > and what his specific constraints are.)
>
> Hmm?
>

​Context: The reason I'm trying to work on a json-path implementation is
that Pavel Stehule suggested it as a good first PG-hacking project for me.
At the time, it sounded like he had a use for the feature.

- C

Re: [HACKERS] Tackling JsonPath support

[Christian Convey]

On Mon, Nov 28, 2016 at 11:23 AM, Nico Williams 
wrote:
...

> JSON Path is not expressive enough (last I looked) and can be mapped
> onto jq if need be anyways.
>

​Hi Nico,

Could you please clarify what you mean by "not expressive enough"?

I ask because I've been struggling to identify clear requirements for the
json-path functionality I'm trying to provide.  It sounds like perhaps you
have something concrete in mind.

Since I myself have no need currently for this functionality, I'm left
guessing about hypothetical users of it.​  My current mental model is:

(a) Backend web developers.  AFAICT, their community has mostly settled on
the syntax/semantics proposed by Stefan Groessner.  It would probably be
unkind for PG's implementation to deviate from that without a good reason.

(b) PG hackers who will eventually implement the ISO SQL standard
operators.  In the standards-committee meeting notes I've seen, it seemed
to me that they were planning to define some operators in terms of
json-path expression.  So it would probably be good if whatever json-path
function I implement turns out to comply with that standard, so that the
PG-hackers can use it as a building block for their work.

(c) Pavel.  (I'm still somewhat unclear on what has him interested in this,
and what his specific constraints are.)

- Christian

Re: [HACKERS] Tackling JsonPath support

[Christian Convey]

On Mon, Nov 28, 2016 at 9:47 AM, Pavel Stehule 
wrote
>
> > I thought by adding my first implementation to "contrib", we could make
> this functionality available to end-users, even before there was a
> consensus about what PG's "official" JSON-related operators should have for
> syntax and semantics.
> >
>
> this time the commiters dislike the contrib dir. It is hard to push there
> anything :-(. You can try it, but it can be lost time.
>

​Thanks for the warning.  I'm okay with  my patch adding the "json_path"
function to the core PG code.​

I would still suggest that we hold off on having my first patch implement
an official JSON-related operator such as "JSON_TABLE".  I would prefer to
have my "json_path" function available to users even before we know how
"JSON_TABLE", etc. should behave.

​Does that sound reasonable?​

Re: [HACKERS] Tackling JsonPath support

[Christian Convey]

On Mon, Nov 28, 2016 at 9:40 AM, Pavel Stehule 
wrote:
​...​

> > ​Hi Pavel,
> >
> > Can you clarify what you meant?  I *think* you're saying:
> >
> > * It's not important for me to match the syntax/semantics of the
> json-path implementations found in MySQL / Oracle / DB2 / ​MS SQL Server,
> and
> >
>
> oh no. the syntax is important. But for start we can have a subset. For
> json table function .. json to relation mapping is important path
> expression. some other features like predicates
> are nice, but can be implemented later.
>
> Im sorry. My English is bad.
>
​Hi Pavel,

You're English is very good, actually.  I think the confusion arises from
me speaking in vague terms.  I apologize for that.  Allow me to be more
specific about what I'm proposing to do.

I propose adding to "contrib" a function with the following characteristics:

* Its signature is "json_path( jsonb from_json, string
json_path_expression) --> jsonb".

* The function will hopefully be a useful building block for PG's
implementation of "official" JSON operators such as "JSON_TABLE".  Once the
PG community agrees on what those operators' syntax/semantics should be.
​
* The function will hopefully be immediately useful to PG users who want
JSONPath -like operations on their "jsonb" objects.

- C

Re: [HACKERS] Tackling JsonPath support

[Christian Convey]

On Mon, Nov 28, 2016 at 5:20 AM, Pavel Stehule 
wrote:
...

> Incremental work is great idea - I like this this style. Instead contrib,
> you can use public repository on github. Minimally for first stage is
> better to live outside core - you are not restricted by PostgreSQL
> development process. When your code will be stabilized, then you can go to
> commitfest. I believe so we need good JSON support. The XML support helps
> to PostgreSQL lot of, JSON will be great too.
>

​Hi Pavel,

Thanks for the suggestion.

I am planning to use one of my own public github repos as the location for
my work.

I thought by adding my first implementation to "contrib", we could make
this functionality available to end-users, even before there was a
consensus about what PG's "official" JSON-related operators should have for
syntax and semantics.

Does my reasoning make sense?

- C

Re: [HACKERS] Tackling JsonPath support

[Christian Convey]

On Mon, Nov 28, 2016 at 5:20 AM, Pavel Stehule 
wrote:
​...​

> Con: "JSON path expression" is a recurring them in the *grammars* of
>> user-facing operators in [1], [2], [3], and [4].  But it doesn't
>> necessarily follow that the function implemented in Step 2 will provide
>> useful infrastructure for PG's eventual implementations of "JSON_TABLE",
>> etc.
>>
>
> We can implement subset only - our XPath based on libxml2 does it too. The
> good target is support of usual examples on the net.
>

​Hi Pavel,

Can you clarify what you meant?  I *think* you're saying:

* It's not important for me to match the syntax/semantics of the json-path
implementations found in MySQL / Oracle / DB2 / ​MS SQL Server, and

* Instead, I should just use examples / explanations on the web as my
guidance.

​Thanks,​
Christian

Re: [HACKERS] Tackling JsonPath support

[Christian Convey]

>From looking at other databases' docs, it seems like the behavior of
various JSON-related operators / functions are described partially in terms
of a "json path expression":

* In Oracle, "JSON_TABLE", "JSON_exists_column", "JSON_value_column": [1]
* In MySQL: [2]
* In DB2: [3]
* In MS SQL Server: [4]
* (Whatever the Standards committee will end up producing.)

If I'm correctly understanding the situation, It sounds like we have two
big unknowns:

(a) The exact syntax/semantics of JSON path searching, especially w.r.t.
corner cases and error handling, and

(b) The syntax/semantics of whatever SQL operators / functions are
currently defined in terms of (a).  E.g., "JSON_TABLE".

If that's correct, then what do you guys think about us taking the
following incremental approach?

Step 1: I'll dig into the implementations described above, to see what's
similar and different between the JSON-path-expression syntax and semantics
offered by each.  I then report my findings here, and we can hopefully
reach a consensus about the syntax/semantics of PG's json-path-expression
handling.

Step 2: I submit a patch for adding a new function to "contrib", which
implements the JSON-path-expression semantics chosen in Step 1.  The
function will be named such that people won't confuse it with any
(eventual) SQL-standard equivalent.

Step 3: PG developers can, if they choose, start defining new JSON operator
/ functions, and/or port existing JSON-related functions, in terms of the
function created in Step 2.

I see the following pros / cons to this approach:

Pro: It gives us a concrete start on this functionality, even though we're
not sure what's happening with the SQL standard.

Pro: The risk of painting ourselves into a corner is relatively low,
because we're putting the functionality in "contrib", and avoid function
names which conflict with likely upcoming standards.

Pro: It might permit us to give PG users access to JSONPath -like
functionality sooner than if we wait until we're clear on the ideal
long-term interface.

Con: "JSON path expression" is a recurring them in the *grammars* of
user-facing operators in [1], [2], [3], and [4].  But it doesn't
necessarily follow that the function implemented in Step 2 will provide
useful infrastructure for PG's eventual implementations of "JSON_TABLE",
etc.

- Christian

[1] https://docs.oracle.com/database/121/SQLRF/functions092.htm#SQLRF56973
[2] https://dev.mysql.com/doc/refman/5.7/en/json-path-syntax.html
[3]
http://www.ibm.com/support/knowledgecenter/ssw_ibm_i_72/db2/rbafzjsonpath.htm
[4] https://msdn.microsoft.com/en-us/library/mt577087.aspx

Re: [HACKERS] Tackling JsonPath support

[Christian Convey]

Hi Pavel,

Can I check a few assumptions about what you're suggesting for this task?

* Our ultimate goal is to give Postgres an implementation of the functions
"JSON_EXISTS", "JSON_VALUE", and "JSON_QUERY" which fully comply with the
SQL standards.

* The best representation of those standards is found here: [1].

* When [1] mentions a "JSON path expression" or "JSON path language", it's
referring to the query language described here: [2].

* Even if other popular DBMS's deviate from [1], or other popular JSONPath
implementations deviate from [2], we remain committed to a faithful
implementation of [1].

* It's okay for my first commit to implement just two things: (a) a
PG-internal implementation of JsonPath, and (b) a user-visible
implementation of "JSON_QUERY" based on (a).  Later commits could add
implementations of "JSON_VALUE", "JSON_EXISTS", etc. in terms of (a).

Thanks,
Christian

[1]
http://jtc1sc32.org/doc/N2501-2550/32N2528-WG3-Tutorial-Opening-Plenary.pdf

[2] http://goessner.net/articles/JsonPath


On Fri, Sep 16, 2016 at 2:28 AM, Pavel Stehule <pavel.steh...@gmail.com>
wrote:

> Hi
>
> 2016-09-15 18:05 GMT+02:00 Christian Convey <christian.con...@gmail.com>:
>
>> On Mon, Sep 5, 2016 at 1:44 PM, Pavel Stehule <pavel.steh...@gmail.com>
>> wrote:
>> ...
>>
>> > I wrote XMLTABLE function, and I am thinking about JSON_TABLE function.
>> But
>> > there is one blocker - missing JsonPath support in our JSON
>> implementation.
>> >
>> > So one idea - implement JsonPath support and related JSON query
>> functions.
>> > This can help with better standard conformance.
>>
>> Hi Pavel,
>>
>> Are you still looking for someone to add the JsonPath support to the
>> JSON implementation?  And if so, how urgently are people waiting for
>> it?
>>
>
> yes - JsonPath support should be great. I hope so this or next commitfest
> the XMLTABLE patch will be committed, and with JsonPath I can start to work
> on JSON_TABLE function.
>
> But the JsonPath can be merged separately without dependency to
> JSON_TABLE. There are more JSON searching functions, and these functions
> should to support JsonPath be ANSI SQL compliant.
>
>
>>
>> I'd be happy to start working on a patch, but since I'm new to PG
>> development, I'm probably not the fastest person to get it done.
>>
>
> It is not problem. Probably you should to do this work without deep
> knowledges about PostgreSQL internals. The work with data types (and
> functions for data types) is well isolated from PostgreSQL engine.
>
> You can learn from current searching on JSON -
> postgresql/src/backend/utils/adt/json.c
>
> And it is good start to be PostgreSQL's hacker - I started with
> implementation of own data type and related functions.
>
> Regards
>
> Pavel
>
>
>> Kind regards,
>> Christian
>>
>
>

[HACKERS] Tackling JsonPath support

[Christian Convey]

On Mon, Sep 5, 2016 at 1:44 PM, Pavel Stehule  wrote:
...

> I wrote XMLTABLE function, and I am thinking about JSON_TABLE function. But
> there is one blocker - missing JsonPath support in our JSON implementation.
>
> So one idea - implement JsonPath support and related JSON query functions.
> This can help with better standard conformance.

Hi Pavel,

Are you still looking for someone to add the JsonPath support to the
JSON implementation?  And if so, how urgently are people waiting for
it?

I'd be happy to start working on a patch, but since I'm new to PG
development, I'm probably not the fastest person to get it done.

Kind regards,
Christian


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [GENERAL] C++ port of Postgres

[Christian Convey]

Some of your patches look useful, but unrelated to C++: 7, 8, 15, 16(?), 20.

I applied that subset to 9.6 and got a clean "make check".

Would it make sense to add them to the next commitfest, regardless of
the C++ effort?

On Wed, Aug 31, 2016 at 9:41 AM, Peter Eisentraut
 wrote:
> [trimmed cc list because of big attachments]
>
> On 8/16/16 4:22 PM, Jim Nasby wrote:
>> Joy, do you have an idea what a *minimally invasive* patch for C++
>> support would look like? That's certainly the first step here.
>
> I developed a minimally invasive patch for C++ support a few years ago
> shortly after I wrote that blog post.  Since there appears to have been
> some interest here now, I have updated that and split it up into logical
> chunks.
>
> So here you go.
>
> To build this, you need to configure with g++ <= version 5.  (4.x works,
> too.)  g++ version 6 does not work yet because of the issues described
> in patch 0013.
>
> Then you also need to edit src/Makefile.custom and set
>
> COPT = -fpermissive -Wno-sign-compare -Wno-write-strings
>
> The -W options are optional just to reduce some noise.  Cleaning up
> those warnings can be a separate project that might also have some
> benefit under C.
>
> The -fpermissive option is a g++ specific option that reduces some
> errors to warnings.  (So this won't work with clang or other compilers
> at all at this point.)  In particular, C++ does not allow casting from
> or to void pointers without a cast, but -fpermissive allows that.  The
> step from this to "real" C++ would be adding a bunch of casts around
> things like malloc and palloc and other places.  That would be mostly
> busy work, so I have excluded that here.
>
> The patches are numbered approximately in increasing order of dubiosity.
>  So 0001 is probably a straight bug fix, 0002 and 0003 are arguably
> minor bug fixes as well.  The patches through 0012 can probably be
> considered for committing in some form.  After that it gets a bit hackish.
>
> --
> Peter Eisentraut  http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [GENERAL] C++ port of Postgres

[Christian Convey]

> P.S. I'm asking because I was planning to review that patch.  But I
>> can't tell if any more review by a non-committer is still required by
>> the commitfest workflow.
>
>
> I think this has gotten enough attention, for the commitfest workflow. The
> workflow is flexible, depending on the nature of patch. But of course, if
> you're interested, feel free to review and comment anyway!

Hi Heikki,

Thanks for the help.  If you think nothing more is needed for this
patch set w.r.t. the 2016-09 commitfest, I'll move on to other things.

- Christian


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [GENERAL] C++ port of Postgres

[Christian Convey]

Hi Heikki,

Could I ask you a newbie-reviewer question about something I'm seeing
here?  https://commitfest.postgresql.org/10/776/

>From some reading I've done (e.g., Stephen Frost's PGCon 2011 slides),
I got the impression that a successful patch would always have this
sequence of states in commitfest:
  1. patch-record created
  ...
  2. Needs Review
  ...
  3. Ready for Committer

But if I'm reading the patch's activity log correctly, it looks like
you marked the patch as "Ready for Committer" (2016-09-06 18:59:02)
without any record of it having been reviewed.

Was that intentional?

Thanks very much,
Christian

P.S. I'm asking because I was planning to review that patch.  But I
can't tell if any more review by a non-committer is still required by
the commitfest workflow.

Kind regards,
Christian

On Tue, Sep 6, 2016 at 3:15 PM, Christian Convey
<christian.con...@gmail.com> wrote:
> On Tue, Sep 6, 2016 at 3:12 PM, Tom Lane <t...@sss.pgh.pa.us> wrote:
>>> (2) It seems like there are still a few big questions about this commit:
>>>- Is it wanted at the moment?  It didn't seem like there's a
>>>  consensus about whether or not this enhancement should be
>>>  merged, even if the patch is pretty minimal.
>>>- It seems like there are two competing patch
>>>  sets in play for this enhancement: Joy's and
>>>  Peter's.  Presumably at most one of them would
>>>  be merged.
>>
>> These are things that reviews should be helping to decide.  It's probably
>> a squishier topic than some patches, but if you're interested, feel free
>> to read code and weigh in.
>
> Thanks. It sounds like worst-case scenario, I perform an unneeded
> review.  I'll give it a shot.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [GENERAL] C++ port of Postgres

[Christian Convey]

> Thanks. It sounds like worst-case scenario, I perform an unneeded
> review.  I'll give it a shot.

Hi guys,

Apologies for more boring process-related questions, but any pointers
would be greatly appreciated...

I'm a bit confused about how PG's code-review process is meant to
handle this C++ port.  My confusion may stem from the combination of
my inexperience with the process, and there being two competing patch
sets.

Here's some background:

* My intention was to review Joy's patch.

* On "commitfest.postgresql.org" (for 2016-09), the only C++
  -related patch I found was Peter's: [1]

* I wrongly assumed that the commitfest entry would be for
  Joy's patch, not Peter's, so I signed up as its reviewer.
  (That's fine - I don't mind reviewing both authors' patch
  sets.)

But here are my questions:

Q1) My understanding of PG's code-review process is that it's a pipeline:
Step 1. The discussion starts on the pgsql-hackers mailing list, where
the author posts a patch.  He/she may also post revised patches
based on the discussion.

Step 2. A subset of those discussions are modeled by new entries in
the commitfest website.

Step 3. A subset of those commitfest items get merged.

   If that's correct, then it sounds like the only way Joy's commit has
   a chance of acceptance is if Peter's commit is rejected.
   Because Peter's commit might be merged as part of the 2016-09
   commitfest, but Joy's can show up until 2016-11 at the earliest.

   Is my understanding correct?

There seems to be a little ambiguity regarding the exact version of
the code to be reviewed.  This is true for both Joy's and Peter's
submissions:
   * Joy's email provides a link to a Github repo, but does not specify
 a particular commit (or even branch) in that repo: [2]

   * In the email thread, Peter did provide a patch set: [3]
 but the corresponding commitfest entry references a github branch: [4]

So I have a few questions here:

Q2) Are authors expected to submit an unambiguous patch to frame the
discussion?  (I.e,. a specific patch file, or a specific git commit
hash, as opposed to a github repo or a github branch.)

Q3) Are authors expected to submit a single patch/commit, or is it
acceptable / desirable for a large patch to be broken up as Peter has
done?

Q4) Do we require that any submitted patches appear as attachments on
the pgsql-hackers email list, or is a github URL good enough?

Q5) (This question is more generic.)  I'm accustomed to using Github's
pull-request system, where I can engage in dialog regarding specifc
lines of a patch.  I haven't noticed anything similar being used for
PG code reviews, but perhaps I'm just looking in the wrong places.
Are all PG code reviews basically just back-and-forth email
conversations on the pgsql-hackers list?

Thanks,
Christian

[1] https://commitfest.postgresql.org/10/776/
[2] 
https://www.postgresql.org/message-id/CABgyVxDBd3EvRdo-Rd6eo8QPEqV8%3DShaU2SJfo16wfE0R-hXTA%40mail.gmail.com
[3] 
https://www.postgresql.org/message-id/bf9de63c-b669-4b8c-d33b-4a5ed11cd5d4%402ndquadrant.com
[4] https://github.com/petere/postgresql/tree/commitfest/c%2B%2B


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Suggestions for first contribution?

[Christian Convey]

On Wed, Sep 7, 2016 at 10:44 AM, Stas Kelvich  wrote:
> There is also a list of projects for google summer of code: 
> https://wiki.postgresql.org/wiki/GSoC_2016
>
> That topics expected to be doable by a newcomer during several months. It is 
> also slightly
> outdated, but you always can check current state of things by searching this 
> mail list on interesting topic.
>
> Also postgres have several internal API’s like fdw[1] or gist[2] that allows 
> you to implements something
> useful without digging too much into a postgres internals.
>
> [1] https://www.postgresql.org/docs/current/static/fdwhandler.html
> [2] https://www.postgresql.org/docs/current/static/gist-extensibility.html

Thanks Stas, I'll have a look at those.  I appreciate the links.

- Christian


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Suggestions for first contribution?

[Christian Convey]

On Wed, Sep 7, 2016 at 10:34 AM, Yury Zhuravlev
<u.zhurav...@postgrespro.ru> wrote:
> Christian Convey wrote:
>>
>> Yury doesn't seem to need help
>> with CMake
>
> Hello.
> I am sorry that the only answer is now.
> I need help but with write CMake code:
> 1. Make ecpg tests
> 2. Make MinGW/Msys support
> 3. many many ...
> all targets and discussion here:
> https://github.com/stalkerg/postgres_cmake/issues
>
> If you can only test CMake for Ubuntu x86_64 that it is not necessary.
> If I not fully understand you - sorry.

Hi Yury, no problem.  It sounds like most of the platform testing you
want requires hardware I don't have, unfortunately.

- Christian


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Suggestions for first contribution?

[Christian Convey]

On Wed, Sep 7, 2016 at 10:14 AM, Aleksander Alekseev
 wrote:
> Here is another idea for a contribution - refactoring.
>
> Currently there are a lot of procedures in PostgreSQL code that
> definitely don't fit on one screen (i.e. ~50 lines). Also many files are
> larger than say 1000 lines of code. For instance, psql_completion
> procedure is more than 2000 lines long! I think patches that would make
> such code easier to read would have a good chance to be accepted.

Thanks for the suggestion!  I can see how refactoring could make a lot
of sense for some functions.  That's probably a task I'd want to take
on after I had more experience with PG's code base.  I think being
familiar with most/all of PG's code would make that task go more
smoothly.

- Christian


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [GENERAL] C++ port of Postgres

[Christian Convey]

On Tue, Sep 6, 2016 at 3:12 PM, Tom Lane  wrote:
>> (2) It seems like there are still a few big questions about this commit:
>>- Is it wanted at the moment?  It didn't seem like there's a
>>  consensus about whether or not this enhancement should be
>>  merged, even if the patch is pretty minimal.
>>- It seems like there are two competing patch
>>  sets in play for this enhancement: Joy's and
>>  Peter's.  Presumably at most one of them would
>>  be merged.
>
> These are things that reviews should be helping to decide.  It's probably
> a squishier topic than some patches, but if you're interested, feel free
> to read code and weigh in.

Thanks. It sounds like worst-case scenario, I perform an unneeded
review.  I'll give it a shot.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [GENERAL] C++ port of Postgres

[Christian Convey]

On Wed, Aug 31, 2016 at 9:41 AM, Peter Eisentraut
 wrote:
>> Joy, do you have an idea what a *minimally invasive* patch for C++
>> support would look like? That's certainly the first step here.
>
> I developed a minimally invasive patch for C++ support a few years ago
> shortly after I wrote that blog post.  Since there appears to have been
> some interest here now, I have updated that and split it up into logical
> chunks.
>
> So here you go.

Hi folks,

I'm ready for my first-ever PG code review, and I was considering
tackling this one.  I.e., https://commitfest.postgresql.org/10/776/

Could someone help me with a few procedural questions?

(1) This page: https://wiki.postgresql.org/wiki/Reviewing_a_Patch
lists the current commitfest's manager as "(vacant)".  But this page:
https://commitfest.postgresql.org/ seems to indicate that a commitfest
is currently in progress.  Is there a commitfest manager at the
moment?

(2) It seems like there are still a few big questions about this commit:

   - Is it wanted at the moment?  It didn't seem like there's a
 consensus about whether or not this enhancement should be
 merged, even if the patch is pretty minimal.

   - It seems like there are two competing patch
 sets in play for this enhancement: Joy's and
 Peter's.  Presumably at most one of them would
 be merged.

   Are these signs that I should find a different commit to review
   for now, so that this one can be further discussed?

Thanks very much,
Christian


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Suggestions for first contribution?

[Christian Convey]

Thanks everyone for the suggestions.

It sounds like the most useful thing I can do at the moment is perform
code reviews.  I assumed I'd need more experience with the PG code
base, but I keep on reading that newcomers' reviews are welcome.
Unless someone has a better idea, I'll start with that.

The projects that Pavel and David mentioned sound very interesting.
Hopefully I can tackle one of them after I've done a few code reviews.

Thanks again,
Christian


On Tue, Sep 6, 2016 at 4:20 AM, Aleksander Alekseev
 wrote:
> Hello, Christian
>
>> Can anyone suggest a project for my first PG contribution?
>
> In my experience good starting points are:
>
> * Doing code review. It's really important for PostgreSQL project.
>   Everyone likes to write code, no one likes to read or test it :)
> * Finding typos, there are a lot of them.
> * Finding bugs: static code analysis, sanitizers, etc. Also subscribe to
>   pgsql-bugs@ mailing list.
> * Fixing bottlenecks. Tricky part here is to find one. Could be
>   difficult unless you are working for a company with many clients or
>   maintain a project that uses PostgreSQL.
>
> Also please note that such activity as answering questions on IRC or in
> pgsql-general@, highlighting project news in your blog, writing articles
> (or maybe even a book) about PostgreSQL internals, organizing local user
> group, etc - all of this are very significant contributions.
>
> Good luck!
>
> --
> Best regards,
> Aleksander Alekseev


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Suggestions for first contribution?

[Christian Convey]

Hi guys,

Can anyone suggest a project for my first PG contribution?

My first two ideas didn't pan out:  Yury doesn't seem to need help
with CMake, and the TODO list's "-Wcast-align" project (now deleted)
appeared impractical.

I can continue trying things from the TODO list, but if someone knows
of a project that's definitely worth doing, all the better.

I'd prefer doing back-end coding, but I'm flexible.  The only two
caveats are: (1) I can't work on speedup-focused changes without my
employer's permission, and (2) I only have access to Linux boxes.

Any advice would be greatly appreciated.

Kind regards,
Christian


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Obsolete TODO item "-Wcast-align" ?

[Christian Convey]

On Sun, Sep 4, 2016 at 5:56 PM, Tom Lane <t...@sss.pgh.pa.us> wrote:
> Christian Convey <christian.con...@gmail.com> writes:
>> I chose this item from the TODO page: "[E] Remove warnings created by
>> -Wcast-align".  It didn't have a check-mark after the "[E]", which I
>> took to mean it's an outstanding issue.
>> However, I'm starting to wonder if it's already been fixed:
>
> No, but you only see it on some platforms/compilers.  On my OS X laptop
> (clang-based not gcc-based compiler), turning that on generates just a
> hair short of 13000 warnings :-(
>
> I think that TODO item is indeed obsolete, but more in the direction
> of "we're never gonna do that".  There are too many places where we
> do need to cast up from generic pointer to specific structure pointer,
> and there doesn't seem to be any practical way to tell a compiler which
> instances are useful to warn about.

Thanks for the response.  I'm unclear about how the TODO list is
curated.  Is there someone whose attention I should direct to this
thread?

- Christian


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Obsolete TODO item "-Wcast-align" ?

[Christian Convey]

Hi guys,

I'm trying to get my feet wet in PG development, but I ran into a snag
with the TODO item I picked.  Could someone please tell me if I'm
making an obvious error?

I chose this item from the TODO page: "[E] Remove warnings created by
-Wcast-align".  It didn't have a check-mark after the "[E]", which I
took to mean it's an outstanding issue.

However, I'm starting to wonder if it's already been fixed:

(1) I wasn't able to reproduce it.  I built PG using:
 * PG source commit 6591f4...  (a recent master/HEAD)
 * Ubuntu's patched version of GCC 4.8.4
 * "-Wcast-align"
but I didn't see anything in the compiler output indicating that the
warning got triggered.

(2) It looks like someone else tried to tackle this TODO item about
two years ago, but he also was unable to reproduce it:
https://www.postgresql.org/message-id/flat/CALgh3eseTDT9RspDi0AfhE0j0A3e%2B5XiS3ykQyhK7Z5RgPQTLA%40mail.gmail.com#calgh3esetdt9rspdi0afhe0j0a3e+5xis3ykqyhk7z5rgpq...@mail.gmail.com

I didn't find any other discussions about that line-item via Google,
or by looking at the official/unofficial TODO-detail pages.

Thanks very much,
Christian Convey


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP: About CMake v2

[Christian Convey]

Hi Yury,

>> I glad to hear it. I suppose you can just try build postgres and send all
>> problems to github tracker.
>> https://github.com/stalkerg/postgres_cmake/issues

FYI, I had success using your "postgres_cmake" repo.  I tested it up
through "make check" and "make install".

Here are the details:

* postgres_cmake commit e7e75160d4533cd8caa9f3f0dd7b485dbd4e7bdf
* compiler = cc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4
* cmake version = 3.5.3

Kind regards,
Christian


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP: About CMake v2

[Christian Convey]

Hi Yury,


On Fri, Aug 19, 2016 at 9:46 AM, Yury Zhuravlev
<u.zhurav...@postgrespro.ru> wrote:
> Christian Convey wrote:
>>
>> I'm interested in helping with your CMake effort.  I don't have any
>> experience contributing to PG, but I do have some free time at the
>> moment.  Please let me know if I can help.
>
> I glad to hear it. I suppose you can just try build postgres and send all
> problems to github tracker.
> https://github.com/stalkerg/postgres_cmake/issues

Thanks, I'll be happy to do that.  There's been a lot of discussion on
this thread regarding the minimum required CMake version.  If you'd
like me to test with a particular version (or versions) of CMake,
please let me know.

- Christian


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP: About CMake v2

[Christian Convey]

Hi Stefan,

>> I ask because I'm curious if/how someone in Yury's situation could
>> predict which minimum version of CMake must be supported in order for
>> his patch to be accepted.  (And if he accepts my offer to pitch in,
>> I'll actually need that particular detail.)
>
> well I personally think the level to meet would be that all the systems
> on the buildfarm that can build -HEAD at the time the patch is proposed
> for a commit should be able to build using the new system with whatever
> cmake version is available in those by default (if it is at all).

What standard would you suggest for those platforms which don't have
an obvious default version of CMake?

I assume this includes all versions of Windows and of OS X, but
perhaps I'm misinformed.

Thanks,
Christian


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP: About CMake v2

[Christian Convey]

>
> I don't think we're interested in maintaining more build systems than we
> already are.  If cmake can replace the current MSVC tooling and
> autoconf, all in one, my impression is that we're in.  If we're
> replacing two tools we've hammered pretty well over the years with two
> tools that we haven't, I doubt we're interested.

Good to know, thanks.

- Christian


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP: About CMake v2

[Christian Convey]

Hi Tom,

>> I ask because I'm curious if/how someone in Yury's situation could
>> predict which minimum version of CMake must be supported in order for
>> his patch to be accepted.  (And if he accepts my offer to pitch in,
>> I'll actually need that particular detail.)
>
> well I personally think the level to meet would be that all the systems
> on the buildfarm that can build -HEAD at the time the patch is proposed
> for a commit should be able to build using the new system with whatever
> cmake version is available in those by default (if it is at all).

I see.  In other projects I've worked on, the configuration of a build
farm has been driven by some list of platforms that were considered
important to support.

Is that the case here as well?  I.e., is the build-farm population
just a convenient proxy for some other source of information regarding
what platforms are important?

Apologies if my questions are so basic that I can find the answers
elsewhere.  I'll happily follow any RTFM links.

Thanks again,
Christian


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP: About CMake v2

[Christian Convey]

Hi Stefan,

>> Yury: Would it make sense to add a call to "cmake_minimum_required" in
>> one or more of your CMakeLists.txt files?
>
> it would make sense nevertheless but I dont think that 2.8.11 is old
> enough - looking at the release information and the feature compatibily
> matrix it would seems we should more aim at something like 2.8.0 or 2.8.3...

I'm new to PG development, so I don't know what ideas the community is
open to.  But I wonder if there's any merit to the following
approach...

* Allow the CMake-based build system to assume a fairly modern version
of CMake.  (Maybe 2.8.12, or 3.0.)

* For systems where the minimum CMake version isn't readily available,
have an alternative build system which is just a simplistic Bash
script that naively performs a full build every time it's invoked.
The idea being that PG contributors are mostly the people who want
efficient rebuilds, and most/all of them could easily install that
minimal CMake version.

*IF* it proved possible to write a clear, maintainable Bash script for
that, perhaps that would eliminate most concerns about CMake not being
well-supported on uncommon platforms / platform versions.

- Christian


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP: About CMake v2

[Christian Convey]

Hi Tom,

Thanks for that information.

Is there some document I can read that explains which platform
versions (e.g., OpenBSD 5.3) are considered strongly supported?

I ask because I'm curious if/how someone in Yury's situation could
predict which minimum version of CMake must be supported in order for
his patch to be accepted.  (And if he accepts my offer to pitch in,
I'll actually need that particular detail.)

Kind regards,
Christian


> As an additional comment, I don't see us accepting a build system
> that doesn't run on pretty old cmakes.  We have never had a policy
> of "latest and greatest is required" for *any* build support tool,
> and cmake isn't likely to be given an exception.
>
> BTW, I just noticed that cmake doesn't seem to be supplied as part of
> Apple's dev tools, at least not up to current (El Capitan) releases.
> That's going to be a rather large minus to be taken into account
> whenever we make the go/no-go decision on this.
>
> regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP: About CMake v2

[Christian Convey]

Hi Karl,

I'll need to let Yury answer your original question regarding the best
way to report CMake-related bugs.

Regarding the errors you're getting...  I just looked at CMake's
online documentation regarding your "target_compile_definitions"
error.

>From what I can tell, the "target_compile_definition" property was
introduced in CMake 2.8.11.  It sounds like your version of CMake is
just a little too old.

Regarding how one can know the required CMake version: My modus
operandi for CMake projects in general is (1) read the project's
how-to-build docs, and if that's not heplful, (2) hope that the
project's CMake files contain a "cmake_minimum_required" call to give
me a clear error message.  I didn't find any such indication in Yuri's
files, although perhaps I missed it.


Yury: Would it make sense to add a call to "cmake_minimum_required" in
one or more of your CMakeLists.txt files?

Kind regards,
Christian

On Thu, Aug 18, 2016 at 3:08 PM, Stefan Kaltenbrunner
<ste...@kaltenbrunner.cc> wrote:
> On 08/18/2016 08:57 PM, Christian Convey wrote:
>> Hi Stefan,
>>
>> I think I've seen similar errors when a project's CMake files assumed
>> a newer version of CMake than the one being run.
>>
>> Which version of CMake gave you those errors?  (Sorry if you provided
>> that detail and I'm just missing it.)
>
>
> % cmake --version
> cmake version 2.8.10.2
>
> a quick look in the docs does not seem to reveal any kind of "minimum"
> cmake version required - and the above is what is packaged as part of
> openbsd 5.3 (which is outdated and unsupported upstream but it is
> currently perfectly fine building all postgresql versions including
> -HEAD and serving as a buildfarm member for years)
>
>
>
> Stefan


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP: About CMake v2

[Christian Convey]

Hi Stefan,

I think I've seen similar errors when a project's CMake files assumed
a newer version of CMake than the one being run.

Which version of CMake gave you those errors?  (Sorry if you provided
that detail and I'm just missing it.)

Kind regards,
Christian

On Thu, Aug 18, 2016 at 2:45 PM, Stefan Kaltenbrunner
 wrote:
> On 06/29/2016 06:23 PM, Yury Zhuravlev wrote:
>> Hello Hackers.
>>
>> I decided to talk about the current state of the project:
>> 1. Merge with 9.6 master. 2. plpython2, plpython3, plperl, pltcl, plsql
>> all work correctly (all tests pass).
>> 3. Works done for all contrib modules. 4. You can use gettext, .po->.mo
>> will have converted by CMake.  5. All test pass under some Linux,
>> FreeBSD, Solaris10 (on Sparc), Windows MSVC 2015. MacOSX I think not big
>> trouble too.  6. Prototype for PGXS (with MSVC support) done.
>> I think is very big progress but I came across one problem.
>> I not have access to many OS and platforms. For each platform need tests
>> and small fixes. I can't develop and give guarantee without it.
>>
>> I think this is very important work which makes it easier further
>> support Postgres but I can not do everything himself. It's physically
>> impossible.
>>
>> I think without community support I can't do significantly more.
>>
>> Current version you can get here:
>> https://github.com/stalkerg/postgres_cmake
>
> hmm how do you actually want reports on how it works?
>
> I just played with it on spoonbill (OpenBSD 5.3/sparc64) and got this:
>
> CMake Error at CMakeLists.txt:1250 (file):
>   file does not recognize sub-command GENERATE
>
>
> CMake Error at src/port/CMakeLists.txt:156 (target_compile_definitions):
>   Unknown CMake command "target_compile_definitions".
>
>
> -- Configuring incomplete, errors occurred!
>
>
> there is also a ton of stuff like:
>
>
> Make Error: Internal CMake error, TryCompile generation of cmake failed
> -- Looking for opterr - not found
> -- Looking for optreset
> CMake Error at CMakeLists.txt:10 (ADD_EXECUTABLE):
>   Target "cmTryCompileExec3458204847" links to item " m" which has
> leading or
>   trailing whitespace.  This is now an error according to policy CMP0004.
>
>
> CMake Error: Internal CMake error, TryCompile generation of cmake failed
> -- Looking for optreset - not found
> -- Looking for fseeko
> CMake Error at CMakeLists.txt:10 (ADD_EXECUTABLE):
>   Target "cmTryCompileExec2628321539" links to item " m" which has
> leading or
>   trailing whitespace.  This is now an error according to policy CMP0004.
>
>
> CMake Error: Internal CMake error, TryCompile generation of cmake failed
>
>
> which I have no idea whether they are an actual problem or just
> "configure" noise
>
>
>
>
>
> Stefan
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP: About CMake v2

[Christian Convey]

Hi Yury,

I'm interested in helping with your CMake effort.  I don't have any
experience contributing to PG, but I do have some free time at the
moment.  Please let me know if I can help.

I have an intermediate level of experience with CMake, Python, and
Bash scripting.  My native environment is Mint 17.3 (i.e., Ubuntu
15.10), but I'd be happy to create additional VM's as needed.  I'm
also happy to look into bugs on other systems (VMS, etc.) if I can get
SSH access.

Kind regards,
Christian Convey


On Fri, Jul 1, 2016 at 5:26 AM, Dmitry Maslyuk <d.mas...@postgrespro.ru> wrote:
> Hi!
>
> I think, we need simple configure script generator for backward
> compatibility and easy using this build system.
> Try playing with cmake build system under Win2008+MinGW. I plan to write
> perl script for automatic build this
> with depends.
>
> On 29.06.2016 19:23, Yury Zhuravlev wrote:
>>
>> Hello Hackers.
>>
>> I decided to talk about the current state of the project:
>> 1. Merge with 9.6 master. 2. plpython2, plpython3, plperl, pltcl, plsql
>> all work correctly (all tests pass).
>> 3. Works done for all contrib modules. 4. You can use gettext, .po->.mo
>> will have converted by CMake.  5. All test pass under some Linux, FreeBSD,
>> Solaris10 (on Sparc), Windows MSVC 2015. MacOSX I think not big trouble too.
>> 6. Prototype for PGXS (with MSVC support) done.
>> I think is very big progress but I came across one problem.
>> I not have access to many OS and platforms. For each platform need tests
>> and small fixes. I can't develop and give guarantee without it.
>>
>> I think this is very important work which makes it easier further support
>> Postgres but I can not do everything himself. It's physically impossible.
>>
>> I think without community support I can't do significantly more.
>>
>> Current version you can get here:
>> https://github.com/stalkerg/postgres_cmake
>>
>> Thanks!
>
>
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Breaking compile-time dependency cycles of Postgres subdirs?

[Christian Convey]

On Sun, Feb 9, 2014 at 8:06 PM, Robert Haas robertmh...@gmail.com wrote:

 On Fri, Feb 7, 2014 at 7:39 AM, Christian Convey
 christian.con...@gmail.com wrote:
  This question is mostly just curiosity...

 As someone very new to this code base, I think these cycles make it a
 little
  harder to figure out the runtime and compile-time dependencies between
 the
  subsystems these directories seem to represent.  I wonder if that's a
  problem others face as well?

 There are probably some cases that could be improved, but I have my
 doubts about whether eliminating cycles is a reasonable goal.
 Sometimes, two modules really do depend on each other.  And, you're
 talking about this not just on the level of individual files but
 entire subtrees.  There are 90,000 lines of code in src/backend/access
 (whose headers are in src/include/access) and more than 38,000 in
 src/backend/storage (whose headers are in src/include/storage);
 expecting all dependencies between those modules to go in one
 direction doesn't feel terribly reasonable.  If it could be done at
 all, you'd probably end up separating code into lots of little tiny
 directories, splitting apart modules with logically related
 functionality into chunks living in entirely different parts of the
 source tree - and I don't think that would be an improvement.


Thanks Robert.  IMHO, whether or not it would be beneficial depends on
which files (or definitions within files) had to be broken out into
additional subdirectories in order to break the cycles.   If it could be
accomplished with at most a few additional subdirectories that were also
intuitively meaningful groupings of files/definitions, it could be a win.
 But if not, I agree it would be a step backwards.

Still, I'm thinking this might be a problem we need to partially solve if
we're going to support a pluggable storage manager, particularly if we
allow a pluggable storage manager to use the system's buffer system and/or
block I/O system.  I guess it depends on exactly what we want from a
pluggable storage manager.

- Christian

Re: [HACKERS] Breaking compile-time dependency cycles of Postgres subdirs?

[Christian Convey]

On Mon, Feb 10, 2014 at 10:28 AM, Tom Lane t...@sss.pgh.pa.us wrote:

 I think if it had been a clear, enforced goal all along, it might've been
 possible to build the system with such a restriction (for the most part at
 least).  At this point though, the amount of work and code churn involved
 seems like it'd far exceed the benefits.


That makes sense to me.  I certainly didn't think it was a slam-dunk that
what I was proposing would be an improvement.  It just seemed like a
question worth asking.  Thanks for your thoughts.

- Christian

[HACKERS] Breaking compile-time dependency cycles of Postgres subdirs?

[Christian Convey]

This question is mostly just curiosity...

There are build-time dependency cycles between some of Postgres' code
subdirectories.  For example, storage and access have such a cycle:
storage/buffpage.h #includes access/xlogdefs.h
access/visibilitymap.h #includes storage/block.h

Has there been any discussion about reorganizing these directories so that
no such cycles exist?

As someone very new to this code base, I think these cycles make it a
little harder to figure out the runtime and compile-time dependencies
between the subsystems these directories seem to represent.  I wonder if
that's a problem others face as well?

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

[Christian Convey]

On Mon, Jan 27, 2014 at 7:14 PM, Kouhei Kaigai kai...@ak.jp.nec.com wrote:

 FDW's join pushing down is one of the valuable use-cases of this interface,
 but not all. As you might know, my motivation is to implement GPU
 acceleration
 feature on top of this interface, that offers alternative way to scan or
 join
 relations or potentially sort or aggregate.


I'm curious how this relates to the pluggable storage idea discussed here
https://wiki.postgresql.org/wiki/PgCon_2013_Developer_Meeting and here
http://www.databasesoup.com/2013/05/postgresql-new-development-priorities-2.html


I haven't seen a specific proposal about how much functionality should be
encapsulated by a pluggable storage system.  But I wonder if that would be
the best place for specialized table-scan code to end up?

Re: [HACKERS] Changeset Extraction v7.3

[Christian Convey]

It seems to me that the terms physical, logical, and binary are
always relative to the perspective of the component being worked on.

Physical often means one level of abstraction below mine, and upon which
my work builds.  Logical means my work's level of abstraction.  And
Binary means data which I'm not going to pretend I know or care how to
interpret.

So I'd suggest block and tuple, perhaps.


On Wed, Jan 29, 2014 at 4:25 AM, Albe Laurenz laurenz.a...@wien.gv.atwrote:

 Andreas Karlsson wrote:
  On 01/28/2014 10:56 PM, Andres Freund wrote:
  On 2014-01-28 21:48:09 +, Thom Brown wrote:
  On 28 January 2014 21:37, Robert Haas robertmh...@gmail.com wrote:
  On Tue, Jan 28, 2014 at 11:53 AM, Robert Haas robertmh...@gmail.com
 wrote:
  The point of Andres's patch set is to introduce a new technology
  called logical decoding; that is, the ability to get a replication
  stream that is based on changes to tuples rather than changes to
  blocks.  It could also be called logical replication.  In these
  patches, our existing replication is referred to as physical
  replication, which sounds kind of funny to me.  Anyone have another
  suggestion?
 
  Logical and Binary replication?
 
  Unfortunately changeset extraction output's can be binary data...
 
  I think physical and logical are fine and they seem to be well known
  terminology. Oracle uses those words and I have also seen many places
  use physical backup and logical backup, for example on Barman's
  homepage.

 +1

 I think it also fits well with the well-known terms physical [database]
 design and logical design.  Not that it is the same thing, but I
 believe that every database person, when faced with the distiction
 physical versus logical, will conclude that the former refers to
 data placement or block structure, while the latter refers to the
 semantics of the data being stored.

 Yours,
 Laurenz Albe

 --
 Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
 To make changes to your subscription:
 http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] alternative back-end block formats

[Christian Convey]

On Tue, Jan 28, 2014 at 5:42 AM, Cédric Villemain ced...@2ndquadrant.comwrote:
...

 As written in the meeting notes, Tom Lane revealed an interest in
 pluggable storage. So it might be interesting to check that.

 https://wiki.postgresql.org/wiki/PgCon_2013_Developer_Meeting


Thanks.  I just read those meeting notes, and also Josh Berkus' blog on the
topic:
http://www.databasesoup.com/2013/05/postgresql-new-development-priorities-2.html

I was only thinking to enable pluggable operations on a single, specified
heap page, probably as a function of which table owned the page.  Josh's
blog seems to describe something a little broader in scope, although I
can't tell from that post exactly what functionality that entails.

Either way, this sounds like something I'd enjoy pitching in on, to
whatever extent I could be useful.  Has anyone started work on this yet?

Re: [HACKERS] alternative back-end block formats

[Christian Convey]

 There are a couple of really huge issues that would have to be argued out
 before any progress could be made.


Is this something that people want to spend time on right now?  As I
mentioned earlier, I'm game.  But it doesn't sound like I'll get very far
without adult supervision.

Re: [HACKERS] alternative back-end block formats

[Christian Convey]

On Tue, Jan 28, 2014 at 12:39 PM, Tom Lane t...@sss.pgh.pa.us wrote:

 TBH, I'd rather we waited till the commitfest is over.  This is certainly
 material for 9.5, if not even further out, so there's no pressing need for
 a debate right now; and we have plenty of stuff we do need to deal with
 right now.


Works for me.  I'll just lurk in the meantime, and see what I can figure
out.  Thanks.

- Christian

Re: [HACKERS] alternative back-end block formats

[Christian Convey]

Hi Craig,

On Sun, Jan 26, 2014 at 5:47 AM, Craig Ringer cr...@2ndquadrant.com wrote:

 On 01/21/2014 07:43 PM, Christian Convey wrote:
  Hi all,
 
  I'm playing around with Postgres, and I thought it might be fun to
  experiment with alternative formats for relation blocks, to see if I can
  get smaller files and/or faster server performance.

 It's not clear how you'd do this without massively rewriting the guts of
 Pg.

 Per the docs on internal structure, Pg has a block header, then tuples
 within the blocks, each with a tuple header and list of Datum values for
 the tuple. Each Datum has a generic Datum header (handling varlena vs
 fixed length values etc) then a type-specific on-disk representation
 controlled by the type output function for that type.


I'm still in the process of getting familiar with the pg backend code, so I
don't have a concrete plan yet.  However, I'm working on the assumption
that some set of macros and functions encapsulates the page layout.

If/when I tackle this, I expect to add a layer of indirection somewhere
around that boundary, so that some non-catalog tables, whose schemas meet
certain simplifying assumptions, are read and modified using specialized
code.

I don't want to get into the specific optimizations I'd like to try, only
because I haven't fully studied the code yet, so I don't want to put my
foot in my mouth.

What concrete problem do you mean to tackle? What idea do you want to
 explore or implement?


My real motivation is that I'd like to get more familiar with the pg
backend codebase, and tilting at this windmill seemed like an interesting
way to accomplish that.

If I was focused on really solving a real-world problem, I'd say that this
lays the groundwork for table-schema-specific storage optimizations and
optimized record-filtering code.  But I'd only make that argument if I
planned to (a) perform a careful study with statistically significant
benchmarks, and/or (b) produce a merge-worthy patch.  At this point I have
no intentions of doing so.  My main goal really is just to have fun with
the code.


  Does anyone know if this has been done before with Postgres?  I would
  have assumed yes, but I'm not finding anything in Google about people
  having done this.

 AFAIK (and I don't know much in this area) the storage manager isn't
 very pluggable compared to the rest of Pg.


Thanks for the warning.  Duly noted.

Kind regards,
Christian

[HACKERS] alternative back-end block formats

[Christian Convey]

Hi all,

I'm playing around with Postgres, and I thought it might be fun to
experiment with alternative formats for relation blocks, to see if I can
get smaller files and/or faster server performance.

Does anyone know if this has been done before with Postgres?  I would have
assumed yes, but I'm not finding anything in Google about people having
done this.

Thanks,
Christian

[HACKERS] Question about Lockhart's book

[Christian Convey]

Hi guys,

I'm starting to poke around the internals of Postgres.  Does anyone know
the extent to which Thomas Lockhart's book, PostgresSQL Programmer's
Guide is accurate with respect to the current state of the code base?

Thanks,
Christian

Re: [HACKERS] Question about Lockhart's book

[Christian Convey]

Hi Andrew,

Thanks for your response.  Sometimes overall software architectures stay
(mostly) unchanged for a long time, and so I figured that's possibly the
case for Postgresql as well.  But I didn't know, which is why I asked.

Kind regards,
Christian


On Fri, Dec 27, 2013 at 11:11 AM, Andrew Dunstan and...@dunslane.netwrote:


 On 12/27/2013 10:55 AM, Christian Convey wrote:

 Hi guys,

 I'm starting to poke around the internals of Postgres.  Does anyone know
 the extent to which Thomas Lockhart's book, PostgresSQL Programmer's
 Guide is accurate with respect to the current state of the code base?



 Umm, that book was published in 2000, from what I can see on Amazon. Would
 you use a book published 13 years ago to educate yourself on, say, the
 Linux code base? 13 years is an eternity in this business.

 cheers

 andrew

Re: [HACKERS] Question about Lockhart's book

[Christian Convey]

Thanks very much Josh.  Those sound like great ideas - I'll try to give
them a shot.


On Fri, Dec 27, 2013 at 1:52 PM, Josh Berkus j...@agliodbs.com wrote:

 On 12/27/2013 08:14 AM, Christian Convey wrote:
  Hi Andrew,
 
  Thanks for your response.  Sometimes overall software architectures stay
  (mostly) unchanged for a long time, and so I figured that's possibly the
  case for Postgresql as well.  But I didn't know, which is why I asked.

 Some things in that book will still be accurate and informative.  The
 problem is that you, as a beginner, won't know which things are still
 good and which are obsolete.

 I'd suggest:

 - Developer documentation in our primary docs
 - Developer FAQ on the wiki
 - Bruce's presentations on various internals
 - Tom's presentations on how the query planner works
 - Various other people's presentations on other aspects, such as foreign
 data wrappers, event triggers, etc.

 Unfortunately, there's no central index of presentations.

 I'm a big fan of learn by doing, and here's a program which would
 bring you up on a LOT of PostgreSQL:

 1. Write a few of your own C functions, including trigger functions and
 an operator.

 2. Write your own foreign data wrapper for something.

 3. Write your own Type, including input/output functions, stats
 estimation and custom indexing.

 --
 Josh Berkus
 PostgreSQL Experts Inc.
 http://pgexperts.com


 --
 Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
 To make changes to your subscription:
 http://www.postgresql.org/mailpref/pgsql-hackers