Re: BUG #15182: Canceling authentication due to timeout aka Denial of Service Attack

[Michael Paquier]

On Tue, Jul 24, 2018 at 01:34:13AM -0400, Alvaro Herrera wrote:
> But I don't see why RangeVarCallbackOwnsTable isn't sufficient.

The set of relkinds checked by truncate_check_rel and
RangeVarCallbackOwnsTable is different (toast and matviews).  And in the
case of VACUUM, partitioned tables can call RangeVarGetRelidExtended
when one is listed as part of a manual VACUUM command.
--
Michael


signature.asc
Description: PGP signature

Re: BUG #15182: Canceling authentication due to timeout aka Denial of Service Attack

[Michael Paquier]

On Tue, Jul 24, 2018 at 02:23:02PM +0900, Michael Paquier wrote:
> Well, letting any users take an exclusive lock on system catalogs at
> will is not acceptable either, so two possible answers would be to fail
> or skip such relations.  The first concept applies if a relation list is
> given by the user, and the second if no list is given.

The first sentence is incorrect.  That's actually "letting any users
attempt to take an exclusive lock which makes others to be stuck as
well".
--
Michael


signature.asc
Description: PGP signature

Re: Usage of epoch in txid_current

[Thomas Munro]

On Tue, Jul 17, 2018 at 1:55 AM, Tom Lane  wrote:
> Andres Freund  writes:
>> On 2018-07-15 16:41:35 -0400, Tom Lane wrote:
>>> Andres Freund  writes:
 On 2018-07-09 19:56:25 -0400, Tom Lane wrote:
> Or, perhaps, use a struct in assert builds and int64 otherwise?
> You could hide the ensuing notational differences in macros.
>
>>> [ bunch of test results ]
>>> Offhand it would seem that we can get away with struct wrappers
>>> on any platform where performance is really of concern today.
>
>> Cool, thanks for checking!

+1

Here's a new version.  I did some cosmetic clean-up, and I dropped the
changes to pg_controldata output, replication epoch/xid processing
code and various similar non-essential changes.  For this patch I want
just the new type + next xid generator + appropriate conversions.

I propose that we get this committed early in the cycle.  That'd
maximise testing time in the tree, fix the bug identified by Amit, and
leave plenty of time for later patches to use FullTransactionId in
more places as appropriate.

Does anyone have specific kinds of validation or testing they'd like to see?

-- 
Thomas Munro
http://www.enterprisedb.com


0001-Track-the-next-xid-using-64-bits-v5.patch
Description: Binary data

Re: BUG #15182: Canceling authentication due to timeout aka Denial of Service Attack

[Michael Paquier]

On Mon, Jul 23, 2018 at 09:51:54PM -0700, Andres Freund wrote:
> On July 23, 2018 9:50:10 PM PDT, Michael Paquier  wrote:
>> Oh, yes, that would be bad.  My mind has slipped here.  I have seen
>> manual VACUUMs on system catalogs for applications using many temp
>> tables...  So we would want to have only VACUUM FULL being
>> conditionally
>> happening?  The question comes then about what to do when a VACUUM FULL
>> is run without a list of relations because expand_vacuum_rel() is not
>> actually the only problem.  Would we want to ignore system tables as
>> well except if allow_system_table_mods is on?  When no relation list is
>> specified, get_all_vacuum_rels() builds the list of relations which
>> causes vacuum_rel() to complain on try_relation_open(), so patching
>> just expand_vacuum_rel() solves only half of the problem for manual
>> VACUUMs.
> 
> I think any such restriction is entirely unacceptable. FULL or not.

Well, letting any users take an exclusive lock on system catalogs at
will is not acceptable either, so two possible answers would be to fail
or skip such relations.  The first concept applies if a relation list is
given by the user, and the second if no list is given.

Do you have any thoughts on the matter?
--
Michael


signature.asc
Description: PGP signature

FailedAssertion on partprune

[Jaime Casanova]

Hi,

I was trying sqlsmith on REL_11_STABLE (commit
1b957e59b92dc44c14708762f882d7910463a9ac) with a database i have at
hand, and got an assertion failure.
It seems to happen during planning on prunning time but only when
tables get bigger than certain size.

I configured it with "--enable-debug --enable-depend
--enable-cassert", attached is the backtrace and a script to reproduce
the problem.

--
Jaime Casanova  www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
set = {__val = {0, 94906079940272, 4613425733568182272, 0, 
4607182418800017408, 0, 0, 0, 0, 0, 4613425733568182272, 0, 0, 0, 
4613425733568182272, 0}}
pid = 
tid = 
#1  0x7f42bbbe642a in __GI_abort () at abort.c:89
save_stage = 2
act = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, 
sa_mask = {__val = {0 }}, sa_flags = 0, sa_restorer = 
0x7f42ae94cc38}
sigs = {__val = {32, 0 }}
#2  0x56510b3ca743 in ExceptionalCondition 
(conditionName=conditionName@entry=0x56510b570418 "!(relid_subplan_map[rti] == 
0)", 
errorType=errorType@entry=0x56510b419c9d "FailedAssertion", 
fileName=fileName@entry=0x56510b56ff67 "partprune.c", 
lineNumber=lineNumber@entry=244) at assert.c:54
No locals.
#3  0x56510b2298f5 in make_partition_pruneinfo 
(root=root@entry=0x56510becb828, partitioned_rels=0x7f42ae94cc38, 
subpaths=0x7f42ae94ca18, 
prunequal=prunequal@entry=0x7f42ae950360) at partprune.c:244
rti = 
targetpart = 0x0
pinfolist = 0x0
doruntimeprune = false
relid_subplan_map = 0x7f42ae950398
relid_subpart_map = 0x7f42ae950430
lc = 
i = 
#4  0x56510b1f4004 in create_append_plan (best_path=0x7f42ae94cb78, 
root=0x56510becb828) at createplan.c:1098
prunequal = 0x7f42ae950360
subplans = 
subpaths = 
rel = 0x56510be5e038
partpruneinfos = 0x0
tlist = 0x7f42ae94d000
#5  create_plan_recurse (root=root@entry=0x56510becb828, 
best_path=0x7f42ae94cb78, flags=flags@entry=1) at createplan.c:390
plan = 
__func__ = "create_plan_recurse"
#6  0x56510b1f3a3f in create_limit_plan (flags=1, best_path=0x7f42ae94ce88, 
root=0x56510becb828) at createplan.c:2406
plan = 
subplan = 
#7  create_plan_recurse (root=root@entry=0x56510becb828, 
best_path=0x7f42ae94ce88, flags=flags@entry=1) at createplan.c:487
plan = 
__func__ = "create_plan_recurse"
#8  0x56510b1f6580 in create_plan (root=root@entry=0x56510becb828, 
best_path=) at createplan.c:318
plan = 
#9  0x56510b204474 in standard_planner (parse=0x56510bb4c728, 
cursorOptions=256, boundParams=) at planner.c:412
result = 
glob = 0x56510bec8798
tuple_fraction = 0
root = 0x56510becb828
final_rel = 
best_path = 
top_plan = 
lp = 
lr = 
#10 0x56510b2bac5d in pg_plan_query (querytree=0x56510bb4c728, 
cursorOptions=256, boundParams=0x0) at postgres.c:809
plan = 
#11 0x56510b2bad3e in pg_plan_queries (querytrees=, 
cursorOptions=cursorOptions@entry=256, boundParams=boundParams@entry=0x0) at 
postgres.c:875
stmt = 
stmt_list = 0x0
query_list = 0x56510becb7c8
#12 0x56510b2bb229 in exec_simple_query (
query_string=0x56510bb254c8 "select  \n  (select pg_catalog.avg(actu_codi) 
from public.actualizar_sistema)\n as c0, \n  sample_0.radi_nume_temp as c1, 
\n  sample_0.radi_usua_ante as c2, \n  (select anex_borrado from public.anexos 
"...) at postgres.c:1050
portal = 
snapshot_set = true
commandTag = 0x56510b537dd6 "SELECT"
completionTag = 
"\000\000\000\000\002\000\000\000\t\000\000\000QV\000\000\000\000\000\000\000\000\000\000Q\000\000\000\000\000\000\000\310T\262\vQV\000\000
 \a\265\vQV", '\000' , "\331r\216\375\177\000"
querytree_list = 
plantree_list = 
receiver = 
format = 0
dest = DestRemote
parsetree_list = 0x56510bb4c5d8
parsetree_item = 0x56510bb4c5b0
save_log_statement_stats = false
was_logged = false
use_implicit_block = false
msec_str = 
"\000\000\000\000\002\000\000\000\t\000\000\000QV\000\000\000\000\000\000\000\000\000\000Q\000\000\000\000\000\000"
__func__ = "exec_simple_query"
#13 0x56510b2bceb5 in PostgresMain (argc=, 
argv=argv@entry=0x56510bb508c8, dbname=, username=) at postgres.c:4153
query_string = 0x56510bb254c8 "select  \n  (select 
pg_catalog.avg(actu_codi) from public.actualizar_sistema)\n as c0, \n  
sample_0.radi_nume_temp as c1, \n  sample_0.radi_usua_ante as c2, \n  (select 
anex_borrado from public.anexos "...
input_message = {
  data = 0x56510bb254c8 "select  \n  (select pg_catalog.avg(actu_codi) 
from public.actualizar_sistema)\n as c0, \n  

Re: BUG #15182: Canceling authentication due to timeout aka Denial of Service Attack

[Andres Freund]

On July 23, 2018 9:50:10 PM PDT, Michael Paquier  wrote:
>On Mon, Jul 23, 2018 at 09:17:53PM -0700, Andres Freund wrote:
>> I might be mis-parsing this due to typos. Are you actually suggesting
>> vacuum on system tables should depend on that GUC? If so, why? That's
>> seems like a terrible idea.  It's pretty normal to occasionally have
>> to vacuum them?
>
>Oh, yes, that would be bad.  My mind has slipped here.  I have seen
>manual VACUUMs on system catalogs for applications using many temp
>tables...  So we would want to have only VACUUM FULL being
>conditionally
>happening?  The question comes then about what to do when a VACUUM FULL
>is run without a list of relations because expand_vacuum_rel() is not
>actually the only problem.  Would we want to ignore system tables as
>well except if allow_system_table_mods is on?  When no relation list is
>specified, get_all_vacuum_rels() builds the list of relations which
>causes vacuum_rel() to complain on try_relation_open(), so patching
>just expand_vacuum_rel() solves only half of the problem for manual
>VACUUMs.

I think any such restriction is entirely unacceptable. FULL or not.

Andres
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: BUG #15182: Canceling authentication due to timeout aka Denial of Service Attack

[Michael Paquier]

On Mon, Jul 23, 2018 at 09:17:53PM -0700, Andres Freund wrote:
> I might be mis-parsing this due to typos. Are you actually suggesting
> vacuum on system tables should depend on that GUC? If so, why? That's
> seems like a terrible idea.  It's pretty normal to occasionally have
> to vacuum them?

Oh, yes, that would be bad.  My mind has slipped here.  I have seen
manual VACUUMs on system catalogs for applications using many temp
tables...  So we would want to have only VACUUM FULL being conditionally
happening?  The question comes then about what to do when a VACUUM FULL
is run without a list of relations because expand_vacuum_rel() is not
actually the only problem.  Would we want to ignore system tables as
well except if allow_system_table_mods is on?  When no relation list is
specified, get_all_vacuum_rels() builds the list of relations which
causes vacuum_rel() to complain on try_relation_open(), so patching
just expand_vacuum_rel() solves only half of the problem for manual
VACUUMs.
--
Michael


signature.asc
Description: PGP signature

Re: BUG #15182: Canceling authentication due to timeout aka Denial of Service Attack

[Andres Freund]

On July 23, 2018 9:14:03 PM PDT, Michael Paquier  wrote:
>- While it would make sense, at least to me, to make VACUUM fall into
>if
>allow_system_table_mods is allowed, 

I might be mis-parsing this due to typos. Are you actually suggesting vacuum on 
system tables should depend on that GUC? If so, why? That's seems like a 
terrible idea. It's pretty normal to occasionally have to vacuum them?


Andres
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: BUG #15182: Canceling authentication due to timeout aka Denial of Service Attack

[Michael Paquier]

On Mon, Jul 23, 2018 at 11:29:33AM -0400, Robert Haas wrote:
> ExecuteTruncate needs to be refactored to use RangeVarGetRelidExtended
> with a non-NULL callback rather than heap_openrv, and
> expand_vacuum_rel needs to use RangeVarGetRelidExtended with a
> callback instead of RangeVarGetRelid.  See
> cbe24a6dd8fb224b9585f25b882d5ffdb55a0ba5 as an example of what to do.
> I fixed a large number of cases of this problem back around that time,
> but then ran out of steam and had to move onto other things before I
> got them all.  Patches welcome.

Thanks for pointing those out, I looked at both code paths recently for
some other work...  The amount of work does not consist just in using
for example RangeVarCallbackOwnsRelation for VACUUM and TRUNCATE.  There
are a couple of reasons behind that:
- While it would make sense, at least to me, to make VACUUM fall into if
allow_system_table_mods is allowed, that's not the case of ANALYZE as I
think that we should be able to call ANALYZE on a system catalog as
well.  So we would basically a new flavor of
RangeVarCallbackOwnsRelation for VACUUM which makes this difference
between vacuum and analyze with an argument in the callback, the options
of VacuumStmt would be nice.  This would not be used by autovacuum
anyway, but adding an assertion and mentioning that in the comments
would not hurt.  There is an argument for just restricting VACUUM FULL
as well and not plain VACUUM, as that's the one hurting here.
- TRUNCATE is closer to a solution, as it has its own flavor of relation
checks with truncate_check_rel.  So the callback would replace
truncate_check_rel but CheckTableNotInUse should be moved out of it.
TRUNCATE already uses allow_system_table_mods for its checks.

Thoughts?
--
Michael


signature.asc
Description: PGP signature

Re: Have an encrypted pgpass file

[Thomas Munro]

On Tue, Jul 24, 2018 at 2:10 PM, Craig Ringer  wrote:
>> Grabbing it from a process's memory is a bit harder than grabbing contents
>> of a file, but not much harder. If the agent is remote then that's harder,
>> but you can just ask the script to decrypt the pgpass for you, so again, not
>> much of a win.
>>
>> Even with a hardware crypto offload device the advantage here seems to be
>> mainly limited to making it harder to capture data from backups or
>> file-lifting attacks. Anything that can execute code or commands on the host
>> can still get the credentials.
>
> To be clear I'm not saying not to do this. I think it'd make more sense to
> do it via an agent and socket, where libpq learns to ask the agent for
> credentials (and certs!). That way we could finally support libnss, etc.
> It's not perfect, and it doesn't magically make unattended storage secure,
> but it sure helps slow down file-based password theft.

Yeah, the idea that you can defend yourself against root is obviously
not a good one.  But the Apple keychain example (see the commands I
showed earlier) does seem to protect you against some threat
scenarios: if your computer is stolen, that password can't be
extracted from the disk.  You'd need to unlock the keychain first by
logging in.  It's not 'unattended': you have to be in attendance to
unlock the keychain.

I'm deeply suspicious of unattended use cases for encrypted passwords
though.  I have heard requests for this.  Encrypting it with a key
that is present in the configuration file, derived from well known
things, or stored in a nearby file doesn't change that.  The encrypted
secret + easily available key is still a secret you must protect
exactly as hard.  I suspect it is a misapplication of the advice that
you should never store (other people's) passwords in your database --
instead you should store something that allows you to check if *they*
have the password, without storing the password itself.  But that
doesn't mean that you should throw away your own passwords: instead
you should recognise that they are secrets, and treat them as such.

An idea I wondered about: if the goal is to allow psql (not libpq) to
use a secret from ,
then perhaps psql could have a machine-friendly
--read-password-from-stdin feature for use by wrapper scripts (it
seems to be hard to send passwords to -W, though maybe I'm just doing
something stupid).  Or if you could wrap it in a script that provides
one end of a named pipe as PGPASSFILE (is that 'on disk'?).  Or uses a
temporary file in tmpfs with swap not configured (is that 'on disk'
yet?  Yeah, I bet that's against the rules...)  Of course you could
write a wrapper script that sets PGPASSWORD, but some people don't
like putting secrets in environment variables.  Supposedly there are
operating systems where anyone can see your environment (which?), but
on the systems I know only root can.  Then you run into the thorny
question of why you don't trust root not to peer at your
/proc/{$PID}/environ (or local equivalent), but do trust them not to
core dump your whole process and kernel.  (It is interesting that
Linux pam_exec.so chooses to pass the username via env var PAM_USER
but send the password via a pipe connected to stdin, considering all
the ways root could intercept that.)

-- 
Thomas Munro
http://www.enterprisedb.com

Re: [report] memory leaks in COPY FROM on partitioned table

[Kohei KaiGai]

Further investigation I did:

CopyFrom() calls ExecFindPartition() to identify the destination child
table of partitioned table.
Then, it internally calls get_partition_for_tuple() to get partition
index according to the key value.
This invocation is not under the per-tuple context.

In case of hash-partitioning, get_partition_for_tuple() calls
hash-function of key data type; which is hash_numeric in my case.
The hash_numeric fetches the key value using PG_GETARG_NUMERIC(0). It
internally calls pg_detoast_datum() which may allocate new memory if
varlena datum is not uncompressed long (32bit) format.

Once this patch attached, PostgreSQL backend process has been working
with about 130MB memory consumption for 20min right now (not finished
yet...)
Before the patch applied, its memory consumption grows up about
10BM/sec, then terminated a few hours later.

P.S,
I think do_convert_tuple() in ExecFindPartition() and
ConvertPartitionTupleSlot() may also allocate memory out of the
per-tuple context, however, I could not confirmed yet, because my test
case has TupleConversionMap == NULL.

Thanks,

2018-07-24 10:43 GMT+09:00 Michael Paquier :
> On Tue, Jul 24, 2018 at 09:41:52AM +0900, Kohei KaiGai wrote:
>> In PG11beta2, my backend process gets terminated during COPY FROM of
>> large text file (544GB) on partitioned table.
>> The kernel log says OOM Killer send SIGKILL due to memory pressure.
>> In fact, 63GB of system physical 64GB was consumed by the PostgreSQL
>> backend just before the termination.
>
> Hmm..  That's not nice.  Let's add an open item.
> --
> Michael



-- 
HeteroDB, Inc / The PG-Strom Project
KaiGai Kohei 


fix-leak-copyfrom.patch
Description: Binary data

Re: Add SKIP LOCKED to VACUUM and ANALYZE

[Michael Paquier]

On Mon, Jul 23, 2018 at 02:27:48PM +, Bossart, Nathan wrote:
> Sorry for the delay on these patches!  This is nearly identical to
> what I started writing last night, so it looks good to me.

Thanks for double-checking.  I have pushed this one to move on with the
rest of the feature.

> +typedef enum ClusterOption
> +{
> + CLUOPT_RECHECK, /* recheck relation state */
> + CLUOPT_VERBOSE  /* print progress info */
> +}ClusterOption;
> 
> It looks like the last line here has a bit of extra whitespace
> compared to the other code in parsenodes.h.

That came from pgindent.  I have just updated typedefs.list to take care
of it.  There could be an argument about moving recheck out of
cluster_rel(), but that would not be nice with any module for example
calling this API, so its contract is unchanged.
--
Michael


signature.asc
Description: PGP signature

Re: Have an encrypted pgpass file

[Craig Ringer]

On 24 July 2018 at 05:53, Jeff Janes  wrote:

> On Wed, Jul 18, 2018 at 5:52 PM, Tom Lane  wrote:
>
>> Thomas Munro  writes:
>> > On Thu, Jul 19, 2018 at 5:46 AM, Marco van Eck 
>> wrote:
>> >> Since .pgpass files contain plain-text passwords, I searched for an
>> >> alternative.
>> >> In the attached patch I've added the possibility to run a command to
>> produce
>> >> the content of the pgpass file, in exactly the same format.
>>
>> > ... Here you side step those questions completely and make that the end
>> > user's problem.   I like it.
>>
>> ... but doesn't this just encourage people to build hacks that aren't
>> really any more secure than the unreadable-file approach?  In fact,
>> I'm afraid this would be an attractive nuisance, in that people would
>> build one-off hacks that get no security vetting and don't really work.
>>
>> I'd like to see a concrete example of a use-case that really does add
>> security; preferably one short and useful enough to put into the docs
>> so that people might copy-and-paste it rather than rolling their own.
>> It seems possible that something of the sort could be built atop
>> ssh-agent or gpg-agent, for instance.
>>
>
> If the goal is not unattended operation but just unannoying operation, I
> think the first example he provided is already that use-case.  If you
> already have gpg configured to use gpg-agent, then it just works.  You get
> encryption-at-rest, and you don't have to type in your password repeatedly
> in the same continuous shell session.
>

... and the attacker steals the key from gpg-agent.

Grabbing it from a process's memory is a bit harder than grabbing contents
of a file, but not much harder. If the agent is remote then that's harder,
but you can just ask the script to decrypt the pgpass for you, so again,
not much of a win.

Even with a hardware crypto offload device the advantage here seems to be
mainly limited to making it harder to capture data from backups or
file-lifting attacks. Anything that can execute code or commands on the
host can still get the credentials.

-- 
 Craig Ringer   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Missing pg_control crashes postmaster

[Craig Ringer]

On 24 July 2018 at 03:31, Brian Faherty 
wrote:

> Hey Hackers,
>
> If a postmaster is running and the pg_control file is removed postgres
> will PANIC.


How does that happen?

"Don't go deleting stuff in pgdata" is pretty fundamental.

Re: Missing pg_control crashes postmaster

[Michael Paquier]

On Mon, Jul 23, 2018 at 07:00:30PM -0400, Tom Lane wrote:
> I would vote to reject any such patch; it's too likely to cause more
> problems than it solves.  Generally, if critical files like that one
> have disappeared, trying to write new data isn't going to be enough
> to fix it and could well result in more corruption.

+1.
--
Michael


signature.asc
Description: PGP signature

Re: pgbench: improve --help and --version parsing

[Michael Paquier]

On Mon, Jul 23, 2018 at 07:47:44AM -0400, Fabien COELHO wrote:
>> I don't think that it is a bad idea to improve things the way you are
> 
> For the record, this is not my patch, I'm merely reviewing it.

Of course, any input is welcome.  It is nice to see that you took some
time to look at the patch.

>> doing, however you should extend program_version_ok() and
>> program_help_ok() in src/test/perl/TestLib.pm so as short options are
>> tested for two reasons:
> 
> Interesting, I did not notice these functions before. I fully agree that
> manual testing is a pain for such a simple change.
>
> Do you mean something like the attached?

You basically have the idea, except that the number of tests in any TAP
files calling program_help_ok and program_version_ok needs to be
updated, and that the test is too verbose :)

>  # Version
> -pgbench('-V', 0, [qr{^pgbench .PostgreSQL. }], [qr{^$}], 'pgbench version');
> +pgbench('-V', 0, [qr{^pgbench \(PostgreSQL\) \d+}], [qr{^$}], 'pgbench 
> version');

This could go away.

> + is($stdout, $stdout2, "$cmd --help and -? have same output");
> + like($stdout, qr{Usage:}, "$cmd --help is about usage");
> + like($stdout, qr{\b$cmd\b}, "$cmd --help is about $cmd");
> + like($stdout, qr{--help}, "$cmd --help is about option --help");
> + like($stdout, qr{-\?}, "$cmd --help is about options -?");
> + like($stdout, qr{--version}, "$cmd --help is about options --version");
> + like($stdout, qr{-V}, "$cmd --help is about options -V");

I would keep things a bit more simple by not parsing the output as you
do, and it would be possible to reduce that to one single test with a
text block as well, say using qq().

Andrei, could you update your patch in this area please?  That will ease
reviews and checks.  Double-checking that the documentation gets the
correct call would not hurt as well.
--
Michael


signature.asc
Description: PGP signature

Re: [report] memory leaks in COPY FROM on partitioned table

[Michael Paquier]

On Tue, Jul 24, 2018 at 09:41:52AM +0900, Kohei KaiGai wrote:
> In PG11beta2, my backend process gets terminated during COPY FROM of
> large text file (544GB) on partitioned table.
> The kernel log says OOM Killer send SIGKILL due to memory pressure.
> In fact, 63GB of system physical 64GB was consumed by the PostgreSQL
> backend just before the termination.

Hmm..  That's not nice.  Let's add an open item.
--
Michael


signature.asc
Description: PGP signature

Re: Fix calculations on WAL recycling.

[Michael Paquier]

On Mon, Jul 23, 2018 at 07:55:57PM +0900, Kyotaro HORIGUCHI wrote:
> At Mon, 23 Jul 2018 15:59:16 +0900, Michael Paquier  
> wrote in <20180723065916.gi2...@paquier.xyz>
>> This is an open item for v11.
> 
> Mmm. Thanks. I wrongly thought this was v10 item. Removed this
> from the next CF.

Thanks for updating the CF app.

>> By doing so, you would notice that the oldest WAL segment does not get
>> recycled after the checkpoint, while it should as the redo pointer is
>> always checkpoint generated.  What happens is that this oldest segment
>> gets recycled every two checkpoints.
> 
> (I'm not sure I'm getting it correctly..) I see the old segments
> recycled. When I see pg_switch_wal() returns 0/12..,
> pg_walfile_name is ...13 and segment files for 13 and 14 are
> found in pg_wal directory. That is, seg 12 was recycled as seg
> 14. log_checkpoint=on shows every checkpoint recycles 1 segment
> in the case.

With your patch applied I see one segment recycled after each
checkpoint, which is correct.  On HEAD, you would see no segments,
followed by 2 segments recycled.  But I also see sometimes one segment
recycled.

>> The calculation of _logSegNo in CreateRestartPoint is visibly incorrect,
>> this still uses PriorRedoPtr so the bug is not fixed for standbys.  The
>> tweaks for ThisTimeLineID also need to be out of the loop where
>> PriorRedoPtr is InvalidXLogRecPtr, and only the distance calculation
>> should be kept.
> 
> Agreed.  While I reconsidered this, I noticed that the estimated
> checkpoint distance is 0 when PriorRedoPtr is invalid. This means
> that the first checkpoint/restartpoint tries to reduce WAL
> segments to min_wal_size. However, it happens only at initdb time
> and makes no substantial behavior change so I made the change
> ignoring the difference.

Yes, same analysis here.

>> 1) and 2) are in my opinion clearly oversights from 4b0d28d, but 3) is
>> not.
> 
> Thank you for the comments and suggestions. After all, I did the
> following things in the attached patch.
> 
> - Reverted the change on timeline switching. (Removed the (3))
> - Fixed CreateRestartPoint to do the same thing with CreateCheckPoint.
> - Both CreateRestart/CheckPoint now tries trimming of WAL
>   segments even for the first time.

Thanks, pushed after some comment tweaks and fixing the variable names
at the top of xlog.c for the static declarations.  Perhaps we can do
more refactoring here by moving all the segment calculation logic
directly in RemoveOldXlogFiles, but that makes the end LSN calculation a
bit blurry so I kept things as you proposed in version 3 of the patch.
--
Michael


signature.asc
Description: PGP signature

[report] memory leaks in COPY FROM on partitioned table

[Kohei KaiGai]

Hello,

In PG11beta2, my backend process gets terminated during COPY FROM of
large text file (544GB) on partitioned table.
The kernel log says OOM Killer send SIGKILL due to memory pressure.
In fact, 63GB of system physical 64GB was consumed by the PostgreSQL
backend just before the termination.

OOM Killer says:
[69267.687791] Out of memory: Kill process 23881 (postgres) score 977
or sacrifice child
[69267.687860] Killed process 23881 (postgres) total-vm:105210868kB,
anon-rss:63372320kB, file-rss:0kB, shmem-rss:126144kB

Configurations are below:
The 'lineorder' partition table has three child tables by
hash-partitioning on lo_orderkey (numeric).
Each child table has its own tablespace. 'lineorder_p0' is built on
the tablespace 'nvme0' on behalf of independent SSD device for
instance.
The query I run is:
copy lineorder from '/opt/nvme0/ssbm/lineorder.tbl' delimiter '|';

So, its contents shall be distributed to individual child tables,
based on in-database evaluation of hash-keys.

To track which memory-context consumes too much memory more than usual
expectations, I put elog() to report dying message prior to OOM
Killer.
See "aset-track.patch". It raises a warning message when memory
consumption per memory-context goes across watermark.

It says 'PortalContext' consumed 25GB at 04:26, then it grows up to
34GB at 05:21, and terminated at 05:51.
It looks to me somewhere allocates memory our of per-tuple memory
context, but I'm still under the investigation.

Any ideas?

2018-07-25 04:26:54.096 JST [23881] WARNING:  memory context
'PortalContext' grows up 25769803784 bytes
2018-07-25 04:26:54.096 JST [23881] CONTEXT:  COPY lineorder, line
1610626836: 
"1610614887|1|18487099|541334|1474684|19980523|3-MEDIUM|0|30|4125930|30528526|2|4043411|82518|8|19980..."
WARNING:  memory context 'PortalContext' grows up 25769803784 bytes
2018-07-25 04:27:07.202 JST [23865] LOG:  checkpoints are occurring
too frequently (25 seconds apart)
:
  
:
2018-07-25 05:21:22.423 JST [23881] WARNING:  memory context
'PortalContext' grows up 34359738384 bytes
2018-07-25 05:21:22.423 JST [23881] CONTEXT:  COPY lineorder, line
2147497762: 
"2147498439|7|910553|962168|773580|19971006|1-URGENT|0|46|5658552|38894795|1|5601966|73807|2|19971201..."
:
  
:
2018-07-25 05:51:07.264 JST [23837] LOG:  server process (PID 23881)
was terminated by signal 9: Killed
2018-07-25 05:51:07.264 JST [23837] DETAIL:  Failed process was
running: copy lineorder from '/opt/nvme0/ssbm/lineorder.tbl' delimiter
'|';


-- 
HeteroDB, Inc / The PG-Strom Project
KaiGai Kohei 


aset-track.patch
Description: Binary data


OOMKiller.log
Description: Binary data

Re: "interesting" issue with restore from a pg_dump with a database-wide search_path

[Tom Lane]

"Regina Obe"  writes:
>> You don't really need any new syntax for this particular case, I think.
>> You can declare the function in the extension like this:
>> create function ... set search_path from current;

> But then the search_path would be local variable to the function.  Wouldn't
> that impact performance?

Yeah, but it would *work*.  Never put performance before functionality.

> We had originally tried that in PostGIS functions (well not that but
> explicitly setting the functions local search path to where postgis is
> installed by adding a search_path variable to the function)
> And things got 10 times slower.

I can imagine that you'd take a noticeable hit for SQL functions that'd
otherwise be inline-able, but I doubt that it makes much difference for
index functions.

regards, tom lane

RE: "interesting" issue with restore from a pg_dump with a database-wide search_path

[Regina Obe]

> 
> > a) In this particular case, I have a function that uses fuzzystrmatch
and is
> used in functional indexes.
> > I unfortunately can't schema qualify the use of soundex, because I
> > don't know where the user may have installed fuzzystrmatch is
> > installed
> > b) Stephen Frost had suggested, perhaps we should have some syntax like
> @extension_loc(fuzzystrmatch)...@ so that one could reference an extension
> dependency location within a function without knowing where it is
installed.
> 
> You don't really need any new syntax for this particular case, I think.
> You can declare the function in the extension like this:
> 
> create function ... set search_path from current;
> 
> which will cause it to absorb the search path that's set while running the
> extension script, which should be what you want.
> 
>   regards, tom lane

But then the search_path would be local variable to the function.  Wouldn't
that impact performance?

We had originally tried that in PostGIS functions (well not that but
explicitly setting the functions local search path to where postgis is
installed by adding a search_path variable to the function)
And things got 10 times slower.

Re: "interesting" issue with restore from a pg_dump with a database-wide search_path

[Tom Lane]

"Regina Obe"  writes:
> Sorry for not posting from the thread.  Paul alerted me to this one and I am 
> aware of the issue.

> 1) I do have fuzzstrmatch listed as a dependency in the control file.  I know 
> because I often install the geocoder with
> CREATE EXTENSION postgis_tiger_geocoder CASCADE;
> And it installs postgis and fuzzystrmatch

OK.

> a) In this particular case, I have a function that uses fuzzystrmatch and is 
> used in functional indexes.
> I unfortunately can't schema qualify the use of soundex, because I don't know 
> where the user may have installed fuzzystrmatch is installed
> b) Stephen Frost had suggested, perhaps we should have some syntax like 
> @extension_loc(fuzzystrmatch)...@ so that one could reference an extension 
> dependency location within a function without knowing where it is installed.

You don't really need any new syntax for this particular case, I think.
You can declare the function in the extension like this:

create function ... set search_path from current;

which will cause it to absorb the search path that's set while running
the extension script, which should be what you want.

regards, tom lane

RE: GSOC 2018 Project - A New Sorting Routine

[Kefan Yang]

Hi Tomas!

I did a few tests on my own Linux machine, but the problem is that my resources 
on AWS(CPU, RAM and even Disk space) are very limited. I considered 
establishing virtual machine on my own PC but the performance is even worse.

My original patch has two main optimizations: (1) switch to heap sort when 
depth limit exceeded (2) check whether the array is presorted only once at the 
beginning. Now I want to test these optimizations separately. On AWS EC2 
instance, regressions on CREATE INDEX cases seems to be less significant if we 
use (1) only, but I can only test up to 10 records and 512MB memory using 
your scripts.

So would you mind re-running the tests using the two patches I provided in the 
attachment? That will be very helpful

Regards,
Kefan

From: Tomas Vondra
Sent: July 18, 2018 2:26 PM
To: Kefan Yang
Cc: Andrey Borodin; Peter Geoghegan; PostgreSQL Hackers
Subject: Re: GSOC 2018 Project - A New Sorting Routine

I don't have any script for that - load the files into a spreadsheet,
create pivot tables and you're done.

regards

On 07/18/2018 11:13 PM, Kefan Yang wrote:
> Hey Tomas!
> 
>  
> 
> I am trying to reproduce the results on my machine. Could you please
> share the script to generate .ods files?
> 
>  
> 
> Regards,
> 
> Kefan
> 
>  
> 
> *From: *Tomas Vondra 
> *Sent: *July 18, 2018 2:05 AM
> *To: *Andrey Borodin 
> *Cc: *Peter Geoghegan ; Kefan Yang
> ; PostgreSQL Hackers
> 
> *Subject: *Re: GSOC 2018 Project - A New Sorting Routine
> 
>  
> 
>  
> 
>  
> 
> On 07/18/2018 07:06 AM, Andrey Borodin wrote:
> 
>> Hi, Tomas!
> 
>>
> 
>>> 15 июля 2018 г., в 1:20, Tomas Vondra  
>>> > написал(а):
> 
>>> 
> 
>>> So I doubt it's this, but I've tweaked the scripts to also set this GUC
> 
>>> and restarted the tests on both machines. Let's see what that does.
> 
>>
> 
>> Do you observe any different results?
> 
>>
> 
>  
> 
> It did change the CREATE INDEX results, depending on the scale. The full
> 
> data is available at [1] and [2], attached is a spreadsheet summary from
> 
> the Xeon box.
> 
>  
> 
> For the largest scale (1M rows) the regressions for CREATE INDEX queries
> 
> mostly disappeared. For 10k rows it still affects CREATE INDEX with a
> 
> text column, and the 100k case behaves just like before (so significant
> 
> regressions for CREATE INDEX).
> 
>  
> 
> I don't have time to investigate this further at the moment, but I'm
> 
> still of the opinion that there's little to gain by replacing our
> 
> current sort algorithm with this.
> 
>  
> 
>  
> 
> [1] https://bitbucket.org/tvondra/sort-intro-sort-xeon/src/master/
> 
> [2] https://bitbucket.org/tvondra/sort-intro-sort-i5/src/master/
> 
>  
> 
> regards
> 
>  
> 
> -- 
> 
> Tomas Vondra  http://www.2ndQuadrant.com
> 
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
> 
>  
> 

-- 
Tomas Vondra  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



check_once.diff
Description: Binary data


use_heap.diff
Description: Binary data

Re: Have an encrypted pgpass file

[Jeff Janes]

On Wed, Jul 18, 2018 at 5:52 PM, Tom Lane  wrote:

> Thomas Munro  writes:
> > On Thu, Jul 19, 2018 at 5:46 AM, Marco van Eck 
> wrote:
> >> Since .pgpass files contain plain-text passwords, I searched for an
> >> alternative.
> >> In the attached patch I've added the possibility to run a command to
> produce
> >> the content of the pgpass file, in exactly the same format.
>
> > ... Here you side step those questions completely and make that the end
> > user's problem.   I like it.
>
> ... but doesn't this just encourage people to build hacks that aren't
> really any more secure than the unreadable-file approach?  In fact,
> I'm afraid this would be an attractive nuisance, in that people would
> build one-off hacks that get no security vetting and don't really work.
>
> I'd like to see a concrete example of a use-case that really does add
> security; preferably one short and useful enough to put into the docs
> so that people might copy-and-paste it rather than rolling their own.
> It seems possible that something of the sort could be built atop
> ssh-agent or gpg-agent, for instance.
>

If the goal is not unattended operation but just unannoying operation, I
think the first example he provided is already that use-case.  If you
already have gpg configured to use gpg-agent, then it just works.  You get
encryption-at-rest, and you don't have to type in your password repeatedly
in the same continuous shell session.

Cheers,

Jeff

Re: Stored procedures and out parameters

[Peter Eisentraut]

On 23.07.18 19:38, Andrew Gierth wrote:
>> "Robert" == Robert Haas  writes:
> 
>  >> However, connecting via Npgsql, which uses the extended protocol, I
>  >> see something quite different. As a response to a Describe
>  >> PostgreSQL message, I get back a NoData response rather than a
>  >> RowDescription message, In other words, it would seem that the
>  >> behavior of stored procedures differs between the simple and
>  >> extended protocols, when INOUT parameters are involved.
> 
>  Robert> I might be wrong, but that sounds like a bug.
> 
> Completely off the cuff, I'd expect 59a85323d might have fixed that;
> does it fail on the latest 11-stable?

Yes, that's supposed to address that.

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: BUG #15182: Canceling authentication due to timeout aka Denial of Service Attack

[Jeff Janes]

On Fri, Jul 20, 2018 at 5:56 PM, Marko Tiikkaja  wrote:

> On Fri, Jul 20, 2018 at 2:17 AM, Jeremy Schneider 
> wrote:
>
>> I'd like to bump this old bug that Lloyd filed for more discussion. It
>> seems serious enough to me that we should at least talk about it.
>>
>> Anyone with simply the login privilege and the ability to run SQL can
>> instantly block all new incoming connections to a DB including new
>> superuser connections.
>>
>
> So..  don't VACUUM FULL pg_authid without lock_timeout?
>

That's like saying the solution to a security hole is for no one to attempt
to exploit it.

Note that you do not need to have permissions to do the vacuum full.  This
works merely from the attempt to do so, before the permissions are checked.

Cheers,

Jeff

Re: Missing pg_control crashes postmaster

[Andres Freund]

On July 23, 2018 12:31:13 PM PDT, Brian Faherty  
wrote:
>Hey Hackers,
>
>If a postmaster is running and the pg_control file is removed postgres
>will PANIC.
>
>Steps to recreate:
>
>1.) start a new cluster
>2.) rm $DATADIR/pg_control
>3.) psql => CHECKPOINT;
>
>PANIC:  could not open control file "global/pg_control": No such file
>or directory
>
>After the PANIC there is no pg_control. Recovery would be difficult
>without a replica or a backup. Instead of crashing we can just write a
>new pg_control file since all the data is in memory at the time.
>
>There does not really seem to be a need for this behavior as all the
>information postgres needs is in memory at this point. I propose with
>a patch to just recreate pg_control on updates if it does not exist.

What's the issue this would solve? Given that there's moments, until the 
control file is rewritten, where you would be toast either way, I don't buy 
this gives much added safety. Nor have you explained which realistic scenarios 
lead to the file missing, without much broader problems being present.


Andres 
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Missing pg_control crashes postmaster

[David G. Johnston]

On Mon, Jul 23, 2018 at 12:31 PM, Brian Faherty <
anothergenericu...@gmail.com> wrote:

> There does not really seem to be a need for this behavior as all the
> information postgres needs is in memory at this point. I propose with
> a patch to just recreate pg_control on updates if it does not exist.


Or at minimum create said file with a different name in PGDATA so an admin
can rename it should they wish to accept the in memory version as being a
valid replacement for whatever ended up happening to the original.

Even if it can be safely rebuilt having pg_control removed out from under a
running server seems like something that shouldn't happen and the server is
in its rights to panic if it does.

David J.
​

Missing pg_control crashes postmaster

[Brian Faherty]

Hey Hackers,

If a postmaster is running and the pg_control file is removed postgres
will PANIC.

Steps to recreate:

1.) start a new cluster
2.) rm $DATADIR/pg_control
3.) psql => CHECKPOINT;

PANIC:  could not open control file "global/pg_control": No such file
or directory

After the PANIC there is no pg_control. Recovery would be difficult
without a replica or a backup. Instead of crashing we can just write a
new pg_control file since all the data is in memory at the time.

There does not really seem to be a need for this behavior as all the
information postgres needs is in memory at this point. I propose with
a patch to just recreate pg_control on updates if it does not exist.

-- 
Brian Faherty
From 80b84792d3c54a90e1e221c818a7ef025fe30b61 Mon Sep 17 00:00:00 2001
From: Brian Faherty 
Date: Mon, 23 Jul 2018 15:08:57 -0400
Subject: [PATCH] Create pg_control on update if not exists

Removing the global/pg_control file on a running postgres cluster will
no longer crash the postmaster during a checkpoint.
---
 src/backend/access/transam/xlog.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 335b4a573d..4721ea0168 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -4758,7 +4758,7 @@ UpdateControlFile(void)
 	FIN_CRC32C(ControlFile->crc);
 
 	fd = BasicOpenFile(XLOG_CONTROL_FILE,
-	   O_RDWR | PG_BINARY);
+	   O_RDWR | O_CREAT | PG_BINARY);
 	if (fd < 0)
 		ereport(PANIC,
 (errcode_for_file_access(),
-- 
2.17.1

Re: How can we submit code patches that implement our (pending) patents?

[Nico Williams]

On Mon, Jul 23, 2018 at 12:12:03PM -0700, Joshua D. Drake wrote:
> On 07/23/2018 12:06 PM, Bruce Momjian wrote:
> >>So, is it FUD?  The core needs paid-for legal advice, not speculation.
> >>
> >>I'm quite certain that a software license can make a patent grant to the
> >>satisfaction of many open source communities, and almost certainly to
> >>the satisfaction of the PG community.  But it will take an IP lawyer to
> >>review or write such a license.
> >And is the payback worth it?  Many don't think so.
> 
> Although Nico is correct, I also think we need to consider what the
> community wants here. Historically, we have always explicitly avoided
> anything to do with patents to the point where some hackers won't even read
> white papers on patented methods. I do think there is a definite
> technological advantage for PostgreSQL if there was a license that core
> could accept that was patent friendly but frankly, I don't think that core
> or the community has the desire to work through the cost of doing so.

Absolutely.

I myself don't want to be tainted when reading PG source code (or even
docs).

I believe PG needs to demand at least royalty-free, world-wide,
non-exclusive, transferrable license, either irrevocable (i.e., without
further conditions) or with a no-lawsuit covenant (i.e., revocable when
the patent owner gets sued by the patent user).  Such terms would be
acceptable to me as a third-party contributor.  Other contributors might
have a different take.  An IP lawyer could tighten that up and turn it
into legalese.

There might exist a license with suitable patent grant clauses that PG
could adopt for new contributions.  (Not GPLv3, of course, since that
one is viral, but still, it has patent grant language that could be
adapted for PG.)

I myself would not welcome patent grants that apply only to PG itself
and not to forks of it.  I would also not welcome grants that apply only
to PG and forks of it, but not new non-PG implementations of
the same patent (whether open source or not) -- i.e., don't taint me.

I suspect most other contributors too would not want to be tainted, so I
think that's probably the most critical requirement, and that does drive
towards the minimum grant breadth described above.

Some contributors may also oppose no-lawsuit covenants, but I believe
PG will benefit much more from allowing them than from disallowing them.

At any rate, PG absolutely should accept code involving patents that are
placed in the public domain provided that the owner places all relevant
patents (in the U.S., in the EU, and so on -- wherever they have them)
in the public domain.  Yes, I'm aware that Germany does not recognize
the idea of "public domain".  This too requires legal advice, which my
advice isn't.

Nico
-- 

RE: "interesting" issue with restore from a pg_dump with a database-wide search_path

[Regina Obe]

> From: Paul Ramsey [mailto:pram...@cleverelephant.ca]
> Sent: Monday, July 23, 2018 2:42 PM
> To: Regina Obe 
> Subject: Fwd: "interesting" issue with restore from a pg_dump with a
> database-wide search_path
> 
> Seen this one?
> P
> 
> 
> -- Forwarded message --
> From: Tom Lane 
> Date: Fri, Jul 6, 2018 at 1:10 PM
> Subject: Re: "interesting" issue with restore from a pg_dump with a
> database-wide search_path
> To: Larry Rosenman 
> Cc: "Joshua D. Drake" , pgsql-
> hack...@lists.postgresql.org
> 
> 
> Larry Rosenman  writes:
> > On Fri, Jul 06, 2018 at 11:35:41AM -0700, Joshua D. Drake wrote:
> >> Knowing the errors would be helpful.
> 
> > pg_restore: [archiver (db)] Error while PROCESSING TOC:
> > pg_restore: [archiver (db)] Error from TOC entry 12; 3079 887963
> > EXTENSION postgis_tiger_geocoder
> > pg_restore: [archiver (db)] could not execute query: ERROR:  function
> > soundex(character varying) does not exist
> > HINT:  No function matches the given name and argument types. You
> might need to add explicit type casts.
> 
> This looks like a problem with the postgis_tiger_geocoder extension.
> It's depending on the fuzzystrmatch extension (which has the soundex
> function), but seemingly this dependency is not declared in the extension's
> control file.  If it were, the search path would've been set to include the
> schema of the fuzzystrmatch extension during CREATE EXTENSION.
> 
> regards, tom lane
[Regina Obe] 

Sorry for not posting from the thread.  Paul alerted me to this one and I am 
aware of the issue.

1) I do have fuzzstrmatch listed as a dependency in the control file.  I know 
because I often install the geocoder with

CREATE EXTENSION postgis_tiger_geocoder CASCADE;

And it installs postgis and fuzzystrmatch

2) I have brought this issue up before and that's why we in fact had to schema 
qualify all postgis functions cause even with postgis within the same 
extension, things like materialized views fail to load.

3) My guess as to how this happens

a) In this particular case, I have a function that uses fuzzystrmatch and is 
used in functional indexes.
I unfortunately can't schema qualify the use of soundex, because I don't know 
where the user may have installed fuzzystrmatch is installed

b) Stephen Frost had suggested, perhaps we should have some syntax like 
@extension_loc(fuzzystrmatch)...@ so that one could reference an extension 
dependency location within a function without knowing where it is installed.


 

Re: How can we submit code patches that implement our (pending) patents?

[Nico Williams]

On Mon, Jul 23, 2018 at 03:06:13PM -0400, Bruce Momjian wrote:
> On Mon, Jul 23, 2018 at 02:02:40PM -0500, Nico Williams wrote:
> > On Mon, Jul 23, 2018 at 01:12:19PM -0400, Bruce Momjian wrote:
> > > On Mon, Jul 23, 2018 at 11:27:49AM -0500, Nico Williams wrote:
> > > > Perhaps patent law [in some countries] requires contracts as opposed to
> > > > licenses?
> > > 
> > > Yes, I really don't know.  I have just seen enough "oh, we didn't think
> > > of that" to be cautious.
> > 
> > So, is it FUD?  The core needs paid-for legal advice, not speculation.
> > 
> > I'm quite certain that a software license can make a patent grant to the
> > satisfaction of many open source communities, and almost certainly to
> > the satisfaction of the PG community.  But it will take an IP lawyer to
> > review or write such a license.
> 
> And is the payback worth it?  Many don't think so.

Wouldn't that be something to decide on a case-by-case basis?

Recall, a contributor might have acquired a patent only for the purpose
of defensive use, and the patent might involve useful-to-PG ideas.

Moreover, if PG re-invents those ideas later it then risks exposure to
lawsuit anyways.  Arguably, accepting suitably-wide patent grants serves
to protect PG from this risk!  Is that FUD from me, or good advice?
You'll need a lawyer to decide.

The important thing is the legal language of the grant.  This is why PG
needs legal advice.

Nico
-- 

Re: Bugs in TOAST handling, OID assignment and redo recovery

[Alvaro Herrera]

On 2018-Apr-11, Peter Eisentraut wrote:

> On 4/10/18 06:29, Pavan Deolasee wrote:
> > One of our 2ndQuadrant support customers recently reported a sudden rush
> > of TOAST errors post a crash recovery, nearly causing an outage. Most
> > errors read like this:
> > 
> > ERROR: unexpected chunk number 0 (expected 1) for toast value 
> 
> While researching this, I found that the terminology in this code is
> quite inconsistent.  It talks about chunks ids, chunk indexes, chunk
> numbers, etc. seemingly interchangeably.  The above error is actually
> about the chunk_seq, not about the chunk_id, as one might think.
> 
> The attached patch is my attempt to clean this up a bit.  Thoughts?

I didn't review your patch carefully, but +1 on the idea.  By all means
let's not forget this for pg12.

-- 
Álvaro Herrerahttps://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: How can we submit code patches that implement our (pending) patents?

[Joshua D. Drake]

On 07/23/2018 12:06 PM, Bruce Momjian wrote:

So, is it FUD?  The core needs paid-for legal advice, not speculation.

I'm quite certain that a software license can make a patent grant to the
satisfaction of many open source communities, and almost certainly to
the satisfaction of the PG community.  But it will take an IP lawyer to
review or write such a license.

And is the payback worth it?  Many don't think so.


Although Nico is correct, I also think we need to consider what the 
community wants here. Historically, we have always explicitly avoided 
anything to do with patents to the point where some hackers won't even 
read white papers on patented methods. I do think there is a definite 
technological advantage for PostgreSQL if there was a license that core 
could accept that was patent friendly but frankly, I don't think that 
core or the community has the desire to work through the cost of doing so.


JD

--
Command Prompt, Inc. || http://the.postgres.company/ || @cmdpromptinc
***  A fault and talent of mine is to tell it exactly how it is.  ***
PostgreSQL centered full stack support, consulting and development.
Advocate: @amplifypostgres || Learn: https://postgresconf.org
* Unless otherwise stated, opinions are my own.   *

Re: How can we submit code patches that implement our (pending) patents?

[Nico Williams]

On Mon, Jul 23, 2018 at 01:12:49PM -0400, Bruce Momjian wrote:
> On Mon, Jul 23, 2018 at 11:37:05AM -0500, Nico Williams wrote:
> > On Mon, Jul 23, 2018 at 11:40:41AM -0400, Bruce Momjian wrote:
> > > Sun Microsystems seemed reasonably trustworthy too.
> > 
> > Are there patent grants from Sun that Oracle has attempted to renege on?
> > Are there court cases about that?  Links?
> 
> No, but I bet there are things Oracle is doing that no one at Sun
> expected to be done, and users who relied on Sun didn't expect to be
> done.

There are questions that the PG core needs help with and which IP
lawyers are needed to answer.  There are also business questions,
because sure, even if a patent owner makes an acceptable grant, how fast
and cheaply you could get a lawsuit by them dismissed on the basis of
that grant is a business consideration.

We, the non-lawyer PG community, can give input such as that which I've
contributed:

 - I won't read/modify source code involving patents whose grants are
   not as wide as X

 - the PG core needs advice from IP lawyers

 - patents placed in the public domain surely are safe for PG

 - there must be patent grant language acceptable to PG

Just merely "but they could do something bad!" from us non-lawyers is
not very good advice.  Already PG incurs the risk that its contributors
could act in bad faith.  For example, a contributor's employer might sue
PG under copyright and/or trade secrecy law claiming the contribution
was not authorized (this is why some open source projects require
contributor agreements).

Nico
-- 

Re: Stored procedures and out parameters

[Vladimir Sitnikov]

Andrew>does it fail on the latest 11-stable

1) Current "build from Git/master PostgreSQL" produces the data row for
both simple and extended queries.

2) Just a side note: `CALL my_proc()` is not suitable for functions. That
looks weird.
Is the client expected to lookup system catalogs in order to tell if
`my_proc` is procedure or function and use either `call my_proc` or `select
* from my_proc()`?
Issuing `call my_function()` fails with 42883 ERROR: my_func(unknown) is
not a procedure

Note: JDBC defines two options to call a stored procedure:
   {?= call [(,, ...)]}
   {call [(,, ...)]}
see
https://docs.oracle.com/javase/9/docs/api/java/sql/CallableStatement.html

There's no notion if the called object is a procedure or function.
Note: PostgreSQL can have a function that `returns void`, and it is hard to
tell if {call test()} refers to a function or procedure.

Can functions and procedures be unified at the backend level?
For instance, support "CALL" for both of them.
Or support "select * ..." for both of them.

Vladimir​

Re: How can we submit code patches that implement our (pending) patents?

[Bruce Momjian]

On Mon, Jul 23, 2018 at 02:02:40PM -0500, Nico Williams wrote:
> On Mon, Jul 23, 2018 at 01:12:19PM -0400, Bruce Momjian wrote:
> > On Mon, Jul 23, 2018 at 11:27:49AM -0500, Nico Williams wrote:
> > > Perhaps patent law [in some countries] requires contracts as opposed to
> > > licenses?
> > 
> > Yes, I really don't know.  I have just seen enough "oh, we didn't think
> > of that" to be cautious.
> 
> So, is it FUD?  The core needs paid-for legal advice, not speculation.
> 
> I'm quite certain that a software license can make a patent grant to the
> satisfaction of many open source communities, and almost certainly to
> the satisfaction of the PG community.  But it will take an IP lawyer to
> review or write such a license.

And is the payback worth it?  Many don't think so.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+  Ancient Roman grave inscription +

Re: How can we submit code patches that implement our (pending) patents?

[Nico Williams]

On Mon, Jul 23, 2018 at 01:12:19PM -0400, Bruce Momjian wrote:
> On Mon, Jul 23, 2018 at 11:27:49AM -0500, Nico Williams wrote:
> > Perhaps patent law [in some countries] requires contracts as opposed to
> > licenses?
> 
> Yes, I really don't know.  I have just seen enough "oh, we didn't think
> of that" to be cautious.

So, is it FUD?  The core needs paid-for legal advice, not speculation.

I'm quite certain that a software license can make a patent grant to the
satisfaction of many open source communities, and almost certainly to
the satisfaction of the PG community.  But it will take an IP lawyer to
review or write such a license.

Nico
-- 

Re: cached plans and enable_partition_pruning

[Andres Freund]

On 2018-07-23 12:03:32 -0400, Alvaro Herrera wrote:
> On 2018-Jul-24, Amit Langote wrote:
> 
> > On Mon, Jul 23, 2018 at 11:20 PM, Andres Freund  wrote:
> 
> > > I think it's correct to check the plan time value, rather than the
> > > execution time value. Other enable_* GUCs also take effect there, and I
> > > don't see a problem with that?
> > 
> > Ah, so that may have been intentional.  Although, I wonder if
> > enable_partition_pruning could be made to work differently than other
> > enable_* settings, because we *can* perform pruning which is an
> > optimization function even during execution, whereas we cannot modify
> > the plan in other cases?
> 
> Well, let's discuss the use-case for doing that.  We introduced the GUC
> to cover for the case of bugs in the pruning code (and even then there
> was people saying we should remove it.)  Why would you have the GUC
> turned on during planning but off during execution?

I think it's even more than that: It'd not be consistent to take it into
account at execution time, and there'd have to be very convincing
reasons to behave differently.

Greetings,

Andres Freund

Re: Making "COPY partitioned_table FROM" faster

[Peter Eisentraut]

On 20.07.18 16:57, David Rowley wrote:
> One final note:  I'm not entirely convinced we need this adaptive
> code, but it seems easy enough to rip it back out if it's more trouble
> than it's worth. But if the other option is a GUC, then I'd rather
> stick with the adaptive code, it's likely going to do much better than
> a GUC since it can change itself during the copy which will be useful
> when the stream contains a mix small and large sets of consecutive
> tuples which belong to the same partition.

I think some kind of way to switch between the two code paths would be
desirable.  For example, with hash partitioning, it's likely that in
many cases you won't find any adjacent candidates in batches of
significant size.  So then you've just made everything 5% slower.
Unless we can make the multi-insert path itself faster.

The particular heuristic you have chosen seems sensible to me, but we'll
have to see how it holds up in practice.

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] logical decoding of two-phase transactions

[Andres Freund]

Hi,

On 2018-07-23 12:38:25 -0400, Robert Haas wrote:
> On Mon, Jul 23, 2018 at 12:13 PM, Andres Freund  wrote:
> > My point is that we could just make HTSV treat them as recently dead, 
> > without incurring the issues of the bug you referenced.
> 
> That doesn't seem sufficient.  For example, it won't keep the
> predecessor tuple's ctid field from being overwritten by a subsequent
> updater -- and if that happens then the update chain is broken.

Sure. I wasn't arguing that it'd be sufficient. Just that the specific
issue that it'd bring the bug you mentioned isn't right.  I agree that
it's quite terrifying to attempt to get this right.


> Maybe your idea of cross-checking at the end of each syscache lookup
> would be sufficient to prevent that from happening, though.

Hm? If we go for that approach we would not do *anything* about pruning,
which is why I think it has appeal. Because we'd check at the end of
system table scans (not syscache lookups, positive cache hits are fine
because of invalidation handling) whether the to-be-decoded transaction
aborted, we'd not need to do anything about pruning: If the transaction
aborted, we're guaranteed to know - the result might have been wrong,
but since we error out before filling any caches, we're ok.  If it
hasn't yet aborted at the end of the scan, we conversely are guaranteed
that the scan results are correct.

Greetings,

Andres Freund

Re: How can we submit code patches that implement our (pending) patents?

[Andres Freund]

On 2018-07-23 13:14:04 -0400, Bruce Momjian wrote:
> On Mon, Jul 23, 2018 at 11:38:47AM -0500, Nico Williams wrote:
> > On Mon, Jul 23, 2018 at 11:55:01AM -0400, Bruce Momjian wrote:
> > > On Mon, Jul 23, 2018 at 11:40:41AM -0400, Bruce Momjian wrote:
> > > > On Mon, Jul 23, 2018 at 08:19:35AM -0700, Andres Freund wrote:
> > > > > I'm fairly sure that I'm right. But my point isn't that we should 
> > > > > "trust
> > > > > Andres implicitly ™" (although that's obviously not a bad starting 
> > > > > point
> > > > > ;)). But rather, given that that is a reasonable assumption that such
> > > > > agreements are legally possible, we can decide whether we want to take
> > > > > advantage of such terms *assuming they are legally sound*. Then, if, 
> > > > > and
> > > > > only if, we decide that that's interesting from a policy POV, we can
> > > > > verify those assumptions with lawyers.
> > > > 
> > > > > 
> > > > > Given we're far from the first project dealing with this, and that
> > > > > companies that have shown themselves to be reasonably trustworthy 
> > > > > around
> > > > > open source, like Red Hat, assuming that such agreements are sound 
> > > > > seems
> > > > > quite reasonable.
> > > > 
> > > > Sun Microsystems seemed reasonably trustworthy too.
> > > 
> > > I realize what you are saying is that at the time Red Hat wrote that,
> > > they had good intentions, but they might not be able to control its
> > > behavior in a bankruptcy, so didn't mention it.  Also, Oracle is suing
> > > Google over the Java API:
> > > 
> > >   https://en.wikipedia.org/wiki/Oracle_America,_Inc._v._Google,_Inc.
> > > 
> > > which I can't imagine Sun doing, but legally Oracle can now that they
> > > own Java via Sun.  Of course, Sun might not have realized the problem,
> > > and Red Hat might have, but that's also assuming that there aren't other
> > > problems that Red Hat doesn't know about.
> > 
> > That's not about patents though, is it.
> 
> No, it is not.  It is just a case of not being able to trust any
> company's "good will".  You only get what the contract says.

> > (I do believe that case is highly contrived.  Sun put Java under the
> > GPL, so presumably Google can fork it under those terms.  I've not
> > followed that case, so I don't really know what's up with it or why it
> > wasn't just dismissed with prejudice.)
> 
> Yes, it is a big case with big ramifications if upheld.

Google didn't originally fork Dalvik from OpenJDK under the GPL though,
they re-implemented under a *more* permissive license (and reused Apache
Harmony bits). The "fix" google pursued is *precisely* to instead go for
an OpenJDK fork, to benefit from GPL protections.  But that doesn't help
them with retroactive claims about the time when that wasn't true.

There was no contract granting patent rights in the relevant case. So I
fail to see what this has to do with the topic at hand.

Greetings,

Andres Freund

Re: Expression errors with "FOR UPDATE" and postgres_fdw with partition wise join enabled.

[Robert Haas]

On Mon, Jul 23, 2018 at 3:49 AM, Etsuro Fujita
 wrote:
> I have to admit that that is not a good idea.  So, I'll update the patch so
> that we don't assume the projection capability of the subplan anymore, if we
> go this way.

Isn't that assumption fundamental to your whole approach?

>> Also, even today, this would fail if the subplan happened to be
>> a Sort, and it's not very obvious why that couldn't happen.
>
> You mean the MergeAppend case?  In that case, the patch will adjust the
> subplan before adding a Sort above the subplan, so that could not happen.

That would only help if create_merge_append_plan() itself inserted a
Sort node.  It wouldn't help if the Sort node came from a child path
manufactured by create_sort_path().

>> I think that's a bad idea.  The target list affects lots
>> of things, like costing.  If we don't insert a ConvertRowTypeExpr into
>> the child's target list, the costing will be wrong to the extent that
>> ConvertRowTypeExpr has any cost, which it does.
>
> Actually, this is not true at least currently, because set_append_rel_size
> doesn't do anything about the costing:

Why would it?  Append can't project, so the cost of any expressions
that appear in its target list is irrelevant.  What is affected is the
cost of the scans below the Append -- see e.g. cost_seqscan(), which
uses the data produced by set_pathtarget_cost_width().

> Some createplan.c routines already change the tlists of their nodes. For
> example, create_merge_append_plan would add sort columns to each of its
> subplans if necessary.  I think it would be similar to that to add a
> ConvertRowtypeExpr above a child whole-row Var in the subplan's tlist.

You have a point, but I think that code is actually not a very good
idea, and I'd like to see us do less of that sort of thing, not more.
Any case in which we change the plan while creating it has many of the
same problems that I discussed in the previous email.  For example,
create_merge_append_path() has to know that a Sort node might get
inserted and set the costing accordingly.  If the callers guaranteed
that the necessary sort path had already been inserted, then we
wouldn't need that special handling.

Also, that code is adding additional columns, computed from the
columns we have available, so that we can sort.  Those extra columns
then get discarded at the next level of the Plan tree.  What you're
trying to do is different.  Perhaps this is too harsh a judgement, but
it looks to me like you're using a deliberately-wrong representation
of the value that you ultimately want to produce and then patching it
up after the fact.  That seems quite a bit worse than what the
existing code is doing.

> I reviewed his patch, and thought that that would fix the issue, but this is
> about the current design on the child tlist handling in partitionise join
> rather than that patch: it deviates from the assumption that we had for the
> scan/join planning till PG10 that "a rel's targetlist would only include
> Vars and PlaceHolderVars", and turns out that there are a lot of places
> where we need to modify code to suppress that deviation, as partly shown in
> that patch.  So, ISTM that the current design in itself is not that
> localized.

I used to think that was the assumption, too, but it seems to me that
Tom told me a while back that it was never really true, and that
assumption was in my head.  Unfortunately, I don't have a link to that
email handy.  Either way, I think the solution is to get the tlist
right from the start and cope with the consequences downstream, not
start with the wrong thing and try to fix it later.  To see an example
of why I think that's a valuable approach, see
11cf92f6e2e13c0a6e3f98be3e629e6bd90b74d5, especially the changes in
the regression test outputs.  The old code discovered after it had
generated an Append that it had the wrong tlist, but since the Append
can't project, it had to add a Result node.  With the new code, we get
the children of the Append to produce the right output from the start,
and then the Append just needs to concatenate all that output, so no
Result node is needed.  As noted in the commit message, it also made
the costing more accurate.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: GiST VACUUM

[Andrey Borodin]

Hi!

> 21 июля 2018 г., в 17:11, Andrey Borodin  написал(а):
> 
> <0001-Physical-GiST-scan-in-VACUUM-v13.patch>

Just in case, here's second part of patch series with actual page deletion.

I was considering further decreasing memory footprint by using bloom filters 
instead of bitmap, but it will create seriously more work for cpu to compute 
hashes.

Best regards, Andrey Borodin.


0002-Delete-pages-during-GiST-VACUUM-v13.patch
Description: Binary data


0001-Physical-GiST-scan-in-VACUUM-v13.patch
Description: Binary data

Re: Stored procedures and out parameters

[Andrew Gierth]

> "Robert" == Robert Haas  writes:

 >> However, connecting via Npgsql, which uses the extended protocol, I
 >> see something quite different. As a response to a Describe
 >> PostgreSQL message, I get back a NoData response rather than a
 >> RowDescription message, In other words, it would seem that the
 >> behavior of stored procedures differs between the simple and
 >> extended protocols, when INOUT parameters are involved.

 Robert> I might be wrong, but that sounds like a bug.

Completely off the cuff, I'd expect 59a85323d might have fixed that;
does it fail on the latest 11-stable?

-- 
Andrew (irc:RhodiumToad)

Re: How can we submit code patches that implement our (pending) patents?

[Bruce Momjian]

On Mon, Jul 23, 2018 at 11:38:47AM -0500, Nico Williams wrote:
> On Mon, Jul 23, 2018 at 11:55:01AM -0400, Bruce Momjian wrote:
> > On Mon, Jul 23, 2018 at 11:40:41AM -0400, Bruce Momjian wrote:
> > > On Mon, Jul 23, 2018 at 08:19:35AM -0700, Andres Freund wrote:
> > > > I'm fairly sure that I'm right. But my point isn't that we should "trust
> > > > Andres implicitly ™" (although that's obviously not a bad starting point
> > > > ;)). But rather, given that that is a reasonable assumption that such
> > > > agreements are legally possible, we can decide whether we want to take
> > > > advantage of such terms *assuming they are legally sound*. Then, if, and
> > > > only if, we decide that that's interesting from a policy POV, we can
> > > > verify those assumptions with lawyers.
> > > 
> > > > 
> > > > Given we're far from the first project dealing with this, and that
> > > > companies that have shown themselves to be reasonably trustworthy around
> > > > open source, like Red Hat, assuming that such agreements are sound seems
> > > > quite reasonable.
> > > 
> > > Sun Microsystems seemed reasonably trustworthy too.
> > 
> > I realize what you are saying is that at the time Red Hat wrote that,
> > they had good intentions, but they might not be able to control its
> > behavior in a bankruptcy, so didn't mention it.  Also, Oracle is suing
> > Google over the Java API:
> > 
> > https://en.wikipedia.org/wiki/Oracle_America,_Inc._v._Google,_Inc.
> > 
> > which I can't imagine Sun doing, but legally Oracle can now that they
> > own Java via Sun.  Of course, Sun might not have realized the problem,
> > and Red Hat might have, but that's also assuming that there aren't other
> > problems that Red Hat doesn't know about.
> 
> That's not about patents though, is it.

No, it is not.  It is just a case of not being able to trust any
company's "good will".  You only get what the contract says.

> (I do believe that case is highly contrived.  Sun put Java under the
> GPL, so presumably Google can fork it under those terms.  I've not
> followed that case, so I don't really know what's up with it or why it
> wasn't just dismissed with prejudice.)

Yes, it is a big case with big ramifications if upheld.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+  Ancient Roman grave inscription +

Re: How can we submit code patches that implement our (pending) patents?

[Bruce Momjian]

On Mon, Jul 23, 2018 at 11:37:05AM -0500, Nico Williams wrote:
> On Mon, Jul 23, 2018 at 11:40:41AM -0400, Bruce Momjian wrote:
> > On Mon, Jul 23, 2018 at 08:19:35AM -0700, Andres Freund wrote:
> > > I'm fairly sure that I'm right. But my point isn't that we should "trust
> > > Andres implicitly ™" (although that's obviously not a bad starting point
> > > ;)). But rather, given that that is a reasonable assumption that such
> > > agreements are legally possible, we can decide whether we want to take
> > > advantage of such terms *assuming they are legally sound*. Then, if, and
> > > only if, we decide that that's interesting from a policy POV, we can
> > > verify those assumptions with lawyers.
> > 
> > > 
> > > Given we're far from the first project dealing with this, and that
> > > companies that have shown themselves to be reasonably trustworthy around
> > > open source, like Red Hat, assuming that such agreements are sound seems
> > > quite reasonable.
> > 
> > Sun Microsystems seemed reasonably trustworthy too.
> 
> Are there patent grants from Sun that Oracle has attempted to renege on?
> Are there court cases about that?  Links?

No, but I bet there are things Oracle is doing that no one at Sun
expected to be done, and users who relied on Sun didn't expect to be
done.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+  Ancient Roman grave inscription +

Re: How can we submit code patches that implement our (pending) patents?

[Bruce Momjian]

On Mon, Jul 23, 2018 at 11:27:49AM -0500, Nico Williams wrote:
> On Mon, Jul 23, 2018 at 09:56:47AM -0400, Bruce Momjian wrote:
> > On Mon, Jul 23, 2018 at 06:31:14AM -0700, Andres Freund wrote:
> > > It explicitly says irrevocable and successors. Why seems squarely
> > > aimed at your concern. Bankruptcy wouldn't just invalidate that.
> > 
> > They can say whatever they want, but if they are bankrupt, what they say
> > doesn't matter much.  My guess is that they would have to give their
> > patents to some legal entity that owns them so it is shielded from
> > bankruptcy.
> 
> Can you explain how a new owner could invalidate/revoke previous
> irrevocable grants?

The question is whether the promises Red Hat makes are part of the
license for free use of their patents or something they fully control
and just promise to do, i.e. is this a promise from the company or
embedded in the patent grant.  I don't know.

But it is a larger question of how such a promise is embedded in the
patent grant and _cannot_ be revoked, even if someone else buys the
patent in a bankruptcy case and does control that promise.

> That's not rhetorical.  I want to know if that's possible.
> 
> Perhaps patent law [in some countries] requires contracts as opposed to
> licenses?

Yes, I really don't know.  I have just seen enough "oh, we didn't think
of that" to be cautious.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+  Ancient Roman grave inscription +

Re: Stored procedures and out parameters

[Robert Haas]

On Mon, Jul 23, 2018 at 2:23 AM, Shay Rojansky  wrote:
> Hi hackers, I've encountered some odd behavior with the new stored procedure
> feature, when using INOUT parameters, running PostgreSQL 11-beta2.
>
> With the following procedure:
>
> CREATE OR REPLACE PROCEDURE my_proc(INOUT results text)
> LANGUAGE 'plpgsql'
> AS $BODY$
> BEGIN
> select 'test' into results;
> END;
> $BODY$;
>
> executing CALL my_proc('whatever') yields a resultset with a "results"
> column and a single row, containing "test". This is expected and is also how
> functions work.
>
> However, connecting via Npgsql, which uses the extended protocol, I see
> something quite different. As a response to a Describe PostgreSQL message, I
> get back a NoData response rather than a RowDescription message, In other
> words, it would seem that the behavior of stored procedures differs between
> the simple and extended protocols, when INOUT parameters are involved.

I might be wrong, but that sounds like a bug.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: How can we submit code patches that implement our (pending) patents?

[Nico Williams]

On Mon, Jul 23, 2018 at 11:55:01AM -0400, Bruce Momjian wrote:
> On Mon, Jul 23, 2018 at 11:40:41AM -0400, Bruce Momjian wrote:
> > On Mon, Jul 23, 2018 at 08:19:35AM -0700, Andres Freund wrote:
> > > I'm fairly sure that I'm right. But my point isn't that we should "trust
> > > Andres implicitly ™" (although that's obviously not a bad starting point
> > > ;)). But rather, given that that is a reasonable assumption that such
> > > agreements are legally possible, we can decide whether we want to take
> > > advantage of such terms *assuming they are legally sound*. Then, if, and
> > > only if, we decide that that's interesting from a policy POV, we can
> > > verify those assumptions with lawyers.
> > 
> > > 
> > > Given we're far from the first project dealing with this, and that
> > > companies that have shown themselves to be reasonably trustworthy around
> > > open source, like Red Hat, assuming that such agreements are sound seems
> > > quite reasonable.
> > 
> > Sun Microsystems seemed reasonably trustworthy too.
> 
> I realize what you are saying is that at the time Red Hat wrote that,
> they had good intentions, but they might not be able to control its
> behavior in a bankruptcy, so didn't mention it.  Also, Oracle is suing
> Google over the Java API:
> 
>   https://en.wikipedia.org/wiki/Oracle_America,_Inc._v._Google,_Inc.
> 
> which I can't imagine Sun doing, but legally Oracle can now that they
> own Java via Sun.  Of course, Sun might not have realized the problem,
> and Red Hat might have, but that's also assuming that there aren't other
> problems that Red Hat doesn't know about.

That's not about patents though, is it.

(I do believe that case is highly contrived.  Sun put Java under the
GPL, so presumably Google can fork it under those terms.  I've not
followed that case, so I don't really know what's up with it or why it
wasn't just dismissed with prejudice.)

Nico
-- 

Re: [HACKERS] logical decoding of two-phase transactions

[Robert Haas]

On Mon, Jul 23, 2018 at 12:13 PM, Andres Freund  wrote:
> My point is that we could just make HTSV treat them as recently dead, without 
> incurring the issues of the bug you referenced.

That doesn't seem sufficient.  For example, it won't keep the
predecessor tuple's ctid field from being overwritten by a subsequent
updater -- and if that happens then the update chain is broken.  Maybe
your idea of cross-checking at the end of each syscache lookup would
be sufficient to prevent that from happening, though.  But I wonder if
there are subtler problems, too -- e.g. relfrozenxid vs. actual xmins
in the table, clog truncation, or whatever.  There might be no
problem, but the idea that an aborted transaction is of no further
interest to anybody is pretty deeply ingrained in the system.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: How can we submit code patches that implement our (pending) patents?

[Nico Williams]

On Mon, Jul 23, 2018 at 11:40:41AM -0400, Bruce Momjian wrote:
> On Mon, Jul 23, 2018 at 08:19:35AM -0700, Andres Freund wrote:
> > I'm fairly sure that I'm right. But my point isn't that we should "trust
> > Andres implicitly ™" (although that's obviously not a bad starting point
> > ;)). But rather, given that that is a reasonable assumption that such
> > agreements are legally possible, we can decide whether we want to take
> > advantage of such terms *assuming they are legally sound*. Then, if, and
> > only if, we decide that that's interesting from a policy POV, we can
> > verify those assumptions with lawyers.
> 
> > 
> > Given we're far from the first project dealing with this, and that
> > companies that have shown themselves to be reasonably trustworthy around
> > open source, like Red Hat, assuming that such agreements are sound seems
> > quite reasonable.
> 
> Sun Microsystems seemed reasonably trustworthy too.

Are there patent grants from Sun that Oracle has attempted to renege on?
Are there court cases about that?  Links?

Nico
-- 

Re: How can we submit code patches that implement our (pending) patents?

[Nico Williams]

On Mon, Jul 23, 2018 at 10:13:48AM -0400, Chapman Flack wrote:
> On 07/23/2018 10:01 AM, Bruce Momjian wrote:
> 
> > And the larger question is whether a patent free for use by software
> > under any license can be used in a defensive way.  If not, it means we
> > have no way forward here.
> 
> Isn't 'defensive', in patent-speak, used to mean 'establishing prior
> art usable to challenge future patent claims by others on the same
> technique'?

Not prior art.  You don't need patents to establish prior art.  Just
publish your idea and you're done.  (The problem with just publishing an
idea is that someone might have a patent application regarding a similar
idea and might modify their application to cover yours once they see it.
Later, when you go claim that you have prior art, there will be a heavy
burden of presumption on you.  This, of course, encourages you to apply
for a patent...)

Patents let you have counter-suit options should you get sued for patent
infringement -- that's the "defensive" in defensive patents.  You need a
large portfolio of patents, and this doesn't work against patent trolls.

There's also no-lawsuit covenants in grants.  You get to use my patent
if you don't sue me for using yours, and if you sue me then my grants to
you are revoked.

Nico
-- 

Re: Should contrib modules install .h files?

[Andrew Gierth]

> "Tom" == Tom Lane  writes:

 Tom> As I said before, I think that we should change the existing
 Tom> contrib modules to be coded likewise, all using a single -I switch
 Tom> that points at SRCDIR/contrib. That'd help give people the right
 Tom> coding model to follow.

I don't see that playing nicely with PGXS?

-- 
Andrew.

Re: How can we submit code patches that implement our (pending) patents?

[Nico Williams]

On Mon, Jul 23, 2018 at 09:56:47AM -0400, Bruce Momjian wrote:
> On Mon, Jul 23, 2018 at 06:31:14AM -0700, Andres Freund wrote:
> > It explicitly says irrevocable and successors. Why seems squarely
> > aimed at your concern. Bankruptcy wouldn't just invalidate that.
> 
> They can say whatever they want, but if they are bankrupt, what they say
> doesn't matter much.  My guess is that they would have to give their
> patents to some legal entity that owns them so it is shielded from
> bankrupcy.

Can you explain how a new owner could invalidate/revoke previous
irrevocable grants?

That's not rhetorical.  I want to know if that's possible.

Perhaps patent law [in some countries] requires contracts as opposed to
licenses?

Nico
-- 

Re: [HACKERS] logical decoding of two-phase transactions

[Andres Freund]

On July 23, 2018 9:11:13 AM PDT, Robert Haas  wrote:
>On Thu, Jul 19, 2018 at 3:42 PM, Andres Freund 
>wrote:
>> I don't think this reasoning actually applies for making HOT pruning
>> weaker as necessary for decoding. The xmin horizon on catalog tables
>is
>> already pegged, which'd prevent similar problems.
>
>That sounds completely wrong to me.  Setting the xmin horizon keeps
>tuples that are made dead by a committing transaction from being
>removed, but I don't think it will do anything to keep tuples that are
>made dead by an aborting transaction from being removed.

My point is that we could just make HTSV treat them as recently dead, without 
incurring the issues of the bug you referenced.

Andres
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Should contrib modules install .h files?

[Tom Lane]

Peter Eisentraut  writes:
> Also, let's recall that the point of this exercise is that you want to
> install the header files so that you can build things (another
> extension) that somehow interacts with those extensions.  Then, even if
> you put things in separate directories per extension, you still need to
> make sure that all the installed header files don't clash, since you'll
> be adding the -I options of several of them.  In a way, doing it this
> way will make things less robust, since it will appear to give extension
> authors license to use generic header names.

Personally, I'd recommend using *one* -I switch and having .c files
reference extension headers with #include "extensionname/headername.h".

As I said before, I think that we should change the existing contrib
modules to be coded likewise, all using a single -I switch that points
at SRCDIR/contrib.  That'd help give people the right coding model
to follow.

regards, tom lane

Re: [HACKERS] logical decoding of two-phase transactions

[Robert Haas]

On Thu, Jul 19, 2018 at 3:42 PM, Andres Freund  wrote:
> I don't think this reasoning actually applies for making HOT pruning
> weaker as necessary for decoding. The xmin horizon on catalog tables is
> already pegged, which'd prevent similar problems.

That sounds completely wrong to me.  Setting the xmin horizon keeps
tuples that are made dead by a committing transaction from being
removed, but I don't think it will do anything to keep tuples that are
made dead by an aborting transaction from being removed.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Remove psql's -W option

[Tom Lane]

Robert Haas  writes:
> I also think your other question is a good one.  It seems like the
> fact that we need to reconnect -- rather than just prompting for the
> password and then sending it when we get it -- is an artifact of how
> libpq is designed rather than an intrinsic limitation of the protocol.

Well, it's a limitation of the libpq API.  The problem is that it's the
application, not libpq, that's in charge of actually asking the user for
a password.  Right now we inform the app that it needs to do that by
passing back a failed PGconn with appropriate state.  We could imagine
passing back a PGconn with a half-finished open connection, and asking
the app to re-submit that PGconn along with a password so we could
continue the auth handshake.  But it'd require changing apps to do that.

Also, doing things like that would incur the risk of exceeding
authentication_timeout while the user is typing his password.  So we'd
also need some additional complexity to retry in that situation.

regards, tom lane

Re: pgbench - remove double declaration of hash functions

[Robert Haas]

On Sun, Jul 22, 2018 at 7:14 PM, Fabien COELHO  wrote:
> I noticed that hash functions appear twice in the list of pgbench functions,
> although once is enough. The code is functional nevertheless, but it looks
> silly. This was added by "e51a04840a1" back in March, so should be removed
> from 11 and 12dev.

Good catch.  Committed and back-patched.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: cached plans and enable_partition_pruning

[Amit Langote]

On Mon, Jul 23, 2018 at 11:20 PM, Andres Freund  wrote:
> Hi,
>
> On 2018-07-23 18:31:43 +0900, Amit Langote wrote:
>> It seems that because enable_partition_pruning's value is only checked
>> during planning, turning it off *after* a plan is created and cached does
>> not work as expected.

[ ... ]

>> Should we check its value during execution too, as done in the attached?
>
> I think it's correct to check the plan time value, rather than the
> execution time value. Other enable_* GUCs also take effect there, and I
> don't see a problem with that?

Ah, so that may have been intentional.  Although, I wonder if
enable_partition_pruning could be made to work differently than other
enable_* settings, because we *can* perform pruning which is an
optimization function even during execution, whereas we cannot modify
the plan in other cases?

Thanks,
Amit

Re: How can we submit code patches that implement our (pending) patents?

[Andres Freund]

Hi,

On 2018-07-23 11:40:41 -0400, Bruce Momjian wrote:
> Sun Microsystems seemed reasonably trustworthy too.

I don't really agree with that characterization (they've a long history
of weird behaviour around open source, LONG before the Oracle
acquisition). But it doesn't really matter, as they've not breached any
of their hard obligations, have they?

Greetings,

Andres Freund

Re: How can we submit code patches that implement our (pending) patents?

[Bruce Momjian]

On Mon, Jul 23, 2018 at 11:40:41AM -0400, Bruce Momjian wrote:
> On Mon, Jul 23, 2018 at 08:19:35AM -0700, Andres Freund wrote:
> > I'm fairly sure that I'm right. But my point isn't that we should "trust
> > Andres implicitly ™" (although that's obviously not a bad starting point
> > ;)). But rather, given that that is a reasonable assumption that such
> > agreements are legally possible, we can decide whether we want to take
> > advantage of such terms *assuming they are legally sound*. Then, if, and
> > only if, we decide that that's interesting from a policy POV, we can
> > verify those assumptions with lawyers.
> 
> > 
> > Given we're far from the first project dealing with this, and that
> > companies that have shown themselves to be reasonably trustworthy around
> > open source, like Red Hat, assuming that such agreements are sound seems
> > quite reasonable.
> 
> Sun Microsystems seemed reasonably trustworthy too.

I realize what you are saying is that at the time Red Hat wrote that,
they had good intentions, but they might not be able to control its
behavior in a bankruptcy, so didn't mention it.  Also, Oracle is suing
Google over the Java API:

https://en.wikipedia.org/wiki/Oracle_America,_Inc._v._Google,_Inc.

which I can't imagine Sun doing, but legally Oracle can now that they
own Java via Sun.  Of course, Sun might not have realized the problem,
and Red Hat might have, but that's also assuming that there aren't other
problems that Red Hat doesn't know about.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+  Ancient Roman grave inscription +

Re: [HACKERS] logical decoding of two-phase transactions

[Nikhil Sontakke]

Hi Andres,

>> We can find out if the snapshot is a logical decoding one by virtue of
>> its "satisfies" function pointing to HeapTupleSatisfiesHistoricMVCC.
> 
> I think we even can just do something like a global
> TransactionId check_if_transaction_is_alive = InvalidTransactionId;
> and just set it up during decoding. And then just check it whenever it's
> not set tot InvalidTransactionId.
> 
> 

Ok. I will work on something along these lines and re-submit the set of 
patches. 

Regards, 
Nikhils

Re: How can we submit code patches that implement our (pending) patents?

[Bruce Momjian]

On Mon, Jul 23, 2018 at 08:19:35AM -0700, Andres Freund wrote:
> I'm fairly sure that I'm right. But my point isn't that we should "trust
> Andres implicitly ™" (although that's obviously not a bad starting point
> ;)). But rather, given that that is a reasonable assumption that such
> agreements are legally possible, we can decide whether we want to take
> advantage of such terms *assuming they are legally sound*. Then, if, and
> only if, we decide that that's interesting from a policy POV, we can
> verify those assumptions with lawyers.

> 
> Given we're far from the first project dealing with this, and that
> companies that have shown themselves to be reasonably trustworthy around
> open source, like Red Hat, assuming that such agreements are sound seems
> quite reasonable.

Sun Microsystems seemed reasonably trustworthy too.

> I find it fairly hubristic to just assume bad faith, or lack of skill,
> on part of the drafters of Apache2, GLPv3, RH patent promise, ... that
> they either didn't think about bankruptcy or didn't care about
> it. They're certainly better lawyers than any of us here. 

True.  It would be nice if we could get an answer about bankruptcy from
Red Hat.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+  Ancient Roman grave inscription +

Re: pgbench-ycsb

[Robert Haas]

On Sun, Jul 22, 2018 at 4:42 PM, Fabien COELHO  wrote:
> Basically I'm against having something called YCSB if it is not YCSB;-)

Yep, that seems pretty clear.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Should contrib modules install .h files?

[Andrew Gierth]

> "Peter" == Peter Eisentraut  writes:

 Peter> Nobody said anything about one-file-per-extension. You can of
 Peter> course have hstore_this.h and hstore_that.h or if you want to
 Peter> have many, use postgis/this.h and postgis/that.h.

So now you want the extension to be able to _optionally_ specify a
subdirectory?

(just having the extension do  HEADERS=foo/bar.h  will not work, because
as with every other similar makefile variable, that means "install the
file foo/bar.h as bar.h, not as foo/bar.h".)

-- 
Andrew (irc:RhodiumToad)

Re: BUG #15182: Canceling authentication due to timeout aka Denial of Service Attack

[Robert Haas]

On Thu, Jul 19, 2018 at 7:17 PM, Jeremy Schneider  wrote:
> I'd like to bump this old bug that Lloyd filed for more discussion. It
> seems serious enough to me that we should at least talk about it.
>
> Anyone with simply the login privilege and the ability to run SQL can
> instantly block all new incoming connections to a DB including new
> superuser connections.
>
> session 1:
> select pg_sleep(99) from pg_stat_activity;
>
> session 2:
> vacuum full pg_authid; -or- truncate table pg_authid;
>
> (there are likely other SQL you could run in session 2 as well.)

ExecuteTruncate needs to be refactored to use RangeVarGetRelidExtended
with a non-NULL callback rather than heap_openrv, and
expand_vacuum_rel needs to use RangeVarGetRelidExtended with a
callback instead of RangeVarGetRelid.  See
cbe24a6dd8fb224b9585f25b882d5ffdb55a0ba5 as an example of what to do.
I fixed a large number of cases of this problem back around that time,
but then ran out of steam and had to move onto other things before I
got them all.  Patches welcome.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Remove psql's -W option

[David Fetter]

On Mon, Jul 23, 2018 at 11:20:46AM -0400, Robert Haas wrote:
> On Sun, Jul 22, 2018 at 9:35 AM, Fabien COELHO  wrote:
> > Otherwise ISTM that "-W/--password" still has some minimal value thus does
> > not deserve to be thrown out that quickly.
> 
> I think I agree.  I don't think this option is really hurting
> anything, so I'm not quite sure why we would want to abruptly get rid
> of it.
> 
> I also think your other question is a good one.  It seems like the
> fact that we need to reconnect -- rather than just prompting for the
> password and then sending it when we get it -- is an artifact of how
> libpq is designed rather than an intrinsic limitation of the protocol.

Am I understanding correctly that doing the following would be
acceptable, assuming good code quality?

- Rearrange libpq so it doesn't force this behavior.
- Deprecate the -W option uniformly in the code we ship by documenting
  it and making it send warnings to stderr.

Best,
David.
-- 
David Fetter  http://fetter.org/
Phone: +1 415 235 3778

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

Re: How can we submit code patches that implement our (pending) patents?

[Andres Freund]

Hi,

On 2018-07-23 17:11:30 +0200, David Fetter wrote:
> Yet again, you are assuming contrary to reality that you can simply
> read and understand how legal code will operate without court cases to
> back it.

Oh, FFS. You're implying serious bad faith here (and not just on my
part, but also on the drafters of apache2, gplv3 etc).  I *explicitly*
said that we should check in with lawyers if we decided we want to go
forward.

You're arguing that we shouldn't even consider anything around patent
grants because YOU don't know of caselaw supporting what's
proposed. You're very clearly not a lawyer. You're doing precisely what
you warn about.


> In the particular instance you're citing, that's what happens
> *after* the contract is upheld as legal, which it could well not be.

In which case we'd likely be screwed anyway, because the right to the
code would quite likely not be effective anymore either. A lot of the
relevant law treats copyright and patents similarly.

Greetings,

Andres Freund

Re: Should contrib modules install .h files?

[Peter Eisentraut]

On 23.07.18 06:15, Tom Lane wrote:
> Michael Paquier  writes:
>> On Sun, Jul 22, 2018 at 09:42:08PM -0400, Stephen Frost wrote:
>>> So, +1 from me for having a directory for each extension.
> 
>> So, like Stephen, that's a +1 from me.
> 
> Same here.  One-file-per-extension is too strongly biased to tiny
> extensions (like most of our contrib examples).

Nobody said anything about one-file-per-extension.  You can of course
have hstore_this.h and hstore_that.h or if you want to have many, use
postgis/this.h and postgis/that.h.  That's how every C package in the
world works.  We don't need to legislate further here other than, use
sensible naming.

Also, let's recall that the point of this exercise is that you want to
install the header files so that you can build things (another
extension) that somehow interacts with those extensions.  Then, even if
you put things in separate directories per extension, you still need to
make sure that all the installed header files don't clash, since you'll
be adding the -I options of several of them.  In a way, doing it this
way will make things less robust, since it will appear to give extension
authors license to use generic header names.

> I don't have a real strong opinion on whether it's too late to
> push this into v11.  I do not think it'd break anything other than
> packagers' lists of files to be installed ... but it does seem
> like a new feature, and we're past feature freeze.

Certainly a new feature.  I suggest submitting it to the next commit fest.

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Remove psql's -W option

[Robert Haas]

On Sun, Jul 22, 2018 at 9:35 AM, Fabien COELHO  wrote:
> Otherwise ISTM that "-W/--password" still has some minimal value thus does
> not deserve to be thrown out that quickly.

I think I agree.  I don't think this option is really hurting
anything, so I'm not quite sure why we would want to abruptly get rid
of it.

I also think your other question is a good one.  It seems like the
fact that we need to reconnect -- rather than just prompting for the
password and then sending it when we get it -- is an artifact of how
libpq is designed rather than an intrinsic limitation of the protocol.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: How can we submit code patches that implement our (pending) patents?

[Andres Freund]

On 2018-07-23 11:06:25 -0400, Bruce Momjian wrote:
> On Mon, Jul 23, 2018 at 07:59:20AM -0700, Andres Freund wrote:
> > Hi,
> > 
> > On 2018-07-23 16:32:55 +0200, David Fetter wrote:
> > > On Mon, Jul 23, 2018 at 06:31:14AM -0700, Andres Freund wrote:
> > > > On July 23, 2018 6:25:42 AM PDT, Bruce Momjian  wrote:
> > > > >Notice this makes no mention of what happens to the patents if the
> > > > >company goes bankrupt.  My guess is that in such a situation the
> > > > >company
> > > > >would have no control over who buys the patents or how they are used.
> > > > 
> > > > It explicitly says irrevocable and successors. Why seems squarely
> > > > aimed at your concern. Bankruptcy wouldn't just invalidate that.
> > > 
> > > Until this has been upheld in court, it's just a vague idea.
> > 
> > To my knowledge this has long been settled. Which makes a lot of sense,
> > because this is relevant *far* beyond just open source.  FWIW, in the US
> > it's explicit law, see https://www.law.cornell.edu/uscode/text/11/365
> > subclause (n).  Anyway, should we decide that this would be a good idea
> > as a policy matter, we'd *OBVIOUSLY* have to check in with lawyers to
> > see whether our understanding is correct. But that doesn't mean we
> > should just assume it's impossible without any sort of actual
> > understanding.
> 
> You are saying that Red Hat's promise is part of the contract when they
> contributed that code --- I guess that interpretation is possible, but
> again, I am not sure.

I'm fairly sure that I'm right. But my point isn't that we should "trust
Andres implicitly ™" (although that's obviously not a bad starting point
;)). But rather, given that that is a reasonable assumption that such
agreements are legally possible, we can decide whether we want to take
advantage of such terms *assuming they are legally sound*. Then, if, and
only if, we decide that that's interesting from a policy POV, we can
verify those assumptions with lawyers.

Given we're far from the first project dealing with this, and that
companies that have shown themselves to be reasonably trustworthy around
open source, like Red Hat, assuming that such agreements are sound seems
quite reasonable.

I find it fairly hubristic to just assume bad faith, or lack of skill,
on part of the drafters of Apache2, GLPv3, RH patent promise, ... that
they either didn't think about bancruptcy or didn't care about
it. They're certainly better lawyers than any of us here. 

Greetings,

Andres Freund

Re: How can we submit code patches that implement our (pending) patents?

[David Fetter]

On Mon, Jul 23, 2018 at 07:59:20AM -0700, Andres Freund wrote:
> Hi,
> 
> On 2018-07-23 16:32:55 +0200, David Fetter wrote:
> > On Mon, Jul 23, 2018 at 06:31:14AM -0700, Andres Freund wrote:
> > > On July 23, 2018 6:25:42 AM PDT, Bruce Momjian  wrote:
> > > >Notice this makes no mention of what happens to the patents if the
> > > >company goes bankrupt.  My guess is that in such a situation the
> > > >company
> > > >would have no control over who buys the patents or how they are used.
> > > 
> > > It explicitly says irrevocable and successors. Why seems squarely
> > > aimed at your concern. Bankruptcy wouldn't just invalidate that.
> > 
> > Until this has been upheld in court, it's just a vague idea.
> 
> To my knowledge this has long been settled. Which makes a lot of sense,
> because this is relevant *far* beyond just open source.  FWIW, in the US
> it's explicit law, see https://www.law.cornell.edu/uscode/text/11/365
> subclause (n).  Anyway, should we decide that this would be a good idea
> as a policy matter, we'd *OBVIOUSLY* have to check in with lawyers to
> see whether our understanding is correct. But that doesn't mean we
> should just assume it's impossible without any sort of actual
> understanding.

Yet again, you are assuming contrary to reality that you can simply
read and understand how legal code will operate without court cases to
back it. In the particular instance you're citing, that's what happens
*after* the contract is upheld as legal, which it could well not be.
There are lots of ways it might not be, even if you don't count bad
will and venue shopping, tactics known to be used ubiquitously by
NPEs.

We know things about the GPL because courts have upheld provisions in
it, not because somebody's idea of the "obvious meaning" of that
document held sway.

> You're just muddying the waters.

Nope. I'm just making sure we understand that when it comes to patent
thickets, it's a LOT easier to get in than out.

Best,
David.
-- 
David Fetter  http://fetter.org/
Phone: +1 415 235 3778

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

Re: Have an encrypted pgpass file

[Robert Haas]

On Wed, Jul 18, 2018 at 11:19 PM, Tom Lane  wrote:
> Sorry, I don't buy that line of argument.  The *only* reason for this
> feature to exist is if it allows ready creation of security solutions
> that are actually more secure than a non-world-readable .pgpass file.
> That's a much higher bar than many people realize to begin with ...
> and if it comes along with huge risk of security foot-guns, I do not
> think that it's going to be a net advance.

I don't think I agree with this objection.  First, not doing anything
won't be a net advance, either.  Second, your objection seems akin to
saying "we're not going to let you drive because you might crash the
car".  There are *some* people who should not be allowed to get behind
the wheel, but your proposal seems analogous to banning *everyone*
from driving on the theory that car crashes are bad.  I think that's
an overreaction.  I agree that there's probably a risk, but why can't
we just document best practices?  Really, I'm not sure that it's right
to suppose that you're calling a shell script specifically.  If it's a
Perl, Python, Ruby, etc. script the risk is probably much less --
you're going to take $ARGV[1] or the equivalent and shove it in a
string variable, and after that it's not really any more or less risky
than any other string variable you've got.  You could of course
perform an ill-considered interpolation into a shell command, but
that's true of any string that originates from a user in any
situation, and if you're a somewhat-knowledgeable programmer you
probably won't.  Generally you have to do *extra* work to make things
safe in the shell, whereas in a scripting language you just have to
not screw up.  foo($thingy) is safe in Perl; foo $thingy is unsafe in
the shell.  Of course mistakes are possible and we can avoid all the
mistakes by not providing the feature, but to me, that doesn't seem
like the way to go.

> One reason I'd like to see a concrete use-case (or several concrete
> use-cases) is that we might then find some design that's less prone
> to such mistakes than "here, run this shell script" is going to be.

I think that the most common use case is likely to be to get the data
from a local or remote keyserver.  For example, when I'm logged in, my
keychain is available to provide passwords; when I log out, those
passwords aren't accessible any more.  Or, when the server is in the
datacenter where it's supposed to be located, it can pull the data
from some other machine in that data center whose job it is provide
said data; when the server is physically stolen from the datacenter
and taken to some other location, the other machine isn't there and
necessary credentials are no longer available (or even if the other
machine *is* there, it probably requires a manually-entered password
to start the key service, which the thief may not have).

> I'm vaguely imagining exec'ing a program directly without a layer
> of shell quoting/evaluation in between; but not sure how far that
> gets us.

It's not a bad thought, although it might not help that much if it
causes somebody who would have written PGPASSCOMMAND="this that" to
instead set PGPASSCOMMAND="thisthat.sh" where that file contains

#!/bin/bash
this that

...which seems like a likely outcome.

> Another question that ought to be asked somewhere along here is
> "how well does this work on Windows?" ...

True.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Alter index rename concurrently to

[Peter Eisentraut]

On 23.07.18 15:14, Andrey Klychkov wrote:
> Moreover, if you rename Table without query locking, it may crushes your
> services that
> do queries at the same time, therefore, this is unlikely that someone
> will be do it
> with concurrent queries to renamed table, in other words, with running
> production.
> So, I think it doesn't have real sense to use the lower lock for example
> for tables (at least by default).
> However, renaming Indexes with the lower lock is safe for database consumers
> because they don't know anything about them.

You appear to be saying that you think that renaming an index
concurrently is not safe.  In that case, this patch should be rejected.
However, I don't think it necessarily is unsafe.  What we need is some
reasoning about the impact, not a bunch of different options that we
don't understand.

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: How can we submit code patches that implement our (pending) patents?

[Bruce Momjian]

On Mon, Jul 23, 2018 at 07:59:20AM -0700, Andres Freund wrote:
> Hi,
> 
> On 2018-07-23 16:32:55 +0200, David Fetter wrote:
> > On Mon, Jul 23, 2018 at 06:31:14AM -0700, Andres Freund wrote:
> > > On July 23, 2018 6:25:42 AM PDT, Bruce Momjian  wrote:
> > > >Notice this makes no mention of what happens to the patents if the
> > > >company goes bankrupt.  My guess is that in such a situation the
> > > >company
> > > >would have no control over who buys the patents or how they are used.
> > > 
> > > It explicitly says irrevocable and successors. Why seems squarely
> > > aimed at your concern. Bankruptcy wouldn't just invalidate that.
> > 
> > Until this has been upheld in court, it's just a vague idea.
> 
> To my knowledge this has long been settled. Which makes a lot of sense,
> because this is relevant *far* beyond just open source.  FWIW, in the US
> it's explicit law, see https://www.law.cornell.edu/uscode/text/11/365
> subclause (n).  Anyway, should we decide that this would be a good idea
> as a policy matter, we'd *OBVIOUSLY* have to check in with lawyers to
> see whether our understanding is correct. But that doesn't mean we
> should just assume it's impossible without any sort of actual
> understanding.

You are saying that Red Hat's promise is part of the contract when they
contributed that code --- I guess that interpretation is possible, but
again, I am not sure.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+  Ancient Roman grave inscription +

Re: How can we submit code patches that implement our (pending) patents?

[Andres Freund]

Hi,

On 2018-07-23 16:32:55 +0200, David Fetter wrote:
> On Mon, Jul 23, 2018 at 06:31:14AM -0700, Andres Freund wrote:
> > On July 23, 2018 6:25:42 AM PDT, Bruce Momjian  wrote:
> > >Notice this makes no mention of what happens to the patents if the
> > >company goes bankrupt.  My guess is that in such a situation the
> > >company
> > >would have no control over who buys the patents or how they are used.
> > 
> > It explicitly says irrevocable and successors. Why seems squarely
> > aimed at your concern. Bankruptcy wouldn't just invalidate that.
> 
> Until this has been upheld in court, it's just a vague idea.

To my knowledge this has long been settled. Which makes a lot of sense,
because this is relevant *far* beyond just open source.  FWIW, in the US
it's explicit law, see https://www.law.cornell.edu/uscode/text/11/365
subclause (n).  Anyway, should we decide that this would be a good idea
as a policy matter, we'd *OBVIOUSLY* have to check in with lawyers to
see whether our understanding is correct. But that doesn't mean we
should just assume it's impossible without any sort of actual
understanding.

You're just muddying the waters.

Greetings,

Andres Freund

Re: How can we submit code patches that implement our (pending) patents?

[Bruce Momjian]

On Mon, Jul 23, 2018 at 10:42:11AM -0400, Chapman Flack wrote:
> On 07/23/2018 10:25 AM, Bruce Momjian wrote:
> 
> >> Isn't 'defensive', in patent-speak, used to mean 'establishing prior
> >> art usable to challenge future patent claims by others on the same
> >> technique'?
> >>
> >> Is there any way that conditions of use, or lack of them, on an
> >> existing patent, would make it unusable in that context?
> > 
> > It doesn't have to be a patent on the same technique;  this URL was
> > referenced in the thread:
> > 
> > https://en.wikipedia.org/wiki/Defensive_termination
> 
> Ah, a very different understanding of defensive use of a patent,
> and one that I can see would lose force if there could be no
> conditions on its use.
> 
> I was thinking more of the use of a filing to establish prior art
> so somebody else later can't obtain and enforce a patent on
> the technique that you're already using. Something along the lines
> of a statutory invention registration[1], which used to be a thing
> in the US, but now apparently is not, though any filed, published
> application, granted or abandoned, can serve the same purpose.
> 
> That, I think, would still work.

Yes, those preemptive patent approaches work too.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+  Ancient Roman grave inscription +

Re: How can we submit code patches that implement our (pending) patents?

[Chapman Flack]

On 07/23/2018 10:25 AM, Bruce Momjian wrote:

>> Isn't 'defensive', in patent-speak, used to mean 'establishing prior
>> art usable to challenge future patent claims by others on the same
>> technique'?
>>
>> Is there any way that conditions of use, or lack of them, on an
>> existing patent, would make it unusable in that context?
> 
> It doesn't have to be a patent on the same technique;  this URL was
> referenced in the thread:
> 
>   https://en.wikipedia.org/wiki/Defensive_termination

Ah, a very different understanding of defensive use of a patent,
and one that I can see would lose force if there could be no
conditions on its use.

I was thinking more of the use of a filing to establish prior art
so somebody else later can't obtain and enforce a patent on
the technique that you're already using. Something along the lines
of a statutory invention registration[1], which used to be a thing
in the US, but now apparently is not, though any filed, published
application, granted or abandoned, can serve the same purpose.

That, I think, would still work.

-Chap

[1]
https://en.wikipedia.org/wiki/United_States_Statutory_Invention_Registration

Re: Log query parameters for terminated execute

[Sergei Kornilov]

Hello

23.07.2018, 17:08, "Tom Lane" :
> Sergei Kornilov  writes:
>>  Please test with logging command tag %i in log_line_prefix. Extended 
>> protocol has three different messages, each can be canceled by timeout. But 
>> here is completely no parameters in PARSE and i did not change BIND in first 
>> patch.
>
> This patch scares me to death. It risks calling user-defined I/O
> functions in all sorts of weird places, particularly outside transactions,
> or in already-failed transactions, or with no ActiveSnapshot.
This is reason why i start thread with question how do it right way
As i wrote in beginning:
> i have no good idea how print ParamListInfo correctly. We can not use 
> OidOutputFunctionCall in all cases, right?
Attached patch is just simple enough to illustrate one possible way.
I can further work with proper design, but i need idea how it should look.

> Separately from that concern: it appears to result in a substantial
> degradation of existing functionality in the places where you did
> s/errdetail/errdetail_log/. What was the reason for that?
This is my second question at thread beginning. Why used errdetail? We assume 
that the user wants to get their own parameters back (if he set 
client_min_messages to LOG)?

regards, Sergei

Re: Indicate anti-wraparound autovacuum in log_autovacuum_min_duration

[Robert Haas]

On Sat, Jul 21, 2018 at 4:22 AM, Michael Paquier  wrote:
> On Sat, Jul 21, 2018 at 09:38:38AM +0300, Sergei Kornilov wrote:
>> Currently log_autovacuum_min_duration log message has no difference
>> between regular autovacuum and to prevent wraparound autovacuum. There
>> are important differences, for example, backend can automatically
>> cancel regular autovacuum, but not anti-wraparound. I think it is
>> useful indicate anti-wraparound in logs.
>
> Yes, a bit more verbosity would be nice for that.  Using the term
> "anti-wraparound" directly in the logs makes the most sense?

I'm not particularly in love with that terminology.  I doubt that it's
very clear to the average user.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: How can we submit code patches that implement our (pending) patents?

[Bruce Momjian]

On Mon, Jul 23, 2018 at 07:32:34AM -0700, Andres Freund wrote:
> On 2018-07-23 10:27:10 -0400, Bruce Momjian wrote:
> > On Mon, Jul 23, 2018 at 07:08:32AM -0700, Andres Freund wrote:
> > > On 2018-07-23 09:56:47 -0400, Bruce Momjian wrote:
> > > > They can say whatever they want, but if they are bankrupt, what they say
> > > > doesn't matter much.  My guess is that they would have to give their
> > > > patents to some legal entity that owns them so it is shielded from
> > > > bankrupcy.
> > > 
> > > Huh? Bancruptcy doesn't simply invalidate licenses which successors then
> > > can ignore. By that logic a license to use the code, like the PG
> > > license, would be just as ineffectual
> > 
> > You are not thinking this through.
> 
> Err.
> 
> 
> > It depends if the patent owner retains some un-released/un-shared
> > rights to the patent, e.g. patent can only be used by others in
> > GPL-licensed code.
> 
> Please point me to any sort of reference that bancruptcy simply
> terminates perpetual irrevocable license agreements; that's simply not
> the case afaik. Even if the patent rights are liquidated, the new owner
> would be bound by the terms.
> Also, as I pointed out above, how would the same not be applicable to
> the code itself?

I am explaining a case where some of the patent rights are given to a
group, but some rights are retained by the company.  They cannot change
the rights given, but can change how the company-controlled rights are
used.

I am just guessing since I have not heard of any such cases, but I know
that companies have limited rights now their assets are sold.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+  Ancient Roman grave inscription +

Re: [HACKERS] possible self-deadlock window after bad ProcessStartupPacket

[Robert Haas]

On Fri, Jul 20, 2018 at 4:53 AM, Heikki Linnakangas  wrote:
> ISTM that no-one has any great ideas on what to do about the ereport() in
> quickdie(). But I think we have consensus on replacing the exit(2) calls
> with _exit(2). If we do just that, it would be better than the status quo,
> even if it doesn't completely fix the problem. This would prevent the case
> that Asim reported, for starters.

+1 for trying to improve this by using _exit rather than exit, and for
not letting the perfect be the enemy of the good.

But -1 for copying the language "if some idiot DBA sends a manual
SIGQUIT to a random backend".  I think that phrase could be deleted
from this comment -- and all of the other places where this comment
appears already today -- without losing any useful informational
content.  Or it could be rephrased to "if this process receives a
SIGQUIT".  It's just not necessary to call somebody an idiot to
communicate the point.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: How can we submit code patches that implement our (pending) patents?

[David Fetter]

On Mon, Jul 23, 2018 at 06:31:14AM -0700, Andres Freund wrote:
> On July 23, 2018 6:25:42 AM PDT, Bruce Momjian  wrote:
> >Notice this makes no mention of what happens to the patents if the
> >company goes bankrupt.  My guess is that in such a situation the
> >company
> >would have no control over who buys the patents or how they are used.
> 
> It explicitly says irrevocable and successors. Why seems squarely
> aimed at your concern. Bankruptcy wouldn't just invalidate that.

Until this has been upheld in court, it's just a vague idea.

> Similarly icenses like Apache 2 grant a perpetual and irrevocable
> patent grant for the use in the contribution.

The "upheld in court" business applies here, too.

I know it's tempting to look at formal-looking texts and infer that
they operate something vaguely like computer code, but that is
manifestly not the case.

Best,
David.
-- 
David Fetter  http://fetter.org/
Phone: +1 415 235 3778

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

Re: How can we submit code patches that implement our (pending) patents?

[Andres Freund]

On 2018-07-23 10:27:10 -0400, Bruce Momjian wrote:
> On Mon, Jul 23, 2018 at 07:08:32AM -0700, Andres Freund wrote:
> > On 2018-07-23 09:56:47 -0400, Bruce Momjian wrote:
> > > They can say whatever they want, but if they are bankrupt, what they say
> > > doesn't matter much.  My guess is that they would have to give their
> > > patents to some legal entity that owns them so it is shielded from
> > > bankrupcy.
> > 
> > Huh? Bancruptcy doesn't simply invalidate licenses which successors then
> > can ignore. By that logic a license to use the code, like the PG
> > license, would be just as ineffectual
> 
> You are not thinking this through.

Err.


> It depends if the patent owner retains some un-released/un-shared
> rights to the patent, e.g. patent can only be used by others in
> GPL-licensed code.

Please point me to any sort of reference that bancruptcy simply
terminates perpetual irrevocable license agreements; that's simply not
the case afaik. Even if the patent rights are liquidated, the new owner
would be bound by the terms.

Also, as I pointed out above, how would the same not be applicable to
the code itself?

Greetings,

Andres Freund

Re: Add SKIP LOCKED to VACUUM and ANALYZE

[Bossart, Nathan]

On 7/22/18, 10:12 PM, "Michael Paquier"  wrote:
> The refactoring for CLUSTER is pretty obvious, and makes the API a bit
> cleaner, so attached is a proposal of patch to do so.  Thoughts?

Sorry for the delay on these patches!  This is nearly identical to
what I started writing last night, so it looks good to me.

+typedef enum ClusterOption
+{
+   CLUOPT_RECHECK, /* recheck relation state */
+   CLUOPT_VERBOSE  /* print progress info */
+}  ClusterOption;

It looks like the last line here has a bit of extra whitespace
compared to the other code in parsenodes.h.

Nathan

Re: How can we submit code patches that implement our (pending) patents?

[Bruce Momjian]

On Mon, Jul 23, 2018 at 07:08:32AM -0700, Andres Freund wrote:
> On 2018-07-23 09:56:47 -0400, Bruce Momjian wrote:
> > They can say whatever they want, but if they are bankrupt, what they say
> > doesn't matter much.  My guess is that they would have to give their
> > patents to some legal entity that owns them so it is shielded from
> > bankrupcy.
> 
> Huh? Bancruptcy doesn't simply invalidate licenses which successors then
> can ignore. By that logic a license to use the code, like the PG
> license, would be just as ineffectual

You are not thinking this through.  It depends if the patent owner
retains some un-released/un-shared rights to the patent, e.g. patent can
only be used by others in GPL-licensed code.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+  Ancient Roman grave inscription +

Re: [HACKERS] logical decoding of two-phase transactions

[Andres Freund]

On 2018-07-23 19:37:46 +0530, Nikhil Sontakke wrote:
> Hi Andres,
> 
> >> > what I'm proposing is that that various catalog access functions throw a
> >> > new class of error, something like "decoding aborted transactions".
> >>
> >> When will this error be thrown by the catalog functions? How will it
> >> determine that it needs to throw this error?
> >
> > The error check would have to happen at the end of most systable_*
> > functions. They'd simply do something like
> >
> > if (decoding_in_progress_xact && TransactionIdDidAbort(xid_of_aborted))
> >ereport(ERROR, (errcode(DECODING_ABORTED_XACT), errmsg("oops")));
> >
> > i.e. check whether the transaction to be decoded still is in
> > progress. As that would happen before any potentially wrong result can
> > be returned (as the check happens at the tail end of systable_*),
> > there's no issue with wrong state in the syscache etc.
> >
> 
> Oh, ok. The systable_* functions use the passed in snapshot and return
> tuples matching to it. They do not typically have access to the
> current XID being worked upon..

That seems like quite a solvable issue, especially compared to the
locking schemes proposed.


> We can find out if the snapshot is a logical decoding one by virtue of
> its "satisfies" function pointing to HeapTupleSatisfiesHistoricMVCC.

I think we even can just do something like a global
TransactionId check_if_transaction_is_alive = InvalidTransactionId;
and just set it up during decoding. And then just check it whenever it's
not set tot InvalidTransactionId.

Greetings,

Andres Freund

Re: How can we submit code patches that implement our (pending) patents?

[Bruce Momjian]

On Mon, Jul 23, 2018 at 10:13:48AM -0400, Chapman Flack wrote:
> On 07/23/2018 10:01 AM, Bruce Momjian wrote:
> 
> > And the larger question is whether a patent free for use by software
> > under any license can be used in a defensive way.  If not, it means we
> > have no way forward here.
> 
> Isn't 'defensive', in patent-speak, used to mean 'establishing prior
> art usable to challenge future patent claims by others on the same
> technique'?
> 
> Is there any way that conditions of use, or lack of them, on an
> existing patent, would make it unusable in that context?

It doesn't have to be a patent on the same technique;  this URL was
referenced in the thread:

https://en.wikipedia.org/wiki/Defensive_termination

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+  Ancient Roman grave inscription +

Re: [HACKERS] Bug in to_timestamp().

[Alexander Korotkov]

On Mon, Jul 23, 2018 at 5:12 PM Arthur Zakirov  wrote:
> On Mon, Jul 23, 2018 at 04:30:43PM +0300, Arthur Zakirov wrote:
> > I looked for some tradeoffs of the patch. I think it could be parsing
> > strings like the following input strings:
> >
> > SELECT TO_TIMESTAMP('2011年5月1日', '-MM-DD');
> > SELECT TO_TIMESTAMP('2011y5m1d', '-MM-DD');
> >
> > HEAD extracts year, month and day from the string. But patched
> > to_timestamp() raises an error. Someone could rely on such behaviour.
> > The patch divides separator characters from letters and digits. And
> > '年' or 'y' are letters here. And so the format string doesn't match the
> > input string.
>
> Sorry, I forgot to mention that the patch can handle this by using
> different format string. You can execute:
>
> =# SELECT TO_TIMESTAMP('2011年5月1日', '年MM月DD日');
>   to_timestamp
> 
>  2011-05-01 00:00:00+04
>
> =# SELECT TO_TIMESTAMP('2011y5m1d', '"y"MM"m"DD"d"');
>   to_timestamp
> 
>  2011-05-01 00:00:00+04
>
> or:
>
> =# SELECT TO_TIMESTAMP('2011y5m1d', 'tMMtDDt');
>   to_timestamp
> 
>  2011-05-01 00:00:00+04

Thank you, Arthur.  These examples shows downside of this patch, where
users may be faced with incompatibility.  But it's good that this
situation can be handled by altering format string.  I think these
examples should be added to the documentation and highlighted in
release notes.

--
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Re: cached plans and enable_partition_pruning

[Andres Freund]

Hi,

On 2018-07-23 18:31:43 +0900, Amit Langote wrote:
> It seems that because enable_partition_pruning's value is only checked
> during planning, turning it off *after* a plan is created and cached does
> not work as expected.
> 
> create table p (a int) partition by list (a);
> create table p1 partition of p for values in (1);
> create table p1 partition of p for values in (2);
> 
> -- force a generic plan so that run-time pruning is used in the plan
> reset enable_partition_pruning;
> set plan_cache_mode to force_generic_plan;
> prepare p as select * from p where a = $1;
> 
> explain (costs off, analyze) execute p (1);
>QUERY PLAN
> 
>  Append (actual time=0.079..0.106 rows=1 loops=1)
>Subplans Removed: 1
>->  Seq Scan on p2 (actual time=0.058..0.068 rows=1 loops=1)
>  Filter: (a = $1)
>  Planning Time: 17.573 ms
>  Execution Time: 0.396 ms
> (6 rows)
> 
> set enable_partition_pruning to off;
> 
> explain (costs off, analyze) execute p (1);
>QUERY PLAN
> 
>  Append (actual time=0.108..0.135 rows=1 loops=1)
>Subplans Removed: 1
>->  Seq Scan on p2 (actual time=0.017..0.028 rows=1 loops=1)
>  Filter: (a = $1)
>  Planning Time: 0.042 ms
>  Execution Time: 0.399 ms
> (6 rows)
> 
> Pruning still occurs, whereas one would expect it not to, because the plan
> (the Append node) contains run-time pruning information, which was
> initialized because enable_partition_pruning was turned on when the plan
> was created.
> 
> Should we check its value during execution too, as done in the attached?

I think it's correct to check the plan time value, rather than the
execution time value. Other enable_* GUCs also take effect there, and I
don't see a problem with that?

Greetings,

Andres Freund

Re: How can we submit code patches that implement our (pending) patents?

[Chapman Flack]

On 07/23/2018 10:01 AM, Bruce Momjian wrote:

> And the larger question is whether a patent free for use by software
> under any license can be used in a defensive way.  If not, it means we
> have no way forward here.

Isn't 'defensive', in patent-speak, used to mean 'establishing prior
art usable to challenge future patent claims by others on the same
technique'?

Is there any way that conditions of use, or lack of them, on an
existing patent, would make it unusable in that context?

-Chap

Re: [HACKERS] Bug in to_timestamp().

[Arthur Zakirov]

On Mon, Jul 23, 2018 at 04:30:43PM +0300, Arthur Zakirov wrote:
> I looked for some tradeoffs of the patch. I think it could be parsing
> strings like the following input strings:
> 
> SELECT TO_TIMESTAMP('2011年5月1日', '-MM-DD');
> SELECT TO_TIMESTAMP('2011y5m1d', '-MM-DD');
> 
> HEAD extracts year, month and day from the string. But patched
> to_timestamp() raises an error. Someone could rely on such behaviour.
> The patch divides separator characters from letters and digits. And
> '年' or 'y' are letters here. And so the format string doesn't match the
> input string.

Sorry, I forgot to mention that the patch can handle this by using
different format string. You can execute:

=# SELECT TO_TIMESTAMP('2011年5月1日', '年MM月DD日');
  to_timestamp  

 2011-05-01 00:00:00+04

=# SELECT TO_TIMESTAMP('2011y5m1d', '"y"MM"m"DD"d"');
  to_timestamp  

 2011-05-01 00:00:00+04

or:

=# SELECT TO_TIMESTAMP('2011y5m1d', 'tMMtDDt');
  to_timestamp  

 2011-05-01 00:00:00+04

-- 
Arthur Zakirov
Postgres Professional: http://www.postgrespro.com
Russian Postgres Company

Re: How can we submit code patches that implement our (pending) patents?

[Andres Freund]

On 2018-07-23 09:56:47 -0400, Bruce Momjian wrote:
> On Mon, Jul 23, 2018 at 06:31:14AM -0700, Andres Freund wrote:
> > On July 23, 2018 6:25:42 AM PDT, Bruce Momjian  wrote:
> > >On Mon, Jul 9, 2018 at 08:29:08AM +, Tsunakawa, Takayuki wrote:
> > >> Thank you for supporting me, Andres.  And please don't mind, David. 
> > >I
> > >> don't think you are attacking me.  I understand your concern and that
> > >> you are also trying to protect PostgreSQL.
> > >>
> > >>   On the other hand, I think TPL seems less defensive.  I read
> > >>   in some report that Apache License and some other open source
> > >>   licenses were created partly due to lack of patent description
> > >>   in BSD and GPLv2.
> > >>
> > >> How can we assure you?  How about attaching something like the
> > >> following to relevant patches or on our web site?
> > >>
> > >> [Excerpt from Red Hat Patent Promise] Red Hat intends Our Promise to
> > >> be irrevocable (except as stated herein), and binding and enforceable
> > >> against Red Hat and assignees of, or successors to, Red Hat’s
> > >> patents (and any patents directly or indirectly issuing from Red
> > >> Hat’s patent applications). As part of Our Promise, if Red Hat
> > >> sells, exclusively licenses, or otherwise assigns or transfers
> > >> patents or patent applications to a party, we will require the party
> > >> to agree in writing to be bound to Our Promise for those patents
> > >> and for patents directly or indirectly issuing on those patent
> > >> applications. We will also require the party to agree in writing to
> > >so
> > >> bind its own assignees, transferees, and exclusive licensees.
> > >
> > >Notice this makes no mention of what happens to the patents if the
> > >company goes bankrupt.  My guess is that in such a situation the
> > >company
> > >would have no control over who buys the patents or how they are used.
> > 
> > It explicitly says irrevocable and successors. Why seems squarely aimed at 
> > your concern. Bankruptcy wouldn't just invalidate that.
> 
> They can say whatever they want, but if they are bankrupt, what they say
> doesn't matter much.  My guess is that they would have to give their
> patents to some legal entity that owns them so it is shielded from
> bankrupcy.

Huh? Bancruptcy doesn't simply invalidate licenses which successors then
can ignore. By that logic a license to use the code, like the PG
license, would be just as ineffectual

Greetings,

Andres Freund

Re: [HACKERS] logical decoding of two-phase transactions

[Nikhil Sontakke]

Hi Andres,

>> > what I'm proposing is that that various catalog access functions throw a
>> > new class of error, something like "decoding aborted transactions".
>>
>> When will this error be thrown by the catalog functions? How will it
>> determine that it needs to throw this error?
>
> The error check would have to happen at the end of most systable_*
> functions. They'd simply do something like
>
> if (decoding_in_progress_xact && TransactionIdDidAbort(xid_of_aborted))
>ereport(ERROR, (errcode(DECODING_ABORTED_XACT), errmsg("oops")));
>
> i.e. check whether the transaction to be decoded still is in
> progress. As that would happen before any potentially wrong result can
> be returned (as the check happens at the tail end of systable_*),
> there's no issue with wrong state in the syscache etc.
>

Oh, ok. The systable_* functions use the passed in snapshot and return
tuples matching to it. They do not typically have access to the
current XID being worked upon..

We can find out if the snapshot is a logical decoding one by virtue of
its "satisfies" function pointing to HeapTupleSatisfiesHistoricMVCC.

>
>> The catalog scan will NOT error out but will return metadata which
>> causes the insert-decoding change apply callback to error out.
>
> Why would it not throw an error?
>

In your scheme, it will throw an error, indeed. We'd need to make the
"being-currently-decoded-XID" visible to these systable_* functions
and then this scheme will work.

Regards,
Nikhils

> Greetings,
>
> Andres Freund



-- 
 Nikhil Sontakke   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Log query parameters for terminated execute

[Tom Lane]

Sergei Kornilov  writes:
> Please test with logging command tag %i in log_line_prefix. Extended protocol 
> has three different messages, each can be canceled by timeout. But here is 
> completely no parameters in PARSE and i did not change BIND in first patch.

This patch scares me to death.  It risks calling user-defined I/O
functions in all sorts of weird places, particularly outside transactions,
or in already-failed transactions, or with no ActiveSnapshot.

Separately from that concern: it appears to result in a substantial
degradation of existing functionality in the places where you did
s/errdetail/errdetail_log/.  What was the reason for that?

regards, tom lane

Re: How can we submit code patches that implement our (pending) patents?

[Bruce Momjian]

On Mon, Jul 23, 2018 at 09:53:26AM -0400, Bruce Momjian wrote:
> On Tue, Jul 10, 2018 at 09:47:09AM -0400, Tom Lane wrote:
> > The core team has considered this matter, and has concluded that it's
> > time to establish a firm project policy that we will not accept any code
> > that is known to be patent-encumbered.  The long-term legal risks and
> > complications involved in doing that seem insurmountable, given the
> > community's amorphous legal nature and the existing Postgres license
> > wording (neither of which are open for negotiation here).  Furthermore,
> > Postgres has always been very friendly to creation of closed-source
> > derivatives, but it's hard to see how inclusion of patented code would
> > not cause serious problems for those.  The potential benefits of
> > accepting patented code just don't seem to justify trying to navigate
> > these hazards.
> 
> Just to add a summary to this, any patent assignment to Postgres would
> have to allow free patent use for all code, under _any_ license. This
> effectively makes the patent useless, except for defensive use, even for
> the patent owner.  I think everyone here agrees on this.
> 
> The open question is whether it is useful for the PGDG to accept such
> patents for defensive use.  There are practical problems with this (PGDG
> is not a legal entity) and operational complexity too.  The core team's
> feeling is that it not worth it, but that discussion can be re-litigated
> on this email list if desired.  The discussion would have to relate to
> such patents in general, not to the specific Fujitsu proposal.  If it
> was determined that such defensive patents were desired, we can then
> consider the Fujitsu proposal.

And the larger question is whether a patent free for use by software
under any license can be used in a defensive way.  If not, it means we
have no way forward here.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+  Ancient Roman grave inscription +

Re: How can we submit code patches that implement our (pending) patents?

[Bruce Momjian]

On Tue, Jul 10, 2018 at 08:20:53AM +, Tsunakawa, Takayuki wrote:
> > One possible answer is that you wouldn't.  But that might reduce the
> > size of the community, or lead to a fork.
>
> Yes, that's one unfortunate future, which I don't want to happen
> of course.  I believe PostgreSQL should accept patent for further
> evolution, because PostgreSQL is now a popular, influential software
> that many organizations want to join.

Why did you say this last sentence?

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+  Ancient Roman grave inscription +

Re: [Proposal] Add accumulated statistics for wait event

[Tom Lane]

Michael Paquier  writes:
> This does not need a configure switch.

It probably is there because the OP realizes that most people wouldn't
accept having this code compiled in.

> What's the performance penalty?  I am pretty sure that this is
> measurable as wait events are stored for a backend for each I/O
> operation as well, and you are calling a C routine within an inlined
> function which is designed to be light-weight, doing only a four-byte
> atomic operation.

On machines with slow gettimeofday(), I suspect the cost of this
patch would be staggering.  Even with relatively fast gettimeofday,
it doesn't look acceptable for calls in hot code paths (for instance,
lwlock.c).

A bigger problem is that it breaks stuff.  There are countless
calls to pgstat_report_wait_start/pgstat_report_wait_end that
assume they have no side-effects (for example, on errno) and
can never fail.  I wouldn't trust GetCurrentTimestamp() for either.
If the report_wait calls can't be dropped into code with *complete*
certainty that they're safe, that's a big cost.

Why exactly is this insisting on logging timestamps and not,
say, just incrementing a counter?  I think doing it like this
is almost certain to end in rejection.

regards, tom lane