Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-11-03 Thread Dimitri Fontaine
Le vendredi 31 octobre 2008, Tom Lane a écrit : With the attached patch, SQL functions support returning the results of INSERT/UPDATE/DELETE RETURNING clauses. Thanks for your work and for having considered user whining in-passing! :) -- dim signature.asc Description: This is a digitally

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-31 Thread Pavel Stehule
2008/10/30 Robert Haas [EMAIL PROTECTED]: With session variables we could implement srf function in plpgsql like current C srf function. Like create or replace function foo() returns record as $$ #option with_srf_context(datatype of srf context) begin return row(...); end; $$

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-31 Thread Tom Lane
Pavel Stehule [EMAIL PROTECTED] writes: RETURN QUERY should be implemented for lazy execution model. And it should be fast and not to much dificult. Really? Consider what happens if it's inside a loop, or an exception block, or any other nesting construct. regards, tom

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-31 Thread Pavel Stehule
2008/10/31 Tom Lane [EMAIL PROTECTED]: Pavel Stehule [EMAIL PROTECTED] writes: RETURN QUERY should be implemented for lazy execution model. And it should be fast and not to much dificult. Really? Consider what happens if it's inside a loop, or an exception block, or any other nesting

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-31 Thread Tom Lane
With the attached patch, SQL functions support returning the results of INSERT/UPDATE/DELETE RETURNING clauses. An INSERT/UPDATE/DELETE statement is always executed to completion before returning (including firing any related triggers or rules), so we always materialize the RETURNING output.

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-30 Thread Tom Lane
I wrote: I'm currently going to have a look at just what it would take to support both lazy and eager evaluation in functions.c (independently of what syntax, if any, we settle on to expose the choice to the user). If it's either really awful or really easy we should know that before arguing

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-30 Thread Robert Haas
With session variables we could implement srf function in plpgsql like current C srf function. Like create or replace function foo() returns record as $$ #option with_srf_context(datatype of srf context) begin return row(...); end; $$ language plpgsql; Oh, sure - but what you can

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-29 Thread Dimitri Fontaine
Le mardi 28 octobre 2008, Pavel Stehule a écrit : 2008/10/28 Dimitri Fontaine [EMAIL PROTECTED]: Hi, In the python language, functions that lazily return collections are called generators and use the yield keyword instead of return.

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-29 Thread Robert Haas
So my idea would be to have the SQL function behavior choose to return values either via tuplestore or via value-per-call, depending on the user setting generator or lazy. Done this way, the user could also choose for the function to be lazy or to use a tuplestore whatever the language in

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-29 Thread Tom Lane
Dimitri Fontaine [EMAIL PROTECTED] writes: Done this way, the user could also choose for the function to be lazy or to use a tuplestore whatever the language in which it's written. The odds of this ever happening for any of the PLs are not distinguishable from zero. It might be nice to have,

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-29 Thread Dimitri Fontaine
Le mercredi 29 octobre 2008, Tom Lane a écrit : So the fact that it's possible for SQL-language functions is an idiosyncrasy of that language, not something we should cram into the general CREATE FUNCTION syntax in the vain hope that having syntax might cause an implementation to appear

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-29 Thread Tom Lane
Dimitri Fontaine [EMAIL PROTECTED] writes: Le mercredi 29 octobre 2008, Tom Lane a écrit : So the fact that it's possible for SQL-language functions is an idiosyncrasy of that language, not something we should cram into the general CREATE FUNCTION syntax in the vain hope that having syntax

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-29 Thread Dimitri Fontaine
Le mercredi 29 octobre 2008, Tom Lane a écrit : Well, call-per-value is *necessary* for lazy evaluation, but it's not *sufficient*. You need a function implementation that can suspend and resume execution, and that's difficult in general. Ok, I think I begin to understand how things are tied

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-29 Thread Tom Lane
Dimitri Fontaine [EMAIL PROTECTED] writes: And I fail to see how the user would control which behavior will get chosen, Oh, I'm sorry, I didn't realize you misunderstood my syntax example. I was suggesting that the SQL function manager recognize some optional non-SQL keywords at the start of a

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-29 Thread Dimitri Fontaine
Le mercredi 29 octobre 2008, Tom Lane a écrit : Now of course the bigger problem with either this syntax or yours is that attaching such a property to a function is arguably the Wrong Thing in the first place. Which one is the best way is likely to depend on the calling query more than it

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-29 Thread Tom Lane
Dimitri Fontaine [EMAIL PROTECTED] writes: Le mercredi 29 octobre 2008, Tom Lane a écrit : However, I see no solution to that problem except function inlining; and if the function gets inlined then all this discussion is moot anyhow. How to inline PLs functions? All of this is

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-29 Thread Hannu Krosing
On Wed, 2008-10-29 at 11:58 -0400, Tom Lane wrote: Dimitri Fontaine [EMAIL PROTECTED] writes: And I fail to see how the user would control which behavior will get chosen, Oh, I'm sorry, I didn't realize you misunderstood my syntax example. I was suggesting that the SQL function manager

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-29 Thread Robert Haas
All of this is pie-in-the-sky for PL functions, and I think properly so: the whole reason for supporting PLs is to enable doing things that SQL does poorly or not at all. So expecting SQL to interoperate very closely with them seems impossible, or at least unreasonably limiting. The real

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-29 Thread Pavel Stehule
2008/10/30 Robert Haas [EMAIL PROTECTED]: All of this is pie-in-the-sky for PL functions, and I think properly so: the whole reason for supporting PLs is to enable doing things that SQL does poorly or not at all. So expecting SQL to interoperate very closely with them seems impossible, or at

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-28 Thread Gregory Stark
Tom Lane [EMAIL PROTECTED] writes: Robert Haas [EMAIL PROTECTED] writes: I'm pretty excited by that example. LIMIT/OFFSET is really useful as a way of paginating query results for display on a web page (show results 1-100, 101-200, etc), and I use it on potentially expensive SRFs just as I

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-28 Thread Simon Riggs
On Sun, 2008-10-26 at 21:49 -0400, Tom Lane wrote: So I'm concluding that we can easily afford to switch to tuplestore-always operation, especially if we are willing to put any effort into tuplestore optimization. (I note that the current tuplestore code writes 24 bytes per row for this

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-28 Thread Tom Lane
Gregory Stark [EMAIL PROTECTED] writes: Tom Lane [EMAIL PROTECTED] writes: I suspect it doesn't help you as much as you think. It's always been the case that SRFs in FROM-items are fed through a tuplestore, and so are plpgsql SRF results. I always thought we considered that a bug though.

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-28 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: On Sun, 2008-10-26 at 21:49 -0400, Tom Lane wrote: So I'm concluding that we can easily afford to switch to tuplestore-always operation, especially if we are willing to put any effort into tuplestore optimization. (I note that the current tuplestore code

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-28 Thread Kenneth Marshall
On Tue, Oct 28, 2008 at 09:28:38AM -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: On Sun, 2008-10-26 at 21:49 -0400, Tom Lane wrote: So I'm concluding that we can easily afford to switch to tuplestore-always operation, especially if we are willing to put any effort into

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-28 Thread Robert Haas
I always thought we considered that a bug though. It sure would be nice if we could generate results as needed instead of having to generate them in advance and store all of them. I suppose, but short of a fundamental rethink of how PL functions work that's not going to happen. There's

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-28 Thread Dimitri Fontaine
Hi, In the python language, functions that lazily return collections are called generators and use the yield keyword instead of return. http://www.python.org/doc/2.5.2/tut/node11.html#SECTION0011100 Maybe having such a concept in PostgreSQL would allow the user to choose

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-28 Thread Pavel Stehule
2008/10/28 Dimitri Fontaine [EMAIL PROTECTED]: Hi, In the python language, functions that lazily return collections are called generators and use the yield keyword instead of return. http://www.python.org/doc/2.5.2/tut/node11.html#SECTION0011100 Maybe having such a concept

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-27 Thread Martijn van Oosterhout
On Sun, Oct 26, 2008 at 09:49:49PM -0400, Tom Lane wrote: So I'm concluding that we can easily afford to switch to tuplestore-always operation, especially if we are willing to put any effort into tuplestore optimization. (I note that the current tuplestore code writes 24 bytes per row for

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-27 Thread Tom Lane
Martijn van Oosterhout [EMAIL PROTECTED] writes: On Sun, Oct 26, 2008 at 09:49:49PM -0400, Tom Lane wrote: So I'm concluding that we can easily afford to switch to tuplestore-always operation, especially if we are willing to put any effort into tuplestore optimization. I thought that the bad

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-27 Thread Robert Haas
I thought that the bad case for a tuplestore was if the set returning function was expensive and the user used it with a LIMIT clause. In the tuplestore case you evaluate everything then throw it away. I'm not terribly excited by that example --- but in any case, the real solution to any

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-27 Thread Tom Lane
Robert Haas [EMAIL PROTECTED] writes: I thought that the bad case for a tuplestore was if the set returning function was expensive and the user used it with a LIMIT clause. In the tuplestore case you evaluate everything then throw it away. I'm not terribly excited by that example --- but in

[HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-26 Thread Tom Lane
We have an open TODO item to support SQL-language functions that return the output of a RETURNING clause attached to an INSERT/UPDATE/DELETE query within the function. This is something that was left undone in the 8.2 development cycle after this thread analyzed the problem: