Re: [HACKERS] WIP Patch: Use sortedness of CSV foreign tables for query planning

2012-08-07 Thread Robert Haas
On Tue, Aug 7, 2012 at 2:02 AM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: I think that optimizations like this are going to be essential for things like pgsql_fdw (or other_rdms_fdw). Despite the thorny semantic issues, we're just not going to be able to get around it. There will even

Re: [HACKERS] WIP Patch: Use sortedness of CSV foreign tables for query planning

2012-08-06 Thread Robert Haas
On Sun, Aug 5, 2012 at 10:41 PM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: I think file_fdw is useful for managing log files such as PG CSV logs. Since often, such files are sorted by timestamp, I think the patch can improve the performance of log analysis, though I have to admit my

Re: [HACKERS] WIP Patch: Use sortedness of CSV foreign tables for query planning

2012-08-06 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Sun, Aug 5, 2012 at 10:41 PM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: I think file_fdw is useful for managing log files such as PG CSV logs. Since often, such files are sorted by timestamp, I think the patch can improve the performance of

Re: [HACKERS] WIP Patch: Use sortedness of CSV foreign tables for query planning

2012-08-06 Thread Robert Haas
On Mon, Aug 6, 2012 at 10:33 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Sun, Aug 5, 2012 at 10:41 PM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: I think file_fdw is useful for managing log files such as PG CSV logs. Since often, such files are

Re: [HACKERS] WIP Patch: Use sortedness of CSV foreign tables for query planning

2012-08-06 Thread Etsuro Fujita
From: Robert Haas [mailto:robertmh...@gmail.com] On Mon, Aug 6, 2012 at 10:33 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Sun, Aug 5, 2012 at 10:41 PM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: I think file_fdw is useful for managing log

Re: [HACKERS] WIP Patch: Use sortedness of CSV foreign tables for query planning

2012-08-05 Thread Etsuro Fujita
Hi Robert, From: Robert Haas [mailto:robertmh...@gmail.com] On Thu, Aug 2, 2012 at 7:01 AM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: The following is a comment at fileGetForeignPaths() in contrib/file_fdw.c: /* * If data file was sorted, and we knew it somehow, we

Re: [HACKERS] WIP Patch: Use sortedness of CSV foreign tables for query planning

2012-08-03 Thread Robert Haas
On Thu, Aug 2, 2012 at 7:01 AM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: The following is a comment at fileGetForeignPaths() in contrib/file_fdw.c: /* * If data file was sorted, and we knew it somehow, we could insert * appropriate pathkeys into the ForeignPath node to

[HACKERS] WIP Patch: Use sortedness of CSV foreign tables for query planning

2012-08-02 Thread Etsuro Fujita
The following is a comment at fileGetForeignPaths() in contrib/file_fdw.c: /* * If data file was sorted, and we knew it somehow, we could insert * appropriate pathkeys into the ForeignPath node to tell the planner * that. */ To do this, I would like to propose new generic

Re: [HACKERS] WIP Patch: Use sortedness of CSV foreign tables for query planning

2012-08-02 Thread Kevin Grittner
Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: I would like to propose single-column sortedness options and insert appropriate pathkeys into the ForeignPath node based on these information: sorted: Boolean option to specify whether data file is sorted by a column. key: Specifies

Re: [HACKERS] WIP Patch: Use sortedness of CSV foreign tables for query planning

2012-08-02 Thread Etsuro Fujita
Hi Kevin, I would like to propose single-column sortedness options and insert appropriate pathkeys into the ForeignPath node based on these information: For character-based columns, don't we need to know the collation? I forgot to add the collate option. I'll add it at the next version