Re: [PyGreSQL] pg.DB() pseudo-cursor to reduce RAM use

2019-04-21 Thread Christoph Zwerschke
For the records, this feature ("iterators as pseudo-cursors") has now finally been committed in r978 and will be available in PyGres 5.1. Thanks to Justin for the discussion and implementation. I have only made a few minor changes and added some missing methods for dicts and named tuples,

Re: [PyGreSQL] pg.DB() pseudo-cursor to reduce RAM use

2018-04-22 Thread Justin Pryzby
On Sun, Apr 22, 2018 at 02:01:59PM +0200, Christoph Zwerschke wrote: > Am 26.03.2018 um 22:01 schrieb Justin Pryzby: > > I have one patch ready for evaluation. > > > > 1: implement fake/pseudo cursors in pg (same as what's in pgdb); > > I proposed not to support any "move()" function in pg, since

Re: [PyGreSQL] pg.DB() pseudo-cursor to reduce RAM use

2018-04-22 Thread Christoph Zwerschke
Am 22.04.2018 um 14:01 schrieb Christoph Zwerschke: > * Would it make sense to make the result row type configurable by > providing get/set_result_type() methods on the module, connection and > query level? We could then get rid of the 6 different accessor methods > for the result result

Re: [PyGreSQL] pg.DB() pseudo-cursor to reduce RAM use

2018-04-22 Thread Christoph Zwerschke
Am 26.03.2018 um 22:01 schrieb Justin Pryzby: > I have one patch ready for evaluation. > > 1: implement fake/pseudo cursors in pg (same as what's in pgdb); > I proposed not to support any "move()" function in pg, since > there's no "fetch" (besides getItem which is newly introduced > at your

Re: [PyGreSQL] pg.DB() pseudo-cursor to reduce RAM use

2018-03-26 Thread Christoph Zwerschke
Am 26.03.2018 um 22:01 schrieb Justin Pryzby: > I have one patch ready for evaluation. > > 1: implement fake/pseudo cursors in pg (same as what's in pgdb); I proposed not > to support any "move()" function in pg, since there's no "fetch" (besides > getItem which is newly introduced at your

Re: [PyGreSQL] pg.DB() pseudo-cursor to reduce RAM use

2018-03-26 Thread Justin Pryzby
I have one patch ready for evaluation. 1: implement fake/pseudo cursors in pg (same as what's in pgdb); I proposed not to support any "move()" function in pg, since there's no "fetch" (besides getItem which is newly introduced at your suggestion). This patch is solid. I also have additional

Re: [PyGreSQL] pg.DB() pseudo-cursor to reduce RAM use

2018-03-18 Thread Justin Pryzby
I wanted to pick this up and finally finish it.. Some opened question in my mind: - should we implement real serverside cursors instead? Or is that off the table for "classic" interface for some reason? - Or should we implement enough of serverside cursors to be sure of what interfaces

Re: [PyGreSQL] pg.DB() pseudo-cursor to reduce RAM use

2017-07-09 Thread Justin Pryzby
On Thu, Jul 06, 2017 at 02:23:37PM +0200, Christoph Zwerschke wrote: > Am 05.07.2017 um 23:12 schrieb Justin Pryzby: > >I'm hoping for some feedback on this - if the interface seems okay and the > >implementation otherwise fine, I can clean up some by pulling > >out the duplicated code from

Re: [PyGreSQL] pg.DB() pseudo-cursor to reduce RAM use

2017-07-09 Thread Christoph Zwerschke
Am 09.07.2017 um 19:07 schrieb Justin Pryzby: Could you elaborate on that ? I would like the idea of a standard function, returning its length, the same thing as ntuples(), but how would one call it ? It seems len() function only works for sequences and mappings. If you implement a __len__

Re: [PyGreSQL] pg.DB() pseudo-cursor to reduce RAM use

2017-07-09 Thread Justin Pryzby
On Thu, Jul 06, 2017 at 02:23:37PM +0200, Christoph Zwerschke wrote: > Am 05.07.2017 um 23:12 schrieb Justin Pryzby: > >I'm hoping for some feedback on this - if the interface seems okay and the > >implementation otherwise fine, I can clean up some by pulling > >out the duplicated code from

Re: [PyGreSQL] pg.DB() pseudo-cursor to reduce RAM use

2017-07-06 Thread Christoph Zwerschke
Am 05.07.2017 um 23:12 schrieb Justin Pryzby: I'm hoping for some feedback on this - if the interface seems okay and the implementation otherwise fine, I can clean up some by pulling out the duplicated code from queryGet/Dict/Result and queryNext(). Hi Justin, some quick feedback: * The

Re: [PyGreSQL] pg.DB() pseudo-cursor to reduce RAM use

2017-07-05 Thread Justin Pryzby
I'm hoping for some feedback on this - if the interface seems okay and the implementation otherwise fine, I can clean up some by pulling out the duplicated code from queryGet/Dict/Result and queryNext(). On Mon, Jun 05, 2017 at 11:28:29AM -0500, Justin Pryzby wrote: > We've run into the problem

Re: [PyGreSQL] pg.DB() pseudo-cursor to reduce RAM use

2017-06-05 Thread Christoph Zwerschke
Hi Justin, thanks for your contribution. I'll look at this when I find some. Might take 2 or 3 weeks though.-- Chris___ PyGreSQL mailing list PyGreSQL@vex.net https://mail.vex.net/mailman/listinfo.cgi/pygresql

[PyGreSQL] pg.DB() pseudo-cursor to reduce RAM use

2017-06-05 Thread Justin Pryzby
We've run into the problem several times where our customers run a large but not unreasonable report returning 100k of rows. Most of the RAM is spent after .query(), and during .getresult(), converting many C arrays to python objects. Here's a recent example: 2017-05-31 17:01:22 starting up