I'm posting this here as it can be useful for other users. Thanks Jacob
for reporting this workaround.
Francesc
---------- Missatge transmès ----------
Subject: Re: Complex query does not compile
Date: Thursday 02 April 2009
From: Jacob Quinn Shenker <jqshen...@gmail.com>
To: Francesc Alted <fal...@pytables.org>
Hi Francesc,
I'm subscribed now! :-)
So, yes, it does seem to be an issue with Python itself, which is very
strange. This query is not really all that complex, so it's ridiculous
that Python chokes. Regardless, changing #define MAXSTACK 1500 to
10000 in parser.c and recompiling Python fixes this.
Thanks!
~ Jacob
A Monday 30 March 2009, Francesc Alted escrigué:
> Hi Jacob,
>
> First of all, please remember to subscribe to this list before to
> post. If not, you are risking that your messages would get lost in
> the middle of tons of spam that we all (not only me) are
> unfortunately receiving.
>
> A Sunday 29 March 2009, escriguéreu:
> > Subject: Complex query does not compile
> > From: Jacob Quinn Shenker <jqshen...@gmail.com>
> > To: pytables-users@lists.sourceforge.net
> > Date: 09-03-29 17:58:51
> >
> > Hi,
> >
> > I'm evaluating PyTables Pro at the moment, and in my testing I find
> > that it cannot handle very complex queries. Unfortunately, this is
> > what my application requires. I have a table sorted on-disk on a
> > CSI index ('IPIX'), and I need to select many intervals on it.
> > Unfortunately, Numexpr complains:
> >
> > s_push: parser stack overflow
> > Traceback (most recent call last):
> > File "bench_table.py", line 39, in <module>
> > main()
> > File "bench_table.py", line 27, in main
> > coords = table.getWhereList(q, {})
> > File "/u/gl/jshenker/lib/python/tables/table.py", line 1309, in
> > getWhereList self._where(condition, condvars, start, stop, step) ]
> > File "/u/gl/jshenker/lib/python/tables/table.py", line 1218, in
> > _where self, compiled, condvars, start, stop, step)
> > File "/u/gl/jshenker/lib/python/tables/_table_pro.py", line 202,
> > in _table__whereIndexed
> > chunkmap = numexpr.evaluate(strexpr, cmvars)
> > File "/u/gl/jshenker/lib/python/tables/numexpr/compiler.py", line
> > 590, in evaluate
> > _names_cache[expr_key] = getExprNames(ex, context)
> > File "/u/gl/jshenker/lib/python/tables/numexpr/compiler.py", line
> > 565, in getExprNames
> > ex = stringToExpression(text, {}, context)
> > File "/u/gl/jshenker/lib/python/tables/numexpr/compiler.py", line
> > 198, in stringToExpression
> > c = compile(s, '<expr>', 'eval')
> > MemoryError
>
> That's curious. The offending function, `compile()`, is one that
> belongs to the Python core, and it seems to work well on my machine
> for arbitrarily large expressions:
>
> In [43]: l = ["((IPIX >= %d) & (IPIX <= %d)) | " % (i, i+1) for i in
> range(1000)]
> In [44]: s = "".join(l)[:-3]
> In [45]: compile(s, '<expr>', 'eval')
> Out[45]: <code object <module> at 0x8ac2530, file "<expr>", line 1>
>
> so, I suppose that the problem is in another place than Python's
> `compile()` function. Which version of Python and operating system
> are you working with? And, how many RAM do you have available?
>
> > I have two questions:
> > 1) How do I get this to work?
>
> So as to answer this, it would help if you can send a small script
> that reproduces the problem.
>
> > 2) Is there a better/faster way to do this selection of many
> > intervals on indices with PyTables?
>
> Well, you may want to try to do the 'and' selections one by one, and
> then join all the results together. Something like:
>
> l = []
> for l1, l2 in zip(lim_inf, lim_sup):
> l.append([r[:] for r in tbl.where("(IPIX >= l1) & (IPIX <= l2)")])
>
> `lim_inf` and `lim_sup` being sequences like:
>
> lim1 = [4, 3, ...]
> lim2 = [10, 12, ...]
>
> and in `l` you would have the results you want. Provided that
> expression above can be cached, this should be reasonably fast.
> Would that work with your setup?
>
> Cheers,
--
Francesc Alted
"One would expect people to feel threatened by the 'giant
brains or machines that think'. In fact, the fightening
computer becomes less frightening if it is used only to
simulate a familiar noncomputer."
-- Edsger W. Dykstra
"On the cruelty of really teaching computer science"
------------------------------------------------------------------------------
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
