On Tue, 2007-02-27 at 17:34 +0100, Dirk Meyer wrote:
> For expat I created a small tree. So the while file is in memory at
> the end. This is why I need 200 MB. See attached file. I wanted it to
> be as close a a DOM as we need it.
I see, ok. But this isn't really necessary in practice.
> The problem is that the expat or SAX parser block until they are
> done.
Well, ok, but this is the case too with libxml2.parseFile.
> To avoid blocking we use a thread. But sqlite is not thread save
> and we need to run add_program in the main loop. So we have several
> choices:
>
> 1. Parse the while file and add later. This requires about 200 MB for
> your TV.xml file.
>
> 2. Jump to mainloop for each program we parse.
>
> IMHO 2. is more or less a good idea. The main loop 'polls' a variable
> the thread is writing.
I agree, 2 is better for both memory usage and net performance (it's
better to continue to parse the XML file in a thread while waiting on
I/O for the sqlite db), and it's not hard to implement.
> Thread:
>
> def end_element(name):
> create Program or channel object
> add object to global queue
>
> MainLoop:
>
> while global queue:
> prog = global queue.pop(0)
> add to epg (prog)
Something like that. Your actual code won't work, since if we parse XML
faster than we add programs to the EPG (very likely) then your
mainthread loop will loop over all programs the first time, effectively
blocking the rest of the app. So better instead to quit after say some
period of time has elapsed. Maybe 0.05s
def dispatcher_handler():
t0 = time.time()
while global_queue and time.time() - t0 < 0.05:
add_to_epg(global_queue.pop(0))
> This should keep the memory usage low, the thread could also block for
> some time if len(queue) is greater x to avoid using too much
> memory.
This sounds like a good idea, yes.
> This logic could also be shared between zaptoit, xmltv and
> epgdata.com parser.
Agreed.
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Freevo-devel mailing list
Freevo-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freevo-devel
