Jason Tackaberry wrote: > On Tue, 2007-02-27 at 17:34 +0100, Dirk Meyer wrote: >> For expat I created a small tree. So the while file is in memory at >> the end. This is why I need 200 MB. See attached file. I wanted it to >> be as close a a DOM as we need it. > > I see, ok. But this isn't really necessary in practice. > >> The problem is that the expat or SAX parser block until they are >> done. > > Well, ok, but this is the case too with libxml2.parseFile. > >> To avoid blocking we use a thread. But sqlite is not thread save >> and we need to run add_program in the main loop. So we have several >> choices: >> >> 1. Parse the while file and add later. This requires about 200 MB for >> your TV.xml file. >> >> 2. Jump to mainloop for each program we parse. >> >> IMHO 2. is more or less a good idea. The main loop 'polls' a variable >> the thread is writing. > > I agree, 2 is better for both memory usage and net performance (it's > better to continue to parse the XML file in a thread while waiting on > I/O for the sqlite db), and it's not hard to implement. > >> Thread: >> >> def end_element(name): >> create Program or channel object >> add object to global queue >> >> MainLoop: >> >> while global queue: >> prog = global queue.pop(0) >> add to epg (prog) > > Something like that. Your actual code won't work, since if we parse XML > faster than we add programs to the EPG (very likely) then your > mainthread loop will loop over all programs the first time, effectively > blocking the rest of the app. So better instead to quit after say some > period of time has elapsed. Maybe 0.05s > > def dispatcher_handler(): > t0 = time.time() > while global_queue and time.time() - t0 < 0.05: > add_to_epg(global_queue.pop(0)) > > >> This should keep the memory usage low, the thread could also block for >> some time if len(queue) is greater x to avoid using too much >> memory. > > This sounds like a good idea, yes. > >> This logic could also be shared between zaptoit, xmltv and >> epgdata.com parser. > > Agreed.
I quick look at the code shows me that we need the result of add_channel but not of add_program. This means when we call add_channel from a thread the thread blocks until add_channel is called from main (MainThreadCallback). This is slow because we need to toggle all the time but the number of channels is small compared to the number of programs. | add_channel = MainThreadCallback(epg.add_channel) or even simpler: | @kaa.notifier.execute_in_mainloop() | def add_channel(self, tuner_id, name, long_name): And a new add_program will work in a queue: | def add_program(self, channel_db_id, start, stop, title, **attributes): | if not kaa.notifier.is_mainthread: | while len(program_queue) > 100: | time.sleep(0.1) | program_queue.append(...) | wakeup(program_queue_poll) | return | current code Dischi -- "There is no reason for any individual to have a computer in their home." (Ken Olsen, Präsident von DEC, 1977)
pgpTSsLCf68dy.pgp
Description: PGP signature
------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________ Freevo-devel mailing list Freevo-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freevo-devel
