Re: [Mixxx-devel] Library scanner refactoring
Hi Nazar sorry for the late answer. Dividing the library scanning into smaller chunks sounds like a good idea. Did you check what others (Amarok,Clementine,Exaile,Banshee,...) do to handle a complete library rescan? I also have some questions about the implementation. What is your DBAccessor? Will this thing life in a separate thread? Will this be the only class that mixxx later uses for database access and by that be replacing direct access to the DAO's that we have all over mixxx? Also how far are you writing test functions for the DAO's? I might want to copy from them :-) best Max On Fri, Jul 12, 2013 at 03:41:05PM -0700, troyane wrote: As you may know, my name is Nazar Gerasymchuk and I’m working on refactoring of library concurrency. All of my work is related with access to database and I’m trying to optimize it. You can see Blueprint connected to this theme on Launchpad https://blueprints.launchpad.net/mixxx/+spec/nonblockingdb . Here you can find information on my project wiki-page http://www.mixxx.org/wiki/doku.php/nonblockingdb_status . I want to consult with others about my further work. We got one long-time operation -- library scanning. I’ll try to implement this scheme instead of one big transaction: - emit (start()); - Inform user about long-time operation. - Block UI-control element (to avoid querying equivalent queries). - Get id for this big operation from DBAccessor - Cycle - - Prepare chunk of data DATA (do some FS-access/calculations/...) - - Mark DATA by id. - - Wrap DATA into one transaction. - - Async send chunk of data to DBAccessor. - Wait until DBAccessor say that big operation with id ended. - Unblock UI-control element. - emit (end()); This can give us possibility to divide all work into several pieces, do some work, collect data in transactions and then one by one send it to DB. (But not lock DB while doing one big transaction). Reading seems to work during active transaction. Next I’ll work on is preparing test-cases: - For SQLite (see https://github.com/troyane/SQLiteTesting) - Create library of files for library scanner -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Get Mixxx, the #1 Free MP3 DJ Mixing software Today http://mixxx.org Mixxx-devel mailing list Mixxx-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mixxx-devel
Re: [Mixxx-devel] Library scanner refactoring
Hi That is really ok. I'd be surprised if you wouldn't have any troubles. Can you later also give what problems you had and why you changed your strategy. And please cc the devel list. best Max On Sat, Jul 27, 2013 at 04:47:08PM +0300, Nazar Gerasimchuk wrote: Hi Max, thanx for your answer! Since I dig into code and tried to implement my ideas, I got some troubles and also deeper understanding of situation. So, really, now I'm trying to implement some other idea. I'm trying to implement pause in scanning process. Till Monday I'll try to prepare some more documentation and how I achieve it. Please, wait for my full description. About test functions -- I haven't implement tests yet. Think that it would be more useful to do after implementing my idea. - - - - - Kind regards, Nazar. troy...@gmail.com 2013/7/27 Max Linke max_li...@gmx.de Hi Nazar sorry for the late answer. Dividing the library scanning into smaller chunks sounds like a good idea. Did you check what others (Amarok,Clementine,Exaile,Banshee,...) do to handle a complete library rescan? I also have some questions about the implementation. What is your DBAccessor? Will this thing life in a separate thread? Will this be the only class that mixxx later uses for database access and by that be replacing direct access to the DAO's that we have all over mixxx? Also how far are you writing test functions for the DAO's? I might want to copy from them :-) best Max On Fri, Jul 12, 2013 at 03:41:05PM -0700, troyane wrote: As you may know, my name is Nazar Gerasymchuk and I’m working on refactoring of library concurrency. All of my work is related with access to database and I’m trying to optimize it. You can see Blueprint connected to this theme on Launchpad https://blueprints.launchpad.net/mixxx/+spec/nonblockingdb . Here you can find information on my project wiki-page http://www.mixxx.org/wiki/doku.php/nonblockingdb_status . I want to consult with others about my further work. We got one long-time operation -- library scanning. I’ll try to implement this scheme instead of one big transaction: - emit (start()); - Inform user about long-time operation. - Block UI-control element (to avoid querying equivalent queries). - Get id for this big operation from DBAccessor - Cycle - - Prepare chunk of data DATA (do some FS-access/calculations/...) - - Mark DATA by id. - - Wrap DATA into one transaction. - - Async send chunk of data to DBAccessor. - Wait until DBAccessor say that big operation with id ended. - Unblock UI-control element. - emit (end()); This can give us possibility to divide all work into several pieces, do some work, collect data in transactions and then one by one send it to DB. (But not lock DB while doing one big transaction). Reading seems to work during active transaction. Next I’ll work on is preparing test-cases: - For SQLite (see https://github.com/troyane/SQLiteTesting) - Create library of files for library scanner -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Get Mixxx, the #1 Free MP3 DJ Mixing software Today http://mixxx.org Mixxx-devel mailing list Mixxx-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mixxx-devel
[Mixxx-devel] Library scanner refactoring
Hi to all! As you may know, my name is Nazar Gerasymchuk and I’m working on refactoring of library concurrency. All of my work is related with access to database and I’m trying to optimize it. You can see Blueprint connected to this theme on Launchpad https://blueprints.launchpad.net/mixxx/+spec/nonblockingdb . Here you can find information on my project wiki-page http://www.mixxx.org/wiki/doku.php/nonblockingdb_status . I want to consult with others about my further work. We got one long-time operation -- library scanning. I’ll try to implement this scheme instead of one big transaction: - emit (start()); - Inform user about long-time operation. - Block UI-control element (to avoid querying equivalent queries). - Get id for this big operation from DBAccessor - Cycle - - Prepare chunk of data DATA (do some FS-access/calculations/...) - - Mark DATA by id. - - Wrap DATA into one transaction. - - Async send chunk of data to DBAccessor. - Wait until DBAccessor say that big operation with id ended. - Unblock UI-control element. - emit (end()); This can give us possibility to divide all work into several pieces, do some work, collect data in transactions and then one by one send it to DB. (But not lock DB while doing one big transaction). Reading seems to work during active transaction. Next I’ll work on is preparing test-cases: - For SQLite (see https://github.com/troyane/SQLiteTesting) - Create library of files for library scanner Thank for advice! -- View this message in context: http://mixxx-devel-list.2283481.n4.nabble.com/Library-scanner-refactoring-tp4641801.html Sent from the Mixxx Devel List mailing list archive at Nabble.com. -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Get Mixxx, the #1 Free MP3 DJ Mixing software Today http://mixxx.org Mixxx-devel mailing list Mixxx-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mixxx-devel
