Re: [Mixxx-devel] Hello, and some patches
Hey Steven, Welcome and thanks for the patch! Some comments: 1) Flushing after every folder is scanned sounds like it could have a significant impact on scanning performance. Using your friend's large collection have you noticed any reduction in scanning speed? In the future we're looking at refactoring everything so that all library operations happen in one thread and all other threads communicate with that thread via message passing. This would allow the scanner to periodically push new tracks to be inserted into the database to the database thread. It also solves the locking problem by forcing all parts of Mixxx to interact with the database asynchronously and have a strategy for waiting for responses to their requests. Hopefully we can work on this change for Mixxx 1.12. 2) RE: returning in BSTM if the table query fails I think this looks like a nice workaround to that bug. It results in bad behavior because the table will still contain the old contents while the sidebar reflects the new view but it's much better than the alternative (assertion failures). We're in the process of releasing 1.11.0 right now and I will be holding a 1.12.0 planning meeting soon. You should join us! Announcements will go out to mixxx-devel for the date/time. thanks, RJ On Wed, Apr 10, 2013 at 1:45 AM, Daniel Schürmann dasch...@mixxx.orgwrote: Hi Steven, welcome to to the Mixxx community! It is nice to here that you had an successful gig using Mixxx. And thank you very much for your patch. I am impressed that you are so familiar with the Mixxx code without any help! I had not the chance to test it, but from the code it looks very good and suitable to fix the bug. Please attach the patch to bug #1090888 and assign the bug to yourselves or mark it as duplicate and use bug #870128 instead. Kind regards, Daniel 2013/4/10 Steven Boswell II ulat...@yahoo.com Hello all! I got a panicked call from a DJ/KJ friend of mine this weekend. He knew I was handy with computers needed help DJing a gig he had that evening. I quickly found mixxx (in the rpmfusion repo for Fedora Core 17), put it through its paces, called him back to tell him I thought I could fake it, spent a few more hours load-testing mixxx and beating it to death...then went to the gig. We pulled it off! So, just like that, I'm a DJ. :-) Enclosed are some patches I've come up with so far. I doubt they're ready to check in; I'm posting them mostly for feedback, and so you can point out any problems with them. I'm building with the 1.11 branch. The patch to src/library/basesqltablemodel.cpp is meant to fix bug #1090888, i.e. https://bugs.launchpad.net/mixxx/+bug/1090888 . The problem is that, if the database is locked, the code continues with an invalid SQL-query result. My solution is to move the query to the top of the method, and return if the query fails. The rest of the patch makes it possible for library-rescan to run in parallel with the rest of the GUI. Before, the database would stay locked during rescan, although nothing prevented the user from trying to use the rest of the GUI, which would lead to the GUI hanging and/or crashing relatively often. My solution is to commit at the end of scanning each individual directory, so that other database clients get a chance to do their work. I also added a pause button to the library-rescan window. I intend to spend part of this evening load-testing my changes. My DJ friend has several hundred GB of music; I tried to scan it right before the gig, but didn't finish. When I had time, I let it run, and it took about 4 hours. Being able to scan it incrementally, and allowing Mixxx to be usable meanwhile, would have been a big help; these changes allow that. I have more to say, but I figure I'll stop here for now...no need to talk your ear off in my first e-mail :-) Steven Boswell -- Precog is a next-generation analytics platform capable of advanced analytics on semi-structured data. The platform includes APIs for building apps and a phenomenal toolset for data science. Developers can use our toolset for easy data analysis visualization. Get a free account! http://www2.precog.com/precogplatform/slashdotnewsletter ___ Get Mixxx, the #1 Free MP3 DJ Mixing software Today http://mixxx.org Mixxx-devel mailing list Mixxx-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mixxx-devel -- Precog is a next-generation analytics platform capable of advanced analytics on semi-structured data. The platform includes APIs for building apps and a phenomenal toolset for data science. Developers can use our toolset for easy data analysis visualization. Get a free account!
Re: [Mixxx-devel] Hello, and some patches
Hi 1) Flushing after every folder is scanned sounds like it could have a significant impact on scanning performance. Using your friend's large collection have you noticed any reduction in scanning speed? I tested it with my small collection and is has a huge impact. The explanation is also fairly easy. Because sqlite force a write to the HD on commit with some consistency checks you can only do ~50 commits/s on an average HD. This writing to the hd is visible when the scanner seems to stall on one title. Reading the sqlitedb shouldn't be affected by the scanner as sqlite allows for multiply reads all the time, also with an open transaction. 2) RE: returning in BSTM if the table query fails I think this looks like a nice workaround to that bug. It results in bad behavior because the table will still contain the old contents while the sidebar reflects the new view but it's much better than the alternative (assertion failures). Is there actually a reliable way to reproduce this error. I sometimes get it at random and so far I failed to reproduce it with ways that other described. As far as I understand the sqlite specs the db should never block to a read request, so I'm puzzled why this happens in mixxx. The pause button is a good idea though. Most users probably won't notice that canceled scan will commit every songs it added so far and that they can just resume by restarting the library scanner. best, Max -- Precog is a next-generation analytics platform capable of advanced analytics on semi-structured data. The platform includes APIs for building apps and a phenomenal toolset for data science. Developers can use our toolset for easy data analysis visualization. Get a free account! http://www2.precog.com/precogplatform/slashdotnewsletter ___ Get Mixxx, the #1 Free MP3 DJ Mixing software Today http://mixxx.org Mixxx-devel mailing list Mixxx-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mixxx-devel
Re: [Mixxx-devel] Hello, and some patches
1) Flushing after every folder is scanned sounds like it could have a significant impact on scanning performance. Using your friend's large collection have you noticed any reduction in scanning speed? I haven't rescanned the collection over again since making this change. I can do that, though. I tested it with my small collection and it has a huge impact. That doesn't surprise me. In the worst case, we can add an incremental library scanning option to the preferences. But the old variant will have to lock the main Mixxx window to prevent using it. Some GUIs do that automatically when a modal dialog is displayed...I guess Qt doesn't? Reading the sqlitedb shouldn't be affected by the scanner as sqlite allows for multiply reads all the time, also with an open transaction. [...] As far as I understand the sqlite specs the db should never block to a read request, so I'm puzzled why this happens in mixxx. AFAIK, multiple readers are OK, but there can be only one writer on a table, and a writer locks out readers. That certainly seems to be how it's behaving, too. Perhaps non-incremental library-scanning can be rewritten to create new tables with temporary names, then at the end, drop the old tables and move the new tables in their place? 2) RE: returning in BSTM if the table query fails I think this looks like a nice workaround to that bug Is there actually a reliable way to reproduce this error. I sometimes get it at random and so far I failed to reproduce it with ways that other described. I can reproduce it pretty easily by trying to click on the Library or Auto DJ playlists during a library rescan. Steven Boswell-- Precog is a next-generation analytics platform capable of advanced analytics on semi-structured data. The platform includes APIs for building apps and a phenomenal toolset for data science. Developers can use our toolset for easy data analysis visualization. Get a free account! http://www2.precog.com/precogplatform/slashdotnewsletter___ Get Mixxx, the #1 Free MP3 DJ Mixing software Today http://mixxx.org Mixxx-devel mailing list Mixxx-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mixxx-devel
Re: [Mixxx-devel] New feature for Auto DJ
Hi Steven, I would be wonderful if you can share your experience in real time audio processing and play list generation with us and join us to make Mixxx even better. We have a some contributions and ideas to auto play list generations. But currently nothing has managed to sneak into the trunk. And as far as I know, non of it is close to your approach. Here is a list of links what I can find and remember. I hope that the others on the list can also give valuable hints. https://bugs.launchpad.net/mixxx/+bug/889898 https://blueprints.launchpad.net/mixxx/+spec/auto-library-filter http://www.mixxx.org/forums/viewtopic.php?f=1t=4924 https://bugs.launchpad.net/mixxx/+bug/1023804 https://bugs.launchpad.net/mixxx/+bug/674067 https://bugs.launchpad.net/mixxx/+bug/894652 Since this malign list is somehow volatile, you should consider open a wishlist bug or a blueprint on Launchpad or a new page on our Wiki to save the ideas. (Or all of it). Btw: Is your djserver open source? Kind regards, Daniel 2013/4/10 Steven Boswell II ulat...@yahoo.com Years ago, I DJed an annual surf contest. At the time, I couldn't find any decent, reliable DJ software, so I wrote my own in 3 weeks. The interface was simple -- I used telnet to connect to my djserver program and I would issue it text commands. Most of the work writing it involved making sure that the sound would never, never, NEVER underflow and pop. I did that with careful threading, careful buffering, and a realtime-priority thread keeping the sound hardware's output buffer full. I could rip new CDs, encode MP3s, etc. on the machine, and the sound would never be affected. (And this is with circa-1998 hardware!) I could give it individual songs to play, and I could also add and remove playlists. However, my handling of playlists was different than Auto DJ's...if there were no individual songs scheduled to play, it would pick a random song out of the totality of loaded playlists. Removing a playlist would take all those songs out of consideration for random selection (unless some of its songs appeared in other loaded playlists, of course). And it would try to play songs that hadn't been played as often -- I gave it a minimum percentage of the total songs available to consider for random selection, so that already-played songs would only be considered if the total number of available songs dropped below that minimum. That would tend to keep more-often-played songs out, but would allow them in so that it wouldn't just play all the songs in the same order. In Mixxx, it seems like this could be implemented by allowing crates to be added to the Auto DJ queue, but as whole entities. If there are no individual songs in the Auto DJ queue, a song will be picked at random from the loaded crates, with the above-described de-prioritization of already-played songs. Any thoughts on this? I intend to implement this sooner rather than later, if I can keep finding the time and energy to work on this, since it's mostly how I intend to use AutoDJ. I won't be DJing ultra-hip ravefests any time soon; I'll be doing pedestrian stuff like birthday parties, happy hours, and car shows. Steven Boswell -- Precog is a next-generation analytics platform capable of advanced analytics on semi-structured data. The platform includes APIs for building apps and a phenomenal toolset for data science. Developers can use our toolset for easy data analysis visualization. Get a free account! http://www2.precog.com/precogplatform/slashdotnewsletter ___ Get Mixxx, the #1 Free MP3 DJ Mixing software Today http://mixxx.org Mixxx-devel mailing list Mixxx-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mixxx-devel -- Precog is a next-generation analytics platform capable of advanced analytics on semi-structured data. The platform includes APIs for building apps and a phenomenal toolset for data science. Developers can use our toolset for easy data analysis visualization. Get a free account! http://www2.precog.com/precogplatform/slashdotnewsletter___ Get Mixxx, the #1 Free MP3 DJ Mixing software Today http://mixxx.org Mixxx-devel mailing list Mixxx-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mixxx-devel
Re: [Mixxx-devel] Hello, and some patches
On Tuesday, April 9, 2013 at 10:45 PM, Daniel Schürmann wrote: And thank you very much for your patch. I am impressed that you are so familiar with the Mixxx code without any help! The library-rescan code had a decent amount of source-code comments, so I figured it out pretty quickly. I always like finding documented code. Hopefully I'll continue to find that as I go through Mixxx's source. :-) I had not the chance to test it, but from the code it looks very good and suitable to fix the bug. Please attach the patch to bug #1090888 and assign the bug to yourselves or mark it as duplicate and use bug #870128 instead. As soon as I figure out how to do all that, I will! On Wednesday, April 10, 2013 at 7:15 AM, RJ Ryan wrote: In the future we're looking at refactoring everything so that all library operations happen in one thread and all other threads communicate with that thread via message passing. [...] That would ultimately be the best solution, but that was a bigger change than I was willing to take on as my first project. :-) We're in the process of releasing 1.11.0 right now and I will be holding a 1.12.0 planning meeting soon. You should join us! Announcements will go out to mixxx-devel for the date/time. I don't know how useful I'd be in such a meeting, but I can try! Maybe I'll just learn by listening. Re: my incremental library-scanning change...right now, I plan for my next version to allow for both incremental and non-incremental scanning, controlled from the same progress dialog. The dialog will gain a checkbox, defaulting to unchecked, that allows for incremental scanning. If scanning is not incremental, the dialog will be modal (i.e. QWidget::setWindowModality()), so that the user can't click on the Mixxx GUI and cause crashes. If scanning is incremental, the dialog will not be modal, and changes will get committed to the database relatively often. Clicking the pause button will also make the dialog non-modal and commit changes to the database. As soon as I have that written, I'll e-mail the patch to the list, for your comment. Steven Boswell-- Precog is a next-generation analytics platform capable of advanced analytics on semi-structured data. The platform includes APIs for building apps and a phenomenal toolset for data science. Developers can use our toolset for easy data analysis visualization. Get a free account! http://www2.precog.com/precogplatform/slashdotnewsletter___ Get Mixxx, the #1 Free MP3 DJ Mixing software Today http://mixxx.org Mixxx-devel mailing list Mixxx-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mixxx-devel
Re: [Mixxx-devel] New feature for Auto DJ
On Wednesday, April 10, 2013 at 2:01 PM, Daniel Schürmann wrote: It would be wonderful if you can share your experience in real time audio processing and play list generation with us and join us to make Mixxx even better. I'll do what I can! I guess I'll find out how much of my experience is still relevant. :-) Here is a list of links what I can find and remember. I will look at those links when I have time...probably tonight and tomorrow. Since this mailing list is somewhat volatile, you should consider opening a wishlist bug or a blueprint on Launchpad or a new page on our Wiki to save the ideas. (Or all of it). I can do that once my ideas get a bit more cooked, i.e. once I understand the target audience better. It seems that Auto DJ isn't the sort of thing that professional DJs make a lot of use of; they're too busy being ultra-hip with record scratching and whatnot. So who knows how much use my playlist changes will see outside of...well, me. :-) Btw: Is your djserver open source? I've never released the code before, but I don't mind releasing it under GPL v2. I just got it compiling and running under Fedora Cores 14 and 16 -- the last time I built it was apparently September 2002. :-) Let me write up a bit of user documentation and I'll e-mail it to the list! Steven Boswell-- Precog is a next-generation analytics platform capable of advanced analytics on semi-structured data. The platform includes APIs for building apps and a phenomenal toolset for data science. Developers can use our toolset for easy data analysis visualization. Get a free account! http://www2.precog.com/precogplatform/slashdotnewsletter___ Get Mixxx, the #1 Free MP3 DJ Mixing software Today http://mixxx.org Mixxx-devel mailing list Mixxx-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mixxx-devel
[Mixxx-devel] Trouble using Auto DJ
I was beating the heck out of auto-DJ last night, and got it to mess up a few times. Sometimes, the played track wouldn't get removed from the list, and the other deck would get loaded with the same track. I thought it was because of shuffling the playlist, or adding new tracks to the top of the queue, or dragging tracks to the decks manually, but after reading the code over today, I don't think that's it -- the code seems to expect that. However, I can cause that problem by dragging a track to the top of the auto-DJ queue manually. The code (i.e. src/dlgautodj.cpp) apparently leaves the track at the top of the auto-DJ queue when it loads the deck, and only removes it when the track starts playing. I wonder, why is that? DlgAutoDJ::shufflePlaylist() has to special-case this, as does WTrackTableView::sendToAutoDJ(). Plus, DlgAutoDJ::toggleAutoDJButton() will mess up if the first track in the queue is on the 2nd deck, and the second track in the queue is on the 1st deck -- neither will get removed from the auto-DJ queue. I think it'd be less error-prone to remove the track from the auto-DJ queue as soon as it's loaded into a deck. The alternative would, at a minimum, have to prevent the user from dragging a track to the top of the auto-DJ queue, which would probably confuse the user. Any thoughts on this? I would do the work of fixing this, once I have your feedback on the issues involved. Steven Boswell -- Precog is a next-generation analytics platform capable of advanced analytics on semi-structured data. The platform includes APIs for building apps and a phenomenal toolset for data science. Developers can use our toolset for easy data analysis visualization. Get a free account! http://www2.precog.com/precogplatform/slashdotnewsletter___ Get Mixxx, the #1 Free MP3 DJ Mixing software Today http://mixxx.org Mixxx-devel mailing list Mixxx-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mixxx-devel
