On Friday, August 19, 2016 at 2:21:13 PM UTC-4, Adrian Sampson wrote: > > Yes, there are potentially-bad ways that beets could crash during import > and leave the DB inconsistent with the placement of files on disk. But it's > actually shockingly difficult to get this right—most filesystems don't have > atomic transactions, so it's nearly impossible to make durable, > simultaneous changes to the SQLite database and the files. >
you can gain an adequate degree of "atomicity" with file copy and rewrite operations using os.rename: https://docs.python.org/3/library/os.html#os.rename. A potential improvement to this behavior would be that if beets does need to persist the to-be-imported file location in the primary database, it could store this with a qualification that it's non-imported, and clean out what is essentially temporary data when it next runs. SQLite is not very good with concurrent writes of course so if it were me I'd likely just use tempfiles for each thread as it does its work (tempfile could itself be a SQLite database) and use a single queue to persist completed write-sets out to the primary database. > > The common way that beets can crash is when things have been added to the > database, are undergoing some additional processing (to add lyrics, for > example), but haven't been moved into the library directory yet. In that > case, there's not really a consistency issue—the paths in the database do > point to files that actually exist—and I think it's actually better than > the alternative, where the files are moved but the database doesn't know > about them yet. A manual `beet move` can come in handy then. > > On Aug 19, 2016, at 12:47 PM, [email protected] <javascript:> wrote: > > > > On Friday, August 19, 2016 at 12:37:01 PM UTC-4, Adrian Sampson wrote: >> >> Aha, thanks for clarifying! >> >> No, that should not happen—if it does, it's a bug. There's one exception: >> if you `beet ls` while an import is currently happening, or if beets >> crashes halfway through, you might see the old paths. >> > > then that is likely what I'm seeing. > > so to further clarify, beets temporarily writes the import file location > into the database? This would be a durability bug, because if the program > crashes, now the DB is basically corrupted ? > > > > > > >> >> On Aug 19, 2016, at 11:33 AM, [email protected] wrote: >> >> >> >> On Thursday, August 18, 2016 at 4:55:19 PM UTC-4, Adrian Sampson wrote: >>> >>> As always, if you think something’s wrong, please provide instructions >>> we can use to reproduce the problem from scratch. >>> >>> In the mean time, if you have files that have been placed somewhere you >>> don’t want them, you can try using `beet move` to have them moved into >>> their default locations. >>> >> >> >> The basic case of, beets config says the files go in directory /foo. >> After an import, beets list shows the path is in /bar. My question is, if >> this were to be demonstrated, would this ever be a correct behavior? I'm >> not asking you to fix it or diagnose it. Just in the abstract, is that >> supposed to happen in some case? thanks for your attention to this >> question. >> >> >> >>> >>> On Aug 18, 2016, at 12:47 PM, [email protected] wrote: >>> >>> in fact it seems to just be not copying / moving files at all from the >>> directory im importing from to the location I've configured. I'm totally >>> lost. >>> >>> beets config: >>> >>> directory: /mnt/synology_beets_music/ >>> library: /mnt/synology_beets_music/library.blb >>> import: >>> copy: yes >>> write: yes >>> incremental: yes >>> paths: >>> default: $albumartist/$album/$track $title >>> singleton: Non-Album/$artist/$title >>> comp: Various Artists/$album%aunique{}/$track $title >>> >>> plugins: fetchart chroma discogs scrub edit >>> chroma: >>> auto: yes >>> >>> >>> command: >>> >>> beet import /mnt/synology_beets_import/ >>> >>> it imports some files, and then I see this: >>> >>> $ beet list American -p >>> /mnt/synology_beets_import/American Music Club/San Francisco/01 >>> Fearless.m4a >>> /mnt/synology_beets_import/American Music Club/San Francisco/02 It's >>> Your Birthday.m4a >>> /mnt/synology_beets_import/American Music Club/San Francisco/03 Can You >>> Help Me_.m4a >>> /mnt/synology_beets_import/American Music Club/San Francisco/04 Love >>> Doesn't Belong 1.m4a >>> /mnt/synology_beets_import/American Music Club/San Francisco/05 Wish The >>> World Away.m4a >>> /mnt/synology_beets_import/American Music Club/San Francisco/06 How Many >>> Six Packs Does It Take To Screw In A Lightbulb.m4a >>> /mnt/synology_beets_import/American Music Club/San Francisco/07 Cape >>> Canaveral.m4a >>> /mnt/synology_beets_import/American Music Club/San Francisco/08 Hello >>> Amsterdam.m4a >>> /mnt/synology_beets_import/American Music Club/San Francisco/09 The >>> Revolving Door.m4a >>> /mnt/synology_beets_import/American Music Club/San Francisco/10 In The >>> Shadow Of The Valley.m4a >>> /mnt/synology_beets_import/American Music Club/San Francisco/11 What >>> Holds The World Together.m4a >>> /mnt/synology_beets_import/American Music Club/San Francisco/12 I Broke >>> My Promise.m4a >>> /mnt/synology_beets_import/American Music Club/San Francisco/13 The >>> Thorn In My Side Is Gone.m4a >>> /mnt/synology_beets_import/American Music Club/San Francisco/14 I'll Be >>> Gone.m4a >>> /mnt/synology_beets_import/American Music Club/San Francisco/15 Fearless >>> (Reprise).m4a >>> >>> I'm lost. Why did it put files *in the library* directly where they are >>> sitting in the import location? Why didn't it *copy* them to the *place >>> where the files go* ? I'm sure we'll figure out some silly reason >>> eventually, but this is just too hard. For this particular album, it >>> also decided to drop one of the files from the library, because the name >>> didn't match up. Different issue. >>> >>> >>> >>> >>> >>> >>> On Thursday, August 18, 2016 at 11:39:01 AM UTC-4, cla...@ >>> zzzcomputing.com wrote: >>>> >>>> Hi - >>>> >>>> I've been working with beets for some weeks now and while it does a lot >>>> of things that are useful, there are many quirks. I'd like to just start >>>> sending them to the list and if I have time I can provide more detail as >>>> I'm assuming this will be necessary. >>>> >>>> First one, beets "copy" will not do a copy if the files you are >>>> importing happen to be in a deep subfolder of the ultimate target >>>> directory, even though the path is wrong and even though the files being >>>> accessed *are not* part of the library yet. Beets should raise an error >>>> if it can't handle this directory setup and it should never *overwrite* a >>>> non-imported file that's not in the correct target location. I >>>> understand this probably is interacting poorly with the fact that you can >>>> do "beet import" at the existing library directory, however because these >>>> files were not present at all in the library beforehand and they are not >>>> in >>>> the correct library position. >>>> >>>> assume you set up this: >>>> >>>> >>>> /mnt/my_music/<no files> >>>> >>>> /mnt/my_music/files_to_import/<all the files I want to import> >>>> >>>> then set up beets config including: >>>> >>>> directory: /mnt/my_music/ >>>> library: /mnt/my_music/library.blb >>>> import: >>>> copy: yes >>>> write: yes >>>> incremental: yes >>>> >>>> then tell beets to import from that subfolder. >>>> >>>> beet import /mnt/my_music/files_to_import/ >>>> >>>> As to why one would do this, I'm working with an NFS-mounted share, and >>>> I wanted to keep all the files on the same side of the share so that the >>>> "copy" operation would proceed much faster. Of course I could have just >>>> made two subdirectories on the share, or used two shares, however the root >>>> of the share is where I ultimately want the music to be, and because beets >>>> seems to code absolute paths in the library.blb file (another poor >>>> practice, as the database file is not portable). >>>> >>>> So the result is that beet *does not copy* the files, it *overwrites >>>> them in place*, thus erasing the tags in the original files which I'd of >>>> course like to keep pristine. That is, in the library itself, when I >>>> do "beet list -p", the files' permanent location is in the >>>> "files_to_import" subfolder which is *not* part of my "paths" >>>> configuration, and this is therefore incorrect. >>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "beets" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "beets" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> >> -- > You received this message because you are subscribed to the Google Groups > "beets" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > For more options, visit https://groups.google.com/d/optout. > > -- You received this message because you are subscribed to the Google Groups "beets" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
