IF the mutable file update problem is due to memory use problems with downloading the file, keeping it in memory then re-uploading it, then (for this use case) it is worth the overhead (to me) for now.
IF there are other issues that I need to be aware of, please let me know. Question: Why was the decision made to download the file before uploading the new version? thanks, jody ---- - Think carefully. - Contra mundum - "Against the world" (St. Athanasius) - Credo ut intelliga - "I believe that I may know" (St. Augustin of Hippo) On Mon, Jan 4, 2010 at 11:33 AM, Peter Secor <[email protected]> wrote: > You are correct Kevan, the hard limit has been removed, sorry I didn't > realize that had been fixed already! > > So, the limit is removed, but Jody you may want to read the ticket #694 > notes, there is a good discussion of why we originally had a hard-limit. > > Ps > > On 1/4/10 10:24 AM, Kevan Carstensen wrote: > > I'm sure someone will correct me if I'm wrong, but I think #694 removed > > the hard limit on mutable files. So your data should be backed up. > > > > (apologies for formatting - I'm on a phone) > > -- > > Kevan Carstensen | <[email protected] <mailto:[email protected]>> > > > > On Jan 4, 2010, at 10:10 AM, Jody Harris <[email protected] > > <mailto:[email protected]>> wrote: > > > >> Ack! > >> > >> So, I'm NOT backing up my data!? (I'm just backing up the first 2MB of > >> each compressed table?) > >> > >> Okay, if immutable files are my only choice, how do I do garbage > >> collection to insure that a minimum of dead space is taken up by > >> "yesterday's" files? > >> > >> jody > >> ---- > >> - Think carefully. > >> - Contra mundum - "Against the world" (St. Athanasius) > >> - Credo ut intelliga - "I believe that I may know" (St. Augustin of > Hippo) > >> > >> > >> On Mon, Jan 4, 2010 at 11:05 AM, Peter Secor < > >> <mailto:[email protected]>[email protected] > >> <mailto:[email protected]>> wrote: > >> > >> Mutable files currently have a small (2MB?) limit and cannot be > >> used for > >> larger files. Immutable files (the default) have a much higher limit > >> (12GB now?) and are much better suited for the purpose you > >> described of > >> large snapshots of data that won't change later. > >> > >> There are ideas behind larger mutable files (#393) but they are not > >> implemented yet. > >> > >> Ps > >> > >> On 1/4/10 7:37 AM, Jody Harris wrote: > >> > Oh, I forgot to answer the "how?" > >> > > >> > ... put --mutable ... > >> > > >> > jody > >> > ---- > >> > - Think carefully. > >> > - Contra mundum - "Against the world" (St. Athanasius) > >> > - Credo ut intelliga - "I believe that I may know" (St. Augustin > >> of Hippo) > >> > > >> > > >> > On Sun, Jan 3, 2010 at 10:59 PM, Jody Harris < > >> <mailto:[email protected]>[email protected] <mailto: > [email protected]> > >> > <mailto: <mailto:[email protected]>[email protected] > >> <mailto:[email protected]>>> wrote: > >> > > >> > I have a script that: > >> > - iterates though my MySQL tables > >> > - dumps each table > >> > - pipes the dump through gzip > >> > - write the file to a temp file > >> > - copies the file to [remote storage] (now Tahoe grid) > >> > - deletes temp file > >> > > >> > This gives me a daily snapshot of all of the active databases on > my > >> > server every night. > >> > > >> > The current backup is ~ 300 MB/night, which isn't bad, but after > one > >> > year, that's 110+GB x 3.3 and counting of consumed tahoe space. > >> > Unless tahoe has a better garbage collection system than I have > come > >> > to believe, that's going to be a problem on a 500 GB grid. > >> > > >> > In my reading of the tahoe docs, I have not come across any alarms > >> > concerning mutable files. Is there something I should be made > >> aware of? > >> > > >> > jody > >> > ---- > >> > - Think carefully. > >> > - Contra mundum - "Against the world" (St. Athanasius) > >> > - Credo ut intelliga - "I believe that I may know" (St. Augustin > of > >> > Hippo) > >> > > >> > > >> > On Sun, Jan 3, 2010 at 10:50 PM, Zooko O'Whielacronx > >> > < <mailto:[email protected]>[email protected] > >> <mailto:[email protected]> <mailto: > >> <mailto:[email protected]>[email protected] > >> <mailto:[email protected]>>> wrote: > >> > > >> > On Sun, Jan 3, 2010 at 10:33 PM, Jody Harris > >> > < <mailto:[email protected]>[email protected] > >> <mailto:[email protected]> <mailto: > >> <mailto:[email protected]>[email protected] > >> <mailto:[email protected]>>> wrote: > >> > > I have set up my web server to store backup files on my tahoe > >> > grid, shifting > >> > > from the use of a server at my house. One thing that > >> > initially alarmed me > >> > > was that the timestamps of the backed up files is not updated > >> > in the > >> > > tahoe-lafs interface. I did confirm that the shares > >> > timestamps are updated. > >> > > >> > Tahoe-LAFS doesn't store timestamps or other metadata on files, > only > >> > on links to files, which links are stored in directories. So if > you > >> > have a directory which contains a link named "README.txt" to > >> > version 1 > >> > of your file, and then you upload version 2 of your file and you > >> > change that directory to link to version 2, then Tahoe-LAFS will > >> > update the timestamp on that link named "README.txt" when it > updates > >> > the link to point to the new file. > >> > > >> > Did you say "mutable files"? You should probably not be using > >> > mutable > >> > files for anything. Immutable files (plus directories) probably > fit > >> > all your needs for backups, and mutable files have significant > >> > performance problems. > >> > > >> > How do you trigger these backups -- are you using "tahoe cp" or > >> > "tahoe > >> > backup" or sending HTTP requests to the WAPI or something else? > >> > > >> > Regards, > >> > > >> > Zooko > >> > _______________________________________________ > >> > tahoe-dev mailing list > >> > <mailto:[email protected]>[email protected] > >> <mailto:[email protected]> <mailto: > >> <mailto:[email protected]>[email protected] > >> <mailto:[email protected]>> > >> > > >> <http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev> > http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev > >> > > >> > > >> > > >> > > >> > > >> > _______________________________________________ > >> > tahoe-dev mailing list > >> > <mailto:[email protected]>[email protected] > >> <mailto:[email protected]> > >> > > >> <http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev> > http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev > >> _______________________________________________ > >> tahoe-dev mailing list > >> <mailto:[email protected]>[email protected] > >> <mailto:[email protected]> > >> <http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev> > http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev > >> > >> > >> _______________________________________________ > >> tahoe-dev mailing list > >> [email protected] <mailto:[email protected]> > >> http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev > > > > > > _______________________________________________ > > tahoe-dev mailing list > > [email protected] > > http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev > _______________________________________________ > tahoe-dev mailing list > [email protected] > http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev >
_______________________________________________ tahoe-dev mailing list [email protected] http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev
