While going over my Accounting work[1] this morning, I had an idea about simplifying the backend storage share-file format. I'd like to remove the lease information from the share files themselves, and use a separate per-server sqlite database (the "LeaseDB") to hold all lease data. I wanted to mention it right away since it interacts with other folk's design work, in particular Least Authority Enterprises (probably for the better: I think it'll make their job simpler).
So far, we've stored all information about leases in the same file as the share data. The general idea is that the share file is canonical, and we have a bunch of Crawlers whose job it is to update/refresh/maintain secondary data structures for faster access. We did this so that we could manually move shares from one server to another by simply copying the backend files with 'scp' or the like, and all the metadata would travel along with them. But having the lease data in those files is a hassle: it's variable length, which means shares for immutable files will change size over time. It requires storing per-server renew/cancel secrets for each share, which both hampers actual migration (the secrets end up being wrong) and means that part of the share file should be kept secret from readers (which was the cause of the security bug that prompted 1.8.3). And Accounting needs that lease data to be in a place where it can be summarized quickly (to answer requests like "find all expired leases", "find all shares that I hold leases on", "how much space am I using", or "Bob shut down his account, cancel all his leases right away"), for which a real database with a proper index works a lot better than a terabyte of share data with tiny bits of lease scattered throughout. So my proposal is this: * make the LeaseDB be the canonical source of lease information * stop updating, ignore all lease info in the share files * build an AccountingCrawler with a schema like this: CREATE TABLE buckets -- I think David-Sarah proposed "sharesets" here ( `storage_index` VARCHAR(26), `id` INTEGER PRIMARY KEY AUTOINCREMENT ); CREATE TABLE leases ( `bucket_id` INTEGER, `account_id` INTEGER, `expiration_time` INTEGER ); * when the crawler sees a share that isn't in the "bucket" table, add it, and add a special "migration lease" that lasts for a month or two. This handles shares that were copied in manually, and also the transition period when the server is first upgraded to this code. * if the crawler sees an entry in the bucket table that has no actual shares on disk, delete the entry. This handles shares that were manually deleted. With that, Accounting can work by adding entries to the "leases" table. Periodic expiration will query the table to find all leases with expiration_time in the past, make a list of their bucket_ids, delete those leases, then walk the list of potential victims to see which ones have no more leases left, and delete the shares (and "buckets" entry) if so configured. This will allow basic manual share migration to work well enough: the requirement is that the client does a deep-add-lease within a month or two to reclaim their shares (which GC requires anyways). It might increase the server's storage burden slightly, if the migrated shares were about to expire on their own (they'll last a month or two extra), but I think that's minor given that manual migration is not likely to happen frequently. And I think we can build better migration tools anyways: imagine a "tahoe storage export SHARE-CRITERIA.." command, which streams data to stdout, to be caught by a corresponding "tahoe storage import" command. The data could include both the share data and the lease information, in a form that is meaningful to the remote end (translating local account_ids, for example, or having the old *server* claim a temporary lease on the shares, attributing the storage burden to them until the real owner takes over). Or a form which copies the export data to a removable disk, to then be manually transported to the new server. I want this because the AccountingCrawler that I've half-implemented so far is worryingly complex, as it's trying to keep a database table in sync with the sharefiles' embedded lease information. And because keeping the leases inside the shares has been a PITA anyways :). Thoughts? -Brian [1]: https://github.com/warner/tahoe-lafs/tree/accounting _______________________________________________ tahoe-dev mailing list tahoe-dev@tahoe-lafs.org http://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev