Re: [tahoe-dev] Thinking about building a P2P backup system

2009-01-09 Thread Shawn Willden
On Friday 09 January 2009 01:33:39 pm Brian Warner wrote: The 'revlog' format that I'd like to use for our eventual LDMF (Large Distributed Mutable Files) design uses forward deltas and an append-only data structure How do you plan to store metadata revisions? This is something I'm struggling

Re: [tahoe-dev] Estimating reliability

2009-01-12 Thread Shawn Willden
On Thursday 08 January 2009 07:56:17 pm Brian Warner wrote: This could be a great academic paper. I've been looking for a grad student to feed (I hear free lunch can get you a lot of paper-writing time :). If you find yourself interested in doing this sort of analysis, I'd love to sit down

Re: [tahoe-dev] Random access to files

2009-01-27 Thread Shawn Willden
On Tuesday 27 January 2009 06:12:19 am zooko wrote: Sounds great! I guess for step 2 we add get_range(offset=0, length=None) and we remove the call to self._download_all_segments in CiphertextDownloader.start(): I need to make sure I understand how this code works. Everything is a callback!

Re: [tahoe-dev] Random access to files

2009-01-28 Thread Shawn Willden
Brian Warner warner-ta...@allmydata.com writes: On Tue, 27 Jan 2009 06:59:49 -0700 Shawn Willden shawn-ta...@willden.org wrote: How hard would it be to add support for random access? Not impossible, but not trivial. Dang. In current Tahoe trunk, a webapi GET call with a Content-Range

Re: [tahoe-dev] Keeping local file system and Tahoe store in sync

2009-02-03 Thread Shawn Willden
On Tuesday 03 February 2009 04:47:15 pm Brian Warner wrote: A few notes about the differences that Shawn mentioned: 4. Backup of metadata in addition to file contents. Permissions, ACLs, resource forks, etc. My ultimate goal is to be able to do whole-system backups and restores, so this is

Re: [tahoe-dev] Keeping local file system and Tahoe store in sync

2009-02-06 Thread Shawn Willden
On Friday 06 February 2009 04:57:33 pm Jeremy Fitzhardinge wrote: BTW, Bryan O'Sullivan wrote a reasonably sensible python binding for inotify as part of Mercurial; not sure if it has an independent existence. There's at least one other inotify binding, but its barking mad. I'm mostly hoping

Re: [tahoe-dev] Keeping local file system and Tahoe store in sync

2009-02-06 Thread Shawn Willden
On Friday 06 February 2009 07:51:42 pm Shawn Willden wrote: I'm mostly hoping that someone will implement an fsevents workalike for Linux. http://lwn.net/Articles/311850/ Looks like there may be something in the works -- fsnotify. The history is a little uninspiring, though; the driving

Re: [tahoe-dev] documenting tahoe's design in standardizable form

2009-02-07 Thread Shawn Willden
On Saturday 07 February 2009 02:00:49 pm Zooko O'Whielacronx wrote: Shawn wants it to store exactly K shares of each file on his mom's computer, if that file is tagged as being a family photo) Even more immediate, I want to be able to tell Tahoe NOT to store any shares on the local Tahoe node.

Re: [tahoe-dev] darcs patch: Allow client-specified keys

2009-02-11 Thread Shawn Willden
in the scrollback buffer: zsh/2 25 % darcs send Creating patch to http://allmydata.org/source/tahoe/trunk-hashedformat;... Tue Jan 13 19:12:35 MST 2009 Shawn Willden shawn-ta...@willden.org * Statistics module Added a statistics module for calculating various facets of share survival statistics. Shall

Re: [tahoe-dev] Tahoe performance

2009-02-13 Thread Shawn Willden
On Friday 13 February 2009 02:57:17 pm Luke Scharf wrote: He's running both sides of rsync locally. You can use rsync to sync local filesystems, which is what he's suggesting. Yeah, that has the benefit that rsync will look at timestamps and sizes. Otherwise it's equivalent to cp -a I used

Re: [tahoe-dev] Unicode issues review

2009-02-17 Thread Shawn Willden
On Tuesday 17 February 2009 10:31:35 am zooko wrote: Ugh -- you mean to tell me that the filesystem itself might not know what encoding a filename is in? Yep! On most (all?) Unix-style systems, locale is an environment setting with a system-wide default, but can be overridden per-user (or

Re: [tahoe-dev] Unicode issues review

2009-02-17 Thread Shawn Willden
On Tuesday 17 February 2009 11:09:09 am zooko wrote: 3. If the filesystem guarantees a specific encoding, use that one, else if it provides a default encoding, then try to decode with that one, and if decoding fails then reject the filename and ask the user to fix it up. 3.b. ... and if

Re: [tahoe-dev] Unicode issues review

2009-02-17 Thread Shawn Willden
Since you control the dirnode format, wouldn't it be easier just to add a this isn't Unicode flag, rather than translating to a reserved range? If the flag is set, the name is opaque binary data. Otherwise, it's UTF-8. Hmm. That suggests another option for Zooko's list: Provide per-file

Re: [tahoe-dev] Unicode issues review

2009-02-17 Thread Shawn Willden
On Tuesday 17 February 2009 09:12:51 pm Kevin Reid wrote: What I'm thinking is: Will supporting unknown-bunch-of-bytes filenames be used sufficiently often to be worth the systemwide complexity in handling them (being not Just Strings), within Tahoe and all client software? If someone knows

Re: [tahoe-dev] [tahoe-lafs] #632: backupdb: override pathname root for snapshots

2009-02-18 Thread Shawn Willden
On Wednesday 18 February 2009 11:07:34 am tahoe-lafs wrote: By the way, I had a good idea for a technique to avoid missed or inconsistent backups of a file at the cost of having delayed (even indefinitely delayed) backup of that file. It is written in this message:

Re: [tahoe-dev] Tahoe performance

2009-02-18 Thread Shawn Willden
On Wednesday 18 February 2009 06:34:16 pm Brian Warner wrote: When rsync is used locally (i.e. no ssh), I don't know whether it uses this algorithm, or if it just compares timestamps and does a one-read/one-write copy when the timestamps don't match. The man page says it goes into whole-file

Re: [tahoe-dev] Tahoe performance

2009-02-19 Thread Shawn Willden
On Wednesday 18 February 2009 08:44:51 pm Brian Warner wrote: True, but the reason our MDMF plans are different (storing the full/deltas in the file object, rather than the directory) is to allow someone to share a single mutable file, without sharing the directory that contains it. Mmm,

Re: [tahoe-dev] [tahoe-lafs] #641: tahoe backup should be able to backup symlinks

2009-02-24 Thread Shawn Willden
On Tuesday 24 February 2009 11:36:12 am tahoe-lafs wrote: Here's a patch which makes tahoe backup ignore symlinks. FYI, that's my backup tool's approach as well. For symlinks I store the symlink metadata (including the link target) but don't traverse the link. On a related issue, I also

Re: [tahoe-dev] [tahoe-lafs] #628: mtime and ctime: I don't think that word means what you think it means.

2009-02-26 Thread Shawn Willden
On Wednesday 25 February 2009 09:02:22 am tahoe-lafs wrote: Also, nobody should do this until Brian comments on this ticket about my proposal that we stop reading the ctime from the local filesystem, or that if we do read the ctime from the local filesystem, that we then switch on platform

Re: [tahoe-dev] buildbot update

2009-03-09 Thread Shawn Willden
On Monday 09 March 2009 03:47:03 pm Brian Warner wrote: For this purpose, we might want to upload them to the allmydata prodgrid instead.. might be more reliable and less stall-sensitive. I dunno. If we want to use the prodgrid, is there a way my box can use the production helper? That

Re: [tahoe-dev] Josh's would-be GSoC proposal

2009-04-04 Thread Shawn Willden
On Friday 03 April 2009 05:25:45 pm zooko wrote: Shawn looked at the The Black Swan, but I think he decided that he knew where it was going after reading only a part of it. No, I didn't. I just commented that in the introduction the author seemed to miss the forest for the trees. After

Re: [tahoe-dev] questions about development priorities for Tahoe-LAFS, summer 2009

2009-04-22 Thread Shawn Willden
On Wednesday 22 April 2009 12:14:45 pm Zooko O'Whielacronx wrote: Shawn Willden is presumably still in his secret underground lair in Utah hacking on GridBackup. He hasn't been very active on this list, but he has published his git repo... Hm, looks like he hasn't updated that git repo

Re: [tahoe-dev] Request for hash-dependency in Tahoe security.

2009-04-29 Thread Shawn Willden
On Wednesday 29 April 2009 11:51:43 am Nathan wrote: I haven't yet read this SHA-1 collisions now 2^52 paper: A paper answering the questions you ask would still be of great interest and value, but I believe Tahoe uses SHA-256, so weaknesses in SHA-1 aren't a concern unless the techniques can

Re: [tahoe-dev] Fwd: [Python-Dev] PEP 383 and GUI libraries

2009-05-03 Thread Shawn Willden
On Saturday 02 May 2009 08:01:45 pm Zooko O'Whielacronx wrote: The best thing to do for Tahoe is likely to define valid UTF-8 as the encoding for all Tahoe file names and to raise an error when people try to stick anything else in there. The problem with this argument is that it pre-supposes

Re: [tahoe-dev] Fwd: [Python-Dev] PEP 383 and GUI libraries

2009-05-03 Thread Shawn Willden
On Sunday 03 May 2009 09:12:56 am Zooko O'Whielacronx wrote: On May 3, 2009, at 8:49 AM, Shawn Willden wrote: On Saturday 02 May 2009 08:01:45 pm Zooko O'Whielacronx wrote: The best thing to do for Tahoe is likely to define valid UTF-8 as the encoding for all Tahoe file names and to raise

Re: [tahoe-dev] [tahoe-lafs] #629: 'tahoe backup' doesn't tolerate 8-bit filenames

2009-05-24 Thread Shawn Willden
On Sunday 24 May 2009 11:03:56 am Zooko Wilcox-O'Hearn wrote: It sounds to me like your design will store enough information to enable any possible future improvement, but that the first version will give mojibake results if you backup from a linux system (even one with all filenames correctly

Re: [tahoe-dev] [tahoe-lafs] #684: let client specify the encryption key

2009-05-24 Thread Shawn Willden
On Sunday 24 May 2009 06:23:22 pm Zooko Wilcox-O'Hearn wrote: On May 24, 2009, at 14:48 PM, Shawn Willden wrote: I need a way to map content hashes to read caps, because my backup log contains content hashes. I can't put read caps in the backup log for a couple of reasons which should

Re: [tahoe-dev] [tahoe-lafs] #684: let client specify the encryption key

2009-05-24 Thread Shawn Willden
On Sunday 24 May 2009 07:00:06 pm Shawn Willden wrote: To compute a read cap you have to (a) pick your FEC parameters and (b) do all of the FEC work so you can generate the UEB so you can hash it. Oh, one more minor issue: Tahoe doesn't provide an easy way for me to perform these calculations

[tahoe-dev] GridBackup moved to Github

2009-05-28 Thread Shawn Willden
Still not usable, but getting closer. Another developer in my area has decided to pitch in, so I put my repository up on Github to make it a little easier to collaborate (so far Github rocks, BTW; I'm very impressed). For anyone interested, the code is at:

Re: [tahoe-dev] GridBackup moved to Github

2009-05-29 Thread Shawn Willden
On Thursday 28 May 2009 04:23:01 pm Shawn Willden wrote: For anyone interested, the code is at: http://github.com/swillden/GridBackup/tree Or, if you want to grab a copy with git, you can clone: git://github.com/swillden/GridBackup.git Just to confuse matters, I changed my

Re: [tahoe-dev] GridBackup moved to Github

2009-05-30 Thread Shawn Willden
On Saturday 30 May 2009 07:58:31 am Arthur Lutz wrote: I tried it out, here a few things : Thanks for looking at it. As you can plainly see, it's far from ready for use. However, if you're willing to do some testing, I'd like to see your results. * no README at the root of project, it

Re: [tahoe-dev] GridBackup moved to Github

2009-05-30 Thread Shawn Willden
On Saturday 30 May 2009 07:58:31 am Arthur Lutz wrote: * I tried backup.py with -h or --help and it just started scanning my entire system, had to CTRL-C to stop it because I didn't know what it was doing. * the tests are hard wired to the default python, so since i'm on ubuntu and only

[tahoe-dev] My buildbot

2009-06-04 Thread Shawn Willden
Hey, I was just looking at the buildbot status page, and I notice that my buildbot is mislabeled or, more precisely, that the machine has changed, but the label has not. The label says that the machine is running Intrepid, but it's running Jaunty. In general, it will always be on the latest

Re: [tahoe-dev] buildbots (Re: My buildbot)

2009-06-05 Thread Shawn Willden
On Friday 05 June 2009 07:21:53 am Shawn Willden wrote: Where do I find that file? NM. I found it. I'm adding the information now. Shawn. ___ tahoe-dev mailing list tahoe-dev@allmydata.org http://allmydata.org/cgi-bin/mailman/listinfo/tahoe

Re: [tahoe-dev] buildbots (Re: My buildbot)

2009-06-05 Thread Shawn Willden
On Friday 05 June 2009 09:12:07 am Brian Warner wrote: Shawn Willden wrote: On Friday 05 June 2009 07:21:53 am Shawn Willden wrote: Where do I find that file? NM. I found it. I'm adding the information now. BTW, bounce the buildslave after changing either info/host or info/admin

Re: [tahoe-dev] Windwos installation problems

2009-06-12 Thread Shawn Willden
On Friday 12 June 2009 11:23:06 am Zooko O'Whielacronx wrote: It is a solution to a slightly different problem, though -- even if we have a good end-user-oriented installer integrated into Tahoe proper, we still want it to be easy and portable to build from source. Agreed. Both are needed.

Re: [tahoe-dev] Mutable or immutable?

2009-06-14 Thread Shawn Willden
On Sunday 14 June 2009 07:39:50 pm Shawn Willden wrote: These files are crucial, so I'm also a little concerned that there be no way to end up with neither. That's not very clear, is it? What I meant to say was that I'm concerned that there be no way to end up with neither the old version nor

[tahoe-dev] Is it useful to truncate mutable files?

2009-06-17 Thread Shawn Willden
Since garbage collection is still some time distant it seems to me that there might be value in trying to reduce storage wasted by deleted files. Without GC, files don't actually disappear, except perhaps eventually by server attrition. However, it seems to me that it is possible to reduce

Re: [tahoe-dev] Is it useful to truncate mutable files?

2009-06-17 Thread Shawn Willden
On Wednesday 17 June 2009 01:19:25 am Brian Warner wrote: But if you've created a lot of large mutable files and then feel like deleting them, then yeah, truncating them first will save you backend space. One more question: I know the SDMF design is intended for smallish files, but is there

[tahoe-dev] GridBackup functional (barely)

2009-07-03 Thread Shawn Willden
After far too long, GridBackup works, for a suitable definition of works. Specifically, it scans the file system, produces and stores the backup logs (full and incremental) containing file metadata, and generates a queue of upload jobs to back up the actual file content. It also backs up the

[tahoe-dev] Client/helper RAM usage

2009-07-03 Thread Shawn Willden
Are files being processed (FEC-encoded, etc.) loaded into RAM? Is it possible to upload a 2 GiB file on a machine with only 512 MiB RAM? Is there any way to estimate the memory consumption that will be required? I'm thinking about putting a client or maybe a helper on a Linux Virtual Server

[tahoe-dev] UEB hash size

2009-07-12 Thread Shawn Willden
Hi guys, What's the rationale for including the full 256-bit UEB hash in the CHK URI? Those URIs could be shortened considerably by truncating it to, say, 128 bits. How difficult would it be to allow Tahoe to operate with either full UEB hashes or abbreviated hashes? I'm thinking I could

Re: [tahoe-dev] UEB hash size

2009-07-12 Thread Shawn Willden
On Sunday 12 July 2009 07:37:12 pm Zooko Wilcox-O'Hearn wrote: I really don't like the idea of adding an insecure mode, if we can help it. Could you explain more -- perhaps even quantify -- what impact the size of the cap has on your system? Hopefully there's a solution which is both secure

Re: [tahoe-dev] UEB hash size

2009-07-12 Thread Shawn Willden
On Sunday 12 July 2009 08:50:43 pm Zooko Wilcox-O'Hearn wrote: By the way, I'm sitting on a good idea that I haven't finished writing up yet for how to combine the encryption key and the integrity-checking hash together so that you have only one value (perhaps of size 256 bits) instead of two

Re: [tahoe-dev] UEB hash size

2009-07-12 Thread Shawn Willden
On Sunday 12 July 2009 09:31:28 pm Zooko Wilcox-O'Hearn wrote: Can you characterize it as something like the difference between a 24-byte immutable read cap and a 32-byte and a 64-byte is X times as many trie node fetches for an average file download? Actually, it's not an issue on the

Re: [tahoe-dev] How does file deletion work?

2009-07-13 Thread Shawn Willden
On Monday 13 July 2009 10:29:31 am Rufus Pollock wrote: Right: so deleting from a directory means unlinking from that directory. If it is linked from elsewhere it won't be deleted and even if i unlink from all directories if there's a read-cap out there it will still stay so how do I *really*

Re: [tahoe-dev] Status of Windows package?

2009-07-21 Thread Shawn Willden
On Tuesday 21 July 2009 09:03:42 am rob kinninmont wrote: At Allmydata we used py2exe to build a (suite of) .exe file(s) including all the bundling of python libraries etc. It's not perfect (it doesn't integrate terribly well with .egg files for instance) but it works pretty well. I have

Re: [tahoe-dev] HTTP and HTTPS access to a node

2009-07-21 Thread Shawn Willden
On Tuesday 21 July 2009 10:04:09 am Brian Warner wrote: Is it possible to configure a node to listen via both HTTP and HTTPS at the same time? Not without code changes. They wouldn't be too hard to implement, though. I'd suggest making web.port take a comma-separated list of strports,

Re: [tahoe-dev] darcs patch: Client support for listening on multiple ports

2009-07-22 Thread Shawn Willden
On Wednesday 22 July 2009 04:22:20 pm Shawn Willden wrote: Wed Jul 22 16:16:54 MDT 2009 Shawn Willden shawn-ta...@willden.org * Client support for listening on multiple ports This still needs unit tests. If someone can point me in the right direction, I'll implement them and then attach

Re: [tahoe-dev] seeking volunteer for an hour's labor Re: [tahoe-lafs] #773: run through installation on Windows

2009-07-25 Thread Shawn Willden
On Saturday 25 July 2009 01:02:00 pm Zooko Wilcox-O'Hearn wrote: Folks: Anybody out there who loves Windows? Okay, how about this: anybody out there who has a working Windows system and is willing to use it for an hour or so to make sure Tahoe-LAFS installs cleanly? I'll fire up a clean XP

Re: [tahoe-dev] seeking volunteer for an hour's labor Re: [tahoe-lafs] #773: run through installation on Windows

2009-07-25 Thread Shawn Willden
On Saturday 25 July 2009 03:53:11 pm Shawn Willden wrote: On Saturday 25 July 2009 01:02:00 pm Zooko Wilcox-O'Hearn wrote: Folks: Anybody out there who loves Windows? Okay, how about this: anybody out there who has a working Windows system and is willing to use it for an hour or so

Re: [tahoe-dev] seeking volunteer for an hour's labor Re: [tahoe-lafs] #773: run through installation on Windows

2009-07-26 Thread Shawn Willden
On Saturday 25 July 2009 11:22:54 pm Shawn Willden wrote: At this point, it looks like my only option is to (a) install cygwin, (b) get python2.5 sources, (c) get python-mingw, (d) build python with mingw and then (e) retry with the mingw-built python. Lots of googling turned up this page

Re: [tahoe-dev] seeking volunteer for an hour's labor Re: [tahoe-lafs] #773: run through installation on Windows

2009-07-26 Thread Shawn Willden
On Sunday 26 July 2009 09:28:30 am Shawn Willden wrote: Ah ha! I looked on that page and there is no egg for python 2.5 win32. There is for python 2.6. I'm off to try that. I'd rather use 2.6 anyway; I only used 2.5 because that's what the installation directions said. Didn't work with 2.6

Re: [tahoe-dev] seeking volunteer for an hour's labor Re: [tahoe-lafs] #773: run through installation on Windows

2009-07-26 Thread Shawn Willden
On Sunday 26 July 2009 03:46:53 pm Zooko Wilcox-O'Hearn wrote: Perhaps you should get a version of pyOpenSSL from after the merge of the working-windows-build branch. How would I get the tahoe build process to use that version? Shawn. ___

Re: [tahoe-dev] darcs patch: Lossmodel updates

2009-07-27 Thread Shawn Willden
On Monday 27 July 2009 04:21:39 pm Shawn Willden wrote: Wed Jul 22 12:20:52 MDT 2009 Shawn Willden shawn-ta...@willden.org * Lossmodel updates Various improvements to the lossmodel paper. Sorry for the size. It seems to me that there's little value in having the paper in the tree

Re: [tahoe-dev] darcs patch: Lossmodel updates

2009-07-28 Thread Shawn Willden
On Tuesday 28 July 2009 02:54:47 pm Brian Warner wrote: On Jul 27, 2009, at 6:30 PM, Shawn Willden shawn-ta...@willden.org wrote: Sorry for the size. It seems to me that there's little value in having the paper in the tree without a PDF of it that people can read. I go back and forth

Re: [tahoe-dev] Tahoe on NAS drives

2009-08-01 Thread Shawn Willden
On Saturday 01 August 2009 10:46:26 am Jan-Benedict Glaw wrote: Sure--that's another possibility. Examples in this group are the Buffalo NAS products, NSLU2, ... However, keep in mind that they typically have quite limitd RAM, so letting them do erasure-encoding on large files will be a pain

[tahoe-dev] Python 2.4 issue

2009-08-02 Thread Shawn Willden
Trying to install 1.5.0 on a box that only has Python 2.4 turned up an issue, pysqlite has some uses of the with keyword, which isn't available in 2.4. This causes build errors. Shawn. ___ tahoe-dev mailing list tahoe-dev@allmydata.org

Re: [tahoe-dev] cleversafe says: 3 Reasons Why Encryption is Overrated

2009-08-05 Thread Shawn Willden
My two cents: On Wednesday 05 August 2009 09:28:59 am Zooko Wilcox-O'Hearn wrote: So, I'm going to try to throw out a few short pieces which hopefully each make sense. I think your comments were very clear. Third, I don't understand why Cleversafe documents claim that public key

Re: [tahoe-dev] No module named python....

2009-08-06 Thread Shawn Willden
On Thursday 06 August 2009 06:53:41 pm Brian Warner wrote: Could sys.path be different on your box? Could be. I don't know how it gets set. Or maybe Twisted isn't installed on the system, so the /var/lib entry shows up earlier and causes the problem? Twisted is installed on the system.

Re: [tahoe-dev] Why no error message when I have only one storage node?

2009-08-10 Thread Shawn Willden
On Monday 10 August 2009 09:48:01 am Zooko Wilcox-O'Hearn wrote: Pretty kloogey! Ahem. I don't normally engage in correcting peoples' spelling, but this one just can't be let go. The word is kludgey. ;-) Note that there is a clump of tickets related to #699. I think my personal favorite

[tahoe-dev] Observations on Tahoe performance

2009-08-10 Thread Shawn Willden
While tuning my GridBackup uploader, I've noticed some things about the perfomance of immutable uploads through a helper that puzzle me. Note that I'm only doing immutable uploads, there are no mutable files, not even dirnodes. First, I should say that I haven't yet instrumented the uploader,

Re: [tahoe-dev] timeouts

2009-08-10 Thread Shawn Willden
On Monday 10 August 2009 01:43:43 pm Brian Warner wrote: If we don't make it to the end of the upload with enough shares in place (shares of happiness), we tell the user that the upload failed, so they can try it again (which should re-do peer-selection and thus pick up a different set of

Re: [tahoe-dev] Observations on Tahoe performance

2009-08-12 Thread Shawn Willden
On Wednesday 12 August 2009 08:02:13 am Francois Deppierraz wrote: Hi Shawn, Shawn Willden wrote: The first observation is that when uploading small files it is very hard to keep the upstream pipe full, regardless of my configuration. My upstream connection is only about 400 kbps, but I

[tahoe-dev] Selecting optimal FEC parameters (also, idea for new peer selection policy)

2009-08-12 Thread Shawn Willden
A key question for anyone setting up a small grid is what FEC parameters to use. In order to conserve space and bandwidth, you want to minimize FEC expansion but increasing reliability requires increasing FEC expansion. There are also download performance implications to consider. What I'd

Re: [tahoe-dev] [tahoe-lafs] #778: shares of happiness is the wrong measure; servers of happiness is better

2009-08-12 Thread Shawn Willden
On Wednesday 12 August 2009 09:22:00 am tahoe-lafs wrote: Comment(by swillden): I have an alternative idea which I described on the mailing list: Note that I'm leaving on vacation in a couple of hours, and I won't have access to Internet (nor cellphone, though I will have my laptop and,

Re: [tahoe-dev] Selecting optimal FEC parameters (also, idea for new peer selection policy)

2009-08-12 Thread Shawn Willden
On Wednesday 12 August 2009 09:49:37 am Zooko Wilcox-O'Hearn wrote: Interesting ideas, Shawn. I'll just respond to a couple of details, below, plus I'll say that I think Kevan should proceed with his plan to implement servers of happiness now, as it is a small enough change that he can finish

Re: [tahoe-dev] So how do *you* manage your keys, then? Re: cleversafe says: 3 Reasons Why Encryption isOverrated

2009-08-18 Thread Shawn Willden
On Tuesday 18 August 2009 01:03:42 am Jason Resch wrote: Users of Tahoe-LAFS are faced with a difficult choice: 1. Keep data highly confidential, by not making copies of the identifiers 2. Keep data highly reliable, by replicating the identifiers It is certainly possible, and not even

[tahoe-dev] IPv6 support

2009-08-21 Thread Shawn Willden
It seems that I remember already asking about this, but I can't find it in the mailing list archives, so maybe I just thought about asking, but didn't actually do it :-) What is the status of Tahoe on an IPv6 network? Is it known not to work? Is there some reason it shouldn't work? Has it

Re: [tahoe-dev] [tahoe-lafs] #778: shares of happiness is the wrong measure; servers of happiness is better

2009-08-23 Thread Shawn Willden
On Saturday 22 August 2009 01:41:22 pm tahoe-lafs wrote: I imagine that Shawn's efforts will result in a web page on the client node which has knobs for encoding parameters and displays probable reliability properties as an output, to help users make those decisions. Or the other way

Re: [tahoe-dev] IPv6 support

2009-08-23 Thread Shawn Willden
On Saturday 22 August 2009 04:37:49 pm Brian Warner wrote: So, not a deep problem, but enough work to be annoying, especially if you have to start with Twisted. I'd love to see foolscap and tahoe be v6-capable.. I don't have a v6 tunnel set up myself, but I'd be willing to figure that part out

[tahoe-dev] Idea for grid-based aliases

2009-08-24 Thread Shawn Willden
There has been some discussion of ways to create aliases for Tahoe URLs, in order to allow users to provide shorter, nicer URLs. I suggested (though I'm sure I wasn't the first) the idea of variable-security aliases that would allow the user to ask Tahoe to generate aliases of arbitrary

Re: [tahoe-dev] Selecting optimal FEC parameters (also, idea for new peer selection policy)

2009-08-24 Thread Shawn Willden
On Wednesday 19 August 2009 03:24:32 pm Brian Warner wrote: It would be neat if you could put your share-placement decisions into a function and get back a couple of predictive numbers: the file-survival probability, the predicted file-download speed (from various places), etc, then compare

Re: [tahoe-dev] Observations on Tahoe performance

2009-08-25 Thread Shawn Willden
On Tuesday 25 August 2009 02:59:10 am Brian Warner wrote: The actual call that Foolscap makes is a transport.write(), which is implemented in Twisted by appending the outbound data to a list and marking the socket as writeable (so that select() or poll() will wake up the process when that data

Re: [tahoe-dev] automatic repair/renewal : where should it go?

2009-08-29 Thread Shawn Willden
On Thursday 27 August 2009 02:18:52 am Brian Warner wrote: I'd like to provoke a discussion about file repair, to see if we can make some progress on improving the process. I'd like to confuse your question by adding more issues to consider ;-) One thing that has been concerning me ever since

Re: [tahoe-dev] automatic repair/renewal : where should it go?

2009-08-29 Thread Shawn Willden
Oops, forgot to address a couple of questions. On Thursday 27 August 2009 02:18:52 am Brian Warner wrote: So.. does this seem reasonable? Can people imagine what the schema of this persistent store would look like? What sort of statistics or trends might we want to extract from this database,

Re: [tahoe-dev] a crypto puzzle about digital signatures and future compatibility

2009-08-31 Thread Shawn Willden
On Thursday 27 August 2009 07:14:32 pm James A. Donald wrote: This is almost precisely the example problem I discuss in http://jim.com/security/prefix_free_number_encoding.html That's very interesting. There are some things in the explanation that aren't entirely clear to me, though. Do you

[tahoe-dev] Troubleshooting node connectivity

2009-09-08 Thread Shawn Willden
While checking my backups I've noticed a problem: two of the nodes in my friendnet aren't reachable by others. I'm pretty sure it's not a NAT/firewall issue, because those nodes ARE reachable by two other nodes. Currently there are seven nodes in my friendnet, three of which are rarely

Re: [tahoe-dev] Troubleshooting node connectivity

2009-09-08 Thread Shawn Willden
On Tuesday 08 September 2009 05:40:56 pm Peter Secor wrote: Hi Shawn, I've had a Windows Tahoe node running pretty reliably as a service for quite some time. You can download the stable production client here: http://allmydata.com/downloads/Allmydata321Setup.exe Very cool! I've been trying

Re: [tahoe-dev] Troubleshooting node connectivity

2009-09-08 Thread Shawn Willden
On Tuesday 08 September 2009 08:40:39 pm Zooko O'Whielacronx wrote: On Tue, Sep 8, 2009 at 6:27 AM, Shawn Willdensh...@willden.org wrote: Any idea what the problem might be?  What can I do to get more visibility into what connections Tahoe is attempting to make (and failing)? I'll try to

Re: [tahoe-dev] Troubleshooting node connectivity

2009-09-09 Thread Shawn Willden
On Wednesday 09 September 2009 01:17:18 am Brian Warner wrote: Remember that Tahoe nodes establish connections bidirectionally, and that once the A-B connection is made, it can also be used B-A. So if your two Ubuntu boxes are behind firewalls, then in fact maybe nobody can establish a

Re: [tahoe-dev] Troubleshooting node connectivity

2009-09-09 Thread Shawn Willden
On Tuesday 08 September 2009 05:40:56 pm Peter Secor wrote: Hi Shawn, I've had a Windows Tahoe node running pretty reliably as a service for quite some time. You can download the stable production client here: http://allmydata.com/downloads/Allmydata321Setup.exe Just edit the introducer and

Re: [tahoe-dev] Upgrading tahoe

2009-09-10 Thread Shawn Willden
On Thursday 10 September 2009 06:41:29 am Dimitri Yioulos wrote: I've installed version 1.4.1 on an OpenSolaris box. I'd like to move to version 1.5. Is there a method to upgrade the current 1.4.1 to 1.5 so that I don't loose my configuration? Just install 1.5.0. Your configuration won't be

Re: [tahoe-dev] Troubleshooting node connectivity

2009-09-11 Thread Shawn Willden
On Thursday 10 September 2009 04:25:01 pm Shawn Willden wrote: I also discovered the 'advertised_ip_addresses' file in the 1.2.0 configuration.txt file, but it doesn't appear to work. Addresses I add there don't show up in the announcement FURL, according to what I see from 'flogtool tail

Re: [tahoe-dev] Newbie help

2009-09-14 Thread Shawn Willden
I don't have a lot of familiarity with the CLI tool, except to say that it is the right tool for what you're trying to do, at least for now. I'm working on another backup tool which I think is much better, but I'm not ready to recommend it to others yet, unless your interest is in helping me

Re: [tahoe-dev] Elk Point design for mutable, add-only, and immutable files

2009-09-16 Thread Shawn Willden
On Tuesday 15 September 2009 09:57:15 pm David-Sarah Hopwood wrote: http://jacaranda.org/tahoe/mutable-addonly-elkpoint-3.svg I'll explain it in more detail tomorrow. I dropped the ability to have write-only caps that do not allow reading, so only needed symmetric encryption. One thing that

[tahoe-dev] Removing the dependency of immutable read caps on UEB computation

2009-10-02 Thread Shawn Willden
I'd like to have a little discussion on whether or not it makes sense in the new immutable cap design to remove the dependency on UEB computation. As background for any who aren't familiar with it, and to confirm my own understanding, the UEB, or URI Extension Block, is a block of hashes that

Re: [tahoe-dev] Removing the dependency of immutable r ead caps on UEB computation

2009-10-04 Thread Shawn Willden
On Saturday 03 October 2009 01:26:16 am Brian Warner wrote: Incidentally, we removed 1 and 2 forever ago, to squash the partial-information-guessing-attack. Makes sense. The diagrams in the docs should be updated. To address these issues, I propose splitting the UEB into two parts

Re: [tahoe-dev] Removing the dependency of immutable read caps on UEB computation

2009-10-04 Thread Shawn Willden
On Sunday 04 October 2009 08:52:38 am Zooko O'Whielacronx wrote: Interesting. The goal is to be able to upload a file with a different K encoding parameter (num shares require to reconstruct) but the same plaintext and the same (?) encryption key and have it match the same immutable file cap?

Re: [tahoe-dev] Removing the dependency of immutable read caps on UEB computation

2009-10-04 Thread Shawn Willden
On Sunday 04 October 2009 02:25:53 pm Brian Warner wrote: So, one suggestion that follows would be to store the immutable share-update cap in the same dirnode column that contains writecaps for mutable files. Perhaps. I think re-encoding caps would have more a more specialized purpose. They

[tahoe-dev] Share rebalancing

2009-10-10 Thread Shawn Willden
I noticed this morning that for the last couple of days my backups have been going nowhere. I have a small friendnet, and for some reason all of the other nodes went down at roughly the same time. My backups continued chugging away, but the only storage node available was my local node -- so

Re: [tahoe-dev] Share rebalancing

2009-10-10 Thread Shawn Willden
On Saturday 10 October 2009 11:24:18 am Zooko Wilcox-O'Hearn wrote: On Saturday,2009-10-10, at 6:15 , Shawn Willden wrote: I have a small friendnet, and for some reason all of the other nodes went down at roughly the same time. My backups continued chugging away, but the only storage node

Re: [tahoe-dev] Python beginner trying to build fordevelopmenttesting

2009-10-19 Thread Shawn Willden
On Monday 19 October 2009 03:56:34 pm Vanderlinden, Chris wrote: Thank you for clearing that up, makes a lot more sense now. So the goal on my end seems to be as such: Install Python Install all dependencies for tahoe Install / build tahoe-lafs One other thing you'll need to do to make it

Re: [tahoe-dev] detecting weak uploads and share rebalancing Re: Share rebalancing

2009-10-19 Thread Shawn Willden
On Monday 19 October 2009 03:36:44 pm Zooko Wilcox-O'Hearn wrote: 4. This thread started because Shawn Willden needed to do some mucking about with his shares, and the permute-per-fileid feature makes it harder for him to muck his shares. This is a real live example of my argument (in e.g

Re: [tahoe-dev] notes about DIR2:CHK deep immutability (maybe Tahoe-LAFS v1.6)

2009-10-21 Thread Shawn Willden
On Tuesday 20 October 2009 11:27:21 pm Zooko Wilcox-O'Hearn wrote: When you create a DIR2:IMM, giving it a set of (childname, childcap) tuples, it should raise an exception if any childcap is not immutable. Cool! One of the things I've been wishing for is a way to create a dirnode already

Re: [tahoe-dev] Tahoe-lafs and nodes behind NAT (behind another NAT)

2009-12-17 Thread Shawn Willden
Another option is to use IPv6. That may seem like crazy talk, but it's not. It's pretty trivial to set up an account with a v6 tunnel broker and to set up a tunnel on WinXP (and, I assume, Vista and 7). Sixxs.net has a graphical utility that supposedly makes it automagic (I haven't tried

Re: [tahoe-dev] Tahoe-lafs and nodes behind NAT (behind another NAT)

2009-12-18 Thread Shawn Willden
On Friday 18 December 2009 08:00:24 am Jody Harris wrote: I'm just desperately brainstorming here. My dreams of my tahoe-lafs grid are still smoldering in the heap that is left after the realization of the problems introduced by non-publics IPs. NAT and the associated loss of the original

Re: [tahoe-dev] Tahoe-lafs and nodes behind NAT (behind another NAT)

2009-12-18 Thread Shawn Willden
On Friday 18 December 2009 10:37:54 pm Jody Harris wrote: If this level of usage is outside the parameters of what tahoe-lafs is trying to address, that's fine. I'm just asking. My opinion is that it's definitely inside the parameters, but I'm not volunteering to do it :-) Shawn

Re: [tahoe-dev] Tahoe-lafs and nodes behind NAT (behind another NAT)

2009-12-20 Thread Shawn Willden
On Sunday 20 December 2009 08:54:10 am Jody Harris wrote: BitTorrent is another open project that has overcome these kinds of problems. Not in my experience... what I see is the various Bittorrent clients I use (mostly Azureus/Vuze, but I've tried several) require me to manually set up port

[tahoe-dev] Test message -- please ignore

2009-12-22 Thread Shawn Willden
Just testing a new mail config. Sorry. ___ tahoe-dev mailing list tahoe-dev@allmydata.org http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev

  1   2   >