[Duplicity-team] [Merge] lp:~ed.so/duplicity/reuse-passphrase-for-signing-fix into lp:duplicity
The proposal to merge lp:~ed.so/duplicity/reuse-passphrase-for-signing-fix into lp:duplicity has been updated. Status: Needs review = Merged For more details, see: https://code.launchpad.net/~ed.so/duplicity/reuse-passphrase-for-signing-fix/+merge/72541 -- https://code.launchpad.net/~ed.so/duplicity/reuse-passphrase-for-signing-fix/+merge/72541 Your team duplicity-team is requested to review the proposed merge of lp:~ed.so/duplicity/reuse-passphrase-for-signing-fix into lp:duplicity. ___ Mailing list: https://launchpad.net/~duplicity-team Post to : duplicity-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~duplicity-team More help : https://help.launchpad.net/ListHelp
Re: [Duplicity-team] [Merge] lp:~mterry/duplicity/tarfile into lp:duplicity
Due to conversations in the mailing list, I've retooled this branch to, instead of using the system tarfile.py, just updating our internal copy to python2.7's version (with changes for 2.4-compatibility). Once we drop 2.4 support, it will be a simple change to drop our internal version and use the system version again. I've run the test suite against this branch (under python 2.4 too), but of course it could use more testing. -- https://code.launchpad.net/~mterry/duplicity/tarfile/+merge/72422 Your team duplicity-team is requested to review the proposed merge of lp:~mterry/duplicity/tarfile into lp:duplicity. ___ Mailing list: https://launchpad.net/~duplicity-team Post to : duplicity-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~duplicity-team More help : https://help.launchpad.net/ListHelp
Re: [Duplicity-team] [Merge] lp:~mterry/duplicity/early-catch-498933 into lp:duplicity
On 23.08.2011 20:25, Kenneth Loafman wrote: The size info is the problem. Each backend returns a list of filenames, but getting size info is problematic. That's going to be a major piece of work to get that info. I am no pythonist per se, but is there a possibility to add a list_extended to backends that would fetch these infos, if possible? So if a backend supports these, they are there in a list structure 0={[name]=,[size]=,...},1={[name]=,[size]=,...},2= if not (or if not implemented by now) only they are missing 0={[name]=},1={[name]=},2= The victim of zero sized files here http://lists.gnu.org/archive/html/duplicity-talk/2011-08/msg00013.html used the ftp backend, which actually could deliver this info. Downloading the file is not a good idea. Performance would take a serious hit, i.e. double the time for backups. If we could get file size info, we would know that the sizes were correct, and that's what we need. agreed, see my verify does this point My guess is that the zero length files are really the last file transmitted before a backup failed. Getting file sizes would help in proving this. exactly my point. ede ...Ken On Tue, Aug 23, 2011 at 1:02 PM, Michael Terry michael.te...@canonical.comwrote: That is a good and sensible idea, but the current backend.list architecture isn't well suited for that, performance wise. (A) We can't ask for information about just one file, we have to list all files each time (B) We can't ask for size information. We'd have to download the file again. Both could be solved by after each upload, just downloading the file again and comparing to the original to 'prove' the round trip worked. That's a decent idea, actually, just seems a lot less performant. I could whip that up, depending on interest. Ken? -- https://code.launchpad.net/~mterry/duplicity/early-catch-498933/+merge/72607https://code.launchpad.net/%7Emterry/duplicity/early-catch-498933/+merge/72607 Your team duplicity-team is requested to review the proposed merge of lp:~mterry/duplicity/early-catch-498933 into lp:duplicity. ___ Mailing list: https://launchpad.net/~duplicity-team Post to : duplicity-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~duplicity-team More help : https://help.launchpad.net/ListHelp -- https://code.launchpad.net/~mterry/duplicity/early-catch-498933/+merge/72607 Your team duplicity-team is requested to review the proposed merge of lp:~mterry/duplicity/early-catch-498933 into lp:duplicity. ___ Mailing list: https://launchpad.net/~duplicity-team Post to : duplicity-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~duplicity-team More help : https://help.launchpad.net/ListHelp
Re: [Duplicity-team] [Merge] lp:~mterry/duplicity/early-catch-498933 into lp:duplicity
true but backends usually list all files anyway. dont they? .. ede They do if you ask them to, and that's the only listing capability the current Duplicity backend.py API has. But if we're talking about adding to that API, I suspect most backends have a query info about a particular file call that doesn't involve listing all files. And when verifying after each volume, that's what we want; we don't need to ask for info on more than just the one volume. -- https://code.launchpad.net/~mterry/duplicity/early-catch-498933/+merge/72607 Your team duplicity-team is requested to review the proposed merge of lp:~mterry/duplicity/early-catch-498933 into lp:duplicity. ___ Mailing list: https://launchpad.net/~duplicity-team Post to : duplicity-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~duplicity-team More help : https://help.launchpad.net/ListHelp
Re: [Duplicity-team] [Merge] lp:~mterry/duplicity/early-catch-498933 into lp:duplicity
On 23.08.2011 21:02, Michael Terry wrote: true but backends usually list all files anyway. dont they? .. ede They do if you ask them to, and that's the only listing capability the current Duplicity backend.py API has. But if we're talking about adding to that API, I suspect most backends have a query info about a particular file call that doesn't involve listing all files. And when verifying after each volume, that's what we want; we don't need to ask for info on more than just the one volume. you are right. let's leave it to the backend. eventually python logic could fish it out of a full listing in case a backend really doesn't support it. seems like ftp and sftp support it that atomically though. we should really implement it. OR at least make resuming optionally. too much cases seem to went wrong sice it was introduced. while i see the advantage, i also see the disadvantage of corrupt backups. and too much users with thin lines do whether verify nor keep their chains short. COMMENTS? back to the issue itself. why not have a get_info() method stuffing all avilable info into a list structure. that way we wont have to reinvent the wheel when theremight be more information necessary. ede -- https://code.launchpad.net/~mterry/duplicity/early-catch-498933/+merge/72607 Your team duplicity-team is requested to review the proposed merge of lp:~mterry/duplicity/early-catch-498933 into lp:duplicity. ___ Mailing list: https://launchpad.net/~duplicity-team Post to : duplicity-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~duplicity-team More help : https://help.launchpad.net/ListHelp
Re: [Duplicity-team] [Merge] lp:~mterry/duplicity/early-catch-498933 into lp:duplicity
ede, making resume optional is a good idea (maybe a --no-resume option). But that's a separate bug/branch. As for get_info() vs get_size(), I'm nervous about getting *all* infor we can. That might be a lot, and some info is more or less expensive on different backends. See, for example, all the info the GIO backend could request of a file: file:///usr/share/doc/libglib2.0-doc/gio/GFileInfo.html I'm sure we could come up with a reasonable subset of metadata to query, but adding more later isn't so hard. What about a get_info() method that returned a dictionary? And for an initial pass, we would only have backends fill out a size entry. That way, if we later discover some piece of metadata we want, we keep the same API and can just edit the backends. Now, as for whether get_info() should return information on a list or a single file... ftp and sftp can operate on multiple files easily, but there would be no savings to do so. We'd ideally want to verify after uploading each file, so we're still making the same number of remote requests. And we're not saving computationally, because getting info on the whole list is asking for more information than we will use. -- https://code.launchpad.net/~mterry/duplicity/early-catch-498933/+merge/72607 Your team duplicity-team is requested to review the proposed merge of lp:~mterry/duplicity/early-catch-498933 into lp:duplicity. ___ Mailing list: https://launchpad.net/~duplicity-team Post to : duplicity-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~duplicity-team More help : https://help.launchpad.net/ListHelp
Re: [Duplicity-team] [Merge] lp:~mterry/duplicity/early-catch-498933 into lp:duplicity
Heh, obviously that GFileInfo link should have been http://developer.gnome.org/gio/stable/GFileInfo.html -- https://code.launchpad.net/~mterry/duplicity/early-catch-498933/+merge/72607 Your team duplicity-team is requested to review the proposed merge of lp:~mterry/duplicity/early-catch-498933 into lp:duplicity. ___ Mailing list: https://launchpad.net/~duplicity-team Post to : duplicity-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~duplicity-team More help : https://help.launchpad.net/ListHelp
Re: [Duplicity-team] [Merge] lp:~mterry/duplicity/early-catch-498933 into lp:duplicity
On 23.08.2011 22:49, Michael Terry wrote: this is a mapped list, right? I'm not familiar with that term, but were you asking what I meant by dictionary? I meant a python dictionary, like: a = {'key': 'value'} a['key'] == 'value' i am no pythonist, so yes a key-value list is called a map in java and was what i meant get_info(path/to/file, (size,last_mod) ) With this API, you'll still have to handle the backend not providing 'last_mod', so why bother telling the backend which bits you want? Backends can just fill in all the bits that duplicity needs (which for right now is just size). because it is the more elegant approach. you were worrying about too much info by default. ok, so we know which info we need, so let's request it. if we can't get it, we'll get a no key=value pair for it, so we can deal with it. performancewise it also makes sense, becuse no info is requested that is not needed. we'll probably end up with backends, that won't/can't tell us the size anyway, so we should design the api with the possibility to deliver a NULL info. we simply cannot know if every backend is going to deliver it. this way we could easily implement a dummy method in backend that returns an empty list and implementing backends can overwrite and return whatever they are capable of is requested. i was just argumenting API-wise, and for the api it'd make sense to be able to return lists instead of atomic data sets. who knows what we'll need it for in the future. I'm only leery of over-engineering. The more we build into the API, the harder it is to write a backend (the ease of which is a nice thing right now). The rest of the current backend API is oriented around single objects. except of list() of course ;) If we don't know why we'd need to be able to query multiple files at once, why optimize for it? that's exactly the reason we don't know! i actually believe that some extra thought in application design will save you the extra hours of doctoring afterwards. so therefor i plead to keep the API open. that does not mean that everyting is to be implemented. it just means that you don't restrict yourself without any need for it. so why restrict the return value to one value set, if we can as well wrap this by default into a list again, just to have the possibility to return a list of dictionaries for several files. in this case it means we add a new interface like get_info() that backends are free to implement. if they do, some extra duplicity functionality kicks in. if not, then it does not ;). another thought: why do we actually log an error here? shouldn't we try num-retries to put the file on the backend and only fail if that does not work out? ede -- https://code.launchpad.net/~mterry/duplicity/early-catch-498933/+merge/72607 Your team duplicity-team is requested to review the proposed merge of lp:~mterry/duplicity/early-catch-498933 into lp:duplicity. ___ Mailing list: https://launchpad.net/~duplicity-team Post to : duplicity-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~duplicity-team More help : https://help.launchpad.net/ListHelp