Re: ra_serf: API and HTTP protocol changes to support delta streaming (was: Re: Proposal: new fsfs.conf properties)

Evgeny Kotkov wrote: > Thank you for this reminder, I have now taken this to a separate thread. Thanks, Evgeny, that all sounds clear now. FWIW, these changes sound fine to me, though my mind-space is far from that side of Subversion so that may not count for much. - Julian

ra_serf: API and HTTP protocol changes to support delta streaming (was: Re: Proposal: new fsfs.conf properties)

Julian Foad writes: > Hi Evgeny. Daniel Shahaf just noticed that r1803143 extended the > svn_delta_editor_t interface. So I took a look and see in the log > message that also: > >> This requires a minor tweak to the Subversion's HTTP protocol, and >> it's the reason why

Re: Proposal: new fsfs.conf properties

Evgeny Kotkov wrote: > I went ahead with implementing a slightly different solution that enables > delta streaming in ra_serf. (Currently, only for "svn import"; commits are > a bit more complicated, but shouldn't be too hard to switch to the new > approach, as all the necessary groundwork is in

Re: Proposal: new fsfs.conf properties

> > A couple of tests show that it's possible to get up to 2x speed improvement > when importing large binary files, although the actual improvement will > vary, depending on the configuration of the server and the client. And, > apart from this, such imports won't be using the free space in

Re: Proposal: new fsfs.conf properties

Evgeny Kotkov writes: > If the svn client would be streaming data as well, it would make the server > and the client process data simultaneously and also avoid the overhead > associated with writing the temporary file to disk. Perhaps, that would > make the

Re: Proposal: new fsfs.conf properties

I think as this thread draws towards a close, that the lz4 modifictions Evgeny made in part address my speed concerns for gigabyte sized files. And that if any more speed for PUT is wanted, that one would set: max-deltification-walk = 0 compression-level = 0 It also looks like the there is

Re: Proposal: new fsfs.conf properties

Philip, > Ah, it's not running out of space in tmp, rather it is failing to find > tmp. There is a behaviour difference between Subversion 1.9 and 1.10 > here: each Apache child process using 1.9 will create two zero-length > files in tmp, the first is APR finding tmp the second is Subversion >

Re: Proposal: new fsfs.conf properties

Paul Hammant writes: > Hey Philip, starting from scratch, I can reproduce "Can't find a temporary > directory" errors in /var/log/apache2/error.log for PUT when the boot drive > is pretty much full, and the repository's drive has 3 spare terabytes. Ah, it's not running out of

Re: Proposal: new fsfs.conf properties

Hey Philip, starting from scratch, I can reproduce "Can't find a temporary directory" errors in /var/log/apache2/error.log for PUT when the boot drive is pretty much full, and the repository's drive has 3 spare terabytes. Can you tell me what to run concurrently that would help you be convinced

Re: Proposal: new fsfs.conf properties

Paul Hammant writes: >> I don't believe Apache/mod_dav_svn uses TMPDIR when processing a PUT. >> > > I can prove that either Apache or ModDavSvn (or the OS) uses TMPDIR during > a PUT of a very large resource. > > Test 1: leave 3GB free on system drive, try to PUT 15GB file

Re: Proposal: new fsfs.conf properties

Philip, > I don't believe Apache/mod_dav_svn uses TMPDIR when processing a PUT. > I can prove that either Apache or ModDavSvn (or the OS) uses TMPDIR during a PUT of a very large resource. Test 1: leave 3GB free on system drive, try to PUT 15GB file thru DAV into it's ultimate destination on a

Re: Proposal: new fsfs.conf properties

Johan Corveleyn writes: > That's good to know. Though that's a bit at odds with what Philip > found when he eliminated deltification on the client side (first by > using svnmucc (delta against empty file only); then by using curl-PUT > + autoversioning (no deltification at

Re: Proposal: new fsfs.conf properties

On Fri, Jul 14, 2017 at 4:31 PM, Evgeny Kotkov wrote: > Johan Corveleyn writes: > >> Nice! >> >> But how about deltification? Has anyone tried / benchmarked the effect >> of turning off deltification (with or without compression), to see >> what

Re: Proposal: new fsfs.conf properties

Philip Martin writes: >> Right now, the support for LZ4 is only implemented for the file:// protocol, >> but support/negotiation for other protocols can be added later. > > That's not right is it? This is a backend change that already affects > all protocols. Indeed,

Re: Proposal: new fsfs.conf properties

Johan Corveleyn writes: > Nice! > > But how about deltification? Has anyone tried / benchmarked the effect > of turning off deltification (with or without compression), to see > what the effect would be on the commit time? Like I suggested in this > thread yesterday (i.e. set

Re: Proposal: new fsfs.conf properties

Paul Hammant writes: > With TMPDIR same drive as Svn's root dir: sometime 6mins, sometimes 14 > mins for the PUT. > With TMPDIR different drive as Svn's root dir: sometime 6mins, sometimes > 14 mins for the PUT. > I can't prove or disprove the benefit of that env var setting.

Re: Proposal: new fsfs.conf properties

Evgeny Kotkov writes: > Right now, the support for LZ4 is only implemented for the file:// protocol, > but support/negotiation for other protocols can be added later. That's not right is it? This is a backend change that already affects all protocols. -- Philip

Re: Proposal: new fsfs.conf properties

Well anecdotally, it wasn't oscillating like that in earlier phases. I'll do p4 obliterate and start over. I mean svn obliterate. Err :-P > > In fact it might be. A full filesystem may have a harder time finding > contiguous empty space, which may translate to more separate > (non-contiguous)

Re: Proposal: new fsfs.conf properties

On 14.07.2017 13:59, Paul Hammant wrote: > I'd like to weigh in with some perf stats on that ever-changing 15GB > file, but I can't: > > With TMPDIR same drive as Svn's root dir: sometime 6mins, sometimes > 14 mins for the PUT. > With TMPDIR different drive as Svn's root dir: sometime 6mins, >

Re: Proposal: new fsfs.conf properties

I'd like to weigh in with some perf stats on that ever-changing 15GB file, but I can't: With TMPDIR same drive as Svn's root dir: sometime 6mins, sometimes 14 mins for the PUT. With TMPDIR different drive as Svn's root dir: sometime 6mins, sometimes 14 mins for the PUT. I can't prove or

Re: Proposal: new fsfs.conf properties

On Fri, Jul 14, 2017 at 1:18 PM, Evgeny Kotkov wrote: > Paul Hammant writes: > >> 1. compression-exempt-suffixes = mp3,mp4,jpeg >> >> 2. deltification-exempt-suffixes = mp3,mp4,jpeg >> >> Regardless of the setting of 'compression-level', #1 above

Re: Proposal: new fsfs.conf properties

Paul Hammant writes: > 1. compression-exempt-suffixes = mp3,mp4,jpeg > > 2. deltification-exempt-suffixes = mp3,mp4,jpeg > > Regardless of the setting of 'compression-level', #1 above two mean certain > things can skip the compression attempt. It must give up at a certain

Re: Proposal: new fsfs.conf properties

Philip Martin wrote on Thu, 13 Jul 2017 21:36 +0100: > Branko Čibej writes: > > > Whether this actually forces a format bump or not is a different > > question which I don't know the answer to. > > I think we would have to bump. The old code could either read the > pre-delta

Re: Proposal: new fsfs.conf properties

Branko Čibej writes: > Whether this actually forces a format bump or not is a different > question which I don't know the answer to. I think we would have to bump. The old code could either read the pre-delta or the post-delta files, depending on how we decided to name

Re: Proposal: new fsfs.conf properties

On 13.07.2017 21:57, Philip Martin wrote: > Branko Čibej writes: > >> Agreed. That's why I didn't propose this. I proposed spawning off a >> daemon that would post-process _one_ commit and exit. It could do all >> sorts of analysis of the content and finding the best (for some

Re: Proposal: new fsfs.conf properties

Branko Čibej writes: > Agreed. That's why I didn't propose this. I proposed spawning off a > daemon that would post-process _one_ commit and exit. It could do all > sorts of analysis of the content and finding the best (for some > definition of "best") source for the delta,

Re: Proposal: new fsfs.conf properties

On 13.07.2017 20:42, Paul Hammant wrote: > A Janitor process that runs independently of Apache isn't a /paradigm > shift/ but it is a huge amount of work to production harden, even if > it was done in today's new hotness Go or Rust. As is the minefield of > maintaining a list that's persistent of

Re: Proposal: new fsfs.conf properties

On 13.07.2017 20:24, Paul Hammant wrote: > Is Mod_Dav_Svn a TSR* thing? Or wholly re-entered for each request > that would require it? > > * https://en.wikipedia.org/wiki/Terminate_and_stay_resident_program > Well since it doesn't run on DOS, it's not TSR :) mod_dav_svn is a shared library

Re: Proposal: new fsfs.conf properties

Is Mod_Dav_Svn a TSR* thing? Or wholly re-entered for each request that would require it? * https://en.wikipedia.org/wiki/Terminate_and_stay_resident_program On Thu, Jul 13, 2017 at 2:11 PM, Branko Čibej wrote: > On 13.07.2017 16:07, Daniel Shahaf wrote: > > On Thu, Jul 13,

Re: Proposal: new fsfs.conf properties

On Thu, Jul 13, 2017 at 06:34:17AM -0400, Paul Hammant wrote: > Or am I really wanting Svn's backend compression and deltification to be > out of band ? > > 1. compression-strategy = defer-to-idle-time-even-if-days-later > 2. deltification-strategy = defer-to-idle-time-even-if-days-later

Re: Proposal: new fsfs.conf properties

Philip Martin writes: > A proper v2 client, such as the Subversion > libraries, will send OPTIONS and parse the response to identify > features. Each transaction has server-side state on disk. A Subversion client will attempt to delete the transaction if an error

Re: Proposal: new fsfs.conf properties

Paul Hammant writes: > Sweet - I'll operationalize that ASAP. The client side of the sync agent > knows whether the item is new or to be changed, already. Protocol details: http://svn.apache.org/repos/asf/subversion/trunk/notes/http-and-webdav/http-protocol-v2.txt POST is

Re: Proposal: new fsfs.conf properties

On Thu, Jul 13, 2017 at 12:34 PM, Paul Hammant wrote: >> >> >> Mmm ... interesting proposition. Also huge -1 because it'd be a really >> awesome abstraction leak. :) > > > > Is there any way to implement the (reasonable in my opinion) feature > request, without transgressing on

Re: Proposal: new fsfs.conf properties

On 13.07.2017 14:12, Paul Hammant wrote: > Sweet - I'll operationalize that ASAP. The client side of the sync > agent knows whether the item is new or to be changed, already. To be quite clear here: If you don't need history, you might consider using a different DAV backend than Subversion.

Re: Proposal: new fsfs.conf properties

Sweet - I'll operationalize that ASAP. The client side of the sync agent knows whether the item is new or to be changed, already. - Paul > > > Sorry I'm gunna be a bit slow here, Philip, but you're saying that > sequence > > of four is a single atomic commit in Subversion? > > Yes. > >

Re: Proposal: new fsfs.conf properties

Paul Hammant writes: > Sorry I'm gunna be a bit slow here, Philip, but you're saying that sequence > of four is a single atomic commit in Subversion? Yes. -- Philip

Re: Proposal: new fsfs.conf properties

I don't mind how it's implemented, and agree this one is weak. I am after a solution though and am open to alternate ideas if compression-exempt-suffixes and its sibling cannot be implemented for some technical reason. On Thu, Jul 13, 2017 at 7:54 AM, Branko Čibej wrote: > On

Re: Proposal: new fsfs.conf properties

On 13.07.2017 13:07, Paul Hammant wrote: > > > The delete+recreate takes the same 5.2s as the original create and is > faster than curl's 6.7s update. > > > New fsfs config prop ? > > delete_and_recreate_instead_of_change_threshold = 50MB You forgot the smiley here, surely you're

Re: Proposal: new fsfs.conf properties

Sorry I'm gunna be a bit slow here, Philip, but you're saying that sequence of four is a single atomic commit in Subversion? - Paul On Thu, Jul 13, 2017 at 7:48 AM, Philip Martin wrote: > Philip Martin writes: > > > The fact that the create

Re: Proposal: new fsfs.conf properties

Philip Martin writes: > The fact that the create is faster than the update means that if > uploading the data is the only thing that matters then breaking the > history and changing the update into a delete+recreate is faster, i.e. > > svnmucc -mm -U

Re: Proposal: new fsfs.conf properties

> > > The delete+recreate takes the same 5.2s as the original create and is > faster than curl's 6.7s update. > > New fsfs config prop ? delete_and_recreate_instead_of_change_threshold = 50MB -ph

Re: Proposal: new fsfs.conf properties

Johan Corveleyn writes: > As I said in my previous response here, I think the reason for Paul to > go for curl+autoversioning is speed, because it eliminates client-side > deltification. It was suggested and demonstrated by Philip here: Using a trunk build, I'm uploading

Re: Proposal: new fsfs.conf properties

> > > > Mmm ... interesting proposition. Also huge -1 because it'd be a really > awesome abstraction leak. :) > Is there any way to implement the (reasonable in my opinion) feature request, without transgressing on programming standards? I think turning off all deltification is a bit brute

Re: Proposal: new fsfs.conf properties

On rationale ... This is a file sync agent. 600 lines of Python that I'm going to open source. The same thing I was working on in 2016 when I chatting on this list. Goals: 1) *not have two copies of everything on the clien*t, 2) have different semantics to pushing back changed files. Secondary

Re: Proposal: new fsfs.conf properties

On Thu, Jul 13, 2017 at 9:36 AM, Johan Corveleyn wrote: > > On Thu, Jul 13, 2017 at 8:27 AM, Branko Čibej wrote: > > On 13.07.2017 04:07, Paul Hammant wrote: > >> I flipped _back_ to Python's requests.put(..) in my solution - from a > >> regular Svn client.

Re: Proposal: new fsfs.conf properties

On Thu, Jul 13, 2017 at 8:27 AM, Branko Čibej wrote: > On 13.07.2017 04:07, Paul Hammant wrote: >> I flipped _back_ to Python's requests.put(..) in my solution - from a >> regular Svn client. That relies on 'autoversioning=on' for it to work >> over DAV, I mean. In that

Re: Proposal: new fsfs.conf properties

Branko Čibej wrote: Paul Hammant wrote: I'd love a --header "svn:message: my message" [...] I'm wondering what you gain [...] Paul, I encourage you to start a new thread if any further discussion of this point is needed. - Julian

Re: Proposal: new fsfs.conf properties

On 13.07.2017 04:07, Paul Hammant wrote: > I flipped _back_ to Python's requests.put(..) in my solution - from a > regular Svn client. That relies on 'autoversioning=on' for it to work > over DAV, I mean. In that configuration it functions like curl, of > course. > > _Commit Messages_ > > I'd love

Re: Proposal: new fsfs.conf properties

On 12.07.2017 21:50, Paul Hammant wrote: > You know, in all seriousness I think the (empty by default) list of > exempted files suffixes the the best way forward. If suffixes is good > enough for Apache itself to use (link provided earlier), it is good > enough in this scenario on the server side

Re: Proposal: new fsfs.conf properties

I flipped *back* to Python's requests.put(..) in my solution - from a regular Svn client. That relies on 'autoversioning=on' for it to work over DAV, I mean. In that configuration it functions like curl, of course. *Commit Messages* I'd love a --header "svn:message: my message" too. I raised it

Re: Proposal: new fsfs.conf properties

On Wed, Jul 12, 2017 at 10:40 PM, Paul Hammant wrote: > I'd be fine with that too if it is also settable via curl --header > "svn:compression: no" for non-client auto-increment operations. I'm wondering whether you'll still need this. You ended up with curl+autoversioning (at

Re: Proposal: new fsfs.conf properties

I'd be fine with that too if it is *also* settable via curl --header "svn:compression: no" for non-client auto-increment operations. On Wed, Jul 12, 2017 at 4:10 PM, Mark Phippard wrote: > I cannot find it in archives so maybe this happened in IRC, but I remember > one time

Re: Proposal: new fsfs.conf properties

I cannot find it in archives so maybe this happened in IRC, but I remember one time suggesting we add a new versioned svn:XXX property to control this. This could then be set by the client based on extension if desired. I recall my suggestion was a compression on|off property that when turned off

Re: Proposal: new fsfs.conf properties

You know, in all seriousness I think the (empty by default) list of exempted files suffixes the the best way forward. If suffixes is good enough for Apache itself to use (link provided earlier), it is good enough in this scenario on the server side of Svn. If the function in question doesn't know

Re: Proposal: new fsfs.conf properties

Branko Čibej wrote on Wed, 12 Jul 2017 12:09 +0200: > I wasn't really proposing to use libmagic on the server. My point is > that instead of using file name suffixes (which the compression and > deltification code don't know about), we'd do some sort of inspection > instead. Detecting ZIP files,

RE: Proposal: new fsfs.conf properties

Hi, From: Johan Corveleyn [mailto:jcor...@gmail.com] > > That's such an easy way to make a malicious client explode the > > repository size. And ... there's realy no reason to complicate. The > > server's storage layer can cheaply do all the necessary checks without > > having to believe the

Re: Proposal: new fsfs.conf properties

On Wed, Jul 12, 2017 at 12:27 PM, Branko Čibej wrote: > On 12.07.2017 12:24, Johan Corveleyn wrote: >> On Wed, Jul 12, 2017 at 12:09 PM, Branko Čibej wrote: >>> On 11.07.2017 22:50, Stefan Sperling wrote: On Tue, Jul 11, 2017 at 09:11:58PM +0200, Branko

Re: Proposal: new fsfs.conf properties

On 12.07.2017 12:24, Johan Corveleyn wrote: > On Wed, Jul 12, 2017 at 12:09 PM, Branko Čibej wrote: >> On 11.07.2017 22:50, Stefan Sperling wrote: >>> On Tue, Jul 11, 2017 at 09:11:58PM +0200, Branko Čibej wrote: Another issue I have with the proposal is the idea to use

Re: Proposal: new fsfs.conf properties

On Wed, Jul 12, 2017 at 12:09 PM, Branko Čibej wrote: > On 11.07.2017 22:50, Stefan Sperling wrote: >> On Tue, Jul 11, 2017 at 09:11:58PM +0200, Branko Čibej wrote: >>> Another issue I have with the proposal is the idea to use file suffixes. >>> That's usually the wrong way to

Re: Proposal: new fsfs.conf properties

On 12.07.2017 12:09, Branko Čibej wrote: > On 11.07.2017 22:50, Stefan Sperling wrote: >> On Tue, Jul 11, 2017 at 09:11:58PM +0200, Branko Čibej wrote: >>> Another issue I have with the proposal is the idea to use file suffixes. >>> That's usually the wrong way to go about things (case in point:

Re: Proposal: new fsfs.conf properties

On 11.07.2017 22:50, Stefan Sperling wrote: > On Tue, Jul 11, 2017 at 09:11:58PM +0200, Branko Čibej wrote: >> Another issue I have with the proposal is the idea to use file suffixes. >> That's usually the wrong way to go about things (case in point: Windows >> does it, with didastrous results).

Re: Proposal: new fsfs.conf properties

On Tue, Jul 11, 2017 at 09:11:58PM +0200, Branko Čibej wrote: > Another issue I have with the proposal is the idea to use file suffixes. > That's usually the wrong way to go about things (case in point: Windows > does it, with didastrous results). It's much better to determine file > format by

Re: Proposal: new fsfs.conf properties

So I'm after a time saving. I'm perfectly happy for the backend to waste space (in my configuration), I just don't want it to take 15 mins to transfer a single 15GB file into Subversion. In my configuration, I'd like to pre-advise Subversion to save as much time as possible for uploads, by

Re: Proposal: new fsfs.conf properties

I'm perfectly happy for the solution to be mime-type based. Maybe we can take the mime-type to *suffix table* from Apache itself to do the translation :- https://svn.apache.org/repos/asf/httpd/httpd/trunk/ docs/conf/mime.types :-P I used it (implicitly) in a Subversion backed wysi-wiki *ten

Re: Proposal: new fsfs.conf properties

On 11.07.2017 15:39, Markus Schaber wrote: > Hi, Paul, > > > From: Paul Hammant [mailto:p...@hammant.org] > >> Markus - I've read your section on deltification and I can see evidence in >> what you wrote that you're concurrently in favor of and against it (the >> file-suffix exclusion idea).

RE: Proposal: new fsfs.conf properties

Hi, Paul, From: Paul Hammant [mailto:p...@hammant.org] > Markus - I've read your section on deltification and I can see evidence in > what you wrote that you're concurrently in favor of and against it (the > file-suffix exclusion idea). Can you re-read and clarify? >> I agree partly.

Re: Proposal: new fsfs.conf properties

Markus - I've read your section on deltification and I can see evidence in what you wrote that you're concurrently in favor of and against it (the file-suffix exclusion idea). Can you re-read and clarify? Thanks, - Paul On Tue, Jul 11, 2017 at 8:53 AM, Markus Schaber

RE: Proposal: new fsfs.conf properties

Hi, Best regards Markus Schaber CODESYS® a trademark of 3S-Smart Software Solutions GmbH Inspiring Automation Solutions 3S-Smart Software Solutions GmbH Dipl.-Inf. Markus Schaber | Product Development Core Technology Memminger Str. 151 | 87439 Kempten | Germany Tel. +49-831-54031-979 | Fax

RE: Proposal: new fsfs.conf properties

Hi, (Sorry, it seems my previous message was sent _very_ prematurely :-( From: Pavel Lyalyakin [mailto:pavel.lyalya...@visualsvn.com] > Hello Paul, > > On Sat, Jul 8, 2017 at 2:51 AM, Paul Hammant wrote: > > > > 1. compression-exempt-suffixes = mp3,mp4,jpeg > > > > 2.

Re: Proposal: new fsfs.conf properties

Hello Paul, On Sat, Jul 8, 2017 at 2:51 AM, Paul Hammant wrote: > > 1. compression-exempt-suffixes = mp3,mp4,jpeg > > 2. deltification-exempt-suffixes = mp3,mp4,jpeg > > Regardless of the setting of 'compression-level', #1 above two mean certain > things can skip the

Proposal: new fsfs.conf properties

1. compression-exempt-suffixes = mp3,mp4,jpeg 2. deltification-exempt-suffixes = mp3,mp4,jpeg Regardless of the setting of 'compression-level', #1 above two mean certain things can skip the compression attempt. It must give up at a certain point right? Same for deltification re #2 I'm