AW: [PATCH] [SVN-4655] configure: add zlib discovery through pkg-config

[Osipov, Michael]

> Michael Osipov  writes:
> 
> > Hi folks,
> >
> > attached you'll find a patch for SVN-4655. The configure script was
> > having problems finding zlib even with a prefix because it is located
> > in
> > /usr/local/lib/hpux32 on HP-UX (--with-zlib=/usr/local won't work).
> > Using pkg-config solves the issue.
> >
> > Tested with default and not default locations on FreeBSD 9.3 and HP-UX 
> > 11.31.
> 
> Thanks for the patch!
> 
> I committed it in r1762338, with a minor stylistic follow-up (r1762339).
> I'll propose it for a backport to 1.9.x.
> 

Evgeniy,

spasibo za podderzhku.

M. Osipov

Re: [PATCH] [SVN-4655] configure: add zlib discovery through pkg-config

[Evgeny Kotkov]

Michael Osipov  writes:

> Hi folks,
>
> attached you'll find a patch for SVN-4655. The configure script was having
> problems finding zlib even with a prefix because it is located in
> /usr/local/lib/hpux32 on HP-UX (--with-zlib=/usr/local won't work).
> Using pkg-config solves the issue.
>
> Tested with default and not default locations on FreeBSD 9.3 and HP-UX 11.31.

Thanks for the patch!

I committed it in r1762338, with a minor stylistic follow-up (r1762339).
I'll propose it for a backport to 1.9.x.


Regards,
Evgeny Kotkov

Re: New SHA1 property for nodes returned 'svn ls --xml' invocations.

[Paul Hammant]

Merkle trees / hashes can help a server maintained graph of objects survive 
"Bitrot" 
(http://arstechnica.com/information-technology/2014/01/bitrot-and-atomic-cows-inside-next-gen-filesystems/
- data in SSD or HD being corrupted by (say) nutrinos over time. See also a 
guy/gal lamenting their corrupted photo collection - 
https://blog.barthe.ph/2014/06/10/hfs-plus-bit-rot/)
 
A svn server, in some background activity could detect that bitrot has happened 
by calculating the hashes afresh, then ask a replica for its version of the 
same file@revision to heal it silently.
 
Of course you could take the policy view that subversion should rely on a file 
system that can deliberately repair corruptions -  
https://blogs.oracle.com/timc/entry/demonstrating_zfs_self_healing - though 
BTRFS may also repair corruptions.  If you do it in Svn itself, you could take 
advantage of *ordinary* file systems, and the fact that people wanting their 
(say) photos to survive a house fire, should probably have their in-house 
file-sync server (say Raspberry Pi + SDCard) replicating to something outside 
the house too.

RE: New SHA1 property for nodes returned 'svn ls --xml' invocations.

[Bert Huijben]

> -Original Message-
> From: Daniel Shahaf [mailto:d...@daniel.shahaf.name]
> Sent: maandag 26 september 2016 09:09
> To: Julian Foad 
> Cc: Paul Hammant ; Subversion Development
> 
> Subject: Re: New SHA1 property for nodes returned 'svn ls --xml' invocations.


> What would content hashes provide that comparing node-rev id's would not?

Just one example:
Stability over dump-load and different subversion filesystems (bdb, fsfs, fsx 
and different versions of those)

If we don't need it for very specific new features I think we should keep 
node-rev id's strictly server/repository side, as the moment we expose them we 
have to promise them to be stable over dump-load, which would probably imply 
another level of redirection. And at that point those IDs might not be the most 
efficient format any more.

Bert

Re: New SHA1 property for nodes returned 'svn ls --xml' invocations.

[Paul Hammant]

Replying to various.  I'm making a Dropbox-a-like client that uses
Svn/WebDav/AutoIncrement as the server. Critical design goal - to *not*
have a classic Svn working tree locally.  Think 50GB of binary files sync'd
down to a client, and a wish to not have that take 100GB of local storage.

> What would content hashes provide that comparing node-rev id's would not?

I can client side detect change to a file, without a subversion working
tree. I store the Sha1 as the server had it. I would calculate that for
every file changed via a inotify/FSEvents/ReadDirectoryChangesW
notification mechanism, before pushing up to the svn server (curl push).

I can't calculate a node id on the client side. That's a function of an
actual commit.  I'd need double the storage to maintain a checkout's
working-copy/tree and that defeats a design goal.

Regardless of you folks implementing the server-side hashes or not, I'm
close to completion of a Python3 script that does all the above. It just
has to do calculations as soon as items come down from svn to the client.

> Node-rev id's get changed on every text change, property change, or copy
of the node itself, but aren't changed when a parent of the node
gets copied.

If you implement Sha1 merkle-trees for items held in Svn, please exclude
properties related to merge-tracking from the amalcgum of what you're
hashing.


As an aside, there's a technology called 'Sparkleshare' for Git (& a Git
remote) that does file sync, that I *also* have a pull request in that
introduces Svn as a backend (svn client required; uses Svn working copy) -
https://github.com/hbons/SparkleShare/pull/1721. For extra shits and
giggles I have a Perforce capability under development too -
https://github.com/ph-hs/SparkleShare/tree/perforce.

Note too, I would love it if y'all would circle back to
https://issues.apache.org/jira/browse/SVN-4454 for an implementation.

- Paul

Re: New SHA1 property for nodes returned 'svn ls --xml' invocations.

[Julian Foad]

Daniel Shahaf wrote:

Julian Foad wrote:
> Hi Paul. I'm +1 on the concept that implementing content hashes in
> Subversion would be useful. I think if we were designing Subversion today,
> the question would be "Why on earth wouldn't we design in a Merkle tree
> content hash?" as it is obviously (to those who have already thought about
> it) useful for these sorts of operation, for people building functionality
> on top of Subversion.

I appreciate that that's your opinion, but I'm going to play devil's
advocate and question it.
The only operation one can do with a content hash is compare it to
another content hash.  Our API already has an object with this property:
svn_fs_id_t.  The equality relation of node-rev id's is a refinement of
the equality relation of content hashes: equal node-rev id's imply equal
content hashes, but the converse is not true.

What would content hashes provide that comparing node-rev id's would not?

1. A node-rev id only exists for a tree that has been committed to the 
repository: there is no way to generate a node-rev id for an external tree of 
content client-side. Note what Paul Hammant wrote about the use case:

"I need to compare to a *local* representation of the same tree that's not under 
subversion control"

2. As you point out, equal content does not imply equal node-rev-ids. The large 
doc-string above svn_fs_id_t says it this way:

"note: Commonly, a node revision will have the same content as some other node 
revisions in the same node and in different nodes. [...]"

Thanks for questioning. That draws out some important points.

Regards,
- Julian

Re: Your project VM needs to be migrated.

[Daniel Shahaf]

[CC -= infra]

Johan Corveleyn wrote on Mon, Sep 26, 2016 at 09:46:55 +0200:
> Thanks infra, and sorry for the delay on my side. I had a busy week
> last week and weekend, but should have some more time on a couple of
> evenings this week. I'll try to get things moving again :-).

Thanks, Johan!

> Additional help is certainly welcome. For starters, we'll need a
> recent svn build (perhaps simply our latest official release, 1.9.4)
> on there for the svn-role activities. I have no experience building
> svn on Linux. I can probably manage when starting from INSTALL, but
> I'm sure there are others here that can do this a lot quicker than me
> :-).

Better than INSTALL would be to start from the incumbent/previous VM's
documentations in /repos/private/pmc/subversion/machines.  They should
have a list of exactly which apt-get packages are needed, what arguments
to pass to configure, and so on.

If we're short on time, we can just install whichever svn is in Ubuntu
packages so we have *something* working before the old VM host is
decommissioned, and upgrade to 1.9.4 later.

Cheers,

Daniel

Re: Your project VM needs to be migrated.

[Johan Corveleyn]

On Mon, Sep 26, 2016 at 8:33 AM, Daniel Shahaf  wrote:
> Johan Corveleyn wrote on Wed, Jul 27, 2016 at 17:00:26 +0200:
>> On Wed, Jul 27, 2016 at 4:36 PM, Daniel Shahaf  
>> wrote:
>> > Daniel Shahaf wrote on Sun, Jul 17, 2016 at 14:59:29 +:
>> >> > https://issues.apache.org/jira/browse/INFRA-12289
>> >>
>> >> dev@: Could someone please take point on this?
>> >
>> > Don't speak all at once, now...
>>
>> :-)
>>
>> /me raises hand.
>>
>> I'll try to take a look at the infra ticket and our docs etc tonight
>> or tomorrow, to see what needs to be done from our side.
>
> Johan, infra has responded on the ticket now — are you still available
> to handle whatever needs doing on our side, or should someone else from
> the PMC take over?

Thanks infra, and sorry for the delay on my side. I had a busy week
last week and weekend, but should have some more time on a couple of
evenings this week. I'll try to get things moving again :-).

Additional help is certainly welcome. For starters, we'll need a
recent svn build (perhaps simply our latest official release, 1.9.4)
on there for the svn-role activities. I have no experience building
svn on Linux. I can probably manage when starting from INSTALL, but
I'm sure there are others here that can do this a lot quicker than me
:-).

Anyhow, I'll take another look tonight ...

-- 
Johan

Re: New SHA1 property for nodes returned 'svn ls --xml' invocations.

[Daniel Shahaf]

Julian Foad wrote on Sun, Sep 25, 2016 at 22:30:54 +0100:
> Daniel Shahaf wrote:
> >Paul Hammant wrote:
> >>[...]  It is easiest to
> >>hit up the root note and ask for a sha1, [...]
> >
> >Can you explain more about your use-case?  [...]
> 
> Hi Paul. I'm +1 on the concept that implementing content hashes in
> Subversion would be useful. I think if we were designing Subversion today,
> the question would be "Why on earth wouldn't we design in a Merkle tree
> content hash?" as it is obviously (to those who have already thought about
> it) useful for these sorts of operation, for people building functionality
> on top of Subversion.
> 

I appreciate that that's your opinion, but I'm going to play devil's
advocate and question it.

The only operation one can do with a content hash is compare it to
another content hash.  Our API already has an object with this property:
svn_fs_id_t.  The equality relation of node-rev id's is a refinement of
the equality relation of content hashes: equal node-rev id's imply equal
content hashes, but the converse is not true.

What would content hashes provide that comparing node-rev id's would not?

Cheers,

Daniel
(Node-rev id's( get changed on every text change, property change, or
copy of the node itself, but aren't changed when a parent of the node
gets copied.)

Re: Your project VM needs to be migrated.

[Daniel Shahaf]

Johan Corveleyn wrote on Wed, Jul 27, 2016 at 17:00:26 +0200:
> On Wed, Jul 27, 2016 at 4:36 PM, Daniel Shahaf  
> wrote:
> > Daniel Shahaf wrote on Sun, Jul 17, 2016 at 14:59:29 +:
> >> > https://issues.apache.org/jira/browse/INFRA-12289
> >>
> >> dev@: Could someone please take point on this?
> >
> > Don't speak all at once, now...
> 
> :-)
> 
> /me raises hand.
> 
> I'll try to take a look at the infra ticket and our docs etc tonight
> or tomorrow, to see what needs to be done from our side.

Johan, infra has responded on the ticket now — are you still available
to handle whatever needs doing on our side, or should someone else from
the PMC take over?