Bug#802919: unison: synchronization incompatibility when built with Ocaml versions pre/post-4.02

[Stéphane Glondu]

On 03/07/2017 21:54, Conrad Hughes wrote:

Ralf> The problem is that any software used to build packages in a
Ralf> distribution (sid, in this case) must also be in the distribution.
Ralf> Hence, providing builds of unison produced with different ocaml
Ralf> versions implies having packages for different ocaml versions
Ralf> in sid.

That's kindof what I thought, but if I understand correctly, unison in
stretch must have been built with a different version of ocaml than the
ocaml that's actually in stretch, which confusingly contradicts this?


Indeed. Unison has been built on 2015-08-18 (on amd64). At that time, 
the current version of OCaml was 4.01.0. So Unison has been built with 
4.01.0. It has never been rebuilt since. Maybe we should have rebuilt it 
at some point...



If I've gotten that right, then the current version of unison on stretch
is built with the "wrong" ocaml (i.e. not stretch's ocaml).  The idea
that it should be built with stretch's ocaml would then argue that
technically unison on stretch needs to be updated to an ocaml-4.02
build.  When that update rolls out, unison on stretch will become
incompatible with everything that it currently works with (except
itself, of course).  So a warning release note would seem appropriate
when that "upgrade" happens..?


Now that stretch is released, the only "upgrade" that can happen is a 
security patch. I think the best we can do is adding a NEWS item when 
that happens...


Care has to be taken during an upgrade of Unison, whatever the release. 
Maybe this should be documented... now that I see it, the README.Debian 
should be updated...



It would be nice if we could persuade upstream to at least give a more
helpful error message when the incompatibility arises.


It would be nice if we could persuade upstream to not rely on OCaml's 
marshalling functions, which is the root of the evil.



Cheers,

--
Stéphane

Bug#802919: unison: synchronization incompatibility when built with Ocaml versions pre/post-4.02

[Conrad Hughes]

Hi Ralf,

Ralf> The problem is that any software used to build packages in a
Ralf> distribution (sid, in this case) must also be in the distribution.
Ralf> Hence, providing builds of unison produced with different ocaml
Ralf> versions implies having packages for different ocaml versions
Ralf> in sid.

That's kindof what I thought, but if I understand correctly, unison in
stretch must have been built with a different version of ocaml than the
ocaml that's actually in stretch, which confusingly contradicts this?
Stretch's unison is only compatible with other pre-ocaml-4.02 unisons,
which suggests it was built with ocaml-4.01 — but stretch's ocaml is
4.02.3-9?  This would seem to be confirmed by the fact that I had to
downgrade ocaml to 4.01 on one of my (non-Debian, more recently updated)
servers when building unison there, so that unison on my jessie and
stretch machines could talk to unison on that server.

Sorry if I'm being slow or missing the point here.

If I've gotten that right, then the current version of unison on stretch
is built with the "wrong" ocaml (i.e. not stretch's ocaml).  The idea
that it should be built with stretch's ocaml would then argue that
technically unison on stretch needs to be updated to an ocaml-4.02
build.  When that update rolls out, unison on stretch will become
incompatible with everything that it currently works with (except
itself, of course).  So a warning release note would seem appropriate
when that "upgrade" happens..?

It would be nice if we could persuade upstream to at least give a more
helpful error message when the incompatibility arises.

Conrad

Bug#802919: unison: synchronization incompatibility when built with Ocaml versions pre/post-4.02

[Ralf Treinen]

Hello,

On Mon, Jul 03, 2017 at 03:05:52PM +0100, Conrad Hughes wrote:

> I'm unfamiliar with how the Debian release process works internally, and
> minimising the work involved in packaging unison is of course essential
> — you're already saving all Debian users a huge amount of work by
> packaging it, so thank you for that.
> 
> At the moment because you're building with ocaml-4.01, the published
> unison is backwards compatible but not forwards compatible; can you
> continue to build unison with 4.01 now that Debian's on ocaml-4.02, or
> is that going to become difficult as well?

The problem is that any software used to build packages in a
distribution (sid, in this case) must also be in the distribution.
Hence, providing builds of unison produced with different ocaml
versions implies having packages for different ocaml versions
in sid.

-Ralf.

Bug#802919: unison: synchronization incompatibility when built with Ocaml versions pre/post-4.02

[Conrad Hughes]

Hi Stéphane,

Thanks for your response.

By this:

> > Possibly some pressure should be put on upstream to (1) catch the
> > relevant exception and flag the ocaml version dependency,

I meant that since this is such a non-obvious failure, which should
never happen normally (I've never seen a unison data stream corrupted in
probably fifteen years of using it), that it might be reasonable and
helpful to catch the exception caused by the serialisation change and to
add a message saying "This exception may be because the two versions of
unison were compiled with different versions of OCaml", with a reference
to some relevant documentation (e.g. bug report or release notes).

--

I'm unfamiliar with how the Debian release process works internally, and
minimising the work involved in packaging unison is of course essential
— you're already saving all Debian users a huge amount of work by
packaging it, so thank you for that.

At the moment because you're building with ocaml-4.01, the published
unison is backwards compatible but not forwards compatible; can you
continue to build unison with 4.01 now that Debian's on ocaml-4.02, or
is that going to become difficult as well?

My initial thought process was that producing two conflicting ocaml
builds, say ocaml_pre_402-4.01.0 and ocaml-4.04.2 (or whatever the
actual versions are), which could *not* be installed simultaneously,
would have been the easiest route to automatically building multiple
versions of unison on some kind of automated build farm (including the
ocaml version in unison package names), but that's probably because I
don't understand what the actual release process is like.

So if two ocaml versions is not a practical way of proceeding, do you
think there's any reasonable way of producing pre- and post-402 releases
of unison for Debian in the future?

If not, then would it be possible to include a release note (the kind
that you get shown and emailed when you upgrade) warning users about the
incompatibility?  I realise that arguably this note should not yet have
been sent since Debian's unison is still backwards compatible; perhaps
you already have it planned for the -4.02 changeover..

Regards,
Conrad

Bug#802919: unison: synchronization incompatibility when built with Ocaml versions pre/post-4.02

[Stéphane Glondu]

On 03/07/2017 00:39, Conrad J.C. Hughes (for Debian package stuff) wrote:

Just to add another voice to this, particularly re: the oddness that stretch
ships with ocaml-4.02, while its unison is compiled with -4.01.


Yes, that's unfortunate indeed.


I realise it's heinous, but I do kindof think the solution has to be to ship
both ocaml-4.01 and -4.02 builds of unison (and possibly as a result, ocaml
itself).


Shipping two versions of OCaml is out of the question.


Possibly some pressure should be put on upstream to (1) catch the
relevant exception and flag the ocaml version dependency,


I don't understand.


and (2) (as mentioned
earlier in this thread) make the ocaml version explicit in unison's "-version"
output and (possibly more a Debian packaging issue?) its executable filenames.


That should be possible.


This mess is certainly not Debian's fault, but it would greatly ease others'
experience if both versions of unison were easily available as part of the
official distribution.


Supporting several versions of OCaml in Debian, and therefore your 
proposal, is too much hurdle. I simply lost faith in Unison 
interoperability between difference releases of Debian (let alone with 
other distributions), and there is no will from upstream to improve things.



With a mixture of stretch, jessie and other machines, I've ended up installing
three different builds of unison on one box.


That may happen indeed. I use testing/unstable's binaries everywhere 
myself. I agree it's unconvenient.



Cheers,

--
Stéphane

Bug#802919: unison: synchronization incompatibility when built with Ocaml versions pre/post-4.02

[Conrad J.C. Hughes (for Debian package stuff)]

Package: unison
Version: 2.48.3-1
Followup-For: Bug #802919

Dear Maintainer,

Just to add another voice to this, particularly re: the oddness that stretch
ships with ocaml-4.02, while its unison is compiled with -4.01.

I realise it's heinous, but I do kindof think the solution has to be to ship
both ocaml-4.01 and -4.02 builds of unison (and possibly as a result, ocaml
itself).  Possibly some pressure should be put on upstream to (1) catch the
relevant exception and flag the ocaml version dependency, and (2) (as mentioned
earlier in this thread) make the ocaml version explicit in unison's "-version"
output and (possibly more a Debian packaging issue?) its executable filenames.
This mess is certainly not Debian's fault, but it would greatly ease others'
experience if both versions of unison were easily available as part of the
official distribution.

With a mixture of stretch, jessie and other machines, I've ended up installing
three different builds of unison on one box.

Conrad



-- System Information:
Debian Release: 9.0
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.9.0-3-amd64 (SMP w/8 CPU cores)
Locale: LANG=en_GB.utf8, LC_CTYPE=en_GB.utf8 (charmap=UTF-8), 
LANGUAGE=en_GB.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages unison depends on:
ii  libc6  2.24-11+deb9u1

Versions of packages unison recommends:
ii  openssh-client [ssh-client]  1:7.4p1-10

Versions of packages unison suggests:
pn  unison-all  

-- no debconf information

Bug#802919: unison: synchronization incompatibility when built with Ocaml versions pre/post-4.02

[Niels Elgaard Larsen]

On Thu, 24 Mar 2016 14:14:21 +0100 Baptiste Jonglez 
 wrote:

As a side note, the current state of unison/ocaml on stretch is confusing:
ocaml 4.02.3 is in stretch, but the unison binary package still seems to
be built against ocaml 4.01.  This is non-obvious and I spent quite some
time wondering why unison from stretch did not interoperate with another
unison client built against ocaml 4.02.




So did I. When I figured out, I just had to build unison from source and 
everything worked again.


At least "unison -version" should tell which version of ocaml it uses.

Bug#802919: unison: synchronization incompatibility when built with Ocaml versions pre/post-4.02

[Baptiste Jonglez]

As a side note, the current state of unison/ocaml on stretch is confusing:
ocaml 4.02.3 is in stretch, but the unison binary package still seems to
be built against ocaml 4.01.  This is non-obvious and I spent quite some
time wondering why unison from stretch did not interoperate with another
unison client built against ocaml 4.02.

Of course, rebuilding the unison package against ocaml 4.02 would break
compatibility because of the bigarray change, so this is most likely not
acceptable...

As a workaround, would it be possible to keep this unison package
unchanged (built against ocaml 4.01), and add a new package
unison-ocaml4.02 built against the new ocaml?

Thanks,
Baptiste

Bug#802919: unison: synchronization incompatibility when built with Ocaml versions pre/post-4.02

[Stéphane Glondu]

Le 25/10/2015 07:10, Mark J. Nelson a écrit :
> Due to a change in ocaml's serialization format, unison built with an
> ocaml pre-4.02 can't synchronize with one built with 4.02 and
> later. This causes a compatibility mess, since the usual requirement
> that both endpoints must have the same unison version is no longer
> sufficient: they now have to have both the same unison version *and*
> be built by compatible versions of ocaml, either both pre-4.02 or both
> post-4.02.
> 
> There is some discussion on the unison-users list here:
> http://marc.info/?l=unison-users=142286809310149=2

More precisely, what changed (at least) is the representation of (char)
big arrays:

  http://caml.inria.fr/mantis/view.php?id=6064

The serialization format itself did not change AFAICT.

This is bad news. It means that we can no longer guarantee
stable<->testing compatibility. The trick used so far was to provide
several unison packages, but all compiled with the same OCaml version.
And providing several ocaml packages (that would probably mean the whole
matrix ocaml version x unison version) is out of question.

> I am not sure what the best solution is. For the near-term future,
> there will probably be a significant number of installations needing
> both to be available. For example any Debian stable server will be
> using the pre-4.02 unison for some time to come. But anyone using
> unison on OSX via Homebrew, Macports, or pkgsrc already needs a
> post-4.02 version to sync with.

For Debian stable<->testing synchronizations, one possibility is to have
a testing chroot on the stable system (or the other way around) so that
both unisons are the same.

I don't see a proper solution for this. It is a design decision of
Unison to depend on OCaml serialization, which is unsafe and not
guaranteed to be stable across OCaml versions. A similar issue can
happen with OCaml 4.03. The compatibility we've observed so far was just
chance. The proper fix would be to use a dedicated serialization library
with stability guarantees.


Cheers,

-- 
Stéphane

Bug#802919: unison: synchronization incompatibility when built with Ocaml versions pre/post-4.02

[Mark J. Nelson]

Package: unison
Version: 2.48.3-1
Severity: normal

Dear Maintainer,

Since ocaml 4.02 recently entered unstable, I thought it might be a
good time to bring up this issue (I haven't seen it discussed in
another bug).

Due to a change in ocaml's serialization format, unison built with an
ocaml pre-4.02 can't synchronize with one built with 4.02 and
later. This causes a compatibility mess, since the usual requirement
that both endpoints must have the same unison version is no longer
sufficient: they now have to have both the same unison version *and*
be built by compatible versions of ocaml, either both pre-4.02 or both
post-4.02.

There is some discussion on the unison-users list here:
http://marc.info/?l=unison-users=142286809310149=2

I am not sure what the best solution is. For the near-term future,
there will probably be a significant number of installations needing
both to be available. For example any Debian stable server will be
using the pre-4.02 unison for some time to come. But anyone using
unison on OSX via Homebrew, Macports, or pkgsrc already needs a
post-4.02 version to sync with.

-Mark

-- System Information:
Debian Release: stretch/sid
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 4.2.0-1-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: sysvinit (via /sbin/init)

Versions of packages unison depends on:
ii  libc6  2.19-22

Versions of packages unison recommends:
ii  openssh-client [ssh-client]  1:6.9p1-2

Versions of packages unison suggests:
pn  unison-all  

-- no debconf information

--
Mark J. Nelson
Anadrome Research
http://www.anadrome.org