Re: gnu: Add syncthing.

[Leo Famulari]

On Fri, Dec 16, 2016 at 01:48:57PM +0100, Petter wrote:
> On 2016-12-16 02:52, Leo Famulari wrote:
> > On Thu, Dec 15, 2016 at 10:06:59PM +0100, Petter wrote:
> > > +(arguments
> > > + `(#:import-path "github.com/syncthing/syncthing"
> > 
> > What do you think about having the go-build-system try to automatically
> > generate the import-path based on the source URL, with the option for
> > the packager to set it manually, as shown here?
> > 
> > For many of the packages in this patch (which will eventually be split
> > into one package per patch ;) ), that auto-generated import-path
> > could be correct.
> > 
> > I think that an (arguments) field indicates that the package's build
> > scripts have deviated from the standard. If a Guix build system requires
> > all of its packages to do something in (arguments), the build system
> > should be extended :)
> 
> This is my goal, and I tried to accomplish this initially, because as
> you can see for git checkouts most of the time import-path is url
> minus scheme://. But I was unable to retrieve the url in the build
> phases. So I did it like this instead, to get something that worked;
> also something like #:import-path is required where import-path can't
> be derived from the url.
> 
> I would need concrete help with this, that is getting the url and
> perhaps what (method) was used.

Does anyone have any advice here?

The issue is that the Go build tools typically want the source code to
be unpacked in a directory structure based on the URL of the Git repo.
We can set it "by hand" in the package definition, which is what the
current go-build-system patches do, but it would be great to have a
default that worked most of the time.

Re: gnu: Add syncthing.

[Leo Famulari]

On Fri, Dec 16, 2016 at 01:48:57PM +0100, Petter wrote:
> On 2016-12-16 02:52, Leo Famulari wrote:
> > On Thu, Dec 15, 2016 at 10:06:59PM +0100, Petter wrote:
> > > I've prefixed most of the packages with "golang-" now. However,
> > > there are
> > > some
> > > packages already starting with "golang-", f.ex.
> > > "golang-org-x-text-unicode-norm",
> > > and I left those alone. It's inconsistent, but I felt this was
> > > preferable to
> > > names like "golang-golang-org-x-text-unicode-norm". What do you think?
> > 
> > That's the right way. It fits the package naming guidelines:
> > 
> > https://www.gnu.org/software/guix/manual/html_node/Package-Naming.html
> 
> I'm able to interpret your answer both ways, should I make it
> "golang-golang-org-x-text-unicode-norm"?

I'm sorry to be unclear :)

I meant to say that the package should be called 
"golang-org-x-text-unicode-norm".

This matches what we do for Python packages:

If a project already contains the word python, we drop this; for
instance, the module python-dateutil is packaged under the names
python-dateutil and python2-dateutil. If the project name starts with py
(e.g. pytz), we keep it and prefix it as described above. 

https://www.gnu.org/software/guix/manual/html_node/Python-Modules.html

> > > Finally, there's a telemetry configuration in Syncthing. It is opt-in;
> > > but it will ask the user after a few hours if they want to join. The
> > > plan is to disable the question, however I suspect I've messed up the
> > > build system in that area, so this takes some more looking in to.
> > 
> > Personally, I'm fine with the upstream "opt-in" nag warning. It only
> > shows up when you open the Syncthing web interface; it's not an
> > intrusive desktop "notification". Once the user has said "yes" or "no",
> > it doesn't appear again until Syncthing makes a change to what
> > information they collect.
> > 
> > But, if people think our package should never ask, I don't mind if we
> > disable the request, as long as we share our changes with the Syncthing
> > project and they don't notice anything broken.
> 
> I'll make an attempt at this while the jury is out, primarily to
> retrieve the ability to edit files. (Either I broke it, or I need to
> learn how to use (snippet).

I applied the following diff to my package, and it made Syncthing
generate new configuration files with telemetry disabled. However, this
requires some of the tests to be adjusted. This caveat makes me wonder
if it's the right place to make the change, although I didn't find
another place with my limited Go knowledge.

diff --git a/gnu/packages/syncthing.scm b/gnu/packages/syncthing.scm
index 0ef38fcae..c67e85039 100644
--- a/gnu/packages/syncthing.scm
+++ b/gnu/packages/syncthing.scm
@@ -61,6 +61,13 @@
  (setenv "GOBIN" (assoc-ref outputs "out"))
  #t))
 
+ (add-after 'unpack 'disable-telemetry
+   (lambda _
+ (with-directory-excursion "src/github.com/syncthing/syncthing"
+   (substitute* "lib/config/optionsconfiguration.go"
+ (("json:\\\"urAccepted\\\"")
+   "json:\"urAccepted\" default:\"-1\"")
+
  (add-before 'build 'increase-test-timeout
(lambda _
  (substitute* "src/github.com/syncthing/syncthing/build.go"

> Yes! Help please. The meta-data part is tricky and time consuming, I'd
> rather spend my Guix time on the build system than descriptions
> etc. Synopsis and descriptions are just stuff i found on their
> home-page.

Okay, I'll work on this and put it on a Git branch somewhere.

> > Don't forget the revision counter :)
> 
> Oh, I didn't see the need for it, with version being:
> "20160522.452c976".
> 
> Should version really be: "20160522-1.452c976"? I don't see revision
> here contributing any value.

[...]

> > > +(define-public golang-github-com-cznic-bufs
> > > +  (let ((commit "3dcccbd7064a1689f9c093a988ea11ac00e21f51"))
> > > +(package
> > > +  (name "golang-github-com-cznic-bufs")
> > > +  (version (string-append "20160818." (string-take commit 7)))
> > 
> > Is this string '20160818' the date of the commit? If the package has no
> > releases, we use '0.0.0' in place of the version.
> 
> Yes. I figure this is best because some later project we'll package
> may be pinned to an earlier commit of a library, and then what?
> Incrementing the revision number would suggest this was of newer date,
> when it's not, and make the older commit appear as the newest version
> of the two to Guix.

We didn't really make a decision in the previous discussion:

https://lists.gnu.org/archive/html/guix-devel/2016-01/msg00422.html

If Go packages have to use specific commits of their dependencies, then
we'll need to specify that version each time we use the package.

If we decide to use the date anyways, remember that commit timestamps
tend not to be chronologically ordered. The "push" timestamp would be
more meaningful.

I don't know the best 

Re: gnu: Add syncthing.

[Petter]

On 2016-12-16 02:52, Leo Famulari wrote:

On Thu, Dec 15, 2016 at 10:06:59PM +0100, Petter wrote:

Hi again,

I've prefixed most of the packages with "golang-" now. However, there 
are

some
packages already starting with "golang-", f.ex.
"golang-org-x-text-unicode-norm",
and I left those alone. It's inconsistent, but I felt this was 
preferable to

names like "golang-golang-org-x-text-unicode-norm". What do you think?


That's the right way. It fits the package naming guidelines:

https://www.gnu.org/software/guix/manual/html_node/Package-Naming.html


I'm able to interpret your answer both ways, should I make it
"golang-golang-org-x-text-unicode-norm"?


Finally, there's a telemetry configuration in Syncthing. It is opt-in;
but it will ask the user after a few hours if they want to join. The
plan is to disable the question, however I suspect I've messed up the
build system in that area, so this takes some more looking in to.


Personally, I'm fine with the upstream "opt-in" nag warning. It only
shows up when you open the Syncthing web interface; it's not an
intrusive desktop "notification". Once the user has said "yes" or "no",
it doesn't appear again until Syncthing makes a change to what
information they collect.

But, if people think our package should never ask, I don't mind if we
disable the request, as long as we share our changes with the Syncthing
project and they don't notice anything broken.


I'll make an attempt at this while the jury is out, primarily to
retrieve the ability to edit files. (Either I broke it, or I need to
learn how to use (snippet).


Many of my comments below are about tedious things. Let us know if you
are getting sick of working on these packages, and I will help :) This
includes improving the descriptions.


Yes! Help please. The meta-data part is tricky and time consuming, I'd
rather spend my Guix time on the build system than descriptions
etc. Synopsis and descriptions are just stuff i found on their
home-page.


Date: Thu, 15 Dec 2016 21:42:03 +0100
Subject: [PATCH] gnu: Add Syncthing.

* gnu/packages/syncthing.scm: New file.


Cool!


+(define-public syncthing



+(source (origin
+  (method git-fetch)
+  (uri (git-reference
+(url "https://github.com/syncthing/syncthing/;)
+(commit (string-append "v" version
+  (file-name (string-append name "-" version 
"-checkout"))


We should use the Syncthing release tarball:

https://github.com/syncthing/syncthing/releases/download/v0.14.14/syncthing-source-v0.14.14.tar.gz


Totally agree! However, tarballs are currently unsupported... due to,
hm, let's say "because very good reasons noone can be blamed for,
particularly and especially not me".

Ok, I messed up the build system :P Removed too much code
apparently. I'm working on getting this back.


+(arguments
+ `(#:import-path "github.com/syncthing/syncthing"


What do you think about having the go-build-system try to automatically
generate the import-path based on the source URL, with the option for
the packager to set it manually, as shown here?

For many of the packages in this patch (which will eventually be split
into one package per patch ;) ), that auto-generated import-path
could be correct.

I think that an (arguments) field indicates that the package's build
scripts have deviated from the standard. If a Guix build system 
requires

all of its packages to do something in (arguments), the build system
should be extended :)


This is my goal, and I tried to accomplish this initially, because as
you can see for git checkouts most of the time import-path is url
minus scheme://. But I was unable to retrieve the url in the build
phases. So I did it like this instead, to get something that worked;
also something like #:import-path is required where import-path can't
be derived from the url.

I would need concrete help with this, that is getting the url and
perhaps what (method) was used.


+   #:phases
+   (modify-phases %standard-phases
+ (replace 'delete-files
+   (lambda _
+ (delete-file-recursively 
"src/github.com/syncthing/syncthing/vendor")))

+
+ (replace 'build
+   (lambda* (#:key inputs #:allow-other-keys)
+ (with-directory-excursion 
"src/github.com/syncthing/syncthing"
+   (zero? (system* "go" "run" "build.go" "install" 
"syncthing" "-no-upgrade")

+
+ (replace 'install
+   (lambda _
+ (copy-recursively 
"src/github.com/syncthing/syncthing/bin/"
+   (string-append (assoc-ref %outputs 
"out") "/bin"))

+ (copy-recursively "pkg"
+   (string-append (assoc-ref %outputs 
"out") "/pkg"))

+ (copy-recursively "src"
+   (string-append (assoc-ref %outputs 
"out") "/src")))

+


Does this package need to use custom build and install phases?


Yes. They use their 

Re: gnu: Add syncthing.

[Leo Famulari]

On Thu, Dec 15, 2016 at 10:06:59PM +0100, Petter wrote:
> Hi again,
> 
> I've prefixed most of the packages with "golang-" now. However, there are
> some
> packages already starting with "golang-", f.ex.
> "golang-org-x-text-unicode-norm",
> and I left those alone. It's inconsistent, but I felt this was preferable to
> names like "golang-golang-org-x-text-unicode-norm". What do you think?

That's the right way. It fits the package naming guidelines:

https://www.gnu.org/software/guix/manual/html_node/Package-Naming.html

> Finally, there's a telemetry configuration in Syncthing. It is opt-in;
> but it will ask the user after a few hours if they want to join. The
> plan is to disable the question, however I suspect I've messed up the
> build system in that area, so this takes some more looking in to.

Personally, I'm fine with the upstream "opt-in" nag warning. It only
shows up when you open the Syncthing web interface; it's not an
intrusive desktop "notification". Once the user has said "yes" or "no",
it doesn't appear again until Syncthing makes a change to what
information they collect.

But, if people think our package should never ask, I don't mind if we
disable the request, as long as we share our changes with the Syncthing
project and they don't notice anything broken.

Many of my comments below are about tedious things. Let us know if you
are getting sick of working on these packages, and I will help :) This
includes improving the descriptions.

> Date: Thu, 15 Dec 2016 21:42:03 +0100
> Subject: [PATCH] gnu: Add Syncthing.
> 
> * gnu/packages/syncthing.scm: New file.

Cool!

> +(define-public syncthing

> +(source (origin
> +  (method git-fetch)
> +  (uri (git-reference
> +(url "https://github.com/syncthing/syncthing/;)
> +(commit (string-append "v" version
> +  (file-name (string-append name "-" version "-checkout"))

We should use the Syncthing release tarball:

https://github.com/syncthing/syncthing/releases/download/v0.14.14/syncthing-source-v0.14.14.tar.gz

> +(arguments
> + `(#:import-path "github.com/syncthing/syncthing"

What do you think about having the go-build-system try to automatically
generate the import-path based on the source URL, with the option for
the packager to set it manually, as shown here?

For many of the packages in this patch (which will eventually be split
into one package per patch ;) ), that auto-generated import-path
could be correct.

I think that an (arguments) field indicates that the package's build
scripts have deviated from the standard. If a Guix build system requires
all of its packages to do something in (arguments), the build system
should be extended :)

> +   #:phases
> +   (modify-phases %standard-phases
> + (replace 'delete-files
> +   (lambda _
> + (delete-file-recursively 
> "src/github.com/syncthing/syncthing/vendor")))
> +
> + (replace 'build
> +   (lambda* (#:key inputs #:allow-other-keys)
> + (with-directory-excursion "src/github.com/syncthing/syncthing"
> +   (zero? (system* "go" "run" "build.go" "install" "syncthing" 
> "-no-upgrade")
> +
> + (replace 'install
> +   (lambda _
> + (copy-recursively "src/github.com/syncthing/syncthing/bin/"
> +   (string-append (assoc-ref %outputs "out") 
> "/bin"))
> + (copy-recursively "pkg"
> +   (string-append (assoc-ref %outputs "out") 
> "/pkg"))
> + (copy-recursively "src"
> +   (string-append (assoc-ref %outputs "out") 
> "/src")))
> +

Does this package need to use custom build and install phases?

> +(define-public golang-github-com-audriusbutkevicius-go-nat-pmp
> +  (let ((commit "452c97607362b2ab5a7839b8d1704f0396b640ca"))

Don't forget the revision counter :)

> +(define-public golang-github-com-bkaradzic-go-lz4
> +  (package
> +(name "golang-github-com-bkaradzic-go-lz4")
> +(version "1.0.0")
> +(source (origin
> +  (method git-fetch)
> +  (uri (git-reference
> +(url "https://github.com/bkaradzic/go-lz4;)
> +(commit "74ddf82598bc4745b965729e9c6a463bedd33049")))

For packages that we build from a Git tag (rather than an untagged
commit), you should do (commit (string-append "v" version)).

Although, if there is a release tarball, it's preferable to use it.

> +(define-public golang-github-com-calmh-xdr
> +  (let ((commit "f9b9f8f7aa27725f5cabb699bd9099ca7ce09143")
> +(revision "1"))
> +(package
> +  (name "golang-github-com-calmh-xdr")
> +  (version (string-append "2.0.0-" revision "." (string-take commit 7)))
> +  (source (origin
> +(method git-fetch)
> +(uri (git-reference
> +  (url "https://github.com/calmh/xdr;)
> +  

Re: gnu: Add syncthing.

[Petter]

Hi again,

I've prefixed most of the packages with "golang-" now. However, there 
are some
packages already starting with "golang-", f.ex. 
"golang-org-x-text-unicode-norm",
and I left those alone. It's inconsistent, but I felt this was 
preferable to

names like "golang-golang-org-x-text-unicode-norm". What do you think?

Also, all the packages have meta-data now: synopsis, description, 
home-page and
license. And the (version) field have been adapted to the Guix 
guidelines.


Also also, a new version of Syncthing with some security fixes was just 
released,

and I've updated the recipe to this version (0.14.14).

Finally, there's a telemetry configuration in Syncthing. It is opt-in; 
but it
will ask the user after a few hours if they want to join. The plan is to 
disable
the question, however I suspect I've messed up the build system in that 
area, so

this takes some more looking in to.

Best,
PetterFrom f6b8324f4200c2dc89967b5f097fc8e28292bd5a Mon Sep 17 00:00:00 2001
From: Petter 
Date: Thu, 15 Dec 2016 21:42:03 +0100
Subject: [PATCH] gnu: Add Syncthing.

* gnu/packages/syncthing.scm: New file.
---
 gnu/packages/syncthing.scm | 1257 
 1 file changed, 1257 insertions(+)
 create mode 100644 gnu/packages/syncthing.scm

diff --git a/gnu/packages/syncthing.scm b/gnu/packages/syncthing.scm
new file mode 100644
index 000..c4da737
--- /dev/null
+++ b/gnu/packages/syncthing.scm
@@ -0,0 +1,1257 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2016 Leo Famulari 
+;;; Copyright © 2016 Petter 
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see .
+
+(define-module (gnu packages syncthing)
+  #:use-module (guix build-system go)
+  #:use-module (guix packages)
+  #:use-module (guix download)
+  #:use-module (guix git-download)
+  #:use-module (guix licenses)
+  #:use-module (gnu packages golang))
+
+(define-public syncthing
+  (package
+(name "syncthing")
+(version "0.14.14")
+(source (origin
+  (method git-fetch)
+  (uri (git-reference
+(url "https://github.com/syncthing/syncthing/;)
+(commit (string-append "v" version
+  (file-name (string-append name "-" version "-checkout"))
+  (sha256
+   (base32 "1c9x7j6pbbd3i34nvisw70v2l3llp3dhw5cvd72jgx3gm0j3v31y"
+(build-system go-build-system)
+(inputs
+ `(("github.com/AudriusButkevicius/go-nat-pmp"
+,golang-github-com-audriusbutkevicius-go-nat-pmp)
+   ("github.com/vitrun/qart/coding" ,golang-github-com-vitrun-qart-coding)
+   ("github.com/vitrun/qart/gf256" ,golang-github-com-vitrun-qart-gf256)
+   ("github.com/vitrun/qart/qr" ,golang-github-com-vitrun-qart-qr)
+   ("github.com/thejerf/suture" ,golang-github-com-thejerf-suture)
+   ("github.com/syndtr/goleveldb" ,golang-github-com-syndtr-goleveldb)
+   ("github.com/sasha-s/go-deadlock" ,golang-github-com-sasha-s-go-deadlock)
+   ("github.com/rcrowley/go-metrics" ,golang-github-com-rcrowley-go-metrics)
+   ("github.com/minio/sha256-simd" ,golang-github-com-minio-sha256-simd)
+   ("github.com/kardianos/osext" ,golang-github-com-kardianos-osext)
+   ("github.com/juju/ratelimit" ,golang-github-com-juju-ratelimit)
+   ("github.com/jackpal/gateway" ,golang-github-com-jackpal-gateway)
+   ("github.com/gogo/protobuf" ,golang-github-com-gogo-protobuf)
+   ("github.com/gobwas/glob" ,golang-github-com-gobwas-glob)
+   ("github.com/calmh/xdr" ,golang-github-com-calmh-xdr)
+   ("github.com/calmh/luhn" ,golang-github-com-calmh-luhn)
+   ("github.com/calmh/du" ,golang-github-com-calmh-du)
+   ("github.com/bkaradzic/go-lz4" ,golang-github-com-bkaradzic-go-lz4)
+   ("github.com/golang/snappy" ,golang-github-com-golang-snappy)
+   ("golang.org/x/crypto/bcrypt" ,golang-org-x-crypto-bcrypt)
+   ("golang.org/x/crypto/blowfish" ,golang-org-x-crypto-blowfish)
+   ("golang.org/x/text/transform" ,golang-org-x-text-transform)
+   ("golang.org/x/text/unicode/norm" ,golang-org-x-text-unicode-norm)
+   ("golang.org/x/net/proxy" ,golang-org-x-net-proxy)
+   ("golang.org/x/net/bpf" ,golang-org-x-net-bpf)
+   ("golang.org/x/net/ipv6" 

Re: gnu: Add syncthing.

[Chris Marusich]

ng0  writes:

> Leo Famulari  writes:
>
>> On Sun, Dec 11, 2016 at 12:25:31PM +0100, Hartmut Goebel wrote:
>>> Am 11.12.2016 um 01:38 schrieb Petter:
>>> > +(define-public github-com-audriusbutkevicius-go-nat-pmp
>>> > +  (package
>>> > +(name "github-com-audriusbutkevicius-go-nat-pmp")
>>> 
>>> I suggest that all `go` packages kill get a prefix like other languages
>>> have. I propose using "golang" and prefix, as Debina does, sicne "go" is
>>> to short and ambiguous.
>>
>> +1 for libraries, excluding user-facing applications like Syncthing.
>
> I'm in support of this naming convention.

Yeah, that seems good to me, too.

-- 
Chris


signature.asc
Description: PGP signature

Re: gnu: Add syncthing.

[Leo Famulari]

On Sun, Dec 11, 2016 at 12:25:31PM +0100, Hartmut Goebel wrote:
> Am 11.12.2016 um 01:38 schrieb Petter:
> > +(define-public github-com-audriusbutkevicius-go-nat-pmp
> > +  (package
> > +(name "github-com-audriusbutkevicius-go-nat-pmp")
> 
> I suggest that all `go` packages kill get a prefix like other languages
> have. I propose using "golang" and prefix, as Debina does, sicne "go" is
> to short and ambiguous.

+1 for libraries, excluding user-facing applications like Syncthing.

Re: gnu: Add syncthing.

[Hartmut Goebel]

Am 11.12.2016 um 12:25 schrieb Hartmut Goebel:
> I suggest that all `go` packages kill get a prefix like other languages

s/kill/will/

-- 
Regards
Hartmut Goebel

| Hartmut Goebel  | h.goe...@crazy-compilers.com   |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: gnu: Add syncthing.

[Hartmut Goebel]

Am 11.12.2016 um 01:38 schrieb Petter:
> +(define-public github-com-audriusbutkevicius-go-nat-pmp
> +  (package
> +(name "github-com-audriusbutkevicius-go-nat-pmp")

I suggest that all `go` packages kill get a prefix like other languages
have. I propose using "golang" and prefix, as Debina does, sicne "go" is
to short and ambiguous.



-- 
Regards
Hartmut Goebel

| Hartmut Goebel  | h.goe...@crazy-compilers.com   |
| www.crazy-compilers.com | compilers which you thought are impossible |