Re: Rebasing guile-daemon branch onto master

[Leo Famulari]

On Sat, Apr 28, 2018 at 07:10:00PM +, Sandeep Subramanian wrote:
> Shall I now rebase by picking only these 30 commits?
> 
> I don't know if what I am doing is right. I could use some guidance and
> criticism now.

Yes, I think that picking only those 30 commits is the right thing to
do.

Please let us know how it goes :)


signature.asc
Description: PGP signature

Re: GSoC-2018

[Caleb Ristvedt]

Hi Sandeep/uniq10, welcome!

I'm the person that worked on the rewrite last year (I like to go by
reepca in conversations and irc). I'm glad that you'll be working on it
this summer. I still feel like I didn't do enough last summer, though,
so if there's any way I can help, I'd be glad to (though the next couple
of days are finals here, so I'll be rather busy just then).

If you have any questions about the code or find yourself reading
through a bunch of C++ you'd rather not be reading through and thinking
to yourself "that other guy probably already had to do this...", feel
free to ask. I'm in #guix pretty much all the time (though only on my
desktop, so if I'm away the response may be delayed), though you'll
probably have to ping me.

- reepca

Re: Rebasing guile-daemon branch onto master

[Sandeep Subramanian]

> I was trying to rebase the guile-daemon branch onto the master branch
> and I have some trouble doing it.

> The interactive git rebase showed me that there are 932 commits to be
> picked but most of them had no relation to the daemon code (Most commits
> were package addition/update commits.) I also tried `git cherry master
> guile-daemon`
> and that too showed 932 "+ commits" and 0 "- commits".

> The graph looks something like:

> (78a5205)  (2472f7a)
>  (master-HEAD)

***-*
>  \   (c4395e7) \
>\   \
> (e338c9b)
>  \
>  *---*-* (6dee54f - guile-daemon HEAD)
>\  (36cc971)
>/

*---**
> (654c8a7) (5e27bfc)
(f5dfbaa)


> A lot of commits on the guile daemon section of the branch are also found
> between 78a5205 and 2472f7a. For example both c4395e7 and 5e27bfc
> are the exact same patches and have the same patch-id. And almost all the
> 932 commits have a duplicate patch.

> I think this is the reason why both rebase and cherry are showing a lot of
> redundant commits.

> To verify, I created an orphaned branch "orphan" from the 78a5205 commit
and
> and did `git cherry-pick 2472f7a..guile-daemon`. Then a `git cherry master
> orphan`
> revealed 30 "+ commits" and 902 "- commits". `git diff guile-daemon
orphan`
> was empty
> and the 30 "+ commits" corresponded to the 30 commits by Caleb Ristvedt.

> Shall I now rebase by picking only these 30 commits?

> I don't know if what I am doing is right. I could use some guidance and
> criticism now.


Fixed the graph formatting.

(78a5205)(2472f7a)(master-HEAD)
 *-***
  \(c4395e7)   \
   \\   (e338c9b)
\*--*-*(guile-daemon-HEAD)
 \ (36cc971) /
  \   /
   *---*--*
(654c8a7)(5e27bfc)   (f5dfbaa)

   --
Sandeep (uniq10)

You say nix, I say guix: Nix 2.0 and Guix

[Chris Marusich]

Hi,

On February 22nd, Nix 2.0 was released:

https://nixos.org/nix/manual/#ssec-relnotes-2.0

It contains a lot of interesting new features.  Are there any plans to
merge some of the nix-daemon changes into our guix-daemon?  Is
compatibility with the nix-daemon a goal of the Guix project?  Can we
take inspiration from any of the non-daemon features and use them in
Guix?  Conversely, is there anything we can upstream to Nix that they
might find useful?

-- 
Chris


signature.asc
Description: PGP signature

Re: Should python-build-system packages have native-inputs?

[Chris Marusich]

Hartmut Goebel  writes:

>> the python-build-system does not cross-compile.
>
> In any case, this is a current limitation only :-)

I see. If we actually do plan on implementing some kind of
cross-compilation support for the python-build-system, then I can
understand why it makes sense to proactively put into the native-inputs
those things which might possibly need to be native-inputs when that
time comes.  Can we at least mention in the manual that the
python-build-system doesn't currently cross-compile, so native-inputs
will be treated the same as inputs for now, but we still recommend
putting the "build-only dependencies" in native-inputs in order to
future-proof our package definitions?  That alone would have helped
clarify things for me when I was starting out.

After reading about Python extension modules a little more, it seems
that they can in fact be cross-compiled.  I didn't look into Python 2,
and I don't know what it would take to enable such cross-compilation in
the python-build-system.

For those following along, here are some related links.

An explanation of Python extension modules:
https://docs.python.org/3/extending/building.html#distributing-your-extension-modules

Some Python open bugs that mention cross-compilation:
https://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus&%40sort=-activity&%40group=priority&%40filter=status=-1%2C1%2C3&%40search_text=cross-compile=search+in+open+issues

-- 
Chris


signature.asc
Description: PGP signature

Re: Mes 0.13 released

[Arne Babenhauserheide]

Hi Jan,

Jan Nieuwenhuizen  writes:
> I am pleased to announce the release of Mes 0.13, representing 45
> commits over 3 weeks.  MesCC can now compile a functional tcc when
> running on Mes (in ~1h45') or on Guile (in ~3min).

That is absolutely awesome! My deepest, deepest respekt for your work!

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken


signature.asc
Description: PGP signature

Rebasing guile-daemon branch onto master

[Sandeep Subramanian]

Hi all,

I was trying to rebase the guile-daemon branch onto the master branch
and I have some trouble doing it.

The interactive git rebase showed me that there are 932 commits to be
picked but most of them had no relation to the daemon code (Most commits
were package addition/update commits.) I also tried `git cherry master
guile-daemon`
and that too showed 932 "+ commits" and 0 "- commits".

The graph looks something like:

(78a5205)  (2472f7a)
  (master-HEAD)
***-*
  \   (c4395e7) \
\   \
(e338c9b)
  \
  *---*-* (6dee54f - guile-daemon HEAD)
\  (36cc971)
/
 *---**
 (654c8a7) (5e27bfc)   (f5dfbaa)


A lot of commits on the guile daemon section of the branch are also found
between 78a5205 and 2472f7a. For example both c4395e7 and 5e27bfc
are the exact same patches and have the same patch-id. And almost all the
932 commits have a duplicate patch.

I think this is the reason why both rebase and cherry are showing a lot of
redundant commits.

To verify, I created an orphaned branch "orphan" from the 78a5205 commit and
and did `git cherry-pick 2472f7a..guile-daemon`. Then a `git cherry master
orphan`
revealed 30 "+ commits" and 902 "- commits". `git diff guile-daemon orphan`
was empty
and the 30 "+ commits" corresponded to the 30 commits by Caleb Ristvedt.

Shall I now rebase by picking only these 30 commits?

I don't know if what I am doing is right. I could use some guidance and
criticism now.

--
Sandeep (uniq10)

Re: GSoC-2018

[Kei Kebreau]

Sandeep Subramanian  writes:

> Hi all,
>
> I am Sandeep Subramanian (uniq10) and I have been selected for the
> GSoC-2018 project "Continue rewrite build daemon in Guile Scheme".
> (Project description:
> https://libreplanet.org/wiki/Group:Guix/GSoC-2018#Continue_rewrite_build_daemon_in_Guile_Scheme
> )
>
> I am very excited to work on this project and am looking forward to
> interacting
> with the community here and on the #guix IRC channel (Nick: uniq10).
> I will mostly be active from UTC 08:00 to UTC 20:00.
>
> I hope to, with your help, successfully complete this project and make a
> meaningful contribution to the community.

Welcome!


signature.asc
Description: PGP signature

New Spanish PO file for 'shepherd' (version 0.3.3-pre1)

[Translation Project Robot]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'shepherd' has been submitted
by the Spanish team of translators.  The file is available at:

http://translationproject.org/latest/shepherd/es.po

(We can arrange things so that in the future such files are automatically
e-mailed to you when they arrive.  Ask at the address below if you want this.)

All other PO files for your package are available in:

http://translationproject.org/latest/shepherd/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

http://translationproject.org/domain/shepherd.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

The Translation Project robot, in the
name of your translation coordinator.

Mes 0.13 released

[Jan Nieuwenhuizen]

I am pleased to announce the release of Mes 0.13, representing 45
commits over 3 weeks.  MesCC can now compile a functional tcc when
running on Mes (in ~1h45') or on Guile (in ~3min).

This means that we are getting very close to a full source bootstrap
of tcc: it is now built without gcc, glibc or guile; using only
MesCC-tools and Mes sources and corresponding ascii/binary seeds.

Attached full dependency graphs from Guix's wip-bootstrap branch.  Note
the tcc-boot dependency on glibc: that's (only) because of the location
of the dynamic linker inserted in the mes-tcc binary.

* About

Mes[0] aims to help create full source bootstrapping for GuixSD[1]
as part of the bootstrappable builds[2] project.

It currently consists of a mutual self-hosting Scheme interpreter
prototype in C and a Nyacc-based C compiler in Scheme.  This C
prototype is being simplified to be transpiled by M2-Planet[3].

The Scheme interpreter prototype (mes.c) has a Garbage Collector,
a library of loadable Scheme modules-- notably Dominique Boucher's
LALR[4], pre-R6RS portable syntax-case[5] with R7RS ellipsis, Matt
Wette's Nyacc[6], Guile's PEG[7] --and test suite just barely
enough to support a simple REPL (repl.mes) and simple C-compiler
MesCC.

Mes+MesCC can compile a modified TinyCC[8] that is close to being
self-hosting.  A GNU Ccc-compiled tcc is known[9] to compile GCC.

Mes is inspired by The Maxwell Equations of Software: LISP-1.5[10]
-- John McCarthy page 13, GNU Guix's source/binary packaging
transparency and Jeremiah Orians's stage0[11] ~300 byte self-hosting
hex assembler.

* Download

git clone https://gitlab.com/janneke/mes

wget https://gitlab.com/janneke/mes/-/archive/v0.13/mes-0.13.tar.gz

Mes runs from the source tree and can also be built, packaged and
installed in Guix[SD] by the usual

guix package -f guix.scm

* Changes in 0.13 since 0.12
 ** Core
 *** Bootstrapped Mes+MesCC can now compile a patched tcc in ~2h30' (~25,000 
LOC).
 *** MesCC scripts for Mes and Guile are now merged; executable is: `mescc'.
 *** Mes now uses only one arena for stop-and-copy; doubles available size.
 *** Mes now has a Guile-like command-line interface (mes.repl has been 
removed).
 *** Mes now boots into a full Scheme by default.
 *** Mes can now be compiled (MES_MINI=1) to boot into a minimal Scheme (~2000 
cells).
 *** Mes now creates less garbage in the reader and in
 append2, append_reverse, reverse, reverse!, vector-for-each, vector-to-list, 
vector-map.
 *** 5 new functions
 append-reverse, chmod, ioctl, isatty, isspace, last_pair, reverse!.
 ** Language
 *** 3 new functions
 char-whitespace?, chmod, isatty?
 ** Noteworthy bug fixes
 *** Two bugs in the jam scraper/garbage collector have been fixed.
 *** equal2_p now uses tail call elimination.
 *** Escaped characters in strings are now read and write'd correctly.
 *** The repl now expands macros again.

Greetings,
janneke

[0] https://gitlab.com/janneke/mes
[1] https://www.gnu.org/software/guix
[2] http://bootstrappable.org
[3] https://github.com/oriansj/m2-planet
[4] https://github.com/schemeway/lalr-scm
[5] https://www.cs.indiana.edu/chezscheme/syntax-case/old-psyntax.html
[6] https://www.nongnu.org/nyacc/
[7] https://www.gnu.org/software/guile/docs/master/guile.html/PEG-Parsing.html
[8] https://gitlab.com/janneke/tinycc
[9] https://lists.gnu.org/archive/html/tinycc-devel/2017-05/msg00103.html
[10] 
http://www.softwarepreservation.org/projects/LISP/book/LISP%25201.5%2520Programmers%2520Manual.pdf
[11] https://github.com/oriansj/stage0




mes-boot-bag.svg
Description: Binary data


tcc-boot-bag.svg
Description: Binary data

-- 
Jan Nieuwenhuizen  | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Should python-build-system packages have native-inputs?

[Hartmut Goebel]

Am 28.04.2018 um 12:11 schrieb Chris Marusich:

I understand your concerns, and I understand why this is hard to get for
a Pythonista. But this is exactly why we added this section to the manual.

> Because the python-build-system never cross-compiles, 

This is an implementation detail which might might change. And if we
remove all inputs now, we need to add again them later. This is a lot of
work, I know since I've cleaned this up for all Python modules. IMHO
it's not a good idea for drop this knowledge from the code.

> If the
> python-build-system actually did support cross-compilation, then this
> might be a different story.

Maybe this is going to change somewhen :-) We should aim to the top, not
the status quo :-)



> My understanding is that the concept of "native-inputs" for a package
> only makes sense when that package uses a build system that can
> cross-compile,

This is my understanding, too. But the python-build-system might be able
to cross-compile somewhen and then this information is essential.

>> And for extension modules it would allow compiling on a faster
>> environment (e.g. x86 vs. ARMv4).
>>
>> (I was not aware of python packages are not cross-compiled, thus I can
>> only guess the reason why this is not possible: Python distutils may not
>> be able to *cross*-compile extension modules. Maybe we could work on this.)
> I am curious about extension modules.  I understand they are tied
> closely to the underlying architecture, but I have little experience
> with them, so I'm not sure how they relate to cross compilation.

Extension modules are simply modules or libraries  written in C/C++ or
other languages. Even modules written in Cython would be counted in
here, since they are translated to C and then compiled into platform
dependent code.

> In any
> case, it doesn't change the fact that today, the python-build-system
> does not cross-compile.

In any case, this is a current limitation only :-)

-- 
Regards
Hartmut Goebel

| Hartmut Goebel  | h.goe...@crazy-compilers.com   |
| www.crazy-compilers.com | compilers which you thought are impossible |




signature.asc
Description: OpenPGP digital signature

Re: Should python-build-system packages have native-inputs?

[Chris Marusich]

Hi Fis and Hartmut,

Thank you for the quick response!

Hartmut Goebel  writes:

> As Fis already wrote:  These native-inputs are for testing and shouldn't
> be installed in normal case.

It's true that for some of the packages that use the
python-build-system, we have been putting the dependencies required for
testing (such as python-pytest) into the package's native-inputs.
However, whether such dependencies are inputs or native-inputs does not
matter.  Because the python-build-system never cross-compiles, all of
the inputs, propagated-inputs, and native-inputs will be included in the
single list that gets passed to each of the build phases via the
#:inputs keyword argument.  You can verify this yourself by inserting
debug statements in the build phases.

In other words, it doesn't matter if we put python-pytest in a package's
inputs or its native-inputs.  The end result is the same.  If the
python-build-system actually did support cross-compilation, then this
might be a different story.  However, the python-build-system doesn't
cross-compile.  As a result, native-inputs and inputs are treated the
same in all of the phases defined in guix/build/python-build-system.scm.

> Please see "Python Modules" in the manual:
>
> Python packages required only at build time---e.g., those listed with
> the @code{setup_requires} keyword in @file{setup.py}---or only for
> testing---e.g., those in @code{tests_require}---go into
> @code{native-inputs}.  The rationale is that (1) they do not need to be
> propagated because they are not needed at run time, and (2) in a
> cross-compilation context, it's the ``native'' input that we'd want.

Thank you for mentioning the manual; I had forgotten that we include
explicit guidance for Python modules.  I've just reviewed the "Python
Modules" section.  I think we should not be advising people to use
native-inputs in packages that use the python-build-system.  There is no
meaningful difference between "native-inputs" and "inputs" in this case,
so asking people to contemplate the difference is like asking them a
kōan.  It's just going to cause confusion.

This is confusing.  And that is precisely why I think we should stop
declaring native-inputs for packages that use the python-build-system.
My understanding is that the concept of "native-inputs" for a package
only makes sense when that package uses a build system that can
cross-compile, such as the gnu-build-system.  Because the
python-build-system never cross-compiles, it doesn't make sense to
declare native-inputs for a package that uses the python-build-system.
Instead, those dependencies should just be declared as inputs.

>> * Are there any circumstances under which it actually WOULD make sense
>>   to cross-compile a Python package?
>
> Of course: Pure-python packages should be able to be cross-compiled
> without any problems, sicne the bytes-code is the same for all
> platforms.

I'm not sure that's the same thing as cross compilation.  When cross
compiling a program for a different architecture, the output of the
build is different for each architecture.  If Python's bytecode is the
same for all platforms, then it sounds like no cross-compilation is
necessary, which suggests that the notion of "cross compilation" does
not make sense for Python code.  Did I misinterpret what you meant?

> And for extension modules it would allow compiling on a faster
> environment (e.g. x86 vs. ARMv4).
>
> (I was not aware of python packages are not cross-compiled, thus I can
> only guess the reason why this is not possible: Python distutils may not
> be able to *cross*-compile extension modules. Maybe we could work on this.)

I am curious about extension modules.  I understand they are tied
closely to the underlying architecture, but I have little experience
with them, so I'm not sure how they relate to cross compilation.  In any
case, it doesn't change the fact that today, the python-build-system
does not cross-compile.

-- 
Chris


signature.asc
Description: PGP signature

Re: Should python-build-system packages have native-inputs?

[Hartmut Goebel]

Am 28.04.2018 um 08:50 schrieb Chris Marusich:
> * Should we change these native-inputs to inputs to prevent confusion?
>   I can personally vouch for the fact that the presence of native-inputs
>   in python-build-system packages confused the heck out of me at first!
 
As Fis already wrote:  These native-inputs are for testing and shouldn't
be installed in normal case. Please see "Python Modules" in the manual:

Python packages required only at build time---e.g., those listed with
the @code{setup_requires} keyword in @file{setup.py}---or only for
testing---e.g., those in @code{tests_require}---go into
@code{native-inputs}.  The rationale is that (1) they do not need to be
propagated because they are not needed at run time, and (2) in a
cross-compilation context, it's the ``native'' input that we'd want.

> * Are there any circumstances under which it actually WOULD make sense
>   to cross-compile a Python package?

Of course: Pure-python packages should be able to be cross-compiled
without any problems, sicne the bytes-code is the same for all
platforms. And for extension modules it would allow compiling on a
faster environment (e.g. x86 vs. ARMv4).

(I was not aware of python packages are not cross-compiled, thus I can
only guess the reason why this is not possible: Python distutils may not
be able to *cross*-compile extension modules. Maybe we could work on this.)

-- 
Regards
Hartmut Goebel

| Hartmut Goebel  | h.goe...@crazy-compilers.com   |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Should python-build-system packages have native-inputs?

[Fis Trivial]

Chris Marusich writes:

> Hi Guix,
>
> I've noticed that a fair number of packages in gnu/packages/python.scm
> using the python-build-system declare native-inputs.  I suspect that in
> every case, these should actually just be inputs.  I also suspect that
> this is benign, except perhaps for the fact that it may confuse
> Pythonistas who (like myself) initially started out by looking at these
> packages as examples of how to get started defining packages in Guix.
>
> The python-build-system's "lower" procedure (in
> guix/build-system/python.scm) explicitly forbids cross-compilation:
>
> --8<---cut here---start->8---
> (define* (lower name
> #:key source inputs native-inputs outputs system target
> (python (default-python))
> #:allow-other-keys
> #:rest arguments)
>   "Return a bag for NAME."
>   (define private-keywords
> '(#:source #:target #:python #:inputs #:native-inputs))
>
>   (and (not target)   ;XXX: no cross-compilation
>(bag
>  (name name)
>  (system system)
>  (host-inputs `(,@(if source
>   `(("source" ,source))
>   '())
> ,@inputs
>
> ;; Keep the standard inputs of 'gnu-build-system'.
> ,@(standard-packages)))
>  (build-inputs `(("python" ,python)
>  ,@native-inputs))
>  (outputs outputs)
>  (build python-build)
>  (arguments (strip-keyword-arguments private-keywords arguments)
> --8<---cut here---end--->8---
>
> As for the native-inputs, they get stored in the bag's build-inputs,
> which eventually find their way to the "inputs" keyword argument used on
> the build side by the various build phases.  In fact, the inputs,
> propagated-inputs, and native-inputs of any package that uses the
> python-build-system are all put into this "inputs" keyword argument.
>
> With this in mind, I have two questions:
>
> * Should we change these native-inputs to inputs to prevent confusion?
>   I can personally vouch for the fact that the presence of native-inputs
>   in python-build-system packages confused the heck out of me at first!
>
> * Are there any circumstances under which it actually WOULD make sense
>   to cross-compile a Python package?
>
> For now, I think the answers to these questions are "sure" and "probably
> not", respectively.  I'm very curious to hear your thoughts about the
> second question, in particular!

I'm confused, some native-inputs are for testing. They shouldn't be
installed in normal case.

Should python-build-system packages have native-inputs?

[Chris Marusich]

Hi Guix,

I've noticed that a fair number of packages in gnu/packages/python.scm
using the python-build-system declare native-inputs.  I suspect that in
every case, these should actually just be inputs.  I also suspect that
this is benign, except perhaps for the fact that it may confuse
Pythonistas who (like myself) initially started out by looking at these
packages as examples of how to get started defining packages in Guix.

The python-build-system's "lower" procedure (in
guix/build-system/python.scm) explicitly forbids cross-compilation:

--8<---cut here---start->8---
(define* (lower name
#:key source inputs native-inputs outputs system target
(python (default-python))
#:allow-other-keys
#:rest arguments)
  "Return a bag for NAME."
  (define private-keywords
'(#:source #:target #:python #:inputs #:native-inputs))

  (and (not target)   ;XXX: no cross-compilation
   (bag
 (name name)
 (system system)
 (host-inputs `(,@(if source
  `(("source" ,source))
  '())
,@inputs

;; Keep the standard inputs of 'gnu-build-system'.
,@(standard-packages)))
 (build-inputs `(("python" ,python)
 ,@native-inputs))
 (outputs outputs)
 (build python-build)
 (arguments (strip-keyword-arguments private-keywords arguments)
--8<---cut here---end--->8---

As for the native-inputs, they get stored in the bag's build-inputs,
which eventually find their way to the "inputs" keyword argument used on
the build side by the various build phases.  In fact, the inputs,
propagated-inputs, and native-inputs of any package that uses the
python-build-system are all put into this "inputs" keyword argument.

With this in mind, I have two questions:

* Should we change these native-inputs to inputs to prevent confusion?
  I can personally vouch for the fact that the presence of native-inputs
  in python-build-system packages confused the heck out of me at first!

* Are there any circumstances under which it actually WOULD make sense
  to cross-compile a Python package?

For now, I think the answers to these questions are "sure" and "probably
not", respectively.  I'm very curious to hear your thoughts about the
second question, in particular!

-- 
Chris


signature.asc
Description: PGP signature