Re: Help wanted with recursive npm import returning #f

[swedebugia]

On 2018-12-04 23:07, Jelle Licht wrote:
> Hi swedebugia,
> 
> Super swell to see you take an interest in this! :D.

I'm so glad to be part of this community and that I finally understand
more about how to implement my ideas. :)

> 
> Some points;
> 
> It seems you wrote your own sanitize-npm-version, but this is not (at
> all) how npm
> deals with versions; I implore you to have a look at
> https://semver.org/ again to see
> what all the silly squigles mean :).

Thanks for the link. I did not mean for it to do anything other that
provide us with a triplet "?.?.?" to append to input-names.

> 
> A general question on your blacklist approach; I like and appreciate
> this attempt 
> 
> at 'breaking the cycle', but could we not just as well define some
> dummy packages?
> As far as the importer is concerned, a dummy package would still be
> seen as 
> 
> "dependency resolved, my work here is done", or am I missing an
> advantage of 
> 
> your approach?

I had not thought about that! Sounds like a good idea. :)

We are going to need quite a few hundred of these dummy packages though.


The advantage of blacklist is to let the import user self determine what
to add and not to the native inputs.

Actually in this version I completely removed the generation of native
inputs and thus also the blacklisting functionallity.

Maybe we can have both dummy-pkg and blacklist. Preferable we should
implement the ability to add a blacklist.scm on the command line like a
manifest. 

> 
> Op di 4 dec. 2018 om 21:44 schreef :
> 
>> Hi
>>
>> Introduction
>> 
>> Inspired by Ricardos commit here I rewrote most of the npm importer.
>>
>> Added memoization, receive, stream->list, values and rewrote the
>> tarball
>> fetcher to use only npm-uri and tarballs from the registry.
>> Additionally
>> I implemented handling of scoped packages (e.g. @babel/core).
> 
>> It contains less lines of code than Jelles importer.
> 
> And hopefully less places for bugs to hide in.

:D It modeled it very very close to the pypi-importer. So they are very
similar now.

> 
>> The single import works and is a lot faster and more reliable than
>> before when fuzzy matching on github was used. See it in action:
>> http://paste.debian.net/1054384/
> 
> I think it is a step in the _wrong_ direction to depend in major ways
> on the npm 
> registry, besides for meta-information where needed. Nonetheless, the
> fuzzy 
> 
> matching was really brittle as you say, and could have been a lot
> faster indeed. 

We could have 2 importers, one for npm-registry and one for npm-github.
As you say we dont know how long the registry will work. All the 200
packages I have looked is on GH.

> 
>> Caveats:
>> 1) we don't know if the registry-tarballs are reproducible.
> 
> Back in the day, they most definitely were not. Seeing as npm-land
> does not put
> an emphasis on this at all, I think it is unwise to assume that any
> reproducible
> features they offer today will still be available in the future.
> 
>> 2) filename is the same as the upstream tarball -> we should convert
>> it
>> to guix-name.
>> 3) we have to download the tarball because sha256 is not among the
>> hashes computed by npm. (I emailed n...@npmjs.org to ask for them to
>> compute it for all their tarballs :) )
> 
> The result of the importer will probably be a package that we will be 
> 
> building in the near future, right?
> 
>> Help wanted
>> ---
>>
>> There is a bug which only affects the recursive importer. I tried
>> hard
>> finding it but I'm in way over my head and my guile-foo does not
>> seem to
>> cut it with this one. :)
> 
>> For recursive output it downloads but outputs #f in the end instead
>> of
>> the sexps. See example output: http://paste.debian.net/1054383/
>>
>> Trying to debug via the REPL I met this:
>> scheme@(guile-user) [1]> (npm-recursive-import "async")
>> $3 = #
>>
>> Any ideas?
> 
> I think I also ran into this. Could you please make your work
> available on a
> public repo somewhere? It would be easier to look at your changes and
> play
> around with it that way.

Good idea. Is this ok? https://gitlab.com/swedebugia/guix/commits/npm

It was my first push ever! :)

-- 
Cheers 
Swedebugia

Re: Help wanted with recursive npm import returning #f

[Jelle Licht]

Op di 4 dec. 2018 om 22:07 schreef Jelle Licht :

> Hi swedebugia,
>
> Super swell to see you take an interest in this! :D.
>
> Some points;
>
> It seems you wrote your own sanitize-npm-version, but this is not (at all)
> how npm
> deals with versions; I implore you to have a look at https://semver.org/
> again to see
> what all the silly squigles mean :).
>
Oops, I meant https://www.npmjs.com/package/semver

>
> [...]
>

Re: Help wanted with recursive npm import returning #f

[Jelle Licht]

Hi swedebugia,

Super swell to see you take an interest in this! :D.

Some points;

It seems you wrote your own sanitize-npm-version, but this is not (at all)
how npm
deals with versions; I implore you to have a look at https://semver.org/
again to see
what all the silly squigles mean :).

A general question on your blacklist approach; I like and appreciate this
attempt
at 'breaking the cycle', but could we not just as well define some dummy
packages?
As far as the importer is concerned, a dummy package would still be seen as
"dependency resolved, my work here is done", or am I missing an advantage
of
your approach?

Op di 4 dec. 2018 om 21:44 schreef :

> Hi
>
> Introduction
> 
> Inspired by Ricardos commit here I rewrote most of the npm importer.
>
> Added memoization, receive, stream->list, values and rewrote the tarball
> fetcher to use only npm-uri and tarballs from the registry. Additionally
> I implemented handling of scoped packages (e.g. @babel/core).
>

> It contains less lines of code than Jelles importer.
>
And hopefully less places for bugs to hide in.


> The single import works and is a lot faster and more reliable than
> before when fuzzy matching on github was used. See it in action:
> http://paste.debian.net/1054384/


I think it is a step in the _wrong_ direction to depend in major ways on
the npm
registry, besides for meta-information where needed. Nonetheless, the fuzzy
matching was really brittle as you say, and could have been a lot faster
indeed.


> Caveats:
> 1) we don't know if the registry-tarballs are reproducible.
>
Back in the day, they most definitely were not. Seeing as npm-land does not
put
an emphasis on this at all, I think it is unwise to assume that any
reproducible
features they offer today will still be available in the future.

2) filename is the same as the upstream tarball -> we should convert it
> to guix-name.
> 3) we have to download the tarball because sha256 is not among the
> hashes computed by npm. (I emailed n...@npmjs.org to ask for them to
> compute it for all their tarballs :) )
>
The result of the importer will probably be a package that we will be
building in the near future, right?

>
> Help wanted
> ---
>
> There is a bug which only affects the recursive importer. I tried hard
> finding it but I'm in way over my head and my guile-foo does not seem to
> cut it with this one. :)
>

> For recursive output it downloads but outputs #f in the end instead of
> the sexps. See example output: http://paste.debian.net/1054383/
>
> Trying to debug via the REPL I met this:
> scheme@(guile-user) [1]> (npm-recursive-import "async")
> $3 = #
>
> Any ideas?


I think I also ran into this. Could you please make your work available on a
public repo somewhere? It would be easier to look at your changes and play
around with it that way.

>
> --
> Cheers
> Swedebugia

Help wanted with recursive npm import returning #f

[swedebugia]

Hi

Introduction

Inspired by Ricardos commit here I rewrote most of the npm importer.

Added memoization, receive, stream->list, values and rewrote the tarball
fetcher to use only npm-uri and tarballs from the registry. Additionally
I implemented handling of scoped packages (e.g. @babel/core).

It contains less lines of code than Jelles importer.

The single import works and is a lot faster and more reliable than
before when fuzzy matching on github was used. See it in action:
http://paste.debian.net/1054384/

Caveats:
1) we don't know if the registry-tarballs are reproducible.
2) filename is the same as the upstream tarball -> we should convert it
to guix-name.
3) we have to download the tarball because sha256 is not among the
hashes computed by npm. (I emailed n...@npmjs.org to ask for them to
compute it for all their tarballs :) )

Help wanted
---

There is a bug which only affects the recursive importer. I tried hard
finding it but I'm in way over my head and my guile-foo does not seem to
cut it with this one. :) 

For recursive output it downloads but outputs #f in the end instead of
the sexps. See example output: http://paste.debian.net/1054383/

Trying to debug via the REPL I met this:
scheme@(guile-user) [1]> (npm-recursive-import "async")
$3 = #

Any ideas? 

-- 
Cheers 
Swedebugia;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2015 David Thompson 
;;; Copyright © 2016 Jelle Licht 
;;; Copyright © 2018 swedebugia 
;;;
;;; This file is part of GNU Guix.
;;;
;;; GNU Guix is free software; you can redistribute it and/or modify it
;;; under the terms of the GNU General Public License as published by
;;; the Free Software Foundation; either version 3 of the License, or (at
;;; your option) any later version.
;;;
;;; GNU Guix is distributed in the hope that it will be useful, but
;;; WITHOUT ANY WARRANTY; without even the implied warranty of
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;;; GNU General Public License for more details.
;;;
;;; You should have received a copy of the GNU General Public License
;;; along with GNU Guix.  If not, see .

(define-module (guix import npm)
  #:use-module (ice-9 binary-ports)
  #:use-module (ice-9 control)
  #:use-module (ice-9 match)
  #:use-module (ice-9 receive)
  #:use-module (ice-9 regex)
  #:use-module (ice-9 rdelim)
  #:use-module (ice-9 vlist)
  #:use-module (gcrypt hash)
  #:use-module (gnu packages)
  #:use-module (guix base32)
  #:use-module (guix build git)
  #:use-module (guix build-system node)
  #:use-module ((guix download) #:prefix download:)
;;  #:use-module (guix import github)
  #:use-module (guix import json)
  #:use-module (guix import utils)
  #:use-module ((guix licenses) #:select (expat))
  #:use-module (guix packages)
  #:use-module (guix serialization)
  #:use-module (guix upstream)
  #:use-module (guix utils)
  #:use-module (json)
  #:use-module (rnrs bytevectors)
  #:use-module (rnrs io ports)
  #:use-module (srfi srfi-1)
  #:use-module (srfi srfi-11)
  #:use-module (srfi srfi-9 gnu)
  #:use-module (srfi srfi-9)
  #:use-module (srfi srfi-26)
  #:use-module (srfi srfi-34)
  #:use-module (srfi srfi-35)
  #:use-module (web uri)
  #:export (npm->guix-package
npm-recursive-import
;; For debugging in the REPL:
npm-fetch
list-requirements
maybe-inputs))
;;;
;;; Comment:
;;; This is the npm importer. 
;;; Native-inputs are not considered by the importer at this stage because the
;;; code is adapted from the pypi importer and because they are for the most
;;; part not needed to build or use the package.

;;; We should compute and include the native-inputs and include them
;;; commented out. To be able to do this we need something (a new
;;; syntax-rule?) to pass comments in the returned sexp like this:
;;; (native-inputs)
;;; `((
;;; ;;("input" ,input)
;;; ))

;;; We should add a flag to the command line to enable import of
;;; devdependencies aka. native inputs if the user desires.
;;; Perhaps a flag to indicate max levels of recursiveness is also useful to
;;; avoid ending up with 100+ records with one command.

;;;
;;; Code
;;;

(define *REGISTRY*  "https://registry.npmjs.org/;)

(define (npm-fetch name)
  "Return metadata from the npm registry for package NAME."
  (json-fetch-alist (string-append *REGISTRY* name)))

(define (npm-tarball alist version)
  "Return the *REGISTRY* tarball url for version VERSION of ALIST"
  (let* ((v (assoc-ref* alist "versions" version))
 (d (assoc-ref* v "dist")))
(assoc-ref* d "tarball")))

;; TODO use this to check the tarball
(define (npm-tarball-sha512 alist version)
  "Return the *REGISTRY* sha512sum for version VERSION of ALIST or #f if not
found"
  (let* ((v (assoc-ref* alist "versions" version))
 (d (assoc-ref* v "dist")))
(assoc-ref* d "integrity")))

(define (npm-latest-release alist)
  "Return a string with the latest