Hi
I programmed the whole day and learned a lot. :D
On 2018-11-30 17:13, swedebugia wrote:
snip
TODO:
* make npm-recursive-import work by not fetching blacklisted packages
See the attached patches :)
* implement keyword blacklisting based on the descriptions
* match not just the whole string of blacklisted packages:
e.g. match also "rollup-plugin" when "rollup" is in the blacklist.
Done. See patch
* get the tarballs from npm-registry instead as they are never missing
(githubs sometimes are) and likely reproducible.
Done, but not tested yet.
Also added a npm-uri (similar to pypi-uri) see patch.
Happy hacking!
--
Cheers Swedebugia
>From 9ad29ede5b26a9052518adb422bc66398c38c9b4 Mon Sep 17 00:00:00 2001
From: swedebugia <swedebu...@riseup.net>
Date: Fri, 30 Nov 2018 20:52:42 +0100
Subject: [PATCH 1/2] guix: Add a "npm-uri" helper method.
* guix/build-system/node.scm (npm-uri): New method.
---
guix/build-system/node.scm | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)
diff --git a/guix/build-system/node.scm b/guix/build-system/node.scm
index b6d362904..ce0d9f84c 100644
--- a/guix/build-system/node.scm
+++ b/guix/build-system/node.scm
@@ -1,5 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2016 Jelle Licht <jli...@fsfe.org>
+;;; Copyright © 2018 swedebugia <swedebu...@riseup.net>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -30,7 +31,22 @@
#:export (npm-meta-uri
%node-build-system-modules
node-build
- node-build-system))
+ node-build-system
+ npm-uri))
+
+;; Commentary:
+;;
+;; Standard build procedure for Node packages using 'packages.json' and
+;; Makefiles. This is implemented as an extension of 'gnu-build-system'.
+;;
+;; Code:
+
+(define* (npm-uri name version #:optional (extension ".tgz"))
+ "Return a URI string for the NPM package hosted on the NPM Registry
+corresponding to NAME and VERSION. EXTENSION is the file name extension, such
+as '.tar.gz'. E.g. https://registry.npmjs.org/async/-/async-1.8.0.tgz";
+ (string-append "https://registry.npmjs.org/"; name "/-/" name "-" version
+ extension))
(define (npm-meta-uri name)
"Return a URI string for the metadata of node module NAME found in the npm
--
2.19.1
>From 54515dbb9e30e3cdf0ac1835c465f157302aefd1 Mon Sep 17 00:00:00 2001
From: swedebugia <swedebu...@riseup.net>
Date: Sat, 1 Dec 2018 00:15:08 +0100
Subject: [PATCH 2/2] guix: Add blacklisting to recursive importer
* guix/import/npm.scm (blacklisted?, npm->guix-package): Remove blacklisted
packages from the list passed to the recursive importer.
* guix/scripts/import/npm.scm (show-help, %options): Document and improve
"recursive" option.
* doc/guix.texi (Running guix import): Document the blacklist function.
Inspired by ac906cb7bb2ec77821ddec291db4857cc812599d by Ricardo
---
doc/guix.texi | 33 ++++++++++++++++++++++++++++++++-
guix/import/npm.scm | 16 ++++++++++++----
guix/scripts/import/npm.scm | 10 ++++++++--
3 files changed, 52 insertions(+), 7 deletions(-)
diff --git a/doc/guix.texi b/doc/guix.texi
index 8f694a20e..ccb3c45f1 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -7258,7 +7258,7 @@ guix import npm minimist
@end example
The importer implements an experimental recursive subsystem, allowing
-for a significant of npm packages to be imported. The command below
+for a significant number of npm packages to be imported. The command below
imports metadata for the @code{optimist} npm package, as well as its
dependencies.
@@ -7266,6 +7266,37 @@ dependencies.
guix import npm -r optimist
@end example
+The importer currently has does not warn when a cyclic dependency has been
+encountered. It will run until the buffer overruns which could take a long
+time. You should generally see output every couple of seconds, if not a cyclic
+dependency has probably been encountered. These are best avoided by first
+packaging an earlier version of one of the packages without the offending
+input to avoid the cycle.
+
+@quotation Note
+We have implemented an experimental blacklist functionality, enabled by
+default, to help you avoid importing a lot of garbage development
+dependencies. You can customize this blacklist by editing
+@file{guix/import/npm-blacklist.scm} only when running @command{guix import
+npm} from a git checkout, see @pxref{Running Guix Before It Is Installed}.
+
+These packages and every package that begins with these 103 names are
+currently blacklisted: (matcha benchmark babel @babel webpack rollup
+browserify async ember broccoli nsp uglifyjs2 uglify-js electron statsd vega
+grunt-release lineman lerna openlayers openpgp yarnpkg wekan etherpad-lite
+meteor keybase docco jsdoc markdown eslint lint markdownlint prettier standard
+jscs @ljharb/eslint-config editorconfig-tools jshint xo tick tsml
+tsd-check @commitlint safe grunt lerna vows husky nps rimraf colors cli
+source-map chai unexpected karma karma-qunit karma-sauce-launcher
+karma-browserify coveralls covert nyc qunit mocha yargs sinon code tape
+colortape airtap tap tap-spec lab mock nyc proxyquire coffe-script cross-swawn
+terst stream-spigot make-generator-function forking-tap bogota faucet gremlins
+jest redux-mock-store webdriverio ava enzyme testem protractor testcafe
+test262 storybook jasmine volkswagen nightwatch supertest istanbul sizzy
+zombie PhantomCSS)
+This expands to thousands of packages.
+@end quotation
+
@item elpa
@cindex elpa
Import metadata from an Emacs Lisp Package Archive (ELPA) package
diff --git a/guix/import/npm.scm b/guix/import/npm.scm
index 2efe81fa9..1d46fd384 100644
--- a/guix/import/npm.scm
+++ b/guix/import/npm.scm
@@ -350,8 +350,12 @@ a git checkout."
,(guix-hash-url temp-dir)))))))))))
(define (blacklisted? name)
- "Check if the pair name-version is blacklisted. RETURN #t if yes, else #f."
- (member name blacklist))
+ "Check if the string name is blacklisted. RETURN #t if yes, else #f."
+ ;; Split the string to enable ut so blacklist scoped packages like
+ ;; @babel/core and packages like eslint-popup without having to type in
+ ;; every single combination.
+ (if (member (car (string-split name (char-set #\- #\/))) blacklist)
+ #t #f))
(define (sanitize-npm-version version)
"Return version without prefixed ^."
@@ -518,8 +522,12 @@ npm list of dependencies DEPENDENCIES."
(npm-dependencies
(append
(extract-npm-dependencies dependencies)
- ;; TODO blacklist!
- (extract-npm-dependencies dev-dependencies)))
+ ;; Remove the blacklisted devdeps to avoid big dependency
+ ;; cycles and all the linters, cli-tools, benchmarking,
+ ;; etc. JS-devs adds to their development environment.
+ (remove
+ blacklisted?
+ (extract-npm-dependencies dev-dependencies))))
(description (assoc-ref package "description"))
(home-page (assoc-ref package "homepage"))
(license (extract-license curr))
diff --git a/guix/scripts/import/npm.scm b/guix/scripts/import/npm.scm
index 1f4ed777c..c540c3574 100644
--- a/guix/scripts/import/npm.scm
+++ b/guix/scripts/import/npm.scm
@@ -1,5 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2015 David Thompson <da...@gnu.org>
+;;; Copyright © 2018 swedebugia <swedebu...@riseup.net>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -25,6 +26,7 @@
#:use-module (srfi srfi-1)
#:use-module (srfi srfi-11)
#:use-module (srfi srfi-37)
+ #:use-module (srfi srfi-41)
#:use-module (ice-9 match)
#:use-module (ice-9 format)
#:export (guix-import-npm))
@@ -41,7 +43,9 @@
Import and convert the npm package for PACKAGE-NAME.\n"))
(display (G_ "
-h, --help display this help and exit"))
- (display (G_ "
+ (display (G_ "
+ -r, --recursive import packages recursively"))
+ (display (G_ "
-V, --version display version information and exit"))
(newline)
(show-bug-report-information))
@@ -88,7 +92,9 @@
`(define-public ,(string->symbol name)
,pkg))
(G_ #f))
- (recursive-import package-name))
+ (reverse
+ (stream->list
+ (npm-recursive-import package-name))))
;; Single import
(let ((sexp (npm->guix-package package-name)))
(unless sexp
--
2.19.1
