On Fri, Feb 14 2020 at 13:46 +0100, Michael Olbrich <[email protected]> wrote: > On Tue, Feb 04, 2020 at 08:53:59AM +0100, Ulrich Ölmann wrote: >> From: Björn Esser <[email protected]> >> >> This is a dummy package to specify and install required Node.js packages for >> the >> target system. >> >> The management of those packages and their dependencies is handled with >> Yarn (run through host-nodejs) to maintain a reproducible offline cache that >> is >> stored in local_src/nodejs_packages. >> >> Signed-off-by: Björn Esser <[email protected]> >> [uol: adjusted commit message and Kconfig menu file's help texts, integrated >> Kconfig symbol NODEJS_PACKAGES_LICENSE, fixed NODEJS_PACKAGES_URL, removed >> one >> square bracket in a shell test and wrapped install/cp into call of execute >> and >> world/execute] >> Signed-off-by: Ulrich Ölmann <[email protected]> >> --- >> v3 --> v4: adjusted commit message and Kconfig menu file's help texts & fixed >> NODEJS_PACKAGES_URL >> Range-diff of v5 against v4: >> 3: ccd2b44a8d92 ! 3: 1a73ebbd6e7e nodejs_packages: new package >> @@ Commit message >> stored in local_src/nodejs_packages. >> >> Signed-off-by: Björn Esser <[email protected]> >> - [uol: adjusted commit message and Kconfig menu file's help texts & >> fixed >> - NODEJS_PACKAGES_URL] >> + [uol: adjusted commit message and Kconfig menu file's help texts, >> integrated >> + Kconfig symbol NODEJS_PACKAGES_LICENSE, fixed NODEJS_PACKAGES_URL, >> removed one >> + square bracket in a shell test and wrapped install/cp into call of >> execute and >> + world/execute] >> Signed-off-by: Ulrich Ölmann <[email protected]> >> >> ## rules/nodejs_packages.in (new) ## >> @@ rules/nodejs_packages.in (new) >> + >> + [1] https://legacy.yarnpkg.com/en/docs/dependency-versions/ >> + >> ++config NODEJS_PACKAGES_LICENSE >> ++ string >> ++ prompt "Licenses of to be installed Node.js packages" >> ++ default "unknown" >> ++ help >> ++ Manually compiled list of licenses that the to be installed >> Node.js >> ++ packages are distributed under, see description of the >> variable >> ++ *_LICENSE in [1]. >> ++ >> ++ ATTENTION: be aware of the fact that Node.js packages usually >> depend >> ++ on additional Node.js packages that Yarn installs as well and >> that >> ++ this propagates recursively resulting in an unknown number of >> Node.js >> ++ packages on the target with an unknown set of licenses that >> they may >> ++ be utilized under. Hence you carefully need to examine all of >> them >> ++ individually to create a reliable list of licenses. >> ++ >> ++ [1] >> https://www.ptxdist.org/doc/dev_manual.html#some-notes-about-licenses >> ++ >> +config NODEJS_PACKAGES_OFFLINE >> + bool >> -+ prompt "lock offline cache for offline/production use" >> ++ prompt "Lock offline cache for offline/production use" >> + help >> -+ This option locks the offline cache for installing the nodejs >> packages >> -+ and allows their installation during the build of the BSP >> without any >> -+ need for internet connectivity. >> ++ This option locks the offline cache for installing the Node.js >> ++ packages and allows their installation during the build of >> the BSP >> ++ without any need for internet connectivity. >> + >> + Enabling this option is highly recommended for production use. >> + >> @@ rules/nodejs_packages.make (new) >> +NODEJS_PACKAGES_URL := >> lndir://$(PTXDIST_WORKSPACE)/$(NODEJS_PACKAGES_LOCAL) >> +NODEJS_PACKAGES_DIR := $(BUILDDIR)/$(NODEJS_PACKAGES) >> +NODEJS_PACKAGES_CACHE := >> $(PTXDIST_WORKSPACE)/$(NODEJS_PACKAGES_LOCAL)/yarn_cache >> -+NODEJS_PACKAGES_LICENSE := unknown >> ++NODEJS_PACKAGES_LICENSE := $(call remove_quotes, >> $(PTXCONF_NODEJS_PACKAGES_LICENSE)) >> + >> +NODEJS_PACKAGES_LIST := $(call remove_quotes, >> $(PTXCONF_NODEJS_PACKAGES_LIST)) >> + >> @@ rules/nodejs_packages.make (new) >> + \"license\": \"UNLICENSED\", \ >> + \"private\": true \ >> + }" > $(NODEJS_PACKAGES_DIR)/package.json >> -+ if [[ ! -f $(YARN_LOCK) ]]; then \ >> ++ if [ ! -f $(YARN_LOCK) ]; then \ >> + touch $(YARN_LOCK); \ >> + ln -fs $(YARN_LOCK) $(NODEJS_PACKAGES_DIR)/yarn.lock; \ >> + fi >> @@ rules/nodejs_packages.make (new) >> + >> +$(STATEDIR)/nodejs_packages.install: >> + @$(call targetinfo) >> -+ install -dm 0755 $(NODEJS_PACKAGES_PKGDIR)/usr/lib >> -+ cp -pr $(NODEJS_PACKAGES_DIR)/node_modules >> $(NODEJS_PACKAGES_PKGDIR)/usr/lib >> ++ @$(call world/execute, HOST_UA_NODESET, \ >> ++ install -dm 0755 $(NODEJS_PACKAGES_PKGDIR)/usr/lib) >> ++ @$(call execute, HOST_UA_NODESET, \ >> ++ cp -pr $(NODEJS_PACKAGES_DIR)/node_modules \ >> ++ $(NODEJS_PACKAGES_PKGDIR)/usr/lib) >> + @$(call touch) >> + >> +# >> ---------------------------------------------------------------------------- >> >> rules/nodejs_packages.in | 73 ++++++++++++++++++++++++++++++ >> rules/nodejs_packages.make | 93 ++++++++++++++++++++++++++++++++++++++ >> 2 files changed, 166 insertions(+) >> create mode 100644 rules/nodejs_packages.in >> create mode 100644 rules/nodejs_packages.make >> >> diff --git a/rules/nodejs_packages.in b/rules/nodejs_packages.in >> new file mode 100644 >> index 000000000000..aea44205da41 >> --- /dev/null >> +++ b/rules/nodejs_packages.in >> @@ -0,0 +1,73 @@ >> +## SECTION=bytecode_engines >> + >> +menuconfig NODEJS_PACKAGES >> + tristate >> + select HOST_YARN >> + select NODEJS >> + prompt "Node.js packages (target) " >> + help >> + Reproducibly install Node.js packages that will be available >> + system-wide on the target. >> + >> + You MUST specify at least one Node.js package in the submenu that is >> + shown after enabling this option. >> + >> + >> +if NODEJS_PACKAGES >> + >> +config NODEJS_PACKAGES_LIST >> + string >> + prompt "Node.js packages to install" >> + help >> + A white-space separated list of Node.js packages including their >> + versions (e.g. [email protected] or socket.io@'>=2.1.0 <=2.3.0'). NPM >> + semvers are fully supported, see [1]. >> + >> + The packages are managed with Yarn and are guaranteed to be >> + reproducible and immutable as long as the download location stays >> + accessible or the offline cache is not altered. >> + >> + If the list of Node.js packages to be installed has been populated for >> + the first time or if it has been modified later on one needs to >> + (re-)build this package to download the desired Node.js packages and >> + update the offline cache. Afterwards it is highly recommended to >> + commit the path '${PTXDIST_WORKSPACE}/local_src/nodejs_packages/' into >> + the version control system and to lock the package cache. >> + >> + As a prerequisite the path '${PTXDIST_WORKSPACE}/local_src/nodejs_ >> + packages/' must exist in the BSP *before* building this package. >> + >> + [1] https://legacy.yarnpkg.com/en/docs/dependency-versions/ >> + >> +config NODEJS_PACKAGES_LICENSE >> + string >> + prompt "Licenses of to be installed Node.js packages" >> + default "unknown" >> + help >> + Manually compiled list of licenses that the to be installed Node.js >> + packages are distributed under, see description of the variable >> + *_LICENSE in [1]. >> + >> + ATTENTION: be aware of the fact that Node.js packages usually depend >> + on additional Node.js packages that Yarn installs as well and that >> + this propagates recursively resulting in an unknown number of Node.js >> + packages on the target with an unknown set of licenses that they may >> + be utilized under. Hence you carefully need to examine all of them >> + individually to create a reliable list of licenses. >> + >> + [1] >> https://www.ptxdist.org/doc/dev_manual.html#some-notes-about-licenses >> + >> +config NODEJS_PACKAGES_OFFLINE >> + bool >> + prompt "Lock offline cache for offline/production use" >> + help >> + This option locks the offline cache for installing the Node.js >> + packages and allows their installation during the build of the BSP >> + without any need for internet connectivity. >> + >> + Enabling this option is highly recommended for production use. >> + >> + Do NOT forget to commit the path '${PTXDIST_WORKSPACE}/local_src/ >> + nodejs_packages/' into the BSP's version control system! >> + >> +endif >> diff --git a/rules/nodejs_packages.make b/rules/nodejs_packages.make >> new file mode 100644 >> index 000000000000..cdde30f38b0c >> --- /dev/null >> +++ b/rules/nodejs_packages.make >> @@ -0,0 +1,93 @@ >> +# -*-makefile-*- >> +# >> +# Copyright (C) 2020 by Bjoern Esser <[email protected]> >> +# >> +# For further information about the PTXdist project and license conditions >> +# see the README file. >> +# >> + >> +# >> +# We provide this package >> +# >> +PACKAGES-$(PTXCONF_NODEJS_PACKAGES) += nodejs_packages >> + >> +# >> +# Paths and names >> +# >> +NODEJS_PACKAGES_VERSION := 0.0.1 >> +NODEJS_PACKAGES := nodejs_packages-$(NODEJS_PACKAGES_VERSION) >> +NODEJS_PACKAGES_LOCAL := local_src/nodejs_packages >> +NODEJS_PACKAGES_URL := lndir://$(PTXDIST_WORKSPACE)/$(NODEJS_PACKAGES_LOCAL) >> +NODEJS_PACKAGES_DIR := $(BUILDDIR)/$(NODEJS_PACKAGES) >> +NODEJS_PACKAGES_CACHE := >> $(PTXDIST_WORKSPACE)/$(NODEJS_PACKAGES_LOCAL)/yarn_cache >> +NODEJS_PACKAGES_LICENSE := $(call remove_quotes, >> $(PTXCONF_NODEJS_PACKAGES_LICENSE)) >> + >> +NODEJS_PACKAGES_LIST := $(call remove_quotes, >> $(PTXCONF_NODEJS_PACKAGES_LIST)) >> + >> +YARN_LOCK := $(PTXDIST_WORKSPACE)/$(NODEJS_PACKAGES_LOCAL)/yarn.lock >> +YARN_OPTS := \ >> + --cwd "$(NODEJS_PACKAGES_DIR)" \ >> + --cache-folder "$(NODEJS_PACKAGES_CACHE)" \ >> + --link-duplicates \ >> + --production=true >> + >> +ifdef PTXCONF_NODEJS_PACKAGES_OFFLINE >> +YARN_OPTS += \ >> + --frozen-lockfile \ >> + --offline >> +endif >> + >> +# >> ---------------------------------------------------------------------------- >> +# Compile >> +# >> ---------------------------------------------------------------------------- >> + >> +$(STATEDIR)/nodejs_packages.compile: >> + @$(call targetinfo) >> + mkdir -p $(NODEJS_PACKAGES_CACHE) >> + printf "{ \ >> + \"name\": \"nodejs_packages\", \ >> + \"version\": \"$(NODEJS_PACKAGES_VERSION)\", \ >> + \"license\": \"UNLICENSED\", \ >> + \"private\": true \ >> + }" > $(NODEJS_PACKAGES_DIR)/package.json >> + if [ ! -f $(YARN_LOCK) ]; then \ >> + touch $(YARN_LOCK); \ >> + ln -fs $(YARN_LOCK) $(NODEJS_PACKAGES_DIR)/yarn.lock; \ >> + fi > > I think we should fail here if NODEJS_PACKAGES_OFFLINE is set and the lock > file is missing. Or will yarn produce a sensible error message in this > case?
Unfortunately Yarn is not stumbling across a missing lock file so I added a check for PTXdist to bail out in that case in v6 of the patch series. >> + yarn $(YARN_OPTS) add $(NODEJS_PACKAGES_LIST) >> + find $(NODEJS_PACKAGES_CACHE) -type f -name '.yarn-tarball.tgz' -delete >> + @$(call touch) >> + >> +# >> ---------------------------------------------------------------------------- >> +# Install >> +# >> ---------------------------------------------------------------------------- >> + >> +$(STATEDIR)/nodejs_packages.install: >> + @$(call targetinfo) >> + @$(call world/execute, HOST_UA_NODESET, \ >> + install -dm 0755 $(NODEJS_PACKAGES_PKGDIR)/usr/lib) > > add -v to make it visible at runtime. Did that in v6 of the patch series. >> + @$(call execute, HOST_UA_NODESET, \ >> + cp -pr $(NODEJS_PACKAGES_DIR)/node_modules \ >> + $(NODEJS_PACKAGES_PKGDIR)/usr/lib) > > Here as well. See v6 of the patch series. Best regards Ulrich > Michael > >> + @$(call touch) >> + >> +# >> ---------------------------------------------------------------------------- >> +# Target-Install >> +# >> ---------------------------------------------------------------------------- >> + >> +$(STATEDIR)/nodejs_packages.targetinstall: >> + @$(call targetinfo) >> + >> + @$(call install_init, nodejs_packages) >> + @$(call install_fixup, nodejs_packages,PRIORITY,optional) >> + @$(call install_fixup, nodejs_packages,SECTION,base) >> + @$(call install_fixup, nodejs_packages,AUTHOR,"Bjoern Esser >> <[email protected]>") >> + @$(call install_fixup, nodejs_packages,DESCRIPTION,missing) >> + >> + $(call install_tree, nodejs_packages, 0, 0, -, /usr/lib/node_modules/) >> + >> + @$(call install_finish, nodejs_packages) >> + >> + @$(call touch) >> + >> +# vim: syntax=make >> -- >> 2.25.0 >> >> >> _______________________________________________ >> ptxdist mailing list >> [email protected] -- Pengutronix e.K. | Ulrich Ölmann | Industrial Linux Solutions | http://www.pengutronix.de/ | Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 | Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 | _______________________________________________ ptxdist mailing list [email protected]
