Re: [yocto] NPM Fetcher & License collection in Yocto Zeus
Hi Olivier,
Thanks for your implication :)
I think the modification you want to make resides in the npm class file:
https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/classes/npm.bbclass?h=zeus
But, starting with dunfell, the npm related mechanism have been reworked
and also the npm class:
https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/classes/npm.bbclass?h=dunfell
Is it possible to test if you have the same issue with dunfell ?
Best Regards,
Jean-Marie
On Fri, Jan 22, 2021 at 10:00 AM Oliver Westermann <
oliver.westerm...@cognex.com> wrote:
> Hey,
>
> We’ve upgraded to zeus some time ago and also updated some of our recipes
> around node-red and it’s nodes. Before I created my own class to easily
> fetch several nodes, now we used devtool to create those recipes. In zeus,
> this happens using the npm:// fetcher.
>
> This worked fine, but we sometimes had QA errors, hard to reproduce.
> Sometimes on some peoples PC, the license files were missing when
> do_populate_lic was running.
>
> ERROR: node-red-contrib-socketio-1.0.7-r0 do_populate_lic: QA Issue:
> node-red-contrib-socketio: LIC_FILES_CHKSUM points to an invalid file:
> /data/workspace/yocto-build/build/cognex-myna/tmp/work/aarch64-poky-linux/node-red-contrib-socketio/1.0.7-r0/npmpkg/node_modules/
> socket.io/node_modules/@types/cookie/LICENSE [license-checksum]
>
> Turned out that do_unpack would download the dependencies of the
> node-red-module (so in the example above, node-red-contrib-socketio depends
> on socket.io), but do_install would package those away/clean them up.
> Since they go into the resulting package, we still have to list those
> licenses, so I fixed it by adding
>
> do_install[depends] += "${PN}:do_populate_lic"
>
> to my packages to fix the order.
>
> However I would like to fix this upstream, but couldn't find the correct
> source location to add this dependency. Can sb help me figure this out as I
> would like to contribute
>
> Best regards, Olli
>
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#52067): https://lists.yoctoproject.org/g/yocto/message/52067
Mute This Topic: https://lists.yoctoproject.org/mt/80026361/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] Parsing recipes: 0% | - hangs for a while
Maybe related to this: meta/conf/distro/include/default-distrovars.inc # The CONNECTIVITY_CHECK_URIS are used to test whether we can succesfully # fetch from the network (and warn you if not). To disable the test set # the variable to be empty. # Git example url: git://git.yoctoproject.org/yocto-firewall-test;protocol=git;rev=master CONNECTIVITY_CHECK_URIS ?= "https://www.example.com/; Try to overwrite CONNECTIVITY_CHECK_URIS in your local.conf by something like google.com or even an empty string and see the result. Best regards, Jean-Marie On Wed, Nov 25, 2020 at 12:46 PM Sergey Ivanov wrote: > > The issue was located. I had overridden to the local path of one recipe via > > "devtool modify -n " /my/local/path > and during parsing it hangs at > "Inheriting /yocto/sources/poky/meta/classes/externalsrc.bbclass" from custom > devtools bbappend for my recipe. > Moreover, "devtool reset -n " turned out into other errors: > > > > NOTE: Reconnecting to bitbake server... > NOTE: Retrying server connection (#1)... > NOTE: Reconnecting to bitbake server... > NOTE: Reconnecting to bitbake server... > NOTE: Retrying server connection (#1)... > NOTE: Retrying server connection (#1)... > NOTE: Reconnecting to bitbake server... > NOTE: Retrying server connection (#2)... > NOTE: Reconnecting to bitbake server... > NOTE: Reconnecting to bitbake server... > NOTE: Retrying server connection (#2)... > NOTE: Retrying server connection (#2)... > NOTE: Reconnecting to bitbake server... > NOTE: Retrying server connection (#3)... (Traceback (most recent call last): > File "/yocto/sources/poky/bitbake/lib/bb/main.py", line 455, in > setup_bitbake > server_connection = bb.server.process.connectProcessServer(sockname, > featureset) > File "/yocto/sources/poky/bitbake/lib/bb/server/process.py", line 490, in > connectProcessServer > sock.connect(os.path.basename(sockname)) > BlockingIOError: [Errno 11] Resource temporarily unavailable > ) > NOTE: Reconnecting to bitbake server... > NOTE: Reconnecting to bitbake server... > NOTE: Retrying server connection (#3)... (Traceback (most recent call last): > File "/yocto/sources/poky/bitbake/lib/bb/main.py", line 455, in > setup_bitbake > server_connection = bb.server.process.connectProcessServer(sockname, > featureset) > File "/yocto/sources/poky/bitbake/lib/bb/server/process.py", line 490, in > connectProcessServer > sock.connect(os.path.basename(sockname)) > BlockingIOError: [Errno 11] Resource temporarily unavailable > ) > NOTE: Retrying server connection (#3)... (Traceback (most recent call last): > File "/yocto/sources/poky/bitbake/lib/bb/main.py", line 455, in > setup_bitbake > server_connection = bb.server.process.connectProcessServer(sockname, > featureset) > File "/yocto/sources/poky/bitbake/lib/bb/server/process.py", line 490, in > connectProcessServer > sock.connect(os.path.basename(sockname)) > BlockingIOError: [Errno 11] Resource temporarily unavailable > ) > NOTE: Reconnecting to bitbake server... > NOTE: Retrying server connection (#4)... (Traceback (most recent call last): > File "/yocto/sources/poky/bitbake/lib/bb/main.py", line 455, in > setup_bitbake > server_connection = bb.server.process.connectProcessServer(sockname, > featureset) > File "/yocto/sources/poky/bitbake/lib/bb/server/process.py", line 490, in > connectProcessServer > sock.connect(os.path.basename(sockname)) > BlockingIOError: [Errno 11] Resource temporarily unavailable > ) > NOTE: Reconnecting to bitbake server... > NOTE: Reconnecting to bitbake server... > NOTE: Retrying server connection (#4)... (Traceback (most recent call last): > File "/yocto/sources/poky/bitbake/lib/bb/main.py", line 455, in > setup_bitbake > server_connection = bb.server.process.connectProcessServer(sockname, > featureset) > File "/yocto/sources/poky/bitbake/lib/bb/server/process.py", line 490, in > connectProcessServer > sock.connect(os.path.basename(sockname)) > BlockingIOError: [Errno 11] Resource temporarily unavailable > ) > NOTE: Retrying server connection (#4)... (Traceback (most recent call last): > File "/yocto/sources/poky/bitbake/lib/bb/main.py", line 455, in > setup_bitbake > server_connection = bb.server.process.connectProcessServer(sockname, > featureset) > File "/yocto/sources/poky/bitbake/lib/bb/server/process.py", line 490, in > connectProcessServer > sock.connect(os.path.basename(sockname)) > BlockingIOError: [Errno 11] Resource temporarily unavailable > ) > NOTE: Reconnecting to bitbake server... > NOTE: Retrying server connection (#5)... (Traceback (most recent call last): > File "/yocto/sources/poky/bitbake/lib/bb/main.py", line 455, in > setup_bitbake > server_connection = bb.server.process.connectProcessServer(sockname, > featureset) > File "/yocto/sources/poky/bitbake/lib/bb/server/process.py", line 490, in > connectProcessServer > sock.connect(os.path.basename(sockname)) >
Re: [yocto] dist folder missing in npm build.
Hi Jose,
Seeing your other email in the list, it seems that you are trying to
build an angular or react application.
Right now the build of these applications is not handled by Yocto yet.
For an angular application you will need to pack the ng native tool
and call it to generate the dist directory.
For a react application you have to run the npm build script but the
output files will be in a build directory.
Best regards,
Jean-Marie
On Fri, Aug 28, 2020 at 11:54 AM Cardenas Jose Antonio (JCARDENA)
wrote:
>
> Hi all,
>
>
>
> i’m trying to reproduce an older recipe that was built with npm. It seems
> that the older recipe copied server files from a “dist” folder that now is
> missing.
>
>
>
> I show the install part of the recipe:
>
> do_install() {
>
> install -d ${D}/usr/httpd/zigbee-ui
>
> cp -r ${S}/dist/* ${D}/usr/httpd/zigbee-ui
>
> }
>
>
>
> Someone knows what happened with this folder?
>
>
>
> Regards
>
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#50402): https://lists.yoctoproject.org/g/yocto/message/50402
Mute This Topic: https://lists.yoctoproject.org/mt/76470414/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] devtool add ignores "devDependencies" from package.json when generates npm-shrinkwrap.json file
Hi Jose, You have to use devtool with the --npm-dev option. Best regards, Jean-Marie On Thu, Aug 27, 2020 at 12:47 PM Cardenas Jose Antonio (JCARDENA) < joseantonio.carde...@niko.eu> wrote: > Hi all, > > > > Trying to create a recipe from scratch with “devtool add” tool, a > npm-shrinkwrap.json file is created from reading of package.json from the > module that is going to be installed. > > > > The building goes fine but I can’t run the application in the target > because nodejs misses the modules placed in “devDependencies” from > package.json. > > I have tried to include NPM_INSTALL_DEV = "1" in the recipe but I get an > error because the modules that appears in “devDependencies” from > package.json are not included in npm-shrinkwrap.json. > > There is any way to regenerate npm-shrinkwrap.json with devtool taking in > account devdependencies or a way to create npm-shrinkwrap.json with > “devtool add” in a way that takes in account devDependencies? > > > > Regards > > Jose > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#50389): https://lists.yoctoproject.org/g/yocto/message/50389 Mute This Topic: https://lists.yoctoproject.org/mt/76448481/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] [yocto-builds] npmsw fetcher error : npm-shrinkwrap.json' has invalid parameters. Unsupported dependency: arr-flatten
Hi Jose and Randy,
Here is a quick and unverified response on my phone.
It seems that your shrinkwrap has no "resolved" fields nor "integrity"
fields. See https://docs.npmjs.com/files/package-lock.json
Try re-generate one manually.
Best regards,
Jean-Marie
Le jeu. 20 août 2020 22:53, Randy MacLeod a
écrit :
> Hi Jose,
>
> On 2020-08-20 5:33 a.m., Cardenas Jose Antonio (JCARDENA) wrote:
>
> Hi all,
>
>
>
> I’m trying to fech packages and dependencies from a nmp-shrinkwrap.json
> file but I get next error:
>
>
>
> *npm-shrinkwrap.json' has invalid parameters. Unsupported
> dependency: arr-flatten*
>
> I don't know but you'll get more people on the main yocto list so I've
> replied there.
>
> People often mistakenly use yocto-builds so we're going to drop that list
> at some point.
>
> ../Randy
>
>
>
>
> The fragment of shrinkwrap file is
>
>
>
>
>
> {
>
> "name": "Gateway_node_webserver",
>
> "version": "2.5.0",
>
> "lockfileVersion": 1,
>
> "requires": true,
>
> "dependencies": {
>
> "chokidar": {
>
> "version": "1.0.6",
>
> "requires": {
>
> "anymatch": "1.3.2",
>
> "arrify": "1.0.1",
>
> "async-each": "0.1.6",
>
> "glob-parent": "1.3.0",
>
> "is-binary-path": "1.0.1",
>
> "is-glob": "1.1.3",
>
> "path-is-absolute": "1.0.1",
>
> "readdirp": "1.4.0"
>
> },
>
> "dependencies": {
>
> "anymatch": {
>
> "version": "1.3.2",
>
> "requires": {
>
> "micromatch": "2.3.11",
>
> "normalize-path": "2.1.1"
>
> },
>
> "dependencies": {
>
> "micromatch": {
>
> "version": "2.3.11",
>
> "requires": {
>
> "arr-diff": "2.0.0",
>
> "array-unique": "0.2.1",
>
> "braces": "1.8.5",
>
> "expand-brackets": "0.1.5",
>
> "extglob": "0.3.2",
>
> "filename-regex": "2.0.1",
>
> "is-extglob": "1.0.0",
>
> "is-glob": "2.0.1",
>
> "kind-of": "3.2.2",
>
> "normalize-path": "2.1.1",
>
> "object.omit": "2.0.1",
>
> "parse-glob": "3.0.4",
>
> "regex-cache": "0.4.4"
>
> },
>
> "dependencies": {
>
> "arr-diff": {
>
> "version": "2.0.0",
>
> "requires": {
>
> "arr-flatten": "1.1.0"
>
> },
>
> "dependencies": {
>
> "arr-flatten": {
>
> "version": "1.1.0"
>
> }
>
> }
>
> },
>
> "array-unique": {
>
> "version": "0.2.1"
>
> },
>
>
>
>
>
> The recipe SRC_URI is :
>
>
>
> SRC_URI = " \
>
>npmsw://${THISDIR}/${PN}/npm-shrinkwrap.json \
>
>"
>
>
>
> Any idea why is failing the fetch?
>
>
>
> Regards.
>
>
> --
> # Randy MacLeod
> # Wind River Linux
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#50311): https://lists.yoctoproject.org/g/yocto/message/50311
Mute This Topic: https://lists.yoctoproject.org/mt/76317070/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] Yocto dunfell - strange error when building npm package without dependence
Hi Edson,
You're right. I have seen seen this too. I have already created a bug
to track this issue:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13901
For now, as a workaround, you can just delete the npmsw:// line in
the generated recipe (and the shrinkwrap file).
Best Regards,
Jean-Marie
On Sat, May 30, 2020 at 4:07 PM Edson Seabra
wrote:
>
> Hi, all
>
> I used recipetool to create the recipe for NPM packages
>
> If the npm package has dependence(s) it works great.
>
> But if not, the error below happens.
>
> I just add a fake dependence in npm-shrinkwrap.json then it works
>
> # cat google-protobuf/npm-shrinkwrap.json
> {
> "name": "google-protobuf",
> "version": "3.3.0",
> "lockfileVersion": 1,
> "requires": true,
> "dependencies": {
> "nan": {
> "version": "2.14.1",
> "resolved": "https://registry.npmjs.org/nan/-/nan-2.14.1.tgz;,
> "integrity":
> "sha512-isWHgVjnFjh2x2yuJ/tj3JbwoHu3UC2dX5G/88Cm24yB6YopVgxvBObDY7n5xW6ExmFhJpSEQqFPvq9zaXc8Jw=="
> }
> }
> }
>
> Another way to make it work is to replace the npmsw with file in SRC_URI
>
> nan.bb ===
> SRC_URI = " \
> npm://registry.npmjs.org;package=nan;version=${PV} \
> file://npm-shrinkwrap.json;subdir=npm \
> "
> S = "${WORKDIR}/npm"
>
> inherit npm
>
> === google-protobuf.bb ==
>
> SRC_URI = " \
> npm://registry.npmjs.org/;package=google-protobuf;version=${PV} \
> npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
> "
>
> S = "${WORKDIR}/npm"
>
> inherit npm
>
> == error ==
>
> ERROR: google-protobuf-3.3.0-r0 do_fetch: Failure expanding variable
> TUNE_FEATURES_tune-core2-64, expression was ${TUNE_FEATURES_tune-x86-64}
> core2 which triggered exception RecursionError: maximum recursion depth
> exceeded while calling a Python object
> ERROR: Logfile of failure stored in:
> /home/edson/ng-trunk/nodegrid/tmp/work/core2-64-poky-linux/google-protobuf/3.3.0-r0/temp/log.do_fetch.33980
> Log data follows:
> | DEBUG: Executing python function extend_recipe_sysroot
> | NOTE: Direct dependencies are
> ['virtual:native:/home/edson/ng-trunk/meta-extended/meta-oe/recipes-oe/node/nodejs_12.14.1.bb:do_populate_sysroot']
> | NOTE: Installed into sysroot: ['nodejs-native', 'pkgconfig-native',
> 'zlib-native', 'brotli-native', 'openssl-native', 'xz-native',
> 'quilt-native', 'python3-native', 'icu-native', 'c-ares-native',
> 'libuv-native', 'gnu-config-native', 'libtool-native', 'automake-native',
> 'autoconf-native', 'ninja-native', 'cmake-native', 'gettext-minimal-native',
> 'readline-native', 'libffi-native', 'bzip2-native', 'sqlite3-native',
> 'libtirpc-native', 'util-linux-native', 'gdbm-native', 'libnsl2-native',
> 'unzip-native', 'texinfo-dummy-native', 'm4-native', 're2c-native',
> 'expat-native', 'ncurses-native', 'curl-native', 'libpcre2-native',
> 'libcap-ng-native']
> | NOTE: Skipping as already exists in sysroot: []
> | DEBUG: sed -e
> 's:^[^/]*/:/home/edson/ng-trunk/nodegrid/tmp/work/core2-64-poky-linux/google-protobuf/3.3.0-r0/recipe-sysroot-native/:g'
>
> /home/edson/ng-trunk/nodegrid/tmp/sysroots-components/x86_64/nodejs-native/fixmepath
>
> /home/edson/ng-trunk/nodegrid/tmp/sysroots-components/x86_64/pkgconfig-native/fixmepath
>
> /home/edson/ng-trunk/nodegrid/tmp/sysroots-components/x86_64/openssl-native/fixmepath
>
> /home/edson/ng-trunk/nodegrid/tmp/sysroots-components/x86_64/quilt-native/fixmepath
>
> /home/edson/ng-trunk/nodegrid/tmp/sysroots-components/x86_64/python3-native/fixmepath
>
> /home/edson/ng-trunk/nodegrid/tmp/sysroots-components/x86_64/icu-native/fixmepath
>
> /home/edson/ng-trunk/nodegrid/tmp/sysroots-components/x86_64/gnu-config-native/fixmepath
>
> /home/edson/ng-trunk/nodegrid/tmp/sysroots-components/x86_64/libtool-native/fixmepath
>
> /home/edson/ng-trunk/nodegrid/tmp/sysroots-components/x86_64/automake-native/fixmepath
>
> /home/edson/ng-trunk/nodegrid/tmp/sysroots-components/x86_64/autoconf-native/fixmepath
>
> /home/edson/ng-trunk/nodegrid/tmp/sysroots-components/x86_64/ncurses-native/fixmepath
>
> /home/edson/ng-trunk/nodegrid/tmp/sysroots-components/x86_64/curl-native/fixmepath
>
> /home/edson/ng-trunk/nodegrid/tmp/sysroots-components/x86_64/libpcre2-native/fixmepath
> | xargs sed -i -e
> 's:FIXMESTAGINGDIRTARGET:/home/edson/ng-trunk/nodegrid/tmp/work/core2-64-poky-linux/google-protobuf/3.3.0-r0/recipe-sysroot:g;
>
> s:FIXMESTAGINGDIRHOST:/home/edson/ng-trunk/nodegrid/tmp/work/core2-64-poky-linux/google-protobuf/3.3.0-r0/recipe-sysroot-native:g'
> -e
> 's:FIXME_PSEUDO_SYSROOT:/home/edson/ng-trunk/nodegrid/tmp/sysroots-components/x86_64/pseudo-native:g'
> -e 's:FIXME_HOSTTOOLS_DIR:/home/edson/ng-trunk/nodegrid/tmp/hosttools:g' -e
> 's:FIXME_PKGDATA_DIR:/home/edson/ng-trunk/nodegrid/tmp/pkgdata/genericx86-64:g'
> -e
> 's:FIXME_PSEUDO_LOCALSTATEDIR:/home/edson/ng-trunk/nodegrid/tmp/work/core2-64-poky-linux/google-protobuf/3.3.0-r0/pseudo/:g'
> -e
>
Re: [yocto] What are the key factors for yocto build speed?
Hi, In my company we have tested some "big Ryzen" configurations to speed up Yocto builds. The conclusion of these tests is that the build time is almost only related to the CPU score: https://www.cpubenchmark.net/high_end_cpus.html The speed (overclock) and size of the RAM does not influence the build time, neither the the use of a SATA SSD or a NVME. For example one of our build servers is using: - AMD Ryzen 9 3900X - ASUS PRIME X570-P - 32Go DDR4 3200 MHZ CL14 - 500Go SSD It is a really good price / build time ratio configuration. Best regards, Jean-Marie On Mar 18, 2020, at 2:29 PM, Paul Barker pbar...@konsulko.com wrote: > On Wed, 18 Mar 2020 at 13:01, Mikko Rapeli wrote: >> >> On Wed, Mar 18, 2020 at 05:52:37AM -0700, Oliver Westermann wrote: >> > Hey, >> > >> > We're currently using a VM on Windows and it's a lot slower than the native >> > linux build (which is expected). >> > We're looking into getting a dedicated build server for our team >> > (basically a >> > self-build tower PC). Any suggestions what to put in that build to get the >> > most >> > out of it? >> > >> > Currently we're looking at a big Ryzen, 64G of RAM and one or multiple >> > SSDs on a >> > "consumer grade" board like the X570. >> >> Drop all virtualization and go for Linux on bare metal. Then make sure there >> is enough(tm) physical RAM for each CPU thread. For a "big Ryzen" the 64G of >> RAM >> sounds >> too little. I'd go higher there, but it all depends what kind of project >> is being compiled. >> >> I would also setup CPU, RAM, disk IO and network IO monitoring on the build >> machines >> and review and monitor the build times and results when the project evolves. >> There are >> times when most CPUs will be idling and there will be times when IO to disk >> is >> happening even when RAM is available. Linux kernel can be tuned to avoid >> disk IO >> if >> RAM is still available for example. > > What Mikko said is excellent advice. > > I'd recommend NVMe drives if you can. My build machine has two large > M.2 NVMe drives, one is used for the working directory and the other > for sstate, downloads and source checkouts to make the best use of the > available bandwidth. I find that XFS is a better fit than ext4 when > you've got fast drives and highly parallel I/O workloads. > > Make sure you spread your RAM across the different memory channels on > your motherboard as this increases memory bandwidth - this usually > means either filling your RAM slots or half-filling them with RAM in > every 2nd slot but you'll need to check the motherboard manual to > confirm the channel layout. > > Let me know if you need a detailed review of your proposed setup. > > Thanks, > Paul > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#48812): https://lists.yoctoproject.org/g/yocto/message/48812 Mute This Topic: https://lists.yoctoproject.org/mt/72047879/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
