Re: [OE-core] npm.bbclass support for deep native modules?
On Mon, 2016-11-28 at 04:35 -0600, Peter A. Bigot wrote: Hi Peter, I'm helping Paul with this. Sadly, I'm not much of an npm expert either, but want to improve support for node.js development. > > Sorry, that wasn't clear. statsd depends on modern-syslog but the > lockdown and shrinkwrap files generated by devtool don't include it. > From the Wiki: > > "Devtool cannot detect native libraries in module dependencies, you > you'll need to manually add packages to recipe" > > The Wiki doesn't go into detail of how that's supposed to be done. Is > the existing infrastructure supposed to be able to find > globally-installed modules? I believe that this refers to native code depending on another native library (e.g. libfoo). In this case, a package for libfoo would have to be available and added to DEPENDS. This is not applicable for modern- syslog. Paul - correct me if I've got this wrong. > I'm wondering whether https://yarnpkg.com/ or one of the other nodejs > dependency managers might be an alternative, as I believe npm's > approach to dependencies is not suited to level of lockdown needed by > Yocto and many other production systems. Good idea, but we need to balance the complexity of adding another tool vs. figuring this out with npm. Back to the problem. I have re-created the statsd build failure, and agree with your diagnosis, but don't have am immediate solution. I have opened a bug #10760, and added you to CC list. Let's use this bug to communicate from hereon. https://bugzilla.yoctoproject.org/show_bug.cgi?id=10760 Henry -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] npm.bbclass support for deep native modules?
On 28/11/16 10:35, Peter A. Bigot wrote: On 11/28/2016 04:11 AM, Paul Eggleton wrote: Hi Peter, On Sat, 26 Nov 2016 18:17:48 Peter A. Bigot wrote: I'm using the current head of morty and trying to get a handle on the new nodejs support in OE. I'm failing to build a recipe for statsd. Starting with this: devtool add 'npm://registry.npmjs.org;name=statsd;version=0.8.0' bitbake statsd produces an error related to the modern-syslog dependency: DEBUG: Executing shell function do_compile | npm ERR! Linux 4.4.0-47-generic | npm ERR! argv "/mnt/devel/oe/omap/build-bb-morty-master/tmp/sysroots/x86_64-linux/usr/bin/ node" "/mnt/devel/oe/omap/build-bb-morty-master/tmp/sysroots/x86_64-linux/usr/bin /npm" "--arch=arm" "--target_arch=arm" "--production" "--no-registry" "install" | npm ERR! node v4.6.1 | npm ERR! npm v2.15.9 | npm ERR! Registry not defined and registry files not found: "/mnt/devel/oe/omap/build-bb-morty-master/tmp/work/cortexa8hf-neon-poky-linu x-gnueabi/statsd/0.8.0-r0/npm_cache/noregistry/modern-syslog/.cache.json", "/mnt/devel/oe/omap/build-bb-morty-master/tmp/work/cortexa8hf-neon-poky-lin ux-gnueabi/statsd/0.8.0-r0/npm_cache/modern-syslog/.cache.json". modern-syslog 1.1.2 needs node-gyp to build a native component and https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM notes that devtool can't detect such things. Doing this works fine to build that package: devtool add 'npm://registry.npmjs.org;name=modern-syslog;version=1.1.2' bitbake modern-syslog but I'm having no luck getting "bitbake statsd" to find the result. I've added: DEPENDS = "modern-syslog" to statsd_0.8.0.bb but that isn't helping. It looks like I need some way to have the recipe install the prepared modern-syslog into the cache (or globally?) before baking statsd, but since the cache gets cleared in npm_do_compile() it's not clear how to make that happen. I'm very rusty with OE (two years away), so am I missing something or is this just beyond what the bitbake infrastructure can currently handle? If so, can somebody suggest a way to hand-patch the recipe, or outline how npm.bbclass might be extended to support this? Disclaimer - I'm the one who has been doing most of the recent work with npm support (aside from the node.js recipe and the original npm fetcher plugin, which were the work of others) however my knowledge of node.js is pretty limited - most of it has been picked up along the way. So unfortunately I can't immediately see why this isn't working. The thing that puzzles me in particular about the error you're seeing though is that we're explicitly telling npm not to look for a registry, so why is it complaining about the lack of a registry? Sorry, that wasn't clear. statsd depends on modern-syslog but the lockdown and shrinkwrap files generated by devtool don't include it. From the Wiki: "Devtool cannot detect native libraries in module dependencies, you you'll need to manually add packages to recipe" The Wiki doesn't go into detail of how that's supposed to be done. Is the existing infrastructure supposed to be able to find globally-installed modules? I'm wondering whether https://yarnpkg.com/ or one of the other nodejs dependency managers might be an alternative, as I believe npm's approach to dependencies is not suited to level of lockdown needed by Yocto and many other production systems. Peter Hi Peter, I'm in a similar boat packaging a custom project with a very large dependency tree. After looking at the available options and our current struggles with npm, yarn was our next point of call. We haven't done anything with it yet, but probably plan to in the near future. Not very helpful, but just a heads up that you're not the only one fighting npm ;) Cheers, Jack. -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] npm.bbclass support for deep native modules?
On 11/28/2016 04:11 AM, Paul Eggleton wrote: Hi Peter, On Sat, 26 Nov 2016 18:17:48 Peter A. Bigot wrote: I'm using the current head of morty and trying to get a handle on the new nodejs support in OE. I'm failing to build a recipe for statsd. Starting with this: devtool add 'npm://registry.npmjs.org;name=statsd;version=0.8.0' bitbake statsd produces an error related to the modern-syslog dependency: DEBUG: Executing shell function do_compile | npm ERR! Linux 4.4.0-47-generic | npm ERR! argv "/mnt/devel/oe/omap/build-bb-morty-master/tmp/sysroots/x86_64-linux/usr/bin/ node" "/mnt/devel/oe/omap/build-bb-morty-master/tmp/sysroots/x86_64-linux/usr/bin /npm" "--arch=arm" "--target_arch=arm" "--production" "--no-registry" "install" | npm ERR! node v4.6.1 | npm ERR! npm v2.15.9 | npm ERR! Registry not defined and registry files not found: "/mnt/devel/oe/omap/build-bb-morty-master/tmp/work/cortexa8hf-neon-poky-linu x-gnueabi/statsd/0.8.0-r0/npm_cache/noregistry/modern-syslog/.cache.json", "/mnt/devel/oe/omap/build-bb-morty-master/tmp/work/cortexa8hf-neon-poky-lin ux-gnueabi/statsd/0.8.0-r0/npm_cache/modern-syslog/.cache.json". modern-syslog 1.1.2 needs node-gyp to build a native component and https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM notes that devtool can't detect such things. Doing this works fine to build that package: devtool add 'npm://registry.npmjs.org;name=modern-syslog;version=1.1.2' bitbake modern-syslog but I'm having no luck getting "bitbake statsd" to find the result. I've added: DEPENDS = "modern-syslog" to statsd_0.8.0.bb but that isn't helping. It looks like I need some way to have the recipe install the prepared modern-syslog into the cache (or globally?) before baking statsd, but since the cache gets cleared in npm_do_compile() it's not clear how to make that happen. I'm very rusty with OE (two years away), so am I missing something or is this just beyond what the bitbake infrastructure can currently handle? If so, can somebody suggest a way to hand-patch the recipe, or outline how npm.bbclass might be extended to support this? Disclaimer - I'm the one who has been doing most of the recent work with npm support (aside from the node.js recipe and the original npm fetcher plugin, which were the work of others) however my knowledge of node.js is pretty limited - most of it has been picked up along the way. So unfortunately I can't immediately see why this isn't working. The thing that puzzles me in particular about the error you're seeing though is that we're explicitly telling npm not to look for a registry, so why is it complaining about the lack of a registry? Sorry, that wasn't clear. statsd depends on modern-syslog but the lockdown and shrinkwrap files generated by devtool don't include it. From the Wiki: "Devtool cannot detect native libraries in module dependencies, you you'll need to manually add packages to recipe" The Wiki doesn't go into detail of how that's supposed to be done. Is the existing infrastructure supposed to be able to find globally-installed modules? I'm wondering whether https://yarnpkg.com/ or one of the other nodejs dependency managers might be an alternative, as I believe npm's approach to dependencies is not suited to level of lockdown needed by Yocto and many other production systems. Peter -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] npm.bbclass support for deep native modules?
Hi Peter, On Sat, 26 Nov 2016 18:17:48 Peter A. Bigot wrote: > I'm using the current head of morty and trying to get a handle on the > new nodejs support in OE. > > I'm failing to build a recipe for statsd. Starting with this: > > devtool add 'npm://registry.npmjs.org;name=statsd;version=0.8.0' > bitbake statsd > > produces an error related to the modern-syslog dependency: > > DEBUG: Executing shell function do_compile > > | npm ERR! Linux 4.4.0-47-generic > | npm ERR! argv > > "/mnt/devel/oe/omap/build-bb-morty-master/tmp/sysroots/x86_64-linux/usr/bin/ > node" > "/mnt/devel/oe/omap/build-bb-morty-master/tmp/sysroots/x86_64-linux/usr/bin > /npm" "--arch=arm" "--target_arch=arm" "--production" "--no-registry" > "install" > | npm ERR! node v4.6.1 > | npm ERR! npm v2.15.9 > > | npm ERR! Registry not defined and registry files not found: > "/mnt/devel/oe/omap/build-bb-morty-master/tmp/work/cortexa8hf-neon-poky-linu > x-gnueabi/statsd/0.8.0-r0/npm_cache/noregistry/modern-syslog/.cache.json", > "/mnt/devel/oe/omap/build-bb-morty-master/tmp/work/cortexa8hf-neon-poky-lin > ux-gnueabi/statsd/0.8.0-r0/npm_cache/modern-syslog/.cache.json". > > modern-syslog 1.1.2 needs node-gyp to build a native component and > https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM notes that devtool > can't detect such things. Doing this works fine to build that package: > > devtool add 'npm://registry.npmjs.org;name=modern-syslog;version=1.1.2' > bitbake modern-syslog > > but I'm having no luck getting "bitbake statsd" to find the result. > I've added: > > DEPENDS = "modern-syslog" > > to statsd_0.8.0.bb but that isn't helping. It looks like I need some > way to have the recipe install the prepared modern-syslog into the cache > (or globally?) before baking statsd, but since the cache gets cleared in > npm_do_compile() it's not clear how to make that happen. > > I'm very rusty with OE (two years away), so am I missing something or is > this just beyond what the bitbake infrastructure can currently handle? > If so, can somebody suggest a way to hand-patch the recipe, or outline > how npm.bbclass might be extended to support this? Disclaimer - I'm the one who has been doing most of the recent work with npm support (aside from the node.js recipe and the original npm fetcher plugin, which were the work of others) however my knowledge of node.js is pretty limited - most of it has been picked up along the way. So unfortunately I can't immediately see why this isn't working. The thing that puzzles me in particular about the error you're seeing though is that we're explicitly telling npm not to look for a registry, so why is it complaining about the lack of a registry? Henry / Brendan - any suggestions? Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] npm.bbclass
On 17 August 2016 at 13:40, Paul Eggletonwrote: > Hey Chris, long time! > Hey :) > On Wed, 17 Aug 2016 11:11:13 Brendan Le Foll wrote: > > >- Is there a reason to split the package like it does? Node projects > > >tend to have huge dependency trees, it makes updating and > distributing > > >node-based applications a bit of a chore if they end up split into > 20 > > >packages, most of which have no use separately. It would be great if > > >there was at least a way to disable this. > > > > Paul added this, the worry was that we wanted to make sure all the > > licenses where tracked properly of the package etc... > > Right, that is why we do the splitting - if you want the licenses and > actual > packages used to be fully represented in the image license manifest then > they > really do have to be split this way. Honestly I view this ugliness more as > an > artifact of how npm works (i.e. every dependency is handled separately > instead > of trying to ensure only one instance of a particular package). > > > I'm not a huge fan either to be honest. Maybe we can have a npm-no- > > split.bbclass would that be ok - Paul? It's in python > > populate_packages_prepend in npm.bbclass. > > Sure, but rather than a separate class I would just add a variable such as > NPM_SPLIT_PACKAGES that defaults to "1" and then you can set it to "0" > either > globally or in each recipe. > This sounds good. A further bug I just encountered (after finally getting a particularly gnarly project packaged), the RDEPENDS that gets generated doesn't seem to be complete. Dependencies of dependencies don't get added it appears. So for example, you package appA that has a dependency on moduleX, which itself has a dependency on moduleY. It all gets packaged, but moduleY is missed from the RDEPENDS of appA (or of moduleX, which I guess is where it would be ideally). At least, I've included appA in an image, and appA is installed, moduleX is installed, but moduleY is missing. I've assumed this is the error, but I could be assuming incorrectly. > > >- Any patches end up getting packaged because they get put in the > > >srcdir. I'm guessing this isn't intentional (or maybe it is?) > > >Just wanted to provide some feedback. It's fantastic that OE has the > > >ability to package node software, and despite the teething > > >difficulties, I've appreciated its availability! > > > > That is a good point, didn't think about it tbh. in npm.bbclass we > > could maybe delete everything that looks like a patch before > > compilation, little bit worried there might be nasty side effects but > > I can try :) > > This is definitely not intentional, but the way we have been dealing with > npm > it makes it easy for stuff like this to leak into the output packages. I > guess > we just need to delete them under ${D} after the files are installed there. > On further thought about this, I actually don't think it's a terrible idea to do this anyway. Unless your patches are huge, it's not much overhead, and it's nice to include to hint at anyone inspecting the filesystem that the distributed files have been modified. It'd definitely be better to package them than to delete them and possibly end up deleting things that shouldn't be deleted. Maybe another thing that could be made optional? Cheers, --Chris -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] npm.bbclass
Hey Chris, long time! On Wed, 17 Aug 2016 11:11:13 Brendan Le Foll wrote: > >- Is there a reason to split the package like it does? Node projects > >tend to have huge dependency trees, it makes updating and distributing > >node-based applications a bit of a chore if they end up split into 20 > >packages, most of which have no use separately. It would be great if > >there was at least a way to disable this. > > Paul added this, the worry was that we wanted to make sure all the > licenses where tracked properly of the package etc... Right, that is why we do the splitting - if you want the licenses and actual packages used to be fully represented in the image license manifest then they really do have to be split this way. Honestly I view this ugliness more as an artifact of how npm works (i.e. every dependency is handled separately instead of trying to ensure only one instance of a particular package). > I'm not a huge fan either to be honest. Maybe we can have a npm-no- > split.bbclass would that be ok - Paul? It's in python > populate_packages_prepend in npm.bbclass. Sure, but rather than a separate class I would just add a variable such as NPM_SPLIT_PACKAGES that defaults to "1" and then you can set it to "0" either globally or in each recipe. > >- Any patches end up getting packaged because they get put in the > >srcdir. I'm guessing this isn't intentional (or maybe it is?) > >Just wanted to provide some feedback. It's fantastic that OE has the > >ability to package node software, and despite the teething > >difficulties, I've appreciated its availability! > > That is a good point, didn't think about it tbh. in npm.bbclass we > could maybe delete everything that looks like a patch before > compilation, little bit worried there might be nasty side effects but > I can try :) This is definitely not intentional, but the way we have been dealing with npm it makes it easy for stuff like this to leak into the output packages. I guess we just need to delete them under ${D} after the files are installed there. Cheers, Paul -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] npm.bbclass
On Tue, Aug 16, 2016 at 05:41:28PM +0100, Christopher Lord wrote: >I've been using npm.bbclass with various node projects we're working on >in the Connected Devices team at Mozilla, and had some >questions/comments about behaviour (note that I'm basing this off of >ostro master, which I unfortunately have to use as no other Intel >Edison bsp is adequate); I'm sorry :/ >- Is there a good reason the npm fetcher only works with a registry? It >seems it could get most (all?) pertinent information from a >package.json in the root of a repository. Not really a good reason, I'm trying to make it so that you can use it with a git fetched source but I haven't had the time to finish it off. It's on the todo list :) >- Is there a reason to split the package like it does? Node projects >tend to have huge dependency trees, it makes updating and distributing >node-based applications a bit of a chore if they end up split into 20 >packages, most of which have no use separately. It would be great if >there was at least a way to disable this. Paul added this, the worry was that we wanted to make sure all the licenses where tracked properly of the package etc... I'm not a huge fan either to be honest. Maybe we can have a npm-no-split.bbclass would that be ok - Paul? It's in python populate_packages_prepend in npm.bbclass. >- The information about packaging non-registry software here: >[1]https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM is incomplete - >you also need to generate a shrinkwrap and install that into the >srcdir, or you're very likely to get failures doing dependency >resolution. Not an OE issue, but I guess Yocto folks also read this >list and it's related :) Agree 100%, that is still WIP, Henry can you make sure you add a how to use the npm lockdown file too in the tutorial? Note - If you use recipetool It does this by default :). And recipetool with node.js pkgs is quite cool imho. >- Any patches end up getting packaged because they get put in the >srcdir. I'm guessing this isn't intentional (or maybe it is?) >Just wanted to provide some feedback. It's fantastic that OE has the >ability to package node software, and despite the teething >difficulties, I've appreciated its availability! That is a good point, didn't think about it tbh. in npm.bbclass we could maybe delete everything that looks like a patch before compilation, little bit worried there might be nasty side effects but I can try :) Thanks for the comments! Brendan -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] npm.bbclass
On 16 August 2016 at 17:41, Christopher Lordwrote: > - Any patches end up getting packaged because they get put in the srcdir. > I'm guessing this isn't intentional (or maybe it is?) > This isn't intentional - I've love to move the patches out of ${S} but that's non-trivial - so npm.bbclass should be skipping that directory. Ross -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core