On 07/21, Daniel Lobato Garcia wrote: > Hi devs, > > Just wanted to give you an update on packaging for npm & webpack > incidentally. We have two PRs in core that are dabbling in a more modern > way of doing frontend development in Foreman. > https://github.com/theforeman/foreman/pull/3433 > https://github.com/theforeman/foreman/pull/3603 > > At the moment, most frontend modules are no longer served as gemified > assets but rather as npm modules. It's difficult for us to develop on a > modern stack of tools & get a nice UX experience so the idea is to move > towards a model where npm will provide all the frontend stuff you need. > Read more on: https://github.com/theforeman/rfcs/pull/3 > > -------------------------- > > Now, regarding packaging (which really has been blocking the two > aforementioned PRs and potential frontend development): > > The issue we face is that we need to serve the node_modules (folder > generated after running npm install) dependencies with Foreman somehow. > I've been trying several things, with different degrees of success. > > 1. Use npm2rpm [1] to create a package for every npm module. Only > 'direct' dependencies are specified in these packages, and they store > merely the dependency itself. > > - Upsides: > - 'the right way' according to rpm guidelines > - no confusion about dependencies and simpler RPM specs > - Downsides: > - this creates close to 1000 new packages > - we'd have to serve several different versions of the same package, > which the OS would refuse to install (e.g if we have uglify-js 0.0.1, > and > 1.0.4 in our repos, the former would be superseded by the latter) > - impossible to review this amount of packages, it must be automated > > - Success with POC? No > > 2. Create 'nodejs-bundle-xxxxx' packages which contain all of the > dependencies for a certain package. e.g webpack Provides > bundle-webpack-core, etc... > > - Upsides: > - similar to how foreman node_modules is structured in development > - Downsides: > - conflicting versions for subdependencies force us to keep track > of the 'dependency tree' in the package. For instance in this spec > > https://github.com/dLobatog/foreman-packaging/blob/5fc7bb8e831fa74f579a41e9bf9f367f6a9527ff/nodejs-webpack/nodejs-webpack.spec > we can't just unpack all 'async' to > %{buildroot}/%{nodejs_sitelib} as there are several 'async' > packages, and the last one to be unpacked will prevail. > - these packages become huge and unwieldy to review > - we have to jump hoops to make it work. for instance we have to > call '%nodejs_fixdep -r dependency' to allow dependencies to not > conflict (this doesn't always work) > - lots of 'cruft' is packed, like examples directories, etc.. > > - Success with POC? No. I stopped when I figured I had to keep track > of the dependency tree before unpackaging.
Update - we can avoid keeping track of the dependency tree by storing it in the npm cache, then npm is able to install the module offline. This also removes the 'jump hoops' & 'packages are unwieldy to review' disadvantages. Props to Dominic Cleal on this as he came up with this idea. Check how that's done here: https://github.com/domcleal/foreman-packaging/commit/149b21195a27eafe8dc36e55454ca1c31c6a2355#diff-2fbc5d49e1f68f9a391153fadade5155R504 npm2rpm nightly is able to produce specs that do this (& download sources automatically) - https://github.com/dlobatog/npm2rpm I have tried to build Foreman itself using these packages, but I also have to create packages for the devDependencies that are webpack plugins before it can run 'webpack:compile'. It looks to me like it'll build just fine after that. Notice we don't require any changes in the pkg:generate_source task that generates the .tar file ------------------ At this point I have working POCs for 3 and 4, however since we're so close to being done with 2, and the person with the most packaging experience here backs it up, I think we can go with this (and it won't delay us). Right now all that's left is: - Rebasing the current nodejs-xxx PRs - Submitting new nodejs-xxx PRs for the webpack plugins (they're automatically created using npm2rpm, nbd) - Merging the Webpack PR & a PR to the foreman.spec that sets the right BuildRequires and runs webpack:compile. - Last but not least: Celebrate! That all could be done in a matter of a few hours - if I can catch some time I'll do all of it (except actually merging the PRs) this afternoon. ----------------- Thanks to all that have commented here for your help & suggestions! > > 3. Put node_modules in a new package (foreman-npm-assets?) that > foreman-assets %install section puts in the right place? > > - Upsides: > - relatively easy to do > - jenkins can automate it > > - Downsides: > - less control or information for users about what's installed with > foreman, although we could specify the dependencies under > "Provides: bundled-npm(xxxx)" > > - Success with POC: haven't tried > > 4. Make Foreman tar file include node_modules so it's part of the > foreman package itself > - Upsides: > - easiest to do > - zero work required by devs > - closest to development > > - Downsides: > - lack of knowledge/control about deps for users, like 3. > - no possibility of depending on system nodejs-xxx packages, but I > don't think we want that. > > - Success with POC: Yes! > https://github.com/dlobatog/foreman-packaging/tree/chainbuild-webpack > is able to run webpack compile just fine after that. We'd need even > less than what's on that branch to do it. > > ---------------------------------------- > > I got kind of stuck with 2. because of the tree dependencies thing, and > honestly I don't see much of a benefit to it now, especially considering > the amount of work there's involved. > > If we can redo this somehow I'd definitely go with 4 which is really > easy and I can't see how that's a problem for us. It'd be a problem if > we were to put Foreman in a Linux distribution but we're not aiming for > that. It is as well more manageable than any other model which requires > hundreds of specs + sources in our foreman-packaging repo. > > Aside from that... > > Why don't we go with 4 at least until we find a better/feasible model? It > would allow development to continue (the two PRs I mentioned + more). > There's plenty of time until 1.13 so I don't see why we can't move > forward, merge these PRs and figure out other packaging strategies if we > need to. > > --------------------------------------- > > If you got this far, you surely know if I'm missing something or you can > make some suggestions about which would be the best process, so please > reply. > > New ideas, comments, critiques, welcome :) > > [1] - https://github.com/dlobatog/npm2rpm > > Best, > > -- > Daniel Lobato Garcia > > @dLobatog > blog.daniellobato.me > daniellobato.me > > GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30 > Keybase: https://keybase.io/elobato -- Daniel Lobato Garcia @dLobatog blog.daniellobato.me daniellobato.me GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30 Keybase: https://keybase.io/elobato -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
signature.asc
Description: PGP signature
