[nodejs] [ANN] yapm: using package.yaml files instead of package.json

Wed, 26 Jun 2013 21:05:43 -0700

 
I'm not talking about how to develop dependencies right now. I'm talking about how to track relevant information about them.
 
When I see a critical bug in a 3rd party npm module, I fix it and send a PR. But PRs sometimes take months to get merged, so I change dependency version to a git fork. When PR gets merged, I return back to npm version. Therefore it's extremely important to preserve information about why was a particular fork was created. I could add this to readme, but that's what comments are for.
 
This is how our dependency list for one project looked a few months ago (right now most of these are resolved and current version doesn't look so impressive). I can't imagine how to write all this information and keep it in sync with actual package data without comments.
 
# main dependencies
dependencies: {
   # require express 3 api
   express: ">= 3.0.1"

   # we depend on this PR: https://github.com/jed/cookies/pull/18
   cookies: ">= 0.3.4"

   async: ">= 0.1.22"

   # date packages
   date-utils: ">= 1.2.12"
   strftime: ">= 0.4.7"

   # severe bug fixed by me, but not yet approved: https://github.com/TooTallNate/node-time/pull/22
   time: "git://github.com/rlidwka/node-time.git"

   # this package is outdated in NPM, trouble with startRecording
   tropo-webapi: ">= 1.0.16"

   # another critical bug here: https://github.com/stevegraham/twilio-js/pull/3
   twilio-js: "git://github.com/rlidwka/twilio-js.git"

   # TODO: unfixed bug: https://github.com/visionmedia/jade/issues/761
   # also waiting for this PR: https://github.com/visionmedia/jade/pull/629
   # the same PR rebased: https://github.com/visionmedia/jade/pull/812
   jade: "git://github.com/rlidwka/jade.git"

   nodemailer: ">= 0.3.31"
   mailparser: ">= 0.2.30"

   # we require "type: raw" for bunyan logs that appears in 0.10
   bunyan: ">= 0.10.2"

   # severe bug fixed, but not yet pushed to the repository: https://github.com/milewise/mongode/pull/6
   # also this one: https://github.com/milewise/mongode/pull/7
   mongode: "git://github.com/rlidwka/mongode.git"

   # bug: https://github.com/andris9/simplesmtp/pull/17
   simplesmtp: ">= 0.1.25"

   # it's my own assets manager, opensource and released as npm module
   asset-pipeline: ">= 0.2.0"

   # assets compilers
   less: ">= 1.3.1"
   ejs: ">= 0.8.3"

   # for development only
   hotswap: ">= 1.0.0"

   hashish: "*"

   # for API reference
   marked: "*"
   highlight: "*"

   # there's a lot of nexmo modules, but none really good; this one at least working
   node-nexmo-api: "*"
}
 
 
--
// alex
 
 
26.06.2013, 13:42, "Floby" <[email protected]>:
Why are you not using git submodules ? this seems like something you'd want to try out if most of your dependencies are git repos.

On Sunday, 23 June 2013 18:12:28 UTC+2, Alex Kocharin wrote:
Hi guys,

TL;DR: I developed a package that monkey-patches npm to work with package.yaml files without any json ever written to a disk. Hope it will be useful.

Published as npm module a month ago: https://npmjs.org/package/ynpm (github: https://github.com/rlidwka/ynpm ), and I think it's ready to use now.

So, 6 months ago I asked in the mailing list about how to place comments in package.json files ( https://groups.google.com/forum/?fromgroups#!topic/nodejs/NmL7jdeuw0M ) and found no good answer for that. So for a long time I used package.js and a Makefile to generate package.json. Rather ugly solution.

We were developing a large and complex project, and I submitted dozens pull requests to other packages. But until these PR were merged, I needed to point these dependencies to my git repositories with these changes in place. And THAT is why I desperately needed comments.

So anyway... these are fundamental issues with JSON:

1. JSON have no comments, you can't comment out why did you put some dependency, but not the other.
2. JSON have no trailing comma. So you can't easily remove an item, add an item or interchange two arbitrary lines in a list.
3. JSON require ugly enquoting both keys and values in object. _javascript_ require enquoting values only, and YAML doesn't require quotes in most cases.

JSON is designed to be written by computers, not humans. Humans could read it easily, but maintaining JSON is a pain.

Why YAML? Well, I would certainly not use XML. :) Anyway, YAML it's easier to read and edit than all other widely known serialization formats, and it solves all issues described above. I would fully support if Isaacs returns package.js back to work (see 3y old github issue https://github.com/isaacs/npm/issues/408 ), it would be good enough, but it doesn't seem to happen.

So, a lot of other people suggested to use YAML. That's the most recent github issue https://github.com/isaacs/npm/issues/3336 , but there were others.

And there are some existing solutions to do that, for example npm-yaml: https://npmjs.org/package/npm-yaml. But all these solutions just pre-compile package.yaml before npm is executed. It's not good enough because json remains written on the disk, and if npm modifies json, yaml remains unmodified.

So, I wrote a wrapper that replaces fs.* calls, so whenever npm reads package.json file, and if there is a yaml file, we compile and return yaml contents. If npm writes package.json, we change yaml instead.

I made sure that all npm functionality is working with this new approach. For example, in order to make `npm version` working I replaced execFile("git", ["add", "package.json"]) with an appropriate substitute. Other features required to replace readdir and file streams, and so on.

If something is wrong with this approach or if something breaks beyond fixing with future npm versions, I'll go for a fork. But as for now things seem to work out nicely.

PS: to guys who write YAML parsers: could you please write a module to change one particular node in YAML file without rewriting the entire file? Because it is the only thing here that ain't very good. I know it can't be done in a general case, but it doesn't mean we shouldn't try.

Regards,
alex

 

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
 
 




-- 

-- 

Job Board: http://jobs.nodejs.org/

Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines

You received this message because you are subscribed to the Google

Groups "nodejs" group.

To post to this group, send email to [email protected]

To unsubscribe from this group, send email to

[email protected]

For more options, visit this group at

http://groups.google.com/group/nodejs?hl=en?hl=en

 

--- 

You received this message because you are subscribed to the Google Groups "nodejs" group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

For more options, visit https://groups.google.com/groups/opt_out.

 

 

Reply via email to