Request for feedback on spec/proposal for distributing package collections via hackage
Hi folks, I'd like to get feedback on a spec/proposal for distributing package collections via hackage. This is currently somewhere beyond vapourware but certainly not a fait accompli and hopefully it is at an appropriate point to get feedback. The basic idea is that package collections are: * useful (IMHO, one of the top two solutions to dependency hell, alongside nix-style package management); and * just as we distribute packages via hackage, we should also be able to easily distribute package collections. One would then use them with tools like cabal and stack. Distributing via hackage (both in the sense of the format/protocol and in the sense of the central community hackage instance) seems natural, and allows taking advantage of much of the infrastructure we have for packages already like: * existing user accounts and management infrastructure on the hackage website * allowing anyone to host collections on their own servers, just as they can host their own package archives currently (either as static file sets or with smart servers) * low barrier for distribution, potentially encouraging more collections to be created potentially covering more use cases * security infrastructure (currently in alpha) * automatic mirroring (currently in alpha) Two obvious examples are stackage-lts and stackage-nightly but if we lower the barrier for distribution then there may well be many more. For example, the existing Linux distros put a lot of effort into selecting and maintain package collections, and some of these collections could be distributed via hackage. In fast several Linux distributions already use Hackage's distro feature to advertise which versions of packages are provided by that distro. One can also imagine special-purpose collections, and there's probably cases we've not thought of yet. Package collections are different things from packages, not like meta packages that one gets in some package systems. A package collection at it's simplest is just a set of source package identifiers (ie names-version pairs). Like packages, package collections have names and versions and are immutable once distributed. The intention is that users can configure their tool to use collection(s), either by nailing down a specific collection version, or by not specifying a version it would default to the latest version of the named collection. (But the specific behaviour is up to the tool) Use cases: * versioned collections. For some collections the policy by which it's defined naturally uses meaningful versions. * daily collections. These can have a date-form version number imposed on them. * live rolling collections. These could have a simple monotonic increasing version with no particular meaning attached. For such collections, clients might be configured to use the latest (by not specifying a version), but it's always possible to pick a specific revision. * special-purpose collections. Not necessarily collections aiming to cover a large number of common packages, but aiming to cover some application area, or related stack of packages (e.g. some of the web frameworks). * negative collections. Collections of packages you may specifically want to avoid (e.g. deprecated by their authors, or known-broken). Using such collections would rely on clients that can be configured to treat it negatively. Specifics: A package collection specifies a set of source package ids (id being name-version pair). It also optionally specifies a (partial) flag assignment for any package name. The collection does not specify how tools should treat them. That is, a collection does not specify if it should be treated as a strong or a soft constraint, inclusive or exclusive, positive or negative. Such things are completely up to the client's policy and configuration. Similarly for flag assignments, collections do not specify whether tools should interpret these as strong or soft constraints. Syntax: Package collection names and versions exactly follow those of package names (but they live in a different namespace). For example, stackage-lts-2.9, or deprecated-343 (the latter being a rolling collection with a meaningless monotonically increasing version). A collection distributed in the archive format is just a text file with one entry per line, such as: foo-1.0 foo-1.1 bar = 3 4 bar +this -that So each line can be one of: * a simple package id * a package version range, using Cabal version range syntax * a package name with a flag assignment, + for on, - for off The interpretation of the above is that: * both foo-1.0 and foo-1.1 are in the collection (ie union not intersection) * all versions of bar between 3 and 4 are in the collection * the
Re: Request for feedback on spec/proposal for distributing package collections via hackage
On Tue, 2015-07-14 at 13:52 +0100, Duncan Coutts wrote: Syntax: Package collection names and versions exactly follow those of package names (but they live in a different namespace). For example, stackage-lts-2.9, or deprecated-343 (the latter being a rolling collection with a meaningless monotonically increasing version). A collection distributed in the archive format is just a text file with one entry per line, such as: foo-1.0 foo-1.1 bar = 3 4 bar +this -that So each line can be one of: * a simple package id * a package version range, using Cabal version range syntax * a package name with a flag assignment, + for on, - for off Oops, one thing I forgot to mention is another entry syntax: baz That is, a package name with no version or range at all. This is shorthand for the version range style with no version constraint (.cabal files have a slightly odd syntax for that, baz -any). This is actually useful if you want to define a negative collection, e.g. all the packages that are deprecated (as a whole, not just single versions). Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/cabal-devel
Re: Request for feedback on spec/proposal for distributing package collections via hackage
On Tue, 2015-07-14 at 12:02 -0700, Edward Z. Yang wrote: Hello Duncan, In my eyes, this proposal looks like some sort of generalization of Stackage; and one further use case is special purpose collection. My big question: how composable are these collections really? I can't put two collections with conflicting versions together (or can I? Do I union?); You're right that some use cases of collections only make sense if clients can reasonably flexibly combine collections, e.g. set ops like union, intersection and inversion or difference. I think in principle taking collection unions or intersections makes sense. For example, in stack, as I understand it, you can add a local version of something that is also in the collection. This is of course a union. So extending narrow collections by unioning them with extra stuff makes sense. And suppose we had other wide collections that didn't nail things down to just one version, then taking an intersection with some other wide or narrow collection makes sense. So in principle, these set-like operations make sense. The code we've got in-progress for cabal-install allows exactly that. But none of that is essential to make use of the general purpose collections like stackage. and is there any point to having a collection without versions in it? To be clear, every collection instance has a version. It's just that for some of them -- live or rolling collections -- there is no particular meaning to the version. So this thing about not specifying a version is completely client side and doesn't affect the spec at all. It's just worth pointing out as a use case. As an example, we might have collections that are automagically defined and updated based on some property of the packages. Those are unlikely to have a meaningful version number. (If Cabal syntax is extended to support depending on collections as well as packages, yes?) I don't think you're confused here, but just to clarify: packages do not talk about collections. Tools like cabal/stack can be configured to use collections. And yes, where currently to configure cabal-install to use a collection you have to explicitly list every member as a constraint in a cabal.config file (see the cabal.config files distributed from the stackage website), the extension in cabal-install is to support these collections as a first class thing, by name-version or just name (and with an expression language to combine them). The classic use-case for package collections is deployment settings, ala Stack, or even Cargo lockfiles / Bundler Gemfile.lock (versioned collections). In all these use-cases package collections are treated as non-compositional things. http://doc.crates.io/guide.html http://bundler.io/v1.7/rationale.html#checking-your-code-into-version-control Libraries (compositional) do NOT publish lockfiles: only executables (non-compositional) DO. Re the file format, it seems fine; suitable for the lockfile use-case and the Stackage use-case. Less sure about the unioning semantics. Right, collections and frozen settings are similar, and for the latter composition doesn't make a lot of sense. We have the latter already of course, in the form of cabal freeze cabal.config files, and similarly for stack with .yml config files (which can be based off of pre-defined collections). It's likely that cabal freeze will switch to use this collection notation as it's somewhat more intentional (than the big sets of raw constraints). Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/cabal-devel
Hackage security alpha
Hi folks, We're doing an alpha release of the hackage security work today and we'd like to invite you all to help test it. In addition to the security improvements it includes automatic use of mirrors (including the server distributing a list of available public mirrors) and includes incremental downloads of the hackage index, so cabal update should be a lot faster. At this alpha stage we would like some but not too many users to try it out, so when things do break we don't have it break for too many people all at once. But subscribers to this list are just the kind of expert users who we'd like to try it out and report issues. In particular we're interested in any problems caused by crazy proxies and annoying things of that ilk. During the beta we'll make the whole thing a bit more user friendly to get more people to try it out. So for the moment you have to grab things from git branches etc. All the details are in this blog post: http://www.well-typed.com/blog/2015/07/hackage-security-alpha/ As it says there, report issues in the github bug tracker. Oh and I don't think we say it in the blog post but the idea is that for any of the new library dependences for the security stuff, if any of them are problematic we can just bundle them with cabal-install (we'll probably just bundle them all). The design deliberately keeps these dependencies to a minimum: SHA256 hashing, ed25519 signing/checking provided by minimal bundled C code. For the alpha the cabal-install integration just uses these as external dependencies. -- Duncan Coutts, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/ ___ cabal-devel mailing list cabal-devel@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/cabal-devel
Re: Hackage security alpha
On Wed, 2015-07-08 at 15:10 +0100, Andres Löh wrote: Hi Roman. On Wed, Jul 08, 2015 at 04:37:44PM +0300, Roman Cheplyaka wrote: Where exactly should I be looking for /snapshot.json? % curl -D - https://hackage.haskell.org/snapshot.json They're provided on the two mirrors, but not on the main Hackage server (yet): https://hackage.haskell.org/security-alpha/mirror1/snapshot.json https://hackage.haskell.org/security-alpha/mirror2/snapshot.json Right, for the alpha we're just providing these two mirrors which are just static file sets (created by the hackage-security tool for managing file based repos). The plan is that for the beta we'll deploy the server side code on hackage.h.o meaning that the main smart server will provide all the required security metadata. -- Duncan Coutts, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/ ___ cabal-devel mailing list cabal-devel@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/cabal-devel
Re: Making cabal-install SSL capable
On Mon, 2015-04-27 at 23:55 -0400, Gershom B wrote: So there are many discussions over various hackage security schemes, and there are a variety of takes on the different elements of how we could make package distribution more secure. However, everyone seems to agree that it would be unambiguously better if the cabal install executable were able to communicate over ssl. Yeah. I looked at the previous discussion on this topic [1], and it seems that HsOpenSSL and tls were both considered. I don’t have any experience with how cross-platform compatible HsOpenSSL is (i.e. if it is sufficiently easy to use for both Windows and OS X that we can just encourage people to “cabal install cabal-install” and things will just work). I don’t know if anyone else can speak to this? Furthermore, of course, redistributing cabal-install binaries could potentially be more of a pain with links to external c libraries. I’m not quite sure how much an issue this would be. Meanwhile, tls is certainly cross-platform, but there is the question about how trustworthy it is, as it is not nearly as widely used and vetted as openssl. Right. I don't have any great suggestion there. This is what has prevented us getting anywhere in the past. Also, we have the option of simply shelling out to curl, wget, or the appropriate powershell command (on windows 7 or above you get those by default). I think that's quite a reasonable 90% solution. I was looking at implementing this the other day after talking about it with Michael. When looking at the code I noticed (as did Michael) that rather embarrassingly the current code is actually using basic auth, when it should have been using digest auth. I've sent a PR: https://github.com/haskell/cabal/pull/2563 So rather than rely on either HsOpenSSL or tls, we could also teach cabal to probe for one of the appropriate executables on first run, save that configuration, and warn if no such executable is available (allowing the user to fall back to http with warnings indefinitely). I don't even think it needs any saved configuration. Cabal is pretty good at probing for and running exes. I would like to pursue getting SSL into cabal by any of these three avenues. What do people feel about the relative tradeoffs of these options? Honestly, I lean towards simply using the tls package, because https is ultimately only going to be a complimentary aspect of our security architecture plans and not central to it. And a pure-haskell dependency is the most logical approach. If people find too much fault with that approach, I would be inclined to shell out as the next option, with HsOpenSSL as the last option only because I worry about too many “unknown unknowns” of the sort I listed above. But if others have more experience with these approaches, proposals are welcome! My suggestion is that in the short term we use an external curl binary if it happens to be available, and fallback to digest auth if not. If/when we are in a position to have dependencies on decent http(s) libraries then we should use those. Also, for those users without curl (or without an HTTPS-enabled version), the digest auth can be improved. We can use digest auth with the auth-int QoP, which includes replay protection and integrity protection of the message body (ie the .tar.gz). It's not as good as TLS, but digest auth (with auth or auth-int QoP) isn't actually known to be broken yet. (It's an oldish standard that relies on MD5, but has both server and client nonces to protect against chosen plaintext attacks). Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/cabal-devel
Re: Cabal and simultaneous installations of the same package
On Mon, 2015-03-23 at 08:45 +, Simon Peyton Jones wrote: Dear Cabal developers You'll probably have seen the thread about the Haskell Platform. Among other things, this point arose: | Another thing we should fix is the (now false) impression that HP gets in | the way of installing other packages and versions due to cabal hell. People mean different things by cabal hell, but the inability to simultaneously install multiple versions of the same package, compiled against different dependencies is certainly one of them, and I think it is the one that Yitzchak is referring to here. But some time now GHC has allowed multiple versions of the same package (compiled against different dependencies) to be installed simultaneously. That's technically true of existing ghc versions, though ghc-pkg does not directly allow registering multiple instances of the same version of a package. As of 7.10 we can actually use ghc-pkg to register multiple instances, using ghc-pkg register --enable-multi-instance Also since 7.10 we can have environment files that say what packages to expose. There can be a per-user default one, or per-user named ones (which can be used via the ghc command line or via env var), or local environments in a directory. While Cabal has always built things with consistent deps and solved the problem of which ByteString do I mean, the same has not been true for GHCi. With this new environment mechanism Cabal can create the environments and then when the user runs ghc/ghci then they get the set of packages previously set up by Cabal. Elsewhere in this thread you say: What you want is for the confusing behaviour to be true of GHCi too. Well that’s simple enough: ensure that the set of exposed packages (ie the ones you say ‘import M’ for), is consistent in the same way. The point is that I may need to install a bunch of packages to build a program. If I’m using Cabal, none of those newly installed packages need be exposed; I simply need them there so I can compile my program (using Cabal). But at the moment I can’t do that. Yes, that is exactly what these environment files are intended for. Myself and Edsko implemented these features (for the IHG) to enable future Cabal versions to take the multi instance approach fully. So all we need to do is to fix Cabal to allow it too, and thereby kill of a huge class of cabal-hell problems at one blow. With these features now implemented in GHC we're in a position to turn attention to Cabal to make use of them. But time has passed and it hasn't happened. Is this because I'm misunderstanding? Or because it is harder than I think? Or because there are much bigger problems? Or because there is insufficient effort available? Or what? There's a number of parts remaining to do. Unless I'm way off beam, this multiple installations of the same package thing has been a huge pain forever, and the solution is within our grasp. What's stopping us grasping it? Yes, we're closer than ever. I covered more of the details in this blog post: http://www.well-typed.com/blog/preview/how-we-might-abolish-cabal-hell-2/ So some of the remaining parts: Cabal needs to assign proper installed package ids, like nix does. These should be based on the hash of all inputs to a package build. In particular that means a hash of the content of the sources. This is easy for tarballs but a bit harder to do accurately for unpacked build trees. There are some new UI issues to deal with. The user interface will have to make this issue of multiple consistent environments be explicit in the user interface. We need to know what env we are in, what is in it, what other envs are available and be able to switch between them. Suppose that we enforce that each environment be fully consistent. Then when I cabal install Q and it cannot find a solution to install everything in the current environment plus the one extra package such that they all have consistent deps, then what should it do? Suppose that Q really could be installed on its own, but cannot be installed consistently with the other things in this env. Should it suggest that you make a new environment? There are some details to work out here so that we don't make a confusing UI. There's also an unresolved issue about when we try to reuse existing installed dependencies. One approach is to say that we make install plans without considering what is already available in the package store, and then only re-use existing ones if the installed package Ids happen to match up. The other approach is to say that the solver should actively try to reuse installed instances when it can. The latter is what we do now, to try and reduce the number of reinstalls. But when there are dozens of versions available this is harder: we need to know more information to know if it is safe to re-use an existing instance. (There are examples where it's clearly not safe to reuse packages.) Or a
Re: Cabal and simultaneous installations of the same package
On Mon, 2015-03-23 at 20:13 +, Simon Peyton Jones wrote: But I'm hazy about why sandboxes are needed at all. As I understand it, they were invented to solve the very problem that is now solved (if only Cabal could take advantage of it). Yes, the nix approach would subsume sandboxes. I cover this in this post http://www.well-typed.com/blog/preview/how-we-might-abolish-cabal-hell-2/ Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/cabal-devel
Re: Cabal and simultaneous installations of the same package
On Mon, 2015-03-23 at 22:10 +, Duncan Coutts wrote: On Mon, 2015-03-23 at 20:13 +, Simon Peyton Jones wrote: But I'm hazy about why sandboxes are needed at all. As I understand it, they were invented to solve the very problem that is now solved (if only Cabal could take advantage of it). Yes, the nix approach would subsume sandboxes. I cover this in this post http://www.well-typed.com/blog/preview/how-we-might-abolish-cabal-hell-2/ Oops, permanent link: http://www.well-typed.com/blog/2015/01/how-we-might-abolish-cabal-hell-part-2/ ___ cabal-devel mailing list cabal-devel@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/cabal-devel
Re: Broken image?
Cheers! Fixed. I've stashed copies of all the Cabal logo files in http://www.haskell.org/cabal/images/ (FYI the cabal website is a darcs repo) On Tue, 2014-04-08 at 13:19 -0700, Jonan Scheffler wrote: Who can fix this? On Apr 8, 2014 11:47 AM, cabal-devel-ow...@haskell.org wrote: You are not allowed to post to this mailing list, and your message has been automatically rejected. If you think that your messages are being rejected in error, contact the mailing list owner at cabal-devel-ow...@haskell.org. -- Forwarded message -- From: Jonan Scheffler jonanscheff...@gmail.com To: cabal-devel@haskell.org Cc: Date: Tue, 8 Apr 2014 11:47:02 -0700 Subject: Broken image? The Cabal logo image on http://www.haskell.org/cabal/ is broken. Is there someplace I could put in an issue/PR? Jonan -- Jonan Scheffler @1337807 http://twitter.com/1337807 jonanscheff...@gmail.com (503) 985-6626 ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Proposal: cabal-install: verify OpenPGP signatures
On Wed, 2014-04-30 at 03:15 +0400, Nikita Karetnikov wrote: Following up on the “cabal-install: Replacing HTTP with HTTPS” thread. I think we can do better. I want to make sure that people will notice if someone compromises the packages on hackage.haskell.org. Here’s a rough plan: 1. Patch ‘hackage-server’ to allow uploading of OpenPGP signatures. 2. Patch ‘cabal-install’ to use GPG for verification. (GPG trust levels could be useful here.) ‘cabal install’ should also support ‘--skip-verification’ or some such to avoid disaster during the adoption stage. In addition, ‘cabal update’ would fetch the list of fingerprints from Hackage and cache each revision. A warning would be raised if a fignerprint cannot be found in the cache. If a maintainer wants to use a new key, it must be signed with the previously used one. If a maintainer loses their private key, for instance, this should be resolved by the admins. For example, an admin (admins?) could sign the new key. After a while, a web of trust would be formed. The fingerprints of active maintainers would be well-known. I’ve been thinking about this for quite a while and don’t see other ways to achive the same level of trust while allowing arbitrary uploads. The proposal also doesn’t require much manual intervention. What do you think? I’m willing to work on this but want to make sure that my time won’t be wasted. Will you accept such a patch? I think optional GPG signatures is a good idea, and I think in principle we would accept the patch. However it does have to be opt-in only: both opt-in for authors signing, and opt-in for clients checking. The idea of GPG signing is complementary to another package signing idea that we have been considering (Mikhail's reference to Austin's plan). GPG signing can provide a higher level of security because it can be end to end. However it's much harder with GPG signing to provide broad coverage of packages and users. It's hard because not all users have GPG (e.g. Windows), not all authors have GPG or can be bothered to sign anything. The web of trust is tricky, especially if we want to minimise the effort on the part of end users. Because of these issues with GPG signing, we thought a better use of our time was to work on this complementary plan where we have the server sign all the packages and the client automatically check. This is less secure because it is not end-to-end, but we can cover every package and we should be able to do it on all platforms, and without any knowledge or action needed on the part of authors or users. However, as I've said, these two security measures are complementary, we can have both. I can imagine a situation in which all packages are signed by the server, but some important ones are also signed by the authors, giving us a higher level of assurance of authenticity and integrity for those packages. So yes, you're welcome to help with either GPG signing or this alternative scheme, whichever you'd prefer to hack on. If you go for GPG, here's some issues to consider: * Who is allowed to sign for each package? Do we place any restriction or is any sufficiently trusted user allowed to upload any package? (I know the server will try to prevent this, but I'm talking about crypto level assurance here, not just server permission checks). If we do try to restrict it then we need some equivalent of the maintainer groups so I can allow other certain people to upload my package. * How to require some packages to be signed while not requiring all packages to be signed. In particular, once we opt-in to signing a particular package, we want to make sure it's signed from then on, and not allow a non-signed version of the package (which would otherwise be an obvious attack). * How to distribute the web of trust among end users (as opposed to authors) to minimise effort. Good luck! Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Hackage 'changelog' display (was: Fwd: [commit: packages/stm] master: Refactor update `stm.cabal` to `cabal-version=1.10` (aa4774b))
On Fri, 2013-10-18 at 16:05 +0200, Herbert Valerio Riedel wrote: Hello Duncan, ...any recommendation on this one? For isntance, how do I best add a link from the description-field pointing to the changelog file? No I think the longer term solution is to treat the changelog properly in some machine-readable form so that hackage can show the changes in a reasonable way, e.g. most recent changes, expand to more detail. I don't think we really want to have the current situation where some packages have these huge descriptions including long changelogs and others have nothing. It's all rather inconsistent. We should probably have hackage show only the first ~20-30 lines of the description and let the user expand that if they want to see more, and do a similar thing for the changelog. Simon makes the point that he likes having the changelog in the .cabal file rather than a separate file because it's less mental effort to update, as you can see it at the same time as the version. So perhaps what we should do is add a changelog field to the .cabal file an allow that to be inline, or to refer to a file, e.g. changelog: or changlog-file:. Either way we can then do a bit of simple syntax checking so we know we can format it ok on hackage and in other tools. Duncan ---BeginMessage--- I'm not keen on losing the changelog from the description. The idea of having the changelog in the description is: - it is prominent on the Hackage page for the package - it's next to the version in the .cabal file, so you're more likely to update it when updating the version. I don't mind splitting it into two - maybe the first few entries in the description and the rest in a separate file, with a link from the description. Cheers, Simon On 14/10/2013 22:38, git-4Dsf34iY/nkouohngz6...@public.gmane.org wrote: Repository : ssh://git-4Dsf34iY/nkouohngz6...@public.gmane.org/stm On branch : master Link : http://git.haskell.org/packages/stm.git/commitdiff/aa4774b7d5f65cbf007803ff54995921a36fbaeb --- commit aa4774b7d5f65cbf007803ff54995921a36fbaeb Author: Herbert Valerio Riedel hvr-mXXj517/z...@public.gmane.org Date: Mon Oct 14 22:02:35 2013 +0200 Refactor update `stm.cabal` to `cabal-version=1.10` As the current code fails to build with GHC 6.10, I've updated the Cabal meta-data to declare support for GHC=6.12 and dropped support for base3. This also moves the changelog from the Cabal description field into a separate `changelog` file and adds a few entries. The code compiles warning-free for all tested GHC versions. Signed-off-by: Herbert Valerio Riedel hvr-mXXj517/z...@public.gmane.org --- aa4774b7d5f65cbf007803ff54995921a36fbaeb Control/Concurrent/STM/TVar.hs |5 -- Control/Sequential/STM.hs | 18 --- changelog | 38 +++ stm.cabal | 103 ++-- 4 files changed, 83 insertions(+), 81 deletions(-) diff --git a/Control/Concurrent/STM/TVar.hs b/Control/Concurrent/STM/TVar.hs index d15896b..6435e17 100644 --- a/Control/Concurrent/STM/TVar.hs +++ b/Control/Concurrent/STM/TVar.hs @@ -43,11 +43,6 @@ import GHC.Weak import Control.Sequential.STM #endif -#if ! (MIN_VERSION_base(4,2,0)) -readTVarIO = atomically . readTVar -#endif - - -- Like 'modifyIORef' but for 'TVar'. -- | Mutate the contents of a 'TVar'. /N.B./, this version is -- non-strict. diff --git a/Control/Sequential/STM.hs b/Control/Sequential/STM.hs index 468e7bd..a644e67 100644 --- a/Control/Sequential/STM.hs +++ b/Control/Sequential/STM.hs @@ -40,35 +40,17 @@ instance Monad STM where x - m r unSTM (k x) r -#ifdef BASE4 atomically :: STM a - IO a atomically (STM m) = do r - newIORef (return ()) m r `onException` do rollback - readIORef r rollback -#else -atomically :: STM a - IO a -atomically (STM m) = do -r - newIORef (return ()) -m r `catch` \ ex - do - rollback - readIORef r - rollback - throw ex -#endif -#ifdef BASE4 throwSTM :: Exception e = e - STM a -#else -throwSTM :: Exception - STM a -#endif throwSTM = STM . const . throwIO -#ifdef BASE4 catchSTM :: Exception e = STM a - (e - STM a) - STM a -#else -catchSTM :: STM a - (Exception - STM a) - STM a -#endif catchSTM (STM m) h = STM $ \ r - do old_rollback - readIORef r writeIORef r (return ()) diff --git a/changelog b/changelog new file mode 100644 index 000..1296a41 --- /dev/null +++ b/changelog @@ -0,0 +1,38 @@ +-*-changelog-*- + +2.4.2.1 Oct 2013 + + * Updated behaviour of `newBroadcastTChanIO` to match + `newBroadcastTChan` in causing an error on a read from + the broadcast channel + + * Add `mkWeakTVar` + +
Re: hackage reverse dependencies
On Mon, 2013-10-14 at 12:18 -0400, Carter Schonwald wrote: So you're saying we need to add a db proper to hackage2 server, like SQLite or Postgres so as to make it more performant for interesting features? What's needed to do that? I don't think that's needed at all. What we need is to look carefully at what we want from a revdeps feature and then look carefully at what info should be stored (ie cached) and what should be calculated on demand. It just needs someone to spend some time on the feature, there's no need to panic about the data storage (especially before looking at what is actually needed). Duncan On Monday, October 14, 2013, Matthew Gruen wrote: Hey, thanks for taking an interest in this. There is kind of a space-time-featureset tradeoff with revdeps, unfortunately. It seems to me like the challenge doesn't come so much from indirect deps, but rather from the sheer number of versions there are! There are 5600+ packages and 33800+ versions, and the set of dependencies of foo-1.2 vs foo-1.3 can be completely different. Not to mention, when you look at the package page for a particular version, you may want to see the revdeps *for that version*, and at this point there are a billion possible combinations. npm's UI, in contrast, has an always in HEAD kind of feel. If you are interested in how the disabled implementation worked, by the way, I have a post here (as gracenotes): https://github.com/haskell/hackage-server/issues/40 So revdeps is great, and there are many different killer uses for it, such as popularity metrics and finding upper version bounds which need updating. In sum, these require keeping around a lot of data. Because there's so much, acid-state is probably not a good place to store it. Not to mention, it's time-consuming to generate it from scratch and space-consuming to keep around data structures to incrementally update it. So there are some tool limitations, but it seems fundamentally tricky to do frugally. Matt On Mon, Oct 14, 2013 at 12:20 AM, Jens Petersen j...@community.haskell.org wrote: Not really complaining :) but one of the things I had been looking forward to with Hackage2 was the display of reverse-dependencies, but I gather it was disabled for now (because it loads the server too much when updating the data iirc?). I was thinking perhaps a compromise would be to show only direct reverse-dependencies rather than including all indirect dependencies too. Perhaps that would make it scale better? I think direct dependency information is the most useful anyway (and also what packdeps provides). Before doing anything I thought I would ask here first if this makes sense and should be work? I for one would still really like to see basic reverse dependencies info listed on hackage (I noticed yesterday that npm has them too). Jens ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel -- Duncan Coutts, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/ ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Lag in new hackage mirroring of old hackage
On Thu, 2013-08-08 at 16:31 -0400, Gregory Weber wrote: Hello, I want to report some lag in the mirroring of old hackage by the new hackage. Note: we are mirroring packages from the old server (currently every 30min) so it is suitable to use as your main hackage server with some caveats: because we are allowing package authors to upload (as well as the mirroring) then you may find a slightly different set of packages on this server. -- http://new-hackage.haskell.org/ viewed on August 2, 2013 But for some elm packages, I just noticed today, the versions on new hackage are considerably more than 30 minutes older than those on old hackage: I realise this is a very late reply, but a little explanation is perhaps still interesting. Yes, the mirroring did fail and we didn't have any monitoring set up or auto restart so we didn't notice for a while. I think I have identified and fixed the cause of the failure (the mirror client was designed to be robust to mirroring problems but this one slipped through). On the new beta site the mirror client is running under an init script and the infrastructure team are going to sort out auto restart and monitoring for us. -- Duncan Coutts, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/ ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Hackage 2 trustees
On Thu, 2013-05-23 at 11:00 +0200, Erik Hesselink wrote: Hi all, I just upgraded our internal hackage to the latest HEAD. I found that the behavior of the 'trustees' group has changed. Previously, they could upload any (existing) package, but that doesn't work anymore. Ah yes. This functionality was very useful for us, since otherwise, we'd have to add all our coders as maintainers to all our packages. I've patched our local code, but my question is: * Is this a bug and should I push a patch? No, it is intended behaviour. For the public server the policy we want is that only maintainers can upload packages. Trustees can tinker with metadata but don't get to completely override maintainers. I see that there is a tension here and you want a different policy because all your local users are trusted. In the short term your patch is the right thing to do. In the longer term we may want to make this more configurable separately, rather than hard coded. * Is this intended and should I push a patch to the text at http://hackage.typlab.com/packages/trustees/, since it doesn't correspond to the current behavior? Yes please. Or better yet, switch that page use a template (like many of the new features use) so that it's easy to customise text like this. -- Duncan Coutts, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/ ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: How can I get SSH access to www.haskell.org to upload the Cabal user guide?
On Fri, 2013-09-06 at 18:36 -0400, Gershom Bazerman wrote: Sorry for not replying to the list. That info is outdated as of last month and we need to update (its now in a vm on rock.haskell.org, the new box) . I've already put Johan in touch with our admins who can set him up. We can also modify that wiki page to note that Johan is 'in charge' of /cabal? I'd be happy to have my haskell.org account restored. I used to do the updates for the /cabal url. Ideally both Johan and I can get access. That'd give us a little redundancy. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: hackage 2: alpha testing, testers wanted!
On Wed, 2013-04-10 at 23:58 +0400, Lennart Kolmodin wrote: 2013/4/5 Duncan Coutts dun...@well-typed.com On Thu, 2013-04-04 at 22:38 +0400, Lennart Kolmodin wrote: I tried to login with my username/pw, but failed. Here's what I did: I go to a package I've published and hit edit package information in the Maintainers' corner. It prompts me to login. If I hit cancel, it shows No authorization provided. If I type a random username/pw it keeps asking me. If I type my actual username/pw it asks me several times, then my webbrowser shows This webpage is not available (ie. not an error message from hackage). At one point it showed me a page where I could upgrade my account (and I haven't yet), but it does not show that page any more. I tried several times, it repeatedly shows me This webpage is not available. What browser are you using? I just tried again, Ta. and it worked. Now I also upgraded the account. I tried again to login to the upgraded account, it worked. Great. So that was after I changed it to use 403. I was using Chrome 26 when I tried to login a few days ago, now I'm using Chrome 27. I'll assume it was the changes I made rather than chrome changes. :-) Duncan I'm going to try changing the status we return in this situation from 401 to 403. That is, if you have an existing account but it has an old-style password that needs to be upgraded, then instead of returning a 401 response with a page linking to the upgrade page, we return a 403. In firefox the current responses work fine, but looking at the RFC it makes it clear that 401 isn't the right response in this case. So I'll try 403. ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: install --builddir
On Tue, 2013-04-09 at 08:46 -0400, Benjamin Scarlet wrote: I'm having trouble with install --builddir. If I run $ cabal install --builddir /some/absolute/path test-framework I get an error after regex-posix is installed, during the build of test-framework: Text/Regex/Posix/Wrap.hsc:141:8: Could not find module `Text.Regex.Base.RegexLike' It is a member of the hidden package `regex-base-0.93.2'. Perhaps you need to add `regex-base' to the build-depends in your .cabal file. Use -v to see a list of the files searched for. But if I run $ cabal install --builddir /some/absolute/path regex-posix $ rm -r /some/absolute/path $ cabal install --builddir /some/absolute/path test-framework I do not get the error. Between experiments I'm removing my ~/.cabal and ~/.ghc and running cabal update. Naively, it looks to me like using an absolute builddir has changed things so successive packages share the same intermediate files, which in turn has caused a later package to erroneously recompile a previously generated file. I'm sure that's exactly what is going on. It is using that directory to store the build artefacts and that is not supposed to be shared with any other package. My ultimate aim is only to be able to use cabal-dev for my development builds, but without writing into my source tree. Is there any way to specify a build directory outside the source of a package without it winding up shared? Could/should cabal instead create a separate subdirectory within the specified builddir for each package to be built, so that specifying an external location didn't break things? You could just call it with builddir set to a fresh directory (e.g. with a wrapper script like cabal-dev). If you want to set it statically and with a shared location then perhaps we could do something like allow vars in the builddir, like we allow vars in the install dirs. A patch to do that shouldn't be too difficult if you'd like to have a go. See the InstallDirs and path template stuff Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: hackage 2: alpha testing, testers wanted!
On Thu, 2013-04-04 at 22:38 +0400, Lennart Kolmodin wrote: I tried to login with my username/pw, but failed. Here's what I did: I go to a package I've published and hit edit package information in the Maintainers' corner. It prompts me to login. If I hit cancel, it shows No authorization provided. If I type a random username/pw it keeps asking me. If I type my actual username/pw it asks me several times, then my webbrowser shows This webpage is not available (ie. not an error message from hackage). At one point it showed me a page where I could upgrade my account (and I haven't yet), but it does not show that page any more. I tried several times, it repeatedly shows me This webpage is not available. What browser are you using? I'm going to try changing the status we return in this situation from 401 to 403. That is, if you have an existing account but it has an old-style password that needs to be upgraded, then instead of returning a 401 response with a page linking to the upgrade page, we return a 403. In firefox the current responses work fine, but looking at the RFC it makes it clear that 401 isn't the right response in this case. So I'll try 403. -- Duncan Coutts, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/ ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Defining CH executables in cabal files
On Thu, 2013-04-04 at 15:46 +0100, Rob Stewart wrote: Hi Duncan, Edsko. I'd like to know where I should most appropriately submit a bug report, or indeed whether the behaviour that I am experiencing is a bug or a user misuse of cabal. It's a consequence of a misunderstanding of how package names affect types and how different cabal constructs give you different package names. I have packaged up a very simple send/receive example into a cabalised archive. In there is a README that describes the problem succinctly, though I will repeat here for convenience. It appears to be weirdness with main-is and hs-source-dirs. In the .cabal file, there are three executables of the same Receiver.hs file - a receiver of a CH message. One appears to be defined in such a way that the CH primitives `match` and `expect` are *not* receiving the message types that are being sent by Sender.hs. The problem is that your three programs are using different types. ch-receiver-1 and ch-receiver-2 use the type cabal-with-ch-executables-1.0:Bar.Baz.Types.Msg while ch-receiver-3 uses the type Bar.Baz.Types.Msg Those are the fully-qualified names of those types and they are different, so they are different types, so have different Typeable fingerprints and so the `expect` does not match them. So as you can see, the fully qualified name of a type includes its package id, and in the first case it has the package name cabal-with-ch-executables-1.0 while in the second case it is the home package, which has no name (or if you like, it's called ). Code in a library has fully qualified names that include the package name. Code in a top level executable is in the home package. The way you've constructed your three programs, the first two put the module in a library and the third puts it in the top level executable, so according to ghc it really is a different module. So lets look at your cabal file: library default-language: Haskell2010 exposed-modules: Bar.Baz.Types , Bar.Baz.Identify build-depends: base = 4 5 , binary = 0.6.3.0 , bytestring = 0.9.2 , distributed-process = 0.4.2 , network , network-transport , network-transport-tcp hs-source-dirs: src GHC-Options: -Wall -- Works Executable ch-receiver-1 default-language: Haskell2010 main-is:Receiver.hs build-depends: base , distributed-process , network-transport-tcp , network-transport , cabal-with-ch-executables hs-source-dirs: examples ghc-options: -Wall -- Works Executable ch-receiver-2 default-language: Haskell2010 main-is:Receiver.hs build-depends: base , distributed-process , network-transport-tcp , network-transport , cabal-with-ch-executables hs-source-dirs: src/Bar/Baz ghc-options: -Wall -- Doesn't work Executable ch-receiver-3 default-language: Haskell2010 main-is: Bar/Baz/Receiver.hs build-depends: base , distributed-process , network-transport-tcp , network-transport , cabal-with-ch-executables -- I don't understand why these below are now needed?? , binary = 0.6.3.0 , bytestring = 0.9.2 , distributed-process = 0.4.2 , network , network-transport , network-transport-tcp hs-source-dirs: src ghc-options: -Wall So you have the first two executables depending on the library and they pick up their Bar.Baz.Types module from the library. Your third executable also depends on the library but the way you have the sources organised means that you're getting the modules from the source files and not from the library. This is because you're using the hs-source-dirs as src, which is where the modules for the library live. ghc will always pick local source files over modules from packages. This is also why you needed those extra dependencies, your executable is including those modules directly (you'll notice they're recompiled for the 3rd exe). The solution is that when you want to have an exe in a package depend on a lib in the same package, you need to separate the src dirs, as you did with the first one (hs-source-dirs: examples). Clear as mud? What would be good is if cabal would warn when the same module can be found both in a local lib that we depend on, and as a source file, because that's probably a mistake. It might also be good if we could tell ghc where to get the modules from explicitly, rather than rely on its behaviour about preferring src modules to package modules. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Code style guide?
On Tue, 2013-03-19 at 07:29 -0700, Johan Tibell wrote: On Tue, Mar 19, 2013 at 6:35 AM, Duncan Coutts duncan.cou...@googlemail.com wrote: Go for it! You'll also notice that the code uses different styles in different places (because of different authors). We've not gone through and tidied stuff up because we're lazy and because there never seems to be a good time to do it (it causes conflicts with patches people are working on). If you need to write a style guide, why not use https://github.com/tibbe/haskell-style-guide/blob/master/haskell-style.md as a starting point. It's widely used in the Haskell community nowadays (e.g. it's a common recommendation on IRC) and it's based on the coding style of Duncan, Bryan, and Don for the most part. BTW, the main thing in that guide that I disagree with is the 4-space indentation. I always use 2, but at the top level there's a slight special case: foo :: a - b foo x = bar x where bar y = ... We have to indent the body of the function here by 4 because the where has to be indented by 2, and the body has to be more than that, this is partly for clarity but also partly because of this problem: foo x = case x of This - That - where bar y = ... IIRC, historically in H98 this attached the where to the top level function foo, but these days with the modified layout rule, the where gets attached to the last branch of the case. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Advance notice that I'd like to make Cabal depend on parsec
On Thu, 2013-03-21 at 17:51 -0400, Isaac Dupree wrote: On 03/18/2013 12:55 PM, Duncan Coutts wrote: [...] it is not simply the outline parser for cabal-style files that we're talking about. We also need parsers/pretty printers for all the various little types that make up the info about packages, like versions, package names, package ids, version constraints, module names, licenses etc etc. (ignorant musing that doesn't help the general difficult of writing a Happy parser: ) Can they not use multiple Happy parsers generated from the same Happy file? http://www.haskell.org/happy/doc/html/sec-multiple-parsers.html Well the compositionality is there for the benefit of other packages, not just as an internal convenience for the Cabal lib. If we dropped that feature then yes we could use monolithic parsers for each of these types. Other packages do use the ability to build new parsers out of old however, in particular cabal-install does. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Code style guide?
On Tue, 2013-03-19 at 00:15 +, Ben Millwood wrote: Hi everyone, Do we have a general notion of what we consider best practices for contributed code to Cabal? Perhaps we should formalise something? One of the reasons I wanted to get the wiki back online was in the hope that it had some pointers, but it doesn't seem to. (I've started migrating stuff anyway: have a look at https://github.com/haskell/cabal/wiki/Migration-from-trac for status). For example, many of the Cabal modules have explicit, and therefore quite long, import lists, with no clear organisation, and hence when adding new imports I have to read the entire list to make sure they are not already there, and then wonder where makes sense to insert them. Usually I just make something up with no pattern to it, and hence perpetuate the problem. I am happy to start things off with a proposal if no-one else wants to do the writing. Go for it! You'll also notice that the code uses different styles in different places (because of different authors). We've not gone through and tidied stuff up because we're lazy and because there never seems to be a good time to do it (it causes conflicts with patches people are working on). Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Advance notice that I'd like to make Cabal depend on parsec
On Sun, 2013-03-17 at 21:04 +0100, Henning Thielemann wrote: On Sun, 17 Mar 2013, Ian Lynagh wrote: I think it would be feasible to stop GHC itself from using the human readable format. The only place I can think of it being used is in the package database, but we could use either Read/Show for that, or just exclusively use the binary format. I already needed the human readable format in order to check what information a custom configure file generated. Or more generally, the classic way to make the pkg info if you were not using the simple cabal build system, but were using configure + make (e.g. wrapped in the cabal make build-type) was to generate the input file using configure/m4 text substitutions. So that did/does need to be human readable. As for the binary format, that's ghc's internal representation and not something I think we would want to standardise between Haskell implementations. Note that other Haskell impls use a package database that just uses these human readable files, with no hc-pkg style program. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Advance notice that I'd like to make Cabal depend on parsec
On Sun, 2013-03-17 at 19:27 +, Ian Lynagh wrote: On Sun, Mar 17, 2013 at 09:57:25AM -0700, Mark Lentczner wrote: Is it essential, or even sensical, that the serialization format GHC needs for storing package info bear any relation to the human authored form? If not, the split out of the package types could be accomplished in a way where GHC uses simple show/read(P) style serialization for storage of package info, where as cabal-lib would use a lovely parsec parser for humans. I'd like this approach. I think it would be feasible to stop GHC itself from using the human readable format. The only place I can think of it being used is in the package database, but we could use either Read/Show for that, or just exclusively use the binary format. The change in functionality to enable that would be that the binary cache would always have to be up to date, so ghc would only ever have to read the cache and never have to read the human-readable package files. Then you can have ghc-pkg depend on Cabal and use that for the human-readable bits, but since that's a program then it doesn't expose the Cabal lib dependency. Then ghc (and hence the ghc lib) would not depend on Cabal, but it would need a copy of the InstalledPackageInfo type and the other types that it uses. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Advance notice that I'd like to make Cabal depend on parsec
On Mon, 2013-03-18 at 12:43 +, Ian Lynagh wrote: On Mon, Mar 18, 2013 at 12:34:16PM +, Duncan Coutts wrote: Then you can have ghc-pkg depend on Cabal and use that for the human-readable bits, but since that's a program then it doesn't expose the Cabal lib dependency. Then ghc (and hence the ghc lib) would not depend on Cabal, but it would need a copy of the InstalledPackageInfo type and the other types that it uses. Right, exactly. But we don't want to have 2 copies of the types, so could we move them into a Cabal-datatypes package which can be shared by both Cabal and GHC please? That would be a rather annoying split. The cabal-lib package itself is supposed to be just types + parsers + pretty printers ( related utils). It'd end up looking like: cabal-types: types: InstalledPackageInfo, PackageName, Version, PackageId, InstalledPackageId, License cabal-lib: parser for InstalledPackageInfo, PackageName, Version, PackageId, InstalledPackageId, License modules Distribution.* cabal-build-simple: modules Distribution.Simple.* It's not as if one could frame this as a the aspects of the Cabal spec that compilers need because the other impls will want the parser + printers as well. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Advance notice that I'd like to make Cabal depend on parsec
On Fri, 2013-03-15 at 10:10 +0100, Herbert Valerio Riedel wrote: Duncan Coutts duncan.cou...@googlemail.com writes: [...] I've been doing regression testing against hackage and I'm satisfied that the new parser matches close enough. I've uncovered all kinds of horrors with .cabal files in the wild relying on quirks of the old parser. I've made adjustments for most of them but I will be breaking a half dozen old packages (most of those don't actually build correctly because though their syntax errors are not picked up by the parser, they do cause failure eventually). btw, why not just keep the current parser as a legacy parser in the code, for older .cabal files (or as a fallback parser, in case the new stricter parsec-parser fails)? I'm satisfied at this point that the number of packages affected by the change is so low that it's not worth the extra maintenance. As I mentioned, most of the ones that break in the new parser are actually already broken in the sense that they will not build (because of mistakes in the .cabal file that just happen not to be caught by the old parser). So the amount of real breakage is trivial. Also, with the new hackage server we will be able to fix .cabal files post-release so if we do care about those few older packages we can actually fix them. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Advance notice that I'd like to make Cabal depend on parsec
On Fri, 2013-03-15 at 12:37 +0800, Conrad Parker wrote: On 14 March 2013 22:53, Duncan Coutts duncan.cou...@googlemail.com wrote: I've been doing regression testing against hackage and I'm satisfied that the new parser matches close enough. I've uncovered all kinds of horrors with .cabal files in the wild relying on quirks of the old parser. I've made adjustments for most of them but I will be breaking a half dozen old packages When you say you've made adjustments for dodgy .cabal files in the wild, do you mean that you'll send those maintainers patches that make their cabal files less dodgy, or do you mean you've added hacks to your parser to reproduce the quirky behaviour? The latter, but the egregiousness of the hacks is actually not too bad in the end. I don't find it revolting. For the worst examples I didn't make adjustments and those ones will break. I think I've made a reasonable judgement about the where to draw the line between the two. I can look into generating warnings in those cases (which is probably better than me emailing them). Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Advance notice that I'd like to make Cabal depend on parsec
On Fri, 2013-03-15 at 05:19 +0100, Bardur Arantsson wrote: On 03/14/2013 11:01 PM, Duncan Coutts wrote: On Thu, 2013-03-14 at 11:15 -0700, Jason Dagit wrote: On Thu, Mar 14, 2013 at 11:01 AM, Bardur Arantsson s...@scientician.netwrote: On 03/14/2013 03:53 PM, Duncan Coutts wrote: Hi folks, Why did I choose parsec? Practicality dictates that I can only use things in the core libraries, and the nearest thing we have to that is the parser lib that is in the HP. I tried to use happy but I could not construct a grammar/lexer combo to handle the layout (also, happy is not exactly known for its great error messages). Just thinking out loud here, but what about ditching the current format for something that's simpler to parse/generate? Like, say, JSON? Of course .cabal files are mainly written by humans, not machines, so we should optimise for them. I though we were mostly talking about InstalledPackageInfo. That could be in $EASILY_PARSEABLE_FORMAT without really breaking anything, right? In principle it could be any format. But it is a format specified in the Cabal spec, and shared between all the Haskell implementations. Unless there's a compelling reason to change all that, I'd rather not. Another option if GHC really also needs to parse .cabal files: That's ok, it doesn't. GHC use Cabal when building ghc, but at runtime it's just using the InstalledPackageInfo type, parser (and perhaps some index utils). -- Duncan Coutts, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/ ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: cabal install --only
On Fri, 2013-03-15 at 12:47 +0100, Mikhail Glushenkov wrote:
Hi,
On Fri, Mar 15, 2013 at 8:14 AM, Andres Löh and...@well-typed.com wrote:
No, please do not remove it. Running install without --only causes all sorts
of things to happen, because install is the magic command. The --only flag
is for use after you have already run build and really only does the
installation, i.e. It behaves comparable to Setup install.
Yes, I see from the code what it does, it's just that I think that the
option is obscure and very rarely used. It's not shown in the --help
output, though, so I guess it's not a big problem.
As other people have pointed out it's rather useful.
The reason it's there is because our UI is confused. It's a wart that
needs fixing not by removing it, but by removing the need for it.
When we had the old runhaskell Setup {configure,build,install}
interface then it was clear that the commands acted only on the local
package and only did one step of the process: you had to have done the
other steps first.
When we moved to cabal-install, we wanted automation and a do
everything command. So the install command will configure, build and
install. But there isn't a proper dependency mechanism and there isn't a
proper tracking of configuration state. So install always reconfigures,
loosing any configuration you did previously with cabal configure. So
that means if you do: cabal {configure --blah,build,install} then you
loose that --blah config. So because of that one does sometimes want a
--only. It's hidden because it's not official UI.
So what would a proper UI look like here? I think one should consider it
in the new context of sandboxes, or more generally the idea that in
future the cabal command line UI does not act on individual packages in
isolation but on packages in the context of an environment containing a
set of source packages.
So when I'm in an environment and I say configure on a package, that
should save semi-persistently the config settings I give (and do package
dependency analysis, if necessary it should (re)configure the other
source packages in the environment that this one depends on). The build
and install commands should work in a make-like fashion in that they
have to be able to discover/check which packages are up to date
(including if they are configured already, and if we already had an up
to date InstallPlan). So install --only is redundant because install
will do the right thing if you've already configured and built.
And that's of course the reason we still have --only, because we don't
have a proper make-like system for checking if packages are already
configured or built etc.
Duncan
___
cabal-devel mailing list
cabal-devel@haskell.org
http://www.haskell.org/mailman/listinfo/cabal-devel
Advance notice that I'd like to make Cabal depend on parsec
Hi folks, I want to give you advance notice that I would like to make Cabal depend on parsec. The implication is that GHC would therefore depend on parsec and thus it would become a core package, rather than just a HP package. So this would affect both GHC and the HP, though I hope not too much. The rationale is that Cabal needs to parse things, like .cabal files and currently we do not have a decent parser in the core libraries. By decent I mean one that can produce error messages with source locations and that doesn't have unpredictable memory use. The only parser in the core libraries at the moment is Text.ParserCombinators.ReadP from the base package and that fails my decent criteria on both counts. Its idea of an error message is (), and on some largish .cabal files we take 100s of MB to parse (I realise that the ReadP in the base package is a cutdown version so I don't mean to malign all ReadP-style libs out there). Partly due to the performance problem, the terrible .cabal file error messages, and partly because Doaitse Swierstra keeps asking me if .cabal files have a grammar, I've been writing a new .cabal parser. It uses an alex lexer and a parsec parser. It's fast and the error messages are pretty good. I have reverse engineered a grammar that closely matches the existing parser and .cabal files in the wild, though I'm not sure Doaitse will be satisfied with the approach I've taken to handling layout. Why did I choose parsec? Practicality dictates that I can only use things in the core libraries, and the nearest thing we have to that is the parser lib that is in the HP. I tried to use happy but I could not construct a grammar/lexer combo to handle the layout (also, happy is not exactly known for its great error messages). I've been doing regression testing against hackage and I'm satisfied that the new parser matches close enough. I've uncovered all kinds of horrors with .cabal files in the wild relying on quirks of the old parser. I've made adjustments for most of them but I will be breaking a half dozen old packages (most of those don't actually build correctly because though their syntax errors are not picked up by the parser, they do cause failure eventually). So far I've just done the outline parser, not the individual field parsers. I'll be doing those next and then integrate. So this change is still a bit of a ways off, but I thought it'd be useful to warn people now. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Advance notice that I'd like to make Cabal depend on parsec
On Thu, 2013-03-14 at 14:53 +, Duncan Coutts wrote: Hi folks, I want to give you advance notice that I would like to make Cabal depend on parsec. The implication is that GHC would therefore depend on parsec and thus it would become a core package, rather than just a HP package. So this would affect both GHC and the HP, though I hope not too much. It's already been pointed out to me that this also implies the following dependencies: text, deepseq, mtl, transformers deepseq is a core package already I think, though ghc doesn't actually depend on it currently. I should also say that I want to make Cabal depend on bytestring and text too. -- Duncan Coutts, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/ ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Advance notice that I'd like to make Cabal depend on parsec
On Thu, 2013-03-14 at 16:06 +0100, Gregory Collins wrote: On Thu, Mar 14, 2013 at 3:53 PM, Duncan Coutts duncan.cou...@googlemail.com wrote: Hi folks, I want to give you advance notice that I would like to make Cabal depend on parsec. The implication is that GHC would therefore depend on parsec and thus it would become a core package, rather than just a HP package. So this would affect both GHC and the HP, though I hope not too much. +1 from me, although the amount of potential knock-on work might be discouraging. The current cabal-install bootstrap process (which is currently pretty easy and is necessary at times) will get a bunch more deps as a result of this change, no? Yes it will, but given that we do have a script it's not too bad I think. And overall I think its worth it to have the better error messages, performance and memory use. Do you have any idea how slow it is to parse all the .cabal files on hackage, and how much memory that takes? You'd be horrified :-) Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Advance notice that I'd like to make Cabal depend on parsec
On Thu, 2013-03-14 at 12:22 -0300, Administrator wrote: This GHC dependency on Cabal is putting a rather troubling constraint in Cabal's evolution, which in my opinion is a serious problem. When I first took a look at the dependencies between GHC and Cabal I found it a bit strange that GHC would depend on Cabal as I would expect GHC to be as low in the dependency tree as possible to avoid exactly these kinds of problems. The problem is that a compiler is a rather sophisticated application and so though you'd like it to have minimal deps, it needs to do so much stuff that it ends up needing lots of deps to support its features. Things would be easier if that were not the case, and it's made harder by the fact that ghc is not just a program, but it's exposed as a library, which exposes all of its dependencies. These GHC dependencies on Cabal are in fact small (see http://hackage.haskell.org/trac/ghc/attachment/ticket/7740/ghc-2.png for a summary) and with a little bit of refactoring it would be possible to split these dependencies into a very small shared package with minimal or no further dependencies. This would liberate Cabal to make the necessary refactoring. Except that the bits of Cabal that ghc needs are exactly the bits that will now need parsec, text etc. The shared part would be the part that defines the InstalledPackageInfo and the parser for that. Also, though the ghc library has only relatively small dependencies on Cabal, the ghc build process uses Cabal extensively, and currently the system is that libraries that ghc needs to build get included as core libraries and shipped with ghc. That itself could change but it's also more work. IMHO, the addition of these new dependencies to Cabal should go together with splitting the GHC-Cabal shared dependencies into a separate package so that there would be no additional coordination needed from then on between these two development efforts (except when dealing with this new package). So I would consider this if I thought it'd make a difference. In particular at some point we'll want to split the Cabal lib into the bit that just defines types and parsers etc, and the part that is a build system. But even that wouldn't save us any dependencies in this situation. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Advance notice that I'd like to make Cabal depend on parsec
On Thu, 2013-03-14 at 09:39 -0700, Jason Dagit wrote: Why did I choose parsec? Practicality dictates that I can only use things in the core libraries, and the nearest thing we have to that is the parser lib that is in the HP. I tried to use happy but I could not construct a grammar/lexer combo to handle the layout (also, happy is not exactly known for its great error messages). Failed attempt aside for a moment, I think you should reconsider happy. Can you learn how to do layout from reading the GHC source? Yes I looked at it, though Haskell's layout is a bit different. The happy documentation that explains how to attach a monad (you could use it to communicate between alex and happy for layout info) is a bit misleading but I have examples I can share with you. Yes, that's what I was doing. I've used happy with monadic lexers with feedback between the lexer and parser before, e.g. when I wrote the C parser now used in language-c. I haven't specifically tackled the layout problem but I could try to make a parser if it would help. One major benefit of using happy is that the productions of the grammar can be analyzed for shift/shift and shift/reduce conflicts. Right, I know and that's great. For example there's no way I could have extended the C89 grammar I started with to cover C99 and GNU C extensions without the aid of that analysis. In this case I could not for the life of me construct a grammar that didn't have conflicts. Now it's plausible that now that I have worked out a grammar using parsec that I could have another go with happy and make it work, though I'd have to do the layout rather differently from how I do it with parsec. I was so pleased to finally have something work, I didn't feel like going back and trying it with happy again. I'd be happy to show you the code I've got with parsec and you can have a go with happy. The equivalent analysis doesn't appear to be possible in parsec. In theory, applicative parsers should allow for this but my understanding is that parsec does not have this feature for its applicative subset. Right, it doesn't. Other benefits are: a) GHC can certainly use parers generated by it, b) the generated code uses common dependencies, c) it's fast, d) it's expressive. Yes, I started with happy for all those reasons. The speed isn't a problem here. I'm using a fast lexer using alex and profiling indicates that still almost all the time is spent in the lexer and very little in the parser. (And that's after I submitted a patch to alex which gets us a 30% perf improvement.) About dependencies. So if we got it working with happy, there is still the issue that we need to parse the individual fields. The way the .cabal (and other files like ghc-pkg input files) work is that we parse the outline and then use individual parsers on the fields. For the latter we use a type class with a parser and pretty printer. That approach using a type class more or less requires that we use a parser combinator approach, rather than a monolithic happy style parser. And it's actually the field parsers that are a large part of the problem: they give us no error messages and their performance is atrocious (that's where we get the massive memory blowups). I think happy just isn't suitable there, so I'd want to use parsec (or any other decent combinator lib) for that part anyway. What is it about happy parser errors that you don't like? Do you know examples where parsec does a better job? Happy doesn't really give parser errors at all as such. It tells you where it failed and you can poke at the token stream and do what you like. It doesn't tell you what production you're in, what set of tokens it was expecting, nothing. Parsec tells us what tokens it was expecting and it tells us what production it was in and it has code to take that info and generate reasonable error messages from it (which I've extended to include the line in question and a visual position indicator). The reason ghc's parser error messages are so bad is exactly because happy doesn't really give us anything to work with. See frown for an example of how we can do better, while still using an LALR(1) approach. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Advance notice that I'd like to make Cabal depend on parsec
On Thu, 2013-03-14 at 16:44 +, Simon Peyton-Jones wrote: Yes I think that'd be a great plan. It's bizarre that GHC depends on *all* of Cabal, but only uses a tiny part of it (more or less the Package data type I think). The sensible way to split it (I think) would be like this: cabal-lib: Distribution.* -- containing definitions of types and parsers pretty printers -- including the InstalledPackageInfo cabal-build-simple Distribution.Simple.* -- the build system for Simple packages cabal -- the program, what is currently called cabal-install And then the ghc package would only depend on the cabal-lib package. But it's that package that is going to use bytestring, text, parsec etc, for its type definitions and parser. The InstalledPackageInfo and its parser is what ghc and ghc-pkg primarily use (though there's the opportunity to share code for handling package indexes) and that type and that parser are also going to end up using text and parsec etc. It'd be possible to split things out further and have InstalledPackageInfo and the types it uses and a special parser just for that with fewer dependencies, but I'm not sure that's really worth it and it would duplicate things (the types and/or parsers shared by InstalledPackageInfo and the source package description). So all in all, the split I suggest above makes sense for its own reasons but it wouldn't help ghc here, and a further split just to help ghc would be rather annoying. Duncan | -Original Message- | From: cabal-devel-boun...@haskell.org [mailto:cabal-devel-boun...@haskell.org] | On Behalf Of Administrator | Sent: 14 March 2013 15:23 | To: Duncan Coutts | Cc: Lentczner; cabal-devel; Haskell Libraries; ghc-d...@haskell.org | Subject: Re: Advance notice that I'd like to make Cabal depend on parsec | | This GHC dependency on Cabal is putting a rather troubling constraint | in Cabal's evolution, which in my opinion is a serious problem. When I | first took a look at the dependencies between GHC and Cabal I found it | a bit strange that GHC would depend on Cabal as I would expect GHC to | be as low in the dependency tree as possible to avoid exactly these | kinds of problems. | | These GHC dependencies on Cabal are in fact small (see | http://hackage.haskell.org/trac/ghc/attachment/ticket/7740/ghc-2.png | for a summary) and with a little bit of refactoring it would be | possible to split these dependencies into a very small shared package | with minimal or no further dependencies. This would liberate Cabal to | make the necessary refactoring. | | IMHO, the addition of these new dependencies to Cabal should go | together with splitting the GHC-Cabal shared dependencies into a | separate package so that there would be no additional coordination | needed from then on between these two development efforts (except when | dealing with this new package). | | | On Thu, Mar 14, 2013 at 12:12 PM, Duncan Coutts | duncan.cou...@googlemail.com wrote: | On Thu, 2013-03-14 at 16:06 +0100, Gregory Collins wrote: | On Thu, Mar 14, 2013 at 3:53 PM, Duncan Coutts | duncan.cou...@googlemail.com |wrote: | |Hi folks, | |I want to give you advance notice that I would like to make Cabal depend |on parsec. The implication is that GHC would therefore depend on parsec |and thus it would become a core package, rather than just a HP package. |So this would affect both GHC and the HP, though I hope not too much. | | | +1 from me, although the amount of potential knock-on work might be | discouraging. The current cabal-install bootstrap process (which is | currently pretty easy and is necessary at times) will get a bunch more deps | as a result of this change, no? | | Yes it will, but given that we do have a script it's not too bad I | think. And overall I think its worth it to have the better error | messages, performance and memory use. Do you have any idea how slow it | is to parse all the .cabal files on hackage, and how much memory that | takes? You'd be horrified :-) | | Duncan | | | ___ | cabal-devel mailing list | cabal-devel@haskell.org | http://www.haskell.org/mailman/listinfo/cabal-devel | | ___ | cabal-devel mailing list | cabal-devel@haskell.org | http://www.haskell.org/mailman/listinfo/cabal-devel ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Advance notice that I'd like to make Cabal depend on parsec
On Thu, 2013-03-14 at 21:29 +0200, Roman Cheplyaka wrote: * Duncan Coutts duncan.cou...@googlemail.com [2013-03-14 17:12:14+] The InstalledPackageInfo and its parser is what ghc and ghc-pkg primarily use (though there's the opportunity to share code for handling package indexes) and that type and that parser are also going to end up using text and parsec etc. Correct me if I'm wrong, but isn't it just a strange coincidence that InstalledPackageInfo is serialised in the format similar to .cabal format? It's not a very strange coincidence. The type is not specific to ghc, it's defined in a compiler-neutral way by the original Cabal spec. So since both the source package and installed package info was defined in the Cabal spec, using the same kind of external syntax and sharing many of the same types, then they both ended up in the Cabal lib and share the same parsers pretty printers. InstalledPackageInfos aren't supposed to be edited by hand and do not need good error reporting. They can be serialized using any serialization library. Right, it doesn't need good error reporting (though it's nice if it's fast, which it isn't currently). The main advantage of the current arrangement is that the source and installed package descriptions get to share the same types and parser/pretty printer. I think there's a slightly more general point here though. Why is it that we don't have any good parser in the core packages? It's not just Cabal that needs to parse things. We have two useless parsers in the base package, ReadS and ReadP. Haskell is famous for its parser combinators and yet our core infrastructure is stuck with only useless ones! Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Advance notice that I'd like to make Cabal depend on parsec
On Thu, 2013-03-14 at 11:15 -0700, Jason Dagit wrote:
On Thu, Mar 14, 2013 at 11:01 AM, Bardur Arantsson
s...@scientician.netwrote:
On 03/14/2013 03:53 PM, Duncan Coutts wrote:
Hi folks,
Why did I choose parsec? Practicality dictates that I can only use
things in the core libraries, and the nearest thing we have to that is
the parser lib that is in the HP. I tried to use happy but I could not
construct a grammar/lexer combo to handle the layout (also, happy is not
exactly known for its great error messages).
Just thinking out loud here, but what about ditching the current format
for something that's simpler to parse/generate? Like, say, JSON?
Of course .cabal files are mainly written by humans, not machines, so we
should optimise for them. The grammar I've got now really isn't that
bad. In fact if we wanted to simplify it we'd rip out the bits that are
designed to make it easier to generate by programs: we'd eliminate the
explicit {} syntax and just use layout. Allowing either is what makes
the grammar more complex. But as I say, I'm satisfied that the grammar
is ok.
I thought I heard someone say that most existing cabal files can be
converted to valid yaml by adding one token at the start? If the change was
that simple it might be doable. I think the trick is that we'd need to
expose this by only treating the file as yaml if the minimum cabal version
is = 1.17 (or so).
I know people have compared it to yaml and suggested we just use yaml,
but I don't think it's that close syntactically. I did look into this
when I started and I think there are too many differences to make it
practical to switch to yaml (or a subset).
In general these sorts of format changes are painful for users and I sense
that now might be a bad time to change it (user morale is already a bit low
with complaints of cabal hell, let's not exacerbate that by breaking
existing .cabal files).
Right. I'm satisfied the format is basically ok, we don't need any
breaking changes.
--
Duncan Coutts, Haskell Consultant
Well-Typed LLP, http://www.well-typed.com/
___
cabal-devel mailing list
cabal-devel@haskell.org
http://www.haskell.org/mailman/listinfo/cabal-devel
Re: hackage trac
On Wed, 2013-02-20 at 12:59 +, Ben Millwood wrote: There's lots of stuff in the trac that still doesn't live anywhere else, and references to it in the README and docs. Are we planning to migrate that content anywhere else? Are we planning to use trac in future? I'd be happy to help move stuff off trac wiki to github wiki, if that's what we decide is best. See also: https://github.com/haskell/cabal/issues/1204 https://github.com/haskell/cabal/pull/1211 I have re-enabled the trac. I deleted the 100k spam wiki changes and compacted the database. I also removed the WIKI_MODIFY permission from authenticated users. I've given you WIKI_ADMIN permissions so that you can move things to the github wiki. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
hackage 2: alpha testing, testers wanted!
Hi folks, I believe we're ready for alpha testing for hackage 2. The server is up at: http://new-hackage.haskell.org/ Key points: * All the old data is imported (including docs) * Uploading is enabled * Live mirroring is also enabled * Per-package maintainer groups are pre-populated * Existing users should be able to log in (after a one-time upgrade) I would like to invite everyone on this list to help out with the alpha testing and to report issues here on this list. Patches are even better. The server is up now so you'll get instant patch gratificationtm if you chip in and fix things. In addition to any bugs you may discover, the server main page gives an overview of the features that are partially complete and could do with some love. When we're satisfied we'll move into beta testing and advertise more widely within the community (reddit etc). The doc builder is not set up yet, but the builder client code exists, just need to get that set up. In general the switch over plan remains that we run the two servers in parallel for long enough to sort out remaining issues (with help from the kind folks who volunteered as admins). Code and wiki = The code is all still in darcs. darcs get http://code.haskell.org/hackage-server/ The plan is to move it to github at some stage. The wiki content is still useful though starting to get a bit out of sync: http://hackage.haskell.org/trac/hackage/wiki/HackageDB/2.0 We had to turn off wiki editing for most users because of spammers. If you'd like to work on the wiki, let me know and I'll give your account permission to edit it. Note also that I think Ben Millwood is planning to move the content to github at some stage, though we can still make improvements to the content in the meantime. Hackage Admins == I believe the following people volunteered to be admins, at least for the alpha/beta testing period: * Erik Hesselink * Ben Millwood * Johan Tibell If you folks (or anyone else who wants to volunteer) would like to confirm that then I'll add you to the admins group. User accounts = Existing users should be fine. That process should now be fairly smooth (testing welcome!). For new users, they can sign up with email confirmation. An administrator is needed to add a user into the uploaders group. Currently there's not a lot you can do with an account if you're not in the uploaders group. That may change in future if we implement social features. Hackage Trustees In addition to admins we have the trustees group. This should eventually be a rather useful job. The idea is that the trustees don't upload packages but look after the health of the package collection as a whole (or some subset they're interested in), a bit like distro maintainers. Currently they can only edit package tags and upload docs (e.g. if the doc builder fails) but the plan is for them (as well as maintainers) to be able to edit package metadata, in particular the constraints on dependencies. Once we get there, that'll be a rather important job. So I'm also looking for volunteers there, both to be trustees and to help finish implementing the metadata-editing feature. -- Duncan Coutts, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/ ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Cabal and cross compilation
On 23 January 2013 05:41, Nathan Hüsken nathan.hues...@posteo.de wrote: Hey, I am working on getting ghc to cross compile to android. When trying to get haskeline to compile. I want to change the cabal file such that it sets a flag when compiling for android. For that I changed cabal so that it recognizes android as a OS. But cabal seems to get its os information from System.Info.os, and from what I can tell this always returns the host os and not the target os. Am I getting this right, is cabal unaware of the target os? How can we change this? That's right, currently Cabal only knows about the host OS arch not the target. Adding proper cross compilation awareness and support into Cabal will require some hacking in the Cabal library, (to pass in the target platform, toolchain etc) Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
current hackage 2 status
Hi folks, You may have noticed the steady stream of hackage 2 patches. The new server is up again at: http://new-hackage.haskell.org/ Key points: * We've got live mirroring going (sync every 30 min). * All the old accounts are imported and the per-package maintainer groups are populated. * Package deprecation info is imported * Distro info is imported * Currently uploading is disabled (pending review of authentication authorisation code). * Documentation is not imported (Ian demoed this before, just haven't got round to it yet for this instance) * The data backup system is much improved. In general the switch over plan remains that we run the two servers in parallel for long enough to sort out remaining issues (with help from the kind folks who volunteered as admins). There was some discussion before about user accounts and the issue of per-package maintainer groups. So the system I envisage for user accounts is that anyone can make an account, but to be added to the uploaders group will require approval from a volunteer administrator (technically speaking there are not yet any features that need an account but don't involve uploading, but I'm assuming some future social features will need this). I do think the per-package maintainer groups are important for security, and now that we can automatically populate them from the old hackage data, I don't think that's a problem. Still missing is a proper process for new account creation. That is, account creation with email validation and all that. There's a couple of options here. We can build it into the server, or we can make a separate page/script to do that (which would make accounts using the appropriate curl -X PUT commands against the server). -- Duncan Coutts, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/ ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: license cabal field for language bindings?
On Sun, 2012-12-16 at 20:14 +0100, Herbert Valerio Riedel wrote: Hello Cabal Devs, Since the recurring topic of software licenses is being discussed on haskell-cafe again, I was wondering what the .cabal license-field should be for packages linking _directly_ to C-libraries, such as Haskell language bindings to C-libraries. To be more specific: As a side-effect of the ongoing discussion, Michael Snoyman implemented a web-based tool for computing the transitive license multiset[1] which made me wonder whether the .cabal license field should describe only the license of the source code proper distributed with the package or whether it should also take into account the licenses of the wrapped/linked libraries? It should be the license of that package's source code only. The full bag of licenses can indeed be calculated. So bindings are not special (unless they actually ship the C source code as part of the package). Calculating the licenses when C libs are involved as dependencies is more tricky because the license info isn't quite so easily available in a machine readable form. You'd probably want to combine info from say the debian package db, find out which debian packages provide which C libs, and we know which C libs are needed by each cabal package. Join those two together and look up the license of the debian packages providing the C libs. PS: A separate but related issue was raised in the haskell-cafe discussion: Many packages depend on the cpphs library as a hack, although they only need the cpphs executable (and therefore don't link against the cpphs package). This also skews the result that tools like [1] report, as the cpphs library is LGPLed which doesn't matter if only the cpphs executable was actually used for building the package. That'd be solved if we did dependencies on programs properly. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: What build environments should Cabal support?
On 14 December 2012 23:33, Johan Tibell johan.tib...@gmail.com wrote: On Fri, Dec 14, 2012 at 3:29 PM, Ben Millwood hask...@benmachine.co.uk wrote: Do we therefore have no interest in compilers other than GHC? Can we drop all instances of ifdef __HUGS__ from the code? I think doing so would require a reasonable amount of work, but would significantly lower the barrier to contributing to Cabal. I think so. To my knowledge there's no one who's actively maintaining and testing the Hugs support in Cabal. It's most likely broken. We should distinguish the compilers that can build Cabal, from those that cabal supports. Cabal supports several compilers that cannot themselves build Cabal. I am happy to drop the support for building Cabal using hugs and nhc as I don't believe they have been used for some years. In particular this will let us fully use new-style IO exceptions, and the new createProcess function. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Cabal and GHC
On 22 November 2012 09:17, Henning Thielemann lemm...@henning-thielemann.de wrote: On Thu, 22 Nov 2012, Simon Peyton-Jones wrote: The solution is obvious: we should make it possible to instally yesod-platform-2.7 twice, once version depending on data-default-0.4 once version depending on data-default-0.5. Does ghc-pkg allow to have two different versions of yesod-platform-2.7 being installed? No, not right now. But the point is, if it did... Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Status of cabal repl
On Wed, 2012-11-07 at 20:36 -0800, Johan Tibell wrote: Hi Duncan, I'm trying to finalize the list of things to include in Cabal/cabal-install 1.18, which I hope to push out by the end of this year. What's the current status of the cabal repl work? Is it close enough to being finished for someone else to pick up and take the rest of the way? I'm reasonably happy with working on this part myself. It's the main cabal hacking I'm doing right now. At a guess I'd say another few weeks. In my naive understanding of the problem, now when we have refactored the way we create GHC command line invocations and gotten the ghci changes into GHC, it should be as simple manner of creating the needed command line (similar to cabal build but with e.g. an --interactive in there) and invoke ghci. This is probably a bit naive so perhaps you could clarify what's needed? The bit I'm working on right now is actually the command line handling. I'm trying to make it easy to use, only requiring disambiguation between targets when really needed. I've just about finished writing a little matching monad. The internals I think are if anything easier. We've now got the graph of components, and we've got the more structured command line handling. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Providing a smooth default user experience
On Fri, 2012-11-02 at 16:52 +0100, Simon Hengel wrote: Duncan, Moreover, it would be awesome if it would regenerate generated source files on :reload. I'm not sure if this is feasible without changes to GHCi, but a custom macro, e.g. :cabal-reload could still work. The GSoC project that started this off did that, and the necessary changes are in ghc now. That is interesting. I'd be interested in how exactly that works. Is there a place where I can read about it? Or if not, can you point me to the related commits? http://lambdasandwich.blogspot.co.uk/2011/08/ending-gsoc.html For the changes to ghci see the ghc git repo. Look for stuff related to .ghci file handling. That was the main extension needed, so we could specify a .ghci file on the command line. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Providing a smooth default user experience
On Fri, 2012-11-02 at 11:25 +0100, Simon Hengel wrote: Recently, I had to add a MIN_VERSION macro to my code because of the old-time / time change in directory, but couldn't test the library in ghci because the macro was not defined. I discovered that configuring ghci with :set -optP-include -optPdist/build/autogen/cabal_macros.h tells ghci about the macro, but I find this even more cumbersome than dist/build/executable/executable. So, if I'm not missing something, it may also be comfortable to add a cabal ghci command which tells ghci about cabal's configuration. You can put the required options in a .ghci file in you project root. I use [3] as a starting point for new projects, [2] and [3] are real world examples. `cabal ghci` could still be useful, but I think a general solution is not trivial. It would probably need to accept qualified targets, e.g. `ghci test:properties` to start ghci in a way suitable for a test-suite with the name properties. Yes, we have been working on this. Last weekend I was updating the code for doing the qualified target stuff cabal build exe:foo etc. Moreover, it would be awesome if it would regenerate generated source files on :reload. I'm not sure if this is feasible without changes to GHCi, but a custom macro, e.g. :cabal-reload could still work. The GSoC project that started this off did that, and the necessary changes are in ghc now. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Providing a smooth default user experience
On Wed, 2012-10-31 at 10:22 -0700, Johan Tibell wrote: Hi all, Sensible defaults are important to provide a smooth user experience, especially for first-time users. There are a few areas where I think cabal could improve its default behavior. In particular, here are a couple of changes I would like to make: *cabal build should imply cabal install --only-dependencies * If you're currently working on a package, the right [1] way to build it is as follows: cabal install --only-dependencies cabal build I've been thinking along similar lines recently. We should be thinking about package environments (which spans multiple packages) rather than individual packages, and the UI should reflect that. Right now it's very much centred on individual packages, not collections. So cabal configure should configure/reconfigure the package environment, not the individual package. It should make an install plan for the current environment (which at minimum is a single target package and its dependencies but could be several packages in the same env). Then of course executing that install plan by building any of the targets should go and build deps as necessary. Of course from a UI pov, we have to make it clear what's going to happen. When you configure an environment, it needs to say that it'll involve installing several things locally. That's also the appropriate point to freeze (or partially freeze) a package environment to give a reproducible build. I think when changing the packages in an environment, we should always enforce complete consistency of dependencies. So you know straight away if you can add two packages to the same environment, and not let people get into the situation of broken packages. You can change stuff and rebuild, but we should always have an install plan for the whole lot, not just a subset. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Gearing up for a cabal-install-1.16.0 release
On 1 October 2012 04:21, Johan Tibell johan.tib...@gmail.com wrote: Hi, I don't want to hold up the release on this issue if it needs more discussion. I'd like to make another Cabal/cabal-install release before end of the year (probably around GHC's release timeline), to include the sandbox stuff once done. Would it be OK to wait until then with these two patches? I think these specific patches do not need discussion now, here's why: the feature comes in two parts, server and client, the bulk of the feature is the server side part which is not implemented yet. There's plenty of time for more discussion of the feature in general before we start actually using it. But if we decide it's ok, then we have to wait even longer to start using it. We can deploy new server side code quickly, but not client side code. If we decide it's not ok, then we don't make use of the feature server-side and then we rip out the client code. So that's why I want to get the client code in now, in advance of the server side implementation. Also note, I have discussed this before with several people (and I think on this list too, a couple years back). So Bryan didn't see it, but it's not like I'm inventing this crazy idea from nowhere and forcing it on people. I'm very happy to discuss the details again, but that's independent of these two patches. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Gearing up for a cabal-install-1.16.0 release
On 30 September 2012 05:26, Bryan O'Sullivan b...@serpentine.com wrote: On Fri, Sep 28, 2012 at 5:55 PM, Duncan Coutts duncan.cou...@googlemail.com wrote: Being able to do this relies on both editing on the server, and for the client to actually use the updated .cabal file. So this patch provides the client side part of the feature. The server side already works (we've actually made quite a few edits to .cabal files on hackage post-release) but the UI for it will improve significantly in the new server which we expect to go live during the lifetime of this next cabal-install release. So getting the feature in now would be a real help to everyone (and ideally, most users will never notice). This feels like a big piece of work to be pushing in right before a release. It worries me because now there's an invisible piece of metadata floating around that neither OS packagers nor library maintainers can see. In a way, this feels strictly worse than having a package just plain break, because it's not hard to imagine a cycle of package breaks; magic invisible dependency update silently fixes it; new version is released; package re-breaks because maintainer never found out about previous breakage. I could be convinced that in fact everything is going to be awesome and it will all somehow work, but in the short term I'd prefer to delay sprinkling invisible dependency pixie dust on packages, and let this wait until the next release, 3 months down the line. I realise there's more to the management side of this feature. But that's something that mostly has to be implemented on the hackage side of things. There's not that much that needs to be in the client. So I'd prefer to have clients support this early, and we can work on the management issues on the server. In particular, hackage will keep track of the revision, and this will be reflected in the .cabal file. Also, all revisions of the .cabal file will be available on the server. So OS packagers and maintainers can see it, and decide if they want to take the original or a revision. The reason I think a scheme like this will work is because it does work in other systems. For example in Gentoo they have ebuilds (much like .cabal files) and they version the ebuilds with an extra revision number, beyond that of the upstream package. Yes, the revision does need to be visible to users so that it doesn't confuse things. And that's where we will probably need some further work in the client, to make this info visible. And yes, there is an issue of how do we communicate fixed back upstream so they get incorporated into the next release. That's another management issue that's mostly a server side issue. So again, we'll work on the management issues on hackage, but having the client support the feature will make it easier and smoother introducing that. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Gearing up for a cabal-install-1.16.0 release
On 27 September 2012 15:44, Johan Tibell johan.tib...@gmail.com wrote: Hi all, There's been a long time since we had a cabal-install release and there are a lot of bug fixes in the repo we need to get out there. After discussing with Duncan we agreed that the cabal-install release should depend on the already released Cabal-1.16.0 package. I've created a new branch, cabal-install-1.16.0, for making the cabal-install-1.16.0 release. The branch was made from the point in the history where Cabal-1.16 was released. There were some patches on the cabal-1.16 branch that never made it into the Cabal-1.16 release. If there are any commits on the cabal-1.16 branch that you think should definitely be in cabal-install-1.16.0, please let me know and I will try to cherry-pick them. Here's the list of commits: I mentioned I'd written a feature to update the .cabal file we use when building with the same one as is in the package archive index. I've now pushed that to the head branch. I think it would be good to have on the 1.16 branch and included in the cabal-install release. For context, as you know there's been a lot of discussion about dependency version bounds and if we should use conservative or optimistic bounds. Something that we've been thinking about for a while that would help here is if we could adjust the dependencies after a release. For example if you used conservative bounds on a dependency and later on a new version of that dependency is released, and it so happens that you were lucky this time and your package does still build and work with the new version. It'd be nice if you (or some other helper monkey) could just adjust the version constraints to reflect reality. Or the other way around, if you were using optimisitic bounds and a new version of a dep is released and you were unlucky and it now fails, then again we can tighten the version bounds. Being able to do this relies on both editing on the server, and for the client to actually use the updated .cabal file. So this patch provides the client side part of the feature. The server side already works (we've actually made quite a few edits to .cabal files on hackage post-release) but the UI for it will improve significantly in the new server which we expect to go live during the lifetime of this next cabal-install release. So getting the feature in now would be a real help to everyone (and ideally, most users will never notice). The patches in question are below, they'll need to be merged into the 1.16 branch but I don't expect any problems there. commit b7565f941b2dd5d61a77e46826fd256358505de7 Author: Duncan Coutts dun...@community.haskell.org Date: Sat Sep 29 01:41:31 2012 +0100 Extend the unpack command for the .cabal file updating By default, cabal unpack blah will also update the .cabal file with the one from the index, so it's consistent with what you get via cabal install. Also added a --pristine flag so you can get the original tarball without the updated .cabal file. commit b92cbb046feb20be6b656752b25d12a2f26abd42 Author: Duncan Coutts dun...@community.haskell.org Date: Sat Sep 29 01:34:58 2012 +0100 On install, update the .cabal file with the one from the index This allows us to make minor changes to packages after they have been released, without changing the package .tar.gz file. We already keep the .cabal file outsite the package in the index and use it for dependency planning. This already lets us do fixes such as making dependency constraints tighter. Currently we cannot make dep constraints more relaxed however, since the original .cabal file is the one used when we get to the actual configure step. So with this change, we now use the updated .cabal file for the configure and build too. So there's more fixes we can do post-release. In particlar, in combination with easier editing on hackage, this should help us address the problems around the PVP and open or closed version constraints. It should allow a system of conservative upper bounds, but allow editing them when new versions of deps are released and we find that they happen to work fine. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Hackage 2
On 5 September 2012 20:22, Erik Hesselink hessel...@gmail.com wrote: Also, we haven't had a single problem that I'm aware of on Ross Paterson's watch as bouncer for Hackage 1.The point I'm trying to make is that a technical solution imposes additional administrative and technical overhead whereas social processes can also be very effective while also handling corner cases more gracefully. I don't see how a technical solution (which is already implemented, by the way) introduces *more* overhead than a manual solution. Also, the fact that we haven't had any problems doesn't mean we won't in the future. We don't have to wait before something goes wrong to fix it. As I think you know, I'm definately in favour of the per-package maintainer group stuff. Let me make one more argument: even if we don't in practice have problems with people uploading packages they shoudn't, it'll make everyone *feel* better (that is, package maintainers and users). We do get a bit of stick for the current lack of security (not just this issue but about the lack of tamper profing / detecting). Additionally, if you decide that you would prefer to allow anyone to upload without having to get manual approval to be in the uploader group, then the per-package maintainer group becomes very useful. You could have more or less a free for all in uploading new names, but nobody can subvert existing names. (We would still have the problem of people taking all the good package names for crappy packages, but that's another issue) I understand we're not planning on importing the accounts from the old server. Could someone explain the issue there? I'd assumed we'd do that for a smoother changeover (and to set up the initial maintainer groups). Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Hackage 2
On 6 September 2012 19:53, Ian Lynagh i...@well-typed.com wrote: On Thu, Sep 06, 2012 at 07:14:53PM +0100, Ross Paterson wrote: On Thu, Sep 06, 2012 at 06:49:36PM +0100, Matthew Gruen wrote: To my knowledge, It's technically possible to import the old accounts. Why is that? I think that's worth exploring, as you'd be losing associations between 1150 users and 24456 uploads of 4442 packages. (Not entirely, because the old names would be there as text, but they wouldn't be connected to user-ids.) I think you misread possible as impossible? FWIW, I assumed that there had been a decision that importing accounts is either impossible or unwanted, as the mirror script doesn't do it and there was no TODO item for it anywhere. But if that's not the case then I can look into importing them. There's two bits: there's the bulk import and then there's the mirroring bot. The bulk import can import the user accounts, given the htpasswd file from the existing server. Obviously the mirroring bot does not have access to that. The mirroring bot can do something with users however, it will try to set the owner of the new package on the destination server to be the same as the owner on the origin server, but I'd have to check the details. Here's the tricky part... (Matthew Gruen and I spent some time investigating this but I'm unsure how much of it is implemented...) The existing hackage server uses http basic authentication and an apache htpasswd file. The new server code uses http digest authentication as this is somewhat better (not as good as ssl but better than basic auth). The passwords from the htpasswd file cannot be converted into the storage format for the digest auth (not without cracking them!). Also, we cannot use either basic and digest depending on the user. So the solution is this... From the point of view of an existing user, they will go to the new site, go to a special page, enter their username and their old password and a new password (which can be the same if they like). This is a one off. After that they use the site as normal with their new password. What is really going on is that for the old users we keep a note that they have an old htpasswd-style password hash. When they go to the special password change page, we authenticate them using http basic auth using their old password. We set their new digest passwd and delete their old basic auth. This ought to be able to be done without affecting the core features, as a separate hackage feature, storing the old htpasswd stuff separately. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Hackage 2 and acid-state vs traditional databases
On 6 September 2012 19:49, Ian Lynagh i...@well-typed.com wrote: Hi all, I've had a bit of experience with Hackage 2 and acid-state now, and I'm not convinced that it's the best fit for us: * It's slow. It takes about 5 minutes for me to stop and then start the server. It's actually surprising just how slow it is, so it might be possible/easy to get this down to seconds, but it still won't be instantaneous. Yes it probably is slower than necessary there. It should be possible to make it as fast as reading the data from disk. For near-instantaneous server code upgrades we would need to use a feature of happs-state that currently isn't implemented in acid-state (afaik). The happs-state allowed clustering so you could start a new server process, let it sync up with the state from the existing process, then kill the old process. * Memory usage is high. It's currently in the 700M-1G range, and to get it that low I had to stop the parsed .cabal files from being held in memory (which presumably has an impact on performance, although I don't know how significant that is), and disable the reverse dependencies feature. It will grow at least linearly with the number of package/versions in Hackage. I think this is solvable. The most costly thing is the package metadata. We can use more compact representations (the Cabal PackageDescription is pretty bad in this respect) and secondly by doing a lot of sharing within the package index. Both of these changes could be done just in the Cabal library without significantly affecting things. The sharing would mean it grows linearly but much more slowly. That said, I don't think 1GB should be considered high. If we want to be able to have good performance, then we want all commonly used data in memory anyway. 1GB for a central community server is not at all unreasonable. * Only a single process can use the database at once. For example, if the admins want a tool that will make it easier for them to approve user requests, then that tool needs to be integrated into the Hackage server (or talk to it over HTTP), rather than being standalone. * The database is relatively opaque. While in principle tools could be written for browsing, modifying or querying it, currently none exist (as far as I know). In both these points, I would argue that we should simply make all the data available via http REST interfaces. Instead of making the data available only locally to someone with direct access to the database, it should be there in machine readable form so that everyone can get it and experiment. In the approving user requests example, that just needs an HTTP PUT/POST. It doesn't need any web form. Totally scriptable with wget/curl etc. * The above 2 points mean that, for example, there was no easy way for me to find out how many packages use each top-level module hierarchy (Data, Control, etc). This would have been a simple SQL query if the data had been in a traditional database, but as it was I had to write a Haskell program to process all the package .tar.gz's and parse the .cabal files manually. As I mentioned on IRC the problem here is really that cabal-install isn't currently available as a library. If it were then loading up the 00-index.tar file with all the .cabal files is just a couple lines of code. Then your query is just a list comprehension. As another example, we should be able to make all the package data available in other formats, e.g. JSON and that provides loads of opportunity for other people to do ad-hoc queries. * acid-state forces us to use a server-process model, rather than having processes for individual requests run by apache. I don't know if we would have made this choice anyway, so this may or may not be an issue. But the current model does mean that adding a feature or fixing a bug means restarting the process, rather than just installing the new program in-place. True. See above about quick restarts. Someone pointed out that one disadvantage of traditional databases is that they discourage you from writing as if everything was Haskell datastructures in memory. Right. This is even more notable with the Cabal types (like PackageDescription) as the types and various utility functions already exist - although it's currently somewhat moot as the current acid-state backend doesn't keep the Cabal datastructures in memory anyway. As I've said, I think that's fixable. The other issue raised is performance. I'd want to see (full-size) benchmarks before commenting on that. There's basically two approaches you can take here: scaling up or out, that is making a single server handle lots and lots of requests, or have lots of machines. It is much simpler (both in terms of code and infrastructure) to have a single server. The performance here can still scale a long way if we can keep all data that requests need in-memory. Additionally, we can scale read
Re: Hackage 2 and acid-state vs traditional databases
On 6 September 2012 21:06, Duncan Coutts duncan.cou...@googlemail.com wrote: On 6 September 2012 19:49, Ian Lynagh i...@well-typed.com wrote: * Only a single process can use the database at once. For example, if the admins want a tool that will make it easier for them to approve user requests, then that tool needs to be integrated into the Hackage server (or talk to it over HTTP), rather than being standalone. * The database is relatively opaque. While in principle tools could be written for browsing, modifying or querying it, currently none exist (as far as I know). In both these points, I would argue that we should simply make all the data available via http REST interfaces. Instead of making the data available only locally to someone with direct access to the database, it should be there in machine readable form so that everyone can get it and experiment. In the approving user requests example, that just needs an HTTP PUT/POST. It doesn't need any web form. Totally scriptable with wget/curl etc. I should probably say this more clearly... The original design of the new server was that it should primarily be a web database (REST and all that), with a user-facing website tacked on. That is, whatever extra fancy features it has, all of the data it stores should be accessible and modifiable via http (i.e. get/put/post) in at least one machine readable format. This alone makes the system extensible, and extensible by third parties. For the most part the current implementation does indeed make all the data available via standard http methods. We can make that better by adding more formats (esp json) and making sure we consistently make everything available, and by making it discoverable that everything is indeed available. The latter should be possible e.g. via an automatically generated site map. We have URL templates for all the resources that are available, and we can simply automatically collate those and make them available on a generated page. So if we've done it properly, there should be no need for direct access to the database. And as I said before, doing it properly has the great advantage that all the data is available to everyone to do cool things we've not thought of yet. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Hackage Accounts
On 5 September 2012 14:34, Lars Viklund z...@acc.umu.se wrote: On Wed, Sep 05, 2012 at 09:19:53AM -0400, Leon Smith wrote: Well, I guess that sort of counts as a refusal. What kinds of requests lead you to ask for more information, and what kinds of information have you asked for? On Wed, Sep 5, 2012 at 9:01 AM, Ross Paterson r...@soi.city.ac.uk wrote: On Wed, Sep 05, 2012 at 01:15:09PM +0100, Leon Smith wrote: I have a question for Ross Paterson, namely do you decline an account request, and if so, what reasons do you have for declining an account request? I haven't refused any requests, but occasionally I ask for more information and don't get a response. This chain of mails looks quite strange to me. Do you have some kind of ulterior motive for these inquiries? It almost sounds like you're trying to get someone to write themselves into a trap. If you're beating around a bush, out with it already. De-drama-ifyingly yours, Lars, the motive is quite simple. The existing hackage server doesn't have much in the way of security restrictions, we basically have to trust all people with upload accounts. So we have a manual, human process for requesting an account. In that context it's perfectly reasonable to ask for more details if the initial request didn't say much. In the new server the security is a little better, but we still want to have a manual step to grant uploader rights. Make sense? It's not a weird conspiracy :-) Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: New cabal-install release (and schedule) needed
On Sun, 2012-08-26 at 22:27 -0700, Bryan O'Sullivan wrote: We need to publish a new cabal-install 0.16 release shortly, ideally before GHC 7.6.1 and the next Haskell Platform come out. At the very least, there's a newish incompatibility between GHC 7.6 and cabal-install that causes all -Werror builds to fail because cabal-install uses a deprecated package flag. This was fixed back in May, but that needs to actually go out into the wild. Stepping back, I'd also like to suggest that we move to a quarterly release cycle on a simple, widely used model. A major goal here is to get code into people's hands on a predictable schedule. We maintain two branches, stable and master. Master freezes a month before a release, with only bugfixes and typo fixes going in. At release time, the stable branch opens (in case any emergency fixes are needed), and master opens up for normal development again. Johan and I are happy to do the work to manage these cycles. I suggest that we start the first cycle with a target release date of the end of September, then go four months to the next one (after all, it would be unrealistic to pretend that we might do a release at the end of December). Does this sound reasonable? I think help with managing the stable branch and making releases would be very helpful indeed. I did used to do more frequent releases but it's a time consuming task. I'd certainly prefer to spend time on patch review, cleaning up old code and adding new features. BTW, I'm no longer on honeymoon and I've nearly finished rewriting the bytestring I/O layer so I hope to have a bit more time for cabal patch review and hacking. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Hackage 2 status
On 3 July 2012 20:38, Johan Tibell johan.tib...@gmail.com wrote: On Mon, Jul 2, 2012 at 3:14 PM, Duncan Coutts duncan.cou...@googlemail.com wrote: Something to keep in mind is memory usage. I know Jeremy is looking at this from the infrastructure side, but I think from the app side there's also some likely culprits. Cabal's GenericPackageDescription type is very large in memory. Having 10's of 1000's of these means lots of memory. One hopefully easy way to save memory here without going to the hassle of redoing Cabal's type definitions is simply to increase sharing. There's a huge amount of repeated information. Start by sharing all the package names and versions. Then there's other meta-data that rarely changes between versions of the same package. This kind of thing should be easy to evaluate, just write a test prog that reads the index file and look at peak memory use. Then try sharing stuff and see how much it drops. This sharing optimisation would still be useful even if later we go and redo GenericPackageDescription to be more compact. This should not hold up the launch of Hackage 2 (which is very important) but I think it's an important issue that we need to address: we don't want to store the perhaps most important data the Haskell community has in an experimental data store! Creating a correct data store (i.e. ACID) that also handles a moderate amount of load is a quite difficult undertaking and it shouldn't be taken lightly. Lets stick the data in some SQL database and spend our energy on other things. :) I still disagree that going with an external SQL db will be easier. The big advantage of the acid-state (and similar) data stores is that they let us use Haskell types properly and don't imply a separate external data model and a marshalling stage. That said, I also do not trust acid-state for long term storage (simply because the binary format it uses isn't sensible) which is why the hackage server already has a system for dumping and restoring to standard formats (like csv, tarballs etc). So if we use this backup system properly (ie in combination with a system for backups to other machines) then I think there's little chance of data loss. Additionally, the really important data (the packages) are stored in the file system. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Hackage 2 status
On Mon, 2012-07-02 at 12:25 +0100, Ian Lynagh wrote: Hi all, I'm planning to spend some time, on behalf of the Industrial Haskell Group, working on Hackage 2 in the coming weeks. [..] Now #913 I assume is not a blocker. #919 I assume is also not a blocker. And #914 and #915 are improvements to the internals, so presumably also not blockers. #426 is not supported by Hackage 1 as far as I can see, so is not a blocker as it is not a regression. I agree with this analysis. So that leaves 3 tickets as blockers: #911: We need to do something here. With Hackage 1, it takes manual approval before you can upload packages, and at the very least Hackage 2 should match that. I have the impression that that is already possible (by restricting package upload to a group, and requiring accounts to be added to that group by an admin), but I haven't confirmed that yet. Right, I don't think we need to do any more than make sure uploaders are in the appropriate group. It *should* currently be the case that only accounts in the package group can upload, and the first time you upload a new named package then you get added as the initial member of the new package group. Currently for testing purposes anyone can register an account and can then upload new packages. We have two options here: restrict account creation to be manual like in hackage 1, or add a new system-wide uploaders group for accounts that are authorised to upload new packages and have a manual admin step to add people to the uploaders group. The latter will allow for registered users who are not uploaders which would be useful later to allow things like non-anonymous commenting etc. #916: At the very least, Hackage 2 needs to support URLs that Hackage 1 supported (unless a conscious decision has been made not to). Ideally we would get the URLs right on the initial release, so that people don't start using the wrong ones. Doesn't sound hard, so may as well do before the switchover. Yes, barring mistakes this should work already. There's a legacy feature module that provides a bunch of redirects. #918: This is the main missing functionality currently missing from the user's point of view. Right. You'll see there's some code for a doc builder. This needs to be improved. Perhaps it can share code with the mirror client which is reasonably robust. Conclusion -- I think the following are the blockers for deploying Hackage 2: * #911 upload perms; may be good enough already * #916 check URLs are OK * #918 build haddock (and HsColour) docs Right. * Show source respository on package pages Should be easy to port that from the old code. * Support the existing Distributions files, and show info on package pages I advocated at the time the feature was added that it should be done differently so that the hackage server does not poll some url, but people in charge of distros push instead. I think it would not be a blocker to not implement the distribution info system as it is now and when eventually spending the time to implement it, switch to doing it in a more sensible way. (plus enough testing to give us confidence in it, of course). One of the main things here is adding tests that the database dump/restore mechanism round trips correctly. Does that match other people's opinions? Did I miss anything? Looks good. Something to keep in mind is memory usage. I know Jeremy is looking at this from the infrastructure side, but I think from the app side there's also some likely culprits. Cabal's GenericPackageDescription type is very large in memory. Having 10's of 1000's of these means lots of memory. One hopefully easy way to save memory here without going to the hassle of redoing Cabal's type definitions is simply to increase sharing. There's a huge amount of repeated information. Start by sharing all the package names and versions. Then there's other meta-data that rarely changes between versions of the same package. This kind of thing should be easy to evaluate, just write a test prog that reads the index file and look at peak memory use. Then try sharing stuff and see how much it drops. This sharing optimisation would still be useful even if later we go and redo GenericPackageDescription to be more compact. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Trac import is complete
On 23 May 2012 23:14, Bryan O'Sullivan b...@serpentine.com wrote: You can now find open issues here: https://github.com/haskell/cabal/issues?state=open Haven't had time to look at it yet (on the road atm) but I just wanted to say a big thank you Bryan for all the hard work! Duncan or Andres, could one of you please chmod -w the files owned by Trac, and add a note pointing people to github for filing bugs? Will do when I get back (unless Andres beats me to it) Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: [Haskell-cafe] Announce: Hackager - new version of hackage-test
Good work David. I used to do something like this for Cabal regression testing but the method I used didn't scale well as hackage grew. I'll look into using your tool next time for testing a major Cabal / cabal-install release. Duncan On 23 April 2012 21:37, David Terei dave.te...@gmail.com wrote: Hi all, I've updated the old hackage-test tool and renamed to hackager. http://hackage.haskell.org/package/hackager Hackager is a tool to automate the compiling of all packages on Hackage. It builds each package on hackage in isolation and records the results. The purpose being to catch regressions caused by changes to GHC (and Cabal although this was not the motivation). Two runs of Hackager can be compared, so the first run is done with a known version of GHC and the next run with a new, experimental version of GHC... ect. The improvements to Hackager over hackage-test are: * Parallelized the build process. Can now specify how many packages to build in parallel, which cuts total run time down greatly (e.g 2 days - 5 hours) * hackage-test and hackage-report are now one tool, 'hackager' that works as a mutli-command tool. * Proper option handling * Fixed some stability issues The new homepage for development can be found here: https://github.com/dterei/Hackager Cheers, David ___ Haskell-Cafe mailing list haskell-c...@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
patch applied (cabal): Fix doc comment for ghcOptSourcePathClear and 11 others
Wed Mar 28 20:08:42 PDT 2012 Duncan Coutts dun...@community.haskell.org * Fix doc comment for ghcOptSourcePathClear Spotted by tibbe M ./Cabal/Distribution/Simple/Program/GHC.hs -1 +2 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20120329030842-5c91e-ac9056beede12f84766d832f0b00e466eab2a942.gz Tue Jul 26 15:29:23 PDT 2011 Thomas Tuegel ttue...@gmail.com * New detailed test suite interface. This patch implements a new interface for detailed test suites based on Duncan's proposal. This implementation differs from his in a few ways: * The constructors of 'Tests' have been renamed. I think 'TestGroup' and 'ExtraTestOptions' are redundant: it is clear from context what sort of Group and what sort of ExtraOptions are being considered and qualified imports can resolve any name conflicts. Group and ExtraOptions have the advantage of improving the brevity of pattern matches on Tests, which are used very often in D.S.Test. * The 'concurrentSafe :: Bool' field of TestInstance has become the 'concurrently' field of Group, allowing package and test framework authors greater control over concurrency. * The 'Finished' constructor of 'Progress' now contains the options used to run the test in addition to the test result. Without returning the options, it would be difficult to extract the RNG seed used to run a test. * A detailed test suite module now exports the symbol 'test :: IO Tests'. This enables the use of IO to enumerate the tests in a group, suggested by Duncan as a way to accomodate the GHC test suite. M ./Cabal/Distribution/Simple/Test.hs -72 +152 M ./Cabal/Distribution/TestSuite.hs -259 +68 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20110726222923-30370-7d42433d44a2b2c3fb88e8a955c76ac5e83a903e.gz Fri Jul 29 08:02:38 PDT 2011 Thomas Tuegel ttue...@gmail.com * Renamed Distribution.TestSuite.Tests to Test. M ./Cabal/Distribution/Simple/Test.hs -4 +4 M ./Cabal/Distribution/TestSuite.hs -4 +4 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20110729150238-30370-a2fc53aa388d175e80faec05723c6e36b2c2b298.gz Fri Jul 29 08:10:01 PDT 2011 Thomas Tuegel ttue...@gmail.com * Removed obsolete LANGUAGE pragma from Distribution.TestSuite. M ./Cabal/Distribution/TestSuite.hs -1 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20110729151001-30370-feefa70db2cb7fce720a0ce73402f0af51884a5a.gz Fri Jul 29 08:42:49 PDT 2011 Thomas Tuegel ttue...@gmail.com * Cleaned for warnings in D.TestSuite and D.S.Test. M ./Cabal/Distribution/Simple/Test.hs -4 +4 M ./Cabal/Distribution/TestSuite.hs -1 +1 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20110729154249-30370-5e7c74f65fce8eb08b408862c1a3894b14dd887c.gz Fri Jul 29 08:48:12 PDT 2011 Thomas Tuegel ttue...@gmail.com * Removed 'optionStringDescription' from OptionType. M ./Cabal/Distribution/TestSuite.hs -1 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20110729154812-30370-915f671e6b9cd23d2286090b69dbcc91432bc061.gz Thu Sep 1 11:26:06 PDT 2011 Thomas Tuegel ttue...@gmail.com * Changed detailed test exported type to [Test]. M ./Cabal/Distribution/Simple/Test.hs -21 +23 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20110901182606-30370-52b1f4413c9e96c4f951d7b483ad8058a6a46a89.gz Tue Sep 6 14:16:11 PDT 2011 Thomas Tuegel ttue...@gmail.com * Clean D.S.Test for unused symbols. The unused declarations were all related to replaying test suites with logged options, but the command-line option for this feature has been disabled for some time. Changing the detailed test suite type to expect tests :: IO [Test] instead of tests :: IO Test made the old method of replaying options clumsy. Since it was already disabled, I chose to remove it, rather than rewrite it again. M ./Cabal/Distribution/Simple/Test.hs -45 +1 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20110906211611-30370-b69b92b2a6d5070add303626eeeac3a9927f2e0f.gz Tue Sep 6 14:55:57 PDT 2011 Thomas Tuegel ttue...@gmail.com * Removed Options from Finished. M ./Cabal/Distribution/Simple/Test.hs -3 +8 M ./Cabal/Distribution/TestSuite.hs -4 +1 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20110906215557-30370-45c2e98be628e5fb739cf536873576fa40430cae.gz Tue Sep 6 14:56:00 PDT 2011 Thomas Tuegel ttue...@gmail.com * Improved documentation for 'concurrently' field of Test. M ./Cabal/Distribution/TestSuite.hs -3 +8 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20110906215600-30370
Re: Cabal's detailed test interface
On 7 September 2011 20:14, Thomas Tuegel ttue...@gmail.com wrote: Greetings once again! I have attached an updated patch with the requested changes to the test interface. (I apologize for the delay; it was caused by the unfortunate conspiracy of moving, not having internet access, and qualifying exams. All of these being done, I will be more responsive on Cabal-related things.) Here is a summary of the patches, with some commentary: I just noticed I'd not applied these yet. Sorry for the delay. It looks good, now applied. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: cabal patch to improve cabal update behavior.
On Tue, 2012-04-17 at 10:32 +0300, Eyal Lotem wrote: I built a preliminary prototype of cabal update enhancements: A) Using a Lazy I/O bytestring with the HTTP download, so that it can do a streaming-decompress on-the-fly (This gives a 30% speedup on my machine) That's interesting. I'm surprised it's that much. So you're decompressing and writing the decompressed .tar to disk? But not unpacking the .tar on the fly right? Is that 30% figure just of the time to write the uncompressed file to disk, or also of the download and unpack? Or are we just talking about the index file here? I guess I typically assume that it makes sense to write the compressed file to disk, and decompresses and unpack on the fly, less disk I/O. That said, for the index we do keep the uncompressed version on disk, so doing the decompression on the fly there would make sense. B) Using the bytestring-progress library to show a useful progress report on the whole process. Unfortunately, due to the PVP, the version constraints are (as usual) over-restrictive, so they need to be relaxed. Until the fix is put on hackage, you have to install bytestring-progress from https://github.com/Peaker/bytestring-progress Interesting. I'm a bit sceptical of it's subtle side effects. I wonder if we can do it in a slightly nicer way. Actually, something that would help is the new interfaces in the zlib binding I've been working on. That'd let you push blocks in one at a time and keep track of the original length, and calculate progress. C) It's quickdirty (Proof of concept), because: C.1) No version constraints on bytestring-progress C.2) Using hSetBuffering on stdout and ANSI terminal codes, which may be problematic in different kinds of terminals C.3) The only way to disable it is setting verbosity=quiet. It probably deserves its own option I hope it's a useful demonstration of the utility of this, though. It makes cabal update something I'd dread a little less to run :) Yes, thanks. BTW, the other way to make that better is to do an incremental index update, and so simply download much less data. If you're interested in hacking on that, lemme know. Also, I was wondering why there's no monad for the passing of Verbosity around to so many functions. Then I could easily add more options that go around to control things like whether to show progress bars... There should be, but you actually want rather more than that otherwise the saving of the verbosity params would be offset by rather a lot of liftIO $ ... What it needs is a shell monad of some kind. I've done a few test designs. It's a major overhaul, but should be done. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
patch applied (cabal-branches/cabal-1.14): Note in the README that you can install using the cabal program and 27 others
Fri Nov 11 05:31:13 PST 2011 Duncan Coutts dun...@community.haskell.org * Note in the README that you can install using the cabal program M ./Cabal/README +11 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal-branches/cabal-1.14;a=darcs_commitdiff;h=2011133113-5c91e-7c31c10875b1e6a21dc1942ac06e12f214fdc0f4.gz Wed Nov 16 16:17:19 PST 2011 Duncan Coutts dun...@community.haskell.org * Now require mtl-2 M ./cabal-install/cabal-install.cabal -1 +1 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal-branches/cabal-1.14;a=darcs_commitdiff;h=2017001719-5c91e-918376fa4b3ec78700ae6a23a650cfc4fe4f51a0.gz Fri Dec 16 06:43:55 PST 2011 Andres Loeh and...@well-typed.com * removing datatype contexts M ./cabal-install/Distribution/Client/Dependency/TopDown/Constraints.hs -2 +1 M ./cabal-install/Distribution/Client/PackageIndex.hs -1 +1 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal-branches/cabal-1.14;a=darcs_commitdiff;h=20111216144355-99ab6-334a83855b850ab4f7eefcee1e1bd43823c3d35d.gz Sun Jan 1 07:32:35 PST 2012 Andres Loeh and...@well-typed.com * removing trailing whitespace M ./cabal-install/Distribution/Client/Dependency/TopDown/Constraints.hs -7 +7 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal-branches/cabal-1.14;a=darcs_commitdiff;h=20120101153235-99ab6-36f99168af28a027d75a343b9101217ca93d0c8e.gz Mon Jan 9 00:09:07 PST 2012 Andres Loeh and...@well-typed.com * Added a comment. M ./cabal-install/Distribution/Client/Dependency/Modular/Solver.hs -2 +1 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal-branches/cabal-1.14;a=darcs_commitdiff;h=20120109080907-99ab6-21b8b4065226ce5a7ab810c26066c4fe26f4662f.gz Tue Jan 10 12:03:56 PST 2012 Andres Loeh and...@well-typed.com * fix compilation with ghc-7.4 Thanks to Tuncer Ayaz. M ./cabal-install/Distribution/Client/Dependency/Modular/Dependency.hs -1 +1 M ./cabal-install/Distribution/Client/Dependency/Modular/Log.hs -1 +2 M ./cabal-install/cabal-install.cabal -5 +5 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal-branches/cabal-1.14;a=darcs_commitdiff;h=20120110200356-99ab6-2cf0e3708254856d4a5b489fa478d3e54158ae88.gz Tue Jan 10 12:30:53 PST 2012 Andres Loeh and...@well-typed.com * updating dependencies in bootstrap.sh Thanks to Tuncer Ayaz. M ./cabal-install/bootstrap.sh -7 +13 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal-branches/cabal-1.14;a=darcs_commitdiff;h=20120110203053-99ab6-d8d2af9861892f82ca3c9848c63684b7a50296e8.gz Tue Feb 7 11:39:58 PST 2012 Thomas Tuegel ttue...@gmail.com * Handle test and benchmark dependencies through the resolver properly. Previously, test and benchmark dependencies were handled by editing the package description to include or exclude those stanzas before running the dependency resolver. Test and benchmark dependencies could only be installed for source packages because no package description is available for named packages before dependency resolution. Now, test and benchmark stanzas are enabled or disabled through constraints passed to the dependency resolver. This way, we can install dependencies for the test suites of target packages without propagating '--enable-tests' through the entire dependency tree; i.e., tests and benchmarks, when enabled, are built only for target packages. Later, this will allow us to automatically run test suites and, e.g., install only upon their success. M ./cabal-install/Distribution/Client/BuildReports/Anonymous.hs -1 +1 M ./cabal-install/Distribution/Client/BuildReports/Storage.hs -2 +2 M ./cabal-install/Distribution/Client/Configure.hs -4 +6 M ./cabal-install/Distribution/Client/Dependency/TopDown.hs -14 +22 M ./cabal-install/Distribution/Client/Dependency/TopDown/Types.hs -3 +5 M ./cabal-install/Distribution/Client/Dependency/Types.hs -1 +2 M ./cabal-install/Distribution/Client/Fetch.hs -1 +1 M ./cabal-install/Distribution/Client/Install.hs -31 +19 M ./cabal-install/Distribution/Client/InstallPlan.hs -4 +3 M ./cabal-install/Distribution/Client/InstallSymlink.hs -3 +3 M ./cabal-install/Distribution/Client/Targets.hs +1 M ./cabal-install/Distribution/Client/Types.hs -3 +19 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal-branches/cabal-1.14;a=darcs_commitdiff;h=20120207193958-30370-364975df3ecb31370856850a3fd166388b3b1046.gz Tue Feb 7 11:45:43 PST 2012 Thomas Tuegel ttue...@gmail.com * Update types in modular dependency solver to compile with new test/benchmark dependency constraints. M ./cabal-install/Distribution/Client/Dependency/Modular/Assignment.hs -3 +6 M ./cabal-install/Distribution/Client/Dependency/Modular/Configured.hs -1 +2 M ./cabal-install/Distribution/Client/Dependency/Modular/ConfiguredConversion.hs
patch applied (cabal): Add brief description of PVP to cabal init generated .cabal files
Sat Sep 24 22:18:15 PDT 2011 Duncan Coutts dun...@community.haskell.org * Add brief description of PVP to cabal init generated .cabal files M ./cabal-install/Distribution/Client/Init.hs -2 +13 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20110925051815-5c91e-cea42fa6abbb5a8eaf931d3f169eba89be536125.gz ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
patch applied (cabal-branches/cabal-1.14): Add brief description of PVP to cabal init generated .cabal files
Sat Sep 24 22:18:15 PDT 2011 Duncan Coutts dun...@community.haskell.org * Add brief description of PVP to cabal init generated .cabal files M ./cabal-install/Distribution/Client/Init.hs -2 +13 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal-branches/cabal-1.14;a=darcs_commitdiff;h=20110925051815-5c91e-cea42fa6abbb5a8eaf931d3f169eba89be536125.gz ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
patch applied (cabal): Convert ghc option construction to use an options record and 7 others
Tue Mar 27 15:52:05 PDT 2012 Duncan Coutts dun...@community.haskell.org
* Convert ghc option construction to use an options record
Using a structured representation of the ghc options rather
than [String] makes it easier to share and reuse the code
for building sets of ghc options, which makes it easier to
add code for calling ghci.
This is based on Sam Anklesaria's GSoC work on cabal repl
M ./Cabal/Cabal.cabal +1
M ./Cabal/Distribution/Simple/GHC.hs -182 +170
M ./Cabal/Distribution/Simple/Haddock.hs -42 +50
A ./Cabal/Distribution/Simple/Program/GHC.hs
View patch online:
http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20120327225205-5c91e-8630eadaf1e697e9186273e6af3d8495fe5bb368.gz
Tue Mar 27 16:32:03 PDT 2012 Duncan Coutts dun...@community.haskell.org
* Break up the component build code a bit
This will make it easier to allow building individual components
M ./Cabal/Distribution/Simple/Build.hs -119 +130
M ./Cabal/Distribution/Simple/LocalBuildInfo.hs +5
View patch online:
http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20120327233203-5c91e-105b034c011c31bb6c8430271f6b4469774e3fd9.gz
Wed Mar 28 14:45:22 PDT 2012 Duncan Coutts dun...@community.haskell.org
* Check for the env var GHC_PACKAGE_PATH
Based on the patch by Keegan McAllister mcallister.kee...@gmail.com
Fixes ticket #335
M ./Cabal/Distribution/Simple/GHC.hs +15
View patch online:
http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20120328214522-5c91e-0f5c4182e6553cd2e518a3375afeababf9d3cb4e.gz
Wed Mar 28 15:05:56 PDT 2012 Duncan Coutts dun...@community.haskell.org
* Fix ticket #731
M ./cabal-install/Distribution/Client/Configure.hs -9 +12
View patch online:
http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20120328220556-5c91e-7854de20600d8b0ca9e11b197875250ae3ff.gz
Wed Mar 28 18:08:37 PDT 2012 Duncan Coutts dun...@community.haskell.org
* Add back deprecated ghcVerbosityOptions and ghcPackageDbOptions
M ./Cabal/Distribution/Simple/GHC.hs +22
View patch online:
http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20120329010837-5c91e-bb2b1de8aa2eda993c83cba04245df90bb327aec.gz
Wed Mar 28 18:46:53 PDT 2012 Duncan Coutts dun...@community.haskell.org
* head cabal-install requires head Cabal
M ./cabal-install/cabal-install.cabal -1 +1
View patch online:
http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20120329014653-5c91e-6fe2258fec400ba162d938d85ed0e528b6f5318c.gz
Wed Mar 28 19:07:20 PDT 2012 Duncan Coutts dun...@community.haskell.org
* Extend the --package-db= feature to allow multiple dbs and start from empty
Previously only one --package-db flag was used (later ones overrode earlier).
Internally we can handle stacks of package dbs, now we expose that fully in
the user interface.
--global sets the initial db stack to [GlobalPackageDB]
--user sets the initial db stack to [GlobalPackageDB, UserPackageDB]
--package-db=clear clears the db stack to []
--package-db=global pushes GlobalPackageDB on the top of the stack
--package-db=user pushes UserPackageDB on the top of the stack
--package-db=${file} pushes SpecificPackageDB ${file} on the top of the stack
So --global and --user just affect the initial db stack, and this can be
extended by the --package-db flag, or reset using --package-db=clear.
Note that although this lets users tell cabal to avoid using the global
package db, we cannot yet stop ghc from looking at the global package db.
See http://hackage.haskell.org/trac/ghc/ticket/5977
Fixes feature request #437.
M ./Cabal/Distribution/Simple/Configure.hs -13 +21
M ./Cabal/Distribution/Simple/GHC.hs -1 +6
M ./Cabal/Distribution/Simple/Setup.hs -9 +20
View patch online:
http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20120329020720-5c91e-bc21b89d0fa7629049b36746a7a7becfd39dcb0a.gz
Wed Mar 28 19:07:52 PDT 2012 Duncan Coutts dun...@community.haskell.org
* Use the new --package-db flag stuff in cabal-install
M ./cabal-install/Main.hs -18 +4
View patch online:
http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20120329020752-5c91e-84dff75cc0f68b7f630f6e19f97150d269c6a9e1.gz
___
cabal-devel mailing list
cabal-devel@haskell.org
http://www.haskell.org/mailman/listinfo/cabal-devel
patch applied (cabal-branches/cabal-1.14): Fix ticket #731
Wed Mar 28 15:05:56 PDT 2012 Duncan Coutts dun...@community.haskell.org * Fix ticket #731 M ./cabal-install/Distribution/Client/Configure.hs -9 +12 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal-branches/cabal-1.14;a=darcs_commitdiff;h=20120328220556-5c91e-7854de20600d8b0ca9e11b197875250ae3ff.gz ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
patch applied (cabal): Fix for #917: Generated Paths module broken with NoImplicitPrelude
Sat Mar 17 13:28:58 PDT 2012 Duncan Coutts dun...@community.haskell.org * Fix for #917: Generated Paths module broken with NoImplicitPrelude import Prelude explicitly M ./Cabal/Distribution/Simple/Build/PathsModule.hs -1 +2 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20120317202858-5c91e-ea47614738e6254722252c4534b0695d0544a419.gz ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: cabal: HOME: does not exist
Hi Jason, You may need to hack the code I'm afraid. It gets the home dir as part of reading the configuration file. Look for where it uses getAppUserDataDirectory. In particular it's used indirectly in baseSavedConfig, though it should mostly get overridden if the config file is found. So additionally you'd need to specify a cabal config file on the command line to avoid it looking for ~/.cabal/config Let us know how it goes, we can integrate changes you make. Duncan On 15 March 2012 05:25, Jason Dusek jason.du...@gmail.com wrote: I am trying to do an automated install of Cabal packages, on EC2 using Ubuntu's cloud-config. The error above was caught in my logs. Cloud-config sets up a minimal environment prior to running any tasks. All the package managers I've worked with so far -- gem, npm, apt -- have no problem with this. It would be nice to find a way to turn off Cabal's user-centric behaviour. -- Jason Dusek pgp // solidsnack // C1EBC57DC55144F35460C8DF1FD4C6C1FED18A2B ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: install with '--upgrade-dependencies' requires '--force-reinstall'
On 11 February 2012 18:03, Antoine Latter aslat...@gmail.com wrote: I'm using a development cut of cabal install from a few weeks ago, so this may not be relevant anymore (I'll upgrade soon to try the changes to --enable-tests and install soon), but I found this behavior odd: When using 'cabal(-dev) install --only-dependecies --solver=modular --upgrade-dependencies' to test my package with new dependencies, I am then told I need to use the flag '--force-reinstalls'. Shouldn't this be implied by '--upgrade-dependencies'? We've not entirely decided what to do with the reinstall check. My preference for the behaviour is that it should check if any packages will be broken by the reinstall (not including the ones being installed themselves). I think this should do the right thing in most cases. Opinions welcome. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Updates of hackage-server and data loss
On 8 February 2012 16:58, Erik Hesselink hessel...@gmail.com wrote: On Mon, Feb 6, 2012 at 19:07, Duncan Coutts duncan.cou...@googlemail.com wrote: On 6 February 2012 15:22, Erik Hesselink hessel...@gmail.com wrote: Hi all, We run hackage-server internally at Silk. Every time we update to a new version, it is unable to read our old data. Is the acid-state being versioned correctly? Or is this expected behavior? It is very annoying for us. If this is expected, is there a way to work around it? Our approach to versioning is not via acid-state by by a database backup/restore method. We export all canonical data to a tarball of text and other standard format files. The reason we do this is because we don't trust that the binary data format is sufficient for long term preservation of all the data. Because we are using dump/restore, we make no attempt to use the acid-state versioning scheme (it's plausible we might do so in future as an extra, but probably not until there are actual releases). The backu[/restore is done via the commands 'hackage-server backup' and 'hackage-server restore'. It is also possible to generate backups while the server is running by sending a USR1 signal (kill -USR1 $the_pid). Note also that testing of the backup/restore method is not quite complete, but the testing framework for it is in place I think. I actually tried the backup at some point, and got the following output: hackage-server: Initializing happstack-state... hackage-server: Server data loaded into memory hackage-server: Preparing export tarball hackage-server: Saving export tarball hackage-server: Shutting down... hackage-server: /home/erik/state/: inappropriate type This was with the old code though, I haven't tried with the new code yet. Ok, like I said it's not fully tested yet, so you may hit some problems that route too, but that's the approach we intend to make work. I just tried sending the signal to the running process, and got: hackage-server: Writing checkpoint... hackage-server: Done There's no backup to be found anywhere, though. Where should it create the backup? Oh, sorry, I was getting myself confused. Yes, that just writes the acid-state checkpoint, not the backup tarball. We can in principle also do online backup snapshots but we don't have a UI for that yet. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Virtual packages
On 2 December 2011 07:19, Edward Z. Yang ezy...@mit.edu wrote: Hello folks, I was thinking a little about how build flags that modify functionality are harmful for dependency resolution (pandoc and xmobar, I'm glaring at you), but authors will still use this feature because it's a lot easier than maintaining a bunch of separate packages for each different flag. Yes, when designing the flags mechanism we explicitly decided not to allow depending on package +/- flags because this is impossible to translate into native distro packages. Therefore, packages are not allowed to change their interface based on flags. If they do so anyway, that's their own fault. We could however consider trying to enforce this more strictly. It suddenly struck me, then, that what we actually needed was a mechanism not unlike what you see in traditional package managers, where a single Cabal file can specify multiple packages. pandoc.cabal might define 'pandoc' and 'pandoc-highlighting', and pandoc-highlighting can have different build dependencies, modules, etc. Module writers still need to arrange their APIs so that the extra functionality should be separable, but I don't see this as being too much of a problem. Mix-and-matching flags can be achieved by simply picking the extra virtual models as dependencies as necessary. What do you think? How is that different from multiple packages? Do we just need better tools for working with multiple related packages? Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Virtual packages
On 2 December 2011 12:21, Chris Dornan ch...@chrisdornan.com wrote: Indeed it looks like a mechanism for making multiple packages easier to manage -- rather like RPM provides (from which the terminology is borrowed I think). It sounds like a really good idea and surely the best way of removing the pressure on providing customization through flags. So what are we looking for exactly? The use case I was thinking of previously was something like gtk2hs which is made up of a bunch of packages and during development you typically want to build all of them together. Do we need multiple .cabal files in one dir sharing the same Setup.hs? Or just a local environment which links to each package so you can build them by name easily? Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
patch applied (cabal): When running haddock, only pass package deps for the component only and 2 others
Wed Nov 16 11:11:39 PST 2011 Duncan Coutts dun...@community.haskell.org * When running haddock, only pass package deps for the component only rather than for the union of all components in the package. M ./Cabal/Distribution/Simple/Haddock.hs -23 +30 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=2016191139-5c91e-bd39fb265f7611cd6d5befeb29115a4cb60b724f.gz Wed Nov 16 11:14:55 PST 2011 Duncan Coutts dun...@community.haskell.org * Include testsuites and benchmarks in external deps when enabled In particular this should fix ticket #871 where the cabal_macros.h is missing info for testsuite components. M ./Cabal/Distribution/Simple/LocalBuildInfo.hs +2 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=2016191455-5c91e-f8a0a5a0c267d238615c3cc0ae233d588c2f942d.gz Wed Nov 16 16:17:19 PST 2011 Duncan Coutts dun...@community.haskell.org * Now require mtl-2 M ./cabal-install/cabal-install.cabal -1 +1 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=2017001719-5c91e-918376fa4b3ec78700ae6a23a650cfc4fe4f51a0.gz ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
patch applied (cabal): Note in the README that you can install using the cabal program
Fri Nov 11 05:31:13 PST 2011 Duncan Coutts dun...@community.haskell.org * Note in the README that you can install using the cabal program M ./Cabal/README +11 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=2011133113-5c91e-7c31c10875b1e6a21dc1942ac06e12f214fdc0f4.gz ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
patch applied (cabal): Fix fallout of Num / Show class changes in ghc-7.4
Thu Nov 3 17:25:10 PDT 2011 Duncan Coutts dun...@community.haskell.org * Fix fallout of Num / Show class changes in ghc-7.4 M ./cabal-install/Distribution/Client/Tar.hs -1 +1 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=2004002510-5c91e-92810a90e395c140984ba7f1e454e4da10f7a08d.gz ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
patch applied (cabal): Bump major version to 1.13
Wed Oct 26 13:43:46 PDT 2011 Duncan Coutts dun...@community.haskell.org * Bump major version to 1.13 Since 1.12 got released with ghc-7.2.x M ./Cabal/Cabal.cabal -1 +1 M ./Cabal/Makefile -1 +1 M ./cabal-install/cabal-install.cabal -2 +2 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20111026204346-5c91e-f15b146fcfb70a0c0402b56a9a5ca109875b9a2d.gz ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
patch applied (cabal): Allow Haddock to be configured from the 'install' command and 1 others
Sat Oct 15 10:01:01 PDT 2011 Max Bolingbroke batterseapo...@hotmail.com * Allow Haddock to be configured from the 'install' command M ./cabal-install/Distribution/Client/Install.hs -12 +14 M ./cabal-install/Distribution/Client/Setup.hs -10 +28 M ./cabal-install/Main.hs -7 +7 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20111015170101-9b787-1f2a1bc3608cc5b29573360f9c8a1a9a3f2f9f0e.gz Sun Oct 16 04:08:52 PDT 2011 Max Bolingbroke batterseapo...@hotmail.com * Add a (substituted) flag to allow configuration of Haddock's --use-contents flag M ./Cabal/Distribution/Simple/Haddock.hs -6 +14 M ./Cabal/Distribution/Simple/InstallDirs.hs +1 M ./Cabal/Distribution/Simple/Setup.hs +11 M ./cabal-install/Distribution/Client/Setup.hs -1 +2 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20111016110852-9b787-e008e5d6e1cf87409e2ec4f71cfe3cb1377ddf69.gz ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
patch applied (cabal): Uploading build reports shouldn't fail if there are no reports
Sun Oct 16 07:38:19 PDT 2011 Max Bolingbroke batterseapo...@hotmail.com * Uploading build reports shouldn't fail if there are no reports M ./cabal-install/Distribution/Client/Upload.hs -11 +14 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20111016143819-9b787-a32f1b435905be00d42ec4b7799fb8e828977f3e.gz ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
patch applied (cabal): Add package checks for benchmarks and 4 others
Wed Oct 12 13:16:04 PDT 2011 Johan Tibell johan.tib...@gmail.com * Add package checks for benchmarks Refactor duplicate names check to avoid having to manually write all O(n^2) possible collision cases between executables, test suites, and benchmarks. M ./Cabal/Distribution/PackageDescription/Check.hs -20 +56 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20111012201604-73595-165c451e997023fc438e37db97b621b0359f4f65.gz Thu Oct 13 15:56:15 PDT 2011 Johan Tibell johan.tib...@gmail.com * Implement 'cabal bench' command The only implement benchmark interface so far is exitcode-stdio-1.0, which forwards the output of the benchmark executable being run to the parent process' stdout/stderr. M ./Cabal/Cabal.cabal +1 M ./Cabal/Distribution/Simple.hs -2 +28 A ./Cabal/Distribution/Simple/Bench.hs M ./Cabal/Distribution/Simple/InstallDirs.hs -1 +4 M ./Cabal/Distribution/Simple/Setup.hs +62 M ./Cabal/Distribution/Simple/UserHooks.hs -3 +14 M ./Cabal/Distribution/Simple/Utils.hs +10 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20111013225615-73595-c92fcd5da380aa88d6597515406d641712994d47.gz Thu Oct 13 16:21:09 PDT 2011 Johan Tibell johan.tib...@gmail.com * Add unit test for 'cabal bench' command-line flags A ./Cabal/tests/PackageTests/BenchmarkOptions/ A ./Cabal/tests/PackageTests/BenchmarkOptions/BenchmarkOptions.cabal A ./Cabal/tests/PackageTests/BenchmarkOptions/Check.hs A ./Cabal/tests/PackageTests/BenchmarkOptions/Setup.hs A ./Cabal/tests/PackageTests/BenchmarkOptions/test-BenchmarkOptions.hs M ./Cabal/tests/PackageTests/PackageTester.hs -1 +15 M ./Cabal/tests/suite.hs -2 +4 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20111013232109-73595-6ab8db41c38aaa13771cf973221d9f0cae3c21d2.gz Thu Oct 13 16:28:47 PDT 2011 Johan Tibell johan.tib...@gmail.com * Fail gracefully if 'cabal bench' is run before 'cabal build' M ./Cabal/Distribution/Simple/Bench.hs +7 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20111013232847-73595-b05ad5f69c791b90f3c26409bc1950df06090101.gz Sun Oct 23 14:39:24 PDT 2011 Duncan Coutts dun...@community.haskell.org * Bump versions of Cabal and cabal-install Latest cabal-install requires latest Cabal due to api addition M ./Cabal/Cabal.cabal -1 +1 M ./cabal-install/cabal-install.cabal -2 +2 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20111023213924-5c91e-2884510f9eb24976527ff8e975e0417ff488d912.gz ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
patch applied (cabal): Add a source package index cache to speed up reading
Sun Oct 23 14:32:53 PDT 2011 Duncan Coutts dun...@community.haskell.org * Add a source package index cache to speed up reading e.g. about 3x faster for cabal info pkgname M ./cabal-install/Distribution/Client/IndexUtils.hs -51 +252 M ./cabal-install/Distribution/Client/Update.hs -1 +2 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20111023213253-5c91e-b47cb652458ae668e6ac89d8609270e40c811b3c.gz ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
patch applied (cabal): Fix source repo subdir name after cabal-Cabal dir rename
Sun Oct 23 14:44:25 PDT 2011 Duncan Coutts dun...@community.haskell.org * Fix source repo subdir name after cabal-Cabal dir rename M ./Cabal/Cabal.cabal -1 +1 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20111023214425-5c91e-343cf0e18ec97f2472005107ec43f3957ac1be1f.gz ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Mirroring package uploader/upload date
On Mon, 2011-10-17 at 10:09 +0100, Max Bolingbroke wrote: Fellow members of the shadowy Cabal, I've been looking at making the mirror client supply the original uploading user/upload time when it mirrors a package. I've developed an approach (attached) that, rather than PUTting a simple tarball to mirror, PUTs a combination of the tarball, user name and upload date in multipart/form-data format. This works (though the code is a bit grungy still). The only thing I'm worrying about before I tidy it up and commit is whether it is sufficiently RESTful: it feels a bit weird that the thing that you PUT is not what you get back from making a GET request to that URL. Yeah, I had the same reaction when considering this previously. An alternative approach would be to expose resources for the uploader/upload time of a package, which the mirror client could then simply PUT to. Yes. I think that is the right thing to do. In principle it's hardly any more expensive (given pipelined http requests) and it's a nicer design. There's some other issues to consider. How does the mirror client pick the user account and make sure it exists. This turns into the more general problem of mapping user accounts between domains. Sadly I don't think there is one policy that fits all circumstances. It depends on what the user knows about the relationship between the servers they're mirroring between. If no policy is given, probably a reasonably default is to not set any uploader account and to just set the upload time (probably that means we should have the uploader account be Nothing rather than set to the mirroring client). The new server identifies user accounts by id. User names are permitted to change, but the userid remains the same (like unix accounts). It exposes both the user id and name in the package index. I'm not sure if we currently expose the set of users and names in some other useful way (ie a single resource providing a name - uid mapping). The old server identifies users only by name. For mirroring from the old server I think the sensible policy is: * take the username from the old server * look it up in the user db on the new server * if it exists, assume these are corresponding accounts * if it does not exist, create a new disabled user account with that name * set the chosen account as the package uploader Another policy that would work for mirroring between new server implementations is to assume the user ids match, or to make use of a supplied uid mapping table. If we don't find a corresponding account and don't want to use a policy of creating disabled accounts then it's probably best to use no uploader name, and just set the upload time. Initially I think we only need the null policy of setting upload time only, and the policy useful for a live public mirror of the central hackage server. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
HEADS UP, REBASING: Re: patch applied (/home/srv/code/hackage-server): First go at adding continuous/live mirroring and 8 others
Sorry everyone, I've decided to back out these patches because it'll make it much easier to merge Max's buildbot patches. Rebasing Max's patches on top of mine is proving a little tricky. So I'm going to rebase the other way round. So if you pulled this weekend, watch out, you'll want to obliterate any patches that you've got that are not upstream, use: darcs push --dry-run if you see any patches (that are not your own local ones you're working on), then obliterate them with darcs obliterate. Duncan On Sun, 2011-10-09 at 01:16 +0100, dun...@lun.haskell.org wrote: Thu Oct 6 11:29:45 BST 2011 Duncan Coutts dun...@well-typed.com * First go at adding continuous/live mirroring Ignore-this: 69423437dcf02091da86ae3480bc128d M ./MirrorClient.hs -32 +105 M ./hackage-server.cabal -1 +1 Thu Oct 6 12:09:20 BST 2011 Duncan Coutts dun...@well-typed.com * Improve the mirror client help messages Ignore-this: 30d7c7730b11b88871fa6a4df3d79014 M ./MirrorClient.hs -5 +35 Sat Oct 8 15:30:24 BST 2011 Duncan Coutts dun...@community.haskell.org * Rework the async var to batch updates and amortise the work Ignore-this: 3f53d68d36f1847b061dcb7ddcb3e23d Previously, every single write to the var was laboriously evaluated and since jobs keep comming in faster than they can be evaluated, then the input queue just grows without bound. But if we're just writing then if we have several new inputs in the queue then we can just skip to the final update and evaluate that one. That's now what we do, with a slight complication to do with incremental updates. M ./Distribution/Server/Util/AsyncVar.hs -16 +34 M ./hackage-server.cabal +1 Sat Oct 8 16:55:51 BST 2011 Duncan Coutts dun...@well-typed.com * Overhaughl the mirror client error handling infrastructure and strategy Ignore-this: 59347decddab335ba0f9f75fd66d3d0f M ./MirrorClient.hs -127 +443 Sat Oct 8 17:01:55 BST 2011 Duncan Coutts dun...@well-typed.com * Only rewrite the mirror client error files if they've changed Ignore-this: b82b608bfcd9a6d2e4473bb616ab563c M ./MirrorClient.hs -2 +4 Sat Oct 8 19:20:12 BST 2011 Duncan Coutts dun...@community.haskell.org * Disable idle GC for the server Ignore-this: 970125139572aee660c2dae06f904464 It doesn't correctly detect when we're really idle and ends up using ~15% CPU when the server is otherwise idle. M ./hackage-server.cabal +2 Sat Oct 8 19:22:16 BST 2011 Duncan Coutts dun...@well-typed.com * Depend on the latest HTTP lib, for bug fixes Ignore-this: a5f411822c140575974829df3e5eacec M ./MirrorClient.hs -3 M ./hackage-server.cabal -1 +1 Sun Oct 9 00:43:15 BST 2011 Duncan Coutts dun...@well-typed.com * Report when a checkpoint is finished writing. Ignore-this: 2f451e601c3d8fb9c1bb45e34d55f0eb It's just that little bit more reassuring. M ./Main.hs +1 Sun Oct 9 01:12:09 BST 2011 Duncan Coutts dun...@well-typed.com * Handle IO errors in mirroring without losing package error info Ignore-this: 718a385e3718d7cc12a6312109648dfc Previously if we got an IO exception, rather than an HTTP level error, then we had no chance to trap it locally so we would lose all the accumulated info about the packages with errors. Now we cheat. Instead of using a proper state monad, we use a reader monad with an IORef that way, we can catch the IO exception in the monad run function, and still read out the latest package error state. Also handle user interrupt ^C similarly. M ./MirrorClient.hs -15 +33 ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
patch applied (cabal): GHC 7.2+ no longer generates _stub.o files
Sat Sep 10 12:53:29 PDT 2011 Duncan Coutts dun...@community.haskell.org * GHC 7.2+ no longer generates _stub.o files So stop looking for them. This could otherwise cause problems if one switches ghc version without cleaning the build dir since we'll pick up the old _stub.o files and end up with duplicate linker symbols. M ./cabal/Distribution/Simple/GHC.hs -3 +7 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20110910195329-5c91e-0172ff56a091947088044f0a9003a111f7d87946.gz ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
patch applied (cabal): Use the configured proxy even for uploading build reports
Wed Sep 28 14:08:59 PDT 2011 Max Bolingbroke batterseapo...@hotmail.com * Use the configured proxy even for uploading build reports M ./cabal-install/Distribution/Client/BuildReports/Upload.hs -3 +1 M ./cabal-install/Distribution/Client/HttpUtils.hs -12 +20 M ./cabal-install/Distribution/Client/Upload.hs -13 +5 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20110928210859-9b787-aa311a80e57a343c26c6b6d0aeb947a311d22cbd.gz ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Documentation building client for cabal-install
On Wed, 2011-09-28 at 22:26 +0100, Max Bolingbroke wrote: Hi Hackagers, I've written a rough-and-ready documentation building service for the Hackage 2.0 effort. It is functional enough to submit build reports and build documentation for a few example packages on my local Hackage instance. Duncan: please review+commit! I also had to make some small changes to cabal-install that should go into the Cabal repo. Ok, all committed now except for one of the cabal-install patches. I did some rebasing of some patches, so you'll want to blow away versions of the patches you sent me, and pull from upstream. You can check first by doing a darcs push --dry-run. Blow away anything local that looks suspicious. Despite your efforts, the mirroring client still doesn't share much infrastructure with the build bot client. In the last few days I've changed all the http get/put/post and http error handling code in the mirror client to make it robust to various kinds of errors. I'm sure that much of it can be shared though. I've not yet applied the cabal-install patch that makes the haddock options available via cabal install. I'm not sure about making them available directly as install commands since it's not clear from context what e.g. cabal install --css means, plus we're likely to get name clashes. I'll have a think about alternative approaches. Alternatively, could the build bot just run each phase separately? As in cabal install --only-dependencies; cabal configure; cabal build; cabal haddock; cabal-install --only Solving the install/haddock command line UI issue would be good, but perhaps we don't have to right now... Or maybe it's better to be forced to find a solution now. :-) Anyway, all good progress. And I've been talking to people about potential public hosting for a testing server. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Hackers hacking on hackage: introductions
Hi all, I'm very pleased to say that recently we've had a number of developers express an interest in helping with the new hackage server, some at the recent hackathons (HacPDX-II, CamHac) and some after my plea for help at the recent Haskell Symposium. I've tried to encourage everyone who's contacted me with some interest to get onto this mailing list [1] and to talk to each other (not just to me). By having everyone aware of each other and talking to each other we can give each other encouragement and advice, code review etc. [1]: http://www.haskell.org/mailman/listinfo/cabal-devel I'll quickly outline some resources for getting started and then do some quick introductions to let everyone know who's indicated they're interested in looking at hackage. Hacker resources Firstly, the wiki is getting into reasonable shape: http://hackage.haskell.org/trac/hackage/wiki/HackageDB/2.0 It has a quickstart hacking guide with links to the darcs repo. The first thing is to try getting it working on your machine then check the TODO file and the todo lists on the wiki. Our initial priority is things needed to get a public server running in parallel with live mirroring. I can give more details on this. People working on hackage - Since I'm introducing people, I'll start with myself... I'm not really the maintainer of the new hackage-server, but since nobody else is yet then I sort-of get stuck with that role. The current design of the new codebase is partly my fault (but in a good way, I think it's a reasonable design). I supervised Matt Gruen last year on his GSoC project who did most of the actual hard work of coding. I have actually been hacking on the codebase myself in recent months, partly at the various hackathons. My involvement is almost entirely in a volunteer capacity, though currently I'm doing a little work on the mirroring side of things for one of the members of the IHG. Matthew Gruen as I mentioned above did his GSoC project on the hackage-server last summer, so the much of the existing code is due to him, in particular the implementation of the resource and dynamic path stuff, plus many of the Features.* modules. Matt has occasionally been able to help out in the last year, in particular he gave us lots of help during the HacPDX-II hackathon a couple months ago. Max Bolingbroke accosted me after my brief talk at the Haskell Symposium and asked what needed to be done to get the new hackage stuff going. As you can see he's already sent in patches for a new documentation build bot. The new server design involves the server being mostly a passive store of data and calls for specialised clients to do things like building documentation and uploading it. Max doesn't know it yet, but he's already got a commit bit on the source repo. I guess he knows now. David Terei also grabbed me at the Haskell Symposium and is keen to help. David Laing contacted me today and is looking for a bigger project than his previous personal side projects. So please help answer his questions and point him in the right direction. And if you want a helper, he might be the person to ask Erik Hesselink is running a hackage server instance internally at his company and is keen to get it more feature complete. Ben Millwood was at CamHac and wrote a number of patches. Stefan Wehr was also at CamHac and did some work on providing access to package changelog files within the package tarballs (via serving the contents of the package tarball). I don't know if Stefan has time to finish this off, but it's a great feature to have. Thomas DuBuisson organised the HacPDX-II hackathon and was keen to have the new hackage server be a focal point of the hackathon. In fact this is what really got the ball rolling again. Thomas also wrote some patches to clean up the package tags feature. David Lazar was also at HacPDX-II and made a number of cleanup patches and fixed some things to do with package deprecation. Jeremy Shaw took part in HacPDX-II and sent us a couple big patches to bring us up to the latest happstack version and switch us from happstack-state to the newer nicer acid-state. Jeremy is a good person to ask about happstack issues. Antoine Latter has made several contributions over the last couple years, including various tricky issues with HTTP. Conrad Parker is keen to see the new server get going. He also wants to deploy it at his company. He's got lots of design ideas for how it should work, and the relation between the package repositories and the cabal clients. Last, but not least, is of course Ross Paterson. Ross wrote the current hackage-scripts which is behind the current hackage.haskell.org site and he has maintained and improved it over the last five years. While people like me have been fussing over new server design ideas, Ross' site has actually been working and has coped with growth from a handful to thousands of packages and hoards of users. No doubt I'm
Re: Proposal: Make it possibly specify the component to build form the command line
On Tue, 2011-10-04 at 19:27 -0700, Johan Tibell wrote: Hi, Here's a another proposal. Now when we start having more components in one package (library, executables, test suites, and soon benchmarks) it would be convient to specify what to build directly from the command line without having to reconfigure e.g. cabal build my-test This would also allow one to enable all components e.g. cabal configure --enable-tests --enable-benchmarks but only build one of them (for faster build times) cabal build my-benchmark Would that work well with how Cabal keeps track of what to build? Yeah, I hope to add this feature once I've integrated the cabal ghci patches. For cabal ghci you need to be able to specify the component you want to load, and we discussed extending this to cover build too. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Proposal: Add a benchmark section to .cabal files
On Wed, 2011-10-05 at 12:09 +0200, Tillmann Rendel wrote: Hi, Johan Tibell wrote: [...] I'm convinced we need support for building benchmark using Cabal. [...] Could test sections be used for benchmarks, too? Step 1. Add basic support for building simple benchmark suites e.g. modelled after test suite sections: This should work immediately. Step 2. Add a detailed benchmark interface that allows the benchmark to associate a real valued *measure* (optionally with upper and lower bounds) with a benchmark name. This could be supported by a protocol where a test reports a real instead of a boolean value. Test bench-foo type: benchmark-1.0 main-is:bench-foo.hs build-depends: base Except for the more specialized keywords, there seems to be no need to distinguish tests from benchmarks. Hmm, that's an interesting idea, to just make benchmarks a different testsuite protocol. Certainly we could prototype it like this. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Proposal: Add a benchmark section to .cabal files
On Mon, 2011-10-03 at 22:59 -0700, Johan Tibell wrote: Proposal Step 1. Add basic support for building simple benchmark suites e.g. modelled after test suite sections: Name: foo Version:1.0 License:BSD3 Cabal-Version: = 1.9.2 Build-Type: Simple Benchmark bench-foo type: stdio-1.0 main-is:bench-foo.hs build-depends: base This doesn't add much on top of normal executable sections, except that it tells Cabal that this isn't an executable it should build by default or install. Just to point out that the closest workaround at the moment is to add an executable but to mark it as not buildable by default. Step 2. Add a detailed benchmark interface that allows the benchmark to associate a real valued *measure* (optionally with upper and lower bounds) with a benchmark name. Benchmark naming would work like test naming (e.g. group/subgroup/name). This would allow us to track performance over time. Step 2 needs to be fleshed out some more, but I think we can get quite a lot of bang-for-the-buck with step 1. In general, yes I think benchmarking is a sensible follow-on from test suites. We can follow the same approach of specifying a protocol so that we can add more sophisticated interfaces later as we gain experience wit them. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Documentation building client for cabal-install
On Wed, 2011-09-28 at 22:26 +0100, Max Bolingbroke wrote: Hi Hackagers, I've written a rough-and-ready documentation building service for the Hackage 2.0 effort. It is functional enough to submit build reports and build documentation for a few example packages on my local Hackage instance. Duncan: please review+commit! I also had to make some small changes to cabal-install that should go into the Cabal repo. Cool, thanks. That was quick! I only just got back from Tokyo. Also got your note about the patch to avoid applying. Comments/suggestions welcome. There is one major known flaw in that the invocation of cabal install performed by the builder is not sandboxed. So arbitrary user code runs with the same privileges as the documentation builder itself. I guess the best solution to this is a chroot (to restrict the writeable part of the file system)+timeout? Yeah, I think this is probably best solved by sys admin stuff rather than trying to build anything sophisticated into the client itself. Someone setting this up on a system would want to use some kind of sandbox, either a vm, a chroot, SELinux policy or whatever else their system provides. We should probably just provide a big fat warning and pointers to documentation on various sandbox/container technologies. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Hackage 2.0: JSON resources?
On Wed, 2011-09-28 at 14:23 +0100, Max Bolingbroke wrote: On 27 September 2011 13:15, Max Bolingbroke batterseapo...@hotmail.com wrote: it would be nicer to expose a machine-readable resource like http://localhost:8080/package/process.json that I could retrieve and extract the relevant information from. I've noticed that the particular data I'm after is available by GETting a URL like http://127.0.0.1:8080/package/array-0.3.0.1/doc/ and testing whether the result is a 404 or not. So this issue is no longer a blocker for me, though JSON resources would be useful more generally to other Hackage clients. Right, in this case doing a GET like that is the simplest RESTful thing I think. In general however we're quite happy for the new server to provide as much info as possible in convenient formats, be that text, xml, json or whatever. You mentioned the json feature. I don't think that's the best way to partition things. That approach suggests having one jsonFeature that serves json format resources for everything. Both xml and json (and any other native format) are fine to be included in each feature, and I think that's much more sensible than trying to separate it out into its own feature. The only reason we have a separate html feature is that nice html for a human website is simply not modular. Whereas for serving resources for non-human clients, it's perfectly modular to provide lots of different, linked, resources in various formats. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
RE: [Hackage] #282: profiling versions of libraries not managed well
On Fri, 2011-09-30 at 08:21 -0500, Chris Dornan wrote: Ooops -- I meant to start another thread. I have found the --package-db flag which Cabal uses. It would be really useful if Cabal could also use an environment variable (e.g., CABAL_PACKAGE_DB) for the purpose. We don't use the GHC_PACKAGE_PATH. You ought to be able to set a global --package-db flag by putting it in the ~/.cabal/config. I don't plan to add any equivalent env var, but we're likely to add support for local cabal config files. What kind of use case are you thinking of? Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
patch applied (cabal): Use a PVP-style version as the default for cabal init
Sat Sep 24 19:17:22 PDT 2011 Duncan Coutts dun...@community.haskell.org * Use a PVP-style version as the default for cabal init M ./cabal-install/Distribution/Client/Init.hs -3 +3 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20110925021722-5c91e-5a67a10fb7e9f18ac636704b2fed43ec8363a0ab.gz ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
patch applied (cabal): Filter autogenerated modules from test suite build info when doing sdist.
Fri Sep 23 13:18:06 PDT 2011 Thomas Tuegel ttue...@gmail.com * Filter autogenerated modules from test suite build info when doing sdist. M ./cabal/Distribution/Simple/SrcDist.hs -1 +3 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=20110923201806-30370-becca78dae5b45ae96b3e978670bfb7e5fb64d2a.gz ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
Re: Cabal's detailed test interface
On Tue, 2011-09-06 at 11:16 +0200, José Pedro Magalhães wrote: Hi Duncan, Is there a planned date for a release which will have the detailed test interface? My hope is that it'll be ready in time to be included in a release of Cabal to go along with GHC 7.4 towards the end of the year. If you'd like to help move things along then you could help us try out the new proposed interface. Thomas posted the latest patches yesterday. Duncan ___ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
