Re: [Haskell-cafe] Cabal dependency hell
Duncan Coutts wrote: On Sun, 2010-04-11 at 18:43 +0200, Maciej Piechotka wrote: - Privacy problem. I don't want the software to call home with data without asking. Obviously it is important that the data be anonymous and that we do not send stuff without the user's knowledge. While there is not any directly identifying information in the existing anonymous build reports, one has to be very careful with how much access the server provides to the reports or it may become possible to infer identifying information. One possibility for mitigating the issues here is to have cabal present the entire message to the user for scrubbing prior to being submitted,[1] similar to how version control systems generally provide a summary of the patch (albeit uneditable) when asking for a patch description. That poses other problems (e.g., reports which are too incomplete to be helpful or which are intentionally erroneous), and doesn't cover everything (e.g., taking advantage of outside knowledge that Duncan is one of the few users on Sparc/Linux), but it helps to solve the declassification problem (i.e., what data the user is willing to reveal to the server). [1] Ideally in a way which allows scripting the scrubbing so folks can just specify preferences once. If we wanted to keep things simple for the implementors, then hooking into $EDITOR and assuming folks know how to script their favorite editor is one approach. Otherwise we'll want a (E)DSL that can be specified in config files. -- Live well, ~wren ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Cabal dependency hell
Ketil Malde wrote: Perhaps it would also be possible to suggest library upgrades likely to remedy the problem in case of a build failure? +1 for good error messages. +2 for should I try upgrading libfoo? [yn] integration (if configurable as AlwaysYes, AlwaysAsk, or AlwaysNo). -- Live well, ~wren ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Cabal dependency hell
On Sun, 2010-04-11 at 18:43 +0200, Maciej Piechotka wrote: * Build reporting in the hackage server The idea here is that cabal sends back anonymous reports to the server to say if a package compiled or not, and against what versions of dependencies. This would make it clearer to maintainers if their dependency versions are correct. Additionally I think it would be useful for hackage to provide tweaked .cabal files for packages with updated constraints, even without new version uploads. We are proposing a GSoC project which would cover some of this. Hmm. I guess there are 2 problems: - Privacy problem. I don't want the software to call home with data without asking. Obviously it is important that the data be anonymous and that we do not send stuff without the user's knowledge. While there is not any directly identifying information in the existing anonymous build reports, one has to be very careful with how much access the server provides to the reports or it may become possible to infer identifying information. For example you might happen to know that I'm one of the few users on Sparc/Linux. Another possible attack is to try to discover timing of uploaded reports by the attacker inserting their own reports at regular intervals. If the server provided direct access to reports, in order of upload time, then this would give away timing of uploads. Similarly, if all reports were kept together then one could identify bunches of packages uploaded at the same time and infer they are from the same user. So yes, it needs careful thought. Perhaps we cannot give clients access to the raw reports at all. Perhaps it's enough to separate reports by package and to delay publishing new ones for a day or two, and to randomise the order when they are made available. - Spoofing problem. Someone can feed wrong data easily without chance of being discovered (for example to prove how grate 'haxor' he is). I don't think that's a major problem. One has to treat the data with that possibility in mind. There's also the option for authenticated non-anoymous reports with build logs. Duncan ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Cabal dependency hell
On Sun, 2010-04-11 at 14:28 -0700, Jason Dagit wrote: I've noticed another type of diamond dependency problem. Suppose I build and install Foo today and it depends on Bar 2.0.0. In a week, I install Bar 2.0.1. Next I installed Baz that also depends on Bar, and it gets Bar 2.0.1. When I install a package that depends on Foo, Baz, and possibly Bar, then cabal won't be able to figure out proper dependencies because Foo needs one version of Bar and Baz needs a different version. That's exactly the problem that cabal-install solves. It will rebuild one or the other of Foo and Baz so that the dependency on Bar is consistent. But if all these packages followed the PVP, then either version of Bar could have been used, if it was used consistently. Yes, but one would still need to be rebuilt. i'm not sure what is the best solution to this problem. Perhaps in this scenario, cabal should ask the user if it should rebuild Foo and/or Baz so they use the same Bar. I think this could lead to a lot of rebuilding, and that rebuilding gets tedious when you're doing it manually. So it happens automatically at the moment. It tries to minimise the amount of rebuilding. I occasionally hear complaints that it is rebuilding things when users did not expect it to. Duncan ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Cabal dependency hell
Duncan Coutts duncan.cou...@googlemail.com writes: Here are a few things which I would like to see implemented that would help all this: * Build reporting in the hackage server The idea here is that cabal sends back anonymous reports to the server to say if a package compiled or not, and against what versions of dependencies. This would make it clearer to maintainers if their dependency versions are correct. It could also be checked before compilation. This would provide an incentive for the user to contribute, since she feels she's already getting something tangible back. And we've already asserted that the network is available. Perhaps it would also be possible to suggest library upgrades likely to remedy the problem in case of a build failure? -k -- If I haven't seen further, it is by standing in the footprints of giants ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Cabal dependency hell
On Mon, Apr 12, 2010 at 4:00 AM, Duncan Coutts duncan.cou...@googlemail.com wrote: On Sun, 2010-04-11 at 14:28 -0700, Jason Dagit wrote: I've noticed another type of diamond dependency problem. Suppose I build and install Foo today and it depends on Bar 2.0.0. In a week, I install Bar 2.0.1. Next I installed Baz that also depends on Bar, and it gets Bar 2.0.1. When I install a package that depends on Foo, Baz, and possibly Bar, then cabal won't be able to figure out proper dependencies because Foo needs one version of Bar and Baz needs a different version. That's exactly the problem that cabal-install solves. It will rebuild one or the other of Foo and Baz so that the dependency on Bar is consistent. Weird. This has not been my experience. Jason ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Cabal dependency hell
On Mon, 2010-03-08 at 13:33 +, Maciej Piechotka wrote: While I love Haskell it's packaging system have some problems - especially with parsec. Currently I'm not able to install an practically anything using cabal due to version mismatches (or at least packages linking to both version of parsec). I found the following problems in various cabal packages (the examples are only examples - please do not be offended): - The constraints are too loose. It is written that package works with '0.6 1' but in 0.8 the API has been changed (example of tagsoup hxt. Fixed hxt versions depends on ghc 6.12... and ghc 6.10 if you change 2 lines in cabal file) - The constraints are too tight. It is written that package works with 'parsec 3' but it can run with 3.0 and 3.1 (a lot of packages) Here are a few things which I would like to see implemented that would help all this: * Build reporting in the hackage server The idea here is that cabal sends back anonymous reports to the server to say if a package compiled or not, and against what versions of dependencies. This would make it clearer to maintainers if their dependency versions are correct. Additionally I think it would be useful for hackage to provide tweaked .cabal files for packages with updated constraints, even without new version uploads. We are proposing a GSoC project which would cover some of this. * A package API tool and greater use of the PVP We do have a package versioning policy that maintainers can choose to follow. This helps users of the package write more accurate dependency version constraints. What we lack is a tool to help maintainers check that they are correctly following the PVP. We also have a GSoC proposal for a package tool. * Private build-depends This would help the parsec 2 vs parsec 3 issue. Most packages that depend on parsec, or QuickCheck, do not export (as part of their public API) functions that use types from the parsec/QC packages. That is, the use of those packages is entirely encapsulated and invisible to clients. In this case, diamond dependency problems are impossible and it would be ok to use different versions of the same package within the same dependency graph. Currently cabal does not know which build-depends are public and which are private. The extension would be to let package authors mark some depends as private (though this would also need to be checked). * Improved dependency resolution algorithm in cabal-install The constraint solver is currently not very smart. It does not do so well with tight version constraints, which is exactly what we will get more of, as people use the PVP more. The resolver could be improved to do significantly better in these situations. I have an algorithm in my head, but not much time to implement it. Volunteers welcome on all tasks. Duncan ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Cabal dependency hell
* Build reporting in the hackage server The idea here is that cabal sends back anonymous reports to the server to say if a package compiled or not, and against what versions of dependencies. This would make it clearer to maintainers if their dependency versions are correct. Additionally I think it would be useful for hackage to provide tweaked .cabal files for packages with updated constraints, even without new version uploads. We are proposing a GSoC project which would cover some of this. Hmm. I guess there are 2 problems: - Privacy problem. I don't want the software to call home with data without asking. - Spoofing problem. Someone can feed wrong data easily without chance of being discovered (for example to prove how grate 'haxor' he is). Regards signature.asc Description: This is a digitally signed message part ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Cabal dependency hell
Am Sonntag 11 April 2010 18:43:30 schrieb Maciej Piechotka: * Build reporting in the hackage server The idea here is that cabal sends back anonymous reports to the server to say if a package compiled or not, and against what versions of dependencies. This would make it clearer to maintainers if their dependency versions are correct. Additionally I think it would be useful for hackage to provide tweaked .cabal files for packages with updated constraints, even without new version uploads. We are proposing a GSoC project which would cover some of this. Hmm. I guess there are 2 problems: - Privacy problem. I don't want the software to call home with data without asking. I think in the default cabal config, build-reporting would be turned off and you'd have to turn it on yourself. Turning it on by default wouldn't be sensible. - Spoofing problem. Someone can feed wrong data easily without chance of being discovered (for example to prove how grate 'haxor' he is). Regards ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Cabal dependency hell
Excerpts from Daniel Fischer's message of Sun Apr 11 16:20:30 -0400 2010: I think in the default cabal config, build-reporting would be turned off and you'd have to turn it on yourself. Turning it on by default wouldn't be sensible. I doubt you'd get very much runtime with that. I'd suggest prompting the user to submit a failed build report if the build fails. Cheers, Edward ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Cabal dependency hell
On Sun, Apr 11, 2010 at 9:26 AM, Duncan Coutts duncan.cou...@googlemail.com wrote: On Mon, 2010-03-08 at 13:33 +, Maciej Piechotka wrote: While I love Haskell it's packaging system have some problems - especially with parsec. Currently I'm not able to install an practically anything using cabal due to version mismatches (or at least packages linking to both version of parsec). I found the following problems in various cabal packages (the examples are only examples - please do not be offended): - The constraints are too loose. It is written that package works with '0.6 1' but in 0.8 the API has been changed (example of tagsoup hxt. Fixed hxt versions depends on ghc 6.12... and ghc 6.10 if you change 2 lines in cabal file) - The constraints are too tight. It is written that package works with 'parsec 3' but it can run with 3.0 and 3.1 (a lot of packages) Here are a few things which I would like to see implemented that would help all this: * Build reporting in the hackage server The idea here is that cabal sends back anonymous reports to the server to say if a package compiled or not, and against what versions of dependencies. This would make it clearer to maintainers if their dependency versions are correct. Additionally I think it would be useful for hackage to provide tweaked .cabal files for packages with updated constraints, even without new version uploads. We are proposing a GSoC project which would cover some of this. * A package API tool and greater use of the PVP We do have a package versioning policy that maintainers can choose to follow. This helps users of the package write more accurate dependency version constraints. What we lack is a tool to help maintainers check that they are correctly following the PVP. We also have a GSoC proposal for a package tool. * Private build-depends This would help the parsec 2 vs parsec 3 issue. Most packages that depend on parsec, or QuickCheck, do not export (as part of their public API) functions that use types from the parsec/QC packages. That is, the use of those packages is entirely encapsulated and invisible to clients. In this case, diamond dependency problems are impossible and it would be ok to use different versions of the same package within the same dependency graph. Currently cabal does not know which build-depends are public and which are private. The extension would be to let package authors mark some depends as private (though this would also need to be checked). * Improved dependency resolution algorithm in cabal-install The constraint solver is currently not very smart. It does not do so well with tight version constraints, which is exactly what we will get more of, as people use the PVP more. The resolver could be improved to do significantly better in these situations. I have an algorithm in my head, but not much time to implement it. I've noticed another type of diamond dependency problem. Suppose I build and install Foo today and it depends on Bar 2.0.0. In a week, I install Bar 2.0.1. Next I installed Baz that also depends on Bar, and it gets Bar 2.0.1. When I install a package that depends on Foo, Baz, and possibly Bar, then cabal won't be able to figure out proper dependencies because Foo needs one version of Bar and Baz needs a different version. But if all these packages followed the PVP, then either version of Bar could have been used, if it was used consistently. i'm not sure what is the best solution to this problem. Perhaps in this scenario, cabal should ask the user if it should rebuild Foo and/or Baz so they use the same Bar. I think this could lead to a lot of rebuilding, and that rebuilding gets tedious when you're doing it manually. Jason ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Cabal dependency hell
Edward Z. Yang ezy...@mit.edu writes: I doubt you'd get very much runtime with that. I'd suggest prompting the user to submit a failed build report if the build fails. Exactly like how Windows keeps prompting you to allow it to send an error report to Microsoft? I don't know about you, but I always found it irritating... -- Ivan Lazar Miljenovic ivan.miljeno...@gmail.com IvanMiljenovic.wordpress.com ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Cabal dependency hell
Excerpts from Ivan Lazar Miljenovic's message of Sun Apr 11 17:37:15 -0400 2010: Edward Z. Yang ezy...@mit.edu writes: I doubt you'd get very much runtime with that. I'd suggest prompting the user to submit a failed build report if the build fails. Exactly like how Windows keeps prompting you to allow it to send an error report to Microsoft? I don't know about you, but I always found it irritating... I think the primary irritation is that Microsoft gets all of these error reports and they disappear into the great black VOID. Some public statistics might help alleviate the irritation. Cheers, Edward ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Cabal dependency hell
Hello Maciej, Monday, March 8, 2010, 4:33:08 PM, you wrote: PS. I understand that content may be flame-gen. I am sorry in advance if such circumstances happen. However I believe that possible improvements in process are worth the risk. i was the author of this idea and i thought that 1) package author should allow to use only versions of dependence that he know will work 2) if new version of dependence arrives, package author should upload minor update of his package that allows to use it -- Best regards, Bulatmailto:bulat.zigans...@gmail.com ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Cabal dependency hell
Hi Maciej, that's why I started hack-nix. You can patch dependencies easily. However you have to install the Nix package manager. It also works with lates versions only because the dependency solves is written in Nix itself. Which package is causing trouble to you? We can't expect package maintainers to test everything. So it must be people like you and me who fixes those changes. The way to go is make hackage allow changing constraints on the fly notifying the author that he can update his repository. This will work in most cases. Bumping versions because a dependency has changed is bad as well. This will cause to much overhead (and it dosen't solve the problem because the old package is still wrong). Specifying dependencies must be decoupled from bumping versions. It's because dep specs do depend on the world which can change.. At least that's what I think. If you're interested in Nix and hack-nix I can show you how everything works using an SSH session. Marc Weber ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
