Re: [Haskell-cafe] Quick Angel User's Survey
You can use cgroups on linux to ensure that everything is shut down. See systemd. Alexander On Sep 14, 2013 9:21 PM, Michael Xavier mich...@michaelxavier.net wrote: Hey Cafe, I am the maintainer of Angel, the process monitoring daemon. Angel's job is to start a configured set of processes and restart them when they go away. I was responding to a ticket and realized that the correct functionality is not obvious in one case, so I figured I'd ask the stakeholders: people who use Angel. From what I know, most people who use Angel are Haskellers so this seemed like the place. When Angel is terminated, it tries to cleanly shut down any processes it is monitoring. It also shuts down processes that it spawned when they are removed from the config and the config is reloaded via the HUP signal. It uses terminateProcess from System.Process which sends a SIGTERM to the program on *nix systems. The trouble is that SIGTERM can be intercepted and a process can still fail to shut down. Currently Angel issues the SIGTERM and hopes for the best. It also cleans pidfiles if there were any, which may send a misleading message. There are a couple of routes I could take: 1. Leave it how it is. Leave it to the user to make sure stubborn processes go away. I don't like this solution so much as it makes Angel harder to reason about from a user's perspective. 2. Send a TERM signal then wait for a certain number of seconds, then send an uninterruptable signal like SIGKILL. There are some caveats with #2. I think I'd prefer the timeout to be configurable per-process. I think I'd also prefer that if no timeout is specified, we assume the user does not want us to use a SIGKILL. SIGKILL can be very dangerous for some processes like databases. I want explicit user permission to do something like this. If Angel generated a pidfile for the process, if it should only be cleaned if Angel can confirm the process is dead. Otherwise they should be left so the user can handle it. So the real question: is the extra burden of an optional configuration flag per process worth this feature? Are my assumptions about path #2 reasonable. Thanks for your feedback! -- Michael Xavier http://www.michaelxavier.net ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ANNOUNCE: MFlow 3.0
Here are some common-lisp web frameworks using continuations: http://common-lisp.net/project/cl-weblocks/ http://common-lisp.net/project/ucw/features.html What always worried me with these frameworks is how they could be made robust in case of failures. Storing all state in a database backend often makes it possible to isolate failures. However, it seems to me that it is be possible to solve this in Haskell where state can be serialized and synchronized between multiple machines using Cloud Haskell, something that is error-prone or impossible in other languages. But that step has never been taken. Alexander On Wed, Jul 10, 2013 at 4:48 PM, Alberto G. Corona agocor...@gmail.comwrote: Thanks Adrian. The racket people where pioneers in this idea I think. There is another web framework in Ocaml, Osigen that it is also continuation based. MFlow is not continuation-based but it also define the navigation as a sequence. But only Seaside (and now MFlow) supports many flows in the same page. See for example this: [PDF] *Seaside* – A *Multiple* Control *Flow* Web Application Frameworkhttp://www.google.es/url?sa=trct=jq=esrc=sfrm=1source=webcd=3cad=rjaved=0CEwQFjACurl=http%3A%2F%2Fscg.unibe.ch%2Farchive%2Fpapers%2FDuca04eSeaside.pdfei=WnPdUYHPI-Ly7Aaa0oHQDAusg=AFQjCNFxwsFQc9QsQCNPRFfD4t6ReQtP5gsig2=_klwtzxIKW5UzAOUDOvFCw There is also other: Apache Coccoon that run in a special kind of JavaScript. The continuation-based frameworks have the reputation of storing a lot of application state and to be non scalable. MFlow uses backtracking and It does not have these problems. 2013/7/10 Adrian May adrian.alexander@gmail.com Oh how nice! I have been looking at MFlow a lot lately and I think it's got something quite special that Yesod, Happstack, etc don't seem to have, at least, not as far as I know. I mean, look at this: sumWidget= pageFlow sum $ do n1 - p Enter first number ++ getInt Nothing ** submitButton enter ++ br n2 - p Enter second number ++ getInt Nothing ** submitButton enter ++ br n3 - p Enter third number ++ getInt Nothing ** submitButton enter ++ br p (The result is: ++show (n1 + n2 + n3)) ++ wlink () b menu ++ p you can change the numbers in the boxes to see how the result changes Is that pretty or what? That's the code for this: http://mflowdemo.herokuapp.com/noscript/fviewmonad To me that's a real technological step over and above the usual servlets paradigm and I'd love to see more people getting involved. It seems like Yesod and Happstack have a lot more manpower behind them, but unless I've missed something, MFlow is going somewhere new and should be helped along. Adrian. PS. Besides Seaside, Racket is playing with the same ideas. They (Jay McCarthy) have something to say about performance but I didn't quite understand it. On 10 July 2013 06:41, Alberto G. Corona agocor...@gmail.com wrote: The third version of MFlow is out. http://hackage.haskell.org/package/MFlow MFlow is an all-heterodox web application framework, but very haskellish. Now MFlow support restful URLs. It is the first stateful web framework to my knowledge that supports it. The type safe routes are implicitly expressed as normal monadic code within a navigation monad. The application look as a normal imperative console application, but the navigation monad goes back and forth to match the path of the URL. The user has control of the state, that can roll-back or not when the navigation goes back depending on the application needs. The state is in the form of normal Haskell variables In a monadic computation, with the weird addition of backtracking. The menu of the application below is implemented as an imperative-like syntax, but the application navigate forward and backward to synchronize with the requests of the web browser: http://mflowdemo.herokuapp.com/ This version support in-page flows. What is that? look at this example: http://mflowdemo.herokuapp.com/noscript/fviewmonad These flows are implemented as formlets with a monad instance, and callbacks which change the look. I call them widgets: http://haskell-web.blogspot.com.es/2013/06/the-promising-land-of-monadic-formlets.html Each page may have many of these active widgets, each one running their own flow. These widgets refresh themselves trough Ajax if they are enclosed in the primitive autoRefresh. If there is no Ajax or JavaScript available, they gracefully degrade by refreshing the entire page: http://mflowdemo.herokuapp.com/noscript/combination http://haskell-web.blogspot.com.es/2013/06/and-finally-widget-auto-refreshing.html The page flows and the multiflow idea was inspired in Seasidehttp://www.seaside.st/, a great Smalltalk web framework and adapted to the pure recursive nature of Haskell and the formlets. It also support some JQuery widgets integrated: modal and not modal dialogs,
Re: [Haskell-cafe] Markdown extension for Haddock as a GSoC project
I see the pluggable markup being pushed in this thread again. I just want to remind everybody that we currently have a flavor of a markup issue on github. The ghc source code uses literal haskell, and it does not work well on github. http://www.haskell.org/pipermail/ghc-devs/2013-April/001099.html Any markup that is not widely supported makes it harder for third parties to support and parse. The solution is *not* to reimplement github in haskell, but to standardize markup as much as possible. Pluggable markup makes the probability that a github-like service, IDEs and similar can make use of the documentation arbitrarily close to zero. Alexander On Mon, Apr 29, 2013 at 8:04 AM, Richard A. O'Keefe o...@cs.otago.ac.nzwrote: I should add that as a consumer of Haddock documentation I can testify that fancier styling (in whatever format) would be of little benefit to _me_. What I need is more plain text and more examples. To be perfectly honest, most of the time when looking at a Haddock page, I end up clicking on the Source button because there are things I need to know that are in the source but not the documentation. So I do agree that markup that doesn't get in the way of a _reader_ who is looking at the source code is an excellent thing. I say this as someone who had to read some Java today and ended up stuffing it through a comment stripper so that I could easily find what I needed to find. This thread is not about the visually lightweight aspect of Markdown. That's a good thing. No argument there. The thread is about how well documented the notation should be. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] GSoC proposal: Haskell AST-based refactoring and API upgrading tool
There is another aspect to this: How do you get maintainers to apply the patches? How should hackage be changed to accomodate large-scale refactorings? There was a discussion on this mailing list related to build regressions on GHC 7.6 last year. All of the regressions could be fixed using perl regexps, and it was only a few hours of work, much less than the work involved in the discussion itself. I downloaded all of hackage and did the fixes using perl. http://www.haskell.org/pipermail/haskell-cafe/2012-August/103155.html However, without the community infrastructure to actually apply the patches, the problem is not solved. I think this is mainly a community/organizational issue. Refactoring is not really the problem, but of course better refactoring abilities are good. Alexander On Mon, Apr 29, 2013 at 9:59 AM, Niklas Hambüchen m...@nh2.me wrote: Hello Malcolm, no, I had indeed not seen this! Thanks for the link. It goes very much in the direction I was thinking of, but it does not seem to maintained and does not cabal install either. It also seems very much focused on interactive editor integration as compared to written-out transformations. Do you know to what extent they have built and a modification-friendly AST? Also, do you know if the people involved in this are still active in the community and interested in working further in this direction? Thanks Niklas On 29/04/13 15:36, Malcolm Wallace wrote: On 29 Apr 2013, at 07:00, Niklas Hambüchen wrote: I would like to propose the development of source code refactoring tool that operates on Haskell source code ASTs and lets you formulate rewrite rules written in Haskell. Seen this? http://www.haskell.org/haskellwiki/HaRe Regards, Malcolm ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Fwd: How to do automatic reinstall of all dependencies?
This is not what you asked for, but reinstalling *all
dependencies*probably isn't such a good idea, because ultimately some
dependencies are
shipped with GHC and you might not be able to reinstall them.
Here is a useful formula I developed to avoid cabal-hell and always *upgrade
* dependencies whenever a new package is installed.
$ cabal install --upgrade-dependencies `eval echo $(ghc-global-constraints
)` package-name
What this does is fix the version of all global packages. These are by
default the ones that are shipped with ghc, so the above command explicitly
excludes those from being upgraded.
The ghc-global-constraints function is something I have in my .bashrc file,
and it looks like this:
function ghc-global-constraints() {
ghc-pkg list --global | tail -n+2 | head -n-1 | grep -v '(' | while
read a; do
VER=${a##*-}
PKG=${a%-*}
echo -n --constraint='$PKG==$VER'
done
}
This technique depends on actually fixing broken package dependencies, but
today that's usually just a github fork away, and often easier than dealing
with multiple cabal-dev installations IMO.
Alexander
On Thu, Apr 25, 2013 at 12:29 AM, capn.fre...@gmail.com
capn.fre...@gmail.com wrote:
-db
- Forwarded message -
From: Captain Freako capn.fre...@gmail.com
Date: Tue, Apr 23, 2013 9:21 pm
Subject: How to do automatic reinstall of all dependencies?
To: haskell-cafe@haskell.org
Hi all,
Does anyone know why the following is not working, as an automatic way of
reinstalling all dependencies?:
dbanas@dbanas-lap:~/prj/AMI-Tool$ cabal install --only-dependencies
--reinstall --force-reinstalls parsec
Resolving dependencies...
All the requested packages are already installed:
Use --reinstall if you want to reinstall anyway.
dbanas@dbanas-lap:~/prj/AMI-Tool$
Thanks,
-db
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] GSoC Project Proposal: Markdown support for Haddock
+1 for concistency.
Also, consider interop with non-haskell environments. For example showing
the documentation of a function in emacs, eclipse, on github, and from a
javascript library.
All of these can be engineered around, and tooling can be provided.
But let me give an example: the other week I was looking for a command-line
tool to extract javadoc to display as contextual information in emacs.
There is no such tool. Javadoc is java only. For me, if I could not
hack it up in an hour, it was too much work. The solution was rather to
craft a specific google search, use I'm feeling lucky, and extract the
subsection containing the documentation for the function.
Often the most useful format for documentation is contextual help in an
IDE/editor, so don't forget that use-case.
Alexander
On Sat, Apr 6, 2013 at 1:04 AM, John MacFarlane j...@berkeley.edu wrote:
I like markdown and use it all the time. While I acknowledge the
problems that have been pointed out, markdown has the advantage of being
easily readable as it is in the source document, and not looking like
markup.
But I do want to point out one problem with markdown as a format for
documentation in Haskell files. Consider:
module MyModule
{-
# Introduction
This is my module
-}
where
import System.Environment
main = getArgs = print
Now try to compile with -cpp, and you'll get an error because of the '#'
in column 1. '#' in column 1 is common in markdown (and even
indispensible for level 3+ headers).
One could work around this by disallowing level 3+ headers, by allowing
the headers to be indented, or by introducing new setext-like syntax for
level 3+ headers, but it is a problem for the idea of using a markdown
SUPERset.
John
+++ dag.odenh...@gmail.com [Apr 05 13 21:59 ]:
I forgot the mention the craziness with the *significant trailing
whitespace*.
On Fri, Apr 5, 2013 at 9:49 PM, [1]dag.odenh...@gmail.com
[2]dag.odenh...@gmail.com wrote:
Personally I think Markdown sucks, although perhaps less than Haddock
markup.
Still:
* No document meta data
* No code block meta data like language for syntax highlighting
* No tables
* No footnotes
* HTML fallback is insecure
* Confusing syntax (is it []() or ()[] for links?)
* Syntax that gets in the way (maybe I don't want *stars* emphasized)
* Above point leads to non-standard dialects like GitHub Markdown
(no, GitHub doesn't use markdown)
* Not extensible, leading to even more non-standard hacks and
work-arounds (GitHub Markdown, Pandoc Markdown, other Markdown
libraries have their own incompatible extensions)
* Not well suited for web input (e.g. four-space indentation for code
blocks), although not that important for Haddock
An important thing to note here is that no, Markdown has *not* won
because no one is actually using *Markdown*. They're using their own,
custom and incompatible dialects.
Only two of the above points apply to reStructuredText (web input and
syntax getting in the way), and those particular points don't apply to
Creole. Therefore I tend to advocate Creole for web applications and
reStructuredText for documents.
On Thu, Apr 4, 2013 at 6:49 PM, Johan Tibell
[3]johan.tib...@gmail.com wrote:
Hi all,
Haddock's current markup language leaves something to be desired
once
you want to write more serious documentation (e.g. several
paragraphs
of introductory text at the top of the module doc). Several features
are lacking (bold text, links that render as text instead of URLs,
inline HTML).
I suggest that we implement an alternative haddock syntax that's a
superset of Markdown. It's a superset in the sense that we still
want
to support linkifying Haskell identifiers, etc. Modules that want to
use the new syntax (which will probably be incompatible with the
current syntax) can set:
{-# HADDOCK Markdown #-}
on top of the source file.
Ticket: [4]http://trac.haskell.org/haddock/ticket/244
-- Johan
___
Haskell-Cafe mailing list
[5]Haskell-Cafe@haskell.org
[6]http://www.haskell.org/mailman/listinfo/haskell-cafe
References
1. mailto:dag.odenh...@gmail.com
2. mailto:dag.odenh...@gmail.com
3. mailto:johan.tib...@gmail.com
4. http://trac.haskell.org/haddock/ticket/244
5. mailto:Haskell-Cafe@haskell.org
6. http://www.haskell.org/mailman/listinfo/haskell-cafe
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
___
Re: [Haskell-cafe] File I/O benchmark help (conduit, io-streams and Handle)
On Fri, Mar 8, 2013 at 9:53 AM, Gregory Collins g...@gregorycollins.netwrote: On Fri, Mar 8, 2013 at 9:48 AM, John Lato jwl...@gmail.com wrote: For comparison, on my system I get $ time cp input.dat output.dat real 0m0.004s user 0m0.000s sys 0m0.000s Does your workstation have an SSD? Michael's using a spinning disk. If you're only copying a GB or so, it should only be memory traffic. Alexander -- Gregory Collins g...@gregorycollins.net ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] RFC: rewrite-with-location proposal
Immediately, the alternative of introducing bound variables in the
environment that is available to rewrite rules comes to mind as a more
general way of doing this.
So this example from the GHC docs:
{-# RULES
map/mapforall f g xs. map f (map g xs) = map (f.g) xs
map/append forall f xs ys. map f (xs ++ ys) = map f xs ++ map f ys
#-}
For some source:
map f (map g xs)
it is translated into:
let location = somefile.hs:234
in map (f.g) xs
So for error:
{-# RULES
error/locationerror s = errorLoc location s
#-}
is translated into:
let location = somefile.hs:345
in errorLoc location s
Alexander
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] RFC: rewrite-with-location proposal
On Mon, Feb 25, 2013 at 12:46 PM, Simon Hengel s...@typeful.net wrote:
On Mon, Feb 25, 2013 at 10:40:29AM +0100, Twan van Laarhoven wrote:
I think there is no need to have a separate REWRITE_WITH_LOCATION
rule. What if the compiler instead rewrites 'currentLocation' to the
current location? Then you'd just define the rule:
{-# REWRITE errorLoc error = errorLoc currentLocation #-}
REWRITE rules are only enabled with -O. Source locations are also
useful during development (when you care more about compilation time
than efficient code and hence use -O0). So I'm not sure whether it's a
good idea to lump those two things together.
I could imagine that source locations being useful when debugging rewrite
rules for example.
I think your argument makes sense, but why not fix that specifically?
{-# REWRITE ALWAYS errorLoc error = errorLoc currentLocation #-}
Alexander
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] package dependencies should distinguish between API and implementation?
On Mon, Feb 25, 2013 at 12:50 PM, Johannes Waldmann waldm...@imn.htwk-leipzig.de wrote: Package dependencies are modelled by a relation A depends-on B. Shouldn't this in fact be two relations: API-depends-on and implementation-depends-on? (meaning that A API-depends-on B iff some type of B is visible to the user of A) There's a third relation, A API-ABI-depends-on B iff some type of B is used in the hidden binary representation that is used in backing the API visible to A. Alexander So what we currently have is implementation-depends-on and API-depends-on is a sub-relation of that. The point being the following: assume * A implementation-depends-on B.1 * not (A API-depends-on B.1), * U implementation-depends-on A * U implementation-depends-on B = 2 Then U (indirectly) implementation-depends on two versions of B but it should still be safe? (e.g., I can install B.3, re-compile U, but keep A) Example: A = template-haskell, B = containers (at least I don't see any mention of Data.Map/Set in th's API, I think the only dependency is in the implementation of PprM http://hackage.haskell.org/packages/archive/template-haskell/2.8.0.0/doc/html/Language-Haskell-TH-PprLib.html ) or in general, A (and B.1) are baked into ghc, but there is some B.2/B.3 out there which U wants to use. Or is this what already happens? (ghc would notice that B.1.foo is different from B.2.foo. cabal-install would warn, but proceed? Then the effect of the proposal would just be to switch off these warnings in some cases?) - J.W. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] RFC: rewrite-with-location proposal
My initial thought as I read the proposal was to represent currentLocation as a lexical bound variable, thus error is rewritten to the expression: let currentLocation = someplace.hs:123 in errorLoc currentLocation there is no referntial transparency issue in that because there is no global function currentLocation, it's a lexically bound variable in the rewrite environment. Btw, I'm happy that people want to implement whatever they feel like. Feel free to do whatever makes sense. My comments are not meant to discourage this :-) Alexander On Mon, Feb 25, 2013 at 2:02 PM, Petr Pudlák petr@gmail.com wrote: 2013/2/25 Michael Snoyman mich...@snoyman.com At that point, we've now made two changes to REWRITE rules: 1. They can takes a new ALWAYS parameters. 2. There's a new, special identifier currentLocation available. What would be the advantage is of that approach versus introducing a single new REWRITE_WITH_LOCATION pragma? Just a remark: 'currentLocation' is not a function (it's a special keyword) but behaves like one - it returns some kind of value. But it's not referentially transparent - it returns a different value depending on where it's used. This is something that I really don't expect from Haskell. So having it return `IO Location` seems therefore much better option. And if someone really wants to get the location as a pure value, (s)he can simply wrap it with `unsafePerformIO`, which signals code readers to be careful with that part. Best regards, Petr ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Ticking time bomb
Forgot the list. On Fri, Feb 1, 2013 at 10:21 AM, Alexander Kjeldaas alexander.kjeld...@gmail.com wrote: Trying to avoid the wrath of Ketil I'll refrain from suggesting to do anything, I'll just explain why git is good at this, and not arbitrary. :-) Most systems that I know of to verify *anything*, use merkle trees, or something very similar. http://en.wikipedia.org/wiki/Hash_tree For example the TPM chip on your motherboard, used for example to ensure the integrity of the Google Chromebook and Windows BitLocker http://en.wikipedia.org/wiki/Trusted_Platform_Module (simplified example: in secure memory it stores H1=hash(microcode), then H2=hash(H1 || BIOS), then H3=hash(H2 || MBR), then H4=hash(H3 || kernel), ...). Or the integrity of the bitcoin currency. https://en.bitcoin.it/wiki/Protocol_specification#Merkle_Trees So these are pretty different systems, but it all boils down to doing cryptographic secure hashes over a previous hash + new data to ensure integrity of the new combined data. Given only one verified hash in such a system, no part of the data, nor its history of mutation can be forged. History can mean which software runs on your computer (TPM), which transactions are valid (Bitcoin), or which commits have been done in a SCM (git, mercurial). So git is not magical, it is just a practical implementation of something that works. Any other *general* solution will be based on similar basic principles. Mercurial does this and there is a GPG extension for it. Bazaar does not use a SHA1-based content addressable storage, so while a signed commit signs the tree, it does not represent the history (no hash of hash, only hash if you look at it as a merkle tree), but it does chain commits. To verify a tree + history, *all* commits must be signed, which is fragile IMO. Regarding Darcs, my understanding is that it deliberately rejects hashing the tree, so it is not clear to me how to verify tree+history. Patches can be signed, but as long as patches are independent, there is no hash of hash component which makes it difficult to see how one can verify the tree. My understanding of darcs is very limited though. But to be *practical* the rest of the workflow should be secure as well, so you need: 1. A way to distribute the merkle tree (git pull/clone/push). Distribution is of the data that is to be signed is required for security, because otherwise the representation of the data itself (web view or 'git diff') can be compromised. Signatures have no meaning if you cannot trust that you know what you sign. 2. A way to sign a change to the merkle tre (git commit -S, git tag -s etc) 3. A way to have multiple signatures on a given hash (i.e. commit, or tag, or whatever it is called in a particular merkle tree implementation). This is required to avoid catastrophic owning of core developers. If required, I do think that multiple signatures can be emulated by a structured set of commits that have single signatures though. 3. A way to reliably do code reviews on the changes to the data (git diff) This is really the same as 1). We cannot reliably do 'git diff' unless the developers do it on their own equipment, thus the system must be distributed. 4. Given the requirement for a distributed merkle tree, some merge strategy is needed. It is thus practical, though not required, to have good support for this. (Btw, even the bitcoin hash chain has a merge strategy - the tree with the most compute power will win, and others are forced to rebase their transactions on that tree) So my choice of git is not arbitrary. The way git works is pretty fundamental to verifying the integrity of stuff. Though when I have looked through the other options, mercurial might be a better fit since it is supported on Windows. Trying to solve this problem from scratch might not be such a good idea, because it might be very close to a reimplementation of git or mercurial. Or maybe it is a good idea for someone who has some time on their hands. Just be aware that the requirements for verifying anything is very close to what existing distributed SCM systems do. Alexander On Fri, Feb 1, 2013 at 3:32 AM, Kevin Quick qu...@sparq.org wrote: Git has the ability to solve all of this. ... 2. Uploads to hackage either happen through commits to the git repository, or an old-style upload to hackage automatically creates a new anonymous branch in the git repository. 3. The git repository is authorative. Signing releases, code reviews etc. all happens through the git repositories. This gives us all the flexibility of a git-style trust model. ... 5. Who owns which package names can be held in a separate meta-tree git repository, and can have consensus requirements on commits. 6. This special meta-tree can also contain suggested verification keys for commits to the other hackage git trees. It can even contain keys
Re: [Haskell-cafe] Ticking time bomb
On Thu, Jan 31, 2013 at 9:26 AM, Vincent Hanquez t...@snarc.org wrote: On 01/31/2013 06:27 AM, Ertugrul Söylemez wrote: In any case there is no valid excuse for the lack of crypto. It's too easy to attack Hackage, so we need some crypto regardless of what we interpret it as. My proposal is: 1. Build the necessary machinery into Cabal to allow signing keys and packages and verifying the signatures, ideally through GnuPG. Cabal would benefit from that even without cabal-install and Hackage. Seems there's lots of suggestion of using gnupg, which is a perfectly valid answer if cabal was unix only, but i'm not sure it's a valid option considering windows. Sure you can install gnupg somehow, but sounds to me it's going the same problem as gtk2hs on windows. One better way, would be to tap in the 2, work in progress, gnupg haskell replacement: http://hackage.haskell.org/**package/openpgphttp://hackage.haskell.org/package/openpgp http://hackage.haskell.org/**package/hOpenPGPhttp://hackage.haskell.org/package/hOpenPGP AFAIK, both packages are not yet handling anything related to WoT, but just do the signing/verification (which is same status as my ad-hoc experiment) In this case I think this is the wrong approach. There must be at least one way to work within a trust model that is not fragile. Whether this is fully supported on all platforms is actually not very important. I have pointed out why simply signing packages is fragile and how git is better suited for this task. We are not going to reimplement all the good infrastructure that already exists (gpg, git), so making that a requirement is not a good idea IMO. Basic verification of signatures should work on Windows, I agree. But the underlying WoT should be a little bit more sophisticated. This means it has to be based on standard tools, or it will never happen. Alexander __**_ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/**mailman/listinfo/haskell-cafehttp://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] FFI - Approaches to C/C++
From my experience, these things are needed to get solid (i.e. not flaky software) results. This is not what normal Haskell bindings look like though: 1. Create an interface over the Haskell RTS if you are going to use any of it from C++, and use dependency injection to choose between mock and real implementations. 2. Create a mock implementation of the Haskell side if it is accessed from C++. 3. Create comprehensive C++ only tests (using mock Haskell) that runs cleanly through valgrind. 4. Create as small an interface between C++ and Haskell as possible. 5. If you have a wide API that has little performance implications between Haskell and C++, consider not using the FFI directly, but a higher-level abstraction such as protocol buffers for this part of your API. Alexander On Thu, Jan 31, 2013 at 9:53 AM, Casey Basichis caseybasic...@gmail.comwrote: Hi Ertugrul, Thank you for the detailed reply. From what you wrote, partial FFI still seems like the way to go. Unfortunately Ogre isn't the only large library I'm using, so difficult several times over sounds like a good way to handicap the project early on. I'm perfectly happy to use Haskell for the strengths that will most benefit my project. I can always go back and try to bring the C++ specific parts into the fold once a prototype is up and running. As it seems there is a great deal of c/c++ to do either way, I would really appreciate so thoughts towards my original question. What practices in C++ are preferred by Haskell users, in the know, for the parts of the app that will not be pure Haskell? Should I be looking to avoid OOP? Dependency Injection? I wont reiterate all the facets of the first post, but it would help me immensely to zero in on a few patterns and strategies that can minimized the damage I inflict in c++ land. Thanks, Casey p.s. With That used to be true, but the reason has nothing to do with the language. The problem was that the libraries weren't there. What do you mean? Which packages should I be looking at? I am on iOS like I said, its a stage 1 GHC compiler so I don't have access to GHCI or template haskell. Casey Basichis caseybasichis at gmail.com wrote: I'm not entirely sure what you mean. I'm intending on using Ogre for GUI - for which there is the Hogre bindings, but after emailing the DEV about it, I didn't get the impression from his advice that I should be using it for production code. Here is what he suggested: It depends, really. Hogre is good for running Ogre from within Haskell, but it has its limitations. The number one thing people have been struggling with is handling input with hogre - there's Hois (Haskell wrapper for OIS) but it's not perfect (it misses input events), and the other option is having to write some C++ glue. Hogre is a solid proof of concept and you can do some demos with it, but if you're e.g. writing a game it might be a bit of a struggle. In the end it's about how much you value being able to write code in Haskell (or how allergic to C++ you are). I'm on iOS so I imagine those difficulties are compounded. I am using several other C++ libraries for which there are no existing bindings and no Haskell alternative packages that are even remotely close. Are you suggesting it would be better to write all my own FFI bindings for all the needed libraries? That's not what I'm suggesting. It was just too little information to properly judge the difficulty of doing everything in Haskell. Binding to Ogre (or C++ in general) is indeed difficult. If Hogre doesn't work or is too limited, your best option might be to write a C wrapper around the Hogre functionality you need. Another option is to use SDL/OpenGL directly, which may be easier or harder depending on your application. However, if you can build the bridge between your rendering library and Haskell, then Haskell is certainly the better choice. Everything I read suggests that Haskells strengths are in transformation and that interaction is not its strong suit. I am interested in your thoughts and I am open to whatever, but you are the first to suggest that the mix is a bad idea. That used to be true, but the reason has nothing to do with the language. The problem was that the libraries weren't there. Nowadays you can write all sorts of interactive applications in Haskell, including GUIs, TUIs, games, simulations and web applications. However, I've long been waiting for useful bindings to Ogre or Irrlicht, but I'm afraid that it's not going to happen any time soon. Ultimately it's your choice. Let me summarize the possiblities: * C wrapper around Ogre. Easy integration, but need to write the rendering code in C/C++. * Full FFI bindings to Ogre. Difficult integration, but you can write your rendering code in Haskell. * Partial FFI bindings to Ogre. Integration may be somewhat easy,
Re: [Haskell-cafe] Ticking time bomb
On Thu, Jan 31, 2013 at 11:48 AM, Vincent Hanquez t...@snarc.org wrote: On 01/31/2013 08:54 AM, Alexander Kjeldaas wrote: On Thu, Jan 31, 2013 at 9:26 AM, Vincent Hanquez t...@snarc.org wrote: On 01/31/2013 06:27 AM, Ertugrul Söylemez wrote: In any case there is no valid excuse for the lack of crypto. It's too easy to attack Hackage, so we need some crypto regardless of what we interpret it as. My proposal is: 1. Build the necessary machinery into Cabal to allow signing keys and packages and verifying the signatures, ideally through GnuPG. Cabal would benefit from that even without cabal-install and Hackage. Seems there's lots of suggestion of using gnupg, which is a perfectly valid answer if cabal was unix only, but i'm not sure it's a valid option considering windows. Sure you can install gnupg somehow, but sounds to me it's going the same problem as gtk2hs on windows. One better way, would be to tap in the 2, work in progress, gnupg haskell replacement: http://hackage.haskell.org/package/openpgphttp://hackage.haskell.org/**package/openpgp http://**hackage.haskell.org/package/**openpgphttp://hackage.haskell.org/package/openpgp http://hackage.haskell.org/package/hOpenPGPhttp://hackage.haskell.org/**package/hOpenPGP http://**hackage.haskell.org/package/**hOpenPGPhttp://hackage.haskell.org/package/hOpenPGP AFAIK, both packages are not yet handling anything related to WoT, but just do the signing/verification (which is same status as my ad-hoc experiment) In this case I think this is the wrong approach. There must be at least one way to work within a trust model that is not fragile. Whether this is fully supported on all platforms is actually not very important. I have pointed out why simply signing packages is fragile and how git is better suited for this task. We are not going to reimplement all the good infrastructure that already exists (gpg, git), so making that a requirement is not a good idea IMO. Basic verification of signatures should work on Windows, I agree. But the underlying WoT should be a little bit more sophisticated. This means it has to be based on standard tools, or it will never happen. I think you misunderstood me. Having a fully working pgp package, means you have full control of the pgp stack, you don't rely on hard-to-get out tools, and it can be integrated with cabal directly for a full WoT experience. Also git doesn't solve the hackage problem, there's not necessarily a one-to-one mapping between packages and their repositories. Regarding git, I don't see anyone even suggesting that hackage should support multiple 'signed-off-by's, signing diffs between packages, and signing releases. I don't see anyone supporting 3rd-parties signing packages at will. All of this is significant engineering effort that just won't happen. And even if it does, it still doesn't solve the problem, because all of the above is required to protect our base ghc, cabal-install tools anyways. If cabal-install is hacked, hackage is hacked. And even when all of this is done, a design where there is a set of trusted developers is flawed. The set of trusted developers will increase over time, and there will be the need to restrict which developers can release which package. How do you protect that information, which keys can sign which packages? That information must itself be signed off by someone. If not, ouch some obscure developer in the trusted set hijacked an important package because he was owned. That signing in itself *requires* either consensus or a trusted third party. Consensus is a superior method and IMO required. Actually, all modifications to important packages requires consensus by at least two developers to avoid the threat posed by owned developers. So although it is not a waste of time to secure hackage, it is always going to be slightly fragile as long as ghc is fragile for example, and it just isn't ever going to *really* solve the problem. Git has the ability to solve all of this. So a better use of resources IMO is to skip securing hackage by own means, but to standardize on using git as the infrastructure for securing hackage. Then the same infrastructure that should secure ghc should secure hackage. To explain this in concrete terms, to do this, all that is needed is *something like* the following: 1. Each package on hackage gets its own repository (say on github or similar). (I've actually put all of hackage in a single git repository - it works fine, so there are no relevant size problems) 2. Uploads to hackage either happen through commits to the git repository, or an old-style upload to hackage automatically creates a new anonymous branch in the git repository. 3. The git repository is authorative. Signing releases, code reviews etc. all happens through the git repositories. This gives us all the flexibility of a git-style trust model. 4. If a package
Re: [Haskell-cafe] Ticking time bomb
On Thu, Jan 31, 2013 at 11:40 AM, Vincent Hanquez t...@snarc.org wrote: On 01/31/2013 10:06 AM, Ertugrul Söylemez wrote: Joachim Breitner m...@joachim-breitner.de wrote: And that may even be more harmful, because an insecure system with a false sense of security is worse than an insecure system alone. Let's do it properly. but don’t overengineer it either. Simply adding to hackage the possibility to store a .asc file next to the tar.gz file that contains the cryptographic signature would be a great start, and allow us to develop a WoT model later on. (I try to resist from wondering whether this could go into hackage1 or only hackage2, and in the latter case, whether that means that we actually have the time to overengineer the system.) In fact, a lot would already be gained by a simple „warn if foo-2.0 is signed with a different key than the version of foo already installed“ on cabal-install and people having a closer look at uploads from different people. Not much infrastructure needed there. That was exactly my suggestion actually. It requires the ability to make and check signatures. The making can be done with external tools like GnuPG, but the checking has to be done by cabal-install. To detect changed keys there also needs to be a trust database, which can be a simple directory in ~/.cabal/ where files are named after the fingerprint of the key it contains. The most important part is a sensible user interface. The whole process should be invisible to the user, until there is a signature error. The first installation of a package will actually generate a handful of signature errors, because the keys are not known yet. This shouldn't be too hard to implement and requires only a small change to Hackage and cabal-install's upload command to begin. That's not a proper solution, and definitively in the warm fuzzy feeling department. What if you install a package for the first time and this package has just been re-uploaded maliciously with a different key and a payload ? What if you're relying on hackage mirrors, what stop this mirror to regenerate all signatures with a new key ? It also make maintainers change difficult, and doing genuine non-maintainer upload. It is still useful to protect most users and detect malicious code really quickly. Someone will always have the package installed and will notice, so the intruder will only be able to do very targeted attacks. Basically this creates at chain of trust for each package. What I think you want is a chain of trust between packages. Alexander -- Vincent __**_ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/**mailman/listinfo/haskell-cafehttp://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Ticking time bomb
On Thu, Jan 31, 2013 at 12:53 PM, Ketil Malde ke...@malde.org wrote: Ertugrul Söylemez e...@ertes.de writes: And that may even be more harmful, because an insecure system with a false sense of security is worse than an insecure system alone. Yes. As is clear to all, the current low level of security means that nobody are _actually_ downloading stuff of Hackage, thank God. Hackage just exists for...well, I forget, but certainly not to distribute software. Right. Sarcasm aside, to some extent, this is true. I used to have a cron job 'cabal install'ing my packages off Hackage to ensure that they would compile with the current offering of their dependencies. But I decided it was way too risky, and don't do it anymore. Let's do it properly. You mean like how it was decisively dealt with when this was discussed in 2008? https://github.com/haskell/cabal/issues/207 Or maybe more the way it was firmly handled when it was brought up again in 2010? http://www.haskell.org/pipermail/haskell-cafe/2010-December/087050.html This looks increasingly like that time of year when the problem is pointed out, the crypto geeks get together to construct the Optimal Solution, and then everybody lose interest and move on to greener pastures for a while. Well, I don't think the perfect solution exists, and even if it could be identified, it might not be implemented, and even if were implemented, it might not be used. Hehe.. hard to argue against history! :-) Alexander We've just been incredibly lucky that nothing really bad has happened so far. Let's hope it lasts. -k -- If I haven't seen further, it is by standing in the footprints of giants ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Ticking time bomb
Not to downplay the significance of this issue, but a primary issue, much more important is to secure ghc, base, cabal-install, and the build process for these. The development process needs to be robust. That process should include signing commits by *two developers*. This is really not a lot of work as a code-review is already done, but more significantly helps fend off a lot of compromised repository issues. There are just a few simple rules to follow: Review the commit on your own equipment, and sign it. That way an attacker will have to compromise two physically different repositories. .http://mikegerwitz.com/docs/git-horror-story.html This is a change that doesn't need any new development, just a procedure change. Wrt Hackage, simply signing packages is going to improve things, but what would be way more awesome would be to have multiple people sign off on the difference between two releases. What I mean is that whenever somebody reviews a new release of some package (the diff from a previous release), the work they did should ideally be represented as a signature on that release, or the commit that is the diff between the two releases. Git can handle this sort of trust issues, but a simple signature scheme will not. Now if a large security-conscious corporation starts using packages from Hackage, they will already have a security team that does code reviews. With a simple signature scheme, the output from their security team will not be possible to use on Hackage. I for one would be much more likely to trust packages signed by the signature by someone who has found numerous security flaws in various packages on Hackage, than some random developer. One signature might mean this is my release, if you trust my machine,repository, +++. Another signature might mean the security team of X Corp has done a security review at level Y. This is cleared for production use.. Simply supporting the first signature is no good. So if we really want to fix this, I suggest moving a large part of the trust infrastructure off of hackage and into git or similar systems that have better support for dealing with trust. Alexander On Wed, Jan 30, 2013 at 8:27 PM, Edward Z. Yang ezy...@mit.edu wrote: https://status.heroku.com/incidents/489 Unsigned Hackage packages are a ticking time bomb. Cheers, Edward ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ANN: monad-bool 0.1
If we step back, I think the lesson here is that Haskell libraries exist, but the concepts are far enough from what you expect to exist given background knowledge from another programming language. So what is actually needed is not monad-bool, but the equivalent documentation that makes a programmer which is not an expert in Haskell connect the dots like you did today. Alexander On Wed, Jan 23, 2013 at 8:10 AM, John Wiegley jo...@fpcomplete.com wrote: John Wiegley jo...@fpcomplete.com writes: Never mind, when/unless + left work just fine for this. You know, it's been a humorous day. First ekmett showed that I can't make a sane Monad instance for AndM or OrM. Then I discovered I can't make a reasonable Monoid (no mempty, given only 'toBool'), so I dropped down to a Semigroup. Further, my combinators for EitherT can be implemented using just 'when' and 'left' to provide the short-circuiting. Already I had very little code left, until he showed me the Applicative instance for Either, plus a little trick: Right 1 * Right 2 * Left 2 * Right 5 Left 2 -- same functionality as my And semigroup let Left x | y = y; x | _ = x in Left 1 | Right 2 | Right 3 Right 2-- same functionality as my Or semigroup And poof, all my code just disappeared... -- John Wiegley FP Complete Haskell tools, training and consulting http://fpcomplete.com johnw on #haskell/irc.freenode.net ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] forkProcess, forkIO, and multithreaded runtime
I think you can test this theory with this patch. If a thread is waiting
on the task-cond condition variable which is matched up with task-lock,
then pthread_cond_destroy will return EBUSY, which must always be a bug in
the RTS.
Alexander
diff --git a/rts/posix/OSThreads.c b/rts/posix/OSThreads.c
index ae31966..0f12830 100644
--- a/rts/posix/OSThreads.c
+++ b/rts/posix/OSThreads.c
@@ -91,7 +91,8 @@ initCondition( Condition* pCond )
void
closeCondition( Condition* pCond )
{
- pthread_cond_destroy(pCond);
+ int ret = pthread_cond_destroy(pCond);
+ CHECKM(ret == 0, RTS BUG! Someone is waiting on condvar %d., ret);
return;
}
On Mon, Jan 21, 2013 at 8:18 AM, Alexander Kjeldaas
alexander.kjeld...@gmail.com wrote:
I just looked at this code and since I don't know the code I can't give
you good solutions, but for others watching this thread the links might
prove interesting.
My main theory is that you do have some other thread in FFI-land while you
are fork()ing. The task-cond, task-lock seems to be related to this (see
quoted comments below).
Also, pthread_mutex_destroy is undefined if the lock is locked, so I am
guessing that the task-lock is somehow locked when it shouldn't be.
It isn't clear from your description whether this is consistently
happening on Linux, or whether this only sometimes happens.
The forkProcess() code seems to hold all capabilities during fork, but
that does not include FFI-land threads AFAIU.
Assuming that this happens only rarely, I am trying to understand what
happens if the thread that is in FFI-land returns to the RTS (in the
parent) after fork(), but before the freeTask() in the child. Based on the
descriptions I read, it seems likely that this thread will try to inspect
task-cap, which requires holding task-lock.
That would in turn make the pthread_mutex_destroy in the child invalid.
https://github.com/ghc/ghc/blob/master/rts/Task.h#L57
...
When a task is migrated from sleeping on one Capability to another,
its task-cap field must be modified. When the task wakes up, it
will read the new value of task-cap to find out which Capability
it belongs to. Hence some synchronisation is required on
task-cap, and this is why we have task-lock.
If the Task is not currently owned by task-id, then the thread is
either
(a) waiting on the condition task-cond. The Task is either
(1) a bound Task, the TSO will be on a queue somewhere
(2) a worker task, on the spare_workers queue of task-cap.
...
freeTask:
https://github.com/ghc/ghc/blob/master/rts/Task.c#L142
the comment in freeTask refers to this test:
https://github.com/ghc/testsuite/blob/master/tests/concurrent/should_run/conc059.hs
That test calls the RTC from C which then forkIOs off actions that are
outstanding when the RTS exits.
in forkProcess, child code
https://github.com/ghc/ghc/blob/master/rts/Schedule.c#L1837
It look like all this code supports the notion that some other thread can
be in foreign code during the fork call.
discardTasksExcept
https://github.com/ghc/ghc/blob/master/rts/Task.c#L305
Alexander
On Mon, Jan 21, 2013 at 12:15 AM, Mark Lentczner mark.lentcz...@gmail.com
wrote:
Sorry to be reviving this thread so long after but I seem to be
running into similar issues as Michael S. did at the start.
In short, I'm using forkProcess with the threaded RTS, and see occasional
hangs:
- I see these only on Linux. On Mac OS X, I never do.
- I'm using GHC 7.4.2
- I noticed the warning in the doc for forkProcess, but assumed I was
safe, as I wasn't holding any shared resources at the time of the fork,
and
no shared resources in the program are used in the child.
- WIth gdb, I've traced the hang to here in the run-time: forkProcess
discardTasksExcept freeTask closeMutex(task-lock)
pthread_mutex_destroy
The discussion in this thread leaves me with these questions:
- Is there reason to think the situation has gotten better in 7.6 and
later?
- Isn't the only reason *System.Process* is safer because it does an
immediate exec in the child? Alas, I really want to just fork()sometimes.
- Is it really true that even if my program has no shared resources
with the child, that the IO subsystem and FFI system do anyway? Surely the
RTS would take care of doing the right thing with those, no?
- There should be no concern with exec w.r.t. library invariants
since exec is wholesale replacement - all the libraries will
reinitialize. Is there a problem here I'm missing?
Alas, I've stopped using the threaded RTS until I understand this better.
- Mark
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] forkProcess, forkIO, and multithreaded runtime
Or this. It seems that you must compile with DEBUG for the mutex check.
This enables error-checking mutexes on posix.
Alexander
diff --git a/rts/posix/OSThreads.c b/rts/posix/OSThreads.c
index ae31966..e07221d 100644
--- a/rts/posix/OSThreads.c
+++ b/rts/posix/OSThreads.c
@@ -91,7 +91,8 @@ initCondition( Condition* pCond )
void
closeCondition( Condition* pCond )
{
- pthread_cond_destroy(pCond);
+ int ret = pthread_cond_destroy(pCond);
+ CHECKM(ret == 0, RTS Bug! Someone is waiting on condvar ret=%d., ret);
return;
}
@@ -165,7 +166,8 @@ initMutex(Mutex* pMut)
void
closeMutex(Mutex* pMut)
{
-pthread_mutex_destroy(pMut);
+int ret = pthread_mutex_destroy(pMut);
+CHECKM(ret == 0, RTS Bug! Destroying held mutex ret=%d, ret);
}
void
On Mon, Jan 21, 2013 at 10:14 AM, Alexander Kjeldaas
alexander.kjeld...@gmail.com wrote:
I think you can test this theory with this patch. If a thread is waiting
on the task-cond condition variable which is matched up with task-lock,
then pthread_cond_destroy will return EBUSY, which must always be a bug in
the RTS.
Alexander
diff --git a/rts/posix/OSThreads.c b/rts/posix/OSThreads.c
index ae31966..0f12830 100644
--- a/rts/posix/OSThreads.c
+++ b/rts/posix/OSThreads.c
@@ -91,7 +91,8 @@ initCondition( Condition* pCond )
void
closeCondition( Condition* pCond )
{
- pthread_cond_destroy(pCond);
+ int ret = pthread_cond_destroy(pCond);
+ CHECKM(ret == 0, RTS BUG! Someone is waiting on condvar %d., ret);
return;
}
On Mon, Jan 21, 2013 at 8:18 AM, Alexander Kjeldaas
alexander.kjeld...@gmail.com wrote:
I just looked at this code and since I don't know the code I can't give
you good solutions, but for others watching this thread the links might
prove interesting.
My main theory is that you do have some other thread in FFI-land while
you are fork()ing. The task-cond, task-lock seems to be related to this
(see quoted comments below).
Also, pthread_mutex_destroy is undefined if the lock is locked, so I am
guessing that the task-lock is somehow locked when it shouldn't be.
It isn't clear from your description whether this is consistently
happening on Linux, or whether this only sometimes happens.
The forkProcess() code seems to hold all capabilities during fork, but
that does not include FFI-land threads AFAIU.
Assuming that this happens only rarely, I am trying to understand what
happens if the thread that is in FFI-land returns to the RTS (in the
parent) after fork(), but before the freeTask() in the child. Based on the
descriptions I read, it seems likely that this thread will try to inspect
task-cap, which requires holding task-lock.
That would in turn make the pthread_mutex_destroy in the child invalid.
https://github.com/ghc/ghc/blob/master/rts/Task.h#L57
...
When a task is migrated from sleeping on one Capability to another,
its task-cap field must be modified. When the task wakes up, it
will read the new value of task-cap to find out which Capability
it belongs to. Hence some synchronisation is required on
task-cap, and this is why we have task-lock.
If the Task is not currently owned by task-id, then the thread is
either
(a) waiting on the condition task-cond. The Task is either
(1) a bound Task, the TSO will be on a queue somewhere
(2) a worker task, on the spare_workers queue of task-cap.
...
freeTask:
https://github.com/ghc/ghc/blob/master/rts/Task.c#L142
the comment in freeTask refers to this test:
https://github.com/ghc/testsuite/blob/master/tests/concurrent/should_run/conc059.hs
That test calls the RTC from C which then forkIOs off actions that are
outstanding when the RTS exits.
in forkProcess, child code
https://github.com/ghc/ghc/blob/master/rts/Schedule.c#L1837
It look like all this code supports the notion that some other thread can
be in foreign code during the fork call.
discardTasksExcept
https://github.com/ghc/ghc/blob/master/rts/Task.c#L305
Alexander
On Mon, Jan 21, 2013 at 12:15 AM, Mark Lentczner
mark.lentcz...@gmail.com wrote:
Sorry to be reviving this thread so long after but I seem to be
running into similar issues as Michael S. did at the start.
In short, I'm using forkProcess with the threaded RTS, and see
occasional hangs:
- I see these only on Linux. On Mac OS X, I never do.
- I'm using GHC 7.4.2
- I noticed the warning in the doc for forkProcess, but assumed I
was safe, as I wasn't holding any shared resources at the time of the
fork,
and no shared resources in the program are used in the child.
- WIth gdb, I've traced the hang to here in the run-time: forkProcess
discardTasksExcept freeTask closeMutex(task-lock)
pthread_mutex_destroy
The discussion in this thread leaves me with these questions:
- Is there reason to think the situation has gotten better in 7.6
and later?
- Isn't the only reason *System.Process
Re: [Haskell-cafe] forkProcess, forkIO, and multithreaded runtime
While trying to dig around this morning I started adding clang-style thread
locking annotations to the source code. These can be very handy and I
found at least one place where the documented locking policy doesn't seem
to match what is happening.
Here is an example with annotations, and what might or might not be a bug.
With these annotations, clang will be able to prove whether the program
obeys the locking regime or not.
But this is of course only one part of the RTS, but the locking can be
pretty interesting in itself.
Does anyone else feel that this sort of annotations would help?
Alexander
diff --git a/rts/Task.c b/rts/Task.c
index e6781a1..1e499dc 100644
--- a/rts/Task.c
+++ b/rts/Task.c
@@ -25,12 +25,12 @@
// Task lists and global counters.
// Locks required: all_tasks_mutex.
-Task *all_tasks = NULL;
+Task *all_tasks GUARDED_BY(all_tasks_mutex) = NULL;
-nat taskCount;
-nat workerCount;
-nat currentWorkerCount;
-nat peakWorkerCount;
+nat taskCount GUARDED_BY(all_tasks_mutex);
+nat workerCount GUARDED_BY(all_tasks_mutex);
+nat currentWorkerCount GUARDED_BY(all_tasks_mutex);
+nat peakWorkerCount GUARDED_BY(all_tasks_mutex);
static int tasksInitialized = 0;
@@ -339,9 +339,11 @@ void updateCapabilityRefs (void)
ACQUIRE_LOCK(all_tasks_mutex);
for (task = all_tasks; task != NULL; task=task-all_next) {
+ACQUIRE_LOCK(task-lock);
if (task-cap != NULL) {
task-cap = capabilities[task-cap-no];
}
+RELEASE_LOCK(task-lock);
for (incall = task-incall; incall != NULL; incall =
incall-prev_stack) {
if (incall-suspended_cap != NULL) {
On Mon, Jan 21, 2013 at 2:14 PM, Chris Dornan ch...@chrisdornan.com wrote:
I am also sorry to be late on this but I have run into the same problem
trying to demonise a programme on 7.4.2. My solution was to get a shell
wrapper to run the daemon in debug mode (I.e., sans fork) and get the shell
script to do the demonising.
Other than this I have found the threaded RTS to be sound and I am quite
reliant on it. So, where things that run —threaded are concerned, no
forkProcess calls for me until I can better understand this better.
If anybody does think they understand what is going on here it would be
great if they could write it up. IMHO, either the current notes on
forkProcess don't go far enough, or there is a bug needing a workaround
until the platform gets fixed.
Chris
From: Mark Lentczner mark.lentcz...@gmail.com
Date: Sunday, 20 January 2013 23:15
To: haskell haskell-cafe@haskell.org
Cc: Mike Meyer m...@mired.org
Subject: Re: [Haskell-cafe] forkProcess, forkIO, and multithreaded runtime
Sorry to be reviving this thread so long after but I seem to be
running into similar issues as Michael S. did at the start.
In short, I'm using forkProcess with the threaded RTS, and see occasional
hangs:
- I see these only on Linux. On Mac OS X, I never do.
- I'm using GHC 7.4.2
- I noticed the warning in the doc for forkProcess, but assumed I was
safe, as I wasn't holding any shared resources at the time of the fork, and
no shared resources in the program are used in the child.
- WIth gdb, I've traced the hang to here in the run-time: forkProcess
discardTasksExcept freeTask closeMutex(task-lock)
pthread_mutex_destroy
The discussion in this thread leaves me with these questions:
- Is there reason to think the situation has gotten better in 7.6 and
later?
- Isn't the only reason *System.Process* is safer because it does an
immediate exec in the child? Alas, I really want to just fork()sometimes.
- Is it really true that even if my program has no shared resources
with the child, that the IO subsystem and FFI system do anyway? Surely the
RTS would take care of doing the right thing with those, no?
- There should be no concern with exec w.r.t. library invariants since
exec is wholesale replacement - all the libraries will reinitialize.
Is there a problem here I'm missing?
Alas, I've stopped using the threaded RTS until I understand this better.
- Mark
___ Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] [ANN] tls-extra 0.6.1 - security update, please upgrade.
On Sun, Jan 20, 2013 at 6:50 AM, Vincent Hanquez t...@snarc.org wrote: Hi cafe, this is a security advisory for tls-extra 0.6.1 which are all vulnerable to bad certificate validation. Some part of the certificate validation procedure were missing (relying on the work-in-progress x509 v3 extensions), and because of this anyone with a correct end-entity certificate can issue certificate for any arbitrary domain, i.e. acting as a CA. This problem has been fixed in tls-extra 0.6.1, and I advise everyone to upgrade as soon as possible. Despite a very serious flaw in the certificate validation, I'm happy that the code is seeing some audits, and would want to thanks Ertugrul Söylemez for the findings [1]. [1] https://github.com/vincenthz/hs-tls/issues/29 Regarding testing, it looks like the Tests directory hasn't been updated to cover this bug. What would really give confidence is a set of tests encoding fixed security vulnerabilities in OpenSSL (and similar libraries). That should also give you a lot of confidence in your library. But anyways, this is fantastic work you're doing. Keep it up! Alexander -- Vincent ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] forkProcess, forkIO, and multithreaded runtime
I just looked at this code and since I don't know the code I can't give you good solutions, but for others watching this thread the links might prove interesting. My main theory is that you do have some other thread in FFI-land while you are fork()ing. The task-cond, task-lock seems to be related to this (see quoted comments below). Also, pthread_mutex_destroy is undefined if the lock is locked, so I am guessing that the task-lock is somehow locked when it shouldn't be. It isn't clear from your description whether this is consistently happening on Linux, or whether this only sometimes happens. The forkProcess() code seems to hold all capabilities during fork, but that does not include FFI-land threads AFAIU. Assuming that this happens only rarely, I am trying to understand what happens if the thread that is in FFI-land returns to the RTS (in the parent) after fork(), but before the freeTask() in the child. Based on the descriptions I read, it seems likely that this thread will try to inspect task-cap, which requires holding task-lock. That would in turn make the pthread_mutex_destroy in the child invalid. https://github.com/ghc/ghc/blob/master/rts/Task.h#L57 ... When a task is migrated from sleeping on one Capability to another, its task-cap field must be modified. When the task wakes up, it will read the new value of task-cap to find out which Capability it belongs to. Hence some synchronisation is required on task-cap, and this is why we have task-lock. If the Task is not currently owned by task-id, then the thread is either (a) waiting on the condition task-cond. The Task is either (1) a bound Task, the TSO will be on a queue somewhere (2) a worker task, on the spare_workers queue of task-cap. ... freeTask: https://github.com/ghc/ghc/blob/master/rts/Task.c#L142 the comment in freeTask refers to this test: https://github.com/ghc/testsuite/blob/master/tests/concurrent/should_run/conc059.hs That test calls the RTC from C which then forkIOs off actions that are outstanding when the RTS exits. in forkProcess, child code https://github.com/ghc/ghc/blob/master/rts/Schedule.c#L1837 It look like all this code supports the notion that some other thread can be in foreign code during the fork call. discardTasksExcept https://github.com/ghc/ghc/blob/master/rts/Task.c#L305 Alexander On Mon, Jan 21, 2013 at 12:15 AM, Mark Lentczner mark.lentcz...@gmail.comwrote: Sorry to be reviving this thread so long after but I seem to be running into similar issues as Michael S. did at the start. In short, I'm using forkProcess with the threaded RTS, and see occasional hangs: - I see these only on Linux. On Mac OS X, I never do. - I'm using GHC 7.4.2 - I noticed the warning in the doc for forkProcess, but assumed I was safe, as I wasn't holding any shared resources at the time of the fork, and no shared resources in the program are used in the child. - WIth gdb, I've traced the hang to here in the run-time: forkProcess discardTasksExcept freeTask closeMutex(task-lock) pthread_mutex_destroy The discussion in this thread leaves me with these questions: - Is there reason to think the situation has gotten better in 7.6 and later? - Isn't the only reason *System.Process* is safer because it does an immediate exec in the child? Alas, I really want to just fork()sometimes. - Is it really true that even if my program has no shared resources with the child, that the IO subsystem and FFI system do anyway? Surely the RTS would take care of doing the right thing with those, no? - There should be no concern with exec w.r.t. library invariants since exec is wholesale replacement - all the libraries will reinitialize. Is there a problem here I'm missing? Alas, I've stopped using the threaded RTS until I understand this better. - Mark ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] forkProcess, forkIO, and multithreaded runtime
On Mon, Jan 21, 2013 at 12:15 AM, Mark Lentczner mark.lentcz...@gmail.comwrote: Sorry to be reviving this thread so long after but I seem to be running into similar issues as Michael S. did at the start. In short, I'm using forkProcess with the threaded RTS, and see occasional hangs: - I see these only on Linux. On Mac OS X, I never do. Previous versions of the linux pthreads library didn't hold any shared resources in locks, so pthread_mutex_destroy could not hang. Now Linux is much improved, and thus it hangs (see pthread_mutex_destroy man page) ;-). - I'm using GHC 7.4.2 - I noticed the warning in the doc for forkProcess, but assumed I was safe, as I wasn't holding any shared resources at the time of the fork, and no shared resources in the program are used in the child. - WIth gdb, I've traced the hang to here in the run-time: forkProcess discardTasksExcept freeTask closeMutex(task-lock) pthread_mutex_destroy The discussion in this thread leaves me with these questions: - Is there reason to think the situation has gotten better in 7.6 and later? - Isn't the only reason *System.Process* is safer because it does an immediate exec in the child? Alas, I really want to just fork()sometimes. If you immediately do exec() in the child, you can use vfork() which blocks the parent. This serializes the actions and makes this whole mess smooth and consistent. - Is it really true that even if my program has no shared resources with the child, that the IO subsystem and FFI system do anyway? Surely the RTS would take care of doing the right thing with those, no? It looks like the RTS is trying to do a lot of things to control all the Haskell threads etc. But I don't think it waits for FFI-land threads before commencing a fork(), so that's why I'm guessing that some interaction between threads using FFI and fork() could be the issue. - There should be no concern with exec w.r.t. library invariants since exec is wholesale replacement - all the libraries will reinitialize. Is there a problem here I'm missing? I think that's right. vfork() + exec() can be serialized and deterministic thus is a lot easier to reason about. Alexander ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] debugging memory corruption
What I've mostly done in similar circumstances (jni) 1. Create an interface (virtual functions or template) for the FFI in C++ that covers everything you use. Then create one test implementation and one real implementation. The test implementation must allocate resources whenever the real FFI does so. Doing memory allocation works. This makes it possible to test all your FFI in C++ using valgrind. 2. Add tracing support to the real implementation and replay support to the test implementation. 3. Upload to Hackage. Alexander On Dec 1, 2012 5:06 PM, Evan Laforge qdun...@gmail.com wrote: Ever since upgrading to 7.6.1 I regularly get panics like this: seq: internal error: evacuate: strange closure type -1958168540 (GHC version 7.6.1 for x86_64_apple_darwin) Please report this as a GHC bug: http://www.haskell.org/ghc/reportabug I've seen some variations, but basically I think it just means someone is corrupting memory and it goes unnoticed until the GC trips over it. This happens infrequently (maybe once in 15m, very roughly, it's extremely inconsistent) in non-optimized code, and frequently (maybe once in 5m) in optimized code. This only happens during interactive use, not during testing or profiling. I had a problem like this once before, and it took a very long time to track down. And in fact, I never really tracked it down, I just got a test that could semi-reliably reproduce it, and then by trial and error discovered that if I changed the alignment of a particular Storable instance from 1 to 4, the problem stopped happening (and 1 should have been correct, it was a struct of chars). Not exactly a satisfying solution, and now I'm thinking all I did was make ghc 6 stop manifesting it, and with 7 it's back again. I'm most suspicious of the FFI usage since it's easy to corrupt memory in C++ and even easier to write a bogus Storable instance that does the same, but I really have no idea what or where. My first thought was to start cutting out portions (especially FFI-using portions) to try to isolate it, but it's slow going because it can sometimes take quite a while to the bug to come out again. My second thought was that I need a more automated way to reproduce it, but it's nontrivial because it only comes up when I'm using the interactive GUI parts, which are also incidentally a big chunk of FFI. And even if I do get a repro, as I did before, it just means I can more quickly poke randomly at things hoping they change it, but even if I can get it to stop happening it doesn't mean I understood it, or even really fixed it. This is also the kind of bug (well, it was last time), which is highly dependent on the code, so add one print and it stops happening. I have a sort of complicated scheme where I pass a wrapped haskell function callback along with a wrapped freeHaskellFunPtr to free the last one, along with itself, maybe it's something to do with that. Anyone out there with ideas or advice on how to track down this kind of bug? My next thought is to try to automate the GUI parts, or maybe just the FFI part, so I can write a program to randomly fuss with it until it crashes. I've also tried valgrind, but it doesn't report anything suspicious. But it also doesn't seem to work on FFI Storable corruption, I've tried intentionally inserting a bad poke and valgrind still won't report it. Thanks in advance for any insight! Actually, there's a whole other discussion which has been nagging at me for a while, though another thread would be more appropriate. But in short it's that it feels like hsc2hs is just too low level, and too error-prone. It's tempting to use because it comes with ghc, but it seems bad to tell people haskell is a safe language, but as soon as you want to talk to C you're writing totally unchecked pokes and peeks. Maybe I should go evaluate the alternatives like c2hs, or maybe safety features can added to hsc2hs. Wouldn't it be nice to have ghc come with a high level and safe FFI language? ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Can a GC delay TCP connection formation?
Jeff, this is somewhat off topic, but interesting. Are telehouse and AWS physically close? Was this latency increase not expected due to geography? Alexander On 28 November 2012 06:21, Neil Davies semanticphilosop...@gmail.comwrote: Jeff Are you certain that all the delay can be laid at the GHC runtime? How much of the end-to-end delay budget is being allocated to you? I recently moved a static website from a 10-year old server in telehouse into AWS in Ireland and watched the access time (HTTP GET to check time on top index page) increase by 150ms. Neil On 27 Nov 2012, at 19:02, Jeff Shaw shawj...@gmail.com wrote: Hello Timothy and others, One of my clients hosts their HTTP clients in an Amazon cloud, so even when they turn on persistent HTTP connections, they use many connections. Usually they only end up sending one HTTP request per TCP connection. My specific problem is that they want a response in 120 ms or so, and at times they are unable to complete a TCP connection in that amount of time. I'm looking at on the order of 100 TCP connections per second, and on the order of 1000 HTTP requests per second (other clients do benefit from persistent HTTP connections). Once each minute, a thread of my program updates a global state, stored in an IORef, and updated with atomicModifyIORef', based on query results via HDBC-obdc. The query results are strict, and atomicModifyIORef' should receive the updated state already evaluated. I reduced the amount of time that query took from tens of seconds to just a couple, and for some reason that reduced the proportion of TCP timeouts drastically. The approximate before and after TCP timeout proportions are 15% and 5%. I'm not sure why this reduction in timeouts resulted from the query time improving, but this discovery has me on the task of removing all database code from the main program and into a cron job. My best guess is that HDBC-odbc somehow disrupts other communications while it waits for the DB server to respond. To respond to Ertugrul, I'm compiling with -threaded, and running with +RTS -N. I hope this helps describe my problem. I c an probably come up with some hard information if requested, E.G. threadscope. Jeff On 11/27/2012 10:55 AM, timothyho...@seznam.cz wrote: Could you give us more info on what your constraints are? Is it necessary that you have a certain number of connections per second, or is it necessary that the connection results very quickly after some other message is received? ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] forkProcess, forkIO, and multithreaded runtime
On 17 October 2012 00:17, Mike Meyer m...@mired.org wrote: On Tue, 16 Oct 2012 21:55:44 +0200 Alexander Kjeldaas alexander.kjeld...@gmail.com wrote: There are variants of this, but the meta-problem is that at the point in time when you call forkProcess, you must control all threads, ensuring that *all invariants hold*. Thus no locks held, no thread is in the C library, no foreign calls active etc. As an example, if one thread is in the C library doing some stdio, then the invariants in that library will not hold, and you cannot expect stdio to work in the child. This means that the only thing you can really do in the child process is call exec. Further, you can only call exec if you make sure that the exec correctly reverts everything back to a state where those invariants hold. Mostly, this is automatic as resources get freed on exec and do the right thing. Locks on file descriptors that aren't closed on exec will leave dangling locks, and locks on files that are closed on exec will unexpectedly close them in the parent. Right. It should be renamed mostlyUnsafeForkProcess, assuming the multi-threaded RTS is mostly the default one. Alexander mike -- Mike Meyer m...@mired.org http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O ascii ribbon campaign - stop html mail - www.asciiribbon.org ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] forkProcess, forkIO, and multithreaded runtime
On 15 October 2012 09:47, Michael Snoyman mich...@snoyman.com wrote: Hi all, I think I have a misunderstanding of how forkProcess should be working. Ultimately this relates to some bugs in the development version of keter, but I've found some behavior in a simple test program which I wouldn't have expected either, which may or may not be related. With the program at the end of this email, I would expect that, once per second, I would get a message printed from each forkIO'd green thread, the forked process, and the master process. And if I spawn 8 or less child threads that's precisely what happens. However, as soon as I up that number to 9, the child process is never run. The process is, however, created, as can be confirmed by looking at the process table. This only occurs when using the multithreaded runtime. In other words, compiling with ghc --make test.hs seems to always produce the expected output, whereas ghc --make -threaded test.hs causes the behavior described above. Having looked through the code for the process package a bit, my initial guess is that this is being caused by a signal being sent to the child process, but I'm not familiar enough with the inner workings to confirm or disprove this guess. If anyone has any ideas on this, I'd appreciate it. Not being familiar with the implementation, I'll just note that combining fork() with threads, as in the multithreaded RTS is going to have subtle bugs, no matter what. The only winning strategy is to not play that game. There are variants of this, but the meta-problem is that at the point in time when you call forkProcess, you must control all threads, ensuring that *all invariants hold*. Thus no locks held, no thread is in the C library, no foreign calls active etc. As an example, if one thread is in the C library doing some stdio, then the invariants in that library will not hold, and you cannot expect stdio to work in the child. This means that the only thing you can really do in the child process is call exec. These issues do not exist in the non-threaded world. Alexander Michael import System.Posix.Process (forkProcess, getProcessID) import Control.Concurrent (forkIO, threadDelay) import System.IO (hFlush, stdout) import System.Posix.Signals (signalProcess, sigKILL) import Control.Exception (finally) main :: IO () main = do mapM_ spawnChild [1..9] child - forkProcess $ do putStrLn starting child hFlush stdout loop child 0 print (child pid, child) hFlush stdout -- I've commented out the finally so that the zombie process stays alive, -- to prove that it was actually created. loop parent 0 -- `finally` signalProcess sigKILL child spawnChild :: Int - IO () spawnChild i = do _ - forkIO $ loop (spawnChild ++ show i) 0 return () loop :: String - Int - IO () loop msg i = do pid - getProcessID print (pid, msg, i) hFlush stdout threadDelay 100 loop msg (i + 1) ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Build regressions due to GHC 7.6
I think you're making this way harder than it really is.
What 99% of people need is that hackage packages builds with the latest
haskell platform, and/or with bleeding edge ghc, and with the latest
versions of its dependencies.
Thus for every dependency there is only one possible version - the latest
one, and there are only a couple of compilers.
Having special interest groups for ghc 6.12 support and old versions of
text is fine, but I think it is a pretty uninteresting problem to solve.
Likewise, supporting/fixing packages where the author for some reason
*requires* use of a non-current version of some other package is also quite
uninteresting (or at least outside the scope of my needs). Such a package
is basically just a relic.
Alexander
On 30 August 2012 22:26, Jay Sulzberger j...@panix.com wrote:
On Thu, 30 Aug 2012, Alexander Kjeldaas alexander.kjeld...@gmail.com
wrote:
This is very unfortunate, but this is crucially a tooling issue. I am
going to wave my hands, but..
Ignore the mapreduce in the following video, but look at the use of clang
to do automatic refactoring of C++. This is *incredibly* powerful in
dealing with updates to APIs.
http://www.llvm.org/devmtg/**2011-11/videos/Carruth_**
ClangMapReduce-desktop.mp4http://www.llvm.org/devmtg/2011-11/videos/Carruth_ClangMapReduce-desktop.mp4
But without all that fancy tech, *just* having all of Hackage source code
in one repository and using perl/regexps, fixing these types of issues is
O(1) instead of O(n).
All of the issues you mention seems to be fixable by a few lines of perl
*if we had the repository*.
Better to do this with sexps.
ad repositories: Part of the general problem of managing a
repository is close to the problem of inferring a good type for
(the value of) an expression. The style of constraints is
similar. Now the design problem is:
1. Arrange a general system for the specification of the
constraints.
2. Design a systematic way of giving both advice and direct
commands to the system. This subsystem would be used to set
up the default policy.
3. Choose a constraint solver.
Maybe worth looking at:
http://en.wikipedia.org/wiki/**Nix_package_managerhttp://en.wikipedia.org/wiki/Nix_package_manager
[page was last modified on 17 July 2012 at 20:20]
oo--JS.
[a few hours later]
Actually, I went and downloaded all of hackage, put it into a git
repository and fixed these issues:
Fix catch
perl -ni -e 'print unless /import Prelude hiding \(catch\)/' $(git grep
'import Prelude hiding (catch)')
Fix CInt constructors (lots of other stuff from Foreign.C.Types not fixed
though)
perl -p -i -e 's/^import Foreign.C.Types(.*)CInt([^(])/**import
Foreign.C.Types${1}CInt(..)${**1}/g' $(git grep -l '^import.*CInt')
Fix bytestring versioning
perl -p -i -e 's/bytestring( +)=([0-9. ]+)([
]*)0.10/bytestring$1=$2${3}**0.11/g' $(git grep 'bytestring.* *0\.')
Patch to hackage:
http://ge.tt/6Cb5ErM/v/0
I understand that this doesn't help anyone, but if there was a way fix,
upload, and get *consensus* on a few regexps like this, then doing API
changes wouldn't be such a headache.
Alexander
On 30 August 2012 07:26, Bryan O'Sullivan b...@serpentine.com wrote:
Since the release of the GHC 7.6 RC, I've been going through my packages
and fixing up build problems so that people who upgrade to 7.6 will have
a
smooth ride.
Sad to say, my experience of 7.6 is that it has felt like a particularly
rough release for backwards incompatibility. I wanted to quantify the
pain,
so I did some research, and here's what I found.
I maintain 25 open source Haskell packages. Of these, the majority have
needed updates due to the GHC 7.6 release:
- base16-bytestring
- blaze-textual
- bloomfilter
- configurator
- criterion
- double-conversion
- filemanip
- HDBC-mysql
- mwc-random
- pcap
- pool
- riak-haskell-client
- snappy
- text
- text-format
- text-icu
That's 16 out of 25 packages I've had to update. I've also either
reported
bugs on, or had to fix, several other people's packages along the way
(maybe four?). So let's say I've run into problems with 20 out of the
combined 29 packages of mine and my upstreams.
The reasons for these problems fall into three bins:
- Prelude no longer exports catch, so a lot of import Prelude hiding
(catch) had to change.
- The FFI now requires constructors to be visible, so CInt has to be
imported as CInt(..).
- bytestring finally got bumped to 0.10, so many upper bounds had to
be relaxed (*cf* my suggestion that the upper-bounds-by-default policy
is destructive).
It has been a lot of work to test 29 packages, and then modify, rebuild,
and release 20 of them. It has consumed most of my limited free time for
almost two weeks. Worse, this has felt like make-work, of no practical
benefit to anyone beyond scrambling to restore
Re: [Haskell-cafe] Build regressions due to GHC 7.6
This is very unfortunate, but this is crucially a tooling issue. I am
going to wave my hands, but..
Ignore the mapreduce in the following video, but look at the use of clang
to do automatic refactoring of C++. This is *incredibly* powerful in
dealing with updates to APIs.
http://www.llvm.org/devmtg/2011-11/videos/Carruth_ClangMapReduce-desktop.mp4
But without all that fancy tech, *just* having all of Hackage source code
in one repository and using perl/regexps, fixing these types of issues is
O(1) instead of O(n).
All of the issues you mention seems to be fixable by a few lines of perl
*if we had the repository*.
[a few hours later]
Actually, I went and downloaded all of hackage, put it into a git
repository and fixed these issues:
Fix catch
perl -ni -e 'print unless /import Prelude hiding \(catch\)/' $(git grep
'import Prelude hiding (catch)')
Fix CInt constructors (lots of other stuff from Foreign.C.Types not fixed
though)
perl -p -i -e 's/^import Foreign.C.Types(.*)CInt([^(])/import
Foreign.C.Types${1}CInt(..)${1}/g' $(git grep -l '^import.*CInt')
Fix bytestring versioning
perl -p -i -e 's/bytestring( +)=([0-9. ]+)([
]*)0.10/bytestring$1=$2${3}0.11/g' $(git grep 'bytestring.* *0\.')
Patch to hackage:
http://ge.tt/6Cb5ErM/v/0
I understand that this doesn't help anyone, but if there was a way fix,
upload, and get *consensus* on a few regexps like this, then doing API
changes wouldn't be such a headache.
Alexander
On 30 August 2012 07:26, Bryan O'Sullivan b...@serpentine.com wrote:
Since the release of the GHC 7.6 RC, I've been going through my packages
and fixing up build problems so that people who upgrade to 7.6 will have a
smooth ride.
Sad to say, my experience of 7.6 is that it has felt like a particularly
rough release for backwards incompatibility. I wanted to quantify the pain,
so I did some research, and here's what I found.
I maintain 25 open source Haskell packages. Of these, the majority have
needed updates due to the GHC 7.6 release:
- base16-bytestring
- blaze-textual
- bloomfilter
- configurator
- criterion
- double-conversion
- filemanip
- HDBC-mysql
- mwc-random
- pcap
- pool
- riak-haskell-client
- snappy
- text
- text-format
- text-icu
That's 16 out of 25 packages I've had to update. I've also either reported
bugs on, or had to fix, several other people's packages along the way
(maybe four?). So let's say I've run into problems with 20 out of the
combined 29 packages of mine and my upstreams.
The reasons for these problems fall into three bins:
- Prelude no longer exports catch, so a lot of import Prelude hiding
(catch) had to change.
- The FFI now requires constructors to be visible, so CInt has to be
imported as CInt(..).
- bytestring finally got bumped to 0.10, so many upper bounds had to
be relaxed (*cf* my suggestion that the upper-bounds-by-default policy
is destructive).
It has been a lot of work to test 29 packages, and then modify, rebuild,
and release 20 of them. It has consumed most of my limited free time for
almost two weeks. Worse, this has felt like make-work, of no practical
benefit to anyone beyond scrambling to restore the status quo ante.
If over half of my packages needed fixing, I'm alarmed at the thought of
the effects on the rest of Hackage.
I'm torn over this. I understand and agree with the impetus to improve the
platform by tidying things up, and yet just two seemingly innocuous changes
(catch and FFI) have forced me to do a bunch of running to stand still.
I don't have any suggestions about what to do; I know that it's hard to
estimate the downstream effects of what look like small changes. And so I'm
not exactly complaining. Call this an unhappy data point.
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Build regressions due to GHC 7.6
On 30 August 2012 15:34, Alexander Bernauer alex-hask...@copton.net wrote: Hi I agree that automatic code migration can solve this issue in large parts. The Python folks have done this to mitigate the transition from version 2 to version 3 [1]. On Thu, Aug 30, 2012 at 03:03:05PM +0200, Alexander Kjeldaas wrote: perl -ni -e 'print unless /import Prelude hiding \(catch\)/' $(git grep 'import Prelude hiding (catch)') I don't think regular expressions are powerful enough. This example does not match on hiding multiple names, for instance. This was just an example, the CInt regexp handles multiple names, so regexps have no problems handling this. And it's a simple git grep 'import.*Prelude.*catch' to see if this actually is a problem or not. My point is that this works, fixes 99% of the cases, and is 1000x less work overall. Alexander But writing proper 'HsModule - HsModule' functions should be doable. And when each release comes with a bunch of such functions, packages could be automatically migrated. Greetings Alex [1] http://docs.python.org/library/2to3.html -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) iEYEARECAAYFAlA/a/oACgkQevm6Dd/q44nJXQCffaxEJ/NZEftgoZ7viAWMuBO3 +jkAnRTw+VCMQn1k9NibyKpkGMtwvrQw =ds3M -END PGP SIGNATURE- ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] createProcess running non-existent programs
On 13 August 2012 23:49, Richard O'Keefe o...@cs.otago.ac.nz wrote: On 13/08/2012, at 11:26 PM, Alexander Kjeldaas wrote: This isn't that hard - a pipe shouldn't be needed anymore. Just require a post-2003 glibc. fexecve is a system call in most BSDs. It is also implemented in glibc using a /proc hack. fexecve is now in the Single Unix Specification, based on POSIX as of 2008, I believe. However, http://www.gnu.org/software/gnulib/manual/html_node/fexecve.html says Portability problems not fixed by Gnulib: * This function is missing on many non-glibc platforms: MacOS X 10.5, FreeBSD 6.0, NetBSD 5.0, OpenBSD 3.8, Minix 3.1.8, AIX 5.1, HP-UX 11, IRIX 6.5, OSF/1 5.1, Solaris 11 2010-11, Cygwin 1.5.x, mingw, MSVC 9, Interix 3.5, BeOS. That warning doesn't seem to be fully up to date. I'm using MacOS X 10.6.8 and fexecve() isn't in the manuals or in unistd.h. FreeBSD 8.0 is covered. OpenBSD not covered OS X not covered http://developer.apple.com/library/mac/#documentation/Darwin/Reference/ManPages/man2/execve.2.html Solaris probably not covered. So support is pretty good, I'd say. For non-modern systems, checking the existence of the file first is possible. The race isn't important, and one can always upgrade to a modern operating system. Alexander ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] createProcess running non-existent programs
On 14 August 2012 17:22, Niklas Larsson metanik...@gmail.com wrote: 2012/8/14 Alexander Kjeldaas alexander.kjeld...@gmail.com: On 13 August 2012 23:49, Richard O'Keefe o...@cs.otago.ac.nz wrote: On 13/08/2012, at 11:26 PM, Alexander Kjeldaas wrote: This isn't that hard - a pipe shouldn't be needed anymore. Just require a post-2003 glibc. fexecve is a system call in most BSDs. It is also implemented in glibc using a /proc hack. fexecve is now in the Single Unix Specification, based on POSIX as of 2008, I believe. However, http://www.gnu.org/software/gnulib/manual/html_node/fexecve.html says Portability problems not fixed by Gnulib: * This function is missing on many non-glibc platforms: MacOS X 10.5, FreeBSD 6.0, NetBSD 5.0, OpenBSD 3.8, Minix 3.1.8, AIX 5.1, HP-UX 11, IRIX 6.5, OSF/1 5.1, Solaris 11 2010-11, Cygwin 1.5.x, mingw, MSVC 9, Interix 3.5, BeOS. That warning doesn't seem to be fully up to date. I'm using MacOS X 10.6.8 and fexecve() isn't in the manuals or in unistd.h. FreeBSD 8.0 is covered. OpenBSD not covered OS X not covered http://developer.apple.com/library/mac/#documentation/Darwin/Reference/ManPages/man2/execve.2.html Solaris probably not covered. So support is pretty good, I'd say. For non-modern systems, checking the existence of the file first is possible. The race isn't important, and one can always upgrade to a modern operating system. The check would be unreliable, the file's existence doesn't imply that it's executable. See access(2) Furthermore it would add unnecessary overhead, createProcess can be run thousands of times in a program and should be lean and mean. Just to keep the bikeshedding doing, I'm going to state as a fact that running performance sensitive *server* workload on any unix other than Linux is purely of theoretical interest. No sane person would do it. Therefore, from a performance overhead, Linux performance is the only important performance measure. But even given the above, the overhead we're talking about is minuscule. A program like '/bin/echo -n ''' which does exactly *nothing*, requires 35(!) system calls to do its job :-). A more complex program like 'id' requires 250 system calls! Also, to see just how minuscule this is, the dynamic linker, ld-linux.so does a few extra access(2) system calls *to the same file*, /etc/ld.so.hwcaps, on startup of every dynamically linked executable. 2 in the 'echo' case, and 8 in the 'id' case above. Even the glibc folks haven't bothered to optimize those syscalls away. Alexander ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] createProcess running non-existent programs
This isn't that hard - a pipe shouldn't be needed anymore. Just require a post-2003 glibc. fexecve is a system call in most BSDs. It is also implemented in glibc using a /proc hack. http://www.kernel.org/doc/man-pages/online/pages/man3/fexecve.3.html Apparently, there are proposals/RFCs to get a system called execveat into the linux kernel which makes this work properly without /proc. http://www.gossamer-threads.com/lists/linux/kernel/1574831 Alexander On 13 August 2012 11:23, David Feuer david.fe...@gmail.com wrote: In Unix, at least, check, then act is generally considered unwise: 1. Something can go wrong between checking and acting. 2. You might not be checking the right thing(s). In this case, the fact that the file exists is not useful if you don't have permission to execute it. You may not be able to determine whether you have the appropriate permissions without fairly deep manipulation of ACLs. 3. Even if the OS has the info at hand, making the system call(s) necessary to get it is not free. Various non-free things happen every time control passes between user-space and kernel-space. On Aug 13, 2012 4:17 AM, Andrew Cowie and...@operationaldynamics.com wrote: On Sun, 2012-08-12 at 23:18 -0700, Evan Laforge wrote: Yes, I ran into the same thing a while back. The problem is that the subprocess has already been forked off before it runs exec() and finds out the file doesn't exist. Given how astonishingly common it is to pass an invalid executable name and/or path, wouldn't it be worth doing a quick probe to see if the file exists before createProcess actually forks? [It's not like the effort the OS is going to do for the stat is going to be thrown away; whether that call pulls it up off of disk or the one after the fork that exec will do doesn't matter] AfC Sydney ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Exceeding OS limits for simultaneous socket connections
On 30 January 2012 14:22, Rob Stewart robstewar...@gmail.com wrote: Hi, I'm experiencing the accept: resource exhausted (Too many open files) exception when trying to use sockets in my Haskell program. The situation: - Around a dozen Linux machines running my Haskell program, transmitting thousands of messages to each other, sometimes within a small period of time. ... $ ulimit -n 1024 This is not an OS limit, this is your freely chosen limit. You should not run with this few file descriptors on a server. Increasing this by 50x is entirely reasonable. However, having too many open TCP connections is not a good thing either. 1024 was an upper limit way way back on the i386 linux architecture for code using the select() system call, that is why it is still a common default. There are a few ways to get out of this situation. 1. Reuse your TCP connections. Maybe you could even use HTTP. An HTTP library might do reusing of connections for you. 2. Since you are blocking in getContents, there is a probability that it is the senders that are being lazy in sendAll. They opened the TCP connection, but now they are not sending everything in sendAll, so your receiver is having lots of threads that are blocked on reading. Try to be strict when *sending* so you do not have too many ongoing TCP connections. 3. On the receiver side, to be robust, you could limit the number of threads that are allowed to do an accept() to the number of file descriptors you have free. You can also block on a semaphore whenever accept returns out of resources, and signal that semaphore after every close. Alexander Indeed, when I experience the accept: resource exhausted (Too many open files) exception, I check the number of open sockets, which exceeds 1024, by looking at the contents of the directory: ls -lah /proc/prod_id/fd It is within the getContents function that, once the lazy bytestring is fully received, the socket is shutdown http://goo.gl/B6XcV : shutdown sock ShutdownReceive There seems to be no way of limiting the number of permitted connection requests from remote nodes. What I am perhaps looking for is a mailbox implementation on top of sockets, or another way to avoid this error. I am looking to scale up to 100's of nodes, where the possibility of more than 1024 simultaneous socket connections to one node is increased. Merely increasing the ulimit feels like a temporary measure. Part of the dilemma is that the `connect' call in `sendMsg' does not throw an error, despite the fact that it does indeed cause an error on the receiving node, by pushing the number of open connections to the same socket on the master node, beyond the 1024 limit permitted by the OS. Am I missing something? One would have thought such a problem occurs frequently with Haskell web servers and the like.. ? -- Rob Stewart ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Cabal issue
Not what you asked for, but.. Although the documentation looks somewhat cryptic, and I have not use it at all, a nix + cabal option seems to be in the works. Search google for nix haskell. I am guessing that their Hydra continuous build system which is built on top of nix also could do wonders for hackage. Alexander On 9 December 2011 12:20, Kevin Jardine kevinjard...@gmail.com wrote: I understand that this may have been addressed before on this list in some form, so I'll be brief: Had problem with deprecated package, was told my only option was to wipe my Haskell install and start over. Is this true and if so, doesn't this mean that Cabal (or the package management system that it is a part of) is broken? Details here: https://plus.google.com/u/0/111705054912446689620/posts/V1186HGWEap Kevin ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Hackage feature request: E-mail author when a package breaks
On 31 October 2011 17:22, Yitzchak Gale g...@sefer.org wrote: Gregory Crosswhite wrote: could [Hackage] have a feature where when a working package breaks with a new version of GHC the author is automatically e-mailed? This would be nice. However, there would have to be a way for it to be turned on and off by the author. (Spam is not nice.) How about sending an email to haskell-package-packate-name@haskell.org, and then people can join that mailing list if they are interested in that sort of stuff? Mailman is good at doing subscribe and unsubscribe. Alexander ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] SMP parallelism increasing GC time dramatically
I am guessing that it is slowdown caused by GC needing to co-ordinate with blocked threads. That requires lots of re-scheduling to happen in the kernel. This is a hard problem I think, but also increasingly important as virtualization becomes more important and the number of schedulable cores unknown. Alexander On 7 October 2011 12:31, Oliver Batchelor saul...@gmail.com wrote: I'm not sure if this is at all related, but if I run a small Repa program with more threads than I have cores/CPUs then it gets drastically slower, I have a dual core laptop - and -N2 makes my small program take approximately 0.6 of the time. Increasing to -N4 and we're running about 2x the time, -N8 and it's taking 20x or more. I guess this is probably more down to the design of Repa rather than GHC itself? Oliver On Sat, Oct 8, 2011 at 1:21 AM, Tom Thorne thomas.thorn...@gmail.comwrote: I have made a dummy program that seems to exhibit the same GC slowdown behavior, minus the segmentation faults. Compiling with -threaded and running with -N12 I get very bad performance (3x slower than -N1), running with -N12 -qg it runs approximately 3 times faster than -N1. I don't know if I should submit this as a bug or not? I'd certainly like to know why this is happening! import Numeric.LinearAlgebra import Numeric.GSL.Special.Gamma import Control.Parallel.Strategies import Control.Monad import Data.IORef import Data.Random import Data.Random.Source.PureMT import Debug.Trace -- subsets s n = (subsets_stream s) !! n subsets_stream [] = [[]] : repeat [] subsets_stream (x:xs) = let r = subsets_stream xs s = map (map (x:)) r in [[]] : zipWith (++) s (tail r) testfun :: Matrix Double - Int - [Int] - Double testfun x k cs = lngamma (det (c+u)) where (m,c) = meanCov xx m' = fromRows [m] u = (trans m') m' xx = fromColumns ( [(toColumns x)!!i] ++ [(toColumns x)!!j] ++ [(toColumns x)!!k] ) i = cs !! 0 j = cs !! 1 test :: Matrix Double - Int - Double test x i = sum p where p = parMap (rdeepseq) (testfun x (i+1)) (subsets [0..i] 2) ranMatrix :: Int - RVar (Matrix Double) ranMatrix n = do xs - mapM (\_ - mapM (\_ - uniform 0 1.0) [1..n]) [1..n] return (fromLists xs) loop :: Int - Double - Int - RVar Double loop n s i = traceShow i $ do x - ranMatrix n let r = sum $ parMap (rdeepseq) (test x) [2..(n-2)] return (r+s) main = do let n = 100 let iter = 5 rng - newPureMT rngr - newIORef rng p - runRVar (foldM (loop n) 0.0 [1..iter]) rngr print p I have also found that the segmentation faults in my code disappear if I switch from Control.Parallel to Control.Monad.Par, which is quite strange. I get slightly better performance with Control.Parallel when it completes without a seg. fault, and the frequency with which it does so seems to depend on the number of sparks that are being created. On Thu, Oct 6, 2011 at 1:56 PM, Tom Thorne thomas.thorn...@gmail.comwrote: I'm trying to narrow it down so that I can submit a meaningful bug report, and it seems to be something to do with switching off parallel GC using -qg, whilst also passing -Nx. Are there any known issues with this that people are aware of? At the moment I am using the latest haskell platform release on arch. I'd like to give 7.2 a try in case that fixes it, but I'm using rather a lot of libraries (hmatrix, fclabels, random fu) and I don't know how to install them for multiple ghc versions On Wed, Oct 5, 2011 at 10:43 PM, Johan Tibell johan.tib...@gmail.comwrote: On Wed, Oct 5, 2011 at 2:37 PM, Tom Thorne thomas.thorn...@gmail.comwrote: The only problem is that now I am getting random occasional segmentation faults that I was not been getting before, and once got a message saying: Main: schedule: re-entered unsafely Perhaps a 'foreign import unsafe' should be 'safe'? I think this may be something to do with creating a lot of sparks though, since this occurs whether I have the parallel GC on or not. Unless you (or some library you're using) is doing what the error message says then you should file a GHC bug here: http://hackage.haskell.org/trac/ghc/ -- Johan ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Enumeratees are killing me
I am waiting for a web service where I can enter the type I want to process, the iterator package to use, and which will spit out the types and an example. Then life will be good. Alexander On 16 September 2011 22:46, tsuraan tsur...@gmail.com wrote: Well, I got it working. It seems to behave sanely when put in a chain with other Enumeratee/Iteratee combinations, so I'm happy. My solution is posted on hpaste: http://hpaste.org/51430 . I'd love any criticism that anybody has, especially since it seems to me to be an ideal building block for the sorts of Enumeratees that I'm building. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] bitSize
On 27 August 2011 21:57, Brandon Allbery allber...@gmail.com wrote: On Sat, Aug 27, 2011 at 06:57, Andrew Coppin andrewcop...@btinternet.comwrote: On 26/08/2011 10:51 PM, Steve Schafer wrote: On Fri, 26 Aug 2011 20:30:02 +0100, you wrote: You wouldn't want to know how many bits you need to store on disk to reliably recreate the value? I can't say that I have cared about that sort of thing in a very long time. Bits are rather cheap these days. I store data on disk, and the space it occupies is whatever it is; I don't worry about it. I meant if you're trying to *implement* serialisation. The Bits class allows you to access bits one by one, but surely you'd want some way to know how many bits you need to keep? I think that falls into the realm of protocol design; if you're doing it in your program at runtime, you're probably doing it wrong. (The fixed size version makes sense for marshaling; it's *dynamic* sizes that need to be thought out beforehand.) All search engines deal with compressed integers, all compressors do, and most people doing bit-manipulation. Golomb, gamma, elias, rice coding, they all need this. Heck, even the Intel engineers chose to optimize this function by including the BSR instruction in the 386 architecture. This is a basic building block. Don't underestimate the bit, it is coming back with a vengeance. Bit-coding is everywhere now, because of the memory hierarchy. No haskeller should be left behind. Alexander ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Replacing stdin from within Haskell
On 10 June 2011 06:43, Richard O'Keefe o...@cs.otago.ac.nz wrote: On 10/06/2011, at 1:11 AM, Erik Hesselink wrote: On Thu, Jun 9, 2011 at 13:40, Neil Davies semanticphilosop...@gmail.com wrote: Anyone out there got an elegant solution to being able to fork a haskell thread and replace its 'stdin' ? If you don't mind being tied to GHC you can use hDuplicateTo from GHC.IO.Handle [1]. You can also use dupTo from the unix package [2], but that ties you to unix-like platforms instead. Erik [1] http://hackage.haskell.org/packages/archive/base/latest/doc/html/GHC-IO-Handle.html#v:hDuplicateTo That replaces stdin for *every* Haskell thread. [2] http://hackage.haskell.org/packages/archive/unix/latest/doc/html/System-Posix-IO.html#v:dupTo That alters file descriptor 0 for *every* Haskell thread. As I understand the original poster's question, it would be paraphrased as can the association between the default input stream used by getChar and the rest be changed for ONE thread in a Haskell program by some method that does NOT have any effect on any other thread in the same OS process. Or to put it another way, do I really have to pass a handle around to all these functions in my thread? And the answer would seem to be there is no reason in principle why this *couldn't* be supported, but it *isn't*; there are no C-style thread-local global variables. Yes, you do have to pass that handle around more than you would like to. Could implicit parameters be used for this purpose? http://www.haskell.org/ghc/docs/7.0.3/html/users_guide/other-type-extensions.html Alexander ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ANNOUNCE: iterIO-0.1 - iteratee-based IO with pipe operators
On 16 May 2011 21:31, dm-list-haskell-c...@scs.stanford.edu wrote:
At Mon, 16 May 2011 10:56:02 +0100,
Simon Marlow wrote:
Yes, it's not actually documented as far as I know, and we should fix
that. But if you think about it, sequential consistency is really the
only sensible policy: suppose one processor creates a heap object and
writes a reference to it in the IORef, then another processor reads the
IORef. The writes that created the heap object must be visible to the
second processor, otherwise it will encounter uninitialised memory and
crash. So sequential consistency is necessary to ensure concurrent
programs can't crash.
Now perhaps it's possible to have a relaxed memory model that provides
the no-crashes guarantee but still allows IORef writes to be reordered
(e.g. some kind of causal consistency). That might be important if
there is some processor arcitecture that provides that memory model, but
as far as I know there isn't.
Actually, in your heap object example, it sounds like you only really
care about preserving program order, rather than write atomicity.
Thus, you can get away with less-than-sequential consistency and not
crash.
The x86 is an example of a relaxed memory model that provides the
no-crashes guarantee you are talking about. Specifically, the x86
deviates from sequential consistency in two ways
1. A load can finish before an earlier store to a different memory
location. [intel, Sec. 8.2.3.4]
2. A thread can read its own writes early. [intel, 8.2.3.5]
[Section references are to the intel architecture manual, vol 3a:
http://www.intel.com/Assets/PDF/manual/253668.pdf]
One could imagine an implementation of IORefs that relies on the fact
that pointer writes are atomic and that program order is preserved to
avoid mutex overhead for most calls. E.g.:
struct IORef {
spinlock_t lock; /* Only ever used by atomicModifyIORef */
HaskellValue *val; /* Updated atomically because pointer-sized
writes are atomic */
};
HaskellValue *
readIORef (struct IORef *ref)
{
return ref-val;
}
void
writeIORef (struct IORef *ref, HaskellValue *val)
{
/* Note that if *val was initialized in the same thread, then by
* the time another CPU sees ref-val, it will also see the
* correct contents of *ref-val, because stores are seen in a
* consistent order by other processors [intel, Sec. 8.2.3.7].
*
* If *val was initialized in a different thread, then since this
* thread has seen it, other threads will too, because x86
* guarantees stores are transitively visible [intel, Sec. 8.2.3.6].
*/
ref-val = val;
}
/* modifyIORef is built out of readIORef and writeIORef */
HaskellValue *
atomicModifyIORef (Struct IORef *ref, HaskellFunction *f)
{
HaskellValue *result;
spinlock_acquire (ref-lock);
result = modifyIORef (ref, f);
spinlock_release (ref-lock);
return result;
}
This is actually how I assumed IORefs worked. But then consider the
following program:
maybePrint :: IORef Bool - IORef Bool - IO ()
maybePrint myRef yourRef = do
writeIORef myRef True
yourVal - readIORef yourRef
unless yourVal $ putStrLn critical section
main :: IO ()
main = do
r1 - newIORef False
r2 - newIORef False
forkOS $ maybePrint r1 r2
forkOS $ maybePrint r2 r1
threadDelay 100
Under sequential consistency, the string critical section should be
output at most once. However, with the above IORef implementation on
x86, since a read can finish before a write to a different location,
both threads might see False for the value of yourVal.
To prevent this deviation from sequential consistency, you would need
to do something like stick an MFENCE instruction at the end of
writeIORef, and that would slow down the common case where you don't
care about sequential consistency. In fact, I would argue that if you
care about S.C., you should either be using atomicModifyIORef or
MVars.
mfence is apparently slower than lock add. see
http://blogs.oracle.com/dave/entry/instruction_selection_for_volatile_fences
so using mfence would make it slower than atomicModifyIORef, and with weaker
guarantees. not a good combination.
Alexander
Can you explain what actually happens inside the real IORef
implementation?
As an aside, these days one sees a lot of hand-wringing over the fact
that CPU clock rates have been flat for a while and the only way to
get more performance is through parallelism. How are we going to
teach programmers to write concurrent code when it's so hard to write
and debug? I've heard numerous people ask.
Haskell could be a major step in the right direction, since in the
absence of variables, it's impossible to have data races. (You can
still have deadlock and other kinds of race condition, such as the one
in maybePrint above, if you had my definition of
[Haskell-cafe] Ur vs Haskell
I've briefly gone through the Ur demo at http://impredicative.com/ur/demo/ Ur looks very impressive, so the natural question I'm asking myself is: How does it stack up against haskell frameworks, and why can't Ur be implemented in Haskell? I'm thinking mainly of the safety guarantees, not necessarily the performance guarantees, GC-less execution, or even non-lazy evaluation. And I can't answer those.. any takers? Alexander ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Type Directed Name Resolution
In most imperative languages understanding x.name requires knowledge of
the type of x to understand what name refers to.
Now with TDNR in Haskell, name x requires knowledge of the type of x to
understand what name refers to.
As a newcomer, I think some of the coding conventions favored by
haskell-coders to be write-only, but in this case I wonder why this is
less readable in Haskell than in, say C?
Alexander
On 10 November 2010 19:05, Albert Y. C. Lai tre...@vex.net wrote:
Typed-directed name resolution brings Haskell closer to a write-only
language; that is, an ambiguous phrase made total sense to the author when
the author wrote it, but an independent reader will need extraordinary
effort to disambiguate.
{-# LANGUAGE TypeDirectedNameResolution #-}
import EnglishMonad
import Cities(buffalo)
import Animals(buffalo)
import Verbs(buffalo,buffalo)
{- why two buffalo's from Verbs? because they are of different types: one
is present verb type and the other is past participle verb type. ever heard
of Type Directed Name Resolution? -}
buffalo = buffalo buffalo buffalo buffalo buffalo buffalo buffalo buffalo
main = runEnglishMonad buffalo
{-
http://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffalo_buffalo_buffalo_Buffalo_buffalo-}
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] Current thinking on CompositionAsDot issue in haskell prime?
Hi haskellers. Reading through the Haskell Prime suggestions, one that caught my eye is the CompositionAsDot issue. I'm especially thinking of the Pro issue: * Paves the way for using . as the selection operator in improved record or module systems I think I recognize this issue from common lisp. Basically the code becomes verbose because accessor functions usually need to redundantly encode the name of the module or struct in its name. The alternative is to require users to import qualified, and that doesn't deal with the occasional intra-module collision. This is a typical large-software-system issue and it annoys me a little that Haskell seems to have inherited the problems from common lisp. I worry that in large systems, a very conservative coding standard is needed with quite verbose naming conventions as that's the only safe way to avoid these conflicts in a large system while allowing refactoring code etc. What is the current thinking on this issue? Thanks, Alexander ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
